[feedback-report] ARF

[Steve Atkins]

On Jul 12, 2008, at 4:45 PM, Jenny Gavin-Wear wrote:

> Hi all,
>
> I have been researching ARF in relation to the AOL FBL.
>
> My situation is that I am a web site developer (Coldfusion) offering  
> hosting and email services to my customers.  Some of those customers  
> are using mail lists.
>
> My problem is that AOL users incorrectly mark mail list emails as  
> spam instead of unsubscribing, despite all of the options I have  
> made available - such as one click unsubscribe.
>
> So, I need to be able to find the email address of the complainant  
> from the AOL ARF report.

You need to embed the information needed to unsubscribe the recipient  
in the email you send out. The usual way to do that is to embed some  
sort of magic cookie which will allow you to identify both the email  
address the mail was sent to and the list the subscription was  
associated with (ideally you'd also be able to identify the specific  
mailing used). This is supported by every decent mailing list manager  
(not least because you need the same data to handle bounces reliably).

>
> I am running two mail servers, (Mdaemon and Smartermail) on Windows  
> 2003 Server Standard.
>
> On researching a solution for the Windows platform I find nothing.   
> Worse still, these is very little information on the Net about the  
> ARF format and on how I might be able to parse it.

If you embed a magic cookie in the mail you send out then it's trivial  
to parse - you don't need to do anything more than pull the message/ 
rfc822 section out of the mail and look for the cookie. 999 times out  
of 1000 you don't even need to parse the MIME at all, you can just  
treat the ARF report as a block of text and look for the magic cookie  
in that block of text.

If you don't include a magic cookie of some sort, and a way of mapping  
from that cookie back to the original email address, you're making a  
lot of unneeded work for yourself.

Mail::ARF - found here http://wordtothewise.com/resources/arfdeveloper.html 
  - is a simple perl module that will parse and validate an ARF report  
and make the information (including the included original message)  
available to a perl script. There's another module in the Email::  
space too that'll do similar things. Perl runs just fine on Windows,  
and that'd be where I'd start if I needed to do a bespoke ARF handler  
on Windows.

>
> I would appreciate any information that can lead me to finding a  
> solution.  It would be my intention to make any script I might be  
> able to create publicly available.
>
> In your development of ARF as a standard, I would like you to  
> consider this, and correct me if I am wrong: AOL state that they are  
> not prepared to provide the email addresses of complainants and will  
> only provide the message ID of the email that generated the  
> complaint - from which I can then find the email address.  Wouldn't  
> it be easier to provide the email address in the ARF report in the  
> first place?

This is unrelated to the ARF standard. ARF just describes a simple,  
consistent MIME format that the data is embedded in. It's just an AOL  
policy thing.

> What could their reasoning be behind their policy?

Scomp reports are sent mechanically. It's possible, and not even that  
unusual, for an scomp report to arrive in the mailbox of someone who  
was not responsible for the original email being sent, and who does  
not have legitimate knowledge of the AOL email address that the  
original mail was sent to. AOL does not give their users email  
addresses to third parties.

By removing the email address from the report AOL is doing basic due  
diligence to reduce the likelihood that their subscribers email  
address will be given to a third party who should not have access to  
it. Any legitimate bulk emailer[1] will have embedded a cookie in  
their outgoing mail such that they, and only they, can identify the  
subscriber given access to the cookie.

This is annoying to sysadmins, occasionally, when you're sending mail  
from a badly written webapp or mail is from an individual user and you  
can't identify the recipient from the scomp report, but that's the  
tradeoff for helping to ensure that the AOL users privacy is protected  
to some degree, but it doesn't inconvenience any legitimate sender of  
bulk email, and they're the recipients of the vast, vast majority of  
legitimate scomp reports.

(Note that I've argued vehemently in the past that this behavior from  
AOL is Bad and Wrong, but I understand why they do it and their  
reasoning is quite sane and fairly compelling, even if I don't always  
agree with it :) ).

>
> I am also curious as to why the ARF report cannot be viewed using  
> standard email clients, such as MS Outlook. Can someone please  
> explain this to me?
>

ARF is a pretty standard MIME format - it uses a multipart/report  
format, containing three parts - one text/plain, one message/feedback- 
report and one message/rfc822.

Any standard email client should be able to handle that format just  
fine. multipart/* is absolutely standard and the only element you need  
to read to handle the report is message/rfc822.

I'm pretty sure that Outlook can read ARF format just fine. If it  
can't, it's a fairly serious bug. Mail.app renders the text/plain and  
message/rfc822 parts inline and makes the message/feedback-report  
section - which is only usually of any use in the case of automated  
handling of reports - available as a mime attachment.

If you want to test for yourself, send an email message to arf-sample at skylist.com 
  and you'll get an ARF formated report about it sent to you  
immediately - I'd be interested in knowing how Outlook handles that.

Cheers,
   Steve

[1] Any legitimate mailer will handle bounces. Any bulk emailer will  
be sending enough email, and receiving enough rejections and bounces  
that they can't handle bounces by hand. Magic cookie based approaches  
such as VERP are the only reliable way to handle bounces mechanically.  
(That there are a few ESPs who don't do this, and try and intuit from  
the bounce messages what the associated email address might be doesn't  
affect the general observation :) ).