Two days ago, Yahoo! made a change to its email servers that caused severe problems for almost all Mailing List Managers, including OnlineGroups.net. Any email sent from a Yahoo! email address to a mailing list could not be delivered to an email address at Gmail, Outlook.com (including MSN, Hotmail and Windows Live Mail), Comcast or Yahoo! itself. By responding quickly, we were able to limit the impact of this change on our users. As from earlier today, deliveries to email addresses at those providers were back to normal.
I will explain the problem that occurred, how it affected Mailing List Managers and their users, including ours, what you can do if the problem affected you and what we did to resolve the problem.
I will also comment on how this incident points the way towards email being more reliable and useful for group collaboration.
There are going to be quite a few acronyms in this post, I will explain them as I go along.
Yahoo!, DMARC and p=reject
Unfortunately it is very easy to send email that looks like it is sent from someone else. Yahoo! has had a lot of problems with people sending spam or phishing email that appear to be from Yahoo! email addresses. To deal with this, Yahoo! changed the way it used DMARC. DMARC is a scheme for checking that email is genuine and not phishing or spam. One part of DMARC is domain alignment.
Domain alignment means that an email has to come from the domain that it says it comes from. To check this, an email server using DMARC first looks at the domain of the From header in the incoming email. The From header is what you normally use to find out who an email is from. The domain is the part after the @ sign. In my email address, it is onlinegroups.net. In a Yahoo! email address it is yahoo.com. The server then tries to verify that the email actually came from that domain. To do this, it uses SPF and DKIM.
SPF is a list of all the IP addresses that are allowed to send email from a domain. Only the owner of a domain can add an IP address to that list. The email server knows the IP address that an email came from so it can easily check if that email is sent from a legitimate IP address. DKIM is a digital signature that is included in the header of the email. The signature is created using the private key of the domain sending the email. The server receiving the email uses that domain’s public key to check that the signature is valid.
There is one more part of DMARC that is important to this story. The p tag. An email server using DMARC can use the p (for policy) tag to tell another email server using DMARC what to do if an email fails the domain alignment checks. The options for p include do nothing, quarantine the email (usually by marking it as spam) or reject. When an email server rejects an email, it simply sends it back where it came from saying no thanks, this failed DMARC.
On Tuesday 8 April, sick of its problems with bad email, Yahoo! changed its p setting to p=reject. This meant that every DMARC compliant mail server that received an email from a Yahoo! email address that did not actually come directly from a Yahoo! mail server, rejected it. Email providers that use DMARC include Gmail, Outlook (née MSN, Hotmail and Windows Live Mail), Comcast and of course Yahoo! themselves. If you have a Yahoo! address (perhaps you did not want to change your address) and use an email server other than Yahoo! to send your email (perhaps you prefer to use another email system), this means that many of the people you email simply won’t get the email.
When you send an email to a Mailing List, the Mailing List Manager makes some changes that mean the original DKIM signature is no longer valid, and then on-sends that email to all the people in the list. Any email set to a list with a Yahoo email address, will then fail DMARC. This means that the email will not be delivered to any intended recipient of that email with a DMARC compliant email server. Furthermore, those email servers will bounce the email back to the Mailing List Manager.
This incident was well reported by John Levine. You can see Yahoo’s DMARC settings with https://dmarcian.com/dmarc-inspector/yahoo.com.In contrast, Google uses ‘quarantine’ https://dmarcian.com/dmarc-inspector/google.com.
How this affected OnlineGroups.net users and what to do
During the period 8 to 10 April, all emails sent from Yahoo! email addresses were not delivered to users of Gmail, Outlook (including MSN, Hotmail and Windows Live Mail), Comcast or Yahoo!
If you think you have missed one or more of these emails, check your online group on the web. The email will be there. If you have sent an important post from a Yahoo! address, send the email again.
Do not worry about your Gmail, Outlook or Yahoo! address being suspended for bouncing messages, OnlineGroups.net requires five bounces on different days over a 50 day period before it sets your email address to unverified.
How we resolved the problem
During the last couple of days, there was a lot of discussion about how Mailing List Managers such as OnlineGroups.net should respond to this problem. DMARC themselves offer some ways to get a mailing list to interoperate with DMARC.
One suggestion was to block Yahoo! email addresses but we do not want to exclude anyone on the basis of their email address. Furthermore, it is quite possible that other email providers will follow Yahoo! and implement strict email checking policies. Another option is to relay the emails without touching them at all. Email servers do this to move email around the Internet. But we want to modify the emails, adding headers, a footer and a subject line prefix to make the email more useful. You can add an Original Authentication Results (aka XOAR) header to claim that the original email came from a trusted source, but more or less anyone can add that so it only works if you are in an elite group of trusted email providers. The remaining option is to change the From address to an address at the domain of the Mailing List Manager. Taking similar approach to Threadable and Al Iverson, that is what we did.
Changing the From address in a post is quite a major change to a Mailing List Manager. It comes with some usability impacts. It also has some advantages, particularly for privacy. The From address in a post to a mailing list is usually the address of the author of the post. This means that a recipient can easily see who is posting what in the list. We have no desire to break that. A recipient can also use that email address to reply privately to another group member who has posted to the list. That is nice for the person sending the private reply, but perhaps the person posting to the list would actually prefer to keep their email address to themselves. For some time, we have had a ticket for a feature that enabled exactly that.
We decided that for now we would change the email addresses of anyone posting to a group from a Yahoo! address. As all group members in a GroupServer site have a profile id, we used that in their email address. Thus if your profile is http://onlinegroups.net/p/1VJQc2umoIC, your email address would become user-1VJQc2umoIC@onlinegroups.net. This means that the email appears to come from us and not Yahoo! so any DMARC compliant mail server will accept it.
To make sure that your name appears on the email, we use the name in your actual email address, or the name in your profile on the site, whichever is longer, in your new email address.
With quick and adept work, Michael JasonSmith pushed out this change on all our hosted systems and made it available to anyone running GroupServer, the open source software that underpins OnlineGroups.net.
How this can make Mailing List Managers better
There has been a fair bit of anger directed towards Yahoo! about making the p=reject setting. By doing this, they were breaking or killing email or at least Mailing List Managers. They were throwing their weight around, using a sledge-hammer to hammer in a nail when they should be taking a more technically sophisticated approach to the problem. At the very least, it would have been nice of Yahoo! to have given us some warning. I do not take particular issue with those views. I take more interest however in how this might be a positive thing.
Email is an inherently insecure and unreliable medium. That is kind of what makes it so easy to use. If you make it harder to use for the bad people, it gets harder to use for the rest of us. But the misuse of email does make it really awful for regular people. We either get swamped with spam or we miss out on legitimate emails. DMARC and Yahoo!’s recent change are part of a trend towards making email more reliable and useful for people.
Mailing List Managers have been around for a long time but that does not mean we can not be part of that trend. What is our job, after all if it is not making email more reliable and useful, in particular for groups? Supporting DMARC is one way we can do that. Developing our role as mediators of email is another. Mailing List Managers, like all software, must always deal with the tension between privacy and usability. Taking charge of the From address is one way in which we can protect the privacy of the author, while doing more to ensure the delivery of the email to the group.
As we reflect on the implications of all this, we will be thinking a lot more about how we deal with the email addresses of group members. Will we obfuscate the email addresses of everyone posting to a group? Or perhaps just those using email servers with strict DMARC policies. Perhaps we will allow group members to choose how private each of their email addresses will be.
And what will we do when people email the addresses we assign group members? Right now, they are treated as posts to a group that does not exist. Clearly that is not great, so we’ll be working on that soon. Will we offer them as an email alias and forward to the original? Or will we somehow make use the Request contact feature on a user’s profile? Perhaps again the group member should get to choose.
However this plays out, I welcome the opportunity to develop the part that Mailing List Managers play in making email more useful and reliable for people who collaborate in groups.