How emails are sent from Xero - Do Beautiful Business

[Pages:11]How emails are sent from Xero

Technical discussion

In June 2013 we made a change to the way emails are sent from Xero. Some of our users have asked us why the change was necessary and whether we are planning to provide more options in the future. Be warned this is a fairly technical topic so take a deep breath.

Xero allows organisations to raise invoices and statements and send them to their customers.

Sending email on behalf of hundreds of thousands of Xero organisations, to the millions of their customers is a significant undertaking, and one that we take seriously.

Recently we changed the mechanism by which we send customer invoice emails. This post is an attempt to describe the new setup, the reasoning behind the decision, and a roadmap for future changes that we hope to make.

Setting the scene:

The email landscape has changed dramatically over the past 5 years, and mail servers and email ISPs have all developed unique practices to try and shield their users from spam and other unsolicited email that currently floods the internet.

Over the past 10 years, several standards and defacto practices have evolved to help mail servers make decisions on which emails to trust:

Sender Policy Framework (SPF) a 10 year old standard that lets the owner of a mail domain (e.g. ) specify which mail servers are allowed to send email

DomainKeys Identified Mail (DKIM) a slightly newer standard that allows the sender of an email to digitally sign their email to prove that it came from them

Grey listing a heuristic that some receiving mail servers use to slow down the receipt of email, on the theory that spammers won't attempt to queue and retry messages

Spam filtering a set of algorithms that receiving mails servers can run on email to try and determine if the content of the email is "spammy"

Blacklisting various databases of IP addresses or email addresses that are known to send spam

Domainbased Message Authentication, Reporting & Conformance (DMARC) a recent specification that combines SPF and DKIM to allow email senders to prove that the mail was sent by them

We realise that our users absolutely, positively want email that they send to be received by their customers their cashflow and business depend on timely payments. Even though email is by design an unreliable communication mechanism, we are trying to make it as robust and reliable

as possible for Xero customers to use email to send invoices to their customers.

Our previous approach:

Our previous approach to sending invoice emails from within Xero was to "impersonate" or send email "on behalf" of the user currently logged in to Xero.

e.g. If joe.bloggs@ had signed up to Xero, and proved that they controlled the email address, then we would happily send out email that looked like this:

From: Joe Bloggs To: A Customer Subject: Here is your invoice, pay me now!

In 2006, this was a fairly reliable solution, and most mail servers would happily accept these emails and deliver them to the recipient.

In the subsequent years, it was possible for Joe Bloggs to add an SPF record to their to indicate that Xero was able to send email on their behalf. This worked reasonably well for a few more years.

The Problem:

It is now no longer possible for SaaS vendors to reliably send email "on behalf" of their customers.

If a recipient mail server receives an email like the above example, the mail server will perform some checks to try and decide whether this email was legitimately sent by Joe Bloggs.

One of the indications the server will use is the IP address of the server that delivered the message to it. If the server does not look like a legitimate server for Joe Bloggs' ISP (), then that's a negative mark against the email. Traditionally a combination of reverse DNS lookups and SPF would be used to perform this check.

These days, setting SPF records is not sufficient to prove the authenticity of an email, and this example email illustrates the reason why:

Envelope Sender: joe.bloggs@ From: Joe Bloggs To: A Customer Subject: Here is your invoice, pay me now!

The "Envelope Sender" address is never displayed to regular email users, but it is actually the

address that SPF uses to test the authenticity of an email. In this example, if this email message arrived from a server that was legitimately run by "", then technically that's all that is needed to be legitimate from an SPF perspective even though the email displayed in the recipient's mail client doesn't indicate it comes from !

For this reason, SPF is no longer the solution to sending email "on behalf" of customers like it was when we launched Xero a few years ago.

We have had this confirmed by email experts from Facebook, Yahoo! and JPMorgan Chase when we spoke to them in February this year. In fact, they confirmed what we were seeing, which is that there's no way to reliably send email while pretending to be someone else.

The previous method of sending Xero emails out by impersonating the Xero user was untenable, and we would see continuing increases in the numbers of emails that would fail to deliver.

Problems with SPF:

We've already discussed how SPF on it's own is not sufficient, and that it relies on the verification of a part of the email message that's not actually displayed to an end user.

Our invoice emails actually looked roughly like this:

Envelope Sender: xyzzy123@notifications. From: Joe Bloggs To: A Customer Subject: Here is your invoice, pay me now!

For this email to pass an SPF test, the IP address of the mail server is compared to the SPF records of the "Envelope Sender" domain of notifications.. Xero has control of these records, and so we were able to make sure that the SPF record for notifications. was always valid.

Some mail servers would also run SPFlike tests against the visible "From" address that displays in the email, even though that's not strictly part of the SPF standard, effectively asking the question: "Is Xero's email server allowed to send email from ?"

In order to pass this test, every Xero customer would need to create an SPF record in their DNS to allow Xero's mail servers to send email on their behalf.

For some Xero customers that are ITsavvy, and in control of their own DNS, this was indeed possible, and over the past few years approximately one hundred Xero customers have managed to do this successfully.

However, for most Xero customers this is infeasible:

Many Xero customers use an "ISP" email address, rather than their own domain name (e.g. Joe.Bloggs@ rather than Joe.Bloggs@). This means that they have no control over the configuration of their ISP.

Customers that had their own domain name (e.g. ) but weren't able to change DNS, had a large company with strict IT policies, or were part of a multinational. These customers aren't easily able to create SPF records.

Customers that use SPF, but didn't include Xero's servers in their SPF records may be hardfailing all impersonated mail from other sources. Our analysis showed this was potentially the case for about 7% of customers.

Corporate customer's mail servers would often reject Xero invoice emails that they sent to themselves, even if SPF was correctly set up. E.g. a corporate mail server for would say "I've recieved mail from 's mail server's that is pretending to be from . I know that I'm the only mail server for , so this is obviously spam" and throw it away.

Creating valid SPF records is actually hard. One only needs to look around the internet and on our own Community to see that people don't understand the nuances of SPF, and a simple mistake in configuration can stop delivery of all your company's email. This goes against our mantra of being the "World's easiest accounting system".

The (2013) solution:

We absolutely want to deliver email for our customers, to their customers. Asking our users to download invoices and send them from their desktop is cumbersome and inefficient although we realise for some customers that it will always be their preferred approach so that they have complete control.

Candidate solutions:

Continue impersonating email Unreliable, not recommended by experts. Requires every Xero user to configure SPF correctly, even though that won't guarantee delivery.

Send email via our customer's email servers Technically infeasible (we'd have to collect login credentials, connect to hundreds of thousands of authenticated mail servers to send email, and deal with the myriad of different ways that could fail). We'd be outsourcing the "email delivery" issues to our customers, rather than being responsible for delivering mail correctly on their behalf.

Deliver email with a . "From" address We can guarantee that messages are correctly authenticated. We control the full infrastructure for the email sending.

 We can track deliverability and bounces.

Ultimately, we made the decision to choose the third option, and change the way that we deliver invoice email. We did consider a hybrid approach, or allowing users to optin to one or other approach, but we decided that there were benefits in having a single, reliable delivery mechanism that required no configuration by our customers.

The change we made in June 2013 changed the format of our invoice emails to look like this:

Envelope Sender: xyzzy123@notifications. From: Joe Bloggs ReplyTo: Joe.Bloggs@ To: A Customer Subject: Here is your invoice, pay me now!

To break down the individual changes, and the rationale behind them:

The "Envelope Sender" address, remains as a unique @notifications. address. This is the domain that is used in SPF to validate that our mail server is allowed to send this email. This address is not displayed in user's email clients, and the unique part at the beginning of the email address allows us to track bounces and failures back to the individual sender.

The "From" address has been broken into two parts: The Display Name "Joe Bloggs", which is now configurable per organisation in Xero The email address messageservice@post., which is no longer configurable

The ReplyTo address is now configurable per organisation (e.g. Joe.Bloggs@). This is the address that any customer replies to your invoice emails will be directed to and displayed to them, if they press the Reply button in their mail client.

Our testing showed that in many mail clients, the recipient of this email would see nothing untoward, and simple actions like reading and replying to emails would not display the "" mail addresses.

In some mail clients (e.g. Outlook 2013), the new email format means that recipients can now see "" in the invoice emails that you send. This is an unavoidable sideeffect of this technical decision, and it was not made for marketing reasons!

The reason that some mail clients display "", is that those clients want to show the user the information they have used to make the "trust decision" that has lead to the email not being marked as spam. Our new email format leverages 's trust and email delivery mechanism, so Outlook wants to make it clear to the user who receives the email the reason why it has appeared in their inbox.

I've labelled this as "The (2013) Solution", because although this the best solution at the moment, the nature of the evolving war against spammers means that this approach is not guaranteed to work forever. Bounce messages There are several different types of replies that might happen to an email. If a user is performing the reply, usually by clicking the "Reply" button in their mail client, then the reply will go to the "ReplyTo" address, and arrive directly in the original sender's inbox. If a user is using an old mail client that doesn't understand the "ReplyTo" address, or they manually reply to the "From" address of messageservice@post., then their reply won't be sent to the original sender, but will rather be sent to the unmonitored mailbox hosted by Xero. We don't expect that this will happen too often, but in those situations that it does happen we want the experience for your customers to be understandable. So what we've done is have a robot "Auto Reply" to all of those messages with an "Oops!" message asking them to resend their email to you rather than Xero:

This mailbox is unmonitored, and we don't read any of the emails that are erroneously sent there. Some time after the autoreply is sent back to the original sender, the email will get deleted from our servers. The other types of replies that will come from an invoice email are sent automatically by the recipient's mail server, and called "Bounces".

Bounce messages will usually happen in one of two ways:

An immediate failure, which we'll send directly to the user (e.g. domain name doesn't exist, or has been mistyped)

A slower failure, where the recipient mail server initially accepts the mail message but then later decides it won't deliver (e.g. the username doesn't exist, or their mailbox is full). We receive these bounce message to the Envelope Sender address of something@notifications., and reroute it back to the original sender at the ReplyTo address.

Implementing the DMARC standard:

Those of you that run your own mail servers will know how hard it is to track the deliverability of email, and to diagnose failures that have occurred.

Thankfully, the new DMARC standard allows mail senders such as Xero to receive delivery reports from the recipient mail servers such as GMail, Yahoo! or Hotmail, and actually diagnose any issues that might have occurred.

We are very excited to have partnered with Agari to begin the process of moving all our invoicing email to conform to the DMARC standard. Agari were a part of the original DMARC working group, and have worked with some enormous US brands to share their advice and implement cuttingedge email trust solutions.

The DMARC standard allows us to conclusively prove to recipient mail servers the invoice emails we send have come from a legitimate source. It blends the older SPF and DKIM standards with a new "alignment" process to make sure that their protection messages apply to all aspects of an email, not just the Envelope Sender address.

The other thing that DMARC and Agari allow us to do is to track trends in our delivered email, and to work on errors in our configuration as reported by the recipient. We're now at the point where Google, Hotmail and other mail providers can send us daily reports telling us why they rejected email, and of any issues that they've had delivering email to their users. This is unprecedented and a completely new approach to give us visibility into what happens to the email we send once it leaves our server.

Our Roadmap:

Changing the way we send email is difficult, but also risky. We absolutely want to avoid making our customers' invoice emails fail to deliver. To this end, we're gradually phasing in our adoption of DMARC and the changes to our email sending process.

Step 1: Monitor email delivery using Agari tools

May 2013

We have implemented a DMARC policy in reportonly mode, which allows us to track the changes we're making without instructing recipient mail servers to reject any email that doesn't conform.

Step 2: Change the from address of outgoing invoice emails to a @post. address. June 2013

This lets us use a unique domain name for invoice emails that isn't reliant on the "reputation" or delivery characteristics of any other email that Xero sends. It also allows us to introduce simpler SPF and DKIM policies, and DMARC policies, because any changes will be restricted just to email that comes from a post. address.

This also means that the visible "From" address now aligns with the "Envelope Sender" address, as they are both subdomains. (Domain alignment is a DMARC requirement)

Next steps:

Step 3: Implement DKIM signing of post. email

This will allow us to cryptographically sign all invoice emails, so that a recipient mail server knows that they definitely came from a Xero server and aren't spam.

Step 4: Turn on DMARC enforcement

This will allow us to instruct recipient mail servers to start enforcing our DMARC policies, meaning that they will definitively know whether email we've sent is from Xero, from a mail server we control, and hasn't been tampered with. This will allow mail providers such as Gmail or Yahoo! to have a much higher level of confidence in our email, and will shortcircuit a lot of their spam filtering.

The end result of these four steps will be that we'll have a much higher quality of email that we're sending, recipient mail servers will have a much higher confidence in the email they receive, and we'll have industryleading monitoring of the deliverability of the email we send, so we can be confident that Xero's customers are sending invoices to their customers successfully!

Frequently Asked Questions:

Is it possible to have emails come from our mail domain rather than post.?

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download