Email Fails – How Testing Can Go Wrong

We created Mailtrap, a tool for safe email testing in development and staging environments, with a strong conviction that email testing must be performed on the pre-production stage. The reason was very simple: our team sent out dozens of test emails to the real customers’ accounts when developing an application. We were aware of the problem and knew that we were not alone as the same issue affected our colleagues or clients. In the worst case scenario, this can be a large scale disaster.

TOP-5 email testing failures

From affecting brand reputation to crashing the whole system – we have gathered a list of the all-time worst email testing failures.

5The Church Community Test. The fifth position goes to The Church Community Builder, a US-based company, which develops church management software. In 2015 they sent out a message with the subject line “TEST – Set Up Your Site!” to their prospects, actual and former customers – everyone who interacted with their intranet system ever. It’s not known how many accounts were affected, and it might not have been a big deal but for the fact that some users reported getting multiple emails. What we liked about this story, was that The Church Community Builder posted real-time updates and apologies on their blog.

The worst thing about testing mistakes is that you most probably won’t even know something has gone wrong until you get complaints from the affected (and irritated) users. This is what happened to the fourth nominee on our list.

4Do you support charity? What about being a steward then? In 2018, Oxfam, the international development charity based in the UK, was testing the capacity of their new system. It is said that they were using a dummy emails database but among the 77,000  fake addresses, there were 22 real ones. Not a whole lot, but they were fund supporters. The message informed them that they had been charged £100 to confirm their participation in the Bearded Theory festival as volunteer stewards. Oops.

Link to the original news

3You’ve got a priority! Oh, wait, you haven’t.  Way back in 2009, the Virgin Blue (now Virgin Australia) airlines emailed its entire Velocity loyalty scheme database (and even those who had unsubscribed from their emails!) a free upgrade to their Gold member status. It offered a free lounge, priority check-in, and extra luggage. Any traveler would be happy to get such features. However, very soon the airlines’ team discovered the mistake and sent another message saying “unfortunately you do not qualify for that upgrade”. Considering that it was a major domestic airline in Australia then, we can just imagine how many customers were upset that day!

Link to the original news

2That’s how to plan a pajama party in the embassy! Also in 2018, the US embassy in Australia accidentally used their real email database when testing their newsletter. And it’s our favorite story! The email test contained a picture of a cat wearing blue pajamas and holding a plate of cookies. There was “Meeting” in the subject line and an option to reply to the invitation. It could have been a great party!

Due to the popularity of Twitter, this email test obviously seems to have been both embarrassing and amusing for the sender. Just have look at the reply of the US embassy in New Zealand!

Link to the original news

1National Health System email collapse. The NHS issue (National Health System in the UK) is the outright winner, though the results here were more disastrous than funny. In 2016,  NHSmail system users received several types of test email at once. Even worse, some of the recipients started replying to those messages using “reply all” feature. It affected 840,00 accounts… Could you imagine the number of needless emails on that day? NHS users started reporting the issue to Twitter and some were mentioning over 800 test-related messages in their inboxes!

As a result, the whole system crashed. It is worth mentioning that the NHS is the largest single-payer health system in the world and counts 1.4 million employees.

Link to the original news

In addition to shame and system outages, mass email sending mistakes and extra activity may affect your domain reputation and lead to your IP getting blocked or spam.

Lessons learned

The stories described in this post demonstrate that testing failures continue to happen, regardless of the size and type of the company. Despite all the technological progress and AI inventions of recent years, there is still no mechanism which is able to prevent email testing errors. But there are best practices you can implement to minimize your risk:

  • Test every change in your email functionality. But please, start with non-production environments. Even if you work with test/dummy emails database there is a risk that it uses real addresses of your customers. To prevent this, use Mailtrap, a fake SMTP server, which imitates work of a real one, and helps you to debug your emails before they reach real recipients.
  • When on production, double-check who the recipients are and be accurate with the “reply all” options. You might have set the wrong list by mistake. To stay on the safe side, you can use your own email address for the campaign check. First, send a test email from production and inspect it in your own inbox. After, you can use Bcc to get all copies of sent messages. However this testing method will cause an absolute deluge in your inbox, so you might want to send them to Mailtrap as well. There is an “Inbox Email Address” feature, which allows you to send messages to the particular inbox in Mailtrap from other email accounts or right from your application.
  • If possible, check and track sent messages. And if despite all the tests you have managed to send a wrong email, apologize for this matter.
  • You might also want to wear a “Fail hat” like this guy 😀