Saturday 9th February was the day we opened sales for TFN 2019... and boy, did they open. We know a lot of people received some service interruptions on the day, so we want to briefly explain what happened, and how we will learn from it.
Short Version:
Your massive support for TFN managed to break the internet. (Not our site. The actual Internet. Well, sort of.)
Long Version:
In 2018 an unprecedented number of attendees logged onto our site on ticket sales day. (Almost 40% of our overall attendance, in fact). This resulted in our servers reaching capacity very quickly, meaning attendees had to queue to get onto our site. We wanted to avoid that ever happening again, so we built a completely new website for this year. We also tested various new server solutions to see what would happen if lots of people logged on at the same time. After looking at figures such as 300 and even 1,000 simultaneous connections, we eventually decided to try something much larger. Last week we tested our website against a Cybertronian-scale 100,000 simultaneous connections. And everything worked fine. The new website really is a bit of a beast.
So, if it wasn't the site or the server, what happened?
The answer is, emails. We use a very well known, internationally recognised email provider. Shortly after sales started, the provider locked our account, after detecting so much activity from different locations. This meant website users were sent an error message as they came to the last stage of booking, because the confirmation email was not able to be sent. That's why so many people thought their bookings had failed when in fact, they had not.
Within a couple of minutes of spotting the issue, we took the decision to suspend ticket bookings whilst we implemented a workaround. The solution was to put a hold on all of the emails, which were then sent out much later, in smaller batches. Once the change was made, we reopened bookings and saw no further issues.
So, what have we learned from this experience?
Firstly, that the support of the TFN community is massive! (But we already knew that)
Secondly, whilst simultaneous email confirmation does bring comfort to a lot of people, a slightly delayed solution should help more people to see fewer delays in future. We're already working on a few changes.
This issue was unexpected - not because we hadn't tested it, but because we had. Our email provider has never blocked any of our tests or live runs in the past, so having the problem this year was unexpected. Nevertheless, that's on us and it's something we can and will learn from for next year. Until then, we can only offer our sincere apologies for any extra stress the delays may have caused you. Apologies, and thanks of course, for your continued, overwhelming support.
At the time of writing, we can confirm that all confirmation emails have now been sent. We have also refunded all tickets which attendees accidentally purchased in duplicate. If you continue to be affected by either of these issues, please contact us at support@tfnation.com (it works now!) and we will get the issue resolved for you.
Thanks for your understanding.
The TFNation Crew