On February 10, 2015, our ad server hosting platform suffered an outage that lasted multiple hours. This incident report has details on the cause of the outage and additional background information.
At 08.46 UTC, our external monitoring system reported several issues with the ad server hosting platform. In fact, the incident had started a few minutes earlier, but since the outage also affected our internal monitoring, the reports from those systems didn’t reach our engineers until after the incident was resolved.
It became clear almost immediately that a massive network issue had occurred, resulting in a complete loss of connectivity for all servers. Under normal circumstances, servers and other devices in the data centers can be managed remotely, but in this case, even that was impossible. Within minutes, a team of our network partner’s engineers was dispatched to go on site at the data centers to work on the issue.
Arriving at the sites in Amsterdam, it was discovered that a DDoS attack was ongoing. Instead of targeting a specific website or server, the attack was focused on key networking devices. This helps explain why it took almost 12 hours to mitigate the attack, all network devices had to be reconfigured manually to block the attack vector.
Our ad server hosting platform came back online at 20.39 UTC. This also meant we could finally access the servers, applications, and databases remotely again. In the hours that followed, every single component was checked and inspected to ensure correct operations. A few short and controlled outages occurred in subsequent hours, for example when a critical device had to be rebooted. The outage can be seen cleared in our online status dashboard for February 2015.
It has since been established that the attack was aimed to bring down the website of the Dutch central government, rijksoverheid.nl, which is hosted and serviced by the same company that cares for our servers and additional infrastructure. As a consequence, the government website but also thousands of other sites (both large and small) went down. The NCSC, National Cyber Security Center, a unit of the Dutch ministry of Justice, has classified this attack as a cyber attack. Pending their investigation, we have been asked not to reveal any details about the method of attack.