Statement from Platform IQ’s owner and General Manager Erik Geurts, posted on Saturday February 14, 2015.
I’m writing this blog post in response to the incident that severely affected our ad server hosting platform, and as a consequence all of our clients, on February 10, 2015. An Incident Report has been posted a few days ago as well.
On my desk next to my keyboard, I keep a faded page from an old inspirational quotes calendar, which I tore out many years ago. It reads:
“Falling on your face is easy, so you should make an art of getting back up”.
On Tuesday, it felt like I was on my face most of the day. Starting in the first half of the morning European time, a crippling network incident took down our ad server hosting platform. It took about 12 hours to get everything up and running again. We have since found out that the incident was the result of a severe DDoS attack, that was aimed directly at the heart of the networking facilities of our hosting and network provider.
When you fall, you can almost always get back up. In this case, it felt like some invisible hand pushed all of us down with force, and kept pushing. I can only begin to imagine how much inconvenience this incident has caused for our clients, and their customers, site visitors, and staff.
In the past 5 years, we’ve invested a lot of time, energy and money into making our own server platform as robust and reliable as we possibly can. Every single component has been setup to be redundant (a term IT people use to indicate that there are multiple devices that can take over from each other in case of trouble). And this paid off, there hasn’t been a single problem in at least 2 years that turned out to be caused by issues related directly to our hardware, software or other forms of infrastructure.
In recent months, there have been a few smaller incidents where a DDoS attack on a site or server that happened to be in the same data center, had an adverse effect on the availability of our platform. It seems that cybercriminals are changing their attack strategies. Tuesday seems to have been their day of triumph, since they were able to take down not only our platform, but also virtually anything that our hosting and network provider operates. Many large websites and applications were down for 10 to 12 hours.
The incident primarily affected the website of the Dutch central government, rijksoverheid.nl, but also many thousands of others. All of these have in common that all or part of their facilities are provided by the same provider that we work with. It could be said that we were the innocent by-standers of a criminal attack. We were in the wrong place at the wrong time. In a press release issued on February 11, the Dutch government confirmed that the cause of the incident has been a cyber attack (in Dutch). The high tech crime unit of the Dutch Ministry of Justice (called NCSC) are investigating the case.
While the incident was ongoing, we tried to keep all of our customers informed about the situation. We sent each customer an e-mail alerting them to the situation and directing them to the Twitter account we have for this type of communication. Unfortunately, we noticed that we had outdated contact details for a few of our clients. People leave or change jobs, and we haven’t received contact new details in all cases. In other cases, we had contact details for business or accounting staff, who have little or no involvement in daily technical operations. We’ll start a project to freshen up our contact list and have a more adequate process of keeping it up to date.
The Twitter account @StatusPltfrmIQ we set up for this type of communication has a few dozen followers, but it is still pretty new (we only started using it in early January 2015), so many people didn’t find it until many hours after the incident began. Because our main website was also down, it was impossible for them to find out where they should go to find up to date information. We will be communicating our emergency contact channels more prominently in our e-mails and other message from now on.
Because of the broad range of clients we have for our services, and the size of our operation, this incident affected many people, probably tens of millions. In many cases, websites that carry advertising that comes from ad serving systems hosted by us for our customers, were not opening or only very slowly. These websites were actually in mint condition, but the web browser used by visitors was unable to display the pages, because it was waiting for the ads to load. I can’t even begin to imagine how frustrating this must have been for so many people. We are going to work closely with the Revive Adserver open source project, with the aim of developing a so-called ‘asynchronous ad tag’, that will not block websites from loading when an ad server is not responding.
I’ve been reading comments on social media and news articles about the incident. Conspiracy theories all over the place. One specific theme that I noticed is the classic game of “blame the victim”. I realize that people are going to search for answers to their questions and concerns during a major incident, especially if they feel that there is not enough information coming from their usual sources. That’s only human. I think we should all realize that criminal acts are performed by criminals, not by their victims. Speculation is never a good source of journalism.
During and after Tuesday’s incident, we received many e-mails, phone calls, and chat message from clients. Many of you were concerned, disappointed and sometimes even a little bit angry. I understand all of that, I can assure you I felt all of those emotions, and more. But most of the times, people were extremely friendly, supportive and patient. I am humbled and grateful that we have such kind and loyal clients. Thank you very much if you reached out to us with a kind word, it made our otherwise very dark day a little bit lighter.
In the coming days, we will be contacting each and every client individually, to make sure all questions get answered. It is a long list, so please don’t be alarmed if it takes a little while before we get in touch.