KML_FLASHEMBED_PROCESS_SCRIPT_CALLS
 

Message from GoGrid Founders Regarding Denial of Service Attack

March 31st, 2009 by - 15,645 views

The message below was sent to GoGrid customers from our CEO/Co-Founder, John Keagy, and CMO/Co-Founder, David Hecht, regarding the Distributed Denial of Service (DDoS) Attack which affected many GoGrid users.

It has been a long couple of days here at GoGrid. We are hearing from many of you that you want more information, more often. Engineers are often working so hard to fix an issue that you they don’t give you enough visibility into what is going on.

In that spirit of improved communication and transparency, I want to continue sending updates via email. You can always see the latest status at http://www.GoGridStatus.com as well.

While some of you have been unscathed by the network attacks and portal issues over the past few days, we know that many of you are frustrated at the downtime and impacts to your Internet infrastructure. We are frustrated as well. We’ve been in the hosting business for over 8 years now, and have generally been able to prevent most incidents from impacting customers as heavily as this attack did.

ONGOING DDoS ATTACK

Our network is currently the target of a large, distributed DDoS attack that began on Monday afternoon.   We took action all day yesterday to mitigate the impact of the attack, and its targets, so that we could restore service to GoGrid customers.  Things were stabilized by 4 PM PDT and most customer servers were back online, although some of you continued to experience intermittent loss in network connectivity.

We had a maintenance window scheduled for 9 PM PDT to do a major expansion of GoGrid’s capacity and roll out some minor feature improvements and bug fixes. Because this maintenance window required the portal being down and support cases would have to be opened by phone, we considered postponing the maintenance to a time when things were calmer.

In the end, the decision was made to proceed with the maintenance because this capacity expansion had been planned for several months and would give us more flexibility in ensuring low utilization across our infrastructure. In hindsight this may have been a poor decision because the maintenance took longer to complete and the maintenance window had to be expanded by several hours.


ROUTING ISSUES THIS MORNING

We spent the night cleaning up servers that were still down, reboots that did not happen properly, and other issues, and continued to develop plans to establish a long term solution to this ongoing issue.

Beginning early this AM PDT, our support team began to get more and more reports of servers that were unreachable from certain parts of the Internet. All of these servers were pingable and accessible from our testing connections outside the GoGrid network, but not to all locations worldwide. There appeared to be a routing issue with some networks not properly announcing GoGrid routes. Some of your web sites appeared offline to most or all of your own customers, while many were unaffected.

CURRENT STATUS

The routing issue was resolved around 11 AM PST today.  Our network engineers localized the problem to an issue with our border routers improperly announcing some routes.  The issue was resolved by clearing our BGP cache on our border routers.  We are not certain at this time the root cause of the issue, and are continuing to investigate and will provide an RFO soon to customers who opened Cases.  We suspect the issue had something to do with the changes we implemented in an emergency maintenance window, yesterday, as part of our efforts to mitigate the DDoS attack.

If you are continuing to see any connectivity issues with your GoGrid servers, we ask that you run a traceroute to your servers’ IP address so you can provide it to our support staff when logging an issue at http://my.GoGrid.com

We appreciate your patience during this difficult time, and thank you for being a GoGrid customer.

I personally would like to apologize for these issues as well. While out of our control for the most part, these DDoS attacks affected many of you and disrupted your livelihood, business and personal life.

The following two tabs change content below.

Michael Sheehan

Michael Sheehan, formerly the Technology Evangelist for GoGrid, is a recognized technology, social media, and cloud computing pundit and blogger who writes regularly about technology news and trends.

4 Responses to “Message from GoGrid Founders Regarding Denial of Service Attack”

  1. Stuart says:

    Thanks for the full explanation and for the apology about the outage, the badly timed maintenance upgrade and the disruption in general. It's good to see such an honest post about the problems. 24 hours of outage was certainly painful for us – but I recognise that the 10000% SLA cost and the negative publicity have also have hurt GoGrid too. Good luck with squashing the remaining part of the DDoS. I'm now looking forwards to growing on an even better GoGrid plaform :)

  2. [...] more poignant reminder of our dependency on the Same Old Stuff™ is the recent DDoS attacks against Cloud provider Go-Grid: ONGOING DDoS [...]

  3. [...] Cloud Computing Anbieter GoGrid ist Opfer eines großen DDoS Angriffes, welcher die Dienste der Hälfte ihrer 1.000 Kunden behindert. Am Montag Nachmittag begannen die Attacken. Während der Nacht wurden von GoGrid Abwehrmassnahmen ergriffen und die Probleme der meisten Kunden waren am darauffolgenden Tag gegen 16 Uhr behoben. “Wir sind seit mehr als 8 Jahren im Webhosting Geschäft und haben es im allgemeinen geschafft unsere Kunden von Auswirkungen solcher Vorfälle, wie diesen aktuellen schweren Angriff, zu bewahren.” (Meldung im GoGrid-Blog) [...]

  4. [...] fecha en la que el gusano Conficker se activaba, varios objetivos fueron alcanzados, entre ellos GoGrid (afectando a la mitad de sus más de 1,000 clientes), así como a UltraDNS, el proveedor de [...]

Leave a reply