The message below was sent to GoGrid customers from our CEO/Co-Founder, John Keagy, and CMO/Co-Founder, David Hecht, regarding the Distributed Denial of Service (DDoS) Attack which affected many GoGrid users.
It has been a long couple of days here at GoGrid. We are hearing from many of you that you want more information, more often. Engineers are often working so hard to fix an issue that you they don’t give you enough visibility into what is going on.
In that spirit of improved communication and transparency, I want to continue sending updates via email. You can always see the latest status at http://www.GoGridStatus.com as well.
While some of you have been unscathed by the network attacks and portal issues over the past few days, we know that many of you are frustrated at the downtime and impacts to your Internet infrastructure. We are frustrated as well. We’ve been in the hosting business for over 8 years now, and have generally been able to prevent most incidents from impacting customers as heavily as this attack did.
ONGOING DDoS ATTACK
Our network is currently the target of a large, distributed DDoS attack that began on Monday afternoon. We took action all day yesterday to mitigate the impact of the attack, and its targets, so that we could restore service to GoGrid customers. Things were stabilized by 4 PM PDT and most customer servers were back online, although some of you continued to experience intermittent loss in network connectivity.
We had a maintenance window scheduled for 9 PM PDT to do a major expansion of GoGrid’s capacity and roll out some minor feature improvements and bug fixes. Because this maintenance window required the portal being down and support cases would have to be opened by phone, we considered postponing the maintenance to a time when things were calmer.
In the end, the decision was made to proceed with the maintenance because this capacity expansion had been planned for several months and would give us more flexibility in ensuring low utilization across our infrastructure. In hindsight this may have been a poor decision because the maintenance took longer to complete and the maintenance window had to be expanded by several hours.
ROUTING ISSUES THIS MORNING
We spent the night cleaning up servers that were still down, reboots that did not happen properly, and other issues, and continued to develop plans to establish a long term solution to this ongoing issue.
Beginning early this AM PDT, our support team began to get more and more reports of servers that were unreachable from certain parts of the Internet. All of these servers were pingable and accessible from our testing connections outside the GoGrid network, but not to all locations worldwide. There appeared to be a routing issue with some networks not properly announcing GoGrid routes. Some of your web sites appeared offline to most or all of your own customers, while many were unaffected.
The routing issue was resolved around 11 AM PST today. Our network engineers localized the problem to an issue with our border routers improperly announcing some routes. The issue was resolved by clearing our BGP cache on our border routers. We are not certain at this time the root cause of the issue, and are continuing to investigate and will provide an RFO soon to customers who opened Cases. We suspect the issue had something to do with the changes we implemented in an emergency maintenance window, yesterday, as part of our efforts to mitigate the DDoS attack.
If you are continuing to see any connectivity issues with your GoGrid servers, we ask that you run a traceroute to your servers’ IP address so you can provide it to our support staff when logging an issue at http://my.GoGrid.com
We appreciate your patience during this difficult time, and thank you for being a GoGrid customer.
I personally would like to apologize for these issues as well. While out of our control for the most part, these DDoS attacks affected many of you and disrupted your livelihood, business and personal life.
Latest posts by Michael Sheehan (see all)
- Get Your Game On in the Cloud - June 11, 2013
- How Software Defined Networking Delivers Next-Generation Success - June 5, 2013
- James Gosling to Speak on Innovation at GoGrid Cloud Meetup on 5/22 - May 16, 2013