This blog post is long overdue but the recent events that affected customers using Amazon Web Services sparked an increased priority in my mind. I have long been an advocate of Amazon’s amazing efforts driving cloud computing to fruition. And the current unfortunate events have not changed my attitude towards them. Amazon is amazing!

Really amazing! And GoGrid is really thankful.

Thank goodness Werner Vogels keeps getting up on stage with incredible success stories. Werner’s slides featuring Amazon’s growth statistics are so awesome that I bet they make Al Gore jealous. And that isn’t a tongue-in-cheek quip because Al Gore gave birth to the Internet. This isn’t a tongue-in-cheek blog-post. This is a hearty high-five to Amazon from GoGrid. The convenient truth for GoGrid is that when Werner takes the stage, it feels like he’ll need a ladder to show us how incredible the adoption is of AWS.

Can you imagine a world without Amazon? The whole “cloud” revolution probably wouldn’t be happening. SalesForce had been doing their thing for about a decade but it wasn’t until Amazon Web Services geared up that the whole “cloud” thing started cranking.

• No great brand-name company was going to bring cloud computing to market because they’ve been watching a bookstore eat their lunch now for years and they still haven’t done anything relevant. I’m talking about computing giants like IBM, HP, Dell, and Cisco or telecom service providers like AT&T and Verizon. Sun was gonna do something relevant about 2 years ago but Larry pulled the plug on that. Amazon has declared an entirely new style of computing that will define the future, and the big-names just seem to be watching. Wake up! What is going through your dino-minds? There is a new Sheriff in town. Do you still think that Amazon is just an attempt to sell excess bookstore infrastructure that they didn’t need at Christmas? Amazon-dot-com faced an incredible IT challenge and so they invented an entirely new style of massively multi-tenant, incredibly scalable, programmable, on-demand infrastructure. Smell the Java ©!
• Countless technology start-ups would never have started. Amazon established an entirely new paradigm that has changed the game for entrepreneurs in several great ways. First of all, Amazon enables bunches of new businesses that simply couldn’t have existed even if they had the capital. The best examples of these are the companies that simply grew too fast to get their needs met without programmable and massively scalable on-demand infrastructure.
• Amazon has done wonders for the psyche of entrepreneurs. Entrepreneurs feel empowered. There is a new consensus that a company can be started for almost nothing whereas it used to feel like a start-up needed $10,000,000. Entrepreneurs are liberated because they know that they can scale their cost of goods sold with their revenues and that they don’t need any CapEx. Amazon is a bigger friend to tech start-ups than venture capital and has become a comparable engine for growth in our economy.
• OpenStack wouldn’t exist. OpenStack was invented to compete with Amazon. RackSpace never would have purchased SliceHost and jumped into the cloud game. RackSpace wouldn’t have needed a “cloud” strategy ‘cause their core business was doing great.
• And last but not least, people wouldn’t care so much about GoGrid. We’re proud to have pioneered cloud computing alongside Amazon, but if we had done it alone it might be comparable to a tree falling in the forest. We had load balanced cloud servers years before Amazon, and load balancing is core to elasticity which is a defining attribute of cloud computing, but not as many people would have noticed. We also were way ahead of Amazon with on-demand Windows servers, real VLANs, dedicated servers, Hosted Private Cloud and bunches of other things but you probably wouldn’t be reading this blog-post if it weren’t for Amazon. We take great pride in having a feature-set that our customers and industry analysts alike consider to be the most comparable to Amazon’s. We’re headquartered in San Francisco and quite good at R&D ourselves and we also are pushing the industry forward. GoGrid got into the game trying to solve a different problem than Amazon, we were trying to automate managed hosting, but we’ve both ended up in nearly the same wonderful place…the cloud!

So thank you Amazon! You’ve done the World a great service!

Love,

John Keagy

CEO
GoGrid

Whoops! Missed a day there. I was busy planning out events for 2010 for GoGrid! I realize now that this is somewhat difficult writing about Cloud News every day so I’m going to start something for Fridays called “This Week in Cloud” which will have some of the bigger Cloud Computing news stories that I came across. I may still do the regular “Cloud News” if there are events or items that warrant coverage. Without further ado, here’s what I read about that got my interest:

• Amazon’s Data Center Outage Reads Like a Thriller
  “When an Amazon Web Services data center lost power early Wednesday, the company wrote about the unfolding event with the brevity and tension of one its bestselling pot boilers.” (Source: CIO/ComputerWorld)
  • Commentary: Ok, we all know that outages happen, whether in the cloud or not. The cloud is under intense scrutiny so when there is even the most minor of hiccups, people scream and yell and pull out their SLA’s and demand immediate recourse. It’s never fun when I read about an outage since we are all birds of a feather working towards a common good. The reason I linked to this article in particular is because of the nature in which it was written…truly like a technology thriller! The funny thing is, I have been through a few outages in various companies that I have worked for and they are never fun. You are torn in many directions of trying to find out internally what is going one, and figuring out the best way to communicate with customers in a way where they won’t freak out but still understand that an “event” is being actively working on. When there is an outage, it truly does unfold organically (and hopefully not catastrophically, causing a cascade effect). Third party monitoring is important as is a good backup and disaster recovery strategy. (GoGrid recently partnered with Stratonomic who provides real-time DR solutions.)  Regardless, the Amazon Data Center outage did not last long and everyone was back on track (but hopefully thinking about making their IT infrastructure more resilient). It was nice to see that GoGrid had (and still seems to have) the highest marks on the Apparent Networks Cloud Provider Scorecard.
• Hackers Find a Home in Amazon’s EC2 Cloud
  “Security researchers have spotted the Zeus botnet running an unauthorized command and control center on Amazon’s EC2 cloud computing infrastructure. This marks the first time Amazon Web Services’ cloud infrastructure has been used for this type of illegal activity, according to Don DeBolt, director of threat research with HCL Technologies, a contractor that does security research for CA.” (Source: PCWorld)
  • Commentary: I don’t want to turn this into an AWS bashing, that is not my intent (even though they are a direct competitor to GoGrid). But this news (old by internet news speed standards) is important to look at. Hackers are an inquisitive (yet destructive) bunch. Recently, I heard about how some hackers created a service (called AutoWhaler) to pull account details from phishing sites. Now if that isn’t innovation, I don’t know what is! Seriously though, hackers can cause a variety of damage to infrastructures that are not protected and actively monitored. The problem is, they are so innovative (or perhaps “creative” is a better word), that it is often difficult to prevent or plan for intrusions within one’s infrastructure. Obviously this and other examples simply prove that while Cloud Computing can help many, we still have a ways to go. On the flipside, this same type of invasion could well have happened within a server farm of physical servers. It’s just the Cloud has the spotlight now.
• Microsoft buys Opalis to strengthen cloud management capabilities
  “Moving to strengthen its management tools related to virtual environments, Microsoft Friday said that it has purchased IT process automation vendor Opalis for an undisclosed sum. Microsoft said the acquisition, which had been rumored for nearly two months, adds to its System Center portfolio needed tools that can manage highly automated and scalable virtual environments. The tools complement Microsoft’s strategy to stretch its management tools across on-premises environments and the cloud.” (source: NetworkWorld)
  • Commentary: And so the Cloud acquisitions begin (or continue). 2010 will have many more of this with M&A activities heating up, especially within the Cloud. Smaller vendors and niche market players will have their 15 minutes of fame next year as they innovate and then potentially get sucked up into the bigger clouds out there. Microsoft is going full steam now into the Cloud with Azure and other Cloud-ish offerings (e.g., Office 2010) so there will be lots to watch in 2010.

Raditha Dissanayake posted a blog entry comparing Amazon EC2 and GoGrid performance. Unfortunately, we think Raditha did not use the most rigorous methodology possible for doing his comparison. It would be inappropriate for GoGrid to performance test Amazon’s EC2. In fact, their Customer Agreement may actually make such activity questionable, but IANAL (I Am Not A Lawyer).

Let’s take a more rigorous look at GoGrid disk subsystem performance.

Framing the Issue

As a start the entire issue is a LOT more complex than can potentially be covered here. Today’s disks, hard drive controllers, and operating systems have many different kinds of caching mechanisms. In addition, virtualization systems like Xen can impact results in unexpected ways. For example, did you know that Xen can be deployed in two major manners?

Either ‘paravirtualized’ or ‘hardware virtualized’. The two different models almost certainly impact any testing methodology. And yes, you guessed it, Amazon and GoGrid don’t configure Xen in the same way. Amazon uses paravirtualization and GoGrid uses hardware virtualization. Beyond this public information neither Amazon nor GoGrid provide significant details about their infrastructure considering it, rightfully so, proprietary intellectual property.

Without a deep understanding of all of the issues it’s difficult to do a test much less a proper comparison.

But we are certain of a few very important things.

Clouds Are Multi-Tenant

First off, it’s hard to do a serious comparison like this using one server on each system. Clouds are inherently multi-tenant systems and since end users have no visibility into who else is using or sharing their disk resources at any given time there is no real way to verify that the results aren’t tainted by other activity.

Use the Right Tool

Secondly, hdparm -t isn’t a very good way to measure disk speed. It’s susceptible to noise from background activity, in fact the man page says:

-t Perform timings of device reads for benchmark and comparison purposes. For meaningful results, this operation should be repeated 2-3 times on an otherwise inactive system (no other active processes) with at least a couple of megabytes of free memory. [...]

As you can see in Raditha’s test, hdparm doesn’t really do enough I/O to get consistent results in a multi-tenant environment. In the tests, hdparm is only active for a very short period of time allowing tenancy to have a dramatic effect on the results.  hdparm requires an inactive system and since that can’t be guaranteed in the cloud it fails the sniff test for a robust tool for cloud performance testing.

Another factor here that is unaccounted for is that hdparm is a utility tuned for real physical disks, not virtual disks.

Better Measurements

Ideally if you want to measure the streaming performance of a block device in a more reliable way in a multi-tenant environment, then use a larger amount of I/O. When doing this I/O you want to try to eliminate:

• Hard disk controller layer cache effects
• Hard disk layer cache effects
• OS level cache effects
• Effects of disk activity from other VMs

All current GoGrid nodes have caches in the storage layer. These are designed to be robust and to absorb burst of write activity. These caches are sufficiently large though that if you do repetitive small I/Os what you end up measuring in the performance in pulling this data out of the storage layers caches, not from the storage itself.

To avoid OS level cache effects use ‘direct I/O’. High performance applications and databases tend to use this internally for similar reasons (because they want to avoid OS level cache pollution and do their own caching). Oracle is probably the most obvious example here.

Testing Performance

On a ‘small VM’ located on a fairly busy node:

[root@foo ~]# dd if=/dev/hda bs=10M of=/dev/null iflag=direct count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 3.50983 seconds, 299 MB/s

[root@foo ~]# dd if=/dev/hda bs=10M of=/dev/null iflag=direct count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 3.06811 seconds, 342 MB/s

[root@foo ~]# dd if=/dev/hda bs=10M of=/dev/null iflag=direct count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 2.14147 seconds, 490 MB/s

That’s using enough I/O to minimize noise from other VM activity and large enough to avoid hitting cache effects.

If the I/O load is small enough you can hit storage layer cache effects:

[root@foo ~]# dd if=/dev/hda bs=10M of=/dev/null iflag=direct count=10
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 0.116491 seconds, 900 MB/s

[root@foo ~]# dd if=/dev/hda bs=10M of=/dev/null iflag=direct count=10
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 0.16058 seconds, 653 MB/s

[root@foo ~]# dd if=/dev/hda bs=10M of=/dev/null iflag=direct count=10
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 0.115701 seconds, 906 MB/s

While this is a fairly contrived example, it’s useful in other ways because it shows you can get very good burst throughput (consider a database updating a few thousand pages).

A larger memory instance (where average performance should be a lot better).

Sustained (large) IO:

[root@ubdev1 ~]# dd if=/dev/hda bs=10M count=100 of=/dev/null iflag=direct
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 1.80415 seconds, 581 MB/s

[root@ubdev1 ~]# dd if=/dev/hda bs=10M count=100 of=/dev/null iflag=direct
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 1.70448 seconds, 615 MB/s

[root@ubdev1 ~]# dd if=/dev/hda bs=10M count=100 of=/dev/null iflag=direct
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 1.6799 seconds, 624 MB/s

Burst (small) IO:

[root@ubdev1 ~]# dd if=/dev/hda bs=10M count=10 of=/dev/null iflag=direct
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 0.105183 seconds, 997 MB/s

[root@ubdev1 ~]# dd if=/dev/hda bs=10M count=10 of=/dev/null iflag=direct
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 0.089827 seconds, 1.2 GB/s

[root@ubdev1 ~]# dd if=/dev/hda bs=10M count=10 of=/dev/null iflag=direct
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 0.090264 seconds, 1.2 GB/s

Don’t take my word for any of this. Try it out. If you’re really bored graph I/O performance vs I/O size and you’ll likely see a step function with a soft edge that will give you some idea of what the storage system is capable of and the degree of I/O variation.

Bottom Line

It’s great that people are kicking the tires of various clouds, but let’s be careful to make sure our testing is rigorous and makes sense for the environment.  If you have questions about how to measure performance on clouds, please send them to us.  Or if you’re a performance and virtualization system guru and have some knowledge to share, please do so.

We always want to improve our cloud and take seriously any feedback that shows a real problem, but in this case the test needs tweaking, not GoGrid.

2008 was an action-packed year for us here at GoGrid and ServePath and we have many accomplishments to be proud of. I thought it would make sense to reflect back on what major things we did over the year as well as a few other notables that happened within the industry. The easiest way for me to do this is through a blog post Chronology (not every post is highlighted):

1st Quarter 2008

• 01.03.08 – GoGrid Blog was launched
• 01.29.08 – “Sneak Peak” at GoGrid
• 02.01.08 – Twitter and Joyent go different ways
• 02.05.08 – Understanding “Clouded” Computer Terms – a post that made a 1st attempt to explain Cloud, Utility, Grid and other Computing terms.
• 02.13.08 – Dilbert does a series on Virtualization (here, here and here)
• 02.15.08 – Amazon’s S3 has major outage (my comments)
• 02.21.08 – GoGrid launches a new public website in anticipation of the product launch
• 03.11.08 – GoGrid Public Beta LAUNCH! After over 2 years of development, GoGrid hits the streets with many Cloud Computing firsts:
  • 1st Cloud Infrastructure provider with a Web GUI
  • 1st to offer Windows Server 2003 in the Cloud
  • 1st to offer Microsoft SQL Server in the Cloud
  • 1st with free Inbound Transfer
  • 1st with free f5 Load Balancing
  • 1st with free 24×7 Support
  • 1st with Persistent Storage
  • 1st with free managed DNS
  • 1st with 100% Uptime SLA
  • 1st with public and private VLANs
• 03.17.08 – Drilling down on the details of new GoGrid accounts
• 03.18.08 – Even I wasn’t initially on board with the whole “Cloud Computing” term. My thoughts have changed obviously.
• 03.28.08 – The initial GoGrid FAQ’s start rolling out.

2nd Quarter 2008

• 04.10.08 – GoGrid launches new templates: multiple RHEL, CentOS, Windows 2003 flavors
• 04.14.08 – GoGrid launches the Wiki (wiki.gogrid.com)
• 04.21.08 – GoGrid exhibits at the Web 2.0 Expo (some of our Expo stunts)
• 05.20.08 – GoGrid starts getting some blog traction
• 05.22.08 – Soliciting feature and images requests from the GoGrid community produced many great ideas
• 06.02.08 – NetworkWorld reviews GoGrid
• 06.04.08 – IP Suggest feature débuts
• 06.10.08 – “Cloud Computing” terms really starts trending upwards
• 06.16.08 – Cloud Computing helps build communities
• 06.19.08 – InfoWorld reviews GoGrid
• 06.24.08 – Introduction of the Cloud Pyramid
• 06.27.08 – GoGrid at GigaOM Structure ‘08 and Rafe Needleman of WebWare talks about GoGrid

3rd Quarter 2008

• 07.07.08 – GoGrid hits 1000th user and coverage by TechCrunchIT
• 07.17.08 – GoGrid launches API
• 07.18.08 – NetworkWorld, C|net & TechCrunchIT cover GoGrid’s new API
• 07.21.08 – InfoWorld does a side-by-side comparison of GoGrid, Amazon’s EC2 and Google App Engine
• 07.22.08 – Teens-in-Tech founder, Daniel Brusilovsky, interview of GoGrid
• 07.31.08 – Google Web Toolkit (GWT) showcases GoGrid
• 08.06.08 – GoGrid WINS LinuxWorld 2008 Best of Show in Product Excellence
• 08.19.08 – GoGrid is the FIRST to launch Windows Server 2008 in the Cloud
• 09.09.08 – the first NoHardware.com video is released
• 09.16.08 – Financial Markets start getting very shaky. Cloud Computing can help stabilize.
• 09.17.08 – GoGrid and RightScale partnership announced
• 09.22.08 – Feature preview of GoGrid’s Cloud Storage (now live)
• 09.23.08 – the second NoHardware.com video is released
• 09.29.08 – The “Original” Cloud Computing in Plain English produced in-house by GoGrid launches
• 09.30.08 – GoGrid and Appistry partnership announced

4th Quarter 2008

• 10.01.08 – Other Cloud providers announce Windows in the Cloud, play catch-up with GoGrid
• 10.03.08 – Recession starts hitting – 10 Tips to Survive it
• 10.07.08 – GoGrid and GigaSpaces partnership announced
• 10.08.08 – the third NoHardware.com video is released
• 10.21.08 – Cloud Computing Product Announcement Bingo Game
• 10.27.08 – GoGrid launches PHP-enabled Windows Server 2008 Clouds
• 11.03.08 – GoGrid & ServePath beta release Cloud Connect (connecting dedicated servers with the Cloud)
• 11.24.08 – Appistry & GoGrid do a joint webinar on Cloud Computing
• 12.02.08 – My 10 Cloud Computing Predictions for 2009, our CEO’s Predictions and our VP of Technology Strategy’s Predictions
• 12.11.08 – GoGrid featured again on Google Web Toolkit (GWT)
• 12.11.08 – GoGrid launches Cloud Storage v0.7
• 12.15.08 – Santa “Cloud” or how Cloud Computing saved Christmas
• 12.29.08 – Some New Years Resolutions

Happy New Year to all of you from us at GoGrid. May 2009 be happy, healthy and prosperous!

The start of a New Year is upon us so it is time to get a list together of things that you will do (or do your best to do) in the coming year. Everybody has their own personal Resolution lists, but what about your Business ones? How are you going to remain competitive? What steps are you going to take to cut your budget to remain lean and mean? Are you going to stick with your current methods or adopt some new strategies?

Here are some “Resolutions” that you can think about as you ready your business for 2009.

1. Invest some time in understanding the term “Cloud Computing” – there are several easy-to-understand definitions and movies that have come out that make Cloud Computing a bit more understandable. This one was done at the 2008 Web 2.0 Expo. Then came the GoGrid “Cloud Computing in Plain English”. Recently, there is a new “In Plain English” from the actual Common Craft folks (whom we got our inspiration from). And here is a more technical presentation that came out recently. Regardless, there are lots of sources out there for quick understandings. I have been maintaining a Bookmark RSS feed as well of many of the Cloud Computing blogs and sites. Subscribe to that feed for updated links. Also, read through the popular Cloud Computing Group on Google. Lastly, you can check Wikipedia for their ever evolving definition of Cloud Computing.
2. Do some research on different Cloud Providers – no Cloud Computing provider is the same, and the differentiation is continuing. Last year (2008), I introduced the idea of the Cloud Pyramid which has Cloud Applications (SalesForce) at the top, then Cloud Platforms (Google App Engine or Microsoft Azure) in the middle and finally Cloud Infrastructure (GoGrid and Amazon EC2) as the bottom foundation. Also hooked into it are Cloud Extenders (e.g., Amazon’s SQS) and Cloud Aggregators (RightScale). It’s pretty obvious that there are many choices to be made and that these are very specific to the type of business you are running. In fact, we will be further segmenting the IaaS (Cloud Infrastructure) section more over the next few weeks. Briefly, GoGrid is now being positioned as a “CloudCenter” (which is essentially, a DataCenter equivalent but in the Cloud). More on that later. In the meantime, compile a series of questions for yourself and for your prospective provider. We will get a list together of things you might want to ask (post to come).
3. Review your IT Budget – If you are like most companies out there, you are going through your 2009 budgeting (or have done so already and are probably on your 10th revision now). One way to make your CFO happy is to reduce your Capital Expenditures (CapEx). The easiest way to do that is to really take a hard look at Cloud Computing. If you can slash your CapEx spend by downsizing your physical server footprints, you can easily upsize that same footprint in the Cloud.
4. Empower your Programmers – Cloud Computing offers something new to Programmers: the ability to programmatically control their IT infrastructure. Using an API, Programmers can skin the functionality provided by Clouds as well as develop “intelligent” applications that scale dynamically, for example.
5. Empower your IT Staff – Be sure that you don’t ignore your IT Staff as you look at the Cloud as a physical IT infrastructure alternative. They have some best practices and standards that should be incorporated in what your IT strategies will be. Let them experiment with the Cloud so that they fully grasp what it can do for your organization. They may tell you that it is a great direction to go in, or, they may say that your current infrastructure simply cannot be ported to the Cloud. There may also be some hybrid solutions (like GoGrid’s Cloud Connect) that will give them the best of both worlds.

These are just a few Cloud Computing New Year’s 2009 Resolutions to get you thinking. What are your business resolutions for 2009?