It’s important to us to clarify GoGrid’s position with regard to cloud computing standards and the Open Cloud Manifesto (OCM). There has been a fair bit of controversy in the ‘blogosphere’ recently over the OCM, which is to be released on Monday.

In particular, myself and Steve Gillmor (of TechCrunch IT fame among others), had a somewhat heated, but friendly exchange over his scathing assessment of the situation. Steve invited me to a “News Gang” podcast of the Gillmor Gang on Friday, which was posted here. During that live podcast he asked us to clarify GoGrid’s position.

This post is really about making sure everyone is on the same page and understands how GoGrid views the OCM and cloud computing standards in general.

Background
It’s unnecessary to recap everything in detail. I think James Urquhart handled this fairly succinctly. Geva Perry also has a nice summary including a link to the draft document. In a nut:

1. Some folks tried to lay down some guiding principles for “open” cloud computing in the Open Cloud Manifesto
2. Some folks reacted badly feeling that the process wasn’t actually “open”
3. Bruhaha ensued

Who cares?

Well, we all should really. From our perspective this is a healthy, yet contentious debate. We think there were good points and missteps on all sides. In particular, we think it’s important to realize that given how interconnected we’re all becoming it’s actually very hard for any given group to monopolize the Internet, the “cloud”, or similar.

We believe the following to be true about what happened:

• The folks involved in the early Open Cloud Manifesto did not intend to “shut out” anyone
• The process around building the Open Cloud Manifesto could have been more ‘open’
• The manifesto is not about setting a standard, but starting a conversation
• This “conversation” is meant to be about principles that already apply to the Internet

Position on Open Cloud Manifesto
We continue to be an enthusiastic supporter of the Open Cloud Manifesto (OCM) and open cloud standards in general. That’s why we licensed our own API under an open license in January. The OCM is an important move forward in the emerging debate about what “Cloud” and “Cloud Computing” mean. We do not support any kind of exclusion in the OCM or of folks who want to be it’s supporters. We believe everyone needs to have a say in these guiding principles. In fact the OCM itself is largely about saying how much “The Cloud” needs to be open, unfettered, and democratized.

That means everyone needs to be involved.

Summary
Simply put: contentious conversations, vibrant arguments, and great people will all eventually yield the right results. We don’t think it’s possible for anyone to cordon off and monopolize this conversation, foist standards on others, and won’t support such efforts. And, we don’t think the Open Cloud Manifesto is anything but a well-intentioned attempt to move the conversation forward. One that was never meant to be ‘closed’ and come Monday when it’s officially released we’ll see that it’s a positive move and all of the folks who worked together on the OCM (including Reuven Cohen, IBM, and many others) should be commended for their attempts to get everyone on the same page even in the face of extreme controversy.

–Randy

Raditha Dissanayake posted a blog entry comparing Amazon EC2 and GoGrid performance. Unfortunately, we think Raditha did not use the most rigorous methodology possible for doing his comparison. It would be inappropriate for GoGrid to performance test Amazon’s EC2. In fact, their Customer Agreement may actually make such activity questionable, but IANAL (I Am Not A Lawyer).

Let’s take a more rigorous look at GoGrid disk subsystem performance.

Framing the Issue

As a start the entire issue is a LOT more complex than can potentially be covered here. Today’s disks, hard drive controllers, and operating systems have many different kinds of caching mechanisms. In addition, virtualization systems like Xen can impact results in unexpected ways. For example, did you know that Xen can be deployed in two major manners?

Either ‘paravirtualized’ or ‘hardware virtualized’. The two different models almost certainly impact any testing methodology. And yes, you guessed it, Amazon and GoGrid don’t configure Xen in the same way. Amazon uses paravirtualization and GoGrid uses hardware virtualization. Beyond this public information neither Amazon nor GoGrid provide significant details about their infrastructure considering it, rightfully so, proprietary intellectual property.

Without a deep understanding of all of the issues it’s difficult to do a test much less a proper comparison.

But we are certain of a few very important things.

Clouds Are Multi-Tenant

First off, it’s hard to do a serious comparison like this using one server on each system. Clouds are inherently multi-tenant systems and since end users have no visibility into who else is using or sharing their disk resources at any given time there is no real way to verify that the results aren’t tainted by other activity.

Use the Right Tool

Secondly, hdparm -t isn’t a very good way to measure disk speed. It’s susceptible to noise from background activity, in fact the man page says:

-t Perform timings of device reads for benchmark and comparison purposes. For meaningful results, this operation should be repeated 2-3 times on an otherwise inactive system (no other active processes) with at least a couple of megabytes of free memory. [...]

As you can see in Raditha’s test, hdparm doesn’t really do enough I/O to get consistent results in a multi-tenant environment. In the tests, hdparm is only active for a very short period of time allowing tenancy to have a dramatic effect on the results.  hdparm requires an inactive system and since that can’t be guaranteed in the cloud it fails the sniff test for a robust tool for cloud performance testing.

Another factor here that is unaccounted for is that hdparm is a utility tuned for real physical disks, not virtual disks.

Better Measurements

Ideally if you want to measure the streaming performance of a block device in a more reliable way in a multi-tenant environment, then use a larger amount of I/O. When doing this I/O you want to try to eliminate:

• Hard disk controller layer cache effects
• Hard disk layer cache effects
• OS level cache effects
• Effects of disk activity from other VMs

All current GoGrid nodes have caches in the storage layer. These are designed to be robust and to absorb burst of write activity. These caches are sufficiently large though that if you do repetitive small I/Os what you end up measuring in the performance in pulling this data out of the storage layers caches, not from the storage itself.

To avoid OS level cache effects use ‘direct I/O’. High performance applications and databases tend to use this internally for similar reasons (because they want to avoid OS level cache pollution and do their own caching). Oracle is probably the most obvious example here.

Testing Performance

On a ‘small VM’ located on a fairly busy node:

[root@foo ~]# dd if=/dev/hda bs=10M of=/dev/null iflag=direct count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 3.50983 seconds, 299 MB/s

[root@foo ~]# dd if=/dev/hda bs=10M of=/dev/null iflag=direct count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 3.06811 seconds, 342 MB/s

[root@foo ~]# dd if=/dev/hda bs=10M of=/dev/null iflag=direct count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 2.14147 seconds, 490 MB/s

That’s using enough I/O to minimize noise from other VM activity and large enough to avoid hitting cache effects.

If the I/O load is small enough you can hit storage layer cache effects:

[root@foo ~]# dd if=/dev/hda bs=10M of=/dev/null iflag=direct count=10
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 0.116491 seconds, 900 MB/s

[root@foo ~]# dd if=/dev/hda bs=10M of=/dev/null iflag=direct count=10
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 0.16058 seconds, 653 MB/s

[root@foo ~]# dd if=/dev/hda bs=10M of=/dev/null iflag=direct count=10
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 0.115701 seconds, 906 MB/s

While this is a fairly contrived example, it’s useful in other ways because it shows you can get very good burst throughput (consider a database updating a few thousand pages).

A larger memory instance (where average performance should be a lot better).

Sustained (large) IO:

[root@ubdev1 ~]# dd if=/dev/hda bs=10M count=100 of=/dev/null iflag=direct
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 1.80415 seconds, 581 MB/s

[root@ubdev1 ~]# dd if=/dev/hda bs=10M count=100 of=/dev/null iflag=direct
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 1.70448 seconds, 615 MB/s

[root@ubdev1 ~]# dd if=/dev/hda bs=10M count=100 of=/dev/null iflag=direct
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 1.6799 seconds, 624 MB/s

Burst (small) IO:

[root@ubdev1 ~]# dd if=/dev/hda bs=10M count=10 of=/dev/null iflag=direct
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 0.105183 seconds, 997 MB/s

[root@ubdev1 ~]# dd if=/dev/hda bs=10M count=10 of=/dev/null iflag=direct
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 0.089827 seconds, 1.2 GB/s

[root@ubdev1 ~]# dd if=/dev/hda bs=10M count=10 of=/dev/null iflag=direct
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 0.090264 seconds, 1.2 GB/s

Don’t take my word for any of this. Try it out. If you’re really bored graph I/O performance vs I/O size and you’ll likely see a step function with a soft edge that will give you some idea of what the storage system is capable of and the degree of I/O variation.

Bottom Line

It’s great that people are kicking the tires of various clouds, but let’s be careful to make sure our testing is rigorous and makes sense for the environment.  If you have questions about how to measure performance on clouds, please send them to us.  Or if you’re a performance and virtualization system guru and have some knowledge to share, please do so.

We always want to improve our cloud and take seriously any feedback that shows a real problem, but in this case the test needs tweaking, not GoGrid.

Amazon’s Web Services (AWS) is not the only way to build scalable Cloud Infrastructures.  There are two emerging methodologies for constructing Infrastructure-as-a-Service (IaaS) AKA “Cloud Infrastructure Services”.  The first is what we call “cloudcenters”, which are essentially datacenters in the sky.  The second is what we call an “Infrastructure Web Service”.  GoGrid was one of the pioneers for cloudcenters, while AWS largely created the second model.

Understanding IaaS means looking closely at these two approaches.  Clearly the notion of cloudcenters embodied by AWS competitors such as ourselves, FlexiScale, ElasticHosts, AppNexus, and others is important.  My colleague, Michael Sheehan, will go into more depth on how we think this distinction modifies his earlier Cloud Pyramid (right) in a follow-on blog posting to this one.

Infrastructure Cloud Models

Understanding these two approaches is important because it directly affects your selection of a Cloud Infrastructure provider.  These two models highlight a difference in core infrastructure and in target markets. Cloudcenters provide a direct equivalent to traditional datacenters and hence are usually more desirable for IT staff, systems operators, and other datacenter savvy folks.  Infrastructure Web Services on the other hand are more analogous to Service-Oriented-Architectures (SOA), require significant programming skills, and are much more comfortable for software developers.

Infrastructure Web Services

I’m going to assume for this article that you are somewhat familiar with Amazon Web Services (AWS), but I’ll briefly re-cap.  AWS provides a number of different ‘Web Services’ that can be consumed individually or put together to support different kinds of applications, usually a batch processing or web application of some kind.  These services include:

• Object-based file storage via Simple Storage Service (S3)
• Servers on demand via Elastic Compute Cloud (EC2)
• Block storage on demand via Elastic Block Storage (a part of EC2)
• Distributed database functionality via SimpleDB (SDB)
• Content distribution and edge-caching via CloudFront
• Messaging & queuing via Simple Queuing Service (SQS)
• Payment processing via Flexible Payment Services (FPS)
• Billing & re-packaging of the above services via Amazon Dev Pay

This is a robust ecosystem of services from which you can use any or all in order to build your application, getting the traditional benefits of Cloud Computing such as self-service, pay-as-you-go, and massive scalability.

Unfortunately, every service above is based on an Amazon standard, not an industry standard.  S3 is not accessible via CIFS or NFS.  EC2 provides Xen hosting, but image management and storage is completely custom.  SQS does not use any standard queuing or messaging protocols such as JMS or STOMP.  SimpleDB now has an ‘SQL-like’ interface, but is essentially a 100% ground up creation of Amazon.

A major advantage of using the Amazon approach however is that greenfield applications developed from scratch have a very powerful set of vetted, scalable services, that can be used to build that application.  This means avoiding the intrinsic and extrinsic costs associated with deploying a separate queuing or database system.

The alternative, of course, is to use the same tools, paradigms and standards that you deploy in an industry standard Enterprise datacenter today.

What’s a Cloudcenter?

Cloudcenters are that solution.  They provide the same kinds of tools that all datacenter and server operators are already accustomed to, but with all the traditional advantages of cloud (i.e. self-service, pay-as-you-go, and scalability).  Instead of creating completely new paradigms, cloudcenters are a methodology by which you, the customer, can have a virtual datacenter hosted ‘in the sky’.  Each virtual datacenter, a ‘grid’ in GoGrid terminology, is hosted in isolation from other customers, in a cloudcenter as shown in the diagram below:

Your grid is akin to a traditional datacenter whereby you have all of the regular components you expect such as hardware firewalls^[1], hardware load balancers, network storage (NAS or SAN^[2] ), virtualized servers, dedicated networks (VLANs), and the option for physical servers for workloads that should not be virtualized.

Logically, a grid looks like:

As you can see, this looks very much like a traditional datacenter infrastructure.

Traditional Datacenters and The Cloud

Just briefly I want to highlight what is in a ‘traditional’ datacenter.  These are all built in the same way.  The following diagram highlights the typical components in a datacenter and their relative dependencies on each other.  (This diagram isn’t perfect, it’s simply meant as a talking point.)

Cloud Infrastructure providers are in the business of providing you the equivalent functionality in the cloud using their scale for cost-effective service delivery.  They must also package this functionality to provide you a high level of control as it’s no longer your datacenter.  Control comes through a user interface (GUI), programming hooks (API), transparency (SAS70 audits), and performance and reliability guarantees (SLAs).

Cloudcenters focus on making your Cloud Infrastructure look very much like infrastructure you already have or are already familiar with, while Infrastructure Web Services ask you to embrace a new paradigm.

Cloudcenter Advantages

Besides the obvious advantage of ‘looking like’ your current datacenter, cloudcenters allow for strategies like using the Cloud for off-site disaster recovery.  It will be much easier to model a copy of your current datacenter to a cloudcenter than it would be to model a copy onto a Infrastructure Web Service.  There are many other advantages to the cloudcenter approach such as:

• Servers meant to be reliable not ‘unreliable-by-design’ on commodity hardware
• Cloudbridging^[3] will be much easier between dedicated VLANs on each side using IPSEC VPNs
• Cloudcenters will look more and more like each other over time, while there will probably never be another AWS
• Datacenter software tools will ‘just work’ with datacenter & cloudcenter standards
• User Interface for controlling the entire datacenter including firewalls and loadbalancers, not just servers^[4]
• Let your cloudcenter provide core infrastructure services like DNS, DHCP, NTP, and image management

Cloud Choices

Ultimately it’s great that there is so much choice when selecting a Cloud Infrastructure provider.  Both Amazon and GoGrid provide compelling solutions.  We believe that GoGrid, the very first public cloudcenter, is the Cloud of choice for sysadmins, operators, and Enterprise IT staff who need infrastructure that looks just their current datacenter infrastructure.

In the future we’ll blog in much more detail contrasting and comparing these two approaches to building Cloud Infrastructure services.

1. To be released in 1Q09 [↩]
2. Scheduled for 2009 [↩]
3. The ability to transparently connect you internal datacenter to your external virtual datacenter or cloudcenter [↩]
4. Imagine VMware VirtualCenter++ [↩]

Hi, my name is Randy Bias, the new VP Technology Strategy at GoGrid.  As the new year approaches, I’m happy to make my first post on this blog.

I haven’t ever provided a New Year prediction list before, so I hope you will indulge.  As the newest member of and the technology visionary on the GoGrid executive team I’m pretty excited to toss a few predictions into the ring for 2009.  Hope this inspires more cloud conversation.  Find more on my thoughts around infrastructure and cloud computing on my personal blog.

1. Cloud-Oriented Architectures (COA) becomes much better understood
   De facto standards drive the adoption of cross-cloud, loosely-coupled, distributed web applications that are connected by REST interfaces.  The community at large comes to understand that this new category of applications are Cloud-Oriented Architectures (COA) and differ from SOA in being deployed on clouds, aware of clouds, and built using grassroots-derived standards instead of top-down standards like SOAP & WS-*.
2. No Cloud Standards Emerge
   Despite hype and hope, no new top-down derived cloud ‘standard’ emerges.  Some forward thinking providers do move the ball forward by opening their platforms and hints of potential standards start to develop by widely embraced, but grass roots developed standards and APIs.
3. Big Iron still has no clue
   The Big Iron folks (Dell, IBM, Sun, and HP) continue to flail at offering ‘cloud’ offerings because they can’t take their focus away from hardware.  Nothing real develops from those folks except failed attempts at ‘standardization’ and cloud offerings.
4. Small and Medium Enterprise (SME) look to clouds & DR
   Economic reality forces the hand of SMEs looking to shave IT costs.  More and more dip their toes into the water, moving their non-production and elastic compute needs to ‘the cloud’.  Disaster Recovery becomes the ‘killer app’ for forward thinking SMEs who want to minimize their exposure while maximizing dollars saved.
5. Broad testing of internal clouds by F500
   Fortune 500 widely tests internal cloud systems using VMware vCloud, EUCALYPTUS, OpenNebula, and related offerings.  Small scale tests, but enough to get a flavor.
6. Content Distribution Networks (CDNs) wake up and realize their business model is at stake
   CDNs finally figure out that as clouds go global the primary barrier to entry for the CDN business, foreign real estate deals, disappears allowing customers to roll their own and a flood of *more* small upstarts enters the already crowded market.  Smart CDNs turn into global cloud providers, further accelerating adoption, or remain clueless and are squeezed from both sides.
7. Hybrid clouds come of age: scale-out on virtual servers and up on iron
   Clue finally sets in that virtualization != clouds and multiple major cloud vendors provide combined virtual+physical server solutions, on-demand just like any other cloud computing offering.  The new hybrid model sets Web 2.0 folks on fire escalating up take for folks reticent to re-engineer for massively distributed databases.  Instead, using big iron for scaling up the DB becomes the de facto solution for anyone who cares more about getting to market and less about faux ‘scaling issues.’[1]
8. Clouds enter Asia
   AWS goes to Asia or a credible competitor arises there capturing mind share and further expanding the global reach of cloud computing providers.  Conversely, clouds entering Asia realize that it’s a fragmented market with expensive intra-country bandwidth, making cost effective traction difficult on a cross-Pacific Rim basis.

 

---

1. I would be remiss if I didn’t mention that we’re leading the charge with one of our latest offerings, Cloud Connect, that allows cross-connecting dedicated servers to your GoGrid cloud.