Amazon S3: The "speed of light" problem
I was interviewed yesterday by Beth Pariseau for an article about Amazon’s S3 at SearchStorage.com. All-in-all I think it’s a good article that covers some of Amazon’s strengths and weaknesses, but would like to clarify some of my quotes in the article.
I’m quoted as having no read speed issues, but having write speed problems. As is common in articles like this, that’s boiling down a long conversation and much is lost in the translation. 🙂 In reality, Amazon has been blazingly fast for us (both reads and writes), relatively speaking, except for the few times they’ve had problems, which I’ve blogged about before. That particular quote, especially about it being less than a 10th of a second, was my attempt to explain the “speed of light” problem, which applies to both read and writes. Even mighty Amazon hasn’t yet figured out how to transfer data at faster-than-light speeds. 🙂
Basically, we’re in California and Amazon isn’t. This means that when we initiate a read or a write to S3, we’re sending bytes to them and they have to cover, at minimum, the physical distance to Amazon’s datacenters (wherever they are) before anything can be done. Assuming that one of their datacenters in on the East Coast, and assuming we have to read or write from that one occasionally, we’re talking 60-80ms of time just to get bits there and back. No-one on Planet Earth can get around this problem, so it bears consideration when you’re planning for S3 usage.
Obviously, our data in our own datacenters suffers from this problem too – only it’s inches, instead of thousands of miles, to our servers, so it’s almost negligible. But we do have clients all over the world, so the problem is still very real. Our friends Down Under, for example, have to wait much longer for their photos to start drawing than our friends at the Googleplex down the street. If we really wanted to solve that problem, we’d have to build or use a CDN (Content Distribution Network). So far, we haven’t wanted to.
Beth mentions how Bob Ippolito at Mochi Media got better performance in Taipei with CacheFly than with Amazon S3. To me, this seems sorta obvious. To my knowledge, S3 doesn’t have a datacenter in Asia at all, and secondly, they’re not a CDN. Let me say that again – they’re not a CDN. Amazon has their issues they need to overcome with S3, but dinging them for lower performance than a CDN is sorta silly. S3 doesn’t provide web search faster than Google either. See my point?
I’m sure Amazon has thought (or is thinking?) about extending S3 to offer CDN services, but I believe the way Amazon builds these things, it’d probably be a separate service that could be layered on top of S3. They’re into offering building blocks which you can mix & match, not complicated services that do too much. (To any would-be Amazon Web Services competitors reading this, the building block approach is the Right Way to do this.)
Beth’s article is right on the money with regards to data transfer costs, though. S3 currently has two sweet spots: small companies who can’t buy large bandwidth, and companies who need a lot of storage but not a lot of transfers. There are, of course, companies which need a lot of transfers but not much storage (CDNs are probably appropriate here), and companies which need a lot of transfers AND a lot of storage. SmugMug potentially falls into this latter category, but you can imagine someone like YouTube falling into it even more than we do. How they solve the different requirements of different companies will be interesting to watch.
Let me reiterate in case it’s not abundantly clear: I love S3. It’s saved us tons of money. I’m a normal, paying customer – not an Amazon shill. It has problems and growing pains, just like every single other online site or service you can name. It may not be right for you – but it’s certainly right for a ton of us.
I address the “speed of light” issue (and some ways of minimizing it) and the whole “sweet spot” pricing issue on my ETech talk (which I’m still working on). If there’s anything specific you’d like to see, be sure to let me know – I’ll be posting the slides here.