S3 outage – We weren't affected
Amazon S3 had an outage today. First I knew about it was reporters emailing and calling me asking if we were knocked out by it.
We weren’t. No customers reported issues, and our systems were all showing typically low and acceptable error rates. To be honest, I’m surprised.
I wasn’t going to blog about it until I understood why we weren’t affected, but I’m really getting inundated with requests now, so I figured this would be a good way to optimize my time rather than spending all day on the phone. 🙂
We’re researching what happened now, but again, I didn’t know about the outage until after it was over, and I haven’t spoken to anyone at Amazon yet. Until I finish my research and speak with Amazon, I’m not going to speculate on what may have happened or why.
I can say, once again, that we pay the same rates everyone else pays and that, other than some early access to upcoming beta services, we don’t get any preferential treatment that I’m aware of.
Some thoughts, though:
- We expect Amazon to have outages. No website I’m aware of doesn’t, whether it’s Google, Amazon, your bank, or SmugMug.
- I’ve written about Amazon S3 outages in the past, but in the last ~12 months, we’ve only seen a single ~2 minute outage window (on January 22nd, 2008 at around 4:38pm Pacific). We also had one recent fairly major hiccup with EC2.
- Yes, I believe there will probably be times where SmugMug is seriously affected, possibly even offline completely, because Amazon (or some other web services provider) is having problems. Today wasn’t that day. Nobody likes outages, especially not us, but we’ve decided the tradeoffs are worth it. You should have your eyes wide open when you make the decision to use AWS or any other external service, though. It will fail from time to time.
- We’ve done our best to build SmugMug in such a way that we handle failures as gracefully as possible. We can’t cover every case, but I think that the fact that we didn’t experience customer-facing outages today is a testament to that. Again, I want to stress that we do expect Amazon to cause us (rare) outages in the future, and that’s unavoidable, but today we dodged that bullet.
- Amazon’s communication about this has been terrible. It took far too long to acknowledge the problem. Fixing a major problem can take forever, which is understandable, but communicating with your customers should happen very rapidly. Amazon’s culture, internally, is very customer focused, so this is a strange anomaly. I will definitely lean on them some about it, and everyone who was affected should rightfully howl too.
- I’ve asked Amazon repeatedly for an “Amazon Web Services Health” page that shows the current expected state of all their services. Then you can tell at a glance (and even poll and work into your own monitoring) whether any of the services are having problems. Something like Keynote’s Internet Health Report would be a good start, but as Jesse Robbins points out, trust.salesforce.com is the gold standard. This page could also double as a mechanism to let customers know what’s being worked on and current ETAs when there are problems.
I’ll try to post a follow-up about why we weren’t affected when I know more. It’s possible that some of the reasons we survived was due to some of our “secret sauce” and I just won’t be able to say, but I kinda doubt it.
Bottom line: While the outage was certainly a big deal to those affected, I think the bigger deal here is how Amazon handled the outage. They need to communicate better about these mission critical services and their health.
If I didn’t answer any questions you’d like me to answer, please post a comment and/or send me an email. I’ll do my best to respond.
UPDATE 1: I’m not sure why there’s all this confusion, but SmugMug *does* use Amazon as our primary data store. We maintain a small “hot cache” in our datacenters of frequently/recently viewed photos and videos, but there are massive numbers of them that are only at Amazon. This is a change from our initial usage of S3, and the change is based on how reliable they’ve been. Yes, we still consider them to be very reliable even after an outage like this. And yes, I suspect our “hot cache” did at least partially enable us to ride out this issue.
SmugMug's Don MacAskill 
My understanding is that you simple use S3 as backup storage, so as long as you are queueing your syncs you should be able to handle fairly long outages on their side. I have never noticed an image URL on your site that is actually coming from S3.
@Brad:
Sorry for the confusion, but S3 is our primary data store, not just backup.
I’ve worked with S3 a bit. How are you serving up images from a http://www.smugmug.com address? Do you house them locally for a certain about of time until they get committed?
Hey Don – I’m glad that you (and my photos) weren’t affected by this downtime. I would echo your suggestion that AWS implement some sort of status page. How about leading by example, though? In this dgrin post, I recommended creating a Smugmug “status” blog that’s hosted in a different DC than the rest of your infrastructure. To my knowledge, no one responded to that suggestion.
Sure, outages are usually reported in Dgrin – that’s not an ideal situation, though, IMHO. If you implemented a status blog, it would make it very simple for your operations staff to post quick updates as they work through problems as well as enabling customers to subscribe to the blog’s RSS feed and get regular updates that way.
Thoughts?
@erik:
I heard you loud and clear during our last outage, and we now have an offsite location to post updates: http://smugmug.wordpress.com/
We just haven’t had an outage since then to use the blog on – but we will!
@Don:
Very nice. I’m already subscribed.
Thanks and keep up the great work!
@Don:
I have subscribed also… thanks for letting us know about the site.
According to Amazon, only one out of three S3 server locations was affected. We (gladly) did not experience any outages today but we are sure, as you stated, that inevitably one day we will 🙂
Speaking of S3, I was wondering if that article on you EC2 use you hinted at some time ago was coming soon… I’d be interested in hearing how you use it (it seems to me like you wouldn’t have enough resizing load to justify it… but I suppose I must be wrong :))
Well I guess we were lucky enough to have our files at the one that went down. Yippee. Our customers certainly appreciated it. There’s sure nothing like waking up to think all your files are gone to start a day right. Don’t worry about us though our reputation will be fine. About the time it’s back to where it was before this they’ll poop again and it’ll be right back down in the trash.
I’m sure your paying customers will appreciate your sentiments around the fact that Amazon is down less than you would have been if you did your own hosting. Does telling them that really work? If so I want your customers because ours are complete the opposite. Of course yours put up images for fun. Not ever customer of S3 is doing this for jollies. Not to lessen what you are doing, but some things are more mission critical. Maybe shame on us then for having an alternative. Have you looked into this Don? Why not send images another S3 like service. Then at least a simple DNS change or URL filter could be used. It doubles the cost of storage but if Amazon ever goes down for say a day it’s going to be hell of a bad day.
Don, the reason that people are under the impression that S3 is only secondary storage for Smugmug is this post from you on 12 August 2006:
@SHG:
That’s so strange to me. Do people still really believe that humans will never fly because there’s a lot written about how impossible it is a few centuries ago? 🙂
That blog post is ancient history. The Wright Brothers showed us how to fly, and SmugMug has moved on with S3. 🙂
Guess I’d better update those old posts, but that seems silly to me that people wouldn’t do a little research rather than trusting some old blog post….
No wonder I had some problems uploading at times on Friday (Aus) and there was an incident I uploaded successfully 5 pics and they didn’t appear in the gallery.
Keep us posted. Thanks
@Arg: if you’re hosting mission critical data on something that openly states they only have 99.9% uptime, then you should be fired for incompetence. 99.9% uptime is NOT mission critical. If the service was advertising 5 9’s, you might have something to gripe about. Right now, you’re basically complaining that you’re having troubles getting a screw out of a board with a hammer.
@Don: If everything is on S3 now, how are there 3 copies of every photo? If it’s all sitting on S3, I guess that scares me. You’re basically entrusting them to never have catastrophic data loss… and I guess I don’t trust any one company that much 😀
@TimC: S3 has 3 different server locations.
@Jorge:
they could have 10 different server locations, if they have a failure in their software which results in data loss, it doesn’t really matter how many different locations the failure occurs at.
@TimC: you asked “how are there 3 copies of every photo?”
^^which would be why I asked originally when it was quoted if they are only using S3 now as I read it to be.
“That’s so strange to me. Do people still really believe that humans will never fly because there’s a lot written about how impossible it is a few centuries ago?”
That was the reason why I choose Smugmug and I can not find the announcement about the change :(.
It ultimately means that I am now paying only for a gallery and Smugmug resells S3 storage to host my photos.
Are you planning any discount/light packages with external S3 accounts? (I mean I am purchasing only the use of the software from Smugmug and the storage from S3)
thanks you
I have subscribed
Да таков наш современный мир и боюсь с этим ни чего невозможно поделать:)
Хороший блог
Думаю эта методика уже не актуальна, есть более новые методы.
Спасибо, очень заинтересовался, будет ли еше что то подобноее?
Что то новенькое, пишите есче очень нравится.
Отличные новости, так держать, удачи в будущем.
Готов разместить вашу сылку у себя на сайте, очень понравился ваш материал.
Стоит ли ждать обновления?.
Могу предложить много инфы по данной теме, нужно?.
Как часто публикуете новости по данной тематике?.
Супер. Спасибо, так давно искал этот материал. Ну просто респектище автору. Никогда не забуду теперь
Я люблю етот блог. Вот я тоже как то об этом думал
Информация подобрана очень успешно, когда будет обновление?
Эх етот кризис все нам портит
Собственно говоря я так и думал, вот про что все толдычут. Мда этож надо так
Очень заинтересовал материал. Что за источник? Я бы еще почитал про сий материал
Кто может мне помоч подробнее в етом разобратся?
Я тоже в блоге про такое пишу, только на тему кино
Поздравте меня у меня родился сын!
Ребят, кто подскажет где можно подробнее узнать.!
Поздравляю всех с наступаюшим праздником!!!
Вах. Какой блог хороший. Хачу постить у вас новости. Как можно данное реализовать. Спасиб
Поздравляю всех, с наступающим праздником, желаю всего наилучшего.
Хорошо что я увидел эту информацию. Очень полезно читать ваши посты
Вы всегда публикуете только лучшую информацию. У вас просто супер блог. Спасибо
Лучшая инфа которую я прочел за последние несколько дней. Очень актуально. Спасибо
Приличная новость. Буду постоянно читать ваши новости
Подскажите пожалуйста, конкретнее где можно посмотреть данную тематику, и подробное описание?
Add your RSS in my reader.
Very nice site!
U vas traffic na saite konchilsia
Hmm, very cognitive post.
Is this theme good unough for the Digg?
Очень хороший пост! Спасибо за проделанную работу!
Отличный у тебя сайтик. Есть свой стиль. А я вот делаю как попало, и не читает меня никто, кроме лохов.
Поздравляю с 8 марта всех читательниц blogs.smugmug.com!!!
Love it! You got me so excited to get one and start shooting video!