Archive
ETech 2007 SmugMug Amazon Slides are Up!
My slides from ETech 2007 about Amazon’s Web Services, especially S3, are up in PDF form.
Holler if something isn’t clear, but hopefully this’ll give anyone who couldn’t make it some good insight into what works and what doesn’t with S3 here at SmugMug.
Enjoy!
ETech Amazon S3 slides are coming
I think they’ll be up later today, I’m just trying to put some of my speaking notes into them, too, so you’re not left wondering what each bullet point means.
So subscribe (see the right sidebar) or come back later.
Sorry they’re not up yet! 🙂
UPDATE: They’re up!
The Enterprise Linux problem
I was talking with a friend last night about commercial OSes, specifically Linux and Solaris. It dawned on me that those of you in the midst of your own startups would find our experience useful. Sorry I didn’t think about blogging this earlier.
First of all, I love open source. I can’t imagine what it would be like to have a major OS problem and not have access to the source code and a community with access to it as well. I love the idea of building companies on open source and then using the success of that company to give back to the people and the community that made it possible.
There are lots of ways of contributing back, but the low hanging fruit for a tiny company like ours is simply buying service and support from one of the open source players, like MySQL or Red Hat. It’s a win-win, theoretically – they get funding to continue to build and test great software, and you get the service and support you need to grow your business.
We’ve been through a few commercial open-source OS distributions at SmugMug. All the gory details after the jump.
Amazon S3: Conferences (ETech, Web 2, MIX07)
ETech is next week, and I’m doing a session called Scalability: Set Amazon’s Servers on Fire, Not Yours. I’m not paid by Amazon or anything (quite the contrary – I’m paying them a lot of money), so this will be a real look at them from someone in-the-trenches. It’ll mostly focus on S3, both business and technical aspects, but I’ll touch on EC2 and some of the others, too. (I had hoped to be using EC2 large-scale by this talk, but ironically, a hardware vendor supply issue is preventing me from rolling out that software). I’ll leave plenty of time for questions, and you can give me some in advance. I will post slides here.
Web 2.0 Expo is coming up, and I’m giving a shorter talk there as one of three speakers on Web-scale Computing. Ditto on slides.
And finally, I just got invited to sit on a “build or buy” panel at MIX07. Since I’ve done both, and ended up with a hybrid approach, I hope I’ll have something intelligent to add. Doubt there are slides for this, but if there’s an audio recording or something, I’ll link to it.
I’m attending, but not speaking at, the MySQL Conference and D (the WSJ uses SmugMug for their D photo sharing, which is cool), too, so if you’re going and want to chat about Amazon or SmugMug, you’ll have to find me in the halls. (I’m definitely a “halls” conference goer, though I lean towards “party” now and then).
As always, if you’re wearing SmugMug gear (you can get some older stuff at CafePress and newer stuff at Zazzle or home-make some if you’re adventurous) at a conference I’m at, I’ll give you a comp’d lifetime Pro account at SmugMug. Just come up and say ‘Hi!’
Oh, yes, and I’ll be Twittering at all of them, assuming Twitter doesn’t collapse under ETech’s load. 🙂
UPDATE: ETech 2007 slides are up.
"Truly viral" premium web services?
The Penny Gap post by First Round Capital that I talked about in my last post got me thinking. He had a provocative statement: “I can’t think of a single premium service that has achieved truly viral distribution.”
As I posted on his blog, I guess it depends on your definition of “truly viral” (and maybe even “premium”, which I took to mean “no free offering”), but there were two mentioned in the comments: Netflix and SmugMug.
Obviously, there’s orders of magnitude difference here, but the point remains. Netflix grew initially much by word-of-mouth using loyal customers and those bright red envelopes of theirs. They’re clearly a massive success, clearly premium, and clearly truly viral.
60-80% of SmugMug subscribers are referrals from other SmugMuggers. There’s a gray area where we don’t have enough data, hence the range, but we think it’s on the high side of that figure. We’ve had >100% growth every year, every customer pays, our subscribers are akin to religious fanatics, and we’ve been profitable since year one. Our marketing budget mostly consists of customer service costs – there’s almost no money spent on traditional marketing. (We view Customer Service, or our Support Heroes as we call them, as a marketing expense. I should blog about that sometime…). By my definition, we qualify.
So that’s two. But surely there are more and I’m just having a brain fart. Can anyone think of any others?
"You're not free? You're gonna die!"
When doing interviews or chatting at conferences, I can always tell who was paying attention during the “last” boom-bust cycle here in the Valley. Sadly, most weren’t. They’re the ones telling me my company is on the brink of death because we’re not free and, thus, going to lose the land-grab that’s sure to ensue. According to them, some free site is always about to steamroll over us. 🙂
Imagine my surprise to find an article today on GigaOM entitled Free: a Tactic, Not a Business Model. Is this a sign of the Apocalypse? Is the latest bubble about to burst?
Probably not, but it’s surprising none-the-less. At nearly every tech conference, the vast majority of the business models seem to be either “Grow fast, sell to Google” or “Grow fast, slap AdWords on it”. While these may actually work from time-to-time, I find it strange that no-one seems to think these are risky approaches.
The article is right on the money: free is a tactic. For some, it’s an incredibly good one. For others, it’s not. At SmugMug, we dabbled with free and found to our great amazement that it damaged our product, our brand, and pissed off our customers. I really need to blog about what we learned one of these days….
The article also references another fascinating blog post entitled The Penny Gap. It’s definitely worth a read, too.
Our mantra? You get what you pay for.
Amazon S3: The "speed of light" problem
I was interviewed yesterday by Beth Pariseau for an article about Amazon’s S3 at SearchStorage.com. All-in-all I think it’s a good article that covers some of Amazon’s strengths and weaknesses, but would like to clarify some of my quotes in the article.
I’m quoted as having no read speed issues, but having write speed problems. As is common in articles like this, that’s boiling down a long conversation and much is lost in the translation. 🙂 In reality, Amazon has been blazingly fast for us (both reads and writes), relatively speaking, except for the few times they’ve had problems, which I’ve blogged about before. That particular quote, especially about it being less than a 10th of a second, was my attempt to explain the “speed of light” problem, which applies to both read and writes. Even mighty Amazon hasn’t yet figured out how to transfer data at faster-than-light speeds. 🙂
Basically, we’re in California and Amazon isn’t. This means that when we initiate a read or a write to S3, we’re sending bytes to them and they have to cover, at minimum, the physical distance to Amazon’s datacenters (wherever they are) before anything can be done. Assuming that one of their datacenters in on the East Coast, and assuming we have to read or write from that one occasionally, we’re talking 60-80ms of time just to get bits there and back. No-one on Planet Earth can get around this problem, so it bears consideration when you’re planning for S3 usage.
Obviously, our data in our own datacenters suffers from this problem too – only it’s inches, instead of thousands of miles, to our servers, so it’s almost negligible. But we do have clients all over the world, so the problem is still very real. Our friends Down Under, for example, have to wait much longer for their photos to start drawing than our friends at the Googleplex down the street. If we really wanted to solve that problem, we’d have to build or use a CDN (Content Distribution Network). So far, we haven’t wanted to.
Beth mentions how Bob Ippolito at Mochi Media got better performance in Taipei with CacheFly than with Amazon S3. To me, this seems sorta obvious. To my knowledge, S3 doesn’t have a datacenter in Asia at all, and secondly, they’re not a CDN. Let me say that again – they’re not a CDN. Amazon has their issues they need to overcome with S3, but dinging them for lower performance than a CDN is sorta silly. S3 doesn’t provide web search faster than Google either. See my point?
I’m sure Amazon has thought (or is thinking?) about extending S3 to offer CDN services, but I believe the way Amazon builds these things, it’d probably be a separate service that could be layered on top of S3. They’re into offering building blocks which you can mix & match, not complicated services that do too much. (To any would-be Amazon Web Services competitors reading this, the building block approach is the Right Way to do this.)
Beth’s article is right on the money with regards to data transfer costs, though. S3 currently has two sweet spots: small companies who can’t buy large bandwidth, and companies who need a lot of storage but not a lot of transfers. There are, of course, companies which need a lot of transfers but not much storage (CDNs are probably appropriate here), and companies which need a lot of transfers AND a lot of storage. SmugMug potentially falls into this latter category, but you can imagine someone like YouTube falling into it even more than we do. How they solve the different requirements of different companies will be interesting to watch.
Let me reiterate in case it’s not abundantly clear: I love S3. It’s saved us tons of money. I’m a normal, paying customer – not an Amazon shill. It has problems and growing pains, just like every single other online site or service you can name. It may not be right for you – but it’s certainly right for a ton of us.
I address the “speed of light” issue (and some ways of minimizing it) and the whole “sweet spot” pricing issue on my ETech talk (which I’m still working on). If there’s anything specific you’d like to see, be sure to let me know – I’ll be posting the slides here.
How we hire at SmugMug
Apologies for the lack of an egregious pun in my title. I’m afraid I don’t have the talent for it that Mike over at Atlassian has, but he called me out in his Life is a Hire Way post so I’m compelled to respond. 🙂
First of all, we don’t have all the answers. In fact, I doubt we have many at all. Hiring is *very* difficult. One could argue that it’s the most difficult thing when building a business. If you’re really good at it, you probably get it really right 33% of the time, really wrong 33% of the time, and the rest of the time you sorta get warm bodies in seats. Not exactly stellar.
SmugMug has 20 employees and we’re in our 6th year. So we haven’t done a lot of it here, but we’ve run other companies, and what we have done here has been interesting. So far, we’re well above the percentages I mentioned above. Almost every single one of our hires has been perfect.
Here are some of my insights:
Get the right people on the bus (and get the wrong people off). Jim Collins wrote the best business book of all time, Good to Great, and his chapter on people is prophetic. You’ve got to hire the right people, no matter what. If the right person knocks on your door, and you don’t have a position for her, hire her anyway. Find a way. Then find a seat for her. Keep moving her from seat to seat until there’s one that’s just right – but get her in the door first. Likewise, if someone doesn’t belong on your bus, get rid of them. Keeping them around longer will only damage your company and morale. One of our very best hires, Andy Williams, kept knocking on our door. We knew we wanted to hire him, we weren’t even sure what we wanted him to do, and we knew we couldn’t afford him. We found a way, we found a seat for him on the bus, and he’s had a profound impact on our company.
Hire for passion first, talent second. Given a choice between a world-class, stellar developer who wasn’t in love with our vision and a passionate hacker with limited experience, I’ll take the passionate hacker any day. Talent can be taught and learned, passion can’t. Of course, there has to be a foundation there – hiring someone who’s never touched code to write heavy-duty software isn’t wise. But you get the point. Passion is vital. (And I think you’ll be pleasantly surprised how often talent AND passion go hand-in-hand once you weed out the dispassionate).
Passion for the job, not passion for the company. I’m thrilled when I talk to people who are passionate about SmugMug and want to work here. I truly am. But what I’m really looking for is a passion for your specific job. Do you love taking care of customers and want to be a SmugMug Support Hero? Do you eat, drink, breathe, and sleep datacenter operations and dream of a fully-managed well-oiled machine devoid of human contact? Does improving customer interaction with their priceless memories make your heart beat faster? You’re our type of person. Passion for the company is icing on the cake. Kathy Sierra’s excellent blog has an interesting take on this concept, too.
Look for failure. SmugMug wouldn’t be here today if I hadn’t tanked another company and wasted lots of my own money. I learn best from failures, and I believe others do, too. I don’t want you to hide your failures – instead, I’d rather you highlight them, explain what went wrong, and most importantly, what you learned from the experience. Too much success without struggle breeds overconfidence and can stangate growth. Everyone blows it. Anyone at SmugMug can attest to the fact that I talk to myself when coding and the most common phrase I utter is “Don, you’re so dumb!” You’re going to fail from time-to-time at SmugMug, too. How you deal with it is important.
Getting stuff done. There is definitely a place in this world for people who do pure research. I love those people, and I love the magic they invent. There isn’t a place for that at SmugMug, though. We’re a family of doers. I spend close to 90% of my time thinking about a problem before I start working on it, but when it comes time to pull that trigger, I *move*. There’s no room at a tiny company for people who don’t move at a frantic pace.
Embrace diversity. Another fantastic book is James Surowiecki’s The Wisdom of Crowds. In it, he’ll open your eyes about group thinking and why innovative ideas come from diverse groups. If everyone thinks the way you do, not only will you never have new innovative ideas, but you’ll never be called upon to really think about and defend your ideas. Having spirited debate inside your company is vital to success. When I’m about to propose a new idea at SmugMug, I *know* people in the company are going to attack my idea before I even walk into the room – so I come mentally prepared to do battle. This sounds scary, but in reality, it’s a good thing. I’ll have thought through my idea to the best of my ability, but the 19 other people at the company will also do the same thing, and they’re all a lot smarter than I am. Vastly more important than their intelligence, though, is the fact that they all think differently than I do. They see the problem differently, they see the solution differently. And the idea gets that much better after it’s been beaten around the room.
Hire your own customers. This has turned out to be the single most important hiring decision we’ve made. And it builds upon all of the other items above. Through our strong user community, we’ve been able to identify stand-out people who are already familiar with our product, passionate about helping people use our product, and in love with the idea of working their magic here at SmugMug. The vocal, passionate true believers – you know who I’m talking about, they’ve cornered you at parties and forced you to buy a TiVo – those are the ones to get on the bus. Get them on the bus, re-arrange some seating if you have to, but likely this is their dream job and likely they’re a dream hire. Almost all of our best hires have come from our user base.
So there you have it. There are probably others that’ll come to mind, but those are the biggies.
And since I think this is a great Meme to spread, I’ll tag 5 other people I know and see if we can get some more insight: Jonathan, Tim, Craig, Stewart, and Ashely. (I’m trying to find out if Ashley has a blog or not, she’s a recruiting manager at Google. If not, I’ll replace her with another tag.)
UPDATE: Craig Newmark just got back to me with his answer. He insists his recommendation is to “turn hiring over to someone who’s good at it” and mentions Jim Buckmaster. Alas, Jim doesn’t blog AFAIK, so I can’t pass the tag along. Oh, and Stewart is traveling. Let’s hope Tim, Jonathan, or Ashely come through. 🙂
Why OpenID at SmugMug?
We announced OpenID support last week. I then responded to some comments asking us why we were a provider first, rather than a consumer. Now, I’m answering some more comments basically asking why they should care about OpenID and how it helps SmugMug customers.
Honestly, I had no idea it wouldn’t be obvious how great this is. To me, the picture seems perfectly clear. There are no dastardly designs or secret agendas – to me, it just makes sense. Here’s are a few reasons why:
We are a pay site. Every SmugMug customer pays for the right to share their photos here. They get what they pay for. That comes with both pluses and minuses where identity is concerned, though. On the one hand, we have a much much stronger relationship with our customers than somewhere free like Hotmail, for example. On the other, we have no good mechanism to interact with viewers, who don’t and shouldn’t have to pay (or even sign up) to see their friends’ photos.
Let’s talk about the pluses first. There’s a much higher level of trust and respect between the customer and SmugMug than a free email provider. They feel secure in knowing that we treat their data carefully and with respect. They consider SmugMug to be their home (or at least a major part of their home) online. They strongly identify with the brand and even more strongly identify with the fact that their memories are stored and shared from our servers. They identify with us.
Do you see where I’m going with this? While everyone has multiple identities online, from email to IM, blogs to photo sharing, the ones where there is a volume of priceless content, such as their photo-sharing site or blog, are the ones our customers identify with the most. Email addresses are “less permanent” since they’re free, easy to forward, etc. Ask your typical passionate Flickr or SmugMug customer, though, and they’ll tell you about their passion for their photos and the pain and anguish it would cause them to move or if the service died. Note that not everyone falls into this category – but those passionate about photo sharing *do* fall into this category, and that describes every SmugMug customer.
Further, I believe the customer should get to choose which site they identify with most. I’d hate to limit them to only their email provider if they happen to hate their email provider. Just like everyone resonates with different brands of cars, jeans, computers, music, etc, they also resonate with different sites. Leave the power in the hands of the customer – let them choose their own identity.
Now, let’s talk minuses. Since there are no free accounts at SmugMug, we can’t interact as well with our viewers. They’re allergic to setting up “yet another account,” something I resonate with, or even passing over their email address. I completely get that – it really really sucks when you go to view someone’s photos at KodakGallery or Shutterfly and they demand your email address so you can get spammed till the end of days. It also sucks when you want to leave a comment at Flickr but can’t without signing up for a Yahoo account. What a pain.
OpenID goes a long way towards solving some of these problems. Comments can now be far more spam-free since identity can be verified, yet the commenter doesn’t have to go through the hassle of signing up for yet-another-account. Access controls to photos and galleries can be specified by the owner of the photos in such a way that sensitive data (like email addresses or passwords) no longer has to be exchanged. Even if we wanted to, SmugMug couldn’t spam someone using their OpenID to leave a comment or view a photo. That’s big – I hate giving my email address out to sites because so many of them *do* spam, you’re never sure which ones are the “good guys” like we are.
OpenID isn’t perfect. There’s no trust here – just identification. There’s still no complete single sign-on. There are issues with dangling stale IDs being left around. Consumer education is going to be interesting. But it’s still a huge step in the right direction. Just verifying that someone has an identity somewhere online gives you the ability to make your apps richer, regulate abuse more easily, and generally improve the user experience.
What’s not to love?
More SmugMug & OpenID
I hope you’ve heard we now support OpenID. If not, now you have. 🙂
A commenter asked politely (after deleting his first rant) why on earth we were being a provider, when being a consumer was more important. I completely disagree, and here’s why:
We should be both. And we will be. But I believe in releasing features quickly and incrementally, so I had to choose one or the other. I chose being a provider. Why?
You can’t sell lightbulbs to those without electricity. You can’t sell gasoline cars to those without access to gas stations. You can’t consume OpenID if there’s no-one who has an OpenID.
Up until AOL’s announcement, LiveJournal/SixApart was the best (and only?) source of OpenIDs on a reasonable scale (100,000+ users), as far as I know. That’s really not very many users. I figured throwing our 100,000+ into the mix would help. I certainly hope it has. AOL, though, threw 63,000,000+ into the mix – so now we’re even closer to a critical mass.
This is a chicken-or-the-egg problem, and I believe you need lots of providers with established user bases to jumpstart it. I’m glad there are tons of brand-new OpenID providers who’ll give you an ID for free – but I believe the real make-or-break metric will be existing services enabling huge chunks of people in one fell swoop.
We just swooped. 🙂
On a side note, while I rarely whine about our lack of coverage (generally, I believe the best way to get coverage online is to buckle down and improve your product), I was surprised to see our OpenID news fall on the net in complete silence. More than 30,000 people read my blog on Friday, but not a single blogger or news source I’m aware of picked it up. Digg can announce that they’ll have support “later this year” and get tons of coverage but we enable more than 100,000 people overnight, making us the 3rd largest OpenID provider on the planet, and no-one cares?
We didn’t do this to get coverage, we did it because I’m a geek and I think OpenID rocks. But a little, tiny bit might have been nice. I guess my “improve the product” theory doesn’t work after all. Can someone enlighten me as to what I’m doing wrong?
Oh, and I’m sorry for the whining. I’ll try not to let it happen again. 🙂
SmugMug's Don MacAskill 