Archive
The Dark Side of the Flickr Acquisition
You asked for it, you got it: SmugMug is offering 50% off to all Flickr refugees. Just sign up for our free trial using the coupon code flickr and if you like what you see, you’ll get 50% off your first year.
We’re getting some email from ‘Old Skool’ Flickr users asking us if they can get a discount because Yahoo’s making some changes they don’t like. Thomas Hawk has more coverage over on his blog, you can read the Flickr Forums for more reactions, and even check out the Flick Off group (aka the Flickr Accounts Mass Suicide Countdown group).
I’m afraid this is the dark side of acquisition, especially by a company that doesn’t seem to resonate with Flickr’s core passionate users. Every time someone inquires about acquiring SmugMug, we shut them down immediately because we’re terrified of exactly this happening – our user experience being damaged by a parent corporation that doesn’t “get” our customers.
I should be clear up-front, though, that while we’re thrilled you want to check us out – we’re not Flickr and we’re not trying to be. We’re short on social networking and long on advanced features like customization, style, and user interface. To make your photos look great, we are the place.
Give us a shot, though. It’s free to try, after all. And we have a track record of listening very closely to customer feedback and implementing what they want. So if you like most of what we have to offer, but we’re missing something crucial, let us know. We just may build it for you. 🙂
UPDATE: It was just pointed out that there’s a Flickr-to-SmugMug migration tool using our open APIs. I’m sure there are others, and I know there are SmugMug-to-Flickr tools, too, should you decide you don’t like our service after all.
Amazon S3: Outages, slowdowns, and problems
First of all, I’m giving a session on Amazon web services (with S3 being the main focus, with a little EC2 and other service love thrown in) at ETech this year. I’ll post a PDF or something of my slides here when I’m done, but if you’re really interested in this stuff, you might want to stop by. Wear some SmugMug gear and I’ll comp your account. 🙂
UPDATE: I’ve posted a call for topics you’re interested in hearing at ETech or in the resulting PDF. Let me know.
So there’s been some noise this month about S3 problems, and I’ve been getting requests about what we do when Amazon has problems and why our site is able to stay up when they do. I’m happy to answer as best I can, and I’d like to remind everyone that I’m not paid by Amazon – it’s the other way around. I pay them a lot of money, so I expect good service. 🙂 That being said, I think they’re getting too much heat, and I’ll explain why.
First, lets define the issues. During our history with Amazon S3 (since April of 2006), we’ve experienced four large problems. The first two were catastrophic outages – they involved core network switch failures and caused everything to die for 15-30 minutes. And by everything, I mean Amazon.com itself was offline, at least from my network view. (Due to DNS caching issues, even GSLB’d sites can look “down” to part of the world while remaining “up” to other parts. I don’t know if this was the case during these two times). We’ve had core network switch failures here at SmugMug, too, and they’re almost impossible to prevent.
The other two were performance-related. Not outages, because the service still functioned, but massively slower than we were used to. In the first case, which happened right as the BusinessWeek cover article hit newstands and during the Web 2.0 Summit, our customers were at our gates with pitchforks and torches. Our paying customers were affected and they could tell there was something wrong. Not good.
The second time, though, was in early January, and our customers had no idea. I emailed the S3 team to let them know we were seeing issues, flipped a switch in our software, and we were fine.
So what was the difference? We’ve been playing with using Amazon in a variety of different roles and scenarios at SmugMug. At first, we were just using them as a backup copy. That provided some great initial savings and a great deal of customer satisfaction as our customers became aware that their photos were safer than ever. As time went on and we grew more confident in Amazon’s ability to scale and keep their systems reliable, though, we moved Amazon into a more fundamental role at SmugMug and experimented with using them as primary storage. The week we started to experiment with that was the first of the two performance issues, and shined a bright glaring light on the downsides of using them in this way. We quickly shifted gears and are now quite happy with our current architecture, both from a cost view and a reliability view.
So what are we doing differently? Simple. Amazon serves as “cold storage” where everyone’s valuable photos go to live in safety. Our own storage clusters are now “hot storage” for photos that need to be served up fast and furious to the millions of unique visitors we get every day. That’s a bit of an oversimplification of our architecture, as you can imagine, but it’s mostly accurate. The end result is that performance problems with S3 are mostly buffered and offset by our local storage, and even outages are mostly properly handled while resyncing after the outage passes. For the curious, this architecture reduces our actual physical disk usage in our own datacenters by roughly 95%.
Further, we also have the ability to target specific Amazon S3 clusters. In January, we noticed that their West Coast cluster seemed to be performing more slowly than their East Coast cluster, even though we’re on the West Coast, so we toggle our primary endpoint to use the East Coast for awhile. This is the switch I mentioned earlier that I flipped, and it worked out beautifully.
Now, though, I think we come to the real meat of the problem. Are we upset about Amazon’s issues? Do we regret using them? Are we looking elsewhere? Absolutely not, and here’s why:
I can’t think of a particular vendor or service we use that doesn’t have outages, problems, or crashes. From disk arrays to networking gear, everything has bad days. Further, I can’t think of a web site that doesn’t, either. It doesn’t matter if you’re GMail or eBay, you have outages and performance problems from time to time. I knew going into this that Amazon would have problems, and I built our software and our own internal architecture to accommodate occasional issues. This is the key to building good internet infrastructures anyway. Assume every piece of your architecture, including Amazon S3, will fail at some point. What will you do? What will your software do?
Amazon does need to get better about communicating with their customers. They need to have a page which shows the health of their systems, and pro-active notification of major issues, a 24/7 contact method, etc. I’m on their Developer Advisory Council, and believe me, they know about these issues. I’m sure they’re working on them.
To put things into perspective, we have vendors which we pay hundreds of thousands of dollars to each year that seem to be incapable of providing us with decent support. Amazon is not unique in terms of providing a great product but average support. If you ask nearly anyone in IT, I think you’ll find that’s far more common in our industry than it should be and not unique to Amazon in particular.
Finally, S3 is a new service and yet remarkably reliable. Since April 2006, they’ve been more reliable than our own internal systems, which I consider to be quite reliable. Nothing’s perfect, but they’re doing quite well so far for a brand-new service. Oh, and their services has also saved our butts a few times. I’ll try to write those up in the near future, too.
Other Amazon articles:
See you at ETech!
Server Analysis – Where's IBM?
Ok, so I’m sorta dumb. For whatever reason, I sorta think of IBM as being a “huge solutions” company where you basically have IBM come in and provide an end-to-end solution for incredibly difficult problems. I don’t think of them as selling service piecemeal and competing with Sun, HP, Dell, etc. But clearly, I should.
I’m sure it’s me being dumb, but maybe there’s some IBM brand positioning at work too? Who knows.
So if there’s anyone from IBM reading this, and you’d like to get in on the action, please contact me. We’ll try the cold-calling route, too, but it’s always nice to talk to someone with context.
Server Vendor Analysis So Far
From my Amazon S3 posts, I know plenty of startups and small companies read my blog, so I thought I’d give you some insight into how our search for a new server vendor is panning out.
It’s interesting, because this space is vastly different than it was even a few years ago. The core hardware (CPU, RAM, disk, networking) is all identical, so the companies are starting to compete on service and differentiation in other ways. That’s good for all of us and very similar to what’s happening in the enterprise software space with open source.
For us, our big hot buttons are:
- Great lights-out management, preferrably over network rather than serial.
- Good order turnaround time (7-14 days).
- Good support contracts (24×7 support, 4hour replacement)
- Price competitive. We’re not necessarily looking for the lowest price, but we expect our final choice to be in the ballpark with the others. We’ll pay a bit of a premium for better management or service or something tangible like that.
On with the show:
Our current main vendor for x86 gear, they’ve finally swung into action and have a good plan of attack for solving our crashing problems. No immediate results, but I’m a developer and an engineer – I understand the need to monitor the problem, gather data, and narrow in. Good stuff.
I’ve been happy with Rackable for most of 4 years, but in the last year they’ve been very slow to get us new hardware and getting some movement on this particular problem was tough in the beginning. I realize, and have always realized, that Rackable isn’t Sun/HP/Dell, but that was sorta the point – when we were tiny, getting attention from the big guys was tough. Now we’re small, rather than tiny, and getting their attention is proving to be tough.
Another knock against them is their management modules are serial based, rather than network based, which we’re growing tired of.
We have at least one HP employee in the servers group who’s a passionate SmugMug customer. Big points there, in my book, since that means we have extra avenues if and when we have problems.
Another *huge* win for HP is their Integrated Lights Out 2 (iLO 2) product. All the vendors offer basic integrated management on some level, but iLO 2 seems to be head-and-shoulders above the competition. It can share a network port with the OS, simplilfying our network architecture, including VLAN support. It does remote full KVM, remote device (DVD/CD/floppy) attachment, and a million other amazing things. Given we’re a small company, strong remote management is a huge bonus for us.
On the downside, our passionate SmugMugger isn’t a sales guy, and the sales contact hasn’t moved as fast as we would like, given our situation. We’re a small company, so I’m not surprised, but Sun and Dell moved faster. We still don’t have much insight into whether there’s an offering at HP that’ll serve this particular need. Finally, HP seems to want to set up a meeting before getting the ball rolling. Personally, I’d prefer to verify that the products and prices are in the right ballpark before spending time organizing and attending a meeting.
No-one at Dell came out of the woodwork in response to my blog entry, but they were easily the fastest on the draw getting a quote (in our preferred format, PDF, no less) in our hands with no mess and no fuss. I think we just cold-called them and they whipped something up fast. Big points, in my book, since it lets us get a feeling for how competitive both their hardware and pricing is.
I believe, though, that their lights-out management is an add-in card, rather than built-in, and it certainly doesn’t share a network port or anything. Dang.
Quite a few SmugMuggers over at Sun, which again, gives me the warm and fuzzies. Nothing quite like having contacts in the company who use, love, and count on your product. Sun also wanted to set up a meeting, rather than whipping out a quick quote, but they were extremely fast and proactive about it, rearranging their schedules and bringing along some big guns. Not only did this flatter us (given our small size), but I think I’m beginning to understand why they’re profitable again and why I’m starting to think that trend will accelerate: they’re trying to give love to small startups because they know some of them will grow into huge companies. If they build a strong relationship early on, the theory goes, they’ll have their business forever. I happen to think this is a very good strategy. There have been some hiccups, as Matt Mullenweg points out, but Jonathan Schwartz responded in a way that I wish every CEO would. Hopefully they work the kinks out.
I love, too, that Sun is changing and innovating once more: embracing x86 & Linux is huge, ZFS is the coolest filesystem ever, and they’ve had an unusually high volume of clever product announcements this year (Thumper, Black Box, Sun Spots, etc).
Yesterday, too, I got a first-hand glimpse of how Sun’s employees feel about Jonathan, their new CEO. They were glowing, speaking in reverent terms about him. That bodes well for their new direction and future.
Alas, though, I have some past experience with their lights-out management, and while it’s adequate, it’s not nearly as good as HP’s. It can’t share a network port and doesn’t offer the other advanced features HP has. At least it is network-based, rather than serial (I believe it does both, actually, which could be a boon to some people).
UPDATE: Dave pointed out in the comments that Sun’s Opteron management lets you have graphical console, keyboard, mouse, DVD, floppy, and ISO support. Plus, it shares one of the GigE ports. I’ve only used the non-Opteron boxes, so I’m probably both out of date and not used to what the architecture differences are. I could have sworn, though, that Sun told us yesterday, in person, that it doesn’t share the network ports. Anyway, sounds like Sun’s doing a better job on management than I initially gave them credit for.
The bad news
The specific platform we want (two CPU sockets, sixteen DIMM slots) doesn’t seem to be readily available from HP, Dell, or Sun. We’re going back-and-forth to make sure, but none of the systems published on their sites seem to offer this. Hopefully someone can deliver something. Rackable, however, does – they’re just crashing for us at the moment.
UPDATE: Sun has one too, the Sun Fire X2200. I just somehow overlooked it. Game on!
In case you’re curious, it’s because 2GB DIMMs are at a price point similar to double 1GB DIMMs, so we’d rather have 10 32GB boxes than 20 16GB boxes for this application.
At this point, I think Sun wins on service & attention while HP wins on product. Which we’ll choose, I still have no idea. 🙂
Have any experience, good or bad, with any of these vendors? Let me know.
Kudos to Jonathan Schwartz
This is a great blog post from Jonathan, CEO at Sun. If more companies were like this, the world would be a better place.
Just imagine if your cell phone provider, for example, actually cared about whether you were happy and whether they were delivering good value to you, their customer? Or how about your broadband provider?
I know, I know, we’d all die of heart attacks from the shock. 🙂
I have to say, I was pleasantly surprised by Sun’s response to my detailed review of the Sun Fire CoolThreads T1000 server and our interest in buying them. Sun assembled a great team of engineers and spent some time with us trying to figure out why our application wasn’t performing up to snuff on their hardware.
For those of you who are still wondering what’s up with that, I’m the one dropping the ball, not Sun – I’ve just gotten swamped. But I’m still interested, so as soon as I come up for air, I’ll try to get more hours put into it. Last time we all worked on it, Sun wasn’t able to get more performance out of it than I was – but they’re anxious to try again and I’m anxious to let them.
In related news, my current server provider, Rackable, seems to have fallen on hard times. We started using them something like 4 years ago, and loved them. Lately, though, their stock is in the toilet, they’ve taken ages to get hardware to us (and a few other major brands I shouldn’t divulge), and worst of all, very expensive brand-new servers from them are failing left and right. Anyone at Dell or HP want our business?
Hello Speed, Beauty & Brains – Goodbye Alexa
Michael Arrington at TechCrunch just broke the story of our latest release and it’s a great write-up. We’re really thrilled to have this puppy out the door and let everyone play with it.
This is a pretty fundamental shift for us, and while I don’t want to give us too much credit, I really think it’s the beginning of a sea change on the web. There have been plenty of apps which launched with 100% AJAX, like GMail, but I can’t think of any that have yet made the plunge to change an existing, entrenched product with lots of users. I’m sure there have been a few, so forgive me if I overlooked you, but certainly not many – most of the big apps are still HTML driven. But I believe that’s going to change because the customer experience just gets so much better.
Everyone is going to do this. The only question is when? (Ok, two questions: And who will be left behind?)
We’ve been playing with 100% JavaScript/AJAX interfaces like this internally for quite some time, but there were some huge pitfalls that kept us from actually releasing it. When we finally solved the last hurdle we got really really excited – this was gonna be great for customers. The minor downside is that I expect our Alexa rank to plummet – we’re no longer really doing page views, which I think they track. (We already get unfairly penalized because so many of our customers use their own custom domain names, but this should really do us in). I could really care less from a business point of view – this is good for customers, after all, but the geek in me thinks that’ll be fascinating to watch and see what happens.
The benefits of this release are obvious: the interface is faster, prettier, and smarter. But the pitfalls are less obvious. Here are a few of the biggies:
- Search engines. I know Google’s been testing a more JavaScript-aware version of Googlebot, but how aware it really is is anyone’s guess. Certainly no crawlers I’m aware of do even a marginal job of crawling AJAX pages. But our customers spend tons of time captioning, describing, and keywording the 120+ million photos at SmugMug.
- Backwards compatiblity. We built our URLs from day one to be “permalinks” so they wouldn’t change if you used them in your blog and forum posts. We had to make sure that things still worked going forward.
- AJAX Permalinks. Now we needed new permalinks that describe various pieces of data for browsing SmugMug, but we also needed to keep them short so people could copy & paste easily, so they wouldn’t wrap in emails, etc.
- Stats tracking. Specifically external sources like StatCounter and Google Analytics which only track page views, not JavaScript UI interactions. Our customers, especially the tens of thousands of hardworking Pros who build their photography businesses at SmugMug, expect to still get useful and meaningful statistics on who’s viewing what.
- Browser interfaces. People expect the Back & Forward buttons to work properly, along with History and Bookmarks. Doing so in all three major browsers was thought to be impossible, and we failed many times. We solved this one, and this was the last biggie. I believe it’s an internet first. Jimmy will be updating his blog about exactly how we do it so anyone else can follow suit. It’s good for the web as a whole for this stuff to move forward.
It was an amazing team effort over here to get this thing done, including tons of our customers. GreenJimmy, our resident Web Superhero, especially drove this project long and hard. Hopefully we can talk more about what we did, technically, so others can avoid making the same mistakes we did.
I really have to also give props to the awesome team over at Yahoo! working on YUI. We couldn’t have pulled this off without their library (easily the best JavaScript library around). They did a profile on us just a week and a half ago, but that was before this release. Now we’re even more hardcore with all the YUI stuff. 🙂
Enjoy the release, and just wait to see what we’ve got coming next…. 🙂
Growth + Happy Customers = Success
Discovered over on Thomas Hawk’s blog that my nPost interview had been posted. I know, I know, I did the interview, I should have noticed when it went up, my bad. Thanks Thomas!
I did the interview over the phone a month or two ago, and thought it was a lot of fun. Looks like there are some minor typos in the transcription (or I didn’t speak clearly enough on the phone), but you’ll get the general idea.
I had no idea I ended up with such a great summary sentence (which I re-used for the title of this blog post), but there it is. My secret formula. 🙂
If you want a pretty good insight into what makes me tick and SmugMug hum, go check it out.
PCworld.ca: Letter to the Editor
PCworld.ca had a story about digital photos a few days ago. I was stoked to hear that we were in it. I wasn’t so stoked, however, once I discovered that our involvement was limited to two sentences because we have a “hefty monthly fee” and “lagging print quality,” of all things!
I whipped off a letter to the editor using their form submission thing, and waited for the reply. Since none has been forthcoming, and I’m getting emails about the article, I thought I’d repost the email I sent them here:
—
Hi Paul,
I’m the CEO and Chief Geek at SmugMug, and I couldn’t help noticing that we were unfairly excluded from your review. I’m hoping I can set the record straight.
You mentioned a “hefty monthly fee” and “lagging print quality” specifically, which raised our eyebrows. A SmugMug subscription with unlimited storage is $39.95 per year, or $3.33 per month. Hardly hefty. 🙂
As for the print quality, we’re renowned in the industry for having the best prints around, bar none. We certainly won’t win on price, but you get what you pay for. We have tens of thousands of professional photographers who build their businesses on SmugMug. As you can imagine, when you’re shooting a wedding or selling multi-hundred dollar prints of Yosemite, the print quality has to be pristine.
As a result, we conduct blind “taste tests” of various printers (10 at last count, including all of those listed in your review) every year. What’s more, we publish the results for all to see, and they’re widely distributed on the net. Some of our competitors have personally thanked us for publishing the results, since it helped them select their own printing partners. You can see for yourself:
http://www.smugmug.com/prints/digital-prints.mg
We use a top-of-the-line professional lab in Georgia called EZPrints at the moment, since they’ve consistently won the “taste test” for 3 years running, but we’re committed to print quality at all costs. Ask any of our customers, or try a Google search for ‘SmugMug print quality’, or better yet, order some prints and see for yourself.
I hope that sets the record straight, and by all means, holler if you have any other questions about our service. Hopefully we can make it into the next one!
Thanks,
Don
—
So there you have it. I wish were were fairly compared in the article, but alas, it wasn’t to be.
National Geographic and MacWorld seem to like us well enough, though, and we were even in The New York Times today, so I suppose I can’t complain. 🙂
Not that I really have to remind you, but if you DO email the editor at PCworld.ca yourself, please be polite. They’re only human, just like the rest of us.
Happy New Year everyone! Here’s to a great 2007!
I love our community.
The fact that I played some small part in building a community that comes together like this makes it all worth it.
My heart goes out to Haley’s family and friends.
Flickr far superior to SmugMug?
It sure is – if you’re not our target customer.
Andy Atkinson has a great write-up of some of the ways Flickr is better than SmugMug. And he’s right about lots of it.
I love reviews like this. First of all, SmugMug doesn’t do any competitive research – we just don’t have time. Instead, we listen voraciously to our customers and our todo list is almost exclusively made up of things our customers want us to add, fix, or change. (Sometimes we have to read between the lines, because they don’t always know exactly how to ask for it, but we do our best). Secondly, we’re awash in positive emails and reviews all the time. They’re nice, but they can give us a false sense of security and obscure the things that we really need to work on. Andy’s review nicely shines light on some areas where we’re weak and gives us a little insight into the competitive landscape at the same time. Thanks Andy!
Andy’s review is particularly refreshing because it’s the first one I can remember, either publicly or privately, where his point of view is that Flickr has more features than we do. Given that we release new features multiple times per month, and often once per week, we frequently (daily?) hear the opposite, and it’d be easy for us to assume we had every Flickr feature our customers wanted.
I left him a comment letting him know just how valuable his write-up is to us, and how much I enjoyed reading it, but he has moderation on. So I thought I’d talk about it here, on my blog, in case he doesn’t actually allow any comments.
As I told him, we’re not trying to be Flickr. We love Flickr, often refer customers that aren’t a great fit with SmugMug, and think it’s a great site that addresses a real mass-market need for photo sharing. But that’s not what SmugMug is – we’re not a mass-market brand, we’re not for everyone, and we think we have a very narrow bead on our target. Andy sure sounds like he’s much more of a Flickr customer than a SmugMug customer, so I’m surprised he lasted this long, but he makes some great points about things we should do better, even given our different focus:
- We don’t make it as easy to get your photos AND metadata back out of SmugMug. This one hit close to home because I’m very passionate about treating your photos as if they’re yours – not ours. We try very hard not to be the photo-sharing equivalent of the roach motel, where photos check in and never check out. We make it very easy to get your photos back out of SmugMug (they are your photos, after all, so you should be able to do whatever you want with them), but we don’t make it nearly as easy to get your metadata, like keywords and captions, back out too. Andy’s right on the money here, and I need to do a better job at this. You can use the API, of course, but we should make it easier than that.
- Our Geotagging interface is falling behind. We were first (we actually had two major releases of our mapping & geotagging stuff long before Flickr), but Flickr’s doing it better. We’re aware of it, and it’s on our radar – we just have to finish our next evolution. This sort of back-and-forth leapfrogging will always happen, I’m afraid. It’s the nature of a competitive business. One company does it best for a few months, then another takes the top spot. Back and forth.
- Our statistics could be better. He’s wrong about us not having per-photo statistics (we do), but he’s right that we don’t offer searching and sorting by other criteria, like comments. Doing better, richer statistics is something we’d like to do, and it’s good to see people like Andy calling us out on it.
- Photo books (and other similar items). He mentions QOOP specifically, but the real issue is that we don’t sell photo books (or calendars, greeting cards, etc). We want to, and we’re working hard on doing it (it’s an active project in the company right now, and has been for awhile), and I wish we’d done it by now, but QOOP just isn’t the answer. Their quality level wasn’t even close to our standards, either in terms of the finished product or the shopping cart experience. This is one area where Flickr’s target customers and ours are a big deciding factor – we’d rather not offer a product for awhile than offer something that’s not high-quality. Many of our customers build their businesses on SmugMug, and if we offer an embarassing level of quality, it reflects badly on them. We take that burden very seriously.
He has plenty of other good, interesting points that we’ll have to think about, but many of them are not really SmugMug’s focus, so I can safely shelve them for a later date. All the points above, though, are solid areas we need to work on. They’re core to our business, they’d enhance our customer’s experience, and we’re clearly not executing on all of them as well or as fast as we should be.
Anyway, great review and a good illustration of the differences between our two sites. I love reading stuff like this, so be sure to let me know if you blog about anything similar. We do, of course, read all of our email every day and usually respond in minutes – so keep the feedback coming!
SmugMug's Don MacAskill 