Archive
The Dark Side of the Flickr Acquisition
You asked for it, you got it: SmugMug is offering 50% off to all Flickr refugees. Just sign up for our free trial using the coupon code flickr and if you like what you see, you’ll get 50% off your first year.
We’re getting some email from ‘Old Skool’ Flickr users asking us if they can get a discount because Yahoo’s making some changes they don’t like. Thomas Hawk has more coverage over on his blog, you can read the Flickr Forums for more reactions, and even check out the Flick Off group (aka the Flickr Accounts Mass Suicide Countdown group).
I’m afraid this is the dark side of acquisition, especially by a company that doesn’t seem to resonate with Flickr’s core passionate users. Every time someone inquires about acquiring SmugMug, we shut them down immediately because we’re terrified of exactly this happening – our user experience being damaged by a parent corporation that doesn’t “get” our customers.
I should be clear up-front, though, that while we’re thrilled you want to check us out – we’re not Flickr and we’re not trying to be. We’re short on social networking and long on advanced features like customization, style, and user interface. To make your photos look great, we are the place.
Give us a shot, though. It’s free to try, after all. And we have a track record of listening very closely to customer feedback and implementing what they want. So if you like most of what we have to offer, but we’re missing something crucial, let us know. We just may build it for you. 🙂
UPDATE: It was just pointed out that there’s a Flickr-to-SmugMug migration tool using our open APIs. I’m sure there are others, and I know there are SmugMug-to-Flickr tools, too, should you decide you don’t like our service after all.
Amazon S3: Outages, slowdowns, and problems
First of all, I’m giving a session on Amazon web services (with S3 being the main focus, with a little EC2 and other service love thrown in) at ETech this year. I’ll post a PDF or something of my slides here when I’m done, but if you’re really interested in this stuff, you might want to stop by. Wear some SmugMug gear and I’ll comp your account. 🙂
UPDATE: I’ve posted a call for topics you’re interested in hearing at ETech or in the resulting PDF. Let me know.
So there’s been some noise this month about S3 problems, and I’ve been getting requests about what we do when Amazon has problems and why our site is able to stay up when they do. I’m happy to answer as best I can, and I’d like to remind everyone that I’m not paid by Amazon – it’s the other way around. I pay them a lot of money, so I expect good service. 🙂 That being said, I think they’re getting too much heat, and I’ll explain why.
First, lets define the issues. During our history with Amazon S3 (since April of 2006), we’ve experienced four large problems. The first two were catastrophic outages – they involved core network switch failures and caused everything to die for 15-30 minutes. And by everything, I mean Amazon.com itself was offline, at least from my network view. (Due to DNS caching issues, even GSLB’d sites can look “down” to part of the world while remaining “up” to other parts. I don’t know if this was the case during these two times). We’ve had core network switch failures here at SmugMug, too, and they’re almost impossible to prevent.
The other two were performance-related. Not outages, because the service still functioned, but massively slower than we were used to. In the first case, which happened right as the BusinessWeek cover article hit newstands and during the Web 2.0 Summit, our customers were at our gates with pitchforks and torches. Our paying customers were affected and they could tell there was something wrong. Not good.
The second time, though, was in early January, and our customers had no idea. I emailed the S3 team to let them know we were seeing issues, flipped a switch in our software, and we were fine.
So what was the difference? We’ve been playing with using Amazon in a variety of different roles and scenarios at SmugMug. At first, we were just using them as a backup copy. That provided some great initial savings and a great deal of customer satisfaction as our customers became aware that their photos were safer than ever. As time went on and we grew more confident in Amazon’s ability to scale and keep their systems reliable, though, we moved Amazon into a more fundamental role at SmugMug and experimented with using them as primary storage. The week we started to experiment with that was the first of the two performance issues, and shined a bright glaring light on the downsides of using them in this way. We quickly shifted gears and are now quite happy with our current architecture, both from a cost view and a reliability view.
So what are we doing differently? Simple. Amazon serves as “cold storage” where everyone’s valuable photos go to live in safety. Our own storage clusters are now “hot storage” for photos that need to be served up fast and furious to the millions of unique visitors we get every day. That’s a bit of an oversimplification of our architecture, as you can imagine, but it’s mostly accurate. The end result is that performance problems with S3 are mostly buffered and offset by our local storage, and even outages are mostly properly handled while resyncing after the outage passes. For the curious, this architecture reduces our actual physical disk usage in our own datacenters by roughly 95%.
Further, we also have the ability to target specific Amazon S3 clusters. In January, we noticed that their West Coast cluster seemed to be performing more slowly than their East Coast cluster, even though we’re on the West Coast, so we toggle our primary endpoint to use the East Coast for awhile. This is the switch I mentioned earlier that I flipped, and it worked out beautifully.
Now, though, I think we come to the real meat of the problem. Are we upset about Amazon’s issues? Do we regret using them? Are we looking elsewhere? Absolutely not, and here’s why:
I can’t think of a particular vendor or service we use that doesn’t have outages, problems, or crashes. From disk arrays to networking gear, everything has bad days. Further, I can’t think of a web site that doesn’t, either. It doesn’t matter if you’re GMail or eBay, you have outages and performance problems from time to time. I knew going into this that Amazon would have problems, and I built our software and our own internal architecture to accommodate occasional issues. This is the key to building good internet infrastructures anyway. Assume every piece of your architecture, including Amazon S3, will fail at some point. What will you do? What will your software do?
Amazon does need to get better about communicating with their customers. They need to have a page which shows the health of their systems, and pro-active notification of major issues, a 24/7 contact method, etc. I’m on their Developer Advisory Council, and believe me, they know about these issues. I’m sure they’re working on them.
To put things into perspective, we have vendors which we pay hundreds of thousands of dollars to each year that seem to be incapable of providing us with decent support. Amazon is not unique in terms of providing a great product but average support. If you ask nearly anyone in IT, I think you’ll find that’s far more common in our industry than it should be and not unique to Amazon in particular.
Finally, S3 is a new service and yet remarkably reliable. Since April 2006, they’ve been more reliable than our own internal systems, which I consider to be quite reliable. Nothing’s perfect, but they’re doing quite well so far for a brand-new service. Oh, and their services has also saved our butts a few times. I’ll try to write those up in the near future, too.
Other Amazon articles:
See you at ETech!
Make video more consumable
As a follow-up to my last post about video online, which was in turn related to the post about Scoble and linking before that, here’s my take on how video online could improve:
I was struck by how much I enjoyed reading Scoble’s rundown of what was in his Intel video. He even included some timestamps of some interesting bits, so you could skip right to it if you’d like. IMHO, this is a step in the right direction. What I’d *really* like to see is yet another step: chapters as separate entities with good, short summaries.
I’d love to see video with great content (I don’t think anyone’s debating whether Scoble gets great content – he clearly does, and lots of it) be available both as the long-form video (in this case, 40 minutes) as well as shorter (1-5minute) chapters that tie together. The chapters would need to be completely separate video files so I don’t have to download the entire 40 minute segment to find the bit that’s important and relevant to me. That’s a biggy so let me elaborate – even if I know the exact timecode for a segment in a longer video, I don’t want to download the whole thing and then jump through it. Instead, I want summaries of the chapters so I can quickly skim through the summaries and watch, say, 10 minutes out of 40 that’s highly focused.
Now, I know this won’t work for everyone. Scoble, for example, is passionate about not editing his videos because they’re conversations and I completely respect that as a viewer, a videographer myself, and an interviewee on his show. But even conversations often have chapters: the business chapter, the competition chapter, the upcoming features chapter, etc. I’m not advocating editing anything more than it’s already been edited – just making it more consumable.
There’s a reason YouTube became so wildly popular, and I don’t think it’s fair to brush the phenomenon off as “that was for fun stuff, but serious video needs to be long.” That’s a load of BS. I consume deep articles on technical subjects all the time, but I often skim for the good bits and jump out of the site to research related items before jumping back in. Both would be made possible by using text-based summaries with hyperlinked chapters as the basis for navigating a given video.
Thoughts?
Videos (and podcasts) suck sometimes
So I already commented on the whole Scoble thing, but I was commenting in general about how linking is usually better than not linking. I think everyone gets that.
But there’s another discussion going on that’s almost as interesting. Paul M. Watson got me thinking with his comment on my original post, which of course, led to me reading his blog. Specifically, his blog entry about how he doesn’t like video. It’s not too hard to find other opinions in like vein, such as Mathew Ingram’s post about video being Scoble’s achilles heel. Just take a peek over at Techmeme and Tailrank and you’ll see there are quite a few discussions, including this one, bubbling up.
I don’t really agree with the specific details (some don’t think his coverage of Intel was that great, others just think video isn’t that great, etc), but I agree partially in spirit. I watch the occasional ScobleShow episode or listen to the occasional podcast – but not often, and not religiously. I read his blog (and dozens of others) almost daily, though. So what’s the difference?
Blogs are massively easier to consume. You can skim them, you get headlines, you have hyperlinks to follow an extra interesting story around the web, etc. The list goes on and on – but what it comes down to, for me, is time. If I’m going to watch a 20 minute video or listen to a 30 minute podcast, I basically can’t do anything else during that time. I also can’t “skip to the good bits” easily.
With text (blogs, articles, reviews, interviews, whatever), those things don’t apply. I can consume blogs in 1-minute chunks of time I have between tasks. I can explore the web to find out more. I can easily find the stuff I’m really interested in.
Don’t get me wrong – I love certain videos and podcasts a great deal. But the signal-to-noise ratio has to be sky-high to get me to invest my time.
Now I’m just gonna sit back and see if any other interesting discussions come up over this furor… I’m sure there will be.
Grab some popcorn, enjoy the show.
Scoble: Throwing himself under busses so I don't have to.
My friend, Robert Scoble, has two great rants up about blogging & linking: Big gadget sites don’t link to blogs followed by Pissing off the blogosphere.
His main point is a valid one – far too many places, whether they be old media (The New York Times) or big blogs (Engadget) or even small bloggers who are afraid people won’t read every word they’ve written, don’t link to external sources.
This is actually a huge deal. The real, true power of the web is just that – it’s a web. Everything can be interconnected, and learning about or researching a subject can be vastly easier online than anywhere else. Using hyperlinks is the very reason content belongs online. If you don’t hyperlink your content, why on earth do you have it online?
People often wonder why “old media” is struggling to find a voice and an audience (and a business plan) for online content. Rarely is the fact that “old media” tend to be stingy linkers mentioned – but I suspect it’s actually a fundamental reason people choose to get much of their content elsewhere.
For me, I’ve never really thought about it in these terms until Robert brought it up. I’ve always tried to link everything and anything, even words like “Google” and “Sun” (surely you know that Google = http://www.google.com and Sun = http://www.sun.com). Why? Because if I were reading my blog, I would want to just be able to click on pertinent details and dig deeper rather than having to open a new window and Google for the subject at hand. It just makes sense. If a word or phrase can be hyperlinked, and it’s more useful that way, it should be.
By the way, this situation also exposes what’s so great about bloggers, especially the big and popular ones – they write about what’s on their mind, often not even thinking about the impact of what they’re saying. Communication is fast, transparent, and emotional.
Preach on, brother Scoble.
Server Analysis – Where's IBM?
Ok, so I’m sorta dumb. For whatever reason, I sorta think of IBM as being a “huge solutions” company where you basically have IBM come in and provide an end-to-end solution for incredibly difficult problems. I don’t think of them as selling service piecemeal and competing with Sun, HP, Dell, etc. But clearly, I should.
I’m sure it’s me being dumb, but maybe there’s some IBM brand positioning at work too? Who knows.
So if there’s anyone from IBM reading this, and you’d like to get in on the action, please contact me. We’ll try the cold-calling route, too, but it’s always nice to talk to someone with context.
Server Vendor Analysis So Far
From my Amazon S3 posts, I know plenty of startups and small companies read my blog, so I thought I’d give you some insight into how our search for a new server vendor is panning out.
It’s interesting, because this space is vastly different than it was even a few years ago. The core hardware (CPU, RAM, disk, networking) is all identical, so the companies are starting to compete on service and differentiation in other ways. That’s good for all of us and very similar to what’s happening in the enterprise software space with open source.
For us, our big hot buttons are:
- Great lights-out management, preferrably over network rather than serial.
- Good order turnaround time (7-14 days).
- Good support contracts (24×7 support, 4hour replacement)
- Price competitive. We’re not necessarily looking for the lowest price, but we expect our final choice to be in the ballpark with the others. We’ll pay a bit of a premium for better management or service or something tangible like that.
On with the show:
Our current main vendor for x86 gear, they’ve finally swung into action and have a good plan of attack for solving our crashing problems. No immediate results, but I’m a developer and an engineer – I understand the need to monitor the problem, gather data, and narrow in. Good stuff.
I’ve been happy with Rackable for most of 4 years, but in the last year they’ve been very slow to get us new hardware and getting some movement on this particular problem was tough in the beginning. I realize, and have always realized, that Rackable isn’t Sun/HP/Dell, but that was sorta the point – when we were tiny, getting attention from the big guys was tough. Now we’re small, rather than tiny, and getting their attention is proving to be tough.
Another knock against them is their management modules are serial based, rather than network based, which we’re growing tired of.
We have at least one HP employee in the servers group who’s a passionate SmugMug customer. Big points there, in my book, since that means we have extra avenues if and when we have problems.
Another *huge* win for HP is their Integrated Lights Out 2 (iLO 2) product. All the vendors offer basic integrated management on some level, but iLO 2 seems to be head-and-shoulders above the competition. It can share a network port with the OS, simplilfying our network architecture, including VLAN support. It does remote full KVM, remote device (DVD/CD/floppy) attachment, and a million other amazing things. Given we’re a small company, strong remote management is a huge bonus for us.
On the downside, our passionate SmugMugger isn’t a sales guy, and the sales contact hasn’t moved as fast as we would like, given our situation. We’re a small company, so I’m not surprised, but Sun and Dell moved faster. We still don’t have much insight into whether there’s an offering at HP that’ll serve this particular need. Finally, HP seems to want to set up a meeting before getting the ball rolling. Personally, I’d prefer to verify that the products and prices are in the right ballpark before spending time organizing and attending a meeting.
No-one at Dell came out of the woodwork in response to my blog entry, but they were easily the fastest on the draw getting a quote (in our preferred format, PDF, no less) in our hands with no mess and no fuss. I think we just cold-called them and they whipped something up fast. Big points, in my book, since it lets us get a feeling for how competitive both their hardware and pricing is.
I believe, though, that their lights-out management is an add-in card, rather than built-in, and it certainly doesn’t share a network port or anything. Dang.
Quite a few SmugMuggers over at Sun, which again, gives me the warm and fuzzies. Nothing quite like having contacts in the company who use, love, and count on your product. Sun also wanted to set up a meeting, rather than whipping out a quick quote, but they were extremely fast and proactive about it, rearranging their schedules and bringing along some big guns. Not only did this flatter us (given our small size), but I think I’m beginning to understand why they’re profitable again and why I’m starting to think that trend will accelerate: they’re trying to give love to small startups because they know some of them will grow into huge companies. If they build a strong relationship early on, the theory goes, they’ll have their business forever. I happen to think this is a very good strategy. There have been some hiccups, as Matt Mullenweg points out, but Jonathan Schwartz responded in a way that I wish every CEO would. Hopefully they work the kinks out.
I love, too, that Sun is changing and innovating once more: embracing x86 & Linux is huge, ZFS is the coolest filesystem ever, and they’ve had an unusually high volume of clever product announcements this year (Thumper, Black Box, Sun Spots, etc).
Yesterday, too, I got a first-hand glimpse of how Sun’s employees feel about Jonathan, their new CEO. They were glowing, speaking in reverent terms about him. That bodes well for their new direction and future.
Alas, though, I have some past experience with their lights-out management, and while it’s adequate, it’s not nearly as good as HP’s. It can’t share a network port and doesn’t offer the other advanced features HP has. At least it is network-based, rather than serial (I believe it does both, actually, which could be a boon to some people).
UPDATE: Dave pointed out in the comments that Sun’s Opteron management lets you have graphical console, keyboard, mouse, DVD, floppy, and ISO support. Plus, it shares one of the GigE ports. I’ve only used the non-Opteron boxes, so I’m probably both out of date and not used to what the architecture differences are. I could have sworn, though, that Sun told us yesterday, in person, that it doesn’t share the network ports. Anyway, sounds like Sun’s doing a better job on management than I initially gave them credit for.
The bad news
The specific platform we want (two CPU sockets, sixteen DIMM slots) doesn’t seem to be readily available from HP, Dell, or Sun. We’re going back-and-forth to make sure, but none of the systems published on their sites seem to offer this. Hopefully someone can deliver something. Rackable, however, does – they’re just crashing for us at the moment.
UPDATE: Sun has one too, the Sun Fire X2200. I just somehow overlooked it. Game on!
In case you’re curious, it’s because 2GB DIMMs are at a price point similar to double 1GB DIMMs, so we’d rather have 10 32GB boxes than 20 16GB boxes for this application.
At this point, I think Sun wins on service & attention while HP wins on product. Which we’ll choose, I still have no idea. 🙂
Have any experience, good or bad, with any of these vendors? Let me know.
Kudos to Sun
Wow, I wasn’t expecting this. I’m very, very impressed. Sun is profitable again.
In case you don’t get it, this is a much much bigger deal than the partnership with Intel they announced yesterday.
I’ve been worried about Sun for years, and for the last year in particular I’ve been telling anyone who’ll listen that Jonathan Schwartz has an incredibly tough job ahead of him. He’s been talking the talk for awhile now, saying all of the right things, but actually turning a large corporation and getting them going in another direction takes an awful lot more than talk. It sure looks like he’s doing it.
But it wouldn’t be fair to heap all the praise on his shoulders alone. Sun employees could easily not get on board with the new direction, making his job impossible. Obviously, that hasn’t happened and everyone must be on-board and embracing change. So kudos to the entire company – this is quite an accomplishment and a rarity in Silicon Valley. Just look at all the server and workstation company corpses around… SGI, anyone?
Keep it up, Sun!
Kudos to Jonathan Schwartz
This is a great blog post from Jonathan, CEO at Sun. If more companies were like this, the world would be a better place.
Just imagine if your cell phone provider, for example, actually cared about whether you were happy and whether they were delivering good value to you, their customer? Or how about your broadband provider?
I know, I know, we’d all die of heart attacks from the shock. 🙂
I have to say, I was pleasantly surprised by Sun’s response to my detailed review of the Sun Fire CoolThreads T1000 server and our interest in buying them. Sun assembled a great team of engineers and spent some time with us trying to figure out why our application wasn’t performing up to snuff on their hardware.
For those of you who are still wondering what’s up with that, I’m the one dropping the ball, not Sun – I’ve just gotten swamped. But I’m still interested, so as soon as I come up for air, I’ll try to get more hours put into it. Last time we all worked on it, Sun wasn’t able to get more performance out of it than I was – but they’re anxious to try again and I’m anxious to let them.
In related news, my current server provider, Rackable, seems to have fallen on hard times. We started using them something like 4 years ago, and loved them. Lately, though, their stock is in the toilet, they’ve taken ages to get hardware to us (and a few other major brands I shouldn’t divulge), and worst of all, very expensive brand-new servers from them are failing left and right. Anyone at Dell or HP want our business?
Hello Speed, Beauty & Brains – Goodbye Alexa
Michael Arrington at TechCrunch just broke the story of our latest release and it’s a great write-up. We’re really thrilled to have this puppy out the door and let everyone play with it.
This is a pretty fundamental shift for us, and while I don’t want to give us too much credit, I really think it’s the beginning of a sea change on the web. There have been plenty of apps which launched with 100% AJAX, like GMail, but I can’t think of any that have yet made the plunge to change an existing, entrenched product with lots of users. I’m sure there have been a few, so forgive me if I overlooked you, but certainly not many – most of the big apps are still HTML driven. But I believe that’s going to change because the customer experience just gets so much better.
Everyone is going to do this. The only question is when? (Ok, two questions: And who will be left behind?)
We’ve been playing with 100% JavaScript/AJAX interfaces like this internally for quite some time, but there were some huge pitfalls that kept us from actually releasing it. When we finally solved the last hurdle we got really really excited – this was gonna be great for customers. The minor downside is that I expect our Alexa rank to plummet – we’re no longer really doing page views, which I think they track. (We already get unfairly penalized because so many of our customers use their own custom domain names, but this should really do us in). I could really care less from a business point of view – this is good for customers, after all, but the geek in me thinks that’ll be fascinating to watch and see what happens.
The benefits of this release are obvious: the interface is faster, prettier, and smarter. But the pitfalls are less obvious. Here are a few of the biggies:
- Search engines. I know Google’s been testing a more JavaScript-aware version of Googlebot, but how aware it really is is anyone’s guess. Certainly no crawlers I’m aware of do even a marginal job of crawling AJAX pages. But our customers spend tons of time captioning, describing, and keywording the 120+ million photos at SmugMug.
- Backwards compatiblity. We built our URLs from day one to be “permalinks” so they wouldn’t change if you used them in your blog and forum posts. We had to make sure that things still worked going forward.
- AJAX Permalinks. Now we needed new permalinks that describe various pieces of data for browsing SmugMug, but we also needed to keep them short so people could copy & paste easily, so they wouldn’t wrap in emails, etc.
- Stats tracking. Specifically external sources like StatCounter and Google Analytics which only track page views, not JavaScript UI interactions. Our customers, especially the tens of thousands of hardworking Pros who build their photography businesses at SmugMug, expect to still get useful and meaningful statistics on who’s viewing what.
- Browser interfaces. People expect the Back & Forward buttons to work properly, along with History and Bookmarks. Doing so in all three major browsers was thought to be impossible, and we failed many times. We solved this one, and this was the last biggie. I believe it’s an internet first. Jimmy will be updating his blog about exactly how we do it so anyone else can follow suit. It’s good for the web as a whole for this stuff to move forward.
It was an amazing team effort over here to get this thing done, including tons of our customers. GreenJimmy, our resident Web Superhero, especially drove this project long and hard. Hopefully we can talk more about what we did, technically, so others can avoid making the same mistakes we did.
I really have to also give props to the awesome team over at Yahoo! working on YUI. We couldn’t have pulled this off without their library (easily the best JavaScript library around). They did a profile on us just a week and a half ago, but that was before this release. Now we’re even more hardcore with all the YUI stuff. 🙂
Enjoy the release, and just wait to see what we’ve got coming next…. 🙂
SmugMug's Don MacAskill 