Archive
LunchGeeks this week
Going to eat at La Fiesta and geek out again this week on May 24th. Come if that sounds like your thing!
Don’t forget about all the Lunch 2.0 events this summer, too. (It must be intern season!)
Flickr still has a human face
I’m solidly in the “Flickr is so cool, I wish they hadn’t sold out to Yahoo” camp. I still don’t understand the decision, but people close the situation tell me it was awesome for all involved, so I’m happy for them. I know for sure they have a bright, passionate team and they deserve to be happy. But I have worried, sometimes publicly, that Yahoo may dehumanize Flickr.
Today I read, with great relief, that it hasn’t happened yet. Stewart’s reply to the Flickr fiasco this week shows that they simply made an honest, human mistake. A painful one, but by being transparent & open, they’re doing the right thing for their customers and community.
I absolutely love to see companies that don’t abuse their customers, particularly in our space. The amount of damage every bad photo sharing site does to consumer confidence is enormous and bad for everyone else involved. There have been sites that have gone dark, with no warning, and deleted millions of photos in an instant. Imagine how easy it is to convince those people that not all photo sharing sites are equal. 😦
Sun Honeymoon Update: Storage
As I mentioned in my review of the Sun X2200 M2 servers we got recently, which we absolutely loved, Sun’s storage wasn’t impressive at all. In fact, it was downright bad. But before I get into the gory details, I feel compelled to mention that I believe Sun’s future, including storage, is bright. Their servers rock, they’re innovating all over the place, and for the most part, the people at Sun have been fantastic to work with – even when they’re being told their storage hardware sucks. That’s impressive. Now, on with the show:
The storage arrays I’m blogging about here are Sun StorEdge 3320 SCSI arrays. For more on why we chose this particular model, you can read about my on-going search for The Perfect DB Storage Array. The bottom line, though, for us is that Speed Matters. Their list price is quite expensive, but Sun was willing to work with us on the price, and we managed to get things into a reasonable ballpark. Reasonable, that is, as long as they performed. 🙂
First, some details. These boxes were destined to be part of our DB layer, with the first few going in as new storage for replication slaves. The goal was to maintain a high number of small (4K) i/o operations per second (IOPS) with an emphasis on writes, since scaling reads is easier (add slaves) than scaling writes (only so many spindles you can add, etc). In this particular case, the writes were being delivered from a MySQL master using InnoDB running Linux on 3 effective 7200rpm spindles, so the Sun array, on paper, should be able to keep up, no sweat. If your needs differ, our story might not be useful – test for yourself.
Installing and configuring them was an adventure. Craig Meakin, our Server Surgeon, was tasked with installing them and immediately ran into a snag. When configured for DHCP management access (which is how they were set up out of the box, exactly how we like them), they wouldn’t actually DHCP an IP address. It took someone at Sun wading through 4 different manuals to determine that not only did the array have to toggled to DHCP, but you also had to write “DHCP” in the IP address spot to make it work. Strike one.
(As an aside, one of Sun’s engineers also told us, after we’d bought them and installed them in the rack, that these storage arrays don’t come with battery backed write caches. Given how expensive they are, I was shocked and furious, but quickly got verification that they do, indeed, have BBWC.)
We brought one online and moved a DB slave snapshot over which was a few hours out-of-date and started replication so it could catch up with the master. Obviously, it wasn’t live and in production, so it was mostly spooling and committing writes from the master, only doing reads as needed for updates and whatnot. A very light load, in other words. With interest, we started timing how fast it would catch up, since it should scream. We were betting at least 2X (15K drives, after all) faster than the master, and on par with our other 15K SCSI slaves. Instead, we measured more than 4X *slower* than our slaves. Strike two.
Ok, no worries. Obviously this is a new array, and we did something terribly stupid setting it up. Sun support to the rescue, right? So we opened up tickets, dumped our config and all other relevant details to them. Nada. Oh, they came back with lots of suggestions and things to try. But none of them helped. Next step was to grab detailed system i/o statistics on production slaves which worked and Sun SE3320s that didn’t, so Sun could compare. And compare they did – their data showed a 6X performance differential between our production slaves (which had $700 off-the-shelf LSI SCSI MegaRAID cards in them) with 15K disks and Sun’s hardware. Sun was 6X slower. Final verdict? “System is performing as designed.” Strike three – they’re out!
Frantically, since the entire reason we had gone with Sun was because Rackable had shipped us a bunch of broken units and we were now months behind on an expansion project, we called Dell and ordered some PowerVault MD3000 SAS arrays. I always give Dell props for fast, efficient delivery, and knock them for a lack of innovation. In this case, they not only got us the gear fast, but the MD3000 turned out to be a fantastic DAS device and nearly perfect for our needs. Thank goodness!
Normally, that would be the end of our little tale, but as luck would have it, when Sun realized they’d laid an egg with the SE3320, they rushed us an engineering sample of their not-yet-announced (then) new StorEdge 2540 array. The good news? It performed neck-and-neck with the Dell array and uses SAS drives, which we prefer over SCSI. The bad news? They weren’t out yet and we needed storage yesterday. I believe they are out now, and I would buy the 100% SAS version, the StorEdge 2530, rather than 2540, for use in our datacenters if we hadn’t gotten the Dells.
So now we’ve got fantastic Sun servers attached to fantastic Dell storage. And our little franken-servers are as happy as can be. Fast, too.
Feed readers: Digg this story
Speed Matters.
As subscribers to my blog have probably already guessed, we spend an inordinate amount of time at SmugMug trying to optimize for speed. As a media-heavy website, that’s a difficult thing to do and there are a lot of pieces. A typical gallery page at SmugMug contains 16 photos (though may contain thousands), plus all of the other graphic elements on the page, JavaScript includes (we use lots of JS), CSS includes, and the page HTML itself.
We’ve long tracked our own internal “page render” time, but once it leaves our servers, it gets more difficult to track. There’s a huge, nasty mess of networking equipment and providers between our servers and each customer. There are paid services that will track some of this for you, but that doesn’t tell you what the actual customer experience is like. We have employees in Utah, Idaho, Ohio, Virginia, New York, London, the Netherlands, and Australia so we can get a decent idea, but nothing beats aggregate data.
Enter Alexa with their excellent service and the data it provides. They get a lot of publicity for their Traffic Rank and Reach stats, but they don’t help us hardly at all (we have tens of thousands of customers who use their own custom domains, for example, among other problems). The stat I really love is the Speed rating. Since Alexa aggregates data from millions of people all over the world, across all page views on a site (heavy and light), we can get a really good view of just how fast our site is:
The usual disclaimers about statistics, particularly Alexa’s, apply: We don’t know exactly what they’re measuring, how much or often they’re measuring it, and how many people are actually measured. But we do know that Alexa’s Speed rating has directly correlated to feedback we get from our customers, and most importantly, our customer satisfaction. That’s good enough for me.
Now, like any Alexa statistic, such as Traffic Rank, it’s best viewed in relation to other sites, rather than alone. So here’s a bunch of photo-sharing sites, both ‘larger’ and ‘smaller’ than SmugMug, and their Speed ratings in rough order of ‘size’ according to Alexa’s Traffic Rank (again, Traffic Rank is notoriously flawed, but we have to order by *something*):
Fotolog:
Flickr:
Photobucket:
Webshots:
PBase:
Kodak Gallery:
SmugMug:
Shutterfly:
Snapfish:
Now, we’re not perfect. SmugMug, like every other site on the net, has problems. But we try very very hard to keep the site speedy and responsive – and I think both the stats above and our customer satisfaction speaks volumes. And I think it’s only fair to note that some of those sites handle more page and photo requests per day than we do – but we’ve left the “small site” size behind long ago, so I wouldn’t discount our size too much. It’s probably only fair to note that with the possible exception of PBase, they all have massive financial resources in comparison to ours, though, too.
We have a huge laundry list of things we can do to speed the site up even more, so look for us to shave more milliseconds off your page load times as we go forward. And I have to thank our Ops team, Andrew Gibbons – Director of Operations & Craig Meakin – Server Surgeon, and programming team, Jimmy Thompson – Web Superhero & Lee Shepherd – SmugSorcerer. Couldn’t have gotten below 1 second without them!
If there’s enough interest, I can do a follow-up post on lots of the tricks we use to get there. I don’t think we do anything earth-shattering, but lots of small things add up. Let me know.
WordPress – an overnight success!
Sorry, Matt, I couldn’t resist. 🙂
Seriously, Matt’s post entitled Meaningful Overnight Relationship really hit home over here. For whatever reason, we’ve started getting the “Wow, you’re an overnight success! How did you do it?” questions a lot here at SmugMug lately. (Answer: “Work really hard for 6 years and maybe, just maybe, people will start to notice” … of course, that leads to the inevitable “Wow, you were so young!” line of questions that Matt is still getting…).
Sam Walton probably said it best when talking about Wal-Mart: “Like most other overnight successes, it was about 20 years in the making.” But Matt says it extremely well, too. I think his story, and Ben & Mena Trott’s, and Howard Schultz’s, and SmugMug’s are much more exciting than a typical get-rich-quick story. These are the kind of stories that power lasting successes and this list is long. Most of your favorite brands these days have a story like this behind them, rather than a “I sold to Yahoogle” story.
On a related note, we’ll send you a free copy of Chicken Soup for the Entrepreneur’s Soul if you’re interested in these sorts of stories. The book is full of great ones from lots of “overnight successes”. 😉
And finally, we do love WordPress. You’re reading a WordPress blog right now.
Bye Yahoo Photos – Hello Flickr (and SmuggLr)!
I got an email months ago saying that the author had written TechCrunch, GigaOM, and Valleywag about Yahoo Photos closing and that they’d ignored it completely. Given that Valleywag posts just about any rumor and that it hadn’t gotten posted, plus Yahoo Photos was so huge, I figured someone was just trying to get me to look stupid on my blog.
Guess not. USA Today broke the story that Yahoo Photo is on it’s way out. TechCrunch has followed up with a post on the subject. One wonders, though, if they wish they’d listened to that email a few months ago? (I know I do!) And where on earth was Valleywag?
The text of the email in question contained a reference to Yahoo only providing migration mechanisms to a single “Web 2.0” company, Yahoo-owned Flickr, and all of the other options being old “Web 1.0” dinosaurs like Kodak and Shutterfly. That seemed brilliant, to me, from a purely business perspective, but poor from a consumer perspective. As great as Flickr is (and I do believe Flickr is great), it’s not perfect for everyone.
They deserve kudos for allowing any non-Flickr migration at all, but the really awesome way to handle this would have been to let any service integrate as a migration partner. (Remember, I’m the CEO & Chief Geek at SmugMug, so I’m biased all over the place!). Rumor has it we were actually up for consideration, but the decision went against us. D’oh!
Regardless, if you give Flickr a try and don’t like it for whatever reason, SmugMug still has a Flickr Refugee 50% Discount in effect. Plus you can use tools like SmuggLr to seamlessly move your photos over.
Part of me is sad, like we’re witnessing the end of an era. So many of the big old photo sharing sites are dead and gone. I believe there are only 6 big ones left from the pre-SmugMug days. Can you name them all?
Thoughts on Silverlight
I was at MIX this week speaking on a panel about web services. Obviously, the big announcements there were around Silverlight, so I’ve been getting emails asking what I think. Here you go, just a raw brain dump:
- No matter what happens, everyone wins. Competition is good for us. Adobe has urgency to improve Flash at a rapid rate, and Microsoft has urgency to catch up. Awesome for you and I – who cares who “wins” or if there even is a “winner”.
- Silverlight looks amazing. I want to consume Netflix movies using the Silverlight interface – it’s gorgeous and fast. But I don’t want to consume Netflix movies on my PC – I want to do it on my TV. That’s why I have an Xbox 360, AppleTV, and TiVo Series 3.
- I would like to develop for Silverlight just to see how neat it could be for our company, but there are two big problems:
- The installation process on Mac OS X is horrendous. And we have a lot of Mac OS X customers.
- Massive chicken-and-the-egg problem. My customers are not technical, and almost always just answer “No” to permission dialogs because they’ve been trained that “Yes” means “I’d love to be infected with a virus!”
- I have some fairly big performance complaints about Flash in some cases, so if Silverlight doesn’t suffer from the same issues, I’ll be pretty thrilled.
- Ray Ozzie & Co didn’t make nearly enough of a big deal about the fact that you can do this stuff in Python and Ruby in addition to C#. That’s huge and should have been the opening headline.
- Offering to host Silverlight videos for free (4GB) is a brilliant way to speed up adoption. Good move, MS!
So, anyway, there we go. I’m excited about it and skeptical about it at the same time. I think if the Mac installation process was a breeze AND I was able to create something really compelling, I’d be willing to show my customers how great it is. So the installation ball is in Microsoft’s court right now.
We’ll see what happens. 🙂
(Oh, and whoever organizes MIX next year, can we please please please have a real grid like every other conference? I missed great sessions because there was no grid)
Going to D? Wanna give me a lift?
Found out a friend of mine got a lift to D: All Things Digital last year. On a VC’s private jet. Lucky duck.
Right now, I’m planning on flying coach into San Diego and driving to D, like last year, but let’s see just how powerful this blog thing is:
Do you have access to a private jet? Are you going to D? Give me a ride! I’ll let you pick my brain about any business or technology I know anything about (SmugMug, bootstrapping to profitability, Amazon Web Services, datacenters for startups, LAMP, whatever) the entire trip. Come on, no-one was going to sit in that extra seat anyway!
Oh, and if you know someone who’s going who fits these criteria, pass my plea along. 🙂
Send all jet offers to don-AT-smugmug. Thanks!
Amazon S3: New pricing model
I’m getting emails about Amazon’s new S3 pricing model, so I guess the news is out. 🙂
For us, this is great. We’ll save money right off the top (we upload a lot, so $0.10/GB uploaded vs $0.20/GB uploaded is a big deal) first of all, and secondly, they finally have tiered download transfer costs. This is a big one for us, because we buy enough bandwidth that $0.20/GB wasn’t cost-effective enough for us.
I’m going to have to run some numbers (I’m at MIX right now) to see if it’s now good enough for us to start serving more content out of S3 or not, but even if it’s still not perfect for us, it’s a major move in the right direction.
Finally, this illustrates a subtle but important point of using S3. When I buy physical disks at SmugMug, those are sunk costs. They’ll never get cheaper because I’ve already paid for them. At Amazon, though, market forces and changes will cause their pricing model to continue to re-adjust downwards. As disks get cheaper, that $0.15/GB/month fee will drop. And instantly all of your storage magically gets cheaper, no sunk costs to worry about.
That happened today, and I’m sure it’ll happen over and over again as storage & bandwidth both get cheaper and Amazon is able to leverage their scale to get better deals. The more people use S3, the more Amazon can drive prices down.
Since we were already saving a ton of money using S3, this is music to my ears. 🙂
SmugMug's Don MacAskill 