Work on Drizzle full-time at Rackspace Mosso!

April 27, 2009 9 comments

This is really cool. Rackspace is hiring people to work on Drizzle full-time for their cloud product, Mosso. Adrian Otto writes the Drizzle mailing list:

I was speaking with Eric Day at the developer conference, and I mentioned that Rackspace is wiling to employ full time developers for the specific purpose of furthering the Drizzle project’s mission. He suggested that I email you on this list becuase he expected there would be interest in this offer. If you work on the project now part time, and want to make it a full time job working exclusively on the Drizzle project, let me know. The Rackspcae Cloud believes in open source, and we want to do our part to make Drizzle a wild success.

I’m super-excited about Drizzle and think this is fabulous for the community at large. I’m not alone – Mark Callaghan and Jeremy Zawodny like the idea too.

So if this sounds like your thing, go do it!

Nasty Safari bug not fixed since December :(

March 26, 2009 33 comments
A rotten little apple

A rotten little apple by Ashley Harding

Apple has had a nasty Safari bug since December which breaks SmugMug, Facebook, Gmail, and lots of US banks.

3 months later, it’s still not fixed. Your only option is to use Firefox if you’re affected.

Apple’s known about the problem since December, and has lots of internal bugs on the issue (30+ I last heard). (For my Apple readers, here’s our bug on the subject and the one it was marked as a duplicate of).

I’ve done everything I know how to get this resolved – Apple employees have been internally working on our behalf, we have an AppleCare Enterprise Support case open (#332101), I tried to open an ADC Premier case (it was denied because they don’t “provide code-level support for content creation issues” whatever that means). Still no luck.

Apparently this is fixed in Mac OS X 10.5.7. We have the latest seed, so we’re going to find out, but 10.5.7 is likely a month or more away from shipping, so expect this stuff to be be broken at least until then. Use Firefox.

Safari happens to be my favorite browser, so this is especially disheartening. The good news is you may not be affected. Not everyone running 10.5.6 with Safari is, for some reason, but lots of you are. You’ll know you are if you see SmugMug galleries which appear to be empty (but aren’t) or see ugly white pages with undecipherable error messages. For that, I apologize – I really wish we could help but we can’t. You can help yourself, though – use Firefox.

Put another way, Eric Schmidt, Google’s CEO, sits on Apple’s board of directors. Gmail has also been broken for 3 months. Apparently he’s powerless too. 😦

Yahoo adds SmugMug support!

January 16, 2009 21 comments
Yahoo! in cloud OR Hadoop? (Яху в облаках)

Yahoo! in cloud OR Hadoop? (Яху в облаках) by Alexander & Natalie

tl;dr: Yahoo adds SmugMug support to Profiles. Windows Live coming. Lots of other services, too.

Wow, what a pleasant surprise! Woke up this morning to this story on TechCrunch about 20 new services they’d added to Yahoo Profiles (here’s mine). Lo and behold, SmugMug is one of them! In fact, in Yahoo’s blog post about the new features, SmugMug was the one mentioned for photos. Cool!

As far as I know, we haven’t talked to Yahoo about this at all – which is part of what makes this so great. Microsoft was supposed to have rolled something like this out to Windows Live profiles awhile ago, but I still haven’t seen it drop. We’re very excited about that, too, but the two company’s approaches were very different: Microsoft came over, chatted with us about the product, then had us sign a contract to participate. That was months ago, and I have no idea when it’s actually coming. Yahoo, on the other hand, seems to have just built it and shipped it.

I can see the arguments for both approaches: Microsoft is probably being extra careful about privacy, and working through their internal rules and regulations about re-using user generated content. Yahoo, on the other hand, is scrambling to catch up now as the underdog. I assume Yahoo realized that SmugMug already has strong privacy controls around our feeds and simply hit the gas – full speed ahead.

Either way, what’s especially heartening is the number of sites, services, and pieces of software that now support SmugMug. At The Crunchies last week, we weren’t nominated (we won for Best Design last year), but it still felt like we were winning – many of the winners use or integrate with us: Google Reader, Windows Live Mesh, Cooliris, lots of companies using Amazon Web Services, lots of apps on the iPhone 3G, and FriendFeed. Very cool.

(And all of that despite what we *know* is terrible and/or nonexistent documentation around our feeds. Yes, we’ll work on that.)

Great things afoot in the MySQL community

December 23, 2008 37 comments

tl;dr: The MySQL community rocks. Percona, XtraDB, Drizzle, SSD storage, InnoDB IO scalability challenges.

For anyone who lives and dies by MySQL and InnoDB, things are finally starting to heat up and get interesting. I’ve been banging the “MySQL/InnoDB scales poorly” drums for years now, and despite having paid Enterprise licenses, I haven’t been able to get anywhere. I was pretty excited when Sun bought MySQL since their future is intrinsically tied to concurrency, but things have been pretty slow going over there this year.

But the community has finally taken up arms and is fighting the good fight. It’s (finally!) a great time to be a MySQL user because there’s been lots of recent progress. Here’re some of my favorites (and highlights of work left to do):

PERCONA

I can’t sing Percona’s praises enough. They’re probably the most knowledgeable MySQL experts out there (possibly even including Sun). Absolutely the best bang for the buck in terms of MySQL service and support – better than MySQL’s own offering. (If I had to guess why that is, I’d bet that MySQL/Sun don’t want to step on Oracle’s toes by fixing InnoDB – but >99% of what we need is related to InnoDB. Percona has no such tip-toeing limitations.) Let me quickly count the ways they’ve helped me in the last few months:

  • They knew of a super obscure configuration setting “back_log“. Have you ever heard of it? I hadn’t. But we started seeing latency on MySQL connections (up to *3 seconds*!) on systems that hadn’t changed recently (exactly 3 seconds sounded awfully suspicious, and sure enough, it was TCP retries). After going through every single kernel, network, and MySQL tuning parameter I know (and I know a lot), I finally called Percona. They dug in, investigated the system, and unearthed ‘back_log’ within an hour or two. Popped that into my configuration and boom, everything was fine again. Whew!
  • We have servers that easily exceed InnoDB’s transaction limits. Did you know InnoDB has a concurrent transaction limit of 1024? (Technically, 1024 INSERTs and 1024 UPDATEs. But INSERT … ON DUPLICATE KEY UPDATE manages to chew up one of each). I know all about it – I’ve had bugs open with MySQL Enterprise for more than 2 years on the issue. What’s more, these are low-end systems – 4 cores, 16GB of RAM – and they’re no-where near CPU or IO bound. It took MySQL months to figure out what the problem was (years, really, to figure out all the final details like the different undo logs for INSERT vs UPDATE). Their final answer? It’ll be fixed in MySQL 6. 😦 Note that 5.1 *just* went GA after years and years. On the other hand, it took Percona one weekend to diagnose the problem, and 13 days to have a preliminary patch ready to extend it to 4072 undo slots. Talk about progress! (And yes, we want Percona to release the patch to the world)
  • Solving the CPU scaling problems. These have been plaguing us for years (we have had some older four-socket systems for awhile … now with quad-core, it’s even worse), and thanks to Google and Percona, this problem is well on its way to being solved. We’re sponsoring this work and can’t wait to see what happens next.
  • XtraDB. This is the biggy. So big it deserves its own heading….

XTRADB

Oracle’s done a terrible job of supporting the community with InnoDB. The conspiracy theorists can all say “I told you so! Oracle bought them to halt MySQL progress” now – history supports them. Which is a shame – Heikki is a great guy and has done amazing work with InnoDB, but the fact remains that it wasn’t moving forward. The InnoDB plugin release was disappointing, to say the least. It addressed none of the CPU or IO scalability issues the community has been crying about for years.

Luckily, Percona finally did what everyone else has been too afraid to do – they forked InnoDB. XtraDB is their storage engine, forked from InnoDB (and then turbocharged!). We’re not running it in production yet, but we are running all of the patches that went into XtraDB and I can tell you they’re great. We’re sponsoring more XtraDB development (and yes, we made sure Percona will be contributing anything they build for us back to the community) with Percona, and I’m sure that’ll continue.

DRIZZLE

I’ve already blogged a bit about Drizzle, but it sure looks like Drizzle + XtraDB might be a match made in heaven. Drizzle can be though of as a MySQL engine re-write with an eye towards web workloads and performance, rather than features. MySQL 4.1, 5.0, and 5.1 added a lot of features that bloated the code without offering anything really useful to web-oriented workloads like ours, so the Drizzle team is ripping all that stuff back out and rethinking the approaches to the things that are being left in. Very exciting.

SSD STORAGE

The advent of “cheap enough” super-fast SSD storage is finally upon us. I’ve got Sun S7410 storage appliances in production and they’re blazingly fast. I have a very thorough review coming, but the short version is that even with NFS latencies, we’re able to do obscene write workloads to these boxes (let alone reads). 10000+ write IOPS to 10TB of mirrored, crazy durable (thanks ZFS!) storage is a dream come true. Once you mix in snapshots, clones, replication, and Analytics – well, it just doesn’t get much better than this.

(Don’t get sticker shock looking at the web pricing – no-one pays anything even remotely like that. Sign up for Startup Essentials if you can, or talk to your Sun sales rep if you can’t, and you can get them much cheaper. I nearly had a heart attack myself until I got “real” pricing. Tell them I sent you – enough Sun people read this blog, it might just help 🙂 ).

STILL NEEDED…

So, all in all, there’s been an awful lot of progress this year, which is great. CPUs are finally scaling under InnoDB, and we finally have storage that isn’t bounded by physical rotation and mechanical arms. Unfortunately, great CPU scaling plus amazing IO capabilities isn’t something InnoDB digests very well. As is common in complicated systems, once you fix one bottleneck, another one elsewhere in the system crops up. This time, it’s IOPS. It was eerie reading Mark Callaghan’s post about this last night – I’d come to the exact same conclusions (from an Operations point of view rather than code-level) just yesterday.

Bottom line: Despite having ample CPU and ample IO, InnoDB isn’t capable of using the IO provided. You can bet we’ll be working with Percona, Google and Sun (read: sitting back and admiring their brilliant work while writing the occasional check and providing production workload information) to look into fixing this.

In the meantime, we’re back to the old standbys: replication and data partitioning. Yes, we’re stacking lots of MySQL instances on each S7410 to maximize both our IOPS and our budget. Fun stuff – more on that later. 🙂

UPDATE: Just occurred to me that there are plenty of *new* readers to my blog who haven’t heard me praise Google and their patches before. Mark Callaghan’s team over at Google definitely deserves a shout-out – they’ve really been a catalyst for much of this work along with Percona.

On Why Auto-Scaling in the Cloud Rocks

December 9, 2008 70 comments
SkyNet Lives - EC2 at SmugMug

In high school, I had a great programmable calculator. I’d program it to solve complicated math and science problems “automatically” for me. Most of my teachers got upset if they found out, but I’ll always remember one especially enlightened teacher who didn’t. He said something to the effect of “Hey, if you managed to write software to solve the equation, you must thoroughly understand the problem. Way to go!”.

George Reese wrote up a blog post over at O’Reilly the other day called On Why I Don’t Like Auto-Scaling in the Cloud. His main argument seems to be that auto-scaling is bad and reflects poor capacity planning. In the comments, he specifically calls SmugMug out, saying we’re “using auto-scaling as a crutch for poor or non-existent capacity planning”.

George is like one of those math teachers who doesn’t “get it”. I was tempted not to write this post because he gets it so wrong, I’d hate to spread that meme. SkyNet auto-scales well. No humans at SmugMug are monitoring it and it just hums along, doing its job. Why is it so efficient? Because I understand the equation. I know what metrics drive our capacity planning and I programmed SkyNet to take these into account. It checks an awful lot of data points every minute or so – this isn’t simply “oh, we have idle CPU, let’s kill some instances.” (I would argue that, depending on the application, simple auto-scaling based on CPU usage or similar data point can be very effective, too, though).

SkyNet has been in production for over a year with only two incidents of note and SmugMug has more than doubled in size and capacity during that time without adding any new operations people. How on earth is this a bad thing?

First 1080p video from Canon's new 5D MkII – Amazing!

November 30, 2008 54 comments

My father and I got our 5D MkIIs on Friday and we could hardly wait for the batteries to charge. He took his to SF to test its vaunted low-light performance and posted this 60-second 1080p clip (along with other resolutions) on his SmugMug site: Click to watch it auto-sized for your monitor or check out the full 1080p resolution (caution – *high* bandwidth! UPDATE: Apologies if you tried to watch 1080p on Windows earlier. My bug made it look terrible. Try again, please?).

Here’s his story:

“I had seen Vincent Laforet’s amazing short film, but only in 720p. I knew what an amazing photographer he is and wondered how close an everyman like me could come to footage like that. Could the clips possibly hold up to viewing in 1080p?”

“So with only an hour’s practice shooting my dog licking peanut butter and the neighbor’s kids running in their yard, I left for the city to compare myself to a Pulitzer Prize-winning photographer with his helicopter, pricey stabilizer, models, set lighting, and post-production experts. I had a few hours and a tripod. What we had in common was the 5D.”

“At first, shooting video on the 5D makes you feel stupid. You’re holding the camera out in front while you look at the LCD on the back and use completely different buttons. I was always wondering if it was in focus, especially at the wide apertures I thought you probably needed at night.”

“How dark was too dark? I’d point at things that seemed impossibly dark, like the fishing boats you saw lit with mostly a string of Christmas lights on the bow of one. But I couldn’t tell how noisy the clips were on the viewfinder, so I held my breath and set the camera at ISO 3200. Why? ‘Cus it was lower than ISO 6400…”

“I had just one sekret weapon, same as Vincent: a Canon 200mm f/2.0 lens, not exactly an everyman item. It made a difference and I used it for maybe half the shots, including the opening clip of the couple kissing at Grace Cathedral, the rotating jewelry in the shop window, the hotel entrances, and the TV reporter. The city skyline was shot with an f/4 lens and it’s noisy. I also used an 85mm f/1.2 for scenes like the cable car, and toys in shop windows.”

“Dog and kid shots look amazing too, but I have to be honest: I missed many shots of fast-moving kids that I would have gotten with my video camera. Maybe I just need figure out how to juggle zooming, focus, and having the controls scattered across the back of the camera, but it felt like I needed three hands and the skillz of a Cirque du Soleil juggler.”

“So which camera for filming my grandkids? Now there’s a question… This calls for some serious 5D time to answer. Even my wife approves of that message.”

BTW, if anyone else out there is shooting 1080p video with cameras like this and would like their SmugMug Pro accounts to allow 1080p video, let us know. That feature is currently in beta, but we’d love to get a few more people using it. 🙂

Nominations open for The Crunchies 2008

November 25, 2008 4 comments
SmugMug wins Best Design at The Crunchies 2007

SmugMug wins Best Design at The Crunchies 2007 by Luca Filigheddu Photography

I loved the idea behind The Crunchies even before we won for Best Design last year so I’m glad to see their triumphant return. And I see that nominations are open for 2008!

It looks like we’re probably eligible for a number of categories, but even if you don’t think we’re worthy, please go nominate your favorite startups. It really means a lot to the teams that work on these companies. Nothing like a little validation for all of our hard work… 🙂

I'm *not* speaking at Cloud Computing Expo 2008

November 17, 2008 11 comments

Just a quick update, I was invited to speak at Sys-Con’s Cloud Computing Expo 2008 West (how’s that for a mouthful?) and accepted, planning on talking about SkyNet, S3, and our future use of cloud computing. Alas, my Inbox is so crazy, I failed to see the handful of emails the conference sent me asking me to sign a contract of some type. So I missed the deadline and they canceled my spot. (BTW, I can’t recall a conference ever asking me to sign something to speak, but this one does and I was full of FAIL.) 😦

So, I’m sorry, despite being listed on the program, I’m *not* speaking there this week. It was my bad – I just missed the emails (as I miss so many emails these days). But still, a phone call from them wouldn’t have hurt, would it?

Who knows if I’ll be on their invitation list next year, but the conference will be great anyway, so have a great time without me!

Sweet new Sun storage stuff on Monday, Nov 10th

November 9, 2008 16 comments

FYI, Sun is announcing some sweet new storage stuff on Monday at 3:30pm PT.

I’m reviewing a few of the things they’re announcing, and hope to publish my thoughts here soon (one of them joins my production network tonight if all goes well). However, I’m at Disneyland with my kids (first trip!) from Monday through Thursday, so I don’t know (yet) when I’ll be able to write them up. Bear with me if it takes a few days.

But the gear is exciting, and the direction Sun is headed is even more exciting!

Categories: datacenter Tags: , , ,

Now is the time to build

October 30, 2008 12 comments
Big Cats

Big Cats by micalngelo

“Every startup CEO is at least thinking about the need to cut back right now” – Michael Arrington

“We simply attempt to be fearful when others are greedy and to be greedy only when others are fearful.” – Warren Buffet

I’ll give you one guess as to which man I’m listening to. So no, not every startup CEO is cutting back. Apple spent their time innovating during the last downturn and look where it got them. I’m thrilled to have just passed out big, healthy profit-sharing bonuses to all of our employees this week for the 5th consecutive year. We think and hope they’ll be even bigger next year.

SmugMug was founded in the middle of the last “nuclear winter” in Silicon Valley. Everyone told us we were crazy, and we knew there was no chance at raising venture capital at a decent valuation, even with our impressive backgrounds. So we did what any good entrepreneur would do: We did it anyway, with both eyes firmly on our business model.

So if you’re running a startup, or thinking of creating one, take heart – downturns are a fabulous time to build and grow businesses. Focus on your revenues and your margins, not your growth rate or # of unique visitors. Find some stable income streams and a customer need. Listen to your customers and give them what they want – and what they’re willing to pay for. And take care of your employees – they’re your most valuable asset.

SmugMug is still hiring Sorcerers, Heroes, and all manner of other mythical beings capable of impossible feats. We filled our last position (quickly, I might add) with a *great* hire (and I’m still sorting through the avalanche of resumes we got to see if we can add a few more), but the job door is never closed at SmugMug for true superstars. Our philosophy is to not let anyone amazing get away, even if we don’t technically have an open position for you.

So if you can make magic and want to work for a company that takes crazy-good care of its employees, let us know.

%d bloggers like this: