Home > datacenter, smugmug > Sun Honeymoon Update: Storage

Sun Honeymoon Update: Storage

May 16, 2007

As I mentioned in my review of the Sun X2200 M2 servers we got recently, which we absolutely loved, Sun’s storage wasn’t impressive at all. In fact, it was downright bad. But before I get into the gory details, I feel compelled to mention that I believe Sun’s future, including storage, is bright. Their servers rock, they’re innovating all over the place, and for the most part, the people at Sun have been fantastic to work with – even when they’re being told their storage hardware sucks. That’s impressive. Now, on with the show:

The storage arrays I’m blogging about here are Sun StorEdge 3320 SCSI arrays. For more on why we chose this particular model, you can read about my on-going search for The Perfect DB Storage Array. The bottom line, though, for us is that Speed Matters. Their list price is quite expensive, but Sun was willing to work with us on the price, and we managed to get things into a reasonable ballpark. Reasonable, that is, as long as they performed. 🙂

First, some details. These boxes were destined to be part of our DB layer, with the first few going in as new storage for replication slaves. The goal was to maintain a high number of small (4K) i/o operations per second (IOPS) with an emphasis on writes, since scaling reads is easier (add slaves) than scaling writes (only so many spindles you can add, etc). In this particular case, the writes were being delivered from a MySQL master using InnoDB running Linux on 3 effective 7200rpm spindles, so the Sun array, on paper, should be able to keep up, no sweat. If your needs differ, our story might not be useful – test for yourself.

Installing and configuring them was an adventure. Craig Meakin, our Server Surgeon, was tasked with installing them and immediately ran into a snag. When configured for DHCP management access (which is how they were set up out of the box, exactly how we like them), they wouldn’t actually DHCP an IP address. It took someone at Sun wading through 4 different manuals to determine that not only did the array have to toggled to DHCP, but you also had to write “DHCP” in the IP address spot to make it work. Strike one.

(As an aside, one of Sun’s engineers also told us, after we’d bought them and installed them in the rack, that these storage arrays don’t come with battery backed write caches. Given how expensive they are, I was shocked and furious, but quickly got verification that they do, indeed, have BBWC.)

We brought one online and moved a DB slave snapshot over which was a few hours out-of-date and started replication so it could catch up with the master. Obviously, it wasn’t live and in production, so it was mostly spooling and committing writes from the master, only doing reads as needed for updates and whatnot. A very light load, in other words. With interest, we started timing how fast it would catch up, since it should scream. We were betting at least 2X (15K drives, after all) faster than the master, and on par with our other 15K SCSI slaves. Instead, we measured more than 4X *slower* than our slaves. Strike two.

Ok, no worries. Obviously this is a new array, and we did something terribly stupid setting it up. Sun support to the rescue, right? So we opened up tickets, dumped our config and all other relevant details to them. Nada. Oh, they came back with lots of suggestions and things to try. But none of them helped. Next step was to grab detailed system i/o statistics on production slaves which worked and Sun SE3320s that didn’t, so Sun could compare. And compare they did – their data showed a 6X performance differential between our production slaves (which had $700 off-the-shelf LSI SCSI MegaRAID cards in them) with 15K disks and Sun’s hardware. Sun was 6X slower. Final verdict? “System is performing as designed.” Strike three – they’re out!

Frantically, since the entire reason we had gone with Sun was because Rackable had shipped us a bunch of broken units and we were now months behind on an expansion project, we called Dell and ordered some PowerVault MD3000 SAS arrays. I always give Dell props for fast, efficient delivery, and knock them for a lack of innovation. In this case, they not only got us the gear fast, but the MD3000 turned out to be a fantastic DAS device and nearly perfect for our needs. Thank goodness!

Normally, that would be the end of our little tale, but as luck would have it, when Sun realized they’d laid an egg with the SE3320, they rushed us an engineering sample of their not-yet-announced (then) new StorEdge 2540 array. The good news? It performed neck-and-neck with the Dell array and uses SAS drives, which we prefer over SCSI. The bad news? They weren’t out yet and we needed storage yesterday. I believe they are out now, and I would buy the 100% SAS version, the StorEdge 2530, rather than 2540, for use in our datacenters if we hadn’t gotten the Dells.

So now we’ve got fantastic Sun servers attached to fantastic Dell storage. And our little franken-servers are as happy as can be. Fast, too.

Feed readers: Digg this story

Categories: datacenter, smugmug
  1. May 16, 2007 at 1:19 pm

    I love these posts!

    Do you store the actual images in your DB or just reference files on the file system / S3? We’re actually reviewing the MD3000 so good to hear you are having good luck with them.

  2. May 16, 2007 at 2:03 pm


    We don’t store our photos in the DB. That’d be way too huge. 🙂

    We store them in Amazon S3 and SmugFS. SmugFS doesn’t use MD3000s (we don’t need 15K disks for that). Instead, we use Apple Xserve RAID for those (and they rock, btw).

  3. May 16, 2007 at 3:12 pm

    I figured. More curious than anything.

    We don’t handle nearly as many images as you do (maybe 5,000-10,000) and are considering moving them into the DB.

    We do have millions of car parts though 😛


  4. Gabe E. Nydick
    May 16, 2007 at 7:09 pm

    I use the Sun SE systems and you have to tune the controller. Play with the tag count for each device and controller, the default values are too low. When I did that, the performance sky rocketed. Probably just an over site by Sun’s integration people.

    – Gabe

  5. May 16, 2007 at 7:39 pm

    I should clarify, we use the SE3510 SAS arrays specifically. Also, I saw some comments about DHCP not working on the x2200 ILOMs. We have those too and they didn’t work unless you used a DHCP reservation by putting the MAC in the client identifier on the DHCP server, in our case a Cisco Catalyst switch, instead of just the usual location.

  6. Bruce Davis
    May 16, 2007 at 8:54 pm

    Did SE3320 the array come with a built-in RAID controller or were you using JBOD with software RAID? What about the PowerVault MD3000 array? Were you comparing apples to oranges?

  7. Mike
    May 16, 2007 at 11:33 pm

    Sun has always made awesome servers, but their storage has always been pretty bad.. The only arrays they got right were the A5200s, and those didn’t even come with a RAID controller. 🙂

  8. May 17, 2007 at 12:03 am

    @Bruce Davis:

    We were using hardware RAID every step of the way, including on the SE3320s. So yes, we were comparing apples to apples, at least as much as is possible with 4 different vendors. (At one point, we had Apple Xserve RAID alongside LSI MegaRAID alongside Sun SE3320s alongside Dell MD3000s. Whew!)

    No JBOD in the mix. Software raid is really freaking fast, btw. Often faster than hardware. But we need reliable battery-backed write caching, which software raid can’t provide (at least, not on any hardware we own).

  9. Neil Corlett
    May 22, 2007 at 3:52 pm

    AFAIK, the SE3320 is re-badged DotHill kit. They were generally pretty good.
    There are Sun blogs floating around that suggest increasing the number of PCI transactions in play speeds things up.
    However, black magic ought not be necessary. This stuff should (preferably) work out of the box (esp. from a systems company) or it should be well documented in the release notes. Too often it is not.

  10. Rob
    May 25, 2007 at 11:46 am

    Sun storage in this range has been awful for years. Way, way, overpriced and nearly impossible to make perform decently. I see even their new line of storage is still way overpriced. At my previous job I bought a bunch of Sun V20Z’s but used Dell storage (connected to LSI MegaRaid cards installed in the V20Z). Before that I was forced to use another departments Sun T-3 Fibre Channel storage. It was hideously slow and ungodly expensive. Finally a Sun tech discovered some sort of misconfiguration and it got to the point of being usuable, but I could still get better write performance on a single UltraSCSI disk.

    The Dell storage attached to the V20Z flew like lightning.

    BTW, has anyone at Sun contacted you about you about your miserable experience?

  11. December 16, 2008 at 3:14 am

    Network Area Storage (NAS) is easier to share files on the network but the price is still too expensive, especially for use only at home.

  12. December 24, 2008 at 8:47 pm

    just came across. Nice blog

  1. May 16, 2007 at 8:12 pm
  2. May 22, 2007 at 11:17 pm
  3. May 23, 2007 at 5:34 pm
  4. October 1, 2007 at 5:03 pm
  5. October 18, 2007 at 5:05 am
  6. October 29, 2007 at 4:48 am
  7. December 14, 2007 at 12:12 pm
Comments are closed.
%d bloggers like this: