Home > business, smugmug > The Enterprise Linux problem

The Enterprise Linux problem

March 23, 2007

I was talking with a friend last night about commercial OSes, specifically Linux and Solaris. It dawned on me that those of you in the midst of your own startups would find our experience useful. Sorry I didn’t think about blogging this earlier.

First of all, I love open source. I can’t imagine what it would be like to have a major OS problem and not have access to the source code and a community with access to it as well. I love the idea of building companies on open source and then using the success of that company to give back to the people and the community that made it possible.

There are lots of ways of contributing back, but the low hanging fruit for a tiny company like ours is simply buying service and support from one of the open source players, like MySQL or Red Hat. It’s a win-win, theoretically – they get funding to continue to build and test great software, and you get the service and support you need to grow your business.

We’ve been through a few commercial open-source OS distributions at SmugMug. All the gory details after the jump.

We started in 2002 with SuSE. At the time, I’d profiled the various filesystems, and XFS was the clear winner for our use case, and SuSE had it. At the time, we had a grand-total of 3 small, used servers that we got for pennies from a failed dot com. We’re bootstrapped, so every dollar we spent came out of our own pockets. The $800/seat price for SuSE with Enterprise support at the time seemed huge and steep (the hardware they were running on was much cheaper), but I choked down the cost and got a few copies.

Bad mistake. SuSE wouldn’t return my phone calls or emails when we ran into our first problem with the product. Literally wouldn’t even listen to see if our problem was legitimate or not. Since then, things over there have only gotten worse – Novell is now in bed with Microsoft, to the detriment of the entire community.

Next up at the plate was Red Hat. This time, I did some research before forking over my dough, and the community seemed in agreement that Red Hat actually responded to customers. Plus, their pricing was more stomachable: $350/seat instead of $800. The really good news? They actually did acknowledge me as a customer, reply to my emails, and best of all, have a fantastic web management interface / software update mechanism in the form of Red Hat Network.

We ended up using Red Hat Enterprise Linux (v3 at first, then later v4) for at least 4 years. During that time, we were mostly happy. The software updates were great, the software was stable and reliable, and we really had very few problems. Those that we did have, though, we were able to work out for ourselves or with a little help from the Linux community more quickly than Red Hat support was able to do. Even with the very most expensive Advanced Premium contracts, the support just wasn’t that useful to us. It does help that we have a lot of Linux experience in-house, so we’re able to take care of lots of things ourselves. Your mileage may vary if you don’t have our same level of expertise on tap 24/7.

When our service contract came up for renewal last year, we decided to evaluate whether it was worth paying or not, as we often do with any contract renewal. After looking at the price (tens of thousands of dollars, not chump change for a small business) and the relative speed and level of support, we balked. We had a dilemma: we loved Red Hat Linux, we loved how good they were at building & testing their software, we loved their mechanism for delivering software updates. We just didn’t need support.

We got on our knees, begging and pleading with Red Hat to let us pay for a “software updates only” license. They wouldn’t have it. “Support comes bundled with updates”, I was told, “no ifs, ands, or buts”. I *want* to pay Red Hat for the valuable service they do for us and the community. I just don’t want to pay for the part we don’t need – human support.

Around this time, Oracle announced Unbreakable Linux. Of course, the dirty little secret is that Unbreakable Linux is simply a repackage of Red Hat Enterprise. They’re identical. Better yet, they do offer a “software updates only” contract, and it’s reasonably priced. Now, this puts us in a strange position, because clearly Red Hat is doing most of the work, and Oracle is trying to profit. Brilliant of Oracle, you could argue, but it doesn’t make us feel that great – we want to give our money to the company actually doing the heavy lifting. Only they won’t take our money.

Nonetheless, we bought a couple of licenses from Oracle and gave it a whirl. It was indeed identical, and was a breeze to migrate our Red Hat boxes to it – we just changed one file. Unfortunately, Oracle released the product half-baked. The update service was broken in multiple ways (can’t have two people administer the same boxes, for example, and software updates inexplicably broke a few times). It quickly became clear that we couldn’t bank on something so unfinished, so we moved on.

I began asking friends of mine, at much larger companies than ours, what they were doing about this problem. Those companies have large, experienced Linux groups on hand to deal with isssues, and thousands of machines, so I couldn’t believe they were paying hundreds or thousands of dollars per Linux box for support they weren’t going to use. Turns out they weren’t – they were all using CentOS.

CentOS, it turns out, is just like Oracle Unbreakable Linux, only it’s free. The community has repackaged a few Red Hat Enterprise packages so Red Hat’s name and logo don’t appear in them, and that’s it. Other than that, it’s basically identical to Red Hat, and closely tracks their patches and updates. Migration from Red Hat Enterprise is just as easy as Oracle – you change a file or two and you’re golden. It’s a breeze. We have a handful of CentOS boxes in our datacenters now, and they’re working out beautifully – no learning curve, feels just like Red Hat, etc.

Now, again, this makes me feel a little slimy. I would really like to pay Red Hat for all their hard work building and testing the software. I’d happily pay $100/seat/year or something to Red Hat for exactly what I’m getting from CentOS for free. It’d be the right thing to do. But Red Hat won’t let me.

I’m certainly not going to waste my money, my shareholders’ money, and most importantly, my customer’s money on something frivolous like support we won’t use. I realize that normally I can’t just dictate prices – I can’t walk into a Honda dealer and tell them I’m buying their minivan for $20K instead of $35K and expect to win. It’s Red Hat’s product, they can sell it for whatever they like. I get that. But with open source, it’s also the community’s product. Which means I have options – including free.

Which brings us to Solaris. Solaris is now open-source, so it’s on my radar again. I love ZFS, I love the fault-tolerant stuff it has for when memory or CPUs go bad, etc. Sure sounds great. It’s a commercial OS, with free updates, and (I assume) good support options. But the last time I gave it a shot (last year), I was lost in userland. Solaris userland and Linux userland differ so greatly, there’s a steep learning curve for someone like me with 14 years of Linux under my belt. But I’m considering taking a closer look.

So at the moment, we’re sorta stuck in limbo. Our CentOS experiments are going well, but I’d rather find a middle ground where I can be part of the solution, rather than the problem. If and when we finally decide on our next course of action, I’ll be sure to let you know.

Love to hear any other experiences, if any, for people using open source software. (There’s no way we’d consider running something mission critical on a non-open source OS. I don’t know how people do it on Microsoft’s stuff).

UPDATE: I’m so dumb. Dag Wieers had a great point in the comments. I can just buy a handful of licenses (whatever I think Red Hat’s hard work on the software is “worth” to us) and not even use them. (And contribute to CentOS in the form of donations, too). Not as elegant as having a ‘software updates only’ license, but still better than nothing. Thanks Dag!

Categories: business, smugmug
  1. March 23, 2007 at 6:51 pm


    I run a good part of the perl.org infrastructure and there we are running on RHEL (gracefully donated by RedHat) and we love the RHN system.

    However, when I needed an OS for a startup I went with Fedora (if I had started now I’d probably have gone with CentOS 5). The cost is just a bit too much for a boot-strapped company with good in-house Linux expertise.

    With Fedora we’ll have to suffer and benefit from more frequent major updates, but my experiences upgrading FC5 to FC6 are pretty good and we’ve worked hard on building systems to distribute configurations and such (csync2 and puppet stuff mostly).

    – ask

  2. March 23, 2007 at 9:24 pm

    I agree – when I was consulting and a company didn’t have in-house Linux expertise, I’d always spec out RHEL.
    However, with my current employer my Linux experience, combined with that of the community is enough, and we use CentOS. One of the solutions I think is to vote with your dollars – show RedHat and the community that you aren’t looking to leach. You mentioned $100 per server as what you’d be willing to pay, so send that to CentOS, Johnny Hughes, et al can certainly make use of it to keep things running smoothly, and CC some of the people at RH.

    That being said – I have had conversations with a few RH employees – they don’t have a problem with CentOS – and aren’t really interested in the market it serves, which is typically the small-medium business. They don’t care about the 20-30 server environments, they want the 2000-3000 server environment. If anything the 20-30 server environment is more painful for them to support.

  3. airtonix
    March 24, 2007 at 2:48 am

    umm what about ubuntu? or am i missing the point?

  4. March 24, 2007 at 6:02 am

    Even when you choose CentOS, you should really consider buying Red Hat support for as much as you think it is worth having CentOS. Without Red Hat there would not be a CentOS, so by not using RHEL you’re actually killing CentOS as well.

    The rule of thumb here is what the liability of downtime/problems are. If you have business critical (or mission critical) systems. These ought to be RHEL systems.

    CentOS by nature can never be better than RHEL. CentOS does not fix software problems. And even when you can fix problems yourself, they will never appear in CentOS of they are not fixed by Red Hat. So if you don’t need the support, then pay Red Hat to include your fixes.

    The only way to do that is to pay for support.

  5. March 24, 2007 at 7:25 am

    @Dag Wieers:

    (Thanks for your RPM repository, btw. It’s a life saver. How can I repay YOU? Donations?)

    I completely, 100% agree. Only Red Hat won’t take my money. I don’t want to use CentOS, and I do want to pay Red Hat.

    I just don’t want to pay for something (being able to call or email a human) that I don’t need and will never use.

    I *do* want to pay for software development, but again, Red Hat won’t take my money. Believe me, I begged and pleaded with them. My money’s no good there. 😦

  6. March 24, 2007 at 7:28 am

    @airtonix: I’m looking for an enterprise-grade OS, with everything that entails.

    Ubuntu has a brand-new Enterprise offering, but since it’s so new, lots of enterprises will likely not use it just yet. I don’t know how to break that chicken-and-the-egg cycle, but hopefully the Ubuntu team does.

    I’d like to see more enterprise editions of Linux out there, but there are currently very few with a reasonable track record.

  7. Jeff
    March 24, 2007 at 8:25 am

    @onethumb: I think Dag was alluding to buying one, two or however many “full support” licenses you value RedHat at your own organization. And just because you purchase the licenses, it doesn’t mean you have to use them — you can continue to use CentOS instead. Thus you’d still be supporting RH with reall $$$.

    Also, if you aren’t aware, in addition to his great RPM repository, Dag also supports CentOS.

    I’d like to offer a hearty THANK YOU to him and all the other people @ CentOS.

  8. March 24, 2007 at 11:15 am

    I’d agree with Jeff, and I know at least one small company that is doing exactly that. They have something like 80 systems running CentOS in production, and 2 running RHEL in the lab (not production, but they’re identical hardware to the production boxes). They pay RedHat for 10 entry level seats (used to be 5, but they upped it last year since they’re doing better, and like you, wanted to pay RH for what they do.) The net result: they get the stable OS they want in production, and can sleep at night because they don’t feel like they’re stealing from RH. They’ve always said that if a problem they couldn’t handle came up, they’d repro it on the RHEL box in the lab and open a support ticket, but so far as I know they’ve never had to actually go there.

  9. March 24, 2007 at 11:47 am


    Holy crap, I’m so dumb. It was so obvious, and Dag was hitting me over the head with it, that I couldn’t even realize it. Haha.

    That’s exactly what I’ll do. I’ll buy a bunch of licenses that I won’t bother using.

    Thanks for clarifying it!

  10. March 25, 2007 at 6:35 pm

    Ubuntu really isn’t there in the enterprise market, sorry kids.

    I dont think any major vendors package for ubuntu, (maybe ibm db2 now), but you need more “stack” to be useful. application ,storage, drivers, backup, operating system.

  11. March 26, 2007 at 6:20 am

    One thing to keep in mind, when you buy your handful of licenses is you can use them. RedHat’s support policy is if it happens on the machine you licensed for, they support it. I even heard the CEO mention this in a recent talk to the press. So, if you have a your handful of licenses, and you run across a problem that you just can’t solve, then try and reproduce the problem on one of your Red Hat machines. If you can, just give support a call and you’ll back in business in no time. If you can’t, then your engineers can study the differences between the two machines to find the problem.

    RedHat is in the business of selling a service, not a bunch of licenses you don’t need. So when you buy the licenses, plan how you can best use them. Are they better to put on your QA machines, where you can attempt to reproduce problems. Or are you best off with them on your most critical production boxes?

    Of course one other thing to consider is with RHEL 5, it is a good time to re-evaluate. Visualization can sometimes reduce your hardware requirements, and thus reduce your support costs considerably.


  12. March 26, 2007 at 6:22 am

    Opps I ment to say virtualization, not visualization.

  13. zuesse
    March 26, 2007 at 11:37 am

    Please feel free to contribute. Go to Gentoo and look up GLEP 19. Solve the problem there and concider yourself a major contributor. I have a few ideas on a supporting architecture but this seem right up your ally.

  14. March 26, 2007 at 2:17 pm

    Whilst I agree with Dag, CentOS team and users do fix bugs and do push bug reports and bug fixes upstream.

  15. March 27, 2007 at 12:23 pm

    If you”re reconsidering Solaris, or an OpenSolaris based distribution, you should, by all means, consider reaching out to the OpenSolaris immigrants community (http://opensolaris.org/os/community/immigrants/). I’m a member there, and would be more than willing to help out during your eval.

    There are definitely differences in between RHEL and Solaris, just like there are between RHEL and Ubuntu or RHEL and Gentoo. There have long been a few choice things you can do to have a system, as far as a user or application is concerned, look and feel much like a Linux distro. Generally it’s things like setting up your path to have the tools you want (most of the GNU tools are supported and in Solaris already) in it and adding any choice packages you’d like to have (i.e. top, if you prefer it to prstat).

    I know there are definitely some larger differences when you start talking about system administration level things, but at that level, at least some of the customers I’ve worked with like the way Solaris operates. For instance, listing wifi networks:
    # dladm scan-wifi
    iwi0 sunwifi 0:18:b0:80:5d:c0 none very good g 54Mb
    iwi0 sunwifi 0:18:b0:80:61:0 none good g 54Mb
    iwi0 — 0:12:17:ac:e:7e none very good b 11Mb

    There are many more, like dealing with multipathing to external storage, multipathing network connections, keeping track of component faults even across reboots, working out how various system services relate to each other, etc.

    It’s definitely a balance, as some Solaris users do not *want* Solaris to become more like Linux. For instance, some folks believe /usr/lib and /usr/bin should not be as full as they are, or commands like ‘tar’ should have a POSIX compliant tar. Yet others think any tar that isn’t flag-compatible with GNU tar is broken. Yet others (including myself) find serious complications in the approach in trying to work with 32/64 bit applications on the same system (or just compiling one over the other) with some Linux distros.

    Thus is the challenge of trying to map the commonalities/differences between various standards and the way people expect them to operate.

    The bright side here is there are solutions, and the OpenSolaris community is working to find the right solutions. For instance, if you find something’s not right in Solaris/OpenSolaris, we’d love to have your input in the GNU Solaris Community (http://www.opensolaris.org/os/community/gnu_solaris/).

    Your blog would work as a feedback mechanism as well. 🙂

    (sorry to be so long-winded)

  16. March 27, 2007 at 3:52 pm

    Can I mention FreeBSD? I’ve used RHEL4 and FC6 at work, but I’ve never felt ‘at home’ on either. Dag’s repo has been a lifesaver in various instances, and thanks to him for having various packages residing there.

    The one problem I have with redhat is their software is always many major releases behind. For example subversion 1.4.3 is out but they still ship 1.1.3. Maybe they like having ancient binaries being distributed, but if you want to use say php5 you have quite a bit of compiling ahead.

    Does anyone know if linux has something similar to the FreeBSD ports tree?

  17. March 28, 2007 at 9:03 pm

    Jacques, some linux distributions use a system very much like FreeBSD’s ports, specifically Gentoo is built around “portage” which was clearly influenced by ports.

  18. Glenn Leavell
    March 28, 2007 at 9:54 pm

    My understanding is that you can purchase real RHEL subscriptions *without* support from some third-party partner vendors. For example, HP sells RHEL subscriptions that tie into the regular Red Hat Network system decoupled from support (which they also sell). I’ve been told that these RHEL subscriptions do *not* need to used on HP servers. Here’s a part number list that shows the different options:


    By the way, with RHEL5, which is just out, you can run virtual guest operating system instances without paying any additional cost for the subscriptions that run within those guests.

    (BTW, I don’t work for HP or Red Hat and don’t have an interest in the companies.)

  19. Tel
    April 1, 2007 at 12:10 am

    You can get software updates for free from RedHat, you just need to download and compile the SRPM packages yourself — not all that difficult. If you really want to spend money on something then spend the same money buying RHAT shares that you would have spent on support… later on you can cash in by selling those shares.

    So if you don’t need the support, then pay Red Hat to include your fixes.

    The only way to do that is to pay for support.

    There’s something backwards about about fixing a problem yourself and paying someone else to use your fix. May I suggest that if you don’t need the support then spend a bit of time instead to carefully document your bugs on http://bugzilla.redhat.com/ and paste your fix there if you have one — if they use it, then well and good… if they don’t, then they will probably fix the bug anyhow and quite likely other people will use your fix if no better fix is available.

    The time spent documenting the problem is good value to yourself because it gives you time to think about the best way to fix it, and you might find someone else with the same problem.

  20. April 2, 2007 at 2:34 am

    I want to remind everyone that Red Hat does not OWN the vast majority of the items that they put in RHEL. Do people who use RHEL feel SLIMY because they are not paying the Mozilla team for Firefox or the OpenOffice.org team for OpenOffice.

    Red Hat rebuilds the majority of the software they use from other places … NOW, don’t get me wrong, they also pay / donate much $$$ and time/code to many of these projects.

    Red Hat is very open source friendly and I recommend that people buy RHEL licenses for their critical services or if a 3rd party application requires it.

    But CentOS does what it does because RHEL is using open source as their business model.

    CentOS also solves problems and reports those to Red Hat … as well CentOS users find problems that CentOS reports to Red Hat if the CentOS team can’t solve the problem. Try a Search in the redhat bugzilla for “centos” and see how many things CentOS has contibuted to RHEL.

    Open Source is good for everybody … and RHEL is a great product. Using CentOS where you are not using RHEL (for cost or other reasons) makes RHEL better … but the CentOS team certianly recommends that you also use RHEL where you need it.

  21. April 5, 2007 at 7:13 am

    Could someone provide a little more info on what Glenn mentioned above? Are VM’s or virtual guests a way around the initial problem discussed in the blog entry?

  22. April 8, 2007 at 9:48 pm

    Consider sending money to a project who’s software you use. Tell them you’d like a feature. Maybe it will show up and make all our lives eaiser. A few thousand dollars might just be enough to allow a developer to take the time to add something good. Or pay your own developer, get exactly what you want, and contribute it back.

    After all, Red Hat employs some developers, but they work on what Red Hat needs, not what you need. Pay a FOSS developer and he benefits, you benefit, and the community benefits. It’s the mark of a good deal.

  23. April 20, 2007 at 9:05 pm

    If you want to try Solaris again, but still have a Linux userland, you should take a look at Nexenta: http://www.gnusolaris.org/gswiki

    Or just put most/all of the gnu stuff in from a package repository like Blastwave: http://www.blastwave.org/

  24. April 20, 2007 at 9:20 pm

    Let me clarify one of my statements for those that don’t know how CentOS and RHEL relate. When I said:

    CentOS by nature can never be better than RHEL. CentOS does not fix software problems.

    I didn’t mean to imply that CentOS is not fixing bugs, but I meant instead instead that CentOS depends on Red Hat to implement the bugs and bugfixes reported by CentOS. Why ? Because that’s the only way to be 100% compatible with Red Hat.

    If CentOS would be fixing its own bugs instead of sending them to Red Hat and waiting for inclusion, compatibility would no longer be guaranteed. CentOS has been, and will be reporting bugs and bugfixes to Red Hat to improve its own products.

  25. April 24, 2007 at 9:23 am

    although this might not be applicable to your case since you are used to use RH-based distros there is also Debian. It is a very very good piece of Linux distribution and widely used in the server environment. I think it is worth mentioning here.

  26. May 26, 2007 at 8:11 pm

    Just FYI, Dell definitely allows customers to buy updates-only RHEL licenses with Dell servers/workstations. These subscriptions are referred to as “Basic” as opposed to “Standard” or “Premium”.

  27. December 3, 2008 at 2:17 am

    Abstract: We describe the LHCb detector simulation application (Gauss) based on the Geant4 toolkit. The application is built using the Gaudi software framework, which is used for all event- processing applications in the LHCb experiment. The existence of an underlying framework allows several common basic services such as persistency, interactivity, as well as detector geometry description or particle data to be shared between simulation, reconstruction and analysis applications. The main benefits of such…

  28. January 18, 2009 at 8:11 am

    sVMGez hi! how you doin?

  29. alain
    January 21, 2009 at 3:49 am

    Another vote for Debian. I switched my servers from a mix of RHEL and SUSE to Debian, after falling in love with Ubuntu on laptops / desktops (Ubuntu is one of a number of Debian derivatives out there). I've found Debian to be rock solid and the community is perhaps the strongest of any Linux distro. I also appreciate the project's strong commitment to software freedom. Still find myself working with other people's RHEL, Fedora, CentOS, SLES, openSUSE boxes and it's a joy to go back to Debian.

    Like you, I've also ventured into Solaris a couple of times (mostly for ZFS) but went back to Linux each time because using that OS is just painful! Can't wait for BTRFS to get to stable! Also, Nexenta seems interesting.

  30. January 21, 2009 at 4:29 pm

    Rg7WRn hi! nice site!

  31. November 23, 2009 at 8:42 pm

    Revitol Review – Skin problems can occur at any age, but there may be a way to avoid them by using revitol and it offers 90 days money back guarantee at http://www.Revitolskincarez.com

  1. March 28, 2007 at 4:18 pm
  2. March 28, 2007 at 9:48 pm
  3. May 17, 2007 at 3:43 pm
  4. April 16, 2008 at 9:47 am
  5. August 8, 2008 at 8:46 pm
Comments are closed.
%d bloggers like this: