Since the release of OmniOS last spring, we've received a lot of feedback from people through email, twitter and at various conferences we've spoken at. Most of this feedback has been very positive: people are happy to see the OpenSolaris family continue on, and are excited to have more options for working with DTrace, ZFS, zones and some of the other exceptional tools available on this platform. Of course, not everyone understands why we did what we did; after all, maintaining your own distribution is not a trivial task. While we've had the chance to talk about what went into our decision making process in person at meetups and other events, we thought it might be worth a look back here to explain some of the thinking that went into creating OmniOS.
We should start by saying that we've never been shy about building things. Whether it's a complex web application or the back-end infrastructure for high scale web services, building things is at the core of what we do. When we run into the limitations of software that isn't quite designed to do what we need, we either change it or make new software that will work for us. OmniTI is a company of engineers, and we have a long history of working with Open Source projects, so taking on an operating system distribution certainly fits our mission.
One of the first things we considered was our need for having a rock solid system to serve highly scaled web services. In the distant past, we used Solaris for this, although the service model was always more limiting. The advent of OpenSolaris allowed us to build out our own hosting infrastructure, which gave us full control over our working environment. It wasn't just about stability; OpenSolaris also gave us deep introspection, which can be critical when dealing with problems at scale.
Of course, there are a lot more people building large-scale web systems these days, and a lot of them do it on other operating systems. We do that for many of our customers as well, but sometimes you get to pick your battles. In the hardware world, you pick things like ECC RAM and SSD's because you want quality tools with which to do your job. When working in web operations, you want things like system tracing and filesystem snapshots--and when it comes to those, dtrace and ZFS are still the systems to beat. And, the implementations available in OpenSolaris, and now Illumos, are the best available; so if we get the chance to use those tools, we do.
Some might argue that tools like System Tap for tracing, BTRFS or LVM, or things like ZFSONLINUX, are equivalent options on Linux. We use these tools too, and we don't agree about their equivalency. Whether it's performance, stability, or ease of use, at best these alternatives leave something to be desired, and at worst may deliver you to disaster. It's not that those problems can't be fixed; we think they probably will be and that some day what's available on Linux may (should?) become the gold standard; but that day isn't today. We think it will be at least another five years before the options on Linux really get close. Five years is a lifetime on the internet. Five years ago there was no Foursquare, no Pinterest and not even a GitHub. Even if the current incarnation of OpenSolaris (aka Illumos) made no additional advancements for five years, we're no worse off by sticking with them and if they do happen to progress, it's gravy.
Of course, rolling our own also meant we got to make sure we had a number of other key technologies:
- zones for lightweight virtualization, so we can build our own "private cloud" systems
- kvm for hardware virtualization, for when we have to run mono-theistic software
- crossbow for network virtualization, which manages a myriad of private networks within our infrastructure
Okay, so we liked the idea of running something Illumos-based; but that still doesn't explain fully why we would roll our own distro rather than just rely on the other options available. When we looked at the current state of the Illumos-based distributions available, there arose two primary factors in our decision to build our own. The first is that what we are targeting with OmniOS doesn't fit the niche of the existing family of Illumos distributions. With SmartOS, Joyent has set their sights on building the next platform for managing private (and public) cloud infrastructure. This vision takes its focus off of the current, common needs around traditional enterprise management issues and strong heterogeneity. Nexenta's version of Illumos focuses strongly on enabling their flagship storage products and, as such, has less focus on the needs of enterprise customers for a general computing platform. OpenIndiana is probably the nearest neighbor to OmniOS, however we've found the build system difficult to work with, and their targets are set differently than ours as well; they don't just want to be a server OS, but also want to provide a fully featured desktop OS too. We have no such illusions, nor can we afford to be distracted by them. ☺ For OmniOS, we want a distro aimed at a minimalist server install that can be installed on both bare metal and virtual machines. That's a problem a lot of people have, and no other Illumos distro is quite focused on solving that problem.
The second reason that we thought we would create our own distro is that we had already been doing most of the heavy lifting for years. When we ran OpenSolaris, we had a full package build system so that we could roll our own version of any software package we needed. We'd already been in the business of managing kernel patches, and dealing with security updates, driver issues and integrating patches from "upstream." In managing our hosting architecture, we were accustomed to managing our own distribution anyway. To be clear, the heavy lifting here is in managing the packages, curating the ecosystem and making sure we had a repeatable installation process for spinning up new systems. An equally important factor was the heavy lifting we didn't have to do; the core operating system was developed--nicely handled by the Illumos team. While we work with that team when we can, we also have the help of all of the aforementioned companies and many others as well.
When Oracle decided to close the future development Solaris, we had to make a choice; limping along with a rotting corpse of Open Solaris wasn't an option, we needed something that was actively developed and that we could count on for the foreseeable future. When we added things up; our technical needs for running a robust, scalable platform; the alternative choices of either converting to a more limited platform (Linux) or working on one of the new Illumos distros, plus our general desire to bring focus to an underserved user base, the decision became clear, and OmniOS was born. We're still new to this, but so far, so good.
