At the center of the GoDaddy controversy is the correlation between advertising and brand identity. As a content provider, the Fox Network understands that their brand will be held responsible for the quality of both the program content and the advertising they deliver. If either aspect of the broadcast is sufficiently offensive or inept, they risk losing viewers to other stations. Consequently, Fox must provide content that is tame enough to avoid outrage from extremely conservative viewers while remaining provocative enough to satisfy the desire of third party companies seeking to push the envelope with their ad strategy. Content providers in any medium will be judged by the quality of the content they provide, and that includes the use and placement of advertising. This can be a tough balancing act, and Fox isn’t alone in walking the tightrope.

When it comes to the Web, the importance of managing advertising content takes on a new dimension. Hackers are increasingly using fraud and social engineering tactics to infiltrate advertising networks, and then utilizing their position within this circle of trust to inject malware, browser redirects, and cross site scripting attacks on unsuspecting visitors. If these attacks are successfully executed, hackers may steal credit cards, social security numbers, banking information, personal photos and anything else that has been digitized on the victim’s computer. Alternatively, a visitor’s computer may be turned into one of many “sleeper cell agents” in a botnet, ready to respond to a few keystrokes at any time and become an active participant in a worldwide Internet attack. It isn’t just web site visitors who are vulnerable. This same strategy can be used by black hat marketing consultants to siphon traffic from one web site to a competing web site or even to blacklist an entire site from the Google search index. The worst part? Hackers are able to target their attacks with profound granularity, making it extremely difficult for anyone within the targeted organization to know the attack is even happening.

The risk posed by vulnerable advertising mediums is not merely theoretical. In 2009, I documented two separate exploits that successfully penetrated highly trafficked and popular web sites. Both sites had full time IT teams who were previously unaware that the exploit was occurring. Furthermore, in a now highly publicized event in September of 2009, the New York Times was the victim of a malware advertiser who legitimately purchased ad space from the Times while pretending to be a representative of Vonage. IT departments and technical staff are trained to watch site visitors for malicious activity, but painfully few are watching the advertisers.

Deconstructing the Hack

Client-side arbitrary code execution is the primary culprit behind advertising based attacks. Attackers first gain trust with the target by posing as a legitimate advertiser. This process may be as simple as paying for space in an automated advertising system, or as involved as calling a major corporation while posing as a sales rep or marketing executive from another company. After the attacker has been approved as an advertiser, they develop a custom script to exploit the medium being used. They may start off displaying advertising that looks legitimate, but inevitably they switch to the malicious ad that begins to infect or otherwise manipulate site visitors.

Depending on the amount of freedom granted to an advertiser, a variety of techniques may be used to deliver the hacker’s payload. Many setups allow advertisers to automatically submit a combination of HTML, CSS and JavaScript code to be embedded within the layout of the publisher’s site. In this scenario, hackers can easily embed malicious scripts by using JavaScript or by including an external Flash SWF in the markup. In the off chance that this code is reviewed at all before publication, it is likely reviewed by someone in marketing or sales who is only examining the submission based on the content currently displayed and is unable to analyze the underlying code for potential security vulnerabilities.

Allowing advertisers to place custom Javascript or Flash files inline as part of a web site’s markup is especially dangerous as the advertiser is no longer restricted by cross-domain access policies. This leaves the advertiser with the power to do virtually anything the web site developers can do, such as posting AJAX requests or altering any element of the DOM. In an attempt to prevent this, some web sites have opted for an alternative setup that links an iframe or traditional frame to a server belonging to the advertiser. However, this approach is also flawed because changes to the advertising code may be made at anytime and the publishing site is powerless to implement a pre-approval process or apply any automated content filtering.

Regardless of the method utilized, if an attacker gains the ability to execute custom code on the target site, Pandora’s box has been opened and virtually all the evil of the Internet may be unleashed. A few fictional yet plausible exploit scenarios include the following:

You Won (Malware)!

Johann is a regular visitor to Finance Magazine’s web site. Like most Finance Magazine visitors, he is an investor with a diversified portfolio in mutual funds, bonds, and individual stocks. While browsing the latest financial news, a popup branded with the magazine logo suddenly appears and announces that he won a free 2 Year subscription to FM and a chance to win lunch with Warren Buffett. Lunches with Buffett are normally valued in the millions, and Johann has been wanting a print subscription to the magazine for some time. He clicks “Accept” and is asked to download the registration form. At first he is a bit suspicious and doesn’t understand why the registration form is a downloadable .exe file, but it is a company promotional from the official web site, so he reasons it must be okay. After downloading the application, he launches it and is asked several questions about his financial net worth. He is then asked for his full name, phone number, address, e-mail address and social security number. He is again a bit suspicious and doesn’t understand why he needs to enter his social security number, but the form does looks very professional. He clicks the info button next to the Social Security Number field and is shown this dialogue message:

You must enter your social security number in order to ensure that “Lunch With Warren” contest participants are limited to one entry per person.

With visions of the “Sage of Omaha” in his head and the promise of the next printed issue of Finance Magazine at his door step, Johann fills out the form and clicks submit.

Several months later, Johann’s financial life is in ruins. His personal information was used to register for several credit cards and a bank loan was issued for a Mercedes SLK in California. The executable file he downloaded included a custom Trojan Horse virus that allowed the attackers to login to his personal machine. They used this access to acquire his banking information and passwords, which they used four months later to wire over $10,000 to an account in the Cayman Islands. Although Johann is now suing for criminal negligence, his life (and his credit) will never be the same.

The Black Hat Who Stole Christmas

Brianna is a successful small business owner with an online niche retail store that averages $25,000 in sales and 500,000 visitors per month. In addition to a product catalog, her site also contains high quality articles and videos that pull traffic from Google and allow her to further monetize her online presence by selling custom advertising. Advertising is sold on a month-by-month contract basis, and advertising providers are given an account that they paste HTML, CSS, and JavaScript in so it can be included directly in the site markup. Usually, Brianna’s sales skyrocket for the entire month of December as Christmas approaches. This year, however, she has seen a 60% reduction in traffic and sales are plummeting. After closely analyzing her site analytics, she realizes that traffic from Google has drastically been reduced. She went from being in the top 10 results for her product niche to not appearing anywhere within the top 5 pages, and she can’t figure out why. What Brianna doesn’t realize is that she sold an advertising slot for December to a Black Hat SEO optimizer working for a competitor. As part of his overall strategy for propelling his client to the top, he decided to cut the legs out from underneath the competitors by running advertising templates on their sites that carefully and skillfully violated nearly every technical SEO guideline required by Google. Brianna made a few hundred dollars for the advertising space, but unknowingly violating Google’s SEO rules would cost her tens of thousands of dollars in the year ahead.

Defending Against Malicious Advertisers

Unfortunately, a foolproof technical solution to malicious advertisement doesn’t exist. However, this fact is not an excuse for apathy. The risk to both organizations and web site visitors can be mitigated by applying these technical steps:

1. Preview Ad Submission

   Within your advertising process, a system should be setup to preview all advertising before it is published live. This process should minimally include the ability to preview the way an ad looks and functions, but ideally it will also involve a quick scan of the actual advertising code to check for any unusual pieces of content (e.g. a few lines of encrypted JavaScript). If ad provider’s can automatically update their content, they should be sure that each new version is also approved before publication.

   This tactic can not be used if ad placement is achieved by embedding frames or iframes that link directly to a third party server.

2. Restrict Dynamic Advertising

   The single most effective method of preventing advertising abuse is to ban your advertisers from executing any dynamic advertising code. This process is as simple as stripping any scripting tags from advertising content and is extremely effective. However, the potency of this tactic comes at the cost of advertising flexibility as it limits all advertising to stylized images and text. In a medium where small games and other interactive content is often used to garner clicks and entertain eyeballs, this is often not a viable tactic. When scripting code is permitted, certain dynamic content should still be banned. For example, anything that involves alert() or document.location() calls in JavaScript could be stripped while leaving other code in place.

   This tactic can not be used if ad placement is achieved by embedding frames or iframes that link directly to a third party server.

3. Sandbox Dynamic Advertising

   In scenarios where dynamic content is permitted, it is useful to place ads within frames or iframes to take advantage of cross-domain safety restrictions. Yet as we have seen, using these constructs to link directly to a third party server is a security vulnerability as advertisers can easily change the content displayed without providing the publishers an opportunity to review the changes or strip malicious code. To obtain the best of both worlds, create an advertising sandbox by designating a domain or subdomain specifically to serve advertising content. Then place frames or iframes on your main site that link directly to the new domain. Because you control the ad domain, you will be able to preview ad submission, restrict dynamic advertising, and benefit from the added security that cross-domain security restrictions provide.

Even when implementing the safeguards above, the decision to grant an advertiser space on your web site should not be taken lightly. Doing so inherently confers a degree of trust upon a third party. Ensure that trust is well placed by implementing a screening policy for all new advertising sign ups. Such a policy could be as simple as calling companies directly to verify that the representative is authorized to sell advertising on their behalf or as involved as requiring all advertisers to provide copies of their business incorporation license or other government issued identification. Regardless, the threat can not be delegated to the IT staff and forgotten. Marketing and sales play an equally important role, and the safest organizations are those who view security as the shared responsibility of all the members within it.

There was a time when good advertising meant entertaining video or amusing copy. It could be judged purely on face value and the ability to generate ROI. That time has passed. In an increasingly interactive world, it is now more important than ever for organizations and individuals to understand that advertising consists of both form and function. Ignoring this fact can result in something far worse than assaulting the sensibilities of your audience; it can devastate their lives. Let content providers and audiences beware: the age of badvertising has begun.

This seems to be a common problem with many professional software development shops. Project managers help to keep the development teams focused, but their goals are still feature-driven with an eye on the next release cycle. The IT and Operations teams are painfully undermanned, left to maintain their systems and services without any training on the care and feeding of their new pet. For the hobbyist or Open Source project, this becomes an annoyance. If you’re running a business, operational neglect can have a dire impact on your bottom line.

Systems Administrators are unconsciously trained to look at everything through a boolean filter. Hosts are up or down. Services are on or off. Their understanding of the application stack is often superficial, limited to the same perspective as that of a typical user. Does the website load? Can I ping the servers? This is a completely logical approach. And yet, it fails to consider those "corner" cases where activity looks normal, but an internal component suffers an unexpected condition. Stealthy failures like these can be missed for months and result in significant lost revenue or wasted overhead.

Monitoring systems have improved over the years with "advanced" features like automatic discovery of hosts and services. Scanning a network, they can identify hosts and differentiate web servers from workstations. Resources are grouped logically. It’s a very turnkey way to add monitoring to your infrastructure. Unfortunately, for many companies, this is where the story ends (and the pain begins). Attentions are subsequently focused elsewhere. Priorities are reestablished. One of the most important resources, the one that makes sure everything else is operating smoothly, becomes forgotten and orphaned. It’s easy to forget about something that doesn’t make your job easier or offer intrinsic value to your bottom line.

A poor economy and high unemployment levels remind us how important it is to optimize our existing architecture. The current trend towards Cloud Computing and Virtualization makes this even more challenging. These technologies are useful for creating highly elastic platforms on a budget, but they complicate engineering by outsourcing data storage and processing to an external black box. In turn, we’re forced to add resiliency in the form of additional processing nodes and redundant storage. This added complexity introduces countless opportunities for disaster. It’s a vicious cycle.

As the Web has become the obvious target for fresh product development, additional layers of abstraction are introduced into the application stack. New technologies and components offer exciting ways to communicate with the end user and from one business to another. The higher we go, the more these layers are decoupled from traditional monitoring proficiencies. The resulting programs are overly intricate and opaque. We need new ways to increase visibility and derive useful data from modern business systems.

Gaining visibility over business operations is probably the easiest improvement any company can make. Quality analytics require a solid understanding of your IT operations and business processes, which come from transparency into your systems. Once these have been established we should be equipped with the tools to streamline and simplify any infrastructure.

1. Key Performance Indictators

   First and foremost, identify the external business metrics that directly affect your revenue. Establish thresholds and put fault-detection monitors into place, just like you would for any server or application. Alerts on business operations (e.g. new user registrations, orders per hour) are more important than the systems that support them. Remember that revenue is an asset, and hardware is a cost. Not the other way around.

2. Review IT Monitors

   Evaluate your existing IT monitoring systems. Ensure that metrics are being gathered for every single host and service. The breadth and depth of data collected now will directly influence the quality of the information that can be extracted later on. It’s paramount to have the metrics to support your decisions, but you won’t know which they are until you can juxtapose them later.

3. Stockpile Data

   Collect as many metrics as possible, for as long as possible. There are no good excuses for not storing metrics indefinitely. Storage is inexpensive, and a variety of technologies allow us to scale capacity with ease. In three years we should be able to look back on data with as much granularity as the information that was collected just yesterday.

4. Highlight Deficiencies

   Graph your metrics. Study their trends and formulate a plan to address the immediate capacity limitations. When deploying new resources, look for hints in the trends that reveal hidden relationships in your network. But remember that this goes beyond planning for the future; this data has inherent value in supporting your ongoing decisions.

5. Build Relationships

   Correlate graphs in ways that represent your business systems. Pinpoint metrics that relate towards a common goal (sales per visit, length of visit, average page size, network latency and webserver load). You might be shocked at the patterns revealed. If your trending application doesn’t allow you to correlate incongruent data easily, find a new one.

6. Empower Stakeholders

   Distribute the accumulated knowledge with individuals and teams within your organization that can take action towards positive change. If possible, give them access to all of the information, not just the data that directly affects them. For large architectures, there is rarely a single person with a holistic view of the entire stack. Trust your partners and there’s a good chance they’ll unearth something you missed previously.

Fault-detection and Trending solutions should return more on investment than high uptime or speedy notifications. They should prepare an organization to increase capacity before limits are reached, realign resources to meet unexpected traffic spikes, help Development & Design teams to better understand your customers, and decrease the maintenance and staffing necessary for normal IT operations. Feedback should be real-time and tuned to the needs of your organization. In a nutshell, it should pay for itself and then some.

Consider Facebook, YouTube, and Digg. These are not “web” companies. They are, respectively, communications, entertainment, and news companies. If tomorrow a new technology emerged (and inevitably it will) that allowed consumers to better achieve their goals than the services offered by these so-called “web” companies, all three organizations would be forced to either embrace that new technology or risk significant loss of market share. The kicker is that they aren’t alone: what is true for Facebook, YouTube, and Digg is also true for you.

Now for the secret: tomorrow may have come a day early. In April of 2009, Apple announced that the iPhone App Store had surpassed one billion downloads in just 9 months¹. Consider that for a moment. It took Firefox, the most used web browser in the world², over 4 years to reach one billion downloads³, and VOIP/IM service Skype served its one billionth download after 5 years⁴. Apple did it in 10 months, and they did it in an emerging market with an entirely unique distribution model. As impressive as that was, Apple has recently outdone themselves once again, reaching the 2 Billion download mark in September of 2009⁵, doubling the total number of downloads served in just 5 months.

This kind of exponential growth is unlikely to slow down anytime soon, and Apple’s success with both the iPhone hardware as well as the iPhone App Store has revolutionized the pace of innovation in the smart phone market. Verizon is projected to carry 18 different smart phones powered by the Google Android OS by the end of 2009⁶, and Blackberry now has an “App World” to provide consumers of their smart phone line with third party applications.

What does this impressive growth mean for business owners, executives, and IT managers? It means that consumers are becoming increasingly mobile, connected, empowered, and, ultimately, accessible. As large segments of the consumer market begin to adopt mobile technology, the organizations who benefit the most will be those who evolve to service them, finding new distribution or advertising channels for existing products and in some cases utilizing this technological shift to create entirely new offerings.

Yet the most profitable organizations will capitalize on the opportunities created by the mobile market without becoming lost within them. As smart phones are beginning to reach critical mass, it is perhaps now more important than ever to realize that much of the excitement in the mobile space is fueled by HTTP and the Web, and Internet usage certainly isn’t going anywhere but up in the foreseeable future. In the United States alone, Internet Service Providers have reached a point of market saturation with 71% of all Americans reporting consistent Internet and Web availability at home or work⁷, and 85% of iPhone users report using mobile Safari to browse the web on a regular basis⁸. Savvy managers will apply many of the lessons learned in previous online ventures to future expansion in the mobile space while continuing to build lasting value wherever their consumers are.

In the past 10 years, I’ve had the pleasure of working on a variety of projects that span multiple mediums. In varying capacities, I’ve directly contributed to television productions, radio broadcasts, print publications, trade show exhibits, stage shows and online advertising campaigns. I’ve also served as the lead engineer on a variety of desktop, web, and mobile software projects. I consider myself to be a graduate of the school of hard knocks, and when you dance with lady experience long enough, you begin to realize that the principles of success transcend all mediums. The following are just a few “Transcendent Themes” that I believe must be applied regardless of the medium an organization chooses to operate within:

1. Launching is Not Enough

   “If you build it, they will come.” It worked great for Kevin Costner’s character in Field of Dreams, but in the real world, merely bringing a product or service to market is rarely enough for it to succeed. Good products require good marketing initiatives behind them, and what is true in conventional mediums is also true on the web and the emerging mobile market.

   From banner ads to social media, today’s marketers have more tools at their disposal than ever before. Finding the right mix can be a challenge, but the solution will likely involve a cross-comparison of proven conversion rates against perceived market presence. In English: find out where your market is digitally congregating, look at the proven effectiveness of the tools at your disposal that are able to reach them, and then start a small initial campaign to test your analysis.

2. Sample It to Sell It

   Shopping mall fast food chains face some of the fiercest competition around. Sales quantity is king in a confined space with direct competition, low profit margins, and a product lifespan measured in hours instead of days or months. The presence of free samples in food courts isn’t primarily motivated by desperation but by survival. Fast food chains continue to offer free samples month after month in shopping centers across America because they understand that the cost of giving away small amounts of their product is adequately covered by the sales those samples generate.

   This same principle is applied in virtually every industry. Clothing stores often allow customers to use in store fitting rooms to try on outfits before purchasing. Automobile dealers allow qualified leads the famed “Test Drive,” and the movie industry produces previews of all new releases to entice the viewer to later enjoy the full experience.

   Samples sell, and this age old axiom may be even more effective when applied to digital products than to physical goods.

   Consider the case study of iCombat, a top 100 iPhone application. The month after the makers of iCombat released a free, “lite” version of their paid application, they were rewarded with an 8.73% conversion rate. The most impressive part? A conversion rate of 8.73% resulted in a monthly sales revenue increase of 496%⁹. From the iCombat creator:

   we had waited months longer than we should have to launch a lite version. There was no point to waiting and sacrificing the initial new release buzz.

   When it comes to profiting from the power of sampling, iCombat isn’t alone. Mobile analytics company Flurry released a study of the impact sample applications had on paid application sales and found that there is on average an 85% “free-to-paid” sales lift generated by application sampling¹⁰. From the Flurry report:

   Among your strongest marketing plays in the App Store is to offer a free trial of your game or application. Not only is the App Store designed for this, but also it’s the best way to reduce consumer risk in trying your application, with the goal of eventually getting that user to purchase the full version. Think: money. And from our data, it’s among the most effective moves you can make.

3. Make a Good First Impression

   1/20th of a second. That’s how long you have to make a good impression on the typical web user according to Dr. Gitte Lindgaard of Carleton University in Ontario, Canada. Dr. Lindgaard published an article in Behaviour and Information Technology in which he describes that a visitor not only forms a cognitive bias in the first 1/20th of a second after visiting a web site, but that this cognitive bias, formally known in psychology as the “halo effect,” would also significantly influence the user’s opinions on the reliability and usability of the web site.

   In the words of Dr. Lindgaard:

   …the strong impact of the visual appeal of the site seemed to draw attention away from usability problems. This suggests that aesthetics, or visual appeal, factors may be detected first and that these could influence how users judge subsequent experience…. Hence, even if a website is highly usable and provides very useful information presented in a logical arrangement, this may fail to impress a user whose first impression of the site was negative.

   While no studies have been published to measure the impact of the "halo effect" in mobile applications, it is undoubtedly an important aspect of the user experience. While a thorough discussion of design principles is beyond the scope of this article, consider the fact that proper use of color alone has been proven to increase brand recognition by up to 80%¹¹. In visual mediums, color certainly matters. Give your users the right impression by abiding by the principles of color psychology shown below:

Success is not limited to any single medium, and neither are your consumers. The companies that realize and apply that to their operations today will be the companies taking consumers into the frontiers of tomorrow. Be among them by placing the needs of your consumers first and then utilizing whatever technology is necessary to serve them.

Footnotes

1. http://www.apple.com/pr/library/2009/04/24appstore.html
2. http://www.w3schools.com/browsers/browsers_stats.asp
3. http://en.wikipedia.org/wiki/Firefox#Market_adoption
4. http://share.skype.com/sites/en/2008/09/celebrating_1_billion_download.html
5. http://www.apple.com/pr/library/2009/09/28appstore.html
6. http://bits.blogs.nytimes.com/2009/05/27/google-expect-18-android-phones-by-years-end/
7. http://www.census.gov/compendia/statab/tables/09s1118.pdf
8. http://macdailynews.com/index.php/weblog/comments/16715/
9. http://www.theapplicationfarm.com/2009/07/just-how-much-does-a-lite-version-help-boost-sales/
10. http://blog.flurry.com/bid/19375/iPhone-App-Store-Marketing-Give-it-Away-to-Get-Paid
11. http://www.colormatters.com/market_whycolor.html

The choice to design an architecture with commodity hardware in mind comes with some enticing benefits. First, instead of one expensive widget, I can afford a bunch of cheaper widgets and spread out my work load among them, which also helps isolate failures and improves the overall continuity of service to my customers. Second, it allows me to scale my solution as the demands of the business grow. Third, money saved by avoiding pricey hardware is freed to be spent in other areas.

Data storage is one market where there is a stark difference between enterprise and commodity hardware. The first time I heard the term RAID, I learned that the “I” stood for “Inexpensive”. Later, I discovered that it is often given as “Independent”. Both make sense in context, but it seems now that the former meaning has been lost when 15K-rpm drives are de rigueur and sit at the top end of the price range. Lowly 7200-rpm or even 5400-rpm SATA drives occupy the low end. This is but one area of computing where commodity drives is not seen as capable of matching more expensive, enterprise drives. However, a holistic systems approach reveals that there are plenty of places where commodity drives makes sense in terms of delivering on business goals and ensuring a high quality of service.

At the high end, dedicated storage arrays with custom hardware controllers filled with 15K-rpm drives define enterprise storage. These are speed demons; the high spindle speed means low latency (around 2ms, compared to 4-5ms for 7200 rpm drives). To drive latency even lower, some arrays use a technique called “short-stroking,” which utilizes only the innermost area of each disk platter, minimizing the distance that the heads must move. Such luxury comes with a steep price. Short-stroking reduces the usable space of each drive, requiring more drives for a given amount of storage. Not only that, but 15K drives max out at 300GB, so obtaining the kind of storage sizes required by large enterprises requires entire cabinets full of disk shelves. 15K drives are power-hungry, hot-running monsters which, at a time when electricity and carbon footprint concerns are becoming increasingly important, means that the luxury of high performance is ever more expensive, outstripping the performance gains of adding more spindles.

By contrast, a commodity storage approach seeks to maximize storage space and minimize costs, both initial and ongoing. The same budget can buy more spindles to keep latencies down and IOPS up. These additional spindles can fit within the same (or less) space and power budget as well.

The commodity storage picture is not all rosy. Enterprise drives are manufactured to withstand the higher level of vibration that comes with putting a lot of drives into the same chassis. They also have lower bit-error rates and higher MTBF than their commodity cousins. Simply put, commodity drives fail more often. “Failure” could be anything from silent data corruption to outright mechanical malfunction. Commodity drives also don’t spin as fast — 7200 rpm at most. That increases latencies, especially on random read loads. Any solution that employs commodity drives must account for these realities and work around them.

Thanks to some excellent, disruptive technology from Sun, namely ZFS, I can design a system around inexpensive, 7200 rpm drives, bolstered by a few SSDs, which provides more capacity with fewer spindles and improves read/write latencies far beyond the capabilities of short-stroked 15K drives. ZFS features such as end-to-end checksums, guaranteed on-disk consistency, intelligent prefetch, and immense scalability make it a good fit for my motley array of cheap disks. If I run low on space, adding more disks is extremely simple and cost-effective. Likewise, features like self-healing data and top-down, metadata-driven resilvering mean that silent data corruption and device failures don’t have to be the ulcer-inducing events they once were.

Storage is just one example of how some careful thought during the design process can yield significant savings during implementation. The same theory applies to entire server farms when deploying web applications. An application that is designed to scale horizontally can be run on a large number of cheap servers rather than a few very expensive ones.

Sometimes it doesn’t make sense to go the commodity route. It is as important to know when you can scale horizontally as it is to know when you should not. An application may require more engineering to rearchitect it to scale out than it would cost to buy a larger single machine to scale it up. I’m not just talking about high-end RISC gear either — some relatively large x86-64 configurations are possible. For example, the upper end of IBM’s System x line can be configured with up to 4 4U chassis linked together in an 8-socket, 48-core, 128-DIMM system. That’s a monster box (cue the Tim Allen grunts). If you can run your app today on one machine, and you can plan its growth to fit into a monster box, then the cost-effective approach may just be to use the larger machine.

What makes this even more complicated is that once you spew forth the result to the end-user you have a daunting set of user-perceptible performance issues remaining to be addressed. This performance challenge happens in a hostile environment: one we do not control (the user’s computer) over a long-haul network we do not control driven via a browser we did not select — a daunting challenge indeed.

We are very fortunate to have excellent tools at our disposal with which to tackle this challenge. Two of my favorites are Yahoo’s YSlow! and Google’s Page Speed tools. Both are extensions to the most excellent FireBug add-on for Mozilla’s FireFox web browser. Both tools will help you dissect the various aspects of the content you deliver to end-users and understand how each bit will contribute to perceived slowness. In the web (and most other things in life) perception is king. A user’s perception drives their response.

Irony: the not-so-delicious kind.

I recently attended the Velocity conference and the first workshop I attended was Steve Souders’ excellent presentation on Website Performance Analysis. Steve Souders is the original author of YSlow! which I use on a daily basis. Steve used YSlow! to show how to analyze website performance (as one might have assumed from his workshop title). I popped open YSlow! on our corporate website and… horror!

While OmniTI has enormous breadth in the Internet space, we are primarily known as an Internet performance and scalability company. This made the fact that we received an F on YSlow! all the more embarrassing. This was a case of the right hand not knowing what the left hand was doing — something we evangelize against. I decided that I would fix that and aim to do it by the end of Steve’s presentation. A play-by-play follows.

No Expires Headers.

It turns out that our images, javascript, and CSS didn’t have expires headers. Our CSS is in a directory /c/, our javascript is located in /js/, and all our images are in /i/. I could do this by content type, but a location-based approach gives me the flexibility of serving dynamic/uncacheable content with those content types if I choose to later:

<Directory "/www/sites/omniti.com/www/i">
    ExpiresActive On
    ExpiresDefault "access plus 1 month"
</Directory>
<Directory "/www/sites/omniti.com/www/c">
    ExpiresActive On
    ExpiresDefault "access plus 1 month"
</Directory>
<Directory "/www/sites/omniti.com/www/js">
    ExpiresActive On
    ExpiresDefault "access plus 1 month"
</Directory>

Using Etags.

Etags are on. This isn’t really a problem in and of itself, but since some of our static content can be served by multiple machines and the Etag in Apache is based off inode, it will be different from machine to machine and cause issues:

<FilesMatch "\.(js|css|gif|png|jpe?g)$">
  FileETag None
</FilesMatch>

Uncompressed content.

This is even easier. We run Apache 2.2, so:

AddOutputFilterByType DEFLATE \
       text/html text/plain text/xml \
       application/javascript text/css

No CDN.

We have a fast CDN-like caching layer residing at s.omniti.net that we can leverage… so I flipped all the images over to that. Technically, this is cheating because you have to add s.omniti.net to the YSlow! configuration to be recognized as a CDN. I was pleased to learn that even without formally moving the images to a known CDN, we still moved to an A rating in YSlow!

Assets served from a domain with cookies.

The move of all static assets to s.omniti.net resolved this issue. This goes to show that even if you don’t have a CDN, simply putting your static assets in a different domain (that has no cookies) can considerably speed performance in two ways: (1) it allows for more concurrency on the network layer and (2) it reduces the upstream payload for quicker requests.

The result?

A noticeably faster web site in under 45 minutes.

Before I fixed things up, it took 486ms to render (over the conference Internet connection).

After, a bit of work, I was able to drop the time-to-render to 315ms over the same link. That’s a 35% reduction and it almost drops the page load time down into the “so fast it doesn’t matter” arena.

There are several things I’d like to do that would further improve page load/render times. The javascript used could be consolidated into a single js file (aside from the web analytics parts). The CSS could also be consolidated from two files to one. On our about page we have thumbnail photos of all our staff, they are all the same size and we could easily turn this into a single image and use CSS sprites; that would dramatically improve the perceived performance of that page.

Some things we did right? Our search is wicked fast as we pull the results in AJAX and make a single DOM manipulation to visualize them.

Next steps.

Go fix your site. Make it faster. Make the web a better place. It took me 45 minutes to make significant positive impact. Granted, if I didn’t know your web application or it was more complicated than our corporate site (which I believe all are), it will take a bit longer. It’s worth it. Do it. Or hire us to do it.

While attending the summit that helps plan this conference, I had two epiphanies:

1. a realization of the lack of a career path for people who do what we do (no standard titles, no standard roles and responsibilities and certainly a lack of sex appeal);
2. a clear lack of terminology for the technology requirements that are so common in these environments.

Terminology is easy, in my opinion — you just argue until someone wins. Of course, arguing is a hobby of mine, so I have bias. On the other hand, defining a career path that is an industry accepted path is hard.

The Career: Web Operations

The term Web Operations was used a lot during this event. While it is not awful, I really do not like this term. The hard part is that the captains, superstars, or heroes in these roles are multidisciplinary experts. They have a deep understanding of networks, routing, switching, firewalls, load-balancing, high availability, disaster recovery, TCP & UDP services, NOC management, hardware specifications, several different flavors of UNIX, several web server technologies, caching technologies, several databases, storage infrastructure, cryptography, algorithms, trending and capacity planning. The issue: how can we expect to find good candidates that have fluency in such a nimiety of technologies? In the traditional enterprise, you have architects which are broad and shallow and their team of experts which are focused and deep. However, the expectation is that your “web operations” engineer be both broad and deep: fix your gigabit switch, optimize your MySQL database and guide the overall architecture design to meet scalability requirements.

I struggle with this. Not everyone can be a superstar. More importantly, no one can really start as a superstar. If we use an apprentice model (which is common in industries without institutional support) we limit the total number of able workers in this field. So, how do we (re)define the requirements for a junior web operations person?

We have to have a plan for hiring on people and progressing them through a career path to make this a legitimate discipline. During conversation, one of my colleagues said they just hire people that they think are agile — “If I tell them to know IOS well enough to configure a router and troubleshoot a problem, I expect them to show up tomorrow with a basic understanding of IOS and ready to start typing in commands at a console.” I agree this sort of “no boundaries” attitude is required for the job, but where do you start?

Another person mentioned that the reason for the lack of sex appeal in the position was due to popular attitude. Many people apply for development positions and “don’t quite make the cut” and are instead offered system administration positions. I personally don’t subscribe to this philosophy and we certainly do not operate like that at OmniTI, but I have seen it in other companies — I hope it is not prevalent.

Basically, this is one of the few positions in the organization that has no boundaries of responsibility. If something breaks, it is your problem. Why isn’t this the case throughout the organization — why is it that even the most junior of developers doesn’t wake up to fix their code when it breaks and causes service degradation in the middle of the night? It is uncommon that this level of responsibility is expected of developers, while it is a quite common expectation of the operations crew.

Circling back, I really do not like the term “web ops.” I realize it is not far off, but it isn’t sexy. Google has a few different roles with this level of responsibility. One I like is called: “Site Reliability Engineer.” However, I would like a set of job titles and a progression through them that makes this an appealing career path for young, ambitious geeks.

In order to define these roles, we should think about what they are responsible for. In our organization I see this as a few things:

Junior

On the junior level, they are responsible for learning. They are responsible for deploying new services and documenting such deployments. They are responsible for instrumenting deployments to make sure that faults are detected and trending is possible.

Mid-level

On the mid-level, they are responsible for all of the above, and more. Effective and complete troubleshooting of failures. Making sense of trending information. Understanding work loads that exist. Tuning systems to better accommodate current workloads and proactive tuning to handle known future workloads. One of the key differences between mid-level and junior is the ability to correctly prioritize remediation of issues during incident response. Staying calm, collected and executing with clarity of thought during an emergency.

What does “complete troubleshooting” mean? I mean troubleshooting without boundaries. I want no shyness in cracking open developer code and telling them what they did wrong and why. Finger pointing at people simply doesn’t work, you have to point your finger at implementation problems, not people. To do that requires the skill to track a performance problem or reliability issue down to a specific line of code or approach.

Senior

On the senior side, technology research and selection is a must. Additionally, they are responsible for incorporating new technologies in the architecture to improve availability and reduce costs, constantly analyzing systems to improve efficiency and capacity planning to understand growth well enough to ensure provisioning and deployment outpace need. Donald Knuth long said that premature optimization is the root of all evil; I’ve long said that the ability to accurately determine what is premature separates senior from junior.

One of the core responsibilities that all engineering disciplines share is assessing the appropriateness of the technologies at hand. For example, a “Web Architect” must ensure that technology selection as well as development and deployment strategy match the business need. This is “hard.”

Above and Beyond

Web operations is a special role. This role is in no way fitting for failed developers, it is for developers/engineers that have outpaced their career path. One that has a deep understanding of how things work: “a complete systemic view of general site architecture.” However, they want more responsibility, they want to make sure that all of it works all of the time: the app, the stack, the hardware, the network. Whatever technology the business needs, it must work, it must performs and it must be able to meet demand. Lastly, in their heart of hearts, they must believe that all problems are equal in their need for resolution and problem prioritization is dictated by business impact and not by flights of fancy (how cool or interesting the problem is).

It is an impossible job requirement: “Knows everything about all technologies deployed in Internet architectures.” While no one fills this requirement, what I want is someone whose career goal is to find out how close they can get.

There are more and more services providing outsourced storage. The concept is simple: you upload a digital asset to the vendor (via some sort of API or tool), they return an identifying key of some sort (sometimes this key is provided by you, the uploader) and they store the asset for you. To retrieve the asset, you use a similar method to the one used to upload. In the simplest terms, you can think of it as a mapped network drive to which you can save assets, and later reconnect to retrieve them.

By no means is this new technology. However, the idea of managing one’s own storage, combined with growing space requirements and fear of loss due to lack of redundancy, have driven people to want to make this particular problem someone else’s. Making this choice — to solve the problem yourself or to outsource — is always the outcome of several factors: cost, convenience, and safety.

Redundancy: The basics

Let’s take a look at the fundamentals of data storage. We all want our data to be safe. It’s pretty obvious that storing exactly one copy of the data isn’t safe, but it’s actually more complex than you would think — storing two copies doesn’t buy you much without taking a few extra steps.

Before we dive in and explore methods for keeping data safe across systems, we need to realize that one of our fundamental assumptions is invalid. We assume that when we write data to a disk, it will have no errors when we read it back. This assumption is fundamentally wrong. There’s this little evil thing called a bit error (basically, one of the zeros or ones that was written came back inverted). How often this type of error occurs is a probability called bit error rate (BER). The BER on modern spinning disks is usually around 10^-13 or 10^-14. Basically, for every 1 to 10 terabytes you write, one of the bits — when read — won’t equal what was written. A single erroneous bit might not matter for some types of data, but for others, such an error could be disastrous. We write a lot of data these days, and bit errors are silent, so the lesson here is: write checksums with your data.

The classic method of ensuring that data is safe is to store multiple copies on different physical media. Inside a single system, this can be accomplished with RAID1 (mirroring), which makes sure all data is on two physically separate disks. With a bit of (somewhat) clever math, we can take that same data, split it into a few pieces and store each piece on a different drive. We can then calculate a block of parity data, and store that on an additional drive. Retracing the same math backwards shows that we can lose any single disk in the set, and we’ll still be able to reconstruct our data. This is the basis for RAID5. Sometimes systems need to be resistant to multiple concurrent disk failures (hence the introduction of RAID6, which uses an erasure code such as Reed-Solomon).

None of these scenarios are designed to reduce the risk of data corruption. Rather, they were designed to prevent data loss due to hardware failure of one or more underlying disks. One issue with using RAID is that you are storing files on a set of drives, those files consist of chunks of data (blocks) which map to physical blocks of bits on the drives, and somewhere along that path we could lose our way. If a specific physical block goes bad, or somehow becomes unreadable, we can’t easily map it back to a logical object, such as a file. We only find out that there’s a problem when we try to read the object. Another problem with this general technique is that all of these disks live in a single system and if that system fails, all of the data is unavailable (or worse, lost).

So, RAID is designed to keep our data somewhat safe within a single system, but it doesn’t address system failures. The most obvious design is to put all of our information on two systems. There are pros and cons with this approach. On the positive side, once we’ve identified which system holds a copy of our asset, we only need to communicate with that single system to retrieve a copy of the asset — simplicity. The downside here is that we’ve used half of our storage as redundancy, and yet if two of our nodes fail, we’ve necessarily made unavailable (or permanently lost) 1/(N*(N-1)) of our assets. With two nodes, this works out to 100% (of course), and with 10 nodes, it’s around 2%.

Taking a different approach altogether allows us to use half of our storage for redundancy, while maintaining dramatically greater availability.

Erasure codes

High availability of assets in light of system failures is achieved by today’s peer-to-peer systems. Their technical description is clear-cut, yet extremely detailed. By using erasure codes, these systems are able to split data into many pieces (similar to RAID5), but instead of calculating simple parity, they calculate unique erasure codes.

Imagine we split our data into 5 pieces, and then calculate 5 additional pieces of data, any of which could be used to reconstruct any of the original 5 pieces were they found to be unavailable — these are erasure codes. So, with the data in 5 fragments + 5 erasure fragments, we’ve consumed twice the space but can now stand to lose any five pieces before the data becomes unavailable and/or lost. The main drawbacks to such a system are that calculating and distributing erasure codes is much more complicated than simply storing two copies of the same data, and that retrieving data requires contacting at least 5 machines to serve an asset.

This erasure code approach assumes a slightly larger network of servers. With two copies and 100 machines we see 99.8% availability with 5 machine faiures. With a 10 fragment (5 data + 5 coded) scenario, if 5 nodes fail, we maintain 100% availability. In the pathological case where 50 of our 100 nodes fail, the two-copy method would result in an availability of approximately 75.3%, whereas the erasure code method would achieve approximatately 98.7% asset availability.

Back to reality

In peer-to-peer systems, where clients enter and leave the network rapidly, the use of erasure codes for high redundancy is quite necessary. However, in a datacenter environment, with redundancy on each system and maintenance windows that we control, the situation is entirely different. Controlling the servers, their configuration and their region of deployment gives us a landscape on which we can build a sufficiently redundant system with all sorts of advantages.

Reduced system complexity and simple distributed processing are significant advantages that result from having whole data objects like images or documents present on a single node. With this model, we can offload some computational processing to the nodes that hold the data and they can act without consuming additional resources such as the CPU time and network bandwidth required to reconstitute whole objects from their distributed pieces.

At the end of the day, a hybrid/adaptive approach between the two would yield the best outcome. I see that being the next thing in distributed storage. Most of us that are faced with storing large amounts of data have already thrown traditional filesystems and POSIX-compliance to the wind and are looking for fresh, more appropriate solutions to our specific problems.

For now, until these merge, the approach of redundantly storing whole assets makes the most sense. It is simple and easy to build, deploy and administer. It is also trivially easy to understand and troubleshoot.

In many cases, web applications can be combined on a single server using virtual hosting facilities in Apache, but this is an imperfect solution. Inevitably the situation arises where you have an application that doesn’t play well in a virtual hosting situation, be it badly written, or requiring specific versions of libraries or modules that conflict with another application. There are also administrative concerns — anybody who has access to one application has access to them all.

The virtual hosting method also eliminates one of the biggest benefits of using virtualization on entire servers. Many virtualization technologies provide some method of transferring a virtual machine between physical hardware — if a particular server is behaving badly, just transfer all the virtual machines onto replacement hardware with little to no loss of service and without having to reinstall the operating system/applications.

Here at OmniTI many of our servers run Solaris, giving us two very powerful features on which we heavily rely when it comes to making use of virtualization: Solaris containers (Zones), and ZFS.

Virtualization using Zones

Zones provide lightweight virtualization for Solaris. Unlike many other virtualization solutions such as VMWare or VirtualBox, Solaris zones don’t emulate physical hardware on which several complete operating systems run; rather, there is one kernel running in the system with multiple partitions (the zones) in which user programs run.

This type of virtualization doesn’t force you to pick a set amount of RAM for each virtual machine, or set up virtual disk images (although resource limits can be set for each zone). Because there is no hardware emulation going on, zones are also incredibly fast — fast enough that we are able to run multiple production services on a single machine without any perceptible slowdown. Even for high traffic sites that can saturate an entire (physical) server, we are still able to make use of zones (with just one non-global zone per server) without any significant performance hit. This allows us to benefit from the ease of moving a zone from one machine to another, either in the event of hardware failure, or to migrate to a more powerful machine.

Zones can also ease administration of multiple servers by centralizing package management. By default, any package installed on the global zone is automatically installed to all non-global zones. You can also specify that certain paths are inherited from the global zone, reducing disk space requirements per zone. The inherited paths become read only, forcing them to be the same across all zones. If all packages are installed from the global zone, and you make use of inherited paths, then you can be assured that every zone has the same software configuration.

However, this doesn’t have to be the case — you also have the option of installing packages in the zones themselves if different zones need different packages installed. To do this, don’t inherit any directories. This creates a 'large' or 'whole root' zone, and you are free to install whatever is needed inside the zone itself.

ZFS and Zones

ZFS has many useful features that put it far ahead of most other filesystems that are available. Several of them are of particular interest in that they make virtualization better: a pooled storage model, snapshots, and the ability to transfer filesystems via the zfs send command.

Pooled storage does away with the idea of having filesystems on individual partitions, and having to guess how much space will be occupied by individual filesystems. You just create one pool across the entire disk (or set of disks) that you want to store your data on. Any filesystems you then create in that pool will only use up as much space as needed to hold the data.

In practice, this means we can create individual filesystems for each of our zones without having to worry about how much space to assign to each. Having each zone on its own filesystem is required to be able to snapshot, backup, restore and transfer zones individually.

Snapshots give you almost instant point-in-time copies of your filesystem, each of which only take up enough space to hold what has changed since the snapshot was taken. The benefits of this are numerous including the ability to roll back to an earlier time and consistent backups (take a backup from the snapshot, and you won’t have files being modified while the backup is in progress). From the point of view of virtualization however, one of the biggest benefits of snapshots is in combination with zfs send.

The zfs send command allows you to send a snapshot of a ZFS filesystem from one machine to another (or on the same machine, if you so desire):

# zfs send data/zones/myzone@somesnapshot | \
    ssh remote_machine zfs receive data/zones/myzone

This allows you to quickly move (or copy) a zone from one machine to another: detach your zone, zfs send the filesystem to another machine, attach the zone, and you have your zone up and running on a completely different machine.

You can also make use of incremental snapshots to minimize the amount of time the zone is down (a zone has to be halted in order to detach it): snapshot the zone’s filesystem and send it across while the zone is still running, shut the zone down, detach it, snapshot the zone’s filesystem once more and send the incremental snapshot across.

Until recently there were issues with upgrading zones that live on a zfs filesystem, but this has been fixed in Solaris 10/08 (u6), and Live Upgrade is now supported. There is now little reason not to use ZFS as the filesystem for zones.

Backing it all up with Zetaback

It doesn’t take much thought to realize that the snapshot/zfs send tools can also be used to take backups of systems, especially when you make use of incremental snapshots. At OmniTI we have developed Zetaback, a backup tool based on zfs that automates much of the work of taking and managing backups.

With Zetaback, you specify a list of hosts, the retention policy, and how often to take a full/incremental backup. Then you just let it go. It connects to each host via ssh, scans the host for filesystems to back up, and by default will back up everything, automatically picking up new filesystems. You can filter the list using regular expressions if you want to limit what is backed up.

In addition to taking backups themselves, Zetaback provides tools to quickly restore zfs filesystems, view the status of backups and generate reports showing which filesystems violate the backup policy (e.g. those that have not had a successful backup in 1 week).

The choice to make use of virtualization is often an easy one, the choice of which solution to go with is somewhat harder. If Solaris meets the needs of your applications, then it is worth considering Zones. Combined with the features of ZFS and Zetaback, they provide a flexible and powerful solution.

Some real-world numbers

We’re a web infrastructure and development shop, so we run a lot of development servers. Each environment needs the flexibility of its own software selection including version. To accommodate that, we run 37 zones on 2 development servers. Each development server has 8GB of RAM and two dual-core 64-bit AMD processors — in financial terms: about $2300 each. Our production boxes, that serve corporate mail, document management, version control, instant messaging, directory services, etc., all run in zones also. For that we have two boxes (just like the development ones) on which 17 zones happily reside. All of our important services run on a rather small set of machines — easy to manage, cheap to power and cool. And for our purposes, it is far more efficient than heavy-weight virtualization like VMWare ESX.

We’ve been running this type of light-weight virtualization for over two years now. We’re pretty happy with it. I suggest you give it a whirl.

ORMs provide an automated link between the application object model and the database model. Practically speaking, this generally means that tables become classes and rows become objects. At the most basic level, this provides per-object persistence services. Most ORMs also handle relationships between objects, turning compositional relationships between objects into foreign key relationships in the database.

In this entry, we present the benefits and pitfalls of ORMs and introduce a new Perl ORM implementation, Class::ReluctantORM. Class::ReluctantORM is “a reluctant ORM for reluctant people.” Its design goals are to create a framework that is unambitious, scalable, and easily circumvented. These goals are not so much technological as philosophical; an approach that has value, as we shall see.

Benefits of ORMs

Why developers love them

Developers see a tremendous productivity gain: persistence no longer has to be hand-crafted into each class. Since all model classes are using the same techniques, there is a large consistency gain as well. Developers also get to keep their head in application-model space, without having to shift gears into database-model space. Context-switching is expensive, and switching between implementation languages can be especially jarring.

Consider getting a list of pirates on a ship:

  my $dbh = DBI->connect(…);
  my $sth = $dbh->prepare(>>EOSQL);
  SELECT p.* 
    FROM highseas.pirates p
      INNER JOIN highseas.ships s ON s.ship_id = p.ship_id
    WHERE s.name = ?
EOSQL
  $sth->execute('Golden Hind');
  my @pirates;
  while (my $row = $sth->fetchrow_hashref()) {
     push @pirates, Pirate->new($row);
  }
  $sth->finish();
  foreach my $pirate (@pirates) {
     # Do something with $pirate
  }

Compared to:

  my $ship = Ship->fetch_by_name('Golden Hind');
  my @pirates = Pirate->search_by_ship($ship);
  foreach my $pirate (@pirates) {
     # Do something with $pirate
  }

The second example is much more legible, and remains entirely in the application model — you don’t have to think about how the database is set up, or how the tables interact. You don’t even need to know SQL.

Additionally, most modern ORMs can shield the business logic from limited changes in the database schema (such as table or column renames). While this sort of change is usually better hidden at the database logical layer using a view, organizations that have restrictive database change policies may appreciate the added flexibility.

Why leads love them

Team leads find ORMs to be very useful for several reasons. The most obvious is the reduced amount of time spent wiring up persistence layers; instead, developers can stay focused on the business problems. This enables new capabilities. For example, using an ORM, it’s much easier to knock out a quick prototype in response to an RFP, or explore an alternative design. Additionally, since the amount of SQL is dramatically reduced, developers need not have SQL skills to be productive.

Why project managers love them

Project managers like ORMs for many of the same reasons that team leads do. Because productivity is increased, bids can be lower, or more features can be delivered for the same schedule. This may lead to more contracts. The reduced skillset needs of an ORM-based project can also help solve staffing problems.

Pitfalls

DBA Gripe #1: ORMS that dictate DB design

Some ORMs dictate database design. These constraints typically center around keys. Commonly, primary keys are required to be be single-column, integer, and auto-incrementing. Foreign keys are often under the same constraints. This leads to the proliferation of artificial keys.

Naming conventions are another sore point. Some ORMs require primary keys to be named id, others require them to be named pirate_id, or even pirate. Tables may be required to be named in the plural, and one ORM’s notion of pluralization may not match that of another ORM (e.g. staff vs staffs vs staves).

These constraints are annoying but tolerable if the database design is new and the ORM-based application is the only client. But in most real-world situations, the ORM-based application is only one consumer of the database. It may be a pre-existing design with several legacy apps already using the schema. It is possible to use views and rules to appease the ORM’s requirements, but that trades the developer’s productivity gain with a busywork task for the DBA.

Finally, there is the issue of schema ownership. Both the ORM and the DB know about the database structure. When a change is needed, where do you make the change? Some ORMs “own” the schema, and will execute DDL to modify the database to match changes in the object model. Others don’t own the schema but instead mirror the database schema in a configuration file. Better ORMs read the database at startup, and configure the object model accordingly (though this has problems of its own, especially related to startup speed).

Class::ReluctantORM — stay agnostic

Class::ReluctantORM is firmly in the “read the schema from the database on startup” camp. Some configuration is still needed — to set up connection handles, declare classes, and to create relationships that cannot be auto-detected. Pushing more of the configuration into auto-configurators, while maintaining overridablility, is an active area of development.

DBA Gripe #2: Opaque, Baroque Query generators

An ORM, by its very nature, must contain some kind of query generation mechanism. SQL is an easy language to generate, but a very difficult language to generate well. There are many dialects. An ORM may choose to generate standards-compliant (but slow) queries, or it may attempt to optimize for the particular database engine. As the optimization increases, the query complexity often increases. Some ORMs choose to punt, generating many simple queries (see DBA gripe #3); others may generate one massive, multi-JOIN query. In either case, at some point you will get the classic complaint that “the database is slow.” A DBA wants to be able to tune and replace these queries with hand-crafted versions. This may or may not be possible. Even if it is, the query generator is buried in the ORM code itself, in developer-land, and often requires both developer and DBA to invest time to optimize a query.

Class::ReluctantORM — query monitors

It is important that the SQL generation process be as transparent as possible. To this end, Class::ReluctantORM provides a unique monitoring facility that provides hooks for several key events in the life of a query, including initiation, SQL generation, execution with bound parameters, result fetching, and teardown of the query.

Monitors may execute arbitrary Perl code at any or all of the events. A monitor may abort a query if needed, or simply log statistics or debugging data. Monitors may be attached at compile time or runtime, and may be attached to a particular class, or all classes in the model.

Class::ReluctantORM ships with six canned monitors, including those for join count, column count, data volume, timing, diagnostic, and one which executes the query under EXPLAIN ANALYZE to predict performance. The developer is free to add new monitors.

DBA Gripe #3: hidden expensive actions

Consider this expression:

  my $jewels = $ship->pirates->first->hideaways->find_by_name('Skull Island')->treasures->first->jewel_count();

While it won’t win any awards for formatting, it is fairly clear: get the number of jewels that the first pirate on my ship has stashed away on Skull Island. It’s easy to imagine a junior programmer writing this, or a journeyman programmer writing a less contrived example.

Does this code, at first glance, look like it is hammering the database? How many database queries will this result in? Depending on the ORM, it may range from 1 to 6. And depending on the database, the 1 query might be better or worse than the 6. In almost every case, the queries involved will pull back more information than they need from the database, so even when the ORM gets the queries right, it’s still likely to have unnecessary overhead.

Or this common case:

  foreach my $ship (@fleet) {
     foreach my $pirate ($ship->pirates()) {
        foreach my $hideaway ($pirate->hideaways()) {
           foreach my $loot ($hideaway->treasures()) {
              tithe_to_queen($loot);
           }
        }
     }
  }

That should have scared you.

Class::ReluctantORM — mandatory prefetching

Looking back at this example:

  # 1-6 queries
  my $jewels = $ship->pirates
                    ->first
                    ->hideaways
                    ->find_by_name('Skull Island')
                    ->treasures
                    ->first
                    ->jewel_count();

This usage is problematic because:

• There is no indication that queries are occurring.
• Any performance issues will be detected in production, not in development.

Class::ReluctantORM does not allow accessors to directly execute queries. Instead, each accessor looks for a cached value, and returns it if found. A cache miss throws a FetchRequired exception. A full-featured prefetching facility is available:

  # One query
  my $ship = Ship->fetch_deep(
    where => 'name' => 'Golden Hind',
    with => {
      pirates => {
        hideaways => {
          treasures => {}
        }
      }
    }
  );
  # Zero queries
  my $jewels = $ship->pirates
                    ->first
                    ->hideaways
                    ->find_by_name('Skull Island')
                    ->treasures
                    ->first
                    ->jewel_count();

This fetch_deep call executes exactly one SELECT SQL statement, JOINing against the related tables. The results are then processed to create one ship object, which has a collection of pirates, each of which has a collection of hideaways, each of which has a collection of treasures. This data is now prefetched, and a long, deep chain of method calls like above is now permissible.

Importantly, if a programmer adds a method call (say, $pirate->parrots) that is not prefetched, an exception will be thrown the first time it is executed. The developer will see this immediately in testing, and add the required clause to the prefetch. This integrates scalability directly into the development process. This feature, unique to Class::ReluctantORM, is what provides its name: it is reluctant to do database fetches.

Software engineering: impedance mismatch rabbit hole

We have a fundamental problem with ORMs: relations aren’t classes, and tuples aren’t objects. This problem is called the “impedance mismatch” between the database model and the application model, and is discussed in detail in several places on the Internet. Some of the more troubling issues include:

• Identity — Two objects referring to the same record are distinct, though there is only one record.
• Partial fetches — Most OOP languages do not have a notion of an object that is only partially populated, but it is perfectly valid (and desirable) to select only a subset of columns from a table.
• Inheritance — There is no clear analogue of inheritance in the database world. Several approaches exist, but they all have severe drawbacks.
• Caching — The object model will be out of date as soon as it leaves the database. Should results be cached? How long?

ORM developers are faced with a few nasty choices. Keep it simple, and let the user of the ORM know that the ORM is an unsynchronized, approximate model of the database. Or gradually add complexity, attempting to patch over the impedance mismatch. The latter path gets into diminishing returns quickly.

Class::ReluctantORM — 90% rule

Some ambitious ORMs try to solve 100% of the object-database problem. Class::ReluctantORM tries to solve the easiest 90%. That means it makes the choice that the impedance mismatch is a very hard problem, and the ORM will do its best, but you still need to be aware of its limitations. This scope limit helps exclude features that would dramatically increase complexity (for example, there is very little support for aggregates).

Skill atrophy

One of the big advantages of ORMs is also a major disadvantage: no, or little, use of SQL. We learn skills through exposure and experience, and if we are never exposed to SQL, we’ll never learn it. Or, if we know some SQL, then use ORMs exclusively, our skills will likely atrophy. In almost every application, the ORM will need to be bypassed at some point, and then SQL skills will be sorely missed.

Class::ReluctantORM — SQL pass-thru

For the remaining 10% of problems outside the scope of Class::ReluctantORM, several avenues are provided to bypass the query generator and use SQL directly. Because it was developed in a shop with a heavy mistrust of ORMs, Class::ReluctantORM is designed to make this bypass as easy as possible. The documentation mentions how to bypass the ORM early.

Avenues of SQL support, ranging from SQL-centric to object-centric:

1. Ask the ORM-managed object or class for a database handle, and execute statements on it. Results are in raw values, not part of the object model.
2. As above, but wrap this into a method call on an ORM object or class, thus integrating SQL into the object model. This is handy for aggregate functions.
3. Future releases aim to provide the ability to override specific ORM-generated queries with your own SQL.
4. Ask Class::ReluctantORM to intepret the SQL into its own representation, and execute. If the translation was successful, return values will be ORM-based objects. This is a Class::ReluctantORM-exclusive feature.
5. Write a query directly using Class::ReluctantORM’s abstract SQL engine. You’re no longer writing SQL directly, but performing method calls on FromClause objects, for example. This is guaranteed to return ORM objects.
6. Use ORM methods and pass SQL fragments as arguments (e.g., a WHERE clause for a search() method).

In all six cases, the developer must use SQL or SQL concepts. This may help reduce SQL atrophy. In many cases, because the SQL can be just “dropped in,” you can have a DBA or SQL expert develop SQL for a specific query with no contact with the ORM.

Framework lock-in

Like any framework, using one is often irreversible. It is very difficult to adapt an application to use a different ORM — even if the interfaces are similar, often times there will be differences among the query specifications, the DB requirements, or the semantics of operations (e.g. do inserts cascade?). ORMs are especially susceptible to lock-in because their footprint is so ubiquitous in the application code. Every time you deal with a relationship between your model objects, you interact with the ORM.

Class::ReluctantORM — unsuprising interface

While there is little that can be done to fight lock-in, Class::ReluctantORM tries to reduce the pain of switching to another ORM, or dropping ORM support altogether, by using common conventions for method names (accessors are named directly after the property, for example). When a new feature is added, the interfaces of other ORMs are studied, and similar conventions are adopted if possible.

Sometimes it’s the wrong tool

ORMs are not good for everything. ORMs by their nature are weaker at these tasks:

• Reporting and summarization — ORMs are good at treating rows as objects. What happens when a column is an aggregate? In this case, raw SQL is much more convenient. Aggregate APIs are often inflexible and complex.
• Anything involving fast startup — If the ORM queries the database for its schema at startup, there will be a lag before the ORM is ready. This isn’t a problem for long-running processes like web servers, but it can be a burden for command-line scripts.

Class::ReluctantORM — lots of fish in the sea

If an ORM isn’t right for your project, Class::ReluctantORM won’t help you. Even if an ORM is a good fit, keep in mind it is a slow-startup ORM. There are others that are fast-startup, large-configuration ORMs, and even some that can cache their configuration.

Conclusion

For all their pitfalls, the tremendous productivity advantages of ORMs will continue to tempt developers to use them. Like any productivity booster, ORMs seem to draw a lot of hype, and it’s important to see through the hype to the realities and shortcomings of the technology. Once those shortcomings have been addressed, however, ORMs can be used conscientiously.

Class::ReluctantORM is a new ORM implementation that seeks to make it harder to fall into the traps. It avoids some “impedance mismatch” issues by narrowing its scope to the most common 90% of use cases. For the more complex situations, numerous SQL bypass avenues are available. Whether queries are ORM-generated or customized, they all pass through the query monitoring system, providing an early warning system for scalability problems. Finally, mandatory prefetching can reduce bad coding practices early in the development cycle.

In the beginning our headquarters was located in Theo’s house. It was very nice but small. We built out a secure data room there in order to satisfy the security requirements of one of our clients. It had plexiglass windows, rack space and a 200-pound metal door. We personally did the build out which was an onerous job, to say the least! It seems mind boggling to think that we now manage thousands of machines in datacenters all around the globe. From that office we upgraded to a suite of three executive offices in Calverton, Maryland. While staying there we located unfinished office space in Columbia and designed it to accommodate our particular work requirements. We were growing fast and needed office space for our additional staff as well as for meeting with clients.

We started with two people and within three years grew to a staff of four. Our clients required services and support 24/7 so the days were long, as were the nights. Holidays, weekends and vacations were commonly workdays, and working into the wee hours of the morning after a full workday was the norm. The work scaled with the increase of staff from 2 to 4 people, so the workload remained the same across the board. Then we reached a point where we were in complete overload. At that point the engine needed overhauling and refitting to stay in the race. So we interviewed and hired 4 new staff - 3 developers and 1 system administrator. This was a major redesign.

The addition of new staff meant we had to make some serious changes to the shop. We needed an HR department and benefits package that would be both attractive and competitive with other companies — yet another upgrade. We also had to have a more comprehensive employment contract and that meant having a labor lawyer to advise us. This was in addition to the corporate counsel who helped us craft our client contracts. The pit crew was growing!

After the move to the new site in Columbia, we immediately increased the size of the team. Before we knew it we were fifteen strong and still growing. During this time we formally defined our product initiatives Ecelerity, Postal Engine and MultiVIP and then proceeded to trademark them. We also began to do business as a separate entity called Message Systems. Now we had two race cars on the track!

As the staff grew we began to understand how instrumental our culture was to our success and started to truly nurture it. This was and remains a unique work atmosphere that permeates the operations of the entire workforce. What exactly is the OmniTI culture? Ask ten people on staff and you probably will get ten different answers. I, on the other hand, have been part of that culture from the get-go and can tell you that, however it is defined, it is the heart and soul of OmniTI and the fuel that feeds the machine. The culture is derived from work principals that were instilled at OmniTI’s inception. These include providing quality services to meet clients’ needs as the number one priority; standing behind our work and being accountable for our mistakes; being passionate about our work; and using mindshare and brainstorming as working tools. Our office is designed specifically to enable and encourage this sort of interactive environment. As a result, the OmniTI culture has attracted some of the best and the brightest in the industry.

After the Columbia office came a second larger Columbia office, and an office down under the Manhattan Bridge (DUMBO) in Brooklyn, New York. We currently have a staff of 7 working in that office.

What about the crew you may ask? Over the years we have been fortunate to have a diverse staff representing a collection of ethnic and cultural backgrounds for which we are all richer. And our team has had many sponsors including trade associations, private industry, not-for-profits, political organizations, and government to name a few. As we zoom around the race track we also take time for the occasional pit stop by having pizza every Thursday, spring cookouts (with serious volley ball games), summer picnics and awesome holiday parties!

So as our race cars become more and more sophisticated they continue to require constant attention to maintain all the working parts. Each day brings new adjustments to the engines and frameworks to keep the motors fine-tuned and the goings smooth.

I’ll take this opportunity to share some helpful hints with the other mechanics out there:

1. Stay organized and, despite this digital age, keep hard copies of everything.
2. Retain legal counsel that understands your business and earns your trust.
3. Set deadlines for everything.
4. If you are going to do something, always take the time to understand how to do it right. If you don’t have time to execute it right, take the time to document the corners you cut and how that is likely to bite you later.

Zoom-zoom!

Newspapers and magazines have a unique opportunity with online publishing. They have the best content. They have the most talented writers, editors, and managers. The industry has survived everything the world has thrown at it since newspapers first emerged in the 16th century. But, making the most of the Web requires specialist help. Whether publications have a dedicated digital team, or integrate print and web together, there is a distinct difference in how editorial, design, and production need to operate. On the one hand, the recent furor about “deep linking” — where the legality of sites linking to individual pages was finally put to rest — showed the fragility of trying to apply standards that are reasonable in print to the Web. On the other hand The Telegraph has shown with its in-house video and audio studios, evolutionary news room and integrated advertising solution, that a radical content strategy can pay dividends. The whole approach for print media on the Web is evolving. It’s definitely a brave new world, but not everything has changed: content is still king!

Doom, gloom, flourish!

As some commentators, fueled by the news of falling readerships and economic woes, prematurely sound the death knell for print, The New Yorker published a beautifully researched article by Jill Lepore (in print and pixel) entitled Back Issues: The Day The Newspaper Died. In it, we relive the last time the death of print loomed large in America: November 1st, 1765. On that day the Stamp Act came into force. It required printers to affix a stamp to each of their pages, pay a halfpenny tax on each half sheet of paper, and a two shilling tax on each advertisement. Ostensibly, the tax was imposed in the “colonies” by the British Parliament to fund the French and Indian wars, but the backlash was severe. Printers were better placed than most to vent their frustrations. It has been argued that the Stamp Tax was one of the sparks that provoked revolution, bringing the issue of taxation without consent sharply into focus, with the ire and vitriol of printers like Benjamin Edes of The Boston Gazette, and Benjamin Franklin thrown in for good measure. They survived that calamity, and some newspapers like The Hartford Courant are still published today both in print and pixel.

Jill Lapore’s article in January coincided neatly with the month the credit crunch came home to roost. It was looming, it was imminent, then suddenly it was here. About the same time we also heard that the Pulitzer Prize-winning Christian Science Monitor was going online-only. The 27-year-old PC Magazine followed suit soon after. Various teen magazines previously had made the switch, including Missbehave, Cosmogirl, Ellegirl, and Teen. In December and January other publications also went online-only, including Hoy Nueva York, AsiaWeek (now redirecting to Time), and the Kansas City Kansan.

Far from “closing”, as some sources seem to suggest, publications are evolving just as they always have. The shift is perhaps a little faster and more pronounced than previous changes, but only a few titles are swapping print production for online-only. The vast majority of newspapers and magazines are augmenting print with online production. How well the transition occurs is the rub. Many will struggle with a half-hearted approach; others, like The Telegraph, are embracing the change, investing and flourishing. It’s also worth affirming that a publication doesn’t have to appear on paper to be worthy, a fact validated by the Pulitzer Prize Board when they announced in December 2008 that they were broadening the competition to allow online-only publications.

Content is almost enough

Print publishers still have the most important advantage: great content. The Web sometimes can seem packed with titillating dross, but people don’t thirst just for a quick bit of light-hearted refreshment; they also hunger for the substantial. Satisfying both with a content strategy specifically for the Web, putting the right infrastructure in place to support it, and understanding the behavior of the Web’s audience are the keys to success.

The Telegraph is investing in user experience design and using it to sell ad space. Get the user experience right and people will become subscribers and readers more readily than they ever have for print. The potential audience is global. The advertising revenue stream is global. The publishers who grasp the nuances of online publishing first will have the competitive edge to evolve into the first truly international news and feature sites. Those who invested early are already ahead: The Guardian launched in America in 2007. The paper predicts its podcasts will be profitable by April this year. Today, it has an almost equal split of readers, with a third from the UK, a third from the U.S., and a third from the rest of the world.

International audiences have very different requirements from content and advertising. Designing production and technical infrastructure that can deliver location-specific material is the future of truly international publications. The same digital content delivery channels (web site, RSS feeds, email, podcasts, and video) are still relevant. The material and style will respond to the context and the audience receiving it. Hot topics emerge much faster in the new era, and publications need the right technology in place to know what they are and be able to react. Some publishers are already providing location-specific content, and being very sophisticated in how they understand and serve their global audience. Advertisers are taking notice.

“Web 2.0” and all that jazz

Print publications provide an enacted narrative. Readers start at the cover, then flip, and read. They observe the story told by the publication in a pre-determined sequence, with only the words and images from the staff to tell the tale. In contrast, the narrative of the Web can be both enacted and emergent. The narrative emerges from both the published material of the professionals and the audience contributions. These contributions can be within the web site, on personal blogs aggregated by services like Technorati, or on social networks like Twitter. When Tim O’Reilly coined the term “Web 2.0”, user-generated content was at the core of his thoughts. Whatever we call it — user-generated content, Web 2.0, emergent narratives, or reader contributions — it requires an approach that is more sophisticated than just opening up a site to comments. There are many ways in which users can and should be able to reuse content, add their own, and participate. There are also many ways in which publciations can pull in content from around the Web to help them tell the tale. In fact, editing such content is a valuable service. You only have to look at the success of sites like Newsvine and Ffffound to see how citizen journalism can contribute to the industry. It’s not like the idea is new. James Franklin, editor of The New England Courant, had this to say about his editorial policy just before American independence:

I hereby invite all Men, who have leisure, Inclination and Ability, to speak their Minds with Freedom, Sense and Moderation, and their Pieces shall be welcome to a Place in my Paper.

Connecting the dots

The overall objective for newspapers or magazines remains the same as it’s ever been: a large audience to attract and retain high-paying advertisers. Meeting that objective online means combining the content with the best user experience. User experience is not a term found in the print world. Paper is a static medium. It’s a passive experience. Letters to the editor have been the traditional interaction of the audience with print publications. The Web is neither static or passive. It’s dynamic, with content being syndicated, read, and shared in new ways. It’s interactive, with content being augmented, reused and commented on as it’s published. Reader behavior has changed. Expectations have changed. People still want to passively read, but they also want to interact, republish, share, and comment — and they want these on demand. They want a different experience. It requires a different kind of strategy that understands the audience expectations and technology, and describes exactly how publications can use both to be successful.

User experience design

The first step is to have a clear set of business objectives in a reasonable timescale. Publications have to invest. How they invest is the big question. All the solutions are already available to bridge the gap between business objectives and audience behavior. Understanding audience behavior is the first step on the path to profitability. User experience design delivers just that. Rather than asking questions, it observes behavior. From that we can build a clear picture of how the audience experience can be optimized. That may involve more than just tweaking the design of the interface. It can encompass elements like content strategy: what is published, how it’s delivered to people, and how it’s written to achieve the business objectives.

Web application development

User experience design is nothing without the right applications to support publishing operations and deliver the content through the various channels. They should actively help journalists, editors, and managers do their job. Applications should make interaction for readers quick, easy and fun — an adventure. The software has to be secure. It has to scale well as (hopefully) increasing numbers of visitors find the great content, and keep coming back. The experience has to be fast, safe, and helpful. Anything less is a disservice to the reader in much the same way that badly printed text or art would be in print.

Internet architecture and infrastructure

Applications need machines to run on. That means intelligent technical architectures and infrastructure. If the audience is international then the infrastructure needs to be. That means multiple locations. It means twenty-four hour monitoring, often using tools written specifically for the task. It means rapid reaction times to fluctuations in traffic.

After our recent work with the award-winning National Geographic, their readership went up by 500%. The applications and infrastructure behind the site also had to handle massive traffic spikes as stories spread virally around the Web. Having infrastructure that can perform when the spikes arrive is almost an art. It can get expensive, and as we all become more conscious about environmental impact, performance is the answer. Friendster’s situation gave us a chance to show how to scale a site properly. They were about to launch in China and predicted they would need twice as many servers to do so. Page loading times were already slow at 9 seconds. With a little help from us, they launched in China with the same number of servers they already had. They doubled the number of users to 60 million, but pages loaded more than twice as fast at 3.5 seconds.

Stacking the deck

As good as the content might be, or the perception of the brand in the real world, when infrastructure or applications fail, or are hacked or threatened, the brand can be irreparably harmed in the eyes of readers. How this happens is often straightforward: using different vendors for design, application development, and infrastructure turns the gaps between them into critical fault lines. Under the pressure of success or failure, the fault lines are amplified. For example, vendors have to communicate with each other, often having very different processes, and contractual obligations. Trying to fix a problem, innovate, or improve performance becomes expensive because each vendor only understands their specialist area. No single vendor can see the whole story to find the most efficient solution. While all this is going on, the experience fails and the audience falls away in frustration.

The business case for separating different production areas no longer exists. Business objectives are best met with a holistic approach to design, development, and infrastructure. They all fundamentally affect the user experience, which is the single most important factor affecting visitor numbers on the Web. Combining great content with holistic technology will save money and encourage innovation. Publishers who do it early and do it right will give their readers the best possible experience, and stack the deck in their favor for years to come.

At OmniTI, we use a number of systems for trending, including Cacti and our very own Reconnoiter. Each system comes with a large number of monitors built in, allowing you to trend anything from network traffic, to system load to disk space. Sometimes however, you need metrics for which there is nothing currently available. This is where these systems’ extensibility comes into play.

The following example shows a situation where we needed information that our current monitoring/trending systems were not able to provide, and we needed to extend them with a custom trending solution:

One of our clients had a website that had become very popular, and was suffering performance issues as a result. We suspected that at least part of the system was I/O bound, and so we wanted to gather metrics on the I/O performance of the system over time. The systems in question were running Solaris 10 with the data on ZFS. The normal iostat command, for which a number of monitors exist, does not give true values for reads and writes performed by ZFS. Iostat can only see read/write requests from filesystems. True I/O statistics can be obtained on the command line by running the zpool iostat command. This works in a similar way, producing output similar to the following:

# zpool iostat rpool 10 5
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
rpool       16.6G  57.7G      1      0  43.5K  2.06K
rpool       16.6G  57.7G      0      0      0      0
rpool       16.6G  57.7G      0      0      0      0
rpool       16.6G  57.7G      0      0      0      0
rpool       16.6G  57.7G      0      0      0      0

The statement above that our monitoring systems were not able to provide the information that we needed isn’t quite true. Elsewhere, we had a monitor that obtained zpool I/O statistics using a long running zpool iostat process, taking the values out and entering them into a database, with a custom script that fetched the values from the database and entered them into cacti. The system in question was a database server, so this method, while clunky, worked well enough for its purpose. For monitoring the web servers however, using the same method just wasn’t practical and we needed something better. We needed something that didn’t require running a long running process and running a database server on the machine just for trending information.

The obvious choice here was to use SNMP. Cacti (as well as pretty much every monitoring/trending package) has built-in support for obtaining data over SNMP, and net-snmp (the snmp agent in use on the server) has various ways of extending functionality to get custom metrics.

Having chosen SNMP, the next decision was how to get the data we needed and present it over SNMP. The seemingly obvious choice would be to run zpool iostat and parse the output as was done previously, presenting those values over SNMP. However, that either requires the long running zpool iostat process, or running it once for a few seconds at a time to get a snapshot of the I/O over that period, which will lead to inaccurate results (it won’t tell us anything about the performance of the system during the time between checks). One of the things that Cacti (or rather rrdtool, which cacti makes use of) is very good at is taking raw data and generating meaningful statistics from it. If we could somehow get raw I/O values rather than already aggregated values such as 'n KB over the past m seconds read' and pass those to cacti, then cacti could do the work and we would get accurate values.

Enter open source. The source code to OpenSolaris is available, including the source code to the zpool command, which it possible to see how the zpool iostat command itself worked. Once you trace the various calls made by that function, it turns out that underneath, the zpool iostat command uses libzfs to fetch the exact raw values we are looking for. It was then a relatively simple matter to take that code and print out the raw values:

#include <stdio.h>
#include <sys/fs/zfs.h>
#include <libzfs.h>

/*
 * Sample code to demonstrate printing of raw zpool io stats.
 * Compile with: cc -lzfs -lnvpair zpoolio.c -o zpoolio
 */

int print_stats(zpool_handle_t *zhp, void *data) {
    uint_t c;
    boolean_t missing;

    nvlist_t *nv, *config;
    vdev_stat_t *vs;

    if (zpool_refresh_stats(zhp, &missing) != 0)
        return (1);

    config = zpool_get_config(zhp, NULL);

    if (nvlist_lookup_nvlist(config, 
        ZPOOL_CONFIG_VDEV_TREE, &nv) != 0) {
        return 2;
    }

    if (nvlist_lookup_uint64_array(nv, 
        ZPOOL_CONFIG_STATS, (uint64_t **)&vs, &c) != 0) {
        return 3;
    }

    printf(
        "pool:%s read_ops:%llu write_ops:%llu " \
            "read_bps:%llu write_bps:%llu\n",
        zpool_get_name(zhp),
        vs->vs_ops[ZIO_TYPE_READ],
        vs->vs_ops[ZIO_TYPE_WRITE],
        vs->vs_bytes[ZIO_TYPE_READ],
        vs->vs_bytes[ZIO_TYPE_WRITE]
    );
    return 0;
}

int main() {
    libzfs_handle_t *g_zfs;
    g_zfs = libzfs_init();
    return(zpool_iter(g_zfs, print_stats, NULL));
}

Once this was done, the next step was to get the values exported over SNMP so that cacti could view them. Net-SNMP has a pass directive that allows you to delegate an OID to an external program, and have that program print out the results it needs. These values are then exported over SNMP, available for any of the above monitoring tools to make use of.

In Cacti, it was then just a matter of creating an appropriate SNMP Data Query, adding some Graph Templates, and wait for the pretty pictures to come flowing in.

This example shows how you might approach developing code to obtain custom metrics, and shows the benefits of having the source code available so that you can learn from tools that do most, but not all of what you are trying to achieve. Sometimes, what you need just isn’t available and you just have to build a solution from the available pieces.

We are Systems Administrators. We are Web Developers. We are Database Engineers and Storage Architects and Pointy-Haired Bosses with hard copies of Schneier’s blog littering our desks. We’ve been told before that Security is our mandate, but to what end? We have heeded the vendor patch announcements and updated our servers. We comply with PCI requirements and pass the Nessus scans. We’ve studied the latest OWASP Top 10 list and applied its principles to combat the cross-site scripting attacks and SQL injections. Haven’t we? Sure we have! Or have we? Have I, as a Systems Administrator, considered the implications of the AJAX interface written by the dev team? Has the DBA considered the impact of an exploit in the chrooted webserver? Have our PHP developers been in touch with the SAN administrator to ensure that he has the capacity to withstand unlimited file uploads, and what effect that might have on our encrypted volume?

A secure infrastructure is not a zero-sum game. While applications on the Internet have become more complex and availability has increased, so have the attack vectors and their impact on the rest of the application stack. We’ve long recognized that an OSI-centric approach to security is a losing proposition. Firewalls are no longer considered a panacea for network attacks. Modern intrusions exercise multiple layers of the defense perimeter. Engineering secure applications becomes akin to a round of Jenga; how many pieces can we lose before the structure collapses? A cohesive approach to Web Application Security mandates a holistic approach across the entire engineering organization. But how many of us think beyond the edges of the envelope that defines our professional skills and aptitude?

Most of us gain our skills through a function-oriented approach. We learn repeatable steps that culminate in the desired result. Often this includes a period of trial-and-error where we assimilate aspects of the misstep and adapt to avoid the failure condition. This is only natural and is reinforced by our trainers or educators in order to attain the prescribed goals. However, these tactics can also be used to seek out the stress points within a system. When you increase your knowledge of the entire application stack, you reveal "opportunities for efficiency" by understanding the relationship of each component to the whole.

Unfortunately, the churn of modern software development fosters a "feature-first" mentality. We’re all familiar with the process. Marketing or Project Management determines the feature set that will drive purchases and upgrades for the client base. Deadlines are aligned with revenue cycles rather than product maturity. In the end, Developers feel the squeeze from Project Managers focused on their calendars and from Customers, frustrated by another release as unwitting QA test subjects. Engineers are forced into specialization, becoming a cog with a narrowed focus. Our aperture begins to close, decreasing our exposure to the application stack and impeding interoperability with other project teams and departments. Although we’ve entered the Information Age, popular software engineering practices are still rooted in the assembly line mentality.

The birth of secure software requires more than a commitment to correct code and elegant program design. A sort of Renaissance man is needed, a polymath who familiarizes himself (or herself) with the belts and pulleys of neighboring components. Someone who has a passion for the whole system. Orthogonal studies should become a desired trait, not a distraction. Database Administrators, Network Engineers, Java Developers and Systems Administrators working in harmony.

Hackers, in the acceptable sense, are a strange breed. By definition we enjoy the art of deconstruction. We want to know what makes things tick. Perhaps it’s something in our biological blueprint that drives our thirst for knowledge. We have an insatiable curiosity for understanding the misunderstood or unknown. An expensive hobby, for certain. Whether that price comes in the form of relationships, money or spare parts, it matters not. The knowledge of how that system works is compelling enough to hold our attention for hours and days and weeks and years.

Or maybe we just like breaking stuff.

After an attack, it’s only natural to wonder what motivated the attacker to focus on us. We comb through the evidence looking for the exposed stress points. But to focus on the bugs is to ignore the problem and merely serves to reinforce the broken processes. The same vulnerability will crawl out of a different hole next time. It will wait for the next hacker. And they will come. The hacker has already created new tools to make it easier next time. The hacker is generous. He likes to share his toys with his hacker friends.

Engineering teams with a foundation in "whole system" design stand a better chance of resisting and recovering from attacks. By studying different layers of the application stack they gain an understanding of the operational complexities and attack vectors. They can predict vulnerabilities in the design and planning phases. They can isolate exploits faster and pinpoint failures in unfamiliar regions. New code becomes inoculated by the changes in philosophy. Junior programmers and administrators pass these principles on to their peers. A new Renaissance begins.

Military planners always factor in casualties when deciding on a plan of action. The failure of an individual component (a soldier, a tank, an airplane) is expected, but the overall goal will still be achieved. Many enterprises do excellent risk management in their business operations but fail to apply those same principles to their IT infrastructure. The mantra of smart investing is diversification; likewise, in the insurance industry, the goal is to spread the company’s risk among a wide population, only a few of whom will actually make a claim in a given year. Yet when it comes to IT planning, all the eggs go into one large (often expensive) basket. No amount of money can ensure that a single point of failure will never fail. That money would be better spent on engineering around failures, to design systems that fail gracefully, or that at least fail only partially, limiting the damage to some subset of users. Put another way, plan for the failure of the most critical piece of infrastructure and engineer service continuity despite that failure.

I like stories. As a systems administrator for more than 10 years, I have my fair share of them, both good and bad, and the really memorable ones have valuable lessons to teach us about how to construct systems that allow most users to see little or no interruption to their service.

In my ISP (Internet Service Provider) days, the company I worked for had one physical server hosting email. It was a relatively large, expensive UNIX server, but it got the job done and had impressive reliability compared to PC hardware of the day. As the ISP business expanded, the demand for email grew beyond what the server could handle, and when the inevitable outage occurred, it affected every single mail user in the system. The solution was, of course, to get more servers. It was not cost-effective to grow with more big UNIX servers, due to a number of factors such as rack space and power, not to mention the capital investment. We needed more (and smaller) servers to store the mail, to both absorb our growing capacity of users and to reduce the impact of an individual server going down. The system we came up with decoupled mail routing from mail delivery and mailbox access. This enabled us to deploy lightweight MX (Mail Exchanger) servers that didn’t need much in the way of local storage, as all incoming mail was delivered to some other host. The MX servers were behind a load balancer, so we could scale them horizontally as required to keep up with demand. The mail storage hosts had more local storage, utilizing RAID (Redundant Array of Inexpensive Disks) to survive disk failures, and had standby hosts to which all mailbox data was replicated in case of host failure. Gluing it all together was a set of proxy hosts backed by LDAP (Lightweight Directory Access Protocol) to locate users’ mail storage host and handle mailbox access. The directory service was also used by the MX hosts for inbound delivery, to locate the appropriate storage host. Users connected to the proxies instead of directly to their mail storage host. We could do quick maintenance or handle short outages without most customers ever realizing there was a problem. For example, POP (Post Office Protocol) clients checking for new mail would be given a "no new mail" response when the backing store was unavailable. This architecture was much more resilient to failures, and in the event of a failure (or even a maintenance event), the existence of the proxy between users and the actual server allowed us to reduce the users’ exposure to the problem.

The next illustrative story comes from a client who operates a large email infrastructure supporting millions of users. Their mail storage sits on a SAN (Storage Area Network), implemented on three expensive, vertically-integrated systems from a major vendor and interconnected on a costly Fibre Channel switching fabric (which is, as ZFS author Jeff Bonwick puts it, "a network designed by disk firmware writers. God help you.") The result is a very high ratio of spindles to control units, so when there is a problem with one unit, that problem affects one-third of their customers, which could run well over several million users. That’s a lot of eggs in one basket. The price of the basket does not guarantee an absence of problems-- the redundant control heads must run the same firmware version, so a firmware bug will wipe out both of them. The cost of the storage platform is sufficiently high that scaling horizontally becomes prohibitively expensive, and doesn’t go very far to address the spindle-to-control-unit ratio. What they need is a fundamental shift in storage planning. More and cheaper baskets to hold fewer eggs each, so fewer eggs are lost when a basket fails. In this case the baskets are commodity servers and direct-attach storage running free software and exporting block devices over iSCSI (Internet Small Computer Systems Interface) to the servers handling client connections. For the cost of one of the vendor-supplied storage systems, we get nine new storage nodes, each with redundant control heads and data storage. These nine nodes provide the same amount of usable space as the three old units, and have capacity to spare. The cost savings enables more nodes to be purchased, and facilitates horizontal scaling to meet demand. Additional cost savings are realized on the interconnects, which can be standard 10Gb Ethernet. The larger number of nodes means a three-fold decrease in the number of users exposed to a node failure, and future scaling only decreases this number further.

Turning away from email, my final story covers data warehousing for a large, web-focused marketing company. Their OLTP (Online Transaction Processing) database that backs the website runs on Oracle. They need a separate place to run intensive data-mining queries and transformations that are not appropriate for the OLTP system, for which the typical solution is an Operational Data Store (ODS), a type of data warehouse. Initially this was another Oracle instance on a single server. When the size of the dataset grew beyond the capacity of the server, a decision had to be made. A server with enough memory and CPU power to handle the load would have exceeded the Oracle product license, but purchasing additional licenses was cost-prohibitive. The solution was two-fold: convert the ODS to the open source PostgreSQL server, and put it on two systems instead of one. The conversion to PostgreSQL is outside the scope of this article, but the decision to use two servers provides several distinct advantages. First, they are not set up as master/slave, which keeps the setup simple. They both replicate from Oracle in parallel, having no awareness of one another. This works fine since the data-mining queries are essentially read-only (some jobs do data transformations, but they operate on temporary tables.) Second, both systems are fast enough to handle the entire operational load, so if one system is down, all its jobs can be shifted to the other with no degradation of service to users. Third, upgrades to PostgreSQL can be tested with live data without disrupting service, as jobs can again be shifted away from the instance being upgraded. Under normal circumstances, both servers are used for production work, yielding the best return on investment.

These stories illustrate the advantages of expecting failure and engineering around it to create robust internet architectures. Failure is inevitable, but dire consequences need not be.

So after one year, what has this marriage produced? Well, they finally pushed out the 5.1 release which had been stuck in development for a couple years, although it didn’t include any major changes from what was on the road-map before the Sun purchase. They also pushed out a fork of the code base, which took the interesting step of removing features, rather than adding the enterprise technology many people were looking for, leaving Sun a bill for $1 Billion dollars but without an "Enterprise" database to call it’s own. The irony of all of this is that, even before the MySQL purchase, Sun already had a product containing technologies similar to today’s leading commercial database, it’s just that the technology lives in a file system, specifically ZFS (Zettabyte Filesystem).

One of the basic tenets of a database system is that you can guarantee that data is safe on disk, and generally that any database will give you a chance to throw away changes if you need to. In the database world, you know this as COMMIT and ROLLBACK, common operations to most people, although missing from the MYISAM technology that Sun purchased from MySQL. In the ZFS world, while not implemented the same way as in a database, these ideas are embodied in the commands zfs snapshot and zfs rollback. Both of the commands work with active data partitions, and work so well that you can use them as protection in large batch command style operations against MYISAM; simply zfs snapshot your system before hand, run your large MYISAM command, and then zfs rollback afterwards if you find you need to go back.

Of course what good is a system, database or any other, if you cannot back it up? The back-up process for MySQL is straightforward, although it’s use of LOCK TABLES makes it a second-rate solution at best. Consider with any sufficiently large system, LOCK TABLES will keep you from providing five nines uptime almost by definition. ZFS on the other hand gives you the ability to make backups with ease. Once you have a snapshot of your system, the ability to clone, promote, or send a snapshot gives you quite a bit of flexibility for backing up your system, and it can all be done on-line.

But it gets even better really. One of the things that many databases deal with is caching data files from the file system, using some algorithm to determine what should be kept in memory. In MySQL, the database only caches index files (data files themselves are left to the OS to handle), and it does so using a simple LRU (least recently used) cache; a caching mechanism where the least recently used data is purged whenever new data requests are made. Again, ZFS contains something more sophisticated. ZFS uses something known as an ARC (adaptive replacement cache), which improves upon the LRU idea by keeping track of not just how recently something is used, but also how frequently it is used. Again the nature of work being done makes for different specifics in implementation, but other database have looked at and implemented ARC systems and seen significant improvements over the LRU method.

And still there are other examples, take the ZFS intent log. The ZFS intent log is used by ZFS to gather systems calls in memory and log them, both for purposes of performance; system calls can be aggregated together before execution; and crash recovery; in the event of a crash, ZFS can examine the log and replay any system calls that did not finish execution. Of course those familiar with databases will recognize this approach, as it is commonly implemented as a transaction log within database systems, for much the same reasons; commits to the database can be aggregated for performance, and in the event of a system crash, the commit log can be replayed to ensure all committed transaction made it to disk. Unfortunately, MYISAM, the storage engine owned by Sun, does not get these benefits.

Now, we must say that MySQL has been around for some time, so it’s users have gone through the trouble of finding workarounds to the lack of functionality we’ve been seeing in ZFS. Luckily Oracle provides a storage engine for MySQL, known as InnoDB, which implements much of the features discussed above. Also MySQL has simplified replication support built in, which allows for users to set up multiple copies of the database without significant effort. In fact, these techniques are encouraged, as you can use the slave database system for taking backups or for crash recovery in case of loss of the primary node. What we think is often overlooked is that here, the database, which should be a model of data integrity and robustness, gives you workarounds and tools like CHECK and REPAIR, while the filesystem, what you typically expect your database to protect you from, is so carefully designed in ZFS to ensure data integrity, that CHECK/REPAIR are unnecessary.

Unfortunately the cynic in us has to wonder if we will ever see some of the more sophisticated ideas from ZFS make their way into MySQL. After all, since the current workarounds tend to require running multiple instances, and Sun is in the business of selling hardware (either multiple servers, or servers large enough to house multiple virtual servers, take your pick), keeping things status quo creates a nice relationship between these two divisions of the company. Given that, maybe there is no irony at after all.

• I see grand plans to make it easier and cheaper to produce foods, but no trend to actually eat less;
• companies make hybrid vehicles and the consumers flock to them without regard for the environmental manufacturing costs and even as these issues are solved and hybrids have less per-mile environmental impact people will still drive too much;
• the fast adoption of florescent light bulbs (and now LED) and yet people still leave their lights on when they don’t need them.

I tend to argue a point where I believe my point is right and the alternative is wrong. In this unique case, I find that the alternative is right, but just not "right enough." We could do better. We should do better. Think complete.

Green Computing through Hardware Optimization

So much focus is placed on making equipment (processors, ram, storage) more energy efficient that people are losing sight of the bigger picture. Energy efficient equipment is certainly one piece of the puzzle. Unfortunately, too many people see that one piece as a fait accompli in their energy conservation efforts.

At OmniTI we’re always careful about fully understanding the power profile of the hardware we install. We are conservative and look for the most power efficient machines we can find that still meet our architectural requirements (which can vary wildly from component to component). Everyone should do this. IBM and HP and Intel are all telling you that you should do it and that they can help. Do it. Let them. But please, don’t stop there.

Green Computing through Virtualization

The next step that is popular in the efforts to save your wallet (and the planet) is consolidation. This is the philosophy that one of today’s machine is powerful enough to accomplish the goals of many of yesteryear’s machines. So, virtualize! Take the old machines, turn them into virtual servers and run them on one machine today. Virtualization (of one type or another) has many advantages including: ease of management, simplistic disaster recovery, flexibility in technology selection, shorter provisioning times and the opportunity for consolidation.

Many of our engineers run VirtualBox or VMWare to quickly launch the platform of their choice. They are allocated one machine each, so they only have the opportunity to use a certain number of watts. Virtualization makes their job a bit faster and a bit easier despite the user experience being ever-so-slightly slower than running native. This use of virtualization does not reduce energy consumption in any significant way though it does increase individual productivity.

We have development environments that are managed by the operations team here that must resemble (as closely as is economically feasible) the production environment to which they deploy. We have many of these and they are all distinct, but not heavily loaded. It is feasible that consolidation could be used in this approach. Our actual situation is that we have to operate 40 isolated development environment. We do this on…

• Two $2300 1U machines
• Solaris Containers (Zones) as the lightweight virtualization technology
• at about 200W run rate, which results in about 3.5 MW-hours per year

If you considered the alternative naive implementation:

• 40 1U machines
• at about 180W run rate, which results in about 63.1 MW-hours per year.

We realize a savings of 59.6 megawatts. Wow! Now, that is an utterly naive method. Instead, let’s look at a popular method like VMWare ESX:

To run 40 VMWare instances…

• I need some substantially bigger hardware at 2GB of RAM per instance (Solaris containers and other similar technologies have some memory sharing efficiencies).
• We only have 40 instances here, so going the blade center route seems less compelling.
• An IBM x3650 should be able to manage about six instances (which aren’t peak and can afford some occasional performance degradation).
• Seven of these at 230W each we burn 14.1 MW-hours per year.
• This assumes you use local storage. If you need a SAN, you’ll have to add that into the power profile too.

One can say they burn 14 megawatts per year instead of 63! But to me, burning 3.5MW is even better. Now, for those financially responsible types, I’ve only spoken to recurring operational costs. If you run the numbers on initial capital investment you’ll see an even more significant savings by simply choosing the right tool for the job (between $80k and $100k by our internal calculations).

This isn’t to say that you should never use VMWare or a similar heavy-weight virtualization technology. Those technologies afford you specific advantages (like the ability to run entirely different operating systems in each instance). You could also consider something slightly lighter-weight like Xen. But, if you find that your virtualization requirements on Solaris will fit in the Containers model (or your Linux needs would be satisfied by OpenVZ) you stand to gain a lot. We only have 40 instances, and the choice saved us 10 megawatts over the next best virtualization solution. Imagine if you had 1000.

These concepts are not likely to be foreign to any reader. Most people have considered virtualization approaches along with hardware replacement to reduce energy costs. But please, don’t stop there.

Green Computing through Performance Optimization

When I look to virtualization technologies for consolidation, there is one requirement — a single machine has enough horsepower to power more than a single virtual instance. At OmniTI we deal with some large Internet architectures that serve millions upon millions of people. The bottom line is, I can completely saturate any piece of hardware you give me. There is no opportunity for consolidation in many of these architectures. The awful thing is that I see people choose hardware that is more energy efficient and simply leave it at that. The logical conclusion everyone has arrived at is: "if I can get the same CPU cycles and I/O operations for less watts, I win!" Yes, you win. No, this is not the conclusion of anything. It is the beginning. I hope your ultimate goal is not to spend CPU cycles, it is to service users. The obvious progression from here is: "if I can serve the same number of users with less CPU cycles and I/O operations, I win!" Now we’re getting some where. That statement starts with the end in mind. This is the land of performance optimization.

I usually try to explain concept through metaphors and analogies, but this multi-resolutioned efficiency concept was a hard one to translate. So hard, that I’m at a loss. Those who know me well, will say: "Theo without a clever analogy at hand?! That’s like Denis Leary without a vulgar rant." Alas, I’ll just give some examples.

• We increased both the functionality and the performance in core XSLT technologies for Friendster and we enabled them increase system performance by a factor of over 2.5. That translates to 60% less hardware or 2.5 times as many users. Armed with that, they chose to enter China.
• We developed a purpose-built content publishing system for National Geographic Magazine and were able to deploy an infrastructure of less than half the size (less than half the power) of the leading competitive offering. This architecture was able to sustain several prolonged front-page exposures on msn.com — delivering, at peak, as many as 3000 new visitors per second.
• We developed the Message Systems MTA that helps the largest of the large ISPs handle incoming mail volume with as much as 80% reduction in infrastructure when replacing competing commercial incumbents and as much as 95% reduction when replacing open source incumbents.

The goal is to get where you are going while spending less. Less of what? Less money, less power, less heat, less CPU cycles, less, less, less. Less of everything. Not only is it better for our planet, it’s simply cheaper. Don’t excessively or wastefully use resources. Be responsible: conserve.

Many popular sites today are popular because they link to articles and news items and photographs and movies all over the Internet; they are "interest aggregation services." And while the Internet has (for now) a decent preservation of net neutrality when it comes to simple web content, not all publishers are on equal footing. Not long ago, anyone could run a server anywhere (their basement) with DSL or cable or (gasp) dial-up — now, the challenge is coping with unexpected attention.

Years ago, the site slashdot coined a term "slashdotted" which meant that a site received so much sudden traffic that service degraded beyond an acceptable point and the site was effectively unavailable. This often happened to sites that were at the end of small pipes (DSL, T1, etc.) and occasionally (though rarely) due to bad engineering. While slashdot might have coined the term, they simply don’t have the viewership numbers that other large sites today have.

At OmniTI, I work on sites that aren’t on the end of T1 lines. Sites with gigabits or tens of gigabits of connectivity. Sites with 50 million or more users. Sites powered by thousands of machines. I also work on sites that service millions of people from just a handful of machines (efficiency certainly has its advantages sometimes). I find it particularly interesting that already popular sites (with significant baseline bandwidth) are seeing these unexpected surges. For a long time, my blog has been on this same machine which is a vhost for several other web sites. I’ve had traffic spikes from places like slashdot, reddit, digg, etc. And, no surprise, I couldn’t actually see the bandwidth jump on the graphs… 10Mbits to 11Mbs? That’s not a spike.

Things are changing. Sites like Digg are becoming ever more popular and people are drawn to them as a means of sifting the waste of the Internet. This means as more people rely on Digg and Reddit and other similar sites, the number of unexpected viewers of your content can rise more sharply.

What does all of this mean? It means that the old rule of thumb that your infrastructure should see 70% resource utilization at peak is starting to falter. The typical trends used to look like this (this is last week’s graph from a retail client with a user base of 3 million):

We see a nice peak, a nice valley. Thursday afternoon, we see a nice traffic spike. Well, this used to be what I called a traffic spike. Now, different services have different spike signatures. It resembles traffic model of classic Internet advertising, except that there is genuine interest and thus dramatically higher conversion rates. It’s a simple combination of placement, frequency and exposure. Because content, unlike ad banners, exists for an extended period of time (sometimes forever), the frequency is very high. Digg and Reddit have excellent placement with very little exposure (things move out quickly). A site like CNN or NYTimes usually provides mediocre placement (unless you are on the front page) and excellent exposure.

Lately, I see more sudden eyeballs and what used to be an established trend seems to fall into a more chaotic pattern that is the aggregate of different spike signatures around a smooth curve. This graph is from two consecutive days where we have a beautiful comparison of a relatively uneventful day followed by long-exposure spike (nytimes.com) compounded by a short-exposure spike (digg.com):

The disturbing part is that this occurs even on larger sites now due to the sheer magnitude of eyeballs looking at today’s already popular sites. Long story short, this makes planning a real bitch.

And the interesting thing is perspective on what is large… People think Digg is popular — it is. The New York Times is too, as is CNN and most other major news networks — if they link to your site, you can expect to see a dramatic and very sudden increase in traffic. And this is just in the United States (and some other English speaking countries)… there are others… and they’re kinda big.

What isn’t entirely obvious in the above graphs? These spikes happen inside 60 seconds. The idea of provisioning more servers (virtual or not) is unrealistic. Even in a cloud computing system, getting new system images up and integrated in 60 seconds is pushing the envelope and that would assume a zero second response time. This means it is about time to adjust what our systems architecture should support. The old rule of 70% utilization accommodating an unexpected 40% increase in traffic is unraveling. At least eight times in the past month, we’ve experienced from 100% to 1000% sudden increases in traffic across many of our clients.

I talk about scalability a lot. It’s my job. It’s my passion. I regularly emphasize that scalability and performance are truly different beasts. One key to scalability is that a "systems design" scales. Architectures are built to be able to scale, they are not built "at scale." It’s just too expensive to build a system to serve a billion people (until you have a billion people). It’s cheap to design a system to serve a billion people. Once you have a billion people accessing your site, you can likely justify executing on your design. Google is successful for this reason: their ideas scale and they can build into them as demand rises. On the flip side, traffic anomalies in the form of spikes are unexpected (by their definition) and scaling a system out to meet the unexpected demand is almost unreasonable. I would even argue that it is more of a performance-centric issue. I want every asset I serve to be as cheap to serve as possible allowing me to handle larger and larger spikes.

The reason I find all of this stuff interesting is that understanding performance and scalability, understanding the principles of scalable systems design and having sound and efficient processes for handling performance issues is becoming crucial for sites regardless of their size. This takes insight and practice and it reminds me of Knuth’s famous saying:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

That’s all well and good, but which 97% of the time? My response to Knuth’s statement (with which I completely agree) is:

Understanding what is and isn’t "premature" is what separates senior engineers from junior engineers.

Let’s add perspective on the word "sudden." Most network monitoring systems poll SNMP devices (like switches, load-balancers, and hosts) once every five minutes (we do this every 30 seconds in some environments). Some people say, "my site scales! bring it on." We see these spikes happen inside 60 seconds and they occasionally induce a ten-fold increase over trended peaks. Often times, this spike can be well underway for several minutes before your graphing tools even pick up on it. Then, before you have time to analyze, diagnose and remediate… poof… it’s gone. Be careful what you wish for.

This, in many ways, is like a tornado. Our ability to predict them sucks. Our responses are crude and they are quite damaging. However, predicting these Internet traffic events isn’t even possible — there are no building weather patterns or early warning signs. Instead we are forced to focus on different techniques for stability and safety. The idea of a DoS, a DDoS or the sometimes similar signature of a sudden popularity spike doesn’t increase my heart rate anymore — it’s just another day on the job. However, I thought I’d share the four guidelines that I believe are key to my sanity in these situations:

1. Be Alert: build automated systems to detect and pinpoint the cause of these issues quickly (in less than 60 seconds).
2. Be Prepared: understand the bottlenecks of your service systemically. Understanding your site inside and out. Contemplate how you would respond if a specific feature or set of features on your site were to get "suddenly popular."
3. Perform Triage: understand the importance of the various services that make up your site. If you find yourself in a position to sacrifice one part to ensure continued service of another, you should already know their relative importance and not hesitate in the decision.
4. Be Calm: any action that is not analytically driven is a waste of time and energy. Be quick, not rash.

Back to those other countries… Enter China and their recently lessened censorship and we have a looming tidal wave for smaller sites that achieve sudden popularity. Spikes of several hundred megabits per second are difficult to account for when your normal trend is around twenty megabits per second. The following graph is traffic induced from a link from a popular foreign news site (that I can’t read). I call it: "ouch:"