The incident itself was nothing exciting. An alert that monitors a critical business process had fired a few times in rapid enough succession, which raised warning flags with our SRE team. As part of our regular alert review, we performed a quick investigation to see if we could learn more about what was going on with this alert. The first step was to look at the current status of the process (everything was fine) and then to look at the error conditions, including what they were, and how long they were in an error state.

In this particular case, the alerts were not a result of the system reporting a failure, but because the check itself returned a null value. Using null alerts on specific checks helps to alert the team that status is unavailable and could mean an error condition that needs investigation. In all of the recent cases, this alert cleared itself (we eventually received a "status ok" message rather than the null) regardless of any involvement by our engineers. Furthermore, in cases when they did investigate, they saw nothing wrong. This obviously raises red flags from an alert management perspective.

It is important to note that, because we are working with complex distributed systems where you can have multiple points of potentially transient failure (ie. anything with networks + 3rd party services), we almost always buffer a null check to some degree. This means we expect to see the null condition reported multiple times before raising an alert. In this case, we checked the alert configuration and saw that it was set to alert only if the null response persisted for at least seven minutes. From the product owner's perspective, this is a pretty reasonable threshold… they are willing to go seven minutes with an unknown status for a critical business process; which could mean downtime for seven minutes before we even start reacting to it; in internet time that is pretty long. However, that wasn't the whole story.

We next looked to see how often we were doing metric collection and this is where we found a problem. It turned out the check in question was checking the status every four minutes, which meant that pretty much any time the system saw a null response, it would never issue a retry fast enough to avoid an alert. This spawned a series of new questions. Could we raise the threshold higher, to something over eight minutes, to ensure at least one retry? Or do we lower the interval for metric collection so that we ensure multiple attempts within our seven-minute window? These are good questions; people often don't think about the overhead of metric collection, and maybe we originally set this to collect at four minutes because asking the status is an expensive question; but not the kind of thing that prompts a blog post.

At one point during this investigation and the subsequent discussion, someone suggested a few times we would get the most business value by actually investigating the transient errors and fixing them. We often say that the point of technical work is to empower business, so a question on how best to serve the business is fair and deserves to be addressed. It's true that transient failures like this can be problematic, and might be a sign of an underlying problem. However it is important to understand that an alert review is not about creating more project work, it is about managing your operational response. When we say to focus your monitoring and alerting around driving business value, that doesn't mean prioritizing business operations over the business operators. We have talked to many a tech worker who tells us this is the kind of situation that their company could easily turn into an "emergency", where the SRE team might get sent down a rabbit hole that they hadn't planned on that day, disrupting who knows what other initiatives. And worse, expecting the team to just suffer through any additional false positives until the situation is fixed. Well, that isn't us; we believe that engineers have business value too.

In this case, we know that from a business perspective:

• 1. The business process is not failing
• 2. The transient null metrics are resolving themselves with no intervention
• 3. The frequency of nulls may be on the rise, but we haven't quantified how bad it is yet
• 4. Waking people up in the middle of the night for something that isn't causing any harm to the business causes harm to our people, which causes harm to the business.

Given the above, we decided the best way to manage the alert was to shorten the interval for metric collection to two minutes (meaning we need three null reports to trigger an alert). We were also able to look at the service history and see that while these alerts have lowered service availability in May, we were still within the boundaries of our service budgets, and overall the system is performing on par with historical trends. Luckily this was pretty easy to show to the folks with business hats since Circonus can auto-generate service uptime reports. (See below) Beyond that, we want to quantify the scope of the null metrics a bit more before deciding when and with what effort to go about troubleshooting and/or repairing that situation. All in all a ten-minute conversation that made for a good reminder that managing alerts is never a one and done type project, but requires regular review on the technical side, and reinforcement on the cultural side.

Figure 1: Circonus Uptime Report

1. Metrics Are A Must

Every website should have at least some basic measurements in place around site traffic and response times. You may also want to try wiring up measurements on HTTP response codes (think 404 or 500's per minute) to track the general health of your website. Having additional metrics on system resources can be a lifesaver if you need to correlate performance issues across different systems. This can also help you to find bottlenecks that you might not otherwise run into except during heavy load.

Think your monitoring setup is already up to speed? Now is the time to work on creating some holiday specific dashboards based around critical and/or heavily used graphs; after all, no one wants to go digging through last year's pinned links in a slack channel to see what's going on. Once you have a few dashboards set up, spend some time walking through them with folks one step removed from your team, so that come crunch time people know where to look to get the information easily.

Concerned you don't have time to set up a bunch of monitoring infrastructure? Lots of monitoring as a service companies have near turn-key solutions that can get you up and running quickly. Many offer free trials for a limited time, or better yet, use something like Circonus which offers free accounts with a cap on the available metrics you can track. This gives you plenty of room to get started, allows you to keep your data for analyzing after the holidays, and won't require a bunch of extra hardware on your end.

2. Sprinkle On The Caching

Ah caching; often touted as a panacea for performance issues, this can often cause as much trouble as it solves. One of the big problems with adding caching into a web application is that it can change the logical workflow for how customers see and interact with data. This can create cases where you have to figure out ways to do cache invalidation which aren't always obvious. That said, there are a couple of cases where you can add caching without having to make significant modifications to your internal app.

The first idea is to simply cache your initial landing page and perhaps nothing more. Depending on your traffic patterns, often time the primary landing page is the item that creates a bottleneck on your site, as people load up this page trying to determine where they need to click next. Even caching this page with a small timeout (1 minute) can make a huge difference in the number of requests and ultimately the amount of resource pressure on your back-end infrastructure. Worried that your landing page includes dynamic content and customizations? Perhaps consider creating a new, static landing page, focused on the most highly sought after information from new or occasional users who are now visiting the site and driving up traffic. Adding a splash page like this in front of your normal page isn't something you want to do in the long term, but for a few days during the holidays, it can buy you a lot of breathing room.

Another idea that can also reap good rewards is setting up a reverse proxy in front of your normal web setup. Depending on your setup, you can use this technique to offload specific items from the application server. Look for work that doesn't require specific state information from your application. Serving static assets like images, CSS, JavaScript files, or even statically rendered pages, is an obvious first choice to serve these items with long-lived cache headers, once again reducing the number of requests your systems have to take.

In both of these cases, you can generally set up a minimal solution using tools like Traffic Server or Nginx, or look to outside CDN services like Fastly or Cloudfront for an initial setup which won't require running your own infrastructure to get started.

3. Have A Scaling Plan

Another idea you should implement is to go through a scaling exercise with your operations team, so you can walk through each component of your system, and understand which parts can be scaled, and what is necessary to scale them. There is nothing worse than thinking you can quickly spin up new web servers only to realize that someone added a new component that is disk bound and now cannot be horizontally scaled without engineering effort to set up data replication between servers. So what you thought would take minutes now takes an hour just to ship the data between servers. I've heard enough people say "I can't believe we did something like that" to know that it's worth at least talking through the idea, and even better to run through it at least once to prove it works.

At a minimum, you want to come up with a run list for scaling options so that decisions are made in advance rather than have to be figured out under pressure. Sure, maybe you can reboot your database server onto a more powerful VM size, but how long does that take, and, more importantly, can you afford the downtime for that? Maybe you should spin up a more powerful secondary ahead of time, and then failover if that becomes necessary. Again, for each component, consider whether you can add more hardware (virtual or physical, scaling the system vertically) or add copies (scaling it horizontally) and decide which scenarios require which reaction. This can also help you understand where your limits are, so you don't spend your time on efforts that cannot bear any fruit.

4. Ain't No Party Like A 3rd Party Service

Everyone loves 3rd party services; they add functionality while making many parts of the service somebody else's problem. Until it becomes your problem. You might be surprised by the number of 3rd party services that don't do adequate load testing, but you shouldn't be; many of the companies running these services are just like yours; under pressure to add new features, with limited resources and budget (aren't we all?). Even Amazon has had cascading failures due to different components going down; trust me it can happen to anyone.

What this means is you need to be aware of how your site will behave when various 3rd party services begin to fail. Better still is if you can come up with ways to mitigate them. Unfortunately, in many cases, this can requires code changes. But before you start down the path of re-implementing a given service, instead try to determine if you can introduce a method for simply turning the service off. Doing this right means isolating the 3rd party code and putting conditionals so that you can easily flip a flag or add a few variables and have the service temporarily turned off. You don't want to be rewriting code like this under pressure, but being able to push out simple config changes can be a lifesaver.

5. The Holidays Are About People

One last suggestion that is always worth remembering, is that it is quite possible that your website is not the only thing that might be adding to the stress of your team this holiday season. Extra busy holiday schedules including concerts, relatives visiting, shopping lists to deal with and who knows what else. Stress levels are often high this time of year, so remember to look out for each other. If your people need flex time or time away during non-emergencies, make sure to advocate for that as much as you can. This also makes the next few weeks a good time to review everyone's ability to work remotely and make sure folks understand where / when communication around issues will take place. Sure it sucks to have to work on a holiday, but it's far worse if you have to do it from the office.

And remember, if you do find yourself dealing with production issues, keep an eye on how much time people spend firefighting and try to rotate people in and out when you can. When trying to solve complex failures, you need fresh minds when you can get them. While people need to take breaks, in a high-stress environment adrenaline kicks in and people may not realize when they have been at it too long. In most cases, service problems seen during the holidays can be recurrent if you aren't prepared (high traffic from Black Friday events will eventually go down, but can often re-manifest over the weekend or again on Cyber Monday), so it's up to you to make sure the team is ready for that.

In Conclusion

There is never enough time to do All The Things™, but if you stay focused, there is still time to make headway. Despite all the hype, it is possible to be successful without rewriting everything into a cloud-based microservices DevOps infrastructure™. But time is running out; you need to stay focused and make sure that what you are prioritizing and working on is intentional, focusing on any immediate concerns and doing as much to prepare as you can in the next couple of weeks. It is those ounces of prevention now that save you from the pounding a cure will require later.

If you have questions or looking for help preparing for (or recovering from) the holidays -- we’re here to help. Let us know if we can assist you this busy season.

The table in question resides in a Postgres database that is about 4TB which restore testing have shown takes up to 48 hours to restore. As a workaround, we have continuous archiving of WAL enabled to a remote server with online backups taken with pg_start/stop_backup and daily ZFS snapshots alongside our normal PITR setup.

Putting Everything Together

So, with a customer crises at hand, and no one looking forward to a 48 hour restore process, I identified the tools at my disposal and mapped out the plan to restore:

1. 1. Restore data directory and xlog directory backup using ZFS clone (which automatically creates the mount-point). The clone command was used here because a copy of the previous night’s backup existed on the server.
   • ·zfs clone data/set/pgdata@lastfull_20170523001500
   • ·data/set/pgdata_lastfull_20170523001500_restore
   • ·zfs clone rpool/xlog@lastfull_20170523001500 rpool/xlog_lastfull_20170523001500_restore


2. 2. Restore the necessary archived wal logs to the local server in a separate directory.


3. 3. Create the recovery.conf file like so :
   • ·standby_mode = 'on'
   • ·restore_command = '/data/path_to_wal/%f "%p"'
   • ·recovery_target_action = pause
   • ·recovery_target_time = '2017-05-23 11:00:00'
   • ·recovery_target_inclusive = false
   Note: The client had given me the time they had run the truncate command, so I set my recovery time a few minutes before that, and set the recovery_target_action to "pause", which will pause replay at that time and allow me to query the system to see if I have the desired data available.


4. 4. Edit postgresql.conf - changing port and archive details like so:
   • ·port = 5434
   • ·shared_buffers = 64MB
   • ·archive_mode = off
   • ·max_wal_senders = 3
   • ·#archive_command = '/omnipitr-archive "%p"'
   Note: When doing an emergency restore like this, make sure you comment out the archive command.


5. 5. Changed pg_xlog symlink to point to the restored xlog directory - this was crucial as I was restoring the backup on the prod server and did not want to cause a data corruption issue. Note: ZFS snapshots are file system snapshots, so when restoring a zfs backup the snapshot will be identical to the original file system that was backed up.
   • ·rm -rf /wal/symlink/in/restored/backup/cause_it_is/pointing/to/prod
   • ·ln -s /point/to/restored/wal/backup/directory

Sometimes it’s never that simple

With my plan in place, I was ready to save the day. I restored the database and checked the table, but it was empty! What happened? I had gone past the truncate ..but how could that be? The client said it was truncated a few minutes after my recovery_target_time. So where is my data? Hmm.

Logging is your friend.

I started to review the pg_logs to identify when the truncate occurred. To my surprise, the client was not 100% accurate in the timing of when the truncate was issued, the logs told the true story. Well, thanks! Now I have to start over.

Tools of the Trade.

But wait, I am using ZFS snapshots; so I am not going to be here all night -- re-creating a clone takes but a few seconds. Once the correct timing of the truncate was identified, I was able to destroy the clone and recreate it, pause recovery and voilà - there was my table with the data I needed. I pg_dumped it out and restored it into the prod database with a new name, and just like that the client was pleased that he was able to create the report sooner than expected.

Conclusion

Of course, we could have solved this without ZFS, but I shudder to think about two day restore times, and can you imagine if after two days you found out that the recovery time you were given was incorrect, and then having to wait another two days to try again? Instead I was able to solve this problem in just a few hours; sure still a solid time investment, but much better than most of the alternatives. You can also imagine how happy the client was to find out they weren't going to have to wait days to receive their data. While I know a lot of companies don't run ZFS, but I am thankful we have the option to use quality tools to help save everyone both time and stress.

Businesses have needs

A few years ago, the company decided to offer their customers an option to upgrade their subscription plans to a different tier. If a user was interested in the upgrade they would click a button to be entered into an upgrade queue. Before each evening’s billing, the list was checked for any subscriptions that had reached their billing cycle date (i.e., are about to be billed). The subscription's plan would then be updated and removed from the upgrade queue. The upgraded subscription would then be billed at the new plan amount. The development history was more complicated, but suffice it to say, changes took place and bugs were found and fixed. This cycle repeated until the system seemed to work well from both business and technical perspectives.

Time affects all things, even unchanging code

Fast forward three years. The company received a complaint from a user who had signed up for the upgraded plan three months prior and was still not in the program. Development was notified and asked to investigate the account. It looked fine. The database was checked, and the subscription was, in fact, flagged to be upgraded. The Oracle process was investigated and the selection query run by hand to see if the subscription would be upgraded. Everything indicated that it should; however, the user had been billed 5 days earlier without being upgraded. Checking the upgrade queue table in detail provided some evidence. There were around 3800 accounts waiting to be upgraded. Past trends indicated this was far too many subscriptions to be in the queue at one time. Something was definitely wrong.

When designing solutions, one must always expect that problems will happen down the line. All of the Oracle database jobs keep logs of when they start, each step they get through, and the final state of the job; so if something ever fails, it is easy to quickly identify the what, where, and how. The summary logs were reviewed.

# -- JOB_ID -- JOB_NAME -- TIMESTAMP -- COMPLETION_STATUS -- COMPLETION_TIMESTAMP

1 -- 60468396  -- PROCESS_UPGRADE -- 20XX-12-25 20:28:39 -- OK -- 20XX-12-25 20:29:08

2 -- 60493968  -- PROCESS_UPGRADE -- 20XX-12-26 20:28:42 -- OK -- 20XX-12-26 20:29:11

3 -- 60519486  -- PROCESS_UPGRADE -- 20XX-12-27 20:28:46 -- OK -- 20XX-12-27 20:29:39

4 -- 60545237  -- PROCESS_UPGRADE -- 20XX-12-28 20:28:50 -- OK -- 20XX-12-28 20:29:15

5 -- 60570885  -- PROCESS_UPGRADE -- 20XX-12-29 20:28:51 -- OK -- 20XX-12-29 20:29:54

6 -- 60596687  -- PROCESS_UPGRADE -- 20XX-12-30 20:28:54 -- OK -- 20XX-12-30 20:29:34

Everything had completed fine -- no errors. The ‘dba_jobs’ table was queried to verify the Oracle schedule for the job.

# -- INTERVAL -- WHAT

1 -- "sysdate + 1", "process_upgrades;"

Everything looked good at first. However, after looking through the previous log it became clear, "sysdate + 1". Looking back at the start times in the logs, they had been increasing by a few seconds each run. How could this be?

"sysdate + 1" is "1 day from now" in Oracle speak. When a job is scheduled to happen and when it actually starts (calculating the next run schedule) are not the same. So when Oracle calculates the next start time it would sometimes be a few seconds later especially if the database was under heavy load. Each day there was a chance for this to creep forward a little bit. The job to process the evening billing was scheduled to run at 19:45, but the job’s start time had drifted to almost 20:30.

The list processing is designed to take place "before each evening billing." When the upgrade runs, it checks to see who is to be billed that evening. Time does not typically travel backward; therefore, when the upgrade process runs after the billing process, naturally, it will not find anyone who needs to billed because... they have already been billed! Thus the accounts would never be upgraded, and never leave the queue.

Bugfixing only fixes the immediate problem

This is a simple fix. If the desire is to run at 19:30 the next day, the interval is set to trunc(sysdate + 1) + 19.5/24. This will always set the next time to 19:30 "tomorrow" regardless of what time of the day it is run. We then searched to make sure we didn’t have this same problem anywhere else. We did. Thankfully most of these were sysdate + 1/24 (an hour), so they did not have any significant business impact from the others. These were fixed anyway to avoid leaving broken windows.

Now for the damage. The process moves forward, on average, 3 seconds every run. For every minute, that equals 20 days. For an hour, that is (60*20=1200 days / 365) roughly 3.25 years. That's close enough to match the originally implemented start time. The billing process starts at 19:45, but does a lot of processing beforehand. By checking against billing logs, it is estimated that the subscriptions are actually checked at about 20:15. So, from the time the job actually starts until the time the subscriptions are checked (20:30 - 20:15) equals 15 mins at 3 seconds a day... approximately 300 days. This has been broken around 10 months.

Looking at the bottom line: the upgrade costs about $5/month. Assuming a 100% success rate and an average retention rate of 3 months will give us a good upper bound. So 3800 subscriptions * $5 * 3 months, means potentially a $57,000 loss of revenue. Luckily, some of the lost revenue can be recovered by processing the accounts still in the queue. A quick check shows 300 of the requested upgrades are still active accounts so (300*5*3) $4500 can be recovered; however, it is still a potential $52,500 loss due to a bug in the code from three years ago.

Consider your options, don’t knee-jerk

This begs the obvious question of what could have been done (or could still be done) to prevent this. A different engineering approach is probably the first thing that comes to mind.

Perhaps instead of using Oracle's job scheduler, the process could have been scheduled as a cronjob. Cronjobs are much more familiar to developers and probably an easier method of scheduling to understand. This type of time shift error would not have occurred using this method. Of course, that would mean this particular database job's scheduling would no longer be in the database or under the DBAs’ direct control. Given none of the other jobs of this nature are scheduled with cron (they all use the Oracle scheduler), this process would become a special snowflake, which opens the application to other risks.

Another option would be to modify the two processes to be made dependent on each other and have the billing process fail or delay until the upgrade job has successfully run. This ties the business logic together in a way which guarantees they either both work, or neither do. This sounds like an obvious win but there are severe negatives to consider as well. If any new problem occurs with the upgrade process the billing process will no longer run. If this delays into the next day it could have serious cash flow implications or even cause unrecoverable loss of revenue/subscriptions. Even though billing is monitored this still increases risk. A partially complete billing is better than no billing for this particular client’s needs.

Still another alternative might be to modify the billing process itself to be updated to handle the queue processing as well. This approach puts all of the logic into one location, which is nice, however the billing system is already quite complex and difficult to manipulate without breaking it. Modifying billing to incorporate this new feature would add even more complexity. In general, keeping processes segmented and less complex decreases the chances of other subtle bugs creeping in, so we wanted to keep these two pieces separate. If the upgrade queue processor needs to be changed in the future, having to fiddle with the core billing system to do it is never a great situation.

Monitoring is insurance against future mistakes

In the end, we decided against these additional changes; we already had a working solution and the goal was to make the software more reliable. It wasn't feasible to spend resources to recreate the wheel without a compelling advantage. What we did do... implement better monitoring. More specifically, business logic monitoring. The previous solution worked well before we encountered the timing error. Since the error was now fixed, the solution would continue to work well. No matter how well you engineer a solution, nothing in a changing environment is immune to an edge case error, or changing assumptions/dependencies. In a constantly changing environment, bugs cannot be totally avoided, but one can plan for increased visibility when they do occur.

The business rule was to change the plan for the account on its next cycle, and remove it from the list of subscriptions to be upgraded. The cycles were 30 days long so a monitor could be added to verify nothing in the list is ever more than 30 days old. If so, then the monitor would alert for investigation. That's it. That simple check, maybe 15 minutes of someone's time to validate the execution of the business rule, would have been worth more than $50,000. If there is a problem, the liability is now limited; and if an error does occur the business can attempt to recover with a higher likelihood of success due to faster response times.

Additional monitors can (and should) be added to check other aspects of this business process. Are items being entered into the list daily? Are the number of those newly listed accounts outside of a standard deviation for the normal trend? These simple additions, together, can catch all sorts of unimagined engineering failures down the road for a small investment of time. Monitoring is an incremental process. Each time a failure occurs the underlying business goal should be considered and an appropriate monitor put in place to validate it is being met. Code only exists to cater to the goals of the business. Do not monitor the code, monitor the rules and goals of the business.

The Usual Solution

When two systems need to talk to each other asynchronously, the typical pattern looks like this:

1. An action is performed in System A that requires a subsequent action in system B.
2. In the code on System A, after the action is performed, insert a row into a queue with all the information needed for the second system.
3. System B periodically polls the queue looking for new entries and processes them.

This works reasonably well but given that the two systems are disconnected from each other, there will be a delay in processing of messages, based on how often System B checks for work to do. Of course, there are issues with this type of system, such as the overhead in constantly checking the queue for unprocessed entries, especially in cases where checking for new records will likely yield zero results. That’s a lot of round-trip queries for nothing. LISTEN/NOTIFY provides an alternative solution where processes can be told what to do, rather than polling for something to do.

What Sorcery Is This?

Despite being available from at least as far back as the original Postgres95 release, LISTEN/NOTIFY is a somewhat unknown notification system that comes bundled inside PostgreSQL. With LISTEN/NOTIFY, multiple listeners can watch a given channel for messages and receive them instantly when another process (or even the same process) issues a notification. For those familiar with RabbitMQ or similar distributed messaging systems, this is analogous to a fanout exchange.

In this project our messaging needs are fairly simple; we would like to have actions that occur within the project tracking system kick off actions in the external cloud accounting system. Passing basic messages around like this is a perfect job for LISTEN/NOTIFY because we can solve the job of message passing without the operational overhead of setting up an additional queuing system.

For basic use a process can register itself to receive messages on a channel by issuing a simple SQL command. Our accounting system listener might do something like this:

LISTEN accounts;

Since PostgreSQL creates channels for us on first use, we don't need to worry about pre-declaring a channel that we want to listen on. Once the process is listening on a channel, any process (even the same one) can send a message on the channel to communicate with the listener. The listener will continue to receive notifications on this channel until stopped with the UNLISTEN command.

Sending a message is also quite simple:

NOTIFY accounts, ‘2 hours posted on Contract ID 1234’;

Anything listening to the ‘accounts’ channel will then immediately receive the following, without needing to check or poll:

Asynchronous notification "accounts" with payload 
"2 hours posted on Contract ID 1234" received from
server process with PID 219.

Using the NOTIFY statement in this way requires a basic string as the payload; we’ll dive a bit more into ways to generate more descriptive payloads later.

A Sample Implementation

Basic usage requires something to issue the notification and something to listen for it. In most cases you will want to set up a long-running listener to wait for activity on a database connection. Many languages provide this sort of construct; in this particular implementation we chose Perl. The basic blueprint looks like this:

use DBI;
use IO::Select;

# Connect to the database
my $dbh = DBI::connect(...);
# Listen to the channel
$dbh->do(‘LISTEN accounts’);
my $fd = $dbh->func(‘getfd’);
my $sel = IO::Select->new($fd);
while (1) {
    $sel->can_read();
    while (my $notify = $dbh->func("pg_notifies")) {
        my $pid = $notify->[1];
        my $payload = $notify->[2];
        print “I got ‘$payload’ from pid $pid\n”;
        # ... and whatever else needs to be 
        done with $payload
    }
}

(original source: How to use LISTEN/NOTIFY in a Perl program)

At this point we have a long-running listener able to receive messages instantly and on demand, however there is a wrinkle. Like most message queuing systems PostgreSQL doesn’t maintain any message history or other way for late-comers to catch up. In our case we want the system to be a bit more robust than that so the system should be able to handle a script crash or other issue that could otherwise result in the loss of a message. For some applications this may not matter but in our case it does - we need to be able to process all messages including any that are sent while the script is offline.

Our Old Friend, The Queue Table

While not the primary way of communicating messages, backing our on-demand message processor with a queue table offers a number of benefits:

• Messages can be retained indefinitely allowing for statistics to be collected
• Stale entries can be detected and processed
• Progress in processing a given message can be reflected in its corresponding queue table entry providing metrics and a way to alert if things fail

That last one is important for monitoring as it provides the ability to catch problems in the message processing system. LISTEN/NOTIFY is fire-and-forget; once the message has been consumed, it’s no longer available for audit/troubleshooting. Queued message processing is handled by a one-time check of the queue table at script startup, catching any messages that might have been issued out of band, and making sure that we are up to date.

It is also beneficial to setup external monitoring of the queue table to catch any anomalies in processing. Messages that stay in a processing state for too long or that are never picked up could indicate a script failure. Retaining message history allows for alerts to be fired to notify someone to investigate.

Automating Messages

Now that we have basic message passing set up and a way to capture missed messages, the next thing we need to focus on is how to trigger messages from the application to the listeners. One option is to issue the NOTIFY command via SQL statements at the appropriate point(s) in the code, but there are a number of problems that arise with that method, such as overhead in preparing and executing the statement, needing to remember all the places in the code to add these new calls, and error mitigation if the statement fails, among other issues.

To avoid all of these possible issues, in our implementation we opted to make use of triggers. Instead of identifying code points that should spawn messages we instead identify the business data that should spawn a message when changed. This takes the application code completely out of the equation and makes it a data-driven solution. This also keeps the application code smaller and the implementation is resistant to future programmers forgetting to code notification points. As a nice plus, messages will be sent if someone modifies data directly in the database, not that anyone would ever do that.

Trigger Function Magic

The approach is twofold; first a trigger on the data table is responsible for forming the message that is inserted into the queue table, then a trigger on the queue table issues the actual NOTIFY to our listening process. Once the listening process gets the NOTIFY, it fetches the queue entry and processes it. There is no repeated polling for jobs and the listening process receives the message almost immediately after the data is changed for the cost of a single query.

A simple implementation for generating an account notice based on changed data might look something like this:

CREATE OR REPLACE FUNCTION modify_account RETURNS trigger AS
$$
BEGIN
  IF (TG_OP = 'INSERT') THEN
    INSERT INTO queue_table (payload, status) 
    VALUES (‘2 hours posted on Contract ID 1234’, ‘NEW’);
    RETURN NEW;
  -- similar conditionals for UPDATE and DELETE
  END IF;
  RETURN NULL;
END;
$$

While this gives us some useful functionality, it is still fairly basic. In many cases our messages will likely need more information, such as potentially containing a parent row ID, perhaps a new/old value combination, or other key/value pairs. Given that the payload sent via NOTIFY must be a flat string, a stringified object format seems sensible. We used JSON to solve this problem, allowing us to transmit as much information as necessary in the payload. Combining these ideas results in the new trigger function below, which runs after any changing operation on affected columns:

CREATE OR REPLACE FUNCTION modify_account RETURNS trigger AS
$$
BEGIN
  IF (TG_OP = 'INSERT') THEN
    INSERT INTO queue_table (payload, status) VALUES (
        (SELECT row_to_json(row) FROM 
(SELECT TG_TABLE_NAME AS table, TG_OP AS operation, 
NEW.id AS id, NOW() AS trigger_timestamp) row), ‘NEW’);
    RETURN NEW;
  -- similar conditionals for UPDATE and DELETE
  END IF;
  RETURN NULL;
END;
$$

Depending on what data is selected for the INSERT, the resulting JSON string might look something like this:

{"table":"accounts","operation":"UPDATE",
"id":6,"trigger_timestamp":"2015-09-14 10:33:45.03627-05"}

So our queue table entry now knows about the triggering row, what operation was performed, the time it was triggered, and we tag the row as NEW to indicate it’s available for processing. PostgreSQL's handy built-in row_to_json() function is used to convert a result row into the JSON string. Any other details about the account being modified are in the business data.

Propagating the message in the queue table out to any listeners could not be simpler. A two line trigger function runs after inserts on the queue table:

CREATE OR REPLACE FUNCTION modify_account RETURNS trigger AS
$$
BEGIN
  PERFORM pg_notify('accounts', ((SELECT row_to_json(row) 
  FROM (SELECT NEW.queue_id AS id, NEW.payload AS payload) 
  row))::text);
  RETURN NEW;
  END;
$$

This grafts the queue table ID onto the existing payload, constructing a deeper JSON object and uses pg_notify() to do the NOTIFY work. Recall that anything other than a flat string won’t work with a straight NOTIFY, so we use a PostgreSQL function instead. Our Perl listener now receives the message on the channel almost instantaneously after the original change commits to the database by way of these trigger functions and is free to act on that message in any way it wishes.

Security Concerns

Compared to a full featured queuing system, LISTEN/NOTIFY is still somewhat primitive. For example, PostgreSQL doesn't provide any additional authentication process to listen on a channel beyond normal database connectivity, so messages on a given channel are available to all database users that know the channel name. If you plan to pass messages that contain sensitive information, the channel name will need to be kept secure and not easily guessed.

One approach you could use would be to generate a random channel name at application startup time and distribute that name to any listening processes. The maximum length of a channel name is 64 characters, so something like an MD5 hash (with proper entropy) makes a suitable choice. Of course, this wouldn't be particularly hard to brute force, so you might want to add additional measures like encrypting the message itself.

Conclusion

By utilizing PostgreSQL’s built-in LISTEN/NOTIFY capabilities, we constructed a system where changes in business data could be communicated to an external process in near real time, while retaining some nice quality-of-life features such as message persistence and audit trails. As other data points were identified in the database, it was simple to hook them up to the notification queue and add corresponding functionality to the listener to handle them. All of this required zero changes to existing application code, and is completely invisible to any end user. Any application systems already using PostgreSQL have this feature built-in, so it is worth a look as a solution to simple messaging needs, especially if those messages can be triggered by changes in business data.

Today, I want to talk about authentication in node.js apps hosted on heroku and demonstrate a way to maintain horizontal scalability with authentication implemented.

Most modern web applications require some sort of authentication, and, luckily, the Passport middleware for node.js does exactly that. Its flexibility not only allows us to implement our own authentication and serialization logic, it supports several authentication strategies out-of-the-box, including SSO with OAuth and OpenID.

More often than not, authentication requires sessions. When developing applications with expressjs as the platform, the default option of storing sessions in memory usually won’t cut it if we want to use more than a single application server - a particular session needs to be accessible from any application dyno. One way to solve this is by using a load balancer that provides “session stickiness” - a mechanism that ensures that the same backend server handles all requests for a particular session - however, this can cause all sorts of problems, and, besides, heroku’s load balancers do not provide such functionality yet. This nudges us to work in a different direction, architecting some sort of centralized storage for our sessions.

There are several ways to store sessions centrally, each with their own pros and cons. I personally prefer something lightweight, non-persistent - ideally, a key-value store with TTL (time-to-live), conveniently accessible from a node.js app. Something like Redis, which we will use as an example in this how-to.

First problem is setting up Redis on heroku. Luckily, redistogo.com offers Redis storage as a heroku add-on.

So, let’s summarize the plan to implement authentication in our app:

1. Set up passport
2. Set up Redis for session storage
3. Profit

Let’s start with the first item on the list - setting up passport to handle our authentication needs.

$ npm install passport

Passport plugs into expressjs just as any other middleware, so invoking the actual authentication is quite simple:

app.post(‘/login’, passport.authenticate(‘local’), function(request, response) {
     // authentication is successful - redirect somewhere else. 
request.user contains the user information:
     response.redirect(‘/some/other/route’);
});

In the example above, ‘local’ is our authentication strategy - we will talk about it shortly. Passport authentication middleware can be instructed to redirect in cases of success or failure, for example:

passport.authenticate(‘local’, {
     successRedirect: ‘/‘,
     failureRedirect: ‘/login'
})

There are even more examples of the passport middleware functionality in the documentation.

Now, our authentication strategy needs to be configured before it can be used, so lets take care of that:

passport.use(new AuthStrategy(function(username, password, done) {
     // this is an example, finding user in the database
     Users.findOne(username, password, function(error, user) {
          // error finding the user - authentication failed
          if(error) return done(error);
          // user not found - authentication failed
          if(!user) return done(null, false);
          // additional checks for authentication are done here
          // otherwise, authentication successful
          return done(null, user);
     });
});

We also need to provide user serialization/deserialization routines. In the simplest scenario, we let things handle themselves:

passport.serializeUser(function(user, done) {
     done(null, user);
});
passport.deserializeUser(function(id, done) {
     done(null, id);
});

In this example, the entire user object is serialized into the session store. Depending on the application logic, you may decide to only store a session id and look up the user by its id somewhere else. The two functions above will be the place to implement any (de)serialization logic we need.

Last thing - initialize passport and let it know about sessions (which, at this point, aren’t set up yet):

app.use(passport.initialize());
app.use(passport.session());

With our simple authentication logic all set up, there is only one missing piece in this puzzle - the actual session store. Moving on to part 2 of our plan - setting up Redis as our session store and using it in our app.

First, let’s add redistogo to our app:

$ heroku addons:add redistogo

That’s it. Your redis instance is ready. Installing the add-on automagically adds a REDISTOGO_URL environment variable to your app, and you will see it if you run:

$ heroku config

And you should see a line similar to this, among other configuration values:

REDISTOGO_URL:
redis://redistogo:7f87e90468f8b1d026ae5495b7735232@hoki.redistogo.com:9271/

Now, moving on to setting up the redis client in your code. I use the node_redis module for the Redis client, and the connect-redis module, which wraps the Redis client as a middleware usable with expressjs. Then, in the application code:

For the purpose of this tutorial and in sake of simplicity, let’s assume you have everything in meaningfully named constants. Let’s set up the session store - it will internally set up the Redis client.

// set the options here
var redisOptions = {
     url: REDIS_URL,
     ttl: SESSION_TTL
     /* There are more options available here, 
          look ‘em up in node_redis documentation */
 };

// create the Redis store, it will set up the Redis client internally
var redisStore = new RedisStore(redisOptions);

// and now, define express sessions with the store we just set up, 
    and some express options
session = express.session( {
     store: redisStore,
     secret: SESSION_SECRET,
     unset: 'destroy',
     proxy: true
});

// and use it in application
app.use(session);

The section above has to run before we initialize passport.

Don’t forget the last remaining pieces: handle logout, user profile, and anything else your app needs. The passport documentation is excellent: http://passportjs.org/guide/.

That’s it - your Redis-based session store is ready, we can now scale our dynos:

$ heroku ps:scale web=2

Now, with this solution, there is the Redis redundancy issue to keep in mind. While the free redistogo.com plan only offers a single database with no extra features, the paid plans do offer sharded setups with redundancy, backups and all the bells and whistles. Resolving the SPOF issue is is a matter of upgrading your plan.

As you see, all this can be done with minimal code. I hope this tutorial helps someone, and happy coding!

Like the rest of our work, we take our beer seriously, and in order to serve our after-hours drinking needs, the kitchen is equipped with a redundant kegerator setup, which provides load balancing and fault tolerance. Initially, in order to show what was on tap, we had a 7" Android tablet showing a web interface in full-screen mode.

The original display.

This did its job well enough, but like anything else that's mission-critical, it needed proper metrics and monitoring. With a couple flow meters, we could measure how much beer is being dispensed from each of the taps, and use that to estimate how much is left in the keg. This would help us to know when we need to buy more beer, but more importantly, it would look awesome. Also, I wanted an excuse to play with electronics.

Initial planning

The first step was to decide on the major parts of the system. I would need a controller board in order to hook everything up and run the code that reads input from the sensors. The Arduino IDE is a popular open-source platform for writing embedded code, and there are many controller boards that can run Arduino code. I ended up choosing the Teensy 2.0 due to its size and low price. While it isn't an official Arduino board, it is capable of running standard Arduino functions using the Teensyduino add-on.

Additionally, several other programs would be necessary: a listener program to communicate with the controller board, a display program to run the monitor, and a web API to store this information in a database.

Software components

Software

Arduino Code

The Arduino code is the part of the program that reads input from the sensors and sends the data to a PC over a USB connection. I used Adafruit's Kegomatic code as a reference for writing this program.

When liquid is flowing through the flow meter, power to its pulse output wire switches on and off. Blowing air through the flow meter also triggers pulses, which allowed for testing without actually installing the meters. According to the specifications, each pulse indicates 2.25 mL of liquid passing through the meter. The Arduino code simply counts these on/off pulses, and then returns that count when asked.

The board is connected to a PC via a USB port, and waits to receive any input over the USB serial interface. Once it receives any input, it builds and returns a packet, and resets its pulse counters. The packet it sends starts with an 8-byte header containing the size of the packet, the number of taps, and the number of temperature sensors. The header is followed by an 8-byte integer for each of the values.

Since the USB module just sends the pulse count, rather than converting to ounces or liters, we can calibrate the flowmeters without having to reprogram the Teensy.

Listener Program

The listener program is a Python program that runs on a PC and communicates with the controller board. The Teensy is connected to the PC via USB, and the listener program uses pySerial to find and communicate with the Arduino program.

The listener program sends updates to the display program when it detects output from a flow meter, and updates the web API after output stops. It also periodically sends readings from the temperature sensors to the web API.

Display Program

The display program is the part of the code that shows which beers are on tap, how much is remaining of each, and when each one is being poured. I wanted the display to update immediately when the taps were active, so a web interface would not be ideal here. Instead, the display is a desktop application written using Tkinter, the most popular GUI programming toolkit for Python.

The new display is also on a much larger monitor than the old Android display. With this additional space, I was able to add more information to the display, such as the beer style and description. Since the program is using the Untappd API for beer information, it was also simple to fetch any Untappd check-ins at OmniTI headquarters and display those on screen.

Web API

The beer list is tracked by a web application, which runs on an OmniOS server, and stores data using SQLite. The web application exposes the current beer list and keg status in JSON format. This endpoint is accessed by the display program to show which beer is on tap, and is also used by an IRC bot that sends out a message to an internal IRC channel when a keg is changed.

The web server also provides an admin interface, which uses OAuth 2.0 and Google's API to authenticate users, and allows the list of beers on tap to be updated.

Users can also pull up the mobile version of the website, which displays beer names and descriptions, and links directly to the beers on Untappd.

A screenshot of the mobile site.

Source Code

I've uploaded this code to GitHub and made it available for use under the MIT License. This was the first Arduino code I had written that's more complex than "Hello World", and it's definitely reinventing the wheel somewhat, but it was a good learning experience.

Hardware

Raspberry Pi

We needed a computer to hook up the flow meters and display. I decided that this would be perfect for a Raspberry Pi, especially since we already had a spare one in a box in the supply closet. I installed the Raspbian Linux distribution, and wrote a script to install all the prerequisites for the display and listener programs.

I also needed a new display. The old beer list was displayed on a 7" tablet, which was a bit too small, and only supported a resolution of 800x480. I was initially looking for something in the 10-12" range. A Tontec 10" LCD would have cost about $50 less, but would have required some assembly, and I was worried that it would still be too small.

I ended up ordering an Asus 21.5" monitor and I'm quite happy with it. It's bigger than we really need, but the larger size made it a lot easier to read, and gave us plenty of real-estate to play with without looking unreasonably huge.

The new screen, prior to hooking up the flowmeters.

First prototype with single meter

After the initial planning, I wanted to create a small proof of concept, before investing into the full kit. I ordered the Teensy 2.0 controller board, a single flowmeter, and a temperature sensor, and hooked them up using a breadboard to write the first draft of the code.

Components for initial prototype

"Practice Run"

After getting the controller program to work with the smaller prototype, I ordered the rest of the flow meters and some wire, and started putting things together. For this design, the Teensy sat directly on top of a Perma-Proto Breadboard. I connected each of the Teensy's inputs to the breadboard by soldering a pin between the Teensy and one of the rows on the breadboard. Then I soldered four 3-pin plug receptacles to the breadboard. These receptacles fit the plug that comes with the flow meter, and allowed me to connect and disconnect the flow meters without having to redo any wiring.

All of the connections worked, so I considered this to be a massive success, especially since I had never soldered anything before. However, there were a few problems with the design:

• I didn't measure beforehand, and the wires between the flow meters and the breadboard weren't long enough to reach from the kegs to a single breadboard.
• I hadn't considered where I was actually going to put the temperature sensors. With this design, I couldn't solder them directly to the breadboard. Adding more of the 3-pin plug receptacles might have been an option for this.
• We have two separate kegerators, and would have had to run nine wires into each of them - three for each of the flow meters, and three for the temperature sensor.
• The power and ground rails and the pre-connected rows of holes were convenient, but forced me to bunch all my wires very close together.

At this point, I thought it was better to scrap it and start over with a better plan, using what I had learned for the next iteration.

Final Design and Assembly

For the final product, I decided to put a small prototype board in each of the kegerators, and connect each of those to a main board with the Teensy. Each of the small boards have a temperature sensor, two ports for connecting flowmeters, and a RJ-45 jack. Ethernet cables were perfect for connecting these to the main board - they have 8 wires each, and we have plenty of them sitting around already. I ran power and ground on the first wire pair, and one signal on each of the other three pairs.

I tried to find out if there was a "correct" way to hook up components on a prototype board, and all the answers were essentially "whatever works is correct". Some instructions recommended using solder bridges to connect power and ground, but this seemed far too difficult. I found it was easier to strip two wires and run them across the length of the board, then connect power and ground to those. Each of the signal wires connected directly from the RJ-45 port.

Basic circuit diagram for the main board (left) and the secondary board (right)

The main board.

One of the secondary boards.

Instead of doing all the wiring by hand, another option would have been to get circuit boards printed through a service like OSH Park or ExpressPCB. This would have cost about $30 more. I still would have had to solder the components to these boards, but I would not have had to do all the manual wiring between these connections. If I ever do anything like this again, I will most likely use this option instead of using prototype boards and manual wiring.

All the pieces ready to be installed. 1: Flow Meters, 2: Secondary Boards, 3: Ethernet Cables, 4: Main Board

Connecting to Hoses

Making the actual connections between the hoses and the flow meters ended up being more of a problem than I had initially anticipated. I needed to cut the existing beer hoses and attach connectors in order to hook up the flow meters, so for four taps, I needed eight connectors. Each flow meter has a 1/2" threaded connector on each end, and our hoses had an internal diameter of 3/16". The only adapters I could find to fit into a 3/16" hose would have required multiple connectors and cost at least $15 each. I ordered a pack of 1/4" hose barbs, hoping they would fit into the hose. Putting the end of the hose in hot water helped to stretch it a bit, and with enough force, I was able to get everything into place.

1/4" hose barb in a 3/16" hose.

Hose barbs installed, hose clamps on, flowmeter connected and tightened.

Flow meter installed in beer line, secondary board taped to side of kegerator

On my first try, the connectors were leaking. After using more teflon tape, tightening them more, and mopping beer off the floor, everything seemed okay.

First working pour! (Old version of interface with formatting issues)

According to the specifications, each pulse is 2.25 mL. Since my code counts both the on and off pulses, it should actually be 1.125 mL per pulse. After converting to sensible and proper units, this is 0.9502 fluid scruples, or 0.0380 fluid ounces.

Because I know you were wondering.

Using this value, the program said we were pouring upwards of 30 ounces into a 12 ounce glass. While this would be nice, it seemed unlikely. I grabbed whatever we had nearby with unit markings and did some basic testing.

We're doing science here.

With the blender, I came up with a rough estimate of 0.02 ounces per pulse, but I wanted to be a bit more exact than that. I ordered a graduated cylinder, put on a snazzy lab coat, grabbed a clipboard, and did some more in-depth testing. The graduated cylinder showed different values for each tap, between 0.0127 and 0.0199 ounces per pulse. I made a small modification to the program to allow it to use a different value for each tap, and updated the configuration file with these new values.

Later, I asked two of our top scientists to peer review my data. Initial results suggest that beer is tasty. Further research on this hypothesis is already underway.

Beer Metrics

The webserver's database keeps track of the amount we have left right now, but I also wanted to make sure we could trend our long-term consumption. To make this happen, the webserver also exposes a JSON endpoint with status information about the kegs. Circonus can access this data and create some useful graphs about our precious liquids.

This graph shows the output from the temperature sensors. When everything is working correctly, the temperature goes up and down as the compressor switches off and on.

But if I don't close the door all the way and don't notice until the following morning, it gets a bit warmer. In the spirit of learning from my own mistakes, I've set up an email alert when the temperature stays too high for more than 30 minutes.

Another useful graph shows an estimate of the amount remaining in each keg. The orange line jumps back up to full in the middle, showing when the keg was replaced. Not all of the kegs were empty when the flow meters were installed, so we didn't have an accurate measurement of the amounts remaining in kegs 3 and 4 (the blue and green lines) until both were replaced on December 13th.

Using these metrics, we could also set up an alert if beer is poured before the end of business hours. This could send out a simple email alert, or (preferably) set off an elaborate shame siren in order to properly admonish the offending party.

This is a stacked graph showing total beer consumption. Each individual tap is shown in a different color, and the combined height of the bars shows the total flow of all four taps combined. The minimum resolution is set to one day, which means that the bars show the total for each day. This gets rid of the large blank spaces that would otherwise show up in the non-drinking hours of every day.

Final Thoughts

This certainly isn't the most useful tool ever built, but I had fun building it and I learned a lot about things I don't normally have a chance to work with. It looks great and I'm incredibly happy with the results, but my beer robot can never be as good as this one.

Transactions are a wonderful tool for web/application developers. If several things in the database need to be modified together, simply wrap them in a transaction. This will guarantee that either all of the changes succeed together, or they all fail together. This helps to remove complicated error conditions of partially successful updates and avoid being stuck in a situation that may no longer possible for the application to clean up.

The following is an example of a simple web transaction in Perl.*

# Turn off AutoCommit so we're in a transaction
local $dbh->{AutoCommit} = 0;
eval {
   my $sth = $dbh->prepare('insert into users (name, email) 
            values (?,?) returning userid');
   $sth->execute( $name, $email );
   $sth->fetch( \$userid ); 
   $sth = $dbh->prepare('insert into addresses (...) 
            values (?,?,?,?,?) returning addressid');
   $sth->execute( $userid, $addr, $city, $state, $zip );
   $sth->fetch( \$addressid ); 
   $sth = $dbh->prepare('insert into user_history (...) 
            values (?,?,?) returning historyid');
   $sth->execute( $userid, $addressid, 'User Created’ );
   $sth->fetch( \$userid ); 
};
if($@) {
    $dbh->rollback();
} else {
    $dbh->commit();
}
# Return AutoCommit back on
$dbh->{AutoCommit} = 1;

The user’s data is inserted into all three tables that this business requires. Without the transaction, it would be possible for the first two statements to execute successfully but have the third fail leaving the user in an incomplete state! Thankfully a transaction provides a guarantee that all statements will succeed or get rolled back to how they were initially.

* Careful readers may have noticed this example isn’t handling errors properly.

Transactions are Hard

While providing great functionality, as transactions become more complex, mistakes are increasingly likely to be made. Error handling is a difficult topic and it is easy for developers to make simple mistakes that can have unpleasant side effects. Modifying the earlier example, a number of common errors that find their way into transactional code, can be illustrated.

Breaking “autocommit” state

In Perl, the DBI module regulates transactions with the “AutoCommit” flag. Normally AutoCommit is set to true, and each query is implicitly wrapped in a “BEGIN; ; COMMIT;” sandwich. To create multi-statement transactions, the AutoCommit flag is turned off. This tells DBI not to send an implicit COMMIT. The application is then responsible for issuing a commit or rollback statement. In the earlier example "local" was used to limit the variable change to the current block scope. Once the block completes, the variable is automatically returned to its previous value. But imagine instead that local had not been used and simply changed that as:

# Turn off AutoCommit so we're in a transaction
$dbh->{AutoCommit} = 0;

If anything unexpected happened, that value would never get reset. This changes that database handle for all future database queries that use it and can cause very long transactions, or unexpectedly large volumes of SQL queries to get rolled back when an error does happen. It’s always important to maintain the state of the AutoCommit flag, and return it back to its previous state.

Including outside operations in a transaction

Especially in larger operations, a developer may not even realize they are in a transaction and put other calls into the execution path. These calls are not part of the transaction, and will not get rolled back in the case of an error. Consider the following change:

eval {
   my $sth = $dbh->prepare('insert into users (name, email) 
            values (?,?) returning userid');
   $sth->execute( $name, $email );
   $sth->fetch( \$userid ); 
   $PARTNER->API->acknowledge_signup( $userid );
   $sth = $dbh->prepare('insert into addresses (...) 
            values (?,?,?,?,?) returning addressid');
   $sth->execute( $userid, $addr, $city, $state, $zip );
   $sth->fetch( \$addressid ); 
};

If an error occurs during the second execute() statement, the first insert will be rolled back. However the Partner has already been notified of the acknowledgement via its API. This API call will not get rolled back causing the Partner to have an acknowledgement of something that never occurred. These acknowledgements should not occur until after the transaction has been successfully committed.

Slowing down other transactions

Updating a value in an MVCC database such as PostgreSQL causes a lock to be taken. Other clients may also want to update that value or table, but until the first transaction is committed or rolled back, the database cannot handle their request. These other clients will have to wait until that lock is released. It is not uncommon to see a thoughtless and expensive operation put in the middle of a transaction block causing other clients to wait needlessly. The API call in the previous example could have a network issue causing the transaction to block until its timeout is reached.

Error handling can have errors

It’s quite difficult to anticipate all of the ways errors can occur. Consider this piece of code:

eval {
   my $sth = $dbh->prepare('insert into users (name, email) 
            values (?,?) returning userid');
   $sth->execute( $name, $email );
   $sth->fetch( \$userid ); 
   $sth = $dbh->prepare('insert into addresses (...) 
            values (?,?,?,?,?) returning addressid');
   $sth->execute( $userid, $addr, $city, $state, $zip );
   $sth->fetch( \$addressid );
};
if($@) {
    $dbh->rollback();
}
$response->send_success();
$dbh->commit();

Here the code does not take into consideration that successful completion of the eval does not mean that the transaction has completed. The commit command itself can return an error if the database connection was broken. In this case it is possible to have sent a success message, but the transaction would still get rolled back by the database.

While only a few are reviewed here, there are many ways that transactions can be done incorrectly. While there are solutions to all of these problems, it would be great for developers if there was a way to avoid most of the issues in the first place by not using transactions, yet still atomically doing the changes.

Enter Writeable CTEs

Common Table Expressions (CTEs) can be thought of like temporary tables that exist only for the life of that query. These are created by using the WITH keyword and are supported by most major databases. These can be quite powerful on their own and I’ve written about the recursive versions of these before. A simple example of a CTE would be:

with naughty_users as (
  select * from users where banned = 1
)
select * from naughty_users;

Writeable CTEs were added to PostgreSQL in version 9.1. These took CTEs and super charged them by allowing the modification of data inside of these expressions. For example, inserting, updating, and deleting. This allows many powerful expressions to be chained together in a manner not possible before. For example being able to delete a set of rows, while being able to insert a subset of those rows into a log table.

with deletions as (
  delete from users where expired is true
  returning *
)
insert into deleted_even_user_archive
select * from deletions where userid % 2 = 0;

It is an important side note that CTEs are “optimization fences” to the query planner. The query planner will not attempt to re-arrange queries across table expression for faster execution plans. For more advanced users, this can be used to force the query planner to use certain indexes. However for the novice, it means the query planner cannot try to optimize a poorly written CTE whereas it may have been able to if it was written in a more traditional manner.

Replacing Insert Transactions with CTEs

This example consists of three separate insert queries:

insert into users (name, email) 
values (?,?) returning userid

insert into addresses (userid, address, city, state, zip) 
values (?,?,?,?,?) returning addressid

insert into user_history (userid, addressid, action) 
values (?,?,?) returning historyid

These three queries could instead be expressed as a single CTE query that returns the ids of all three inserted rows.

with userdata as (
  insert into users (name, email) values (?,?)
  returning userid
), addressdata as (
  insert into addresses (userid, address, city, state, zip)
  select userid,?,?,?,?
  from userdata
  returning addressid 
), historydata as (
  insert into user_history (userid, addressid, action)
  select userid, addressid,?
  from userdata, addressdata 
  returning historyid
)
select userid, addressid, historyid 
from userdata, addressdata, historydata;

Doing this as a single query avoids the transaction issue completely. The entire query is executed and succeeds or fails atomically. If part of the query throws an error the entire query is rolled back, like can be expected of any other query. Wrapping these inserts into the single CTE query allows:

• avoiding complicated transaction code
• avoiding complicated error handling code
• reduced query overhead
• minimization of locks duration (because the transaction runs faster, see below)

Performance Impact

In theory, this should provide a significant amount of back-and-forth savings on the network by running all three queries at once instead of serially. But nothing beats real world metrics. To test this, a free Postgres instance on Amazon RDS was created with the following settings:

Free Tier Postgres RDS instance of Postgres 9.3.3
Multi-AZ Deployment : No
db.t2.micro - 1 vCPU, 1GB RAM
General Purpose SSD storage, 20 GB

A free t1.micro instance running OmniOS r151012 AMI was created to test with. To reproduce these results instructions and scripts are available at https://github.com/bdunavant/cte_demo

The ‘demo_web_inserts.pl’ script runs three different scenarios 100,000 times to get the average time spent for each set of inserts. The results were pretty close to expected.

Traditional without transaction took 758 seconds to run.
758 / 100000 = 7.58ms per set.
Traditional with transaction took 737 seconds to run.
737 / 100000 = 7.37ms per set.
CTE took 289 seconds to run.
289 / 100000 = 2.89ms per set.

It is interesting to note that the transaction version of the 3 inserts actually runs slightly faster than the non-transactional version. This has to do with Postgres not having to write the WAL logs to disk until the commit is issued. Less disk access means faster run times. The CTE version of the query gets to take advantage of this same fact since all three inserts are in the same commit.

Overall the CTE version of the inserts give an approximately 60% decrease in run time; which is to be expected if 1/3rd of the number of queries are being run.

To further explore the level of savings, a simple user insert is run for multiple iterations; 2 inserts vs CTE, 3 inserts vs CTE, and so on up til 10 inserts, to verify the savings scale upwards. This test is in demo_variable_inserts.pl. Each set is run with 100,000 iterations on the same setup as the previous test. Towards the end the test cases take significant time to complete (hours per iteration set). This is simply total time / iterations to determine an average (including outliers). EC2’s variable performance throws off the data but the trends appear pretty clearly.

Caveats

There’s no such thing as a free lunch. So there are reasons to not suddenly go in and update all of a project’s inserts statements to wrap them in CTEs.

• CTEs are often unsupported in ORMs requiring developers to step outside of the framework and supply the SQL themselves. This can break encapsulation in some cases or cause other side effects. Make sure the code doesn’t break by moving all of these queries together.
• CTE queries take some learning to write correctly, especially regarding this different method of returning the ids. Developers must fully understand how the SQL statements work. This is a tradeoff between less complex app code, for slightly more complex SQL statements.
• Postgres executes sub-statements and the main query concurrently making insert order unpredictable. This could have important implications to an application. In the tests, the insert order is forced by referencing the previous sub-statements.
• While this technique can technically also be done with update statement, updates have their own challenges (such as deadlocks) which are out of scope for this article.

Conclusion

Common Table Expressions are a great way to simplify some common database cases helping to reduce mistakes in transaction handling by avoiding transactions entirely. In addition they can improve performance out of a series of queries by merging them into a single query.

As you can imagine, seeing doctor after doctor was very difficult for us, and during that time, I came to a horrifying realization: doctors are just people. Just like everyone else, the level of performance between any two of them can vary wildly. Of course, there can be many reasons for poor performance, but in my experience the more common ones are:

• I don’t know: although I am tasked with helping you, I don’t have the knowledge necessary to do so.
• I don’t care: although I am tasked with helping you, it is not a priority for me. I want to get you through the process as soon as possible so I can get back to what is important to me.
• I can’t: although I am tasked with helping you, my hands are tied by external forces: time constraints, policies, costs, etc.

Not Knowing

Years ago, a team I managed had an engineer, who was in charge of a sizable Ruby on Rails application, leave and I was left without anyone with that certain set of skills. I went to one of our team members and said “Hey - I know you don’t know much about Ruby or Rails, but please support this app as best you can.” So going into it, I knew, and he knew that he didn’t know. And, when you know you don’t know, that’s a better place to be than when you don’t know you don’t know.

Seriously.

Because we knew we didn’t know, when acute or chronic problems presented themselves I was able to tell the project manager, “We are doing the best we can with an engineer who has little experience with this technology.” This isn't the type of thing you want to say, but at least it allows you to set everyones expectations correctly.

As we were working through our son's illness, I think some of the doctors we took Alex to see genuinely did not know why my son was stuck in this cycle of illness, but that didn't stop them from prescribing something that gave us an expectation that *this* would be the solution we were looking for.

Not Caring

Fast forward a few years, and another engineer on my team was trying to solve a problem with one of our web applications. He had been restarting the app routinely, trying to deal with some intermittent issues. Then came the automated script that ran once-a-day to restart the app. Then once-an-hour. I finally set aside some time and had a look at the problem myself.

As it turned out, the restart command being issued wasn’t even restarting the application. I felt compelled to ask, “Did you even check to see if the application was being restarted?” As an engineer, if I don’t verify that my fix is at the very least doing what I intended, can it really be said that I care? As a doctor, and really in any profession, I have to say that I think the same rules apply. While I can’t say for certain that any of the doctors we saw didn’t care enough to give my son the very best treatment, I did get that feeling more than once.

My Hands Are Tied

I see this all the time. We are not finished with the product, but it has to be working in production like YESTERDAY, PEOPLE! We produce code for businesses. That code makes them money, or saves them money, or keeps track of their money. In any case: money. If we have to tack on a piece of code with some duct tape and a few nails we had laying around at the last minute, sometimes we do it. We hope to find the right balance, because we are constantly trying to do just that. But, as Leonardo da Vinci is credited with saying, “Art is never finished, only abandoned.”

The point is, as software engineers, sometimes we have to decide between implementing the quick solution or the right solution. There are a finite amount of resources over time that can be applied to solve a problem; that’s just reality. However, there is a balance to be sought. And, when you are the one being affected by limitations, being aware of them doesn’t do much to ease the pain.

Of course the same thing applies to medical care. If you consider the state of medicine in the U.S. over the last several decades, whatever your views are, I think we can agree there is an imbalance which leads to insufficient resources to adequately care for people. I remember a specific conversation I had with one of Alex’s doctors at the HMO; I left with the feeling that rules and costs were inhibiting her from doing more in-depth treatment that could have helped my son.

No News Is BAD NEWS

Of course, even if you have people with the right knowledge, who care about the problem, and have the resources to work on it, communication is still CRUCIAL. I have seen emotional and procedural escalations occur because of a lack of communication more than any other single aspect of a customer relationship. As a manager, one thing I counseled on repeatedly is that you must answer communications in a timely manner. I know it is more fulfilling to send an email saying, “I completed this task, and it’s in production and working great!” rather than some variation on why it’s not done yet, but you need to keep the customer informed.

No communication leaves the customer not knowing anything about the state of their issue [1]. Is it fixed or finished? Is my issue even being worked on? Did the engineer assigned to it quit and move to Djibouti under an assumed name to escape my wrath? I can honestly think of very few instances where communication of bad news caused a freakout. I can remember many instances where no communication did.

Taking my son through the medical process, we rarely received communication in detail, especially in response to our questions and concerns. We were left in the dark far too often, which was not just unfortunate, but it left us feeling totally alone.

You Are NOT on Your Own

One thing I was taught at a customer service job many years ago was to never let the customer feel like they are on their own. The most basic example being, “I’m going to transfer you to department N. If you get cut off or they are not able to help you, here is a number where you can reach me directly. Please call me back and I will continue to help.” This is comforting to the customer, and they will remember your company in a positive light. I have tried to incorporate this strategy into every aspect of my dealings with customers, both internal and external. “You are not alone,” is one of the best feelings you can give another human being.

The Worst News

Once, an internal customer (a group from within our same company) sent us this email:

“It’s been three months now since the promised delivery date for this product has passed and we still don’t have it. We need to know now if you can provide this to us or not.” Although we had kept them up-to-date with frequent communications and regular meetings, our lateness had finally exceeded their patience.

Unfortunately, this particular software project turned out to be much more involved than we had initially thought. After talking through the issues with our lead developer, I sent this reply:

“We are very sorry, but after encountering many unforeseen complications in the existing data structure, we no longer believe we will be able to deliver this product.”

Wow. That was not a fun email to send. Was there blowback? Surprisingly, none at all. In fact, I don’t think we ever heard from them again. It left a sick feeling in the pit of my stomach for a while, because we wanted to “do good” but this time, we failed. And yet, I think there is a strength, help, and hope in this reply. “We can’t help you,” stated in no uncertain terms, frees you to try another path. It also frees us to stop sweating something we are losing ground on every day. I never heard “I can’t help you” from any of the doctors we saw, but can you see where that might have helped us?

Trying Different Things

It’s been said that the definition of insanity is doing the same thing over and over, and expecting different results. If that’s true, I have seen a lot of insanity.

Earlier I had mentioned a case where we had an issue that we tried to solve through restarting a web application. As it happened, the way this problem manifested was that the application would just stop accepting connections, and the customer's browser would just sit and wait. Our support engineer would restart the application manually, and in the interest of fostering good communication, would inform the customer that the problem was fixed. This was fine the first few times, but after:
“I restarted it.”
“I restarted it.”
“I restarted it.”
. . .understandably, they started to become upset. When they asked the support engineer, he would say, “Well, I can’t tell what’s wrong with it after I restart it.” This was a bit of a catch–22. The customer needed it working again right away, but they wanted the chronic problem fixed too.

I finally said: “The next time the app stops responding, don’t restart it. We need to examine it while it’s broken.” And I communicated this intention to the customer, both beforehand and during the following outage.

Do you want to know what the root cause was? So do I. We never figured it out. But because we took the time to fully analyze the broken app, we could set up some solid monitoring that could detect when the problem occurred and instantly restart the application. Once that was in place, we found that the application rarely restarted, and when it did, it was within a minute of the app failing to respond. Outages became insignificant to the customer, and they were satisfied.

Substitute antibiotics for “I restarted it” and you have our medical dilemma exactly. You “fixed” it, Doctor, but have you really fixed it? No. You kicked the can down the road, and we’ll have to deal with the problem again soon.

Dr. Black

In our quest to find a doctor who could fix our son, my wife and I finally found Dr. Black. When we talked to him the first time about Alex’s recurring illness, he listened. I mean he really listened. And I think we left with antibiotics, because Alex still needed them, but we also left with something new: Flonase (for nasal congestion, where the infection would always start.)

And guess what? That didn’t fix his problem. He was still sick a few weeks later, BUT. . .

A Cut Above the Rest

What made this doctor exceptional was that every single time we saw him, his approach was, “That didn’t work? Well, let’s try something else.” It was such a relief to have someone working on finding a real solution. He may not have known what the problem was, but he let us know that he was as determined to find a solution as we were, and that he would work with us until we did. And a few tries later, he found it. It was Rhinocort, a different medicine in the same family as the first medicine we tried. He pointed out that some people have success with one and not the other.

Since this was preventative medicine, we could give it to my son whenever he started to get congested and not have to give him a course of antibiotics every month. Eventually we were able to stop altogether and he has been very healthy for more than a decade now.

I will never forget The Amazing Dr. Black, and I will always try to emulate his conscientious work ethic.

1.   OK, sometimes they can tell it’s still not working, but most customers will be unsatisfied with this amount of information.

A few months ago I attended Node Summit in San Francisco, dedicated to success stories around node.js implementations in large architectures. There were a couple of very educational war stories (particularly, the so-called, "boring" Walmart Black Friday case study, presented by Eran Hammer), but, this rant is not going to be about how awesome node.js is. This rant is going to be about bad decisions. Those that are still being made in the world of technology, driven by false premises, wrong reasons and buzzword bingo.

With increased availability of new technological choices, people tend to make the mistake of jumping on "hot" technology without a good reason. Reading industry blogs and aggregators, engineers and managers alike hear about "the next best thing" and jump on whatever that is without considering the consequences (or need), justifying the move with a lot of, well, bullshit. For example, I've seen one of the largest media companies in the world make a decision to replace all existing web platforms, serving hundreds of millions of people, with Django. A week after release of version 1.0. Without a single engineer familiar with Django. Or with Python. After recently investing a significant amount into their current architecture that works like a well-oiled machine. The “official” reason? The success behind the launch of The Washington Post's new, also Django based, platform. Nevermind that that was the only production (relative) success at scale at the time. Nevermind that The Washington Post hired one of the creators of Django for that initiative, to ensure it succeeded. Nevermind all that — one success story was enough to pitch the raw idea to, and get approval from, management. There is a saying: If a man jumps off a 10 story building and survives without a scratch, what do you call it? An accident. What if he dusts off, climbs back up and jumps again with the same result? A coincidence. But what if he does it for the third time? A habit. So make your decisions based on repeatable success, not on a one-hit wonder.

Another saying that always comes to mind when seeing poor technology decisions is what we’ve all been told by our parents when we were kids: “If everyone jumped off the bridge, would you just follow?” I had a conversation over a few drinks with the founder of a relatively successful startup, who was looking to add profile management to his suite of online applications. We went pretty deep into an architecture design discussion, talking about the best way to leverage Single Sign On (SSO) and tie the profile into multitude of offered services. Eventually the conversation turned to the technology stack of choice. He told me that node.js was a strong candidate (for all the right reasons), but to my surprise, he also thought MongoDB was a right fit. Given his application suite was already using a RDBMS, and the nature of data that has to be stored is highly relational, why would anyone introduce a new, non-relational storage component to the list of existing tools in this situation? His answer: "I heard that node.js works well with MongoDB." Now, to be fair, it is a better response than "I've read on that blog from that guy who said that MongoDB is webscale,” but not far from it. I’m not hating on MongoDB (in this particular case). In fact, at OmniTI we run some of the largest MongoDB clusters in the world; but when it comes to our high volume node.js deployments, supporting 100,000 requests/sec, those have been backed by Riak and PostgreSQL. Why? Because those technologies provided a better solution for the particular problems at hand. Always select each of the architecture components to fit the role they are going to play, not just go with the newest or shiniest solution. Or the one that people claim is the best. Which brings me to my next point.

People (yes, even technical ones) often make their technology choices based upon marketing materials that usually have very little to do with the reality. There are infamous MongoDB performance benchmarks, supporting the company’s claim that it is the fastest database, that were done without actually writing the data. Yeah. Database that doesn’t guarantee data storage. That makes sense. Once you turn on guaranteed writes, performance plummets. Yet, people make assumptions solely based on those product marketing numbers, negatively impacting whole organizations—from IT to finance. Consider this: A company with an architecture of a couple of hundred servers needed file system encryption for all the application and data storage nodes to comply with security regulations. They selected a "promising" product, installed it, configured it based upon the product company’s recommendations, and noticed about 25% performance drop, impacting core business flows. Not surprisingly, the company found this unacceptable. In conversations, they mentioned that they expected 4% performance decline. Strangely precise number. As it turned out, said company never had filesystem encryption running on any instances of the application, not even development (not that it would be sufficient or comparable), and the performance benchmark expectations came straight from the marketing pamphlet that the product company provided. This harsh realization came after the company invested money in this product, made promises to their own investors and set a budget for hardware—all based on a theoretical number in a brochure. Pro tip: as a general rule, the only performance numbers that matter are the production performance numbers. "It worked fast on my laptop" is not a valid argument for. . .well, anything. Neither are benchmark numbers written in a marketing brochure. Or on a bathroom wall, for that matter.

What comes next is a justification of technology/product selection, which brings me back to Node Summit, which in turn, inspired this rant. A very positive takeaway was that large companies are adopting node.js and contributing back, which brings validity to the technology itself and promises good things to follow. But there were also presenters and panelists giving all the wrong reasons for node.js selection and adoption, which cheapens the achievements, as impressive as they may be. You really cannot take seriously any testimonial on the awesomeness of a decision to switch to node.js if it is based on the fact that “now, it takes much less than 18 months to change background color for all the web properties.” Similarly, if you claim that your application is faster in node.js without CDN than it was in its previous incarnation with CDN, I cannot take you seriously. In case it's not obvious: the problem is not with a poor technology selection to begin with, the problem is with the decision-making process and poor architectural decisions. You can't blame technology for your poor decisions. Similarly, you can't praise a technology because you didn’t repeat the same mistakes the second time around. All it means is that you've learned something from your own mistakes. Or hired smarter people.

The "shininess factor" and pseudo math both do damage, and in the tech industry, you need to keep up with everything. But, with the number of different technologies that hit the market every month (if not day), each backed by media hype and promoted by investors, it is easy to fall into a trap of buying into a promise of a product instead of the product itself. A word of advice: don't base your decision on success stories but rather on the stories of failures. Those are told by people who had to deal with similar choices, experienced the pains firsthand, had to overcome them, and can provide the only factoid that matters—production experience and performance. So do your research, challenge assumptions, and remember the wise words of Bruce Lee: mistakes are always forgivable, if one has the courage to admit them.

Recently, I had an interesting problem to solve myself that I turned into a “big data” problem for curiosity reason. For the problem at hand, I had to analyze two or three days of PostgreSQL logs to make decisions about different issues, like finding out from where clients connect, what they do and how long sessions last.

To start working, I logged onto the server and noticed that each day’s worth of logs was 500Mb, so I had to analyze about 1.5GB of logs. With the help of awk I did the job, but as a geek, I wanted to take the problem to the next level: What would I do if I had to analyze logs from last month, and what would happen if I had more than one server? How could I scale the solution? A month would have 45Gb worth of logs, and from 10 servers it would be something like 450Gb: more than 5TB of logs over the course of one year. That’s a lot of data to analyze with sed and awk, it would take ages just to answer a simple question, and I would have to store that data somewhere. It was now obvious that I was venturing into big data territory.

Big Data

At Surge 2013, I heard the term, “Big Data” a number of times, and got the best description of it. Someone said that there is no real definition of big data and that it’s defined by the individual: it can be anything from images to documents (or as in my case, it can be log files), but no matter what the data might look like it all shares a common attribute. When the data-set is so large and complex, that it becomes difficult to process using a single database tool or a traditional data processing application. Quite recently, I implemented PostgreSQL backups into Hadoop with the use of OmniPITR, so I thought it a good opportunity to use that knowledge to solve my current problem. I got chance to work with Hadoop earlier, so I decided to solve my problem using HDFS and it’s map-reduce functionality to achieve parallelism and get results in reasonable time.

Why Hadoop?

Being an engineer, I divided the bigger problem of processing a large volume of data into three, smaller tasks.

I had to come up with a solution to answer following questions:

1. Where should I store the data?
2. How can I get answers in a reasonable time frame?
3. Would the solution be able to scale in proportion to data-set growth in future?

Hadoop answers all of these questions. If my problem wasn’t theoretical, I would simply go to my colleagues and ask them to install Hadoop, sparing me as much disk space as possible. So there’s the solution to questions 1 and 3 (and let’s not forget: that’s with no additional software costs for my company!) I would create a big Hadoop cluster with my colleagues’ corporate PCs to run map/reduce functions. This solution can be scaled up with zero - little cost, and I could run almost all of the queries I wanted on a big data-set. Now I had a solution, but I had to find a use case to make sure that this would work as I imagined.

The Case!

Again, with scalability in mind, I wanted to answer the following question. I have three fictional data centers across the globe: one in Europe, another in the United States, and another in Asia, and I want to know which one serves whom and how my connections distribute. A relatively simple question on a difficult data-set, but if I was able to answer that, then no matter how many servers were used or how big the logs were, I would be able to answer anything.

Putting Everything Together

With a real use-case and a possible solution in mind, I had to make a checklist of all the different things I would need, like test data, to make a meaningful comparison between one and two Hadoop node runtimes. I would need machines--two Debian VMs running Hadoop seemed like a good starting point--and last, but not least, I would need a MapReduce function. I wrote a log generator to create 1GB worth of data, I setup the VMs and decided to use Hadoop streaming so I could avoid writing code in Java. I chose to use Python and coded two functions that would return something like:

- 

The aggregation would show me all my servers, who each one serves and the number of connections... perfect!

I got first timing results with Hadoop, it was close to what a simple shell script would give me. I scaled it up to the second node, and I got 25% less time. Not bad, but not great either. Then, I decided to use all possible machines that I could put my hands on to see what would happen. Two laptops and two desktops--one of them being remote. With the increased number worker threads, I got a result in one-third of the time! I had the output data answering my first question in a scalable way and with $0 cost.

Now what?

Ok, I have aggregated data, but what did it mean? How could that give me answers to real business questions, like: where are these clients located? which country or even city uses my databases the most? which server do I have to upgrade first and monitor closely? where should I put my next server?

At this point the result of my map/reduce functions was structured data that would fit into a database nicely. Even if I had one connection from each IP in the world (approximately 4.3 billion addresses), a PostgreSQL database would handle them without any problem. I would be able to answer every possible query and I would be able to do something extraordinary: code Geo-functions that would tell me where each IP came from. I used plsh to create some functions to call a whois service and return the country, the region, and most importantly, the longitude and latitude of each IP. The possibilities are infinite! I could pinpoint groups of IPs and make usage patterns, like Heat maps and other neat stuff. To take this project even further one last thing was missing, a good way to visualize my data. I had zero experience with Geo data -- and by zero I mean zero! I found that I could do what I wanted with google map’s API but Google has some strange licensing model that I didn’t understand so I decided to go with “Open Street Maps”. Again Open Street Maps is open source, like everything used so far. All I needed was some basic knowledge of Javascript to use their API in order to create heat maps. It took me some time, but later the same day I had pretty output with the answers I wanted:

Conclusion

If, as they say, information is gold, then open source is the alchemy of the 21st century: making gold out of iron.

What began as a simple question, turned out to become something more--some kind of “business usable solution.” No matter how much data, no matter how many clients or servers, no matter the origination of the logs, this “patchwork of projects” would give answers, it would scale and, as a result, it would answer real questions!

Appendix

Technologies used:

• PostgreSQL
• Hadoop
• Python
• Plsh
• SQL
• Shell scripting
• sed & awk
• Apache
• Javascript
• Open Street Map

Code Repository:

• https://github.com/vventirozos/log_reader

Setting up the New Nodes

Before any cluster changes can take place, the new nodes have to be set up. Versions 1.2+ have excellent clustering tools, and depending on what version the existing cluster is running, there may be extra configuration necessary to handle the intermediate mixed-version cluster if an upgrade is to take place. Be sure to refer to the documentation Basho provides for installing Riak.

When migrating a live cluster, the network is of utmost importance. Keeping latency low and predictable will keep the impact small. If a fast, reliable network can not be established, a live migration may not be possible. Additionally one should have some level of confidence that there will be no network splits between the clusters. While Riak is designed to handle such problems, removing them from the equation will greatly increase the chances of a smooth transition.

After installation, there are a few things to check before starting Riak. Of particular importance is that the Erlang key in /etc/riak/vm.args is the same as the one that the existing nodes use. The node name should be changed from the default node@127.0.0.1 to use the correct non-local ip. If Riak (1.2+) has been started before the name is changed, the following command has to be run even though it is a stand-alone node:

riak-admin cluster replace  

Older Riak versions should have the /var/lib/riak/ring directory emptied before a restart in that situation.

In /etc/riak/app.config, the http (or pb, if using protobuffs) should be configured with an appropriate non-local ip. More info on configuration files can be found here. In addition, you should make sure that there are no firewall rules blocking communication between the new nodes and the existing cluster.

The Important Commands

Before touching the existing cluster, it is important to become familiar with the tools available and necessary for the migration ahead of time. There are some differences depending on the version of Riak being used.

Pre-1.2

Versions prior to 1.2 have immediately acting cluster methods. All modifications are done individually with the following commands.

# Join this node to the cluster with 
riak-admin join 

# Remove this node from the cluster
riak-admin leave

1.2+

Riak 1.2 introduced the staged cluster modifications. Rather than each action occurring immediately and separately, they are all staged first for review. From that point, the changes can either be thrown out or committed as a single action. The following commands will be used to handle and monitor the migration.

# Join this node to the cluster with 
riak-admin cluster join 

# Remove this node from the cluster
riak-admin cluster leave

# Move partitions from 1 to 2, then remove 1
riak-admin cluster replace  

# See currently staged cluster modifications
riak-admin cluster plan

# Clear all staged changes
riak-admin cluster clear

# Commit all staged changes
riak-admin cluster commit

# Status of cluster members (ownership, etc.)
riak-admin member-status

# Additional cluster status information
riak-admin ring-status

All Versions

There are two monitoring commands that are available with all versions.

# Whether all nodes agree on the state of the ring
# and who the members are
riak-admin ringready

# Status of partitions pending transfer
riak-admin transfers

Example Setup

For the purposes of illustration, assume the following production setup:

• Existing Cluster
  • old1@192.168.1.2
  • old2@192.168.1.3
  • old3@192.168.1.4
• New Nodes / Final Cluster
  • new1@192.168.1.5
  • new2@192.168.1.6
  • new3@192.168.1.7
• Load balancer in front of Riak, routing queries to cluster nodes

Approach 1: Join And Leave

This method will work with all versions of Riak. All of the new nodes will join, assuming ownership of part of the ring, resulting in a large intermediate cluster. The old nodes will then leave, relinquishing ownership of the remainder of the ring, leaving the desired final cluster. Patience will be key.

Join the New Nodes

The first step is to join the new nodes to the existing cluster. This will form a single, large cluster and cause a rebalance where ownership of some partitions are transferred to the new nodes. With Riak 1.2+, the following steps should be run (note the host on which the join commands are run).

# Stage join new1 to the cluster
new1$ riak-admin cluster join old1@192.168.1.2

# Stage join new2 to the cluster
new2$ riak-admin cluster join old1@192.168.1.2

# Stage join new3 to the cluster
new3$ riak-admin cluster join old1@192.168.1.2

# View staged changes
new3$ riak-admin cluster plan

# Make the changes
new3$ riak-admin cluster commit

Check that all nodes are now part of the cluster with ringready. At this point, the cluster will decide which nodes should have which partitions and begin to work toward that state. The current and desired (pending) percent ownership of the ring can be monitored with member-status. Total pending transfers and (1.2+) information about active transfers will be seen with transfers. Depending on the size of the cluster, amount of data and load, this can take a while to finish. Look for the percent ring ownership to settle and there to be no remaining transfers. Once the cluster has settled, the new nodes can be added to the load balancer to take point on incoming requests.

Riak 1.1 and earlier is a similar process, though each node will have to be added individually. Wait for all transfers to finish before adding the next node.

Leave the Old Nodes

The next step is to remove the old nodes from the cluster. It is essentially the opposite of what was done to join the new nodes. First remove the old nodes from the load balancer to prevent them from handling incoming requests as they leave.

Then, with Riak 1.2+, run:

# Stage removal of old1 from the cluster
old1$ riak-admin cluster leave

# Stage removal of old2 from the cluster
old2$ riak-admin cluster leave

# Stage removal of old3 from the cluster
old3$ riak-admin cluster leave

# View staged changes
old3$ riak-admin cluster plan

# Make the changes
old3$ riak-admin cluster commit

Follow along progress with member-status and transfers until the old nodes have completely given up ownership and left the cluster. Confirm with ringready.

Again, Riak 1.1 and earlier will have old nodes leave one at a time, waiting for rebalancing to complete in between each.

Approach 2: The Replace

With Riak 1.2+, it's possible to do the migration by using the new replace command. In this case, both the join and the replace are staged and committed together. This will have to be done for each node to be replaced, but will avoid having to rebalance the entire cluster. The current node to be replaced should be removed from the load balancer prior to starting the replacement.

new1$ riak-admin cluster join old1@192.168.1.2 # Stage join new1 new3$ riak-admin cluster replace old1@192.168.1.2 new1@192.168.1.5 # Stage replacement 1 new3$ riak-admin cluster plan # View staged changes new3$ riak-admin cluster commit # Make changes

Once the change settles, as indicated by the new node returning true for riak-admin ringready, the new node can be added to the load balancer and the next node replacement can be performed.

The example here has an easy one-to-one mapping. If the cluster sizes are different, simply stage additional joins or leaves as needed after the replacements are done, similarly to Approach 1.

In Case of Stuck Ownership Transfer, Break Glass

Every once in a while, an issue during the rebalance may cause a transfer to get stuck. Using transfers and ring-status, one can confirm which nodes are involved and where in the transfer process things are stuck. Keep an eye on transfers with "unknown" rates and long runtimes (listed next to the start timestamp), or handoffs waiting on something similar to "[riak_pipe_vnode]".

First step is to try restarting each of the two nodes involved, one at a time. Sometimes a node can take a while to come back online fully. Use the following command to monitor for a node to be ready after a restart:

riak-admin wait-for-service riak_kv 

If that doesn't help, try attaching to the console to the original partition owner and run the following commands. NOTE: Do not use Ctrl+C while attached; it will kill the node. Use Ctrl+D to detach when finished instead.

old1$ riak attach
> riak_core_ring_manager:force_update().
> riak_core_vnode_manager:force_handoffs().

That should get things moving again. If not, try running this function for handoff stalls outlined in the Riak 1.0.2 Release Notes. Despite being for an older version of Riak, we've found that it can do the trick on newer versions as well.

Conclusion

Migration of a Riak cluster is simple and straightforward, particularly with the 1.2+ cluster commands. Basho has supplied excellent documentation for Riak if further information is needed on any command or process.

Let's start with the JPEG format.

JPEG: Joint Photographic Experts Group

The JPEG compression algorithm dates back to the early ‘90s and is used in a wide variety of formats, but here we only care about the commonly used JPEG format. The JPEG algorithms are the most sophisticated compression technique of the three formats we'll be looking at, and are a type of "lossy" compression. This means that some information is lost each time you save and compress a JPEG image. This is important to remember: if you have a JPEG image which you open and re-save, you're losing additional information each time you re-save it. If you re-save a JPEG file too many times, it will begin exhibiting visual artifacts from the re-compression process, even if you're saving it at a high quality setting.

On the left is the original image, in the middle it’s been saved once at 80% quality, and on the right it’s been saved ten times at 80% quality. Even using a high quality setting such as 80%, you can see that resaving further worsens the visual quality.

JPEG compression uses mathematical algorithms to imitate human vision, prioritizing overall color tonal changes over precise detail reproduction. It subsamples your image into small squares, usually either 8x8 pixels or 16x16 pixels (more common), which means you'll get the most efficient compression if your image dimensions are multiples of 16. (Another benefit of sticking to multiples of 16 is that it allows editing applications to rotate your image 90 or 180 degrees without re-compressing it and throwing away more information.)

Because of the way JPEG algorithms work, the JPEG format is really only suited to photographs. JPEGs most commonly use a 24-bit RGB color space (the best for photographs), but don't support transparency, and should not be used for graphics or most non-photographic images. If you want to compress graphics for your application interface, use either the GIF or PNG formats.

On the left is a photo saved with JPEG compression, and on the right is a graphic saved the same way. Even when blown up significantly, the photo doesn’t have many visible compression artifacts, whereas the graphic is noticeably degraded.

Note: when saving JPEG files for web use (and thus trying to minimize the file size), including a color profile makes the file unnecessarily larger. When you compress and save JPEGs for web use, convert your photos to use the sRGB color profile and then be sure you don't include the color profile when you save.

GIF: Graphics Interchange Format

First let's get the pronunciation out of the way. There is a debate as to whether it's pronounced with a "hard g" or "soft g," but according to the format's creator, Steve Wilhite, "GIF" is pronounced just like the peanut butter, Jif. "Choosy developers choose GIF" was a common catchphrase among early GIF developers.

The GIF format is the oldest format here, dating back to 1987, and has enjoyed a resurgence in popularity recently due to the current animated GIF craze (for example, see http://www.gif.tv). But the GIF format is really the workhorse of the group, still popular after a quarter century of use. GIF images are great for compressing graphics and other high contrast, non-photographic imagery because they use lossless LZW compression. "Lossless" compression means you can re-save the image multiple times without losing information. However, this doesn't mean that information isn't discarded when the GIF is initially compressed and saved.

Although the GIF format supports the 24-bit RGB color space, you cannot use all possible colors in a single GIF—you're restricted to a palette of 256 colors (8-bit color). But this is not a downside, it's an upside. . .it's what makes a GIF so tiny when used properly! For the first few years of GIF's existence, we were limited to only the 256 pre-selected "web safe" colors which could be simultaneously rendered by the limited graphics technology of the day, and thus were guaranteed to be accurate on most people's monitors. Now, you can choose any 256 colors from the entire 24-bit color space. If you're compressing an image with lots of colors, you may need all 256 colors, but if your color range is limited then you can make your image even smaller by using a smaller palette (sometimes as small as 4 or 8 colors). If a limited palette forces you to reduce the number of colors used when you export your image, you can usually apply dithering to make the colors appear to blend and look more like the original image. Use dithering sparingly, though, because it makes your GIFs larger depending on the particular technique used.

GIF's limited color palette is important to remember when you're making image sprites, too. When you make your sprites (by combining several small images into a single larger image), take careful consideration of the colors used in the images. Because you're working with a limited palette, it's best to group images that use the same colors, that way they're not competing for space in the limited color palette.

When these blue, red, and green graphics were compressed in the same image, their gradients were dithered due to being forced to share a single color palette.

When the blue, red, and green graphics are compressed in separate images (denoted by the dotted lines), their gradients are smoother due to the increased color palette space.

GIF does support transparency, but only 1 bit per pixel. This means a pixel can be either opaque or transparent, but not partially transparent. Because of this, GIFs are usually only suitable for transparency if the edges of objects are straight horizontal, vertical, or 45 degree angled lines. Objects with curved edges or angles other than 45 degrees will appear jagged, and may have a “halo” of color around them. Also, if you choose to use transparency in GIFs, note that it takes up a location in your color palette so you'll be limited to a maximum of 255 colors.

PNG: Portable Network Graphics

The third and last format we'll look at is the PNG format. If you need images with partial transparency, this is the format for you. The PNG format was designed in the mid-90s to be a royalty-free alternative to the GIF format, and it uses lossless ZIP DEFLATE compression, making it a great option for graphics and non-photographic imagery.

PNG images are a good all-around choice for graphics due to their flexibility. A PNG can be either a full 32-bit RGB image (24-bit color with an 8-bit alpha channel supporting partial transparency), or it can be a paletted 8-bit image just like a GIF. (Note: Internet Explorer started supporting 8-bit transparency in PNGs with IE7, so as long as you don't need to support IE6, you can use 8-bit transparency with no concerns.)

When planning on using an 8-bit paletted image, it's good to compare the difference in file size between PNG and GIF versions. Although an 8-bit image will look the same in either format, one format will result in a smaller file than the other because they use different compression schemes (LZW vs. ZIP). A general rule is that a GIF will be smaller when working with small images and a PNG will be smaller when working with medium and large images. But you'll have to do a comparison in each case to see how the formats perform.

It's very enticing to be able to lay out an image sprite with full 32-bit color and transparency, but be careful. When working with full 32-bit PNGs, you don't have the advantages of JPEG compression or an 8-bit palette to help save space, so 32-bit PNGs can get quite large. Use them sparingly. (Since PNGs don't use JPEG compression, don't use full 32-bit PNGs for photographs if you can help it; photographs should always be saved in the JPEG format.)

JPEG vs. GIF vs. PNG

JPEG, GIF, or PNG… if you’re not knowledgeable about their differences, it can be tricky figuring out which format to use in a given situation. All three image formats are useful in different circumstances; you just have to know their strengths and weaknesses. Here's a reference table to help remember the differences:

Resources

For further information about the JPEG, GIF, and PNG formats (including more technical details about how their compression schemes work), wikipedia has some great articles: Wikipedia: JPEG, Wikipedia: GIF, & Wikipedia: PNG

Although most designers use Adobe software such as Photoshop or Fireworks for compressing web images, those tools may not be practical options for all budgets. If you’re without access to either of those, GIMP is a nice alternative for manipulating and preparing images. And for those who prefer command line interfaces, ImageMagick can be a great tool.

Prerequisite knowledge

The ecosystem of PHP extension is heavily centered around C, so you will need a solid grasp of the language. The canonical reference here would of course be the K&R C book (ISBN: 978-0131103627), but there are many references and tutorial resources available out there. You will also likely need to be very familiar with your development environment’s debugger (see section below for gdb-specific tips).

Extending and Embedding PHP by Sara Golemon

Sara Golemon’s book (ISBN-13: 978-0672327049) is probably the single best printed resource available for modern (PHP 5+) extension development. This is not to say that it is a book without flaws, but it is something that I found myself referencing quite a bit during a recent PECL project. At seven years old as of this writing, it does miss some of the most recent developments in the PHP internals space (e.g., the mechanisms for calling user-defined PHP functions from C, as you would need to invoke a user-provided callback, changed in 5.3).

Marcus Börger’s Extension Talks

http://talks.somabo.de/ contains an archive of various PHP conference talks given by Marcus Börger; the extension-related ones are good summaries of the topics to hand and may serve as a useful adjunct to the other resources listed here. In particular, his coverage of PHP5 object/class support from within a C extension is useful as a supplement to the coverage in Extending and Embedding PHP.

Kristina Chodorow's PHPLovecraftian Blog Posts

http://www.kchodorow.com/blog/2011/08/11/php-extensions-made-eldrich-installing-php/ A series of several blog posts introducing writing PHP extensions; very worthwhile reads (the above is the first article in the series). Heed her advice particularly in the first one on how to set up PHP for extension development and debugging, because it will make your life much easier. Honestly, it is worth spinning up a VM to isolate all this from your regular environments for this reason alone. (VirtualBox, vagrant, etc., are your friends here, but I imagine as a working developer in 2013 you probably already are familiar with using VMs in some fashion.)

LXR: sometimes the source is the only documentation you’ll have

Sadly, there will be parts of the PHP extension process that are completely undocumented in any form. When you run into something like this, the best resource is http://lxr.php.net to conveniently read the source to PHP itself. Particularly, you will be making heavy use of sometimes opaque C macros, and LXR is the best place to turn to find out what they actually do behind the scenes (e.g., MAKE_STD_ZVAL).

GDB and .gdbinit

If you spend time writing PHP extension code, it is a virtual certainty that you will spend time trying to understand what’s happening inside of the PHP interpreter, and your extension code, using gdb. If you are not familiar with gdb, there are a wide number of gdb tutorials available, and I wouldn't hazard a guess as to the best one. That said, what I can highly recommend is the use (or inclusion, if you already have one) of: https://github.com/php/php-src/blob/master/.gdbinit

If you put that in your home directory the next time you start gdb you’ll have all manner of extremely handy debugging macros at your fingertips. “printzv” in particular for easily displaying a ZVAL will save you from having to type things like “print *((zval *)(*((*((*((zval *)(*(thing_ctx*)foo)->extended_value)).value.ht)).pListHead)).pDataPtr)”

Be wary of Zend’s built-in data structures

Just a general warning; it might seem like the path of least resistance to use the pre-existing Zend data structures to pass data around within the domain of your extension (“meh, it’s a hashtable, beats having to write one of my own. . .”). In my experience however, this way lies more headaches than you would think (e.g., add_assoc_* will nul-terminate your keys behind the scenes, which may or may not be what you want). It’s easier in the long run to route around this for your own internal C-land use and use more transparent mechanisms for data storage (i.e., use your own structs/arrays/ pointers/hashtables/linked lists/etc. and don’t rely too heavily on Zend’s), converting back into whatever type of ZVAL is appropriate only when needed to give values back to the PHP side of the fence.

If you do spend any significant amount of time needing to interact with HashTable ZVALs in particular, you will probably want to write utility functions along the lines for add_assoc_* for the other HashTable operations (get current key/value, etc.) simply for sake of saving keystrokes. You likely will need to spend time getting familiar with HashTables as they are the underlying data structure for all arrays in PHP, both numerically indexed and associative, and are common return values.

Testing

PHPT tests are the easiest by far to distribute with your PECL extension as they integrate natively with “make test.” http://qa.php.net/write-test.php gives an overview of the file format and suggestions for how to write good ones.

C++

As mentioned earlier, the PHP extension ecosystem is heavily centered around C. There is some, but not much, information available online for interfacing PHP and C++. “Wrapping C++ classes in a PHP extension” by Paul Osman: http://devzone.zend.com/1435/wrapping-c-classes-in-a-php-extension/ appears to be one of the better resources.

PECL

Once you’ve gone through the trouble of writing and debugging your extension, you may want to share the fruits of your labor with the rest of the world. PECL (http://pecl.php.net/) is the extension repository companion to PEAR, which you may already be familiar with in the context of pure PHP packages, and would be a good place to look for information on how best to do this.

Good luck and Godspeed

Writing a PHP extension can be intensely frustrating and confusing at times, but it’s magical once you’ve gotten things working for the first time. It can also give you a much better understanding of how PHP works by becoming familiar with the internals of the language, which is a worthy goal in it's own right. I hope the resources above help get you through the rough patches along the way.

10,000 hours

Sometimes when I have a few minutes of downtime, I'll go hang out on IRC; you can usually find me in the #postgres channel on freenode. This is an old habit that stems from back when I changed my career path from roles where I was primarily doing web development into roles where I was managing databases full time. I spent a lot of time asking questions, and studying the questions that others asked. I learned countless things there, both about errors within Postgres and the types of problems people ran into; it was an invaluable experience. If it really does take 10,000 hours to become an expert, I suspect many of my hours were logged right there.

Building Better Gurus

A few months ago, one of our DBA's asked me to take a quiz. It had arisen from an interaction he had had with a new prospect, who was looking for a Microsoft SQL Server "Guru". Now, I wouldn't claim to be a guru in MSSQL, but it isn't totally foreign to me either; I've personally developed applications that hit against MSSQL, and done trigger development as well. I've also debugged replication systems that ran against it and had to debug extremely complicated locking behaviors; in short, I have helped a fair number of companies find success with MSSQL problems, and even to this day we still work with a few companies who make use of it, so I figured I might as well give it a go. The test sounded like a typical database problem; a query was not making use of an index when the prospect had thought it should. I started down the normal paths; have the statistics been updated? Yes. And we know there isn't corruption? Yes. The cardinality of the data within the column is such that the index should be used? Yes. They had even tried adding a hint, but it was ignored. At this point, I felt I needed to see the query and the table structure, and maybe even an explain plan.

BOOM! I failed. I was not a guru.

The opinion of the prospect was that a guru should just know what the answer to that was, so since I couldn't solve the question straight out, clearly I was not a guru. Turns out I'm ok with not being a MSSQL Guru, but I do think that the idea that this is how you measure gurus is wrong. Or, maybe that definition is fine, but it doesn't really tell you anything about whether someone that you are talking to can help solve your problem, which is the thing you really want.

What's the difference?

The problem here is that this person was conflating two distinctly different skills sets, but attributing (I believe incorrectly) the expectations of one skill to the other. What the person in the quiz wanted was someone who was good at pattern recognition, not someone who could troubleshoot. Don't get me wrong, pattern recognition is certainly useful, and can be extremely valuable, but it also has limited applicability. If you get a "failed to re-find parent key" error in Postgres, I'm probably (well, hopefully) one of the few people who have seen that error in production, and I have the burn marks to remind me about it. If you are asking on IRC at the same time I am hanging out there, you're going to get a much quicker path to a solution than otherwise. This is great, and maybe some people would say I am a guru because of that, but I sometimes find these pattern recognition skills to be a liability. If you have a connection error to your system, there are less than half a dozen problems it will likely be, and if I'm trying to deduce it, I know I sometimes just fall back to applying the patterns and seeing which one fits. More often than not this is actually going to result in a fix faster than if I tried to troubleshoot the problem. But what happens when you come across something that doesn't match the patterns?

Troubleshooting is really the ability to follow a (repeatable) methodology, step by step, to find a problem point. For something like a connection error, you should be able to start on one end of the connection and walk the path that connection should take, verifying that things work at each point along the way, step by step. You don't check the client credentials and then jump to permissions on the server; you actually verify that the connection is initiated by the client first; then you verify that the request makes it to the server. Not just to the host machine, but do you see a connection being received by the server itself? Yes, this takes longer. And if you know that 40% of all problems are incorrect credentials and you know that another 40% are permissions settings in the server, skipping things like checking if a firewall is getting in the way is probably going to slow you down, but if you really want to be a "guru", you need to be able to handle that other 20%.

Have you ever had the opportunity to watch someone debug website slowness with curl and tcpdump? That is someone who is working to become a true guru; not just understanding patterns of operation, but understanding how things really work. This is one reason why we encourage lower level systems knowledge and a polyglot approach to each part of the stack. When we started building Node.js websites, we were able to take fundemental skills we had learned building scalable sites in Perl and PHP and apply them to this new technology. And that's where you want your guru to be. Even when in completely unfamiliar with a technology, with no pattern matching built in, to be able to understand and solve problems in a new domain by applying the fundamental tools that you know along with troubleshooting skills. That is a "guru".

Intended Audience

In previous entries, we examined best practices in developing Chef cookbooks. In this article, we shift our focus to a powerful, but often overlooked application of Chef: its use as a fixture generation system for application quality assurance testing. Only basic Chef knowledge is needed, though some familiarity with the software development lifecycle is helpful. We’ll be showing how we integrate git, Chef-solo, Vagrant, rspec, jUnit and Jenkins into one QA provisioning stack. To be clear, this article is not about testing Chef cookbooks - there are several good tools and tutorials available for those looking for that.

Repeatability: Not just for Scale-Out

Chef’s main user base is composed of operations engineers. They generally see it as a tool that brings massive automation to their provisioning tasks. If you can configure one machine with Chef, you can easily configure a thousand. Repeatability is key; you don’t want the servers to vary, except in very specific ways that you know about and control. So, Chef lets you define trees of build parameters (called attributes) that you can vary in numerous ways.

Away from Ops, QA practitioners are dealing with similar problems. Instead of scaling out for production load reasons, the QA team wants to build and discard many systems, testing permutations of build, operating environment, seed data, test set and so on. The QA systems load is also highly elastic - in the final days of testing a release, the ability to run many variations simultaneously is crucial. This gets us away from having “the QA server” that is in a precious state - we can always spin up more QA environments.

This assumes that systems and application deployment are integrated. Chef offers many avenues for this; you can use application deployment tools like Capistrano inside Chef, or use Chef-specific tools like the “artifact” cookbook to deploy applications from tarballs or other distribution media. For shops that do not use an application deployment tool, Chef itself can be a very effective tool to deploy applications (performing code checkouts, loading SQL into databases, starting webservers, etc).

Choosing a Platform for QA Provisioning

Choice of platform is largely dictated by the needs of both the developers and the operations team. The QA group will want to be able to allocate and deallocate environments on-demand to respond to testing needs. If self-hosting, this may require tooling or coordination with the operation group. Two additional methods that might be more appealing include cloud-based hosting (which makes a great deal of sense if the production application is cloud-based anyway) and the desktop virtualization harness, Vagrant.

Vagrant allows you to use existing hypervisors like VirtualBox, while executing full Chef runs and performing testing against the VM. VMs can also be brought up in clusters, and have private networking between them, enabling testing of distributed systems in a more realistic fashion. The most recent versions of Vagrant include support for multiple hypervisor platforms, including cloud-based providers such as AWS. Vagrant also allows you to run an environment locally (ideal for developers). We found it to be an ideal platform, but missing several key pieces.

Vagrant Cookbook Fetcher

The first piece that we found missing was a way of managing which cookbooks, roles and data bags were needed for a project. Berkshelf and Librarian both fill this role to a certain degree, but we found their learning curve to be rather steep - a “too big” tool that didn’t entirely meet our needs. We wanted to be able to share our roles, databags, handlers and other auxiliary Chef configuration across projects - all of the cookbook distribution methods out there seem to only focus on sharing the cookbook itself.

We developed a vagrant 1.0 plugin, vagrant-cookbook-fetcher, which reads a CSV -formatted file (specified by a URL) listing each checkout to perform. Because Chef allows multiple cookbook paths, but only one role path, we then created a combined directory in which it symlinks to the various roles. The plugin configures Vagrant’s Chef-solo provisioner to find the roles, etc., in the new combined location. A similar approach is used for databags, handlers, tests and other shared features. Finally, hooks are added to the ‘up’ and ‘provision’ vagrant commands, to trigger the checkout at provision time. The tool is designed to work identically to the Chef-solo-helper tool that we use when bootstrapping production VMs.

Using vagrant-cookbook-fetcher, we can easily recreate any environment, even if the Chef configuration is spread over several repos that may have different access policies.

Vagrant + rspec

The next step is to be able to write tests against the VM that we have created. In the Ruby world, there are several test platforms (minitest, rspec, and cucumber) covering a continuum from unit testing, explicit behavior-driven-testing, and natural-language behavior-driven testing. Of the three, rspec is the happy median; you can use several testing paradigms in rspec, and support for it is broad.

Again, we turned to creating a vagrant 1.0 plugin, vagrant-rspec-ci. Here, our goal was to be able to write specfiles in a project repo that expresses tests against a VM, run the tests easily, and produce test results in a format that could be consumed by our continuous integration server, Jenkins.

The vagrant-rspec-ci tool lets us run rspec tests, then saves the output of those tests in jUnit format (using ci_reporter) in a directory. There are several Vagrantfile configuration variables you can use to control its output and behavior; the most interesting are listed below.

config.rspec.enable_ci_reporter

Defaults to true - whether to produce jUnit reports

config.rspec.tests = [ ‘*_spec.rb’ ]

Array of globs that specify which test files to run.

config.rspec.rspec_bin_path

Defaults to finding the rspec binary in vagrant’s gemset, then falls back to plain ‘rspec’.

All of the tests are executed outside the VM. We chose this approach for several reasons:

• we did not want to alter the system build to include testing tools, like rspec
• we want to test the behavior of the application server as a unit - generally, a blackbox
• vagrant offers several avenues to probe inside the VM if needed, without actually moving the entire test execution there

Probing A Vagrant VM

When testing the behavior of a VM, there are several tasks that come up repeatedly. Some are test predicates we would want to be able to use repeatedly; some are simply informational queries against Vagrant or VirtualBox.

We have captured several of these tasks into a set of Ruby classes to represent a Vagrant VM as a test subject. vagrant-test-subject is typically used as follows:

The first interesting line is the attach() method call.

This looks for the Vagrant VM running in the current directory, and extracts some key information:

• VirtualBox VM GUID
• VirtualBox OS type
• ssh options
• port redirection map

The call to attach() is actually a factory method, and will return an OS-specific subclass of VagrantTestSubject::VM.

Next, we define several tests (or “examples”, as they are called in rspec). Each examines the VM, using various methods defined on the object. Some highlights are below.

@vm.map_port(80)

Returns the port on the VM host that is being redirected to the given port on the VM guest, if any.

@vm.has_running_service?(“httpd”)

Using OS specific means, determines if the named service is running and healthy. Facilities are provided to allow OS-specific aliases for service names.

@vm.process_name_listening('127.0.0.1’, 80)

If possible, determines the command portion of the process listening on a particular IP address and port.

@vm.http_get(url_path)

Returns a Net::HTTPResponse that contains the result of GETing the URL path on port 80.

Together with rspec extensions (like rspec-http), vagrant-test-subject allows the tester to write fluent tests against the VM. vagrant-test-subject is a very young package, and is evolving rapidly.

Parameterizing The Build

As discussed in part 2, we try to decompose our Chef roles to be orthogonal along certain dimensions, such as OS choice, datacenter location, hypervisor, etc. This means that we can vary the Vagrant build by simply changing which of these roles is used in the Vagrantfile. Because a Vagrantfile is Ruby, we have many choices for such a mechanism; but the simplest working approach would be to select each role based on an environment variable.

Excerpt from a Vagrantfile:

To build and test our subject under CentOS, we simply use the default:

To verify that an OmniOS build works just as well:

The use of environment variables as the mechanism for varying the build has an additional benefit - our CI server, Jenkins, can parameterize build jobs using the same mechanism. So we simply have a Jenkins job that exposes VAGRANT_HINT_OS with two different values, ‘centos’ and ‘omnios’.

Fixtures as Recipes and Roles

The last piece of the puzzle needed for testing is some way of initializing the test subject with state; i.e., test fixtures. Types of state in fixtures include filesystem assets, database records, cache entries, queue state and other project-specific needs. Because the fixture types vary by project, and the process by which to install them varies as well, we need a generic way of running variety of tasks within the VM to converge to a known state. Obviously, Chef recipes are a good fit here.

Following the attribute definition practices we outlined in part 2, we implemented a recipe for each fixture type. Each recipe expects to find details about the fixture in the node’s attributes. Together, that allows us to define a specific set of fixtures using Chef roles. A simple example will help illustrate.

The Fontdeck project’s CDN nodes store fonts on the filesystem. When testing the font serving system, we’d like to be able to load a single known font into the test subject VM representing the CDN, then try to access the font as both an authorized and unauthorized user.

We start by defining a recipe to manage installing the font files by walking an attribute tree and using the ‘tar’ community cookbook to download tarballs and expand them in the asset area.

We then define a companion recipe that purges the fixture. While not entirely needed (we could just destroy the VM and start over, or use VirtualBox’s snapshotting facility) this approach is simple and fast. There are safeties in place to ensure that the purge will not run unless specifically authorized.

Now that we can install and remove font assets, we define a specific fixture as a role.

Typically, a project will also include a purge-all role, which simply calls each purge recipe.

Revisiting the Vagrantfile, we add a section to dynamically load fixture roles from environment variables:

Finally, we’re ready to set up the fixtures and run the tests:

Conclusion

The worlds of Chef, Vagrant, and Jenkins are experiencing rapid changes. Together, they can be used to solve serious problems in QA, allowing us to view entire systems - or groups of systems - as test subjects. To reach that goal, a lot of glue code is still required; and some of the best tools have yet to emerge. Even developing tools in-house, we’ve had some payoff from the process - we halted the release of a trafficserver module because it failed a test in QA. One month ago, the module was considered to be “only testable in production”. We look forward to rolling out this approach to more projects, and reaping more benefits as we go along.

Intended Audience

This is not a Chef tutorial, nor a journal of one person’s experience with Chef over a month. You will not learn what a resource is or enough Ruby for Chef. Instead, we’re assuming you have used Chef enough to know what it is, what it does, and at least one way to do what you want to do. You may have authored several cookbooks for your organization’s internal use, and taken slightly different approaches as you learned what works and what doesn’t over time.

Understanding Cookbook Maturity

As cookbooks develop over time, they go through several stages. It can be helpful to understand where a cookbook resides on the development path, to know when investing more time would increase its usefulness. Not every improvement will provide a return.

Early

• May be written for a specific project: no initial intent to re-use across the company or release as open-source
• Values hardcoded, no attributes used; or attributes peculiar to the project
• Depending upon the skill level of the author, may not be aware of existing Resources that can do the job (e.g., might use a “bash” Resource to create a directory)
• No readme, or simply boilerplate

Mid-Life

• Generalized to be a stand alone, non-project-specific cookbook, with bundled templates
• Provides a basic readme and metadata file; may have to read source for details
• Well-defined use of attributes in its own namespaced tree; common cases covered, though rare cases may not be parameterized yet
• Provides defaults for its attributes in a cookbook attribute file
• Provides an attribute-walking default recipe
• Some test scripts, though incomplete; not likely to be portable
• Released company-wide

Mature

• Readme contains sections on recipes, attributes, use cases, resources
• Standardized interaction with data bags, via databag-walking recipe
• Related functionality provided in auxiliary recipes
• Provides ability to operate either via attributes or via databags, with configurable databag names and well documented databag item formats.
• Push software action into LWRPs so recipe authors can use resources without tampering with attributes
• Push functionality into libraries if needed for DRY
• Extensive testing, including for portability across platforms
• Open-sourced

Of course, different organizations will see variations on this path. For example, your cookbook may begin its life on github from day one; you may use TDD and write tests before you even write the first recipe.

Define Attribute Trees Carefully

As adoption of Chef spread within our organization, we found that most users fell into the “black box” category - those who did not have time to learn any Ruby, and wanted to limit their need to learn Chef terminology and concepts. By exposing as much functionality as possible via attributes, we were able to empower this group of users.

That means that the attribute trees effectively become the user interface for many people. The Ruby learning requirement has thus been reduced to editing the literals in the attribute definitions in the role files. Syntactically, this is a low barrier; but there are some massive conceptual barriers to using attributes that still remain. Since the attribute tree is wide open - you can put anything anywhere, for any reason - we tried a variety of approaches, and developed some conventions that helped reduce needless confusion and complexity.

Attribute Trees Should Be Namespaced with the Cookbook Name

For example, if we have a cookbook that installs packages named “packages,” we would use the attribute tree to store lists of packages to install, under the “packages” branch. In a role file, this might look like:

Each cookbook is thus free to define whatever structures are needed for that individual cookbook.

Like any naming convention, it is most powerful when applied consistently. We encountered numerous bugs and moments of confusion because our sudo recipe used a nonstandard namespace ( node[:authorization][:sudo] ). Users created roles that populated values under the node[:sudo] tree, which is what you would expect; but because the cookbook was looking in node[:authorization][:sudo], the attributes did not have any impact. Because the majority of users only make occasional changes to the attributes, they cannot be expected to remember arbitrary differences in namespaces.

Each Project Should Have a Bespoke Cookbook with its own Namespace

Every project seems to be a little different: this one has memcached, mysql and PHP; that one has Java app container, with a complex vanity hostname setup. Each project will thus need to break out of the attributes defined by the stock, shared cookbooks, to define special values that are used in the project’s custom recipes.

Without guidelines, we found that each project tended to define multiple top-level trees, often with very generic names (:cdn_info, :load_balancing, :users, etc). This quickly became a problem, causing collisions with new shared cookbooks (:users, especially). When we moved staff between projects, it was difficult to tell what was a company-standard attribute tree, and what was unique to the new project. This made it difficult to leverage experience gained on past projects.

We found that giving each project its own top-level namespace solved many of these problems.

Prefer Hashes of Attributes to Arrays of Attributes

When one has a list of similar objects, an array is the natural choice for the data structure. Examples might include operating system packages, or database users. However, due to the precedence merging algorithm Chef uses, Arrays are very difficult to work with. You cannot replace an individual element from an array; you can only append to the array (same level of precedence) or overwrite the array entirely (higher level of precedence).

Neither option is attractive. If you’re overriding you almost certainly want to replace an element; and if you recreate the entire array, you now have duplicated configuration data, which is a common cause of errors (as one list gets updated, but not the other).

We have found that in most cases, using hashes is more readable (the keys are self-documenting) and much more friendly to using overrides, because each element is addressable by name within the attribute tree. You can thus override the setting(s) in a higher-precedence call, or even in a later, same-precedence call.

There are some tradeoffs here:

• You still can’t delete individual elements, but you can override their child values. Here, an :enable flag is being set to disable the entry (presumably, the ponies cookbook would look for and respect that value).
• Hashes are unordered. If order of the elements is important, you may need to add an :order key-value pair to the child attributes.
• It’s not as concise as the array form - each entry must have a name and some kind of value, which in its degenerate form may be an empty hash. That’s more verbose, but it allows more room for future functionality - adding a :color attribute to the ponies, for example. Additionally, the keys make the code more self-documenting - which is ideal for people who are only occasional users of Chef.

Because we can now “knock out” individual elements while still remaining at the ‘default’ precedence level, we have found that we very rarely need to use the ‘normal’ or ‘override’ levels. This is a big win for our black-box users; the precedence merging rules are complex and hard to remember. By keeping things at one level of precedence, we can simplify the learning curve for our largest group of users.

Provide Defaults for Multi-Element Attribute Trees

In the above example, you may have observed that if we are disabling Rarity by setting :enable to false, there must be an implicit assumption that :enable is true by default. Implicit assumptions are a big source of errors; how can we make the defaults explicit?

We can’t use the normal attribute precedence mechanisms, because they operate when the attribute path is known in advance. If you have an attribute tree that is expected to have a dynamically changing population of child elements, we can’t construct a default that will match it using the normal methods.

To solve this problem, we wrote a short library[a] that performs dynamic merging in the attribute tree at compile-time (just prior to convergence). It is exposed as a method call on Chef::Recipe, merge_attribute_tree().

More typically, we would place the call to merge_attribute_tree in a special recipe, fixup_attributes, that performs the merge(s) and may also do things alike attribute validation. This fixup_attributes recipe would then get pulled in via include_recipe in each recipe in a cookbook.

This approach has proven to be a powerful tool in the effort to de-duplicate configuration data, while still being intuitive to the occasional Chef user.

Keeping the Roles Sane

Role files are wonderful. They are simple, reusable, and easily understood in isolation. Unfortunately, role files end up being our only tool in many situations, and we end up pounding a lot of nails with them.

One of our client’s project has about 45 different roles. A wide variety of technologies are in play, but there are a number of commonalities; so each node typically uses 4-7 role files, some recursively. This can quickly become unwieldy, especially for the on-call ops staff trying to track down an incorrect attribute in the wee hours of the morning.

Consider using Ruby Nodefiles

Because we use chef-solo, we are dependent on our JSON nodefiles. This is a point of frustration for several reasons:

• The node files are stored apart from the role files; you have to look in two different places to trace an attribute or runlist
• node files must be in JSON, but role files can be in Ruby; if they are different, you can’t cut-paste to move attributes between roles and nodes.
• JSON is somewhat user-hostile. While the same can be said for Ruby literals, at least you can have comments, your choice of quotes, and in some cases, trailing commas and “reasonable” error messages. Preferences vary as to which is more awful.

One solution we have found is to have a nearly-empty JSON nodefile: {“runlist”:[“role[node-foomachine]”]}

The corresponding role file acts exactly as the nodefile would have, but now enjoys the (perceived) benefits of Ruby.

Name Your Role Files Along Several Dimensions

We found that people are using role files for several common, cross-cutting needs. We then leveraged that fact to devise a naming scheme that reduces surprises about which roles a machine would have.

We discovered the following dimensions - you may find you need more or fewer:

• dc: Data Center, the physical location and hosting environment. Often involves things like settings for LDAP servers, routes, etc.
• env: Deployment environment - Different DSNs, fixture files for database testing, etc. Should only include attributes, empty runlist.
• task: Application task - Usually has the “real” runlists. The classic chef role notion.
• vm: Hypervisor peculiarities. We use this especially for Vagrant, controlling memory allocations.
• os: which platform we’re on. Chef has other mechanisms for this which can be used in concert.

We then chose to name each role as follows:

Looking in the roles directory for the OmniTI Website project, you might thus see:

This simplifies answering a lot of questions:

• What roles would I expect a production database server in Chicago to have? (ows-dc-chicago, ows-env-prod, ows-task-db)
• I need all the webservers to also do a geoip database update. Where should I put that? ows-task-www
• Does this project care what hypervisor it runs under? Apparently not - no ows-vm-* files.
• Where should I start tracing for a problem on corpweb1? Start with node-corpweb1, and if you need to make a snowflake change, it’s safe to make it there.
• Our dev environment is EXACTLY like prod, right? (That’s never true - but by comparing the two files, you have explicit knowledge)

This has another important effect: many settings are driven out of “hiding” in the cookbook attributes file, and into roles. We like to make the cookbook attribute files intentionally broken (i.e., providing a “CHANGEME” dummy value) in some cases - so that you MUST include an env role explicitly.

The above naming scheme works well enough for us, but we are constantly refining it and finding new things that don’t fit. Having some kind of organization is better than none, however.

A Plea: Better Attribute Tracing Tools

Most frustrations I hear from both developers and operations engineers concern the difficulty of tracing an attribute. It’s easy to add debugging statements: Chef::Log.info(“My foo is: “ + node[:foo].inspect())

But that will only tell you the final value of node[:foo] - which is usually not actionable information by itself. Instead, what we want to know is:

• what file first set that value?
• at what precedence was it set?
• which files are overwriting - or overriding - the value?
• if I change the value in this role, which nodes are affected?
• what version control revision introduced this change?
• what other values exist in the attribute tree nearby (can help you detect misplaced subtrees)?

Imagine a tree browser tool - something that lets you start at a node, and descend into the attribute tree, optionally viewing the “history” of each key in the tree as it was merged. Such a tool would have tremendous value, both as a diagnostic tool and as an educational tool.

Conclusion

While recipes get a lot of the glory, attributes are truly the workhorses of Chef. Most interaction with chef - in terms of making changes to a deployment - can be handled through attributes if the cookbook is well-designed. Since most users are using the attributes - and not seeing the lower-level pieces like recipes, LWRPs, and the like - it’s important to present a attribute space that can be lived with over time.

In our next installment, we’ll explore applying software engineering techniques to Chef deployments, and vice versa.]]> http://omniti.com/seeds/preserving-nodejs-packages-and-sanity http://omniti.com/seeds/preserving-nodejs-packages-and-sanity Thu, 28 Feb 2013 16:18:08 GMT Software development is often an exercise in standing on the shoulders of giants. This is particularly true with Node.js, which is very much geared toward modular code. Managing the dependencies in a sane, predictable fashion is critical for smooth deploys and restful nights. There isn't a perfect solution, and no approach will work in all aspects, but npm's package.json in conjunction with npm's shrinkwrap feature will get the job done cleanly in most situations.
example |-- config |-- lib | `-- foo.js |-- node_modules | |-- A | |-- B | `-- Z |-- test |-- LICENSE.md |-- package.json `-- README.md

Node.js modules are self-contained, reusable pieces of code written either in pure javascript or a mix of C++ and javascript, the latter of which must be compiled before use. They reside in the node_modules directory and are included elsewhere by using require("foo"). The require() method will walk up the directory structure from the calling file, check each level for a node_modules directory and then for the module. This should be kept in mind, as it is what allows things like modules to each have their own dependencies, but can also cause unexpected behavior if more than one node_modules exists in the main project.

The vast majority of public modules can be found in one of two places: GitHub and npm. GitHub is a popular hosting service for code projects using Git revision control. Using a module from there would require a git clone of the repo and an npm build if compilation is necessary. The other option, npm, is a package manager for Node.js and is installed automatically with the core. Simply running npm install will create a node_modules directory in the current working directory, download the module, and then compile it automatically. Often modules will be hosted and maintained on GitHub and published to npm, but not always.

Manual installation is fine and dandy when developing, but eventually the project is going to have to be deployed or distributed and some way to replicate all of that work will be needed. One's first instinct might be to check node_modules into the project repository. The upside is absolute consistency, but the downside includes a larger repo with more noise, as well as issues with compiled modules being reused across systems; modules will generally need to be rebuilt anyway on each system.

{ "name": "example", "version": "1.0.0", "author": "John Doe ", "description": "An example project", "main": "./lib/foo.js", "repository": { "type": "git", "url": "https://github.com/some github user/example.git" }, "keywords": [ "example", "foo", "bar" ], "dependencies" : { "A" : "2.5.0", "B" : "0.1.2", "Z" : "git://github.com/some github user/Z.git" }, "license": "MIT" }

A much better approach, though not perfect on its own, is to use npm's built in features. In a node project, configuration options, including dependencies, can be specified in a package.json file, as shown above. Dependencies can be saved as npm module names, npm module names and version numbers, or as git links. If npm install is run, it will examine the package.json file and attempt to install any dependencies listed that are not already installed to the node_modules directory. In this case, it will install modules A, B, and Z. Additionally, npm rebuild can be run to recompile any native modules, which is useful (and necessary) after a node version update.

There are, however, two major drawbacks to just using npm and package.json. The first is that the system it is run on will need access to whatever servers are hosting the modules, whether it is the npm servers, github, or somewhere else. This may not be possible under certain security situations.

The second is that, while the dependency versions can be specified in package.json, and npm install will install those versions, this does not guarantee consistent subdependencies, as shown in the diagram below.

Installed on Jan 1

node_modules |-- A @ 2.5.0 (installs 2.5.0) | `-- C @ >= 1.0.0 (installs 1.0.1) `-- B @ 0.1.2 (installs 0.1.2) |-- D @ * (installs 2.4.1) `-- E @ 1.5.1 (installs 1.5.1)

Installed on Dec 1

node_modules |-- A @ 2.5.0 (installs 2.5.0) | `-- C @ >= 1.0.0 (installs 2.1.1) `-- B @ 0.1.2 (installs 0.1.2) |-- D @ * (installs 3.0.7) `-- E @ 1.5.1 (installs 1.5.1)

Each dependency module will have its own package.json, which often does not specify versions for all of its dependencies, meaning the latest versions will be installed. One can control the version of A and B, but not C, D, and E in the example. It is possible that the subdependencies installed are a new enough version that they are no longer compatible with the dependency versions specified in the main package.json or behave slightly differently. This can lead to some nasty surprises. Enter shrinkwrap.

Shrinkwrap is a way to lock down the entire dependency tree. After the base package.json is built and modules installed, running npm shrinkwrap will create a file called npm-shrinkwrap.json. This file will contain the exact version of every dependency and subdependency in the tree. When npm install is run again, it will check both package.json and npm-shrinkwrap.json, using the latter to install the specific versions of everything. The shrinkwrap file can be checked in to the repository, along with package.json, and one can be confident that modules will be consistent across systems.

There are still some caveats, however. It does not solve the issue of requiring access to the dependency hosting. The package.json, npm-shrinkwrap.json, and installed modules (node_modules) must be kept in sync. If there is something in node_modules that isn't listed in package.json, or one is missing, or the versions are different, then shrinkwrapping will fail. This means that all modules in node_modules must be handled through the package.json; any modules to be loaded outside that process need to be kept separate, which must be remembered when requiring modules in the code. When versions are changed in package.json and node_modules, a new shrinkwrap must not be forgotten. None of these are particularly problematic, but are extra considerations when working with the codebase.

Using the outlined methods to combine npm's package.json dependencies with shrinkwrap's version lockdown, we have found that our deploys have become smoother, and less hair has been lost when dealing with Node.js.

]]> http://omniti.com/seeds/broken-windows http://omniti.com/seeds/broken-windows Wed, 30 Jan 2013 19:12:10 GMT One of the tools we use when developing production web services is a script that runs nightly to compile a list of any errors in the webservers' error logs, and email anything abnormal to the developer list. This often helps us to catch broken links, uninitialized value warnings, and code paths that we may have missed in testing. Over the years, we've seen our fair share of issues come and go, but on one particular morning, I opened the email, and was greeted with the following:

sh: York: not found

Now, this is quite possibly the greatest line I've ever seen in an error log. It's four simple words, it makes no sense, it communicates that arbitrary shell commands are being executed somewhere, and there is absolutely no way to search the codebase for it. It's a beautiful nightmare in its simplicity.

After several hours and three smashed keyboards, we found the offending code. It was a mess. The page had started its life as a tangled web of hacks, and every modification added more on top. Eventually a second copy of the page had been created, and modified to work in a slightly different manner. When the second page failed to find the necessary data, it fetched the data from the first page--using a command line call to curl, with the "state" argument passed to the command line. So any two-word state would attempt to execute the second word as a separate command, hence "York: not found" when the Empire State is selected.

Consider a building with a few broken windows. If the windows are not repaired, the tendency is for vandals to break a few more windows. Eventually, they may even break into the building, and if it's unoccupied, perhaps become squatters or light fires inside. Or consider a sidewalk. Some litter accumulates. Soon, more litter accumulates. Eventually, people even start leaving bags of trash from take-out restaurants there or breaking into cars.¹

Now, consider a project with poorly written, unreadable code. Variables have names like $foo and $bar, spacing and indentation styles vary every few lines, and large blocks of code are repeated multiple times throughout. It is obvious that some of the functions don't work, but as far as anyone knows, they never get called anyway. There could even be a few goto statements hidden away in the codebase’s dark alleys. Everything just looks like a previous developer came along and vomited code into the repository. In the absence of directed code refactoring, the tendency is for developers to ignore the need to write elegant code and continue to write in the same style as the rest of the project. If nobody else cares, why should they? They continue to leave the code unindented. They use short, unintelligible variable names. They continue copying and pasting blocks of code, instead of creating function calls. Comments are completely out of the question.

Eventually, some minor bugs are discovered. In the spirit of the project, they're fixed with yet another hack. Since nobody is really sure what most of the code does, each new revision creates a few more bugs. There are innumerable potential security exploits. Because all the code is so entangled, it's impossible to refactor in small pieces. Nobody is sure what the original purpose of the code is, or if it does what it was supposed to do in the first place. Some people will argue that, as long as it works, who cares what it looks like? The users can't see the code, so it doesn't matter to them if there are five different indentation styles used between nearly identical functions that have been copied around the codebase at random. There are plenty of arguments against this mentality that I won't repeat yet again, but there is one downside to writing bad code I want to stress: it leads to more bad code being piled on top, even by developers who wouldn't normally do that. And after enough garbage accumulates, someone sets it on fire.

The developer who made the changes in the earlier example wrote good code when working on modules that were better organized. But, in this situation, he added to the pile of garbage and that lead to a potential exploit being pushed live. He thought to himself, “well, there are broken windows here, I guess I might as well commit a crime." And this certainly was a crime.

Yes, the need to use “best practices" must be balanced against deadlines and business needs, but once you've developed good coding habits for yourself, it doesn't usually take much longer to do things right. And it may save you the headache of fixing a problem caused by someone else who assumed that nobody cared.

1 James Q. Wilson and George L. Kelling. "BROKEN WINDOWS: The police and neighborhood safety" Retrieved 2012-05-16.

]]> http://omniti.com/seeds/seeds-our-experiences-with-chef-adoption-challenges http://omniti.com/seeds/seeds-our-experiences-with-chef-adoption-challenges Wed, 09 Jan 2013 19:56:38 GMT The chef system deployment and configuration tool from OpsCode is a powerful, flexible tool.  While OpsCode’s documentation and marketing tries to focus on its ease of use, they provide little guidance on patterns of real-world use. Many different organizations are experimenting with ways of using Chef. While it’s still a bit early for best practices to emerge, collecting experiences across a variety of business types can help to discover what works and what doesn’t.

Intended Audience

This is not a Chef tutorial, nor a journal of one person’s experience with Chef over a month. You will not learn what a resource is or enough Ruby for Chef. Instead, we’re assuming you have used Chef enough to know what it is, what it does, and at least one way to do what you want to do. You may have authored several cookbooks for your organization’s internal use, and taken slightly different approaches as you learned what works and what doesn’t over time.

Our Multi-Tenant Bootstrapper

Opscode offers several ways of using Chef. Most of them revolve around using a central store of cookbooks and configuration information, known as the Chef server. Chef server includes powerful cross-node search features. Thus, any node can query any other node for information, such as installed software and (depending on storage mechanism) credentials to various services.

Because OmniTI is a consulting company providing a range of development, managed services and hosting offerings, we must be able to isolate client assets from each other. Every tool we use must have excellent multi-tenancy support. At the same time, we have many tools and configuration data that we must share across clients that are specific to our environment. We want to share not only cookbooks, but roles and some data bags; and have fine-grained control over which nodes may access each.

There are essentially two options for running Chef Server. One of those is Hosted Chef, an Opscode IAAS offering, includes access control mechanisms. Cookbooks and node information are bundled into Organizations and cross-organization access is not permitted. Cookbook re-use is permitted via cookbook dependency resolution, but role and data bag re-use is not possible.

The open-source version of Chef-server provides no access control mechanisms, making it an unacceptable multi-tenant solution for security reasons.  

The alternatives were:

1. Invest in developing an ACL system for the open source release of Chef Server. Unlike the Hosted Chef ACL system, the proposed functionality would include inter-organization access to data bags, cookbooks and roles, with ACL enforcement. This would introduce a high likelihood of incompatible changes as the chef-server codebase develops.
2. Run multiple, single-tenant chef-server instances. We still have to keep copies of shared roles, databags and cookbooks on each Chef server, not to mention maintain them all.
3. Use chef-solo. Since we now cannot rely on chef-server to distribute cookbooks, nodes, data bags and roles, we must invent a mechanism to distribute that data securely. Because we already had this material in git repos, with appropriate ACL mechanisms, this was a fairly low-cost option. We pay a price in complexity and learning curve for new projects, because our custom bootstrapping code is peculiar to our organization and is only used at project startup time.

For the time being we are using the third option, but we may push that complexity back into the Chef server and run single-tenant Chef servers with some additional magic (without going so far as to patch chef-server itself). You can get the chef-solo-helper toolkit here.

For more on this topic, see threads here.

Support Varying Levels of Expertise

Deployment touches on a lot of stakeholders. Each group has different skill sets and experience levels. You’ll need to use an approach that supports a variety of uses, and presents a smooth, shallow learning curve.

We’ve had generally good outcomes thinking of our Chef users in four general camps:

Black-box users. Don’t care how Chef works internally at all; want to be able to work entirely in attributes. The default behavior of the recipes is good enough, and any tweaking should be done through attribute settings. May need support determining how roles should be decomposed, and how precedence works.

Bespoke Recipe Authors. Need to do peculiar things that no other project requires. Uses attributes, but also cookbook defaults, and thinks in terms of resources. Familiar with the broad spectrum of resources available. Knows how to template, and to fallback to bash resources where needed. Needs support to make sure that common functionality across projects gets properly formalized into a shared cookbook. They need to have access to repos for their Chef cookbooks that are private to the project.

Shared Cookbook Authors. Implements functionality around a specific software system (for example, Apache or Postgres). Often a topic expert on that system, or works closely with someone who is an expert. Intermediate Ruby knowledge. Defines an attribute space, then creates LWRPs and libraries to create multiple layers of reusable, generalized functionality.

Bootstrappers. In our environment, our Ops team is responsible for provisioning the computing environments (be they Solaris-style Zones, VMs or cloud instances). These people deal with the bootstrapping process, in which the chef-solo (or chef-client) software is installed, and the machine’s Chef configuration is a checkout of version control using our multi-repo chef-solo-helper tool. Their perspective is broad and shallow; they will have limited familiarity with the custom cookbooks of any one project, but deep familiarity with shared services like LDAP and network configuration. They want the VCS access control mechanisms to be straightforward.

Version Control Systems and Authorization

We need to store the Chef configuration data in a version control system, because it is important work that must be tracked and versioned. Some of this data is shared (because it is common to our hosting environment, regardless of client/project) while some of it is proprietary information (the particulars of how a certain client runs their web stack; credentials to access the source code of their custom-developed proprietary web application). Thus, we must be able to have flexible, but secure access to the various repos.

As a development services company, we pride ourselves on our flexibility. Some clients have an internally-hosted VCS system; some want us to host the VCS system, but dictate the choice of VCS; other leave it entirely up to us. When it comes to authentication and authorization, we have a similar situation. We try to keep as much in our internal LDAP system as possible, but still must accommodate other solutions, for historical and political reasons.

Flexibility is the enemy of standardization, and automation thrives in highly standardized environments. These disparate VCS and authentication systems are friction points, especially to the users who bootstrap systems. Anyone who works across projects may find themselves dealing with a missing SSH key one day, or a bad htpasswd entry the next. Realizing that “oh yeah this project uses authentication system X which I have to debug using procedure Y” is a drag on the process, and increases ramp-up time each time people switch projects. We have had some success fighting the good fight: re-examining our reasons for using an oddball approach; automating where possible; documenting everywhere. We’ve also established standard operating procedures for new projects, and only deviate from that at client demand, communicating the increased cost back to them.

Address Fears of Templating

Another major hurdle to adoption of configuration management is a fear of loss of control over configuration files. Of course, the control over config files isn’t lost, but rather shifted. Prior to Chef, most projects managed config files using version control. Two fiefdoms were established: “system” config files, like sudoers, resolv.conf; and “application” config files, like httpd.conf, postgresql.conf, my-custom-application-properties.conf. The system config files were typically less of a point of contention - the Ops team was more interested in having Chef manage these files directly.  

I Can’t Tell Which Files Will Be Overwritten

It can be a nasty surprise to find that a file you just edited got overwritten by Chef on its next run.  We addressed this by placing, in every template file, a large warning at the top:

  # This file is maintained by chef - LOCAL CHANGES WILL BE LOST
  # This project’s chef repo is at: <%= node[:motd][:chef_repo] %>
  # Use the project chef repo to set attributes which affect
  # the rendering of this template.
  # This template is from the dev-support chef repo, at
  #  <%= node[:motd][:chef_repo] %>
  #   in the file cookbooks/jenkins/templates/default/bash-profile.erb
  

The warning includes pointers to the correct repo and file in which to make edits properly.

I have to do hotfixes through Chef, too?

The scenario here is the classic “it’s 3 in the morning and the client is losing money; I just need this to WORK.”  So you make a manual edit to a config file, even though you know Chef will overwrite it in 30 minutes (we use chef-solo under cron).  

First, we did accommodate this by adding a killswitch option to our chef-solo-helper - /var/chef-solo/killswitch exists, the run will be aborted. We then back that up with monitoring, so we know when Chef has been manually disabled.

But in general, this is a weak complaint that gets weaker as Chef is adopted more widely in an organization. It presumes that making changes properly in Chef is somehow difficult. That is true at first, if the individual is new to Chef, or if the processes or project is unfamiliar. As a company focused on a devops approach, we expect basic Chef competency from most staff - certainly those senior enough to be the ones responding to the late-night emergency.

Additionally, this fear is only realistic on very small projects. If there are more than a handful of machines, making a manual edit is impractical and error-prone. On most of our larger projects, with dozens or hundreds of nodes, “oh no I have to make the change in Chef” becomes “thank goodness I can just make that change via Chef.”

Chef overwrites a config file completely.  I want partial edits.

This desire comes up frequently. Chef’s ‘template’ resource supports two modes: overwrite the file if missing, and overwrite the file if different. Often, we want some other process to be able to edit some part of the file, typically VCS, a web service or an OS piece.  

There are a number of ways to handle this.

1. Make sure there isn’t an existing cookbook that solves your problem. Editing hosts, resolv.conf, interfaces, and crontabs are all handled by excellent publically available cookbooks.
2. Depending on the file format being configured, you may be able to use an Include directive (or similar) to pull in additional files. You can then have one file be under Chef control, one under web service control, etc.; the master file can then be Cheffed or not.
3. Use a partial editing resource. There is a file_edit resource out there; and some people have tried various things using tools like xmlstarlet. We look at this as a last resort, as it is very sensitive to changes.

But I use VCS branches to differentiate environments!

Some projects will have a branch for dev, a branch for prod, etc. A configuration file will then exist in each branch, with the environmental differences present only in each branch.

While this may have made some sense prior to the availability of tools like Chef, we feel this is a bad practice once Chef is in use. Environments are defined by Chef - not VCS branches. It’s a much better tool for the job, because environmental differences can be captured and applied throughout the stack, not just at the application configuration level.In addition, storing environments in VCS branches means that merges must always be cherrypicked - moving dev’s edits to the conf file to the prod branch must be done carefully and manually.  

Instead, we found that a one-time merge is more useful - creating a template based on the merge of the various environment branches, then parameterizing each conflicting value. Then, environment roles are created to store the values. This makes it very clear what the differences are between the two environments, and makes it easy to provision new environments that are a mix (for example, a UAT environment).

Make it Safe to Learn

One particular challenge with Chef is that when you make a change, the work product is a complete system (or a group of systems). We had good results using Vagrant, a tool that integrates chef with VirtualBox. This allowed individuals to make changes to local chef repos, test them in Vagrant, and then push. Because no actual machines are involved, individuals may make mistakes without consequence - enabling learning.  

Two other major points about learning Chef are worth mentioning, though they will be covered in detail in later articles.  

1. When you make an attribute change in chef, you have no inherent context as to where that change will have an impact. Each organization must develop practices (naming conventions, role decomposition guidelines and good comments) that make clear the scope of a change. We’ll explore attribute tree best practices in detail in part two of this series.
2. Chef tries to turn your system configuration into code. That means you now inherit all the woes of software engineering: making changes in a coordinated manner and ensuring that changes integrate well are now an even greater concern. In part three of this series, we’ll look at applying software quality assurance and release management  practices to Chef cookbooks and roles.

Conclusion

Our experiences with Chef have been at times exhilarating (“I can’t believe that change was so easy to make!”) and at times terrifying (ditto).  Nearly all of the technical challenges were easily overcome with a bit of ingenuity - but the changes to processes and mindsets have been more disruptive, requiring perseverance and a variety of approaches.  In some cases - like the multi-tenant bootstrapper - a purely technological solution was appropriate; in others a mix of technical and process adaptations were needed.  As best practices emerge in the Chef community, we’ll continue to adapt and grow.

]]> http://omniti.com/seeds/the-way-of-the-intercepting-gantt-chart http://omniti.com/seeds/the-way-of-the-intercepting-gantt-chart Wed, 28 Nov 2012 19:02:39 GMT


Right after college, when I was starting off on my own in the world, I lived with a roommate who was an avid fan of Bruce Lee. Through osmosis I came to learn quite a bit about this intriguing pop culture icon of the 1970's - Bruce Lee, not my roommate. For those of you who may not have heard of Bruce Lee, he was a martial artist and filmmaker of some renown in the late 60's and early 70's. His life was cut short by a freak brain aneurysm at age 33. Prior to this, Bruce (as a fan, I feel I can call him Bruce) revolutionized the American view of the Asian culture through his movies, and is largely credited with reshaping the global view of martial arts. As is often the case with highly successful individuals, it was not just his skill at his craft, but his underlying philosophy that made him so extraordinary. He would have also made an awesome project manager.

In his early years, Bruce focused on one style of martial arts. However, as he progressed he became increasingly frustrated on the limits imposed by practicing just one type of discipline. His goal was to be the ultimate fighter; able to adapt and defeat any opponent no matter what the situation or the style his adversary employed. He believed he needed to develop his own style that borrowed from all existing styles. The style that Bruce developed was called Jeet Koon Do, which literally translates to "The Way of the Intercepting Fist". It's why I named this article as such and why I think he would have made an excellent project manager.

I began my work in technology consulting as an extraordinarily average software engineer (though labeling me as such truly discredits software engineers). As I progressed in my career and realized the world's systems were better served (and safer) by me managing rather than developing them, I came to realize how important adaptability is, particularly in consulting. As technology consultants our processes and procedures have to be dynamic to handle each client and each engagement. I bristle when I encounter anyone who states - "we only do [insert methodology here]" because I feel strongly that one methodology cannot possibly be the best approach for every software project. Certainly all projects have the same basic questions that should be asked - "What are the requirements?"; "Which technologies work best to meet the requirements?"; "What are the biggest risks?"; etc. But the most successful teams understand what aspects of methodologies to jettison or go lightly on, and which aspects require more focus and rigidity for the task at hand. Additionally, as consultants we sometimes face the challenge of following our basic tenets while also adapting them to fit a client's existing process.

At OmniTI, what we do involves just about every aspect of IT consulting. We have assembled professionals with skills in graphic design, software engineering, database design/administration, and hosting/network administration. We build custom applications; help customers re-imagine the look and feel of their websites; and provide hosting and scalability consulting in the areas of database and networks. Some engagements bring all those disciplines together and others require a subset of our skills. Our approach is to employ the necessary aspects of our development life cycle based on the circumstances of the engagement and the people involved. I am constantly urging our teams not to be a slave to a particular tool, process, or methodology, but rigidly enforce basic thought processes and tenets. If the client wants to use SharePoint as a document repository then we need to adapt and use SharePoint. The key is that we still have an understanding of the project definition, scope and requirements and store that in an easily accessible location. Maybe we combine all three into one document, maybe they are large enough to require 3 separate documents - I empower the individual PMs and their teams to make that call. The key is that we do them in some fashion and store them somewhere. The same goes for task management, issue tracking, quality assurance, change control and all other aspects of a project.

This approach is very much akin to the philosophy of Bruce Lee's Jeet Koon Do. Sure you have to know how to kick, how to punch, how to block, how to use nunchaku (nunchucks). But how these and other fundamentals of martial arts are employed and in what combination depends on the opponent and the environment. His teachings focus on the fundamentals and the efficient execution of those fundamentals (no process bloat for Bruce). He believed that combat was spontaneous and every aspect of an encounter could not possibly be predicted, thus the need for ultimate flexibility - something not offered by a single approach. A key aspect of Jeet Koon Do is minimal movement with maximum effect and extreme speed. In short, Bruce Lee's philosophy to his craft would have made him a project manager that actually gives us project managers a good name - not the kind that hands out a software design template to be filled out when moving data from one table to another....true story. I also believe that a roundhouse kick to the head would do wonders to control scope creep (not that I advocate violence as an acceptable method of change control, but still....).

One of my favorite quotes from Bruce sums up my philosophy for project management. It was during an interview on the Pierre Berton show in 1971 that Bruce explained the fundamental philosophy of Jeet Koon Do:

Empty your mind, be formless, shapeless - like water. Now you put water into a cup, it becomes the cup, you put water into a bottle, it becomes the bottle, you put it in a teapot, it becomes the teapot. Now water can flow or it can crash. Be water, my friend.




When you start the planning phase of that next engagement, remember - "Be water, my friend" and try to avoid the "crash".

]]> http://omniti.com/seeds/improving-developer-interviews http://omniti.com/seeds/improving-developer-interviews Thu, 01 Nov 2012 13:22:18 GMT Hiring is hard. You're taking a gamble that the person you select has the right skills to be productive, the right personality to mesh well with your team, and the right drive to make a difference. The interview process tries to bring to light as much information as possible about the candidate to help you make informed decisions, but there's only so much that can be extracted in a limited amount of time. Sadly for the traditional developer, interviews frequently devolve into tasks or tests that are, at best, proxies for the actual competencies you're trying to quantify. While development is a logical activity, seldom do developers face questions during the course of their actual job such as, "why are manhole covers round?" or, "what's the minimum number of drops it would take, given two eggs and a 100-story building, to determine the floor at which they'd break?" (c.f. this blog post for a humorous take on this.) Similarly, the typical developer does not produce code by wearing a nice suit, standing in front of a whiteboard and talking for thirty minutes, while drawing unreadable diagrams. (Maybe we'd produce better code if design was argued more, but that's a rant for a different day.)

The benefits of more directly testing the core skills of a developer are obvious: you're getting a much closer picture of their competency with the kinds of tasks you'll actually expect them to do, and from the other side of the table (putting on my engineer hat for a moment instead of the manager one), it's a heck of a lot more fun to write actual code than answer yet again the dreaded questions: "Oh, so you know Java? What's the difference between an abstract class and an interface?"

Some companies incorporate this by including programming take home assignments, but I think this is suboptimal on two levels. First, there is no guarantee that the candidate can't get inappropriate external help. Second, putting on my engineer hat again; respect my time please. If every company wanted me to grind out several hours for coding assignments, in addition to all the mechanics of getting to and from face-to-face engagements while I was looking for work, it doesn't take long before the number of interviews in a given week would suddenly become more than a full-time job. Given the current market for qualified technical staff, any individual seeking a new position will be forced to manage a tsunami of interest, so it is unwise to give them incentives to skip your HR process (like the natural inclination towards avoiding tedious busywork). You should be able to get a good feel for the core applicant/position fit in an hour at most; speaking frankly, approximately 60-70% of the candidates I see are a clear no hire within the first 5-10 minutes and that rises to 85+% by the half hour mark. Why drag it out?

So, it is a good idea to test skills directly, but how should you go about doing it efficiently? Two approaches come in very handy for this, one fairly lightweight and suitable for phone interviews, and one perhaps a bit better reserved for in-depth, face-to-face ones. Each are covered in turn in the next two sections.

Collaborative Text Editing for Phone Screens

The first approach has to do with incorporating collaborative text editing into a phone screen. This does require that the applicant be near a computer (accordingly they should be informed as such beforehand), but "being near a computer" doesn't seem like a high barrier to impose on a software developer ("being away from one" might be a bigger issue). Essentially, it is good practice to include some "FizzBuzz" type coding problems in your interview process to quickly weed out the people who, shall we say, wouldn't find much if you gave them the proverbial flashlight and free use of both hands. The literal "FizzBuzz" is probably too common now and could be regurgitated via rote memorization, but it isn't hard to come up with some sort of coding challenge that a competent person could write a solution to in at most 10 minutes. Any first year CS curriculum that publishes their problem sets would be a good source of questions like these.

The twist is that with the right sort of collaborative editing tools, both the applicant and the interviewer can watch the person write code in real time. This can be surprisingly illuminating. Do they type the solution quickly? Do they sketch out some example outputs for given sample inputs to a function for testing? These are all good signs. Speaking for myself, I don't tend to penalize applicants for minor syntax errors, but it is also a good sign if they produce perfect code, of course (e.g., remembering semicolons, indenting appropriately, etc.). If you ask for something C-like and they forget both braces and semicolons in every instance where they would be required, you can safely raise an eyebrow at any claims of being "detail-oriented" on their resume.

Two free tools that are available and should work across most any platform capable of running a reasonably modern web browser: Google Docs and collabedit.com. The biggest challenge with collabedit, surprisingly enough, has proven to be communicating the URL over the phone for whatever reason, so including it in the initial stage of the process in an email might work better. Creating a new Google Docs word processor document for each applicant is slightly more heavyweight in the sense that it's more than the one button click required for collabedit, but it's also easier to save them in a collection for future reference if need be that way (they'll auto-save so you only have to create them in the collection to begin with). It helps to have problem definitions pre-typed along with any supporting data so that they can be quickly pasted into the editing interface during the interview.

A third option for collaborative text editing that is specific to Unix-based environments is a shared screen or tmux session and a command line editor (e.g., vim, emacs, nano, et al.). This is covered in more detail in the next section.

Unix Tools for Maintaining Interview Environments

Of course, collaborative editing in a text editor doesn't have the same level of fidelity to actual work responsibilities as a full, live working environment. This section addresses that with the goal of getting a candidate as close as possible to the sort of day-to-day environment they'd be using. Although I try to keep many of my process-oriented articles as technology agnostic as possible, this section is written with the assumption that you are a Unix-based shop and that the majority of your work occurs on the command line. This is a natural fit, in the web development space, for most of the LAMP or LAMP-like shops that use some form of scripting language as their main server-side application development platform. Organizations that rely more heavily on IDEs (e.g., companies who have decided to base their work around the Microsoft toolchain) could still use some of the techniques listed here, but would need to adapt them.

It is handy to have a dedicated "interviewee" or "candidate" account. If your company is commonly interviewing many candidates in parallel, these may require numbering (e.g., candidate1 ... candidateN) and scheduling/coordination much like physical conference rooms for an in-person interview. Depending upon your organization's security requirements, these could reside on a shared development machine or on a dedicated host. (Note that, of course, with the prevalence of virtualization technologies, that "machine" and "host" could just as easily be a virtual machine running somewhere on your network.) For the typical interview usage profile, resource needs for a dedicated environment would likely be very modest unless your technology stack happens to require something extraordinarily resource hungry. Even a low-end developer's workstation (as of this writing in 2012) could easily handle spinning up a 32 bit linux install with say half a gigabyte of ram in VirtualBox and be perfectly adequate. (NB: you'd have to set up port forwarding to connect a port externally visible to the running VM to allow ssh access.)

For face-to-face interviews, the practicum environment described above is obviously useful when combined with a computer and projector. Speaking from personal experience, nothing quite so adequately simulates the need to work under pressure in a real environment like writing a solution with an interviewer sitting at each shoulder watching every keystroke. This is easy to pair with the traditional "sketch your design on a whiteboard" approach, allowing you to get a more full picture of the candidate's skills in both design and implementation. It may seem like overkill to have a pre-set interview environment when any developer workstation might do, but it is useful to have a standardized, pre-configured place for candidates to work both to save time (there is no requirement to try to figure out where to have them work or what to work on) and to create a level playing field for all applicants.

For remote interviews (whether for briefer format phone screens or more in-depth second stage interviews that can't be conducted in person), it is possible to use terminal multiplexing tools like GNU screen or tmux to set up collaborative viewing of a terminal session. For GNU screen, the procedure is:

1. chmod +s /usr/bin/screen (assuming that is the path to your screen binary)
2. chmod 755 /var/run/screen
3. add the following lines to the candidate account's .screenrc:
   • multiuser on
   • acladd {viewing username} (once for each viewing account)
4. (from the candidate account, e.g. automatically via shell init script on login): screen -S interview (or some other fixed name, as desired)
5. (from the allowed observing account) screen -x {candidate account name}/interview

For tmux, the process is simpler:

1. (from the candidate account, e.g. automatically via shell init script on login): tmux -S /tmp/interview (or other convenient name for the socket, as desired; note that this is not deleted when the program exits)
2. sudo tmux -S /tmp/interview attach (optionally, you could also chmod 777 the socket and anyone could do the above without needing sudo)

Note that in both the screen and tmux use cases, the observing user can send input events to the shell as well (e.g., moving the cursor to a given line to highlight a region of code in order to ask questions about it).

Now that the candidate has a place to log in and work during the interview (and a way for their work to be observed by interviewers), it's important to spend a few words talking about what they'll be working with, and on. None of this is vital, but all of it will help make life smoother for both you and your interviewees.

First, as a matter of kindness, the shell environments for the candidate account should be reasonably set up: a complete usable PATH environment variable, reasonable editor configuration, etc. You don't have to go overboard customizing these, but if you're giving an interview testing (for example) perl knowledge, making the candidate find the perl binary on your system or figuring out which one is appropriate ("usr bin? opt nytdbin? usr local bin?") is probably a bit excessive. It is helpful for coordination purposes to set up editors, if possible, to display line numbers for easy reference, e.g. "set nu" and "set ruler" in the interview account's .vimrc. Shell initialization scripts may also be useful to ensure that the working environment is clean (e.g. cleaning out standard account-specific tmp or data directories that could be modified during the course of an interview), either on login or logout as appropriate. Similarly if the interview involves modifying other resources like a database system, restoration to a known good condition as a part of that process would be advisable.

Second, it is extremely useful to keep an interview problem set and all associated working data in source control. This could include program fragments to debug, data files to parse and report upon (e.g., a csv), SQL files to apply to an interview database, and so forth. Retrieval of these resources from source control could be automated as part of the environment setup/cleanup shell initialization script work mentioned above. The source control system used is not terribly important as the likely working set size for interview data is a few tens of files at most and branching/merging/other advanced use cases are not likely.

Conclusion

So, putting all of the pieces together, you'll be able to do things like asking a developer candidate to perform tasks like ssh to a Unix account, use command line tools to debug a program, write a new one, develop a quick and dirty report about a data file, write an ETL script to dump a database table to csv or whip up a quick RESTful web service, and more. Doesn't that sound a bit closer to what you hope to pay them to do every day rather than figuring out how many races you have to conduct with N horses of group size M to find the fastest one?

]]> http://omniti.com/seeds/an-introduction-to-terminal-multiplexing-with-gnu-screen http://omniti.com/seeds/an-introduction-to-terminal-multiplexing-with-gnu-screen Wed, 31 Oct 2012 00:00:00 GMT What is "terminal multiplexing" and why should I, J. Random Hacker, care?

Have you ever had your network connection die in the middle of a long-running job or edit session, causing you to lose work? Have you ever thought "OK, I have 50 xterms open right now for five different projects, how do I group these to make some sense out of this mess?" Have you ever wished you could walk away from your workstation in one place and seamlessly resume all of that project state elsewhere (e.g., for those late night "Aha!" moments, long after you've gone home from work)? If the answer to any of those questions is "yes," terminal multiplexing might make your life much easier.

A terminal multiplexer is, in essence, a program that transparently "stands between" your active terminal connection and N-number of spawned terminal sessions. See the figure below for a "before" and "after" of how a given user is interacting with a server (assuming SSH for remote connection):



Imagine in the diagram above that the network (represented by the vertical dotted line) suddenly went away. Maybe the wifi died in the coffee shop, maybe a backhoe found your network cable, maybe your laptop ran out of power and shut down on you, whatever. The user in the "before" scenario just lost their active connection to each of their shell sessions, something to which many programs do not respond well, and likely lost at least some amount of work in addition to the bother of having to restore the state of their project work ("Hmm, I was editing foo.c, and bar.pl, and I had a psql session going to bazdb. . ."). Feel free at this juncture to doodle a little cloud of steam coming out of the poor fellow's ears.

The user in the second scenario is in a much better situation however. Since the terminal multiplexer is running on the server end of the connection and managing all of the spawned sub-sessions, no state has been lost. All of his programs (editors, database shells, whatever) are still happily running with no idea than an interruption has occurred. So, once the network is restored, all the second user has to do is re-login to the server and resume his terminal multiplexer session. (An important note here: naturally if something happens to the server itself, the terminal multiplexer can't help you.)

The user in the first scenario, even under normal conditions, must maintain N-number of terminal windows open. For one or two concurrent sessions, this is not a big deal. However, this quickly becomes disorganized as the need for concurrent sessions increases (imagine a web developer working with an editor, a watched log file and a database shell related to each of four projects; at a minimum this would be 12 terminal windows). The second user need only have one window for all of their sessions (e.g., they could choose to have one terminal multiplexed 12 ways, or four terminals multiplexed three ways, to continue the previous example). This "bundling" of persistent project state into one window radically cuts down on the number of terminal sessions to keep track of during a busy work day.

Combining the prior two paragraphs yields perhaps the nicest day-to-day use for terminal multiplexers: portable project state. Since all you need to do to resume your project work exactly where you left of is log into the server and resume your terminal multiplexer session, you are not tied to any particular client machine or location. The user in the second scenario could walk away from their workstation at work, have an epiphany over dinner at home, and (via something like a VPN, ssh, or similar) log back into the central server to continue their work. The first user, if they wanted to work remotely, would have to try to recreate all of their project state by logging in and starting programs anew in separate sessions (hoping that no bad behavior would be observed, e.g., editors fighting over modifying the same open files).

There are many other potential uses of terminal multiplexing, this article attempts only to scratch the surface with the above scenarios. For example, it is possible to configure many such programs to allow more than one user to view the same session, allowing for things like remote pair programming and observed practical technical interviews. These uses are left as exercises for the reader as they lie beyond the scope of this introductory article.

GNU Screen: Open Source Terminal Multiplexing

GNU screen is an open source program that provides terminal multiplexing facilities. It is available for many Unix and Unix-like platforms. The GNU screen homepage includes a comprehensive user manual along with download links for the source itself (though it is available pre-packaged for many operating systems; check with your vendor's package repositories and/or Google for " gnu screen package"). This article attempts to give a tutorial on GNU screen usage in light of the overview presented in the first section (in other words, how to achieve all of those good things with screen). In the process of evangelizing about the virtues of using screen, I have, in the past, managed to condense this information onto Post-It notes and once even into a 140 character tweet. I'll try to elaborate a bit more, but at its core, the day-to-day operation of screen is not complicated.

GNU screen is invoked from the command line by entering "screen" at your shell prompt. By default, screen will show a bit of information about itself, and when you press any key, presents you with a shell session (e.g. bash). From this point forward, interaction with the running screen is done by means of meta-character input (in other words a specific key sequence alerting screen that what is to follow is a screen command, and not regular input to be passed through to the sub-process you are interacting with). The default command meta-character is ctrl-a (that is, pressing the control and A simultaneously). A table of common GNU screen commands is shown here (with "ctrl-a" abbreviated as "c-a", and "c-a x" means press control and a simultaneously, release, then press x); there are many other features but this is enough to start:

GNU screen Command	Action
c-a c	"C"reate a new sub-process window (defaults to new shell session)
c-a n	Switch to "n"ext window in the circular list of windows 0 .. (N-1)
c-a p	Switch to "p"revious window in the circular list of windows
c-a (a number)	Switch to window number (a number)
c-a A (shift-a)	Retitles this window (default is running process name e.g. bash)
c-a " (shift ')	Present list of windows w/ titles, allowing selection via arrow/enter
c-a d	"D"etach from the running screen session (see below)
c-a ?	Display a list of commands and other help

Detaching and resuming form the remainder of the core functionality of GNU screen. Once you have detached from GNU screen (or if your network dies, or your client machine does), it remains running on the server. You can see a list of your running screen sessions by entering "screen -ls" (this prints out all running screens and their names, if any, and exits, rather than starting a screen session). To resume a GNU screen session that is already running, enter "screen -dr" (detach running session and reattach). There are other methods of increasing sophistication that can be used with these options, consult the man page for more information. Typically I use -dR always as that will detach and resume, or create if one is not present, so that I only have to enter "screen -dR" (actually, I tend to use "screen -dRAU" as that enables Unicode support and Adapts all sub-process windows to be the same size as the terminal window I am currently using; as this is a bit much to type every time I set up a shell alias like, 'alias sr="screen -dRAU"' ["s"creen "r"esume] to reduce typing). With detaching and reattaching, you could stop working on something, fly to a different continent, log back in and be right where you left off.

One wrinkle that sometimes arises during use of GNU screen is terminal settings, and the TERM environment variable in particular. The TERM environment variable is essentially how running programs (e.g., editors, database shells, less/more) know what kind of terminal you are using so that they can emit the appropriate output format. GNU screen by default causes the TERM environment variable to be set to "screen", which on some operating systems and in some environments is not a recognized value. Commonly this manifests as glitchy display from things like full screen editors, less/more type paging programs warning that "X is not a recognized terminal type" or similar. This can be enough of a complicated question in terms of what the right value should be and how it is set that if you run into this, it probably pays to turn to your local systems administrator for help. For example, on Solaris systems, I sometimes have to do things like "export TERM=dtterm" in bash to get sensible behavior from things running under GNU screen. This issue tends to be less commonly encountered on the core free Unix operating systems like Linux and the BSDs, in my experience.

tmux: An Open Source Competitor to GNU Screen

Open source is nothing if not full of options, and terminal multiplexing is no exception. A relatively recent addition to the field options, tmux is a BSD-licensed terminal multiplexing program. In terms of feature sets it is broadly comparable to the older GNU screen, with the first visible difference being that the command meta-character is control-b rather than control-a. The tmux homepage has extensive documentation and references for further usage. In terms of benefits, power users may find that tmux is easier to configure than GNU screen and it makes it easier to make named groups of windows/programs that should start when it does (e.g., allowing you to define a set of programs to automatically start up for a given project name, for instance a shell in the right root directory of the source code, a database shell pointing to the right database). There are other differences, but they all lie in the realm of uses more advanced than what would be within scope for this article. The downside, of course, to a newer program is that it is less commonly installed and may not be present in all operating systems' precompiled package repositories.

Conclusion

Terminal multiplexing may sound arcane at first blush, but the theory and practical use of it are fairly straightforward as illustrated above, and I believe will make life considerably nicer and more productive for developers working in command-line oriented environments. This article only scratches the surface of the power available through GNU screen and tmux; a great deal of flexibility and configurability is available for the motivated power user.

]]> http://omniti.com/seeds/javascript-tips-for-non-specialists http://omniti.com/seeds/javascript-tips-for-non-specialists Thu, 23 Aug 2012 20:44:23 GMT Javascript coding is a rather unique skill in the web development world—if you are not experienced with writing Javascript for the browser (especially if you are a coder who works primarily with other languages such as Perl or PHP), it may seem that the language has a mind of its own, refusing to behave in the way a language should. This, combined with the myriad Javascript tutorials on the web which teach sub-par coding practices, makes improving your Javascript skills a daunting task. It's also something that developers usually don't have much time for if they only write Javascript occasionally. All this makes the language appear as an enigma, wrapped in a riddle, nestled inside a sesame seed bun of mystery.

When writing this article, I wanted to provide some tips for non-specialists about ways to make your Javascript faster and easier to maintain, and in some cases, easier on the end user. If you're primarily a Javascript/UI/UX/front-end developer then I hope these concepts are old hat, but for all the other developers out there: this is for you.

Just a quick note: I don't assume jQuery usage throughout this article, but I will reference it from time to time since it's widely used and is a good tool for smoothing out cross-browser differences.

1. Understand Variable Scope and Initialization

Many programmers with backgrounds in other languages have a hard time understanding how variable scope works in Javascript. It's actually very simple; it just seems tricky because it's different from most other languages. There are two things to remember:

• The only variable scope is the function you're in. Period. Of course, there's always the global scope, but there is no block scope for variables…only the function scope.
   
• Each function has a "scope chain" containing parent functions' scopes. If a variable or function is used but not declared in the current scope, each link back up the chain is examined, one at a time, until a declaration is found. If you're trying to assign a value to a variable but no declaration is found, then a property of the same name is set on the global scope, since the global scope is the top link of every scope chain. Note: a new link is formed in the scope chain when a function is declared inside another function.

Now that we know that there is only the function scope (but that it has access to parent functions' scopes via its scope chain), we need to know how functions are initialized to fully understand why variables behave the way they do. Each function (as well as the global scope) is initialized in the following manner:

1. Function parameters are declared and values set. Any parameters passed to a function are declared first, with values set.
    
2. Functions are declared. Next, we declare any sub-functions (remember: they will have the current scope as a higher link in their scope chain).
    
3. Variables are declared. All variables that are declared anywhere in the current function's scope are now declared (with undefined values), before procedural code is evaluated. This leads to what's commonly known as "variable hoisting," as variables declared at the end of a function are actually available anywhere in the function. (Note: if a variable has the same name as a function, it's not declared twice, but simply references the function until another value is set in step 4.)
    
4. Variable values are set as procedural code is evaluated. Last, the procedural code in the current scope is evaluated, giving variables their values, etc.

I hope this simplified view gives you insight as to why Javascript variables act the way they do. For a more technical look into how Javascript works, check out the writings of Dmitry Soshnikov on the ECMA-262-3 specification. Now let's move on to some more practical tips.

2. Don't Write Inline Javascript

This one is simple, but I'm including it because I still see it happening. Behavioral Separation is a guiding principle for all front-end developers, and it should be the philosophy of anyone who writes Javascript, even if only occasionally. Javascript is the "behavior layer" of a web site/application and should not be mixed with the HTML "content layer." Thus you should never write:

<button id="my_btn" onclick="doThis();">Submit</button>

Having Javascript inline like this makes maintenance horrendously difficult: there's no single place to look and see all the scripted actions on the page. When a new developer comes onto the project it takes them a long time to search through the markup, find the Javascript, and learn how everything works. And that's not even considering more general maintenance efforts; maintaining inline Javascript simply takes more work than maintaining Javascript which has been kept separate.

If a page needs to initialize page-specific event bindings, it probably needs a page initialization function to be called at the bottom of the page. Thus we only have to look up that initialization function to view all of the Javascript actions and bindings for the entire page. Here is the same button action as above, bound properly from a page initialization function:

<button type="button" id="my_btn">Submit</button>

<!-- These scripts go before the </body> tag. -->
<script type="text/javascript" src="/s/init.js"></script>
<script type="text/javascript">
    initializePage();
    // pretend that these functions are in /s/init.js
    function initializePage() {
        document.getElementById("my_btn")
            .addEventListener("click", doThis);
    }
    function doThis(e) {
        // do cool stuff here
    }
</script>

Just a quick note: the addEventListener() method isn't available in Internet Explorer prior to version 9 (old versions of IE use proprietary methods), so if backwards compatibility is important, you may want to use a library like jQuery to make your cross-browser event bindings easier (requiring just one method call for all browsers including legacy IE).

3. Use Intentional Event Bindings

Since we don't want to use inline event triggers, we have to trigger events with event bindings. With or without libraries like jQuery, it's easy to bind your event handlers with a simple line of code:

// vanilla JS event binding
document.getElementById("my_btn")
    .addEventListener("click", doThis);

// jQuery event binding
$("#my_btn").bind("click", doThis);

Before we bind, though, we need to stop and consider what we intend to make happen. I've seen a lot of "click" event bindings on form submit buttons, when that's not actually what the developer intends. The developer actually wants the event handler to be triggered when the form is submitted, and they mentally associate the submit button with that action. But what if the user is editing a text input and hits "enter?" That would submit the form and completely bypass the submit button binding. Also, what if you want to submit the form via Javascript? It wouldn't make sense to trigger a fake click on the submit button; it's far better to trigger the submission of the form directly. So in this case, since the developer actually intends to capture the form's "submit" event, the submit button shouldn't get a "click" binding. Instead, the form should get a "submit" event binding, which can be triggered in different ways.

4. Use Fast Selectors

There are several different ways to select elements on the page: getElementById, getElementsByClassName, getElementsByTagName, querySelector, and querySelectorAll are the most common (if you're not using jQuery). If you're only dealing with a few elements on the page, then it's not very important how you select your elements. In this case, feel free to use querySelector and querySelectorAll as long as you don't need IE7 compatibility. But if you're manipulating an unknown number of elements that could reach into the hundreds or thousands (or if you need IE7 compatibility), you'll want to be sure to use fast methods, which are usually the oldest and most cross-browser compatible. In this case, I recommend using getElementById, which is supported by all browsers and is the fastest method available (it's an order of magnitude faster than the generic querySelector methods which let you use CSS syntax to select elements).

If you're using jQuery for your selection work, it's still best to use IDs and classes to get the best performance. Many selectors (such as "input[name=first_name]") require string matching on attribute values or other performance-killing techniques, so stick with ids and classnames for your selectors as much as possible…it'll help you stay out of speed traps.

5. Don't Touch the Doc (Unless You Have To)

This is perhaps the most important tip to remember for maximizing your Javascript performance: don't touch the document unless you absolutely have to do so. This is the biggest bottleneck in most Javascript-heavy sites/applications. Your application could be screaming along faster than any app in history, but when you touch the document it's like hitting a traffic jam (especially if you are manipulating hundreds of elements or more).

This is where the DRY mantra, "Don't Repeat Yourself" comes in handy. The best way to minimize document-contact is to save your element references; the first time you use an element, save the element (or jQuery object) in a variable so you'll be able to use it again later without having to touch the document to find it. If you use jQuery, a common pattern for variable naming is to use the "$" character to prefix the names of variables that reference jQuery objects (e.g. $my_button), that way you can tell them apart from other types of variables.

Another important way to minimize contact with the document (thus maximizing performance) is to save attribute values that will be used repeatedly. So instead of this:

var name = document.getElementById("first_name");
if (name.value && ("Chuck" !== name.value)) {
    doThis(name.value);
}

We write this:

var name = document.getElementById("first_name"),
    name_val = name.value;
if (name_val && ("Chuck" !== name_val)) {
    doThis(name_val);
}

Bonus Tip: Equivalency and Truthy/Falsey

Here's an extra tip that will make your life easier when writing "if" statements. When dealing with equivalency tests in a language, you have to know how it tests variables and how to use that to your advantage. In addition to standard equivalency (==, !=) Javascript also has strict equivalency (===, !==). To illustrate the difference, here are some examples:

• "5" == 5 but "5" !== 5
• 0 == false but 0 !== false
• null == undefined but null !== undefined

Although Javascript has actual boolean true and false values, all non-boolean variables have inherent truthy/falsey values. For example, all objects are truthy, so any object, array, or function will always give truthy evaluations (i.e. if (my_obj) ). In addition to 0, null, and undefined being falsey, empty strings are also falsey. So instead of writing if (my_str !== "") or if (my_str.length), you can just write if (my_str). (Note: all non-empty strings are truthy, even "0" despite the fact that "0" == 0 and 0 is falsey!)

In Conclusion

Even if you only write Javascript occasionally, it's still a good practice to be able to write code that will perform well and not place a huge maintenance burden upon future developers. Hopefully, these tips will also help you understand Javascript that others have written, so you'll feel more prepared if you need to debug (or just understand) an existing script. If you don't write Javascript every day—or even every week—you may not need to become a black-belt Javascript ninja, but at least you can feel confident in your code.

]]> http://omniti.com/seeds/why-omnios http://omniti.com/seeds/why-omnios Thu, 02 Aug 2012 20:03:52 GMT Since the release of OmniOS last spring, we've received a lot of feedback from people through email, twitter and at various conferences we've spoken at. Most of this feedback has been very positive: people are happy to see the OpenSolaris family continue on, and are excited to have more options for working with DTrace, ZFS, zones and some of the other exceptional tools available on this platform. Of course, not everyone understands why we did what we did; after all, maintaining your own distribution is not a trivial task. While we've had the chance to talk about what went into our decision making process in person at meetups and other events, we thought it might be worth a look back here to explain some of the thinking that went into creating OmniOS.

We should start by saying that we've never been shy about building things. Whether it's a complex web application or the back-end infrastructure for high scale web services, building things is at the core of what we do. When we run into the limitations of software that isn't quite designed to do what we need, we either change it or make new software that will work for us. OmniTI is a company of engineers, and we have a long history of working with Open Source projects, so taking on an operating system distribution certainly fits our mission.

One of the first things we considered was our need for having a rock solid system to serve highly scaled web services. In the distant past, we used Solaris for this, although the service model was always more limiting. The advent of OpenSolaris allowed us to build out our own hosting infrastructure, which gave us full control over our working environment. It wasn't just about stability; OpenSolaris also gave us deep introspection, which can be critical when dealing with problems at scale.

Of course, there are a lot more people building large-scale web systems these days, and a lot of them do it on other operating systems. We do that for many of our customers as well, but sometimes you get to pick your battles. In the hardware world, you pick things like ECC RAM and SSD's because you want quality tools with which to do your job. When working in web operations, you want things like system tracing and filesystem snapshots--and when it comes to those, dtrace and ZFS are still the systems to beat. And, the implementations available in OpenSolaris, and now Illumos, are the best available; so if we get the chance to use those tools, we do.

Some might argue that tools like System Tap for tracing, BTRFS or LVM, or things like ZFSONLINUX, are equivalent options on Linux. We use these tools too, and we don't agree about their equivalency. Whether it's performance, stability, or ease of use, at best these alternatives leave something to be desired, and at worst may deliver you to disaster. It's not that those problems can't be fixed; we think they probably will be and that some day what's available on Linux may (should?) become the gold standard; but that day isn't today. We think it will be at least another five years before the options on Linux really get close. Five years is a lifetime on the internet. Five years ago there was no Foursquare, no Pinterest and not even a GitHub. Even if the current incarnation of OpenSolaris (aka Illumos) made no additional advancements for five years, we're no worse off by sticking with them and if they do happen to progress, it's gravy.

Of course, rolling our own also meant we got to make sure we had a number of other key technologies:

• zones for lightweight virtualization, so we can build our own "private cloud" systems
• kvm for hardware virtualization, for when we have to run mono-theistic software
• crossbow for network virtualization, which manages a myriad of private networks within our infrastructure

Okay, so we liked the idea of running something Illumos-based; but that still doesn't explain fully why we would roll our own distro rather than just rely on the other options available. When we looked at the current state of the Illumos-based distributions available, there arose two primary factors in our decision to build our own. The first is that what we are targeting with OmniOS doesn't fit the niche of the existing family of Illumos distributions. With SmartOS, Joyent has set their sights on building the next platform for managing private (and public) cloud infrastructure. This vision takes its focus off of the current, common needs around traditional enterprise management issues and strong heterogeneity. Nexenta's version of Illumos focuses strongly on enabling their flagship storage products and, as such, has less focus on the needs of enterprise customers for a general computing platform. OpenIndiana is probably the nearest neighbor to OmniOS, however we've found the build system difficult to work with, and their targets are set differently than ours as well; they don't just want to be a server OS, but also want to provide a fully featured desktop OS too. We have no such illusions, nor can we afford to be distracted by them. ☺ For OmniOS, we want a distro aimed at a minimalist server install that can be installed on both bare metal and virtual machines. That's a problem a lot of people have, and no other Illumos distro is quite focused on solving that problem.

The second reason that we thought we would create our own distro is that we had already been doing most of the heavy lifting for years. When we ran OpenSolaris, we had a full package build system so that we could roll our own version of any software package we needed. We'd already been in the business of managing kernel patches, and dealing with security updates, driver issues and integrating patches from "upstream." In managing our hosting architecture, we were accustomed to managing our own distribution anyway. To be clear, the heavy lifting here is in managing the packages, curating the ecosystem and making sure we had a repeatable installation process for spinning up new systems. An equally important factor was the heavy lifting we didn't have to do; the core operating system was developed--nicely handled by the Illumos team. While we work with that team when we can, we also have the help of all of the aforementioned companies and many others as well.

When Oracle decided to close the future development Solaris, we had to make a choice; limping along with a rotting corpse of Open Solaris wasn't an option, we needed something that was actively developed and that we could count on for the foreseeable future. When we added things up; our technical needs for running a robust, scalable platform; the alternative choices of either converting to a more limited platform (Linux) or working on one of the new Illumos distros, plus our general desire to bring focus to an underserved user base, the decision became clear, and OmniOS was born. We're still new to this, but so far, so good.

]]> http://omniti.com/seeds/histograms-monitor-hot-spots http://omniti.com/seeds/histograms-monitor-hot-spots Thu, 12 Jul 2012 17:48:33 GMT Describing a set of data over time, with sufficient detail in a concise enough format to be useful, is no easy task. There are line graphs of counts and derivatives, where each unit of time corresponds to a single data point. With sets of values per time, however, compromises have to be made to fit the same format, consolidating information with averages, medians, minimums and maximums. Unfortunately, even with all of those calculated variables, important patterns will be lost. Histograms can save the day.

A histogram is a way of describing frequency within a data set. Values are grouped based upon ranges, as fine-tuned or widely encompassing as one desires, and then the group totals are graphed in some way. For a single set of data, the typical graph is a bar graph, with height proportional to count. If, instead of height viewed from the side, frequency counts were represented by color from above, then it becomes trivial to show multiple sets over time by stacking these colored representations together. This visualization, the histogram heatmap, is essential for monitoring.



Network problems are notoriously difficult to identify and describe. Above, is a contrived example of latency data and a corresponding graph. With a simple average graph, one would have no idea about actual latency distribution; though the sets vary wildly, the averages are more or less the same, and it would be assumed that results are consistent.



With additional maximum and minimum graphs, the picture is slightly more accurate, but one won't see the magnitude of the problem. Are they outliers or something more frequent? This risks either over- or under-reacting, solving non existent problems and ignoring fundamental issues. Disproportionate action is going to cost unnecessary time and money somewhere along the line.



Applying this to real response times using Circonus' histogram feature, the benefit becomes apparent. The graph above contains response times (ms) for calls to a specific network service over time, with the darker green representing more frequent results. The blue line is the average response time. There are some points above 500 ms, but for visual clarity, the scale has been limited. Generally, the average hovers around 30 ms, near the more prominent lower band. This completely masks the existence of the second band in the 400 ms area. Such a low average, might lead one to conclude the maximums to be outliers in the overall picture. With the histogram, however, it becomes clear that a small, but not negligible, portion of requests was fairly consistently coming in an order of magnitude above the average, indicating that something was amiss with that service. After investigation and a code change, that higher band goes away completely, confirming the fix worked.

The process for creating histograms differs, depending upon the monitoring and graphing solution you use. For the above, we used Circonus, which makes this a breeze. With Circonus, the most straightforward way is to use an http trap type check and send it messages of the JSON format:

{
  "key" : {
    "_type" : "n",
    "_value" : [1,2,3]
  }
}

Each key will correspond to a metric in the http trap check. The type is "n" for numeric. Value is an array of values, so it's both possible and encouraged to aggregate application side rather than pushing each individual value separately. After the data is flowing, enable the metric and then go and enable the histogram collection (icon of three little blocks). All that remains is to add it to a graph.

Whenever you have changing sets of data over time, histograms are the way to go. They condense information into an easily consumable graph without sacrificing detail, shedding light on any number of problems.]]> http://omniti.com/seeds/writing-readable-code http://omniti.com/seeds/writing-readable-code Wed, 16 May 2012 19:21:59 GMT

Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. Code for readability.

- Martin Golding

Many coders have run across this statement and dismissed it as a bit of folklore, but it rings true. You will almost certainly never be the only person to look at code you've written. Even if that time is many months or years down the road, you owe it to future developers to write code that is easy to read, even if the logic behind it is complex.

Inevitably you will run into people who say "who cares, my code works, that's the bottom line, right?"  Well, yes and no. Of course your code should work -- even the most beautiful code is rendered ineffective if it contains a crippling bug. Unless you're coding in a vacuum though, you or someone else will have to look at it again. If you can't understand what's going on because the code is written poorly you are costing yourself (and likely your clients) time. You also increase the chances of introducing bugs and/or syntax errors which further slow you down. Not good.

So how do we get the job done and still have readable code?

Use a Sensible Indentation Scheme

This one might seem obvious, but you should choose an indentation scheme that makes sense and stick to it. Typical schemes include hard-tabs, two- or four-space soft-tabs, or whacking the space bar a couple of times. Whichever method you choose though, be consistent. Being able to vertically scan a section of code and see where indentation levels match levels of code is extremely useful.

Maintaining consistency is easy if your editor of choice supports a macro or keybind so that indenting is reduced to a single keystroke. Hard-tabs are already done for you. If you go the soft-tabs route, set your editor to use your chosen number of spaces whenever you press Tab for maximum convenience. Indent each code block or logical function so that things visually line up.

Block Separators

Where you place block separators is entirely up to you, but again be consistent. For example, both of the following are acceptable:

while ($condition) {
    # code
}

... or ...

while ($condition)
{
   # code
}

If your language of choice allows it, you should also do this for function calls with significant numbers of arguments, such as:

my $result = super_awesome_function(
    color => 'red',
    awesome => 1,
    frogs => [ 'out', 'gonk' ]
);

You might have noticed a tendency to avoid long lines here. This is intentional. One stat often trotted out is that "stock" terminal windows are 80 characters wide, and therefore you should not go over this for a single line of code. That's a good starting point, but instead try to think in sensible chunks. If you have a long string being passed in as a function argument, then don't sweat having that wrap the terminal window. Just don't put all your arguments on one giant line. That's hard to read, and difficult to deal with.

Crack the Whip Now and Then

So you're doing your best to write consistently readable code, with a sensible indentation scheme and well laid out lines. Your project grows and a couple junior developers come on, and let's say they're not quite as consistent as you are. Consider using a style enforcement tool on your code to keep things in line.

Perltidy is a good example. JSBeautifier is handy to drop blocks of JavaScript into. Other languages have their own, and if you use a GUI to code, consider using one that auto-indents for you so you don't have to worry about it. You can get creative here too -- source control programs like SVN support commit hooks which could auto-prettify your code before check-in, or you could set up a cron job. The idea here is that once you embark down the road of maintaining a readable codebase, try your best to stick with it. Use tools to your advantage.

Use Descriptive Variable Names

Large blocks of code with variable names like, $x, $y, $thing, $o, will certainly work, but give very little insight into what's actually going on. You should use variable names that describe the type of value being stored.

So instead of these:

• $res
• $c
• $rv
• $o

Use these:

• $reservation
• $row_count
• $return_value
• $customer_object

This will make your code significantly more readable and provide in-line, semantic hints when tracing logic.

Similarly, if your language supports implicit function arguments and loop variables, avoid these. For example, in Perl, the following doesn't tell you much.

foreach (@stuff) {
    $_->do();
}

But this does:

foreach my $record (@customer_records) {
    $record->process();
}

It should always be easy to tell what's going on, and what is being acted upon, implicitly. Non-obvious variable names that are used consistently across your application are fine as long as this is well-documented. For example you might want to use $c for a database connection handle to save typing, or $u for a user object. Shortcuts are fine as long as everyone speaks the same language.

Use Natural Language for Subroutine Names

Similarly, your subroutine names should describe the actions being taken as descriptively as possible. This helps to not only segment program functionality, but to find relevant functionality when enhancements need to be made. Consider the following code:

foreach my $record (@customer_records) {
    $record->verify_eligibility();
    $record->process_enrollment();
    $record->check_for_and_process_errors();
    $record->finalize_and_archive();
}

It is very easy to follow exactly what's going on there. No functionality is hidden behind obscure subroutine names, and each action being taken on a customer record is clearly defined. If business rules change, for example, requiring additional customer data before enrollment, it's clear where something like $record->populate_secondary_customer_data() would have to go.

Making defined breaks like this also helps with proper code design -- you wouldn't put archiving code in $record->process_enrollment() for example, but you might be inclined to do so if it were simply named "process."

Write Code Comments for Non-Obvious Things

Very often, as we're in the thick of things, we write complex bits of code that seem perfectly fine, but turn out to be incomprehensible blocks of logic later. You should get into the habit of writing code comments for each logical chunk of code unless it's painfully obvious what's happening. Adding comments to your code costs nothing, but improves the readability of your code considerably. Don't go overboard though. Too many comments will clutter your code, and can actually have a more serious drawback -- teaching your colleagues to ignore your comments. It's sort of like the alarm that keeps going off and just gets acknowledged without being looked at. When developers are routinely forced to wade through lots of unhelpful comments, they'll miss the good ones. Quality over quantity is the rule of thumb here.

You should definitely comment in situations that seem obvious but have their roots in non-obvious business logic. Yes, we can see that you're adding a dollar to the transaction charge on Fridays. But why? Document that business logic; maybe the client has a specific need to do so that won't be so obvious later on.

Know When To Write Many Smaller Statements Instead of One, Huge One

This one is a particular problem in languages like Perl that make it easy to chain many things together using implicit parameters into one monolithic statement. Consider the following code.

my @result = grep { $_->{priority} > 2}
map { { priority => get_priority($_->id()), customer => $_ } }
grep { $_->last_name() =~ /^H/ } @customers;

Basically, we're looking for all customers with a last name beginning with H, who are at priority level 3 or higher and jamming this into some temporary struct. This works great -- but the following is much easier to read:

my @customers = subset_customers('last_name', 'H', @customers);
my @result = find_customers_at_priority(3, @customers);

Now you can hide the complex logic inside your utility functions, and the main program flow is much easier to follow. This is marginally slower to execute given we're storing results and making subroutine calls instead of passing through transient loop variables, but the gain in readability outstrips the loss.

Why You Should Do All of This

Writing readable code makes you a better team player. The next person who picks up your code will be able to tell what's going on, how your design flows, and will know the reasons behind any tricky bits. You'll reduce ramp-up time and the inevitable "what's going on here" questions. Consider that if the next person on your project can't figure out what's going on, they're probably going to call you and make you help. It's in your best interests then to make sure that doesn't happen.

And remember: the violent psychopath who comes along next just might be you.

]]> http://omniti.com/seeds/your-code-may-be-elegant http://omniti.com/seeds/your-code-may-be-elegant Tue, 28 Feb 2012 21:10:41 GMT

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies.
- C.A.R. Hoare, The 1980 ACM Turing Award Lecture

I often get criticized for my mantra toward the development approach. Your code may be elegant, by mine f***ing works. In response, I hear statements ranging anywhere from "You don't understand best practices" to "You hate testing!" In an effort to avoid repeating myself on a regular basis, I decided to write down my point of view on the topic. Adhere to it or not -- your choice.

First of all, let me make a blanket statement here: the sentence "the project may be late — but the code [looks better/is easier to maintain/is cleaner]" (pick any that apply) is inherently flawed. If the project is late, it's not done. Period. There is no way to justify being late by using code quality/elegance as an argument. If the client needs a Christmas promotion, and you deliver the best product in the history of promotions -- on December 29th -- it's worthless.

Now, let's address the "best practices" argument, which implies that in order to produce more maintainable code one needs to take more time. I will keep using the phrase "best practices" in quotes because standards vary across the board (despite some common misconceptions), outside of very common Best Practices 101 that every programmer should have imprinted in their brain before the first "Hello World" is written. With that said, "best practices," however you define it, should be a natural coding standard for any decent developer. If you need to bake more time into the project time-line to make your code comply - you are, at best, new to the programming scene. To give a trivialized example, any seasoned programmer should instinctively know not to call the variables $a, $b, $c, etc., and should properly indent code lines. Granted, there are more advanced "best practice" standards that one might mention, but the point stands - "best practices" is not a good excuse for exceeding your development timeline. And taking it a step further, a veteran programmer should know when and, most importantly, how to cut corners, if needed, to meet the deadline. Which brings me to my next point: over-engineering.

Like any experienced engineer, I understand the desire to build the best, most flexible and robust system for every project. I do. But I also understand the common business constraints of every project: time and money. Most projects have a definite deadline and/or a specific budget that must be met and, often times, building something grand is just not feasible within either them. This is where the developer must make a conscious decision to limit creativity to meet the goals. There is no excuse for spending a week to setup a "proper" caching layer for database queries on a 20-row table, that is only used from the administrative panel by three publishers. Understand the use cases. As cool as it may be to build a flexible and expendable XHR framework to support variable simultaneous requests; you don't need to invest in it if the only feature that will be using it is an update to a visitors counter on one page. Understand the scope. I cannot stress it enough: a good engineer is not the one who knows how to build the most advanced system, but the one who knows when not to build that system.

In the world of software development, time-to-market is the driving force of the business. In the world of Internet application development, it's even more apparent because of the dynamics of the web. When time is of the essence, the simplest solution is not often the best solution; it is always the best solution. And this brings us to our final point of controversy: technical debt.

I often hear the argument that cutting corners anywhere in the development process will accrue irreconcilable technical debt in the long-term, and the cost of that debt should be heavily factored in when making any decision in the process. In reality, this argument supports my point of view. This is exactly the reason the ability to assess when and how to cut corners is crucial when working on deadline-driven projects. There are different types of "technical debt", and the quickest solution, given some thought behind it, will not necessary add to your technical debt. Similarly, over-engineering will not make you debt-free. The ability to make those decisions, often mid-project, is what separates veterans from rookies.

Additionally, often times people who speak to the dangers of technical debts do not take the business implications into account. Technical debt should be weighted against the actual ROI, because in many cases it is more cost effective to launch early. This way, though you may be accruing technical debt, you are also accruing revenue immediately, and you can reconcile the debt over time. This may be preferable to delaying your time to market so that you end up (arguably) debt-free in tech, but losing market opportunities and/or an immediate revenue opportunity which may be far larger over time than the cost of technical debt itself.

As software developers, we often think our job is to develop software, but, really, that is just the means to an end, and the end is to empower business to reach their goals. Your code may be elegant, but if it doesn't meet the objectives (be they time or business) it doesn't f***ing work.

]]> http://omniti.com/seeds/bending-forms-to-your-will http://omniti.com/seeds/bending-forms-to-your-will Wed, 08 Feb 2012 18:53:37 GMT If you're developing an application that requires a lot of forms, you'll often find yourself repeating the same code. There are good reasons to avoid using a form package: loading a bunch of classes and running through layers of validation will never be as fast as just working with markup and post values.  But if your project doesn't have to be incredibly fast, and you're already taking the standard precautions against resource drain (smart caching, efficient class loading, minimal bootstrapping), a form package can save you coding time, or help you standardize forms across an application without relying on copy/paste.

Once you've decided to use a form package, you'll have to figure out which one is best for the task at hand. As with so many other programming decisions, rolling your own is the way to ruthlessly optimize for the things you care about, but it requires the kind of time investment that might not be feasible. Choosing an existing package may be as simple as noticing one that's part of a library you're already using, or it might take some extensive research on what's out there and what the strengths are for each.

For the purposes of this article, we'll be focusing upon the two big tasks of form building: display and validation. The examples will use Zend_Form, which is large and complex enough to include most of the common ways of doing each. And we'll talk about custom elements, which can be incredibly useful and require attention to both.

Views

One important question to ask yourself when choosing or building a form package is how it's going to fit into the rest of your architecture, and into the rhythms of development for the project.

Most packages offer you the option of auto-building the form markup from the element objects you've added. This has a lot of appeal for back-end developers, for whom writing markup is the least interesting part of the endeavor, and you can get a lot of mileage out of them on plenty of projects.

An administrative back-end is ideal for this kind of use. It's used only by people intimately familiar with the terminology you'll be using for labels, so you're unlikely to have do much UI engineering that would require fine-grain control over the layout. Admin areas tend to have a lot of forms that should all share a look and feel. Using auto-build views means you can describe the look in your base class--any tweaks to the default view can be applied to every form.

You can also make use of auto-build views for a project where you must do demos while you're working on the underlying API: people (particularly people without programming knowledge, but sometimes even programmers do it, too) instinctively associate the design polish of a form with the state of completion of the code underneath it. The raw look of default forms can work in your favor to remind everyone that your API is still under development.

There are, however, good reasons not to let it persist into your production code. If you've inherited a project that uses a form package in this way within an MVC framework, you'll find view blocks that look something like this:

<div id="some-form">
    <?php echo $this->form->render(); ?>
</div>

This is not precisely helpful when you or your front-end developer have been asked to add an image next to the CVV field. Where is all the markup? How do you know where to add it? Instead, you need to find your way into the form object to have a look at some abstracted markup:

$cvv = new Zend_Form_Element_Text('cvv');
$cvv->addValidator(new My_Custom_CVV_Validator());
$cvv->addDescription('A 3 or 4 digit number located on the back of the card.');
$cvv->addLabel('CVV/Security Code');
$cvv->addDecorator('HTMLTag', array('tag' => 'span', 'class' => 'short-entry'));
$this->addElement($cvv);
$this->addDisplayGroup(
   array('card_name','card_type','card_num','exp_month','exp_year','cvv'), 
   'credit_card', 
   array('displayGroupClass' => 'card')
   );

This sort of complexity has a tendency to accumulate over time, too. Before long, you may find yourself with code that breaks a form into columns, wraps tags around elements or just shoehorns blocks of markup in wherever they'll fit. At that point, I'd recommend sleeping in locked rooms, because your front-end developer owns several large kitchen knives and knows where you live.

How, then, do you balance your client's need for you to bring up forms quickly with your front-end developer's need to have a form's markup all in one file?

Personally, I like to keep the overall layout in the view with the important parts plugged in. For example:

<form id="new_widget"
    method="<?php echo $this->form->getMethod(); ?>"
    action="<?php echo $this->form->getAction(); ?>"
    enctype="application/x-www-form-urlencoded"<?php
    if ($this->form->isErrors()) { echo ' class="errors"'; } ?>>
<fieldset class="column">
    <legend>Details</legend>
    <ul class="elements">
        <li id="elem_title">
        <label for="title">Title:</label>
        <p class="note">(will be used to build the url slug)</p>
        <?php echo $this->form->title->renderViewHelper(); ?>
        <?php echo $this->form->title->renderErrors(); ?>
    </li>

. . .and so on. You can see how easy it would be to loop through the elements rather than grabbing them individually, if you were so inclined.

Validation

Let's consider a simple scenario: you must validate a postal code based on country: U.S. against /^\d{5}(-\d{4})?$/, CA against /^[ABCEGHJKLMNPRSTVXY]\d[A-Z] \d[A-Z]\d$/, etc. There are four common places to handle validation:

1. Using validators attached to the elements
2. Using a central method for the form
3. Outside of the form class entirely, in the controller code
4. Using javascript automatically built by the form package
5. Using javascript elsewhere (e.g., at the bottom of the view)

Element-level validation is the way to go if your country is coming from somewhere outside the form-if, for example, you're using country code subdomains. You can have your form class accept a country code on build, and set the appropriate regex for your element when you add it. (Most packages come with a regex validator, and it's a simple one to write if you're rolling your own.)

But what if you're getting the country from a select box? You'll need a set of regexes and a way to switch between them based upon the value of the select. For this, you'll need custom validation. Some packages have a built-in "callback" option for element validation, while others use custom classes. Zend Form sends the whole form values array to its validation classes as $context, but this is relatively uncommon. Most packages require you to fall back to the central validation method instead.

Centralized validation is typically implemented as a method you can extend (the parent version runs all of the element-level validation) to include rules that apply to the form as a whole. You can use it to switch out which fields are required, too, if the package allows you to update that information on the fly. The main disadvantage to making this your primary method of customized validation is that it's not very re-use friendly. The logic to test a postal code based upon country will have to be copied between forms, or made into a function or class that can be included. (This is another advantage of Zend Form: the validators extend Zend_Validate rather than anything in the Form package, so they can be used on non-form data, without requiring any of the Zend Form classes.)

For obvious reasons, breaking out of the form to do validation outside of it isn't ideal. If it ends up there, it should be an indicator that your package is too hard to use.

Auto-generated javascript is often more trouble than it's worth: you don't usually have much control over how the error messages are displayed, and just like auto-build views, they make front-end developers sad. Worst of all, they don't save you much time. If you're only using the validators that are already part of the package, it's relatively painless, but any custom validation has to be written twice-once for the backend and once for the front end. That gets old pretty fast.

You can get most of the benefit of automatic Javascript by using an AJAX call on submit, but of course it only saves you the difference between the cost of a page load and the cost of an AJAX call. You don't get the benefit of not contacting the server at all, but it's usually sufficient for most forms. Using jquery as an example

var formvals = {}; // grab these with .val() as necessary
$('#myform').submit(function (e) {
    e.preventDefault();
    $.post({
        url: '/path/to/script',
        data: formvals,
        error: function (ret) { showGenericError(ret); },
        success: function (ret) {
            if (ret.success) {
                // go to the next page, show an message, etc
            } else if ('messages' in ret) {
                // show them on the form
            } else {
                showGenericError(ret);
            }
        }
    });
});

You would just need to add something to the backend to pull up your form class, run the validation method on the post data, and return some json to indicate success and or failure, with error messages keyed by element name. Your front-end dev can do whatever he or she wants with them in the callback. Note that using a real submit button also means it will degrade gracefully; if the user has javascript turned off, the form will behave as usual.

Custom javascript is usually best written ad-hoc, per form, with reusable parts included in a central file. Combining the AJAX method above with custom non-ajax validation (like checking password strength as you type) means the front-end and back-end remain separate, which can be a significant benefit when the front-end and back-end developers must work independently.

Custom elements

Most of the time, regular HTML form elements are fine (and if they aren't, there's usually a jquery plugin to fix that), but occasionally you'll have to do something much more complicated and then include it on multiple forms-for example, an image selector that taps into a pool of user-uploaded images. Often the code to support it will be extensive and involve not only markup but also validation, display, and filtering logic. Rather than try to keep a library of functions to do each piece, you can build a custom element that knows about all those parts and slot itself into the regular form flow.

In Zend_Form, this usually means creating a very simple custom form class:

class My_Element extends Zend_Form_Element {
    public $helper = 'myElement';
    public function init () {
        parent::init();
        $this->addValidator(new My_Validate_MyElement());
    }
}

And a view helper (assuming you've already set the view helper path):

class My_Helper_MyElement extends Zend_View_Helper_FormElement {
   public function myElement($name, $value='', $attribs=array(), $options=null) {
     $info = $this->_getInfo($name, $value, $attribs, $options);
     $content = '<div id="my_elem_' . $info['name'] . '">';
     // show the value in an interesting way...
     // load a jquery plugin, whatever
     $content .= '</div>';
     return $content;
   }
}

And, if necessary, a validator:

class My_Validate_MyElement extends Zend_Validate_Abstract {
    const MYBAD = 'myBad';
    protected $_messageTemplates = array(
        self::MYBAD => "%value% is wrong because...',
    );
    public function isValid($value, $context = array()) {
        if (/* some test */) {
            $this->_error(self::MYBAD);
            return false;
        }
        return true;
    }
}

Then you can add this element wherever you need it. . .like so:

$this->addElement(new My_Element());

Much less painful all around.

In conclusion

As you go about bending forms to your will, bear in mind that the choices you make will affect your whole team, and anyone who works on the code after you. Separating layout from your class structure will ensure that your front-end developer doesn't have to learn about the Decorator Pattern, and modular validation will make for a easy transition when you have to offer the form as a web service. These packages aren't as annoying as they can sometimes seem, and they can provide speed now and flexibility later, provided you use them with care.

]]> http://omniti.com/seeds/why-nosql-does-not-mean-nodba http://omniti.com/seeds/why-nosql-does-not-mean-nodba Thu, 19 Jan 2012 19:55:29 GMT Whether you like it or not, NoSQL is changing the world. Granted, it's not even clear what NoSQL means sometimes, but there is no doubt that, for better or worse, we are in a renaissance of non-relational database systems right now. For me personally, I tend to ignore the hype, study these systems with a critical eye, and then deploy them where traditional RDBMS software struggles. I do occasionally bump into people who babble on about how NoSQL will put DBAs out of business. When I hear this kind of comment, I just nod my head and smile: It's hard to convince people that their beloved paradigm shift is just more of the same, and also very seldom worth the effort. However, I was recently talking to an Oracle DBA, and he made some comments about how he was concerned that new companies would have no use for a DBA because they were all switching to NoSQL. This surprised me a little, actually. I figured if there was one group of people who wouldn't buy into the NoSQL hype, it would be the stalwart Oracle crowd. Et tu, Brute? If the hype has gotten to them, I guess it's time for me to speak up. Whatever you think about NoSQL, the death of the DBA is a ludicrous idea.

One of the little known secrets of NoSQL systems is that they are used to hold data. Most NoSQL systems try to trumpet the ease of pushing data in and out of the system: "just push your JSON object to the server and your data is instantly stored, regardless of structure." It's Oh, So Magical. The problem, of course, is that easily dumping data into a system doesn't mean much if you can't get it back out. This is where a data model comes into play.

In a traditional RDBMS environment there is usually someone who designs a data model ahead of time, breaking down information into a relational design, sometimes even drawing up an ERD diagram for people to reference, so they can find their information. This would be turned into DDL, committed to the database and then enforced rather strictly. Try to insert the wrong type of data, and you'll get an error. Try to query for a column that doesn't exist, and you'll get an error. I've seen many a developer complain that the database is "too strict" because they couldn't get their queries right, but the truth is you still need a data model even though you went to a NoSQL system. You won't hear much about these problems from NoSQL advocates because the type of shops that use them are often smaller shops, where there are a small number of developers, and everyone knows what everyone else is doing, more or less. Or they are using NoSQL on new projects--where the scope of the system still fits in head-space pretty easily. However, as code bases grow, and projects stretch on for years, and developers come and go, not knowing what data is stored where means that you start to wish you had a data model. You recognize this the first time you hit a bug because you assumed that not getting a value back in the "items" key meant the user had no items in their cart, only to realize later that you should have been looking at the "item" key. If you're lucky, this type of bug pops up on information retrieval rather than in information storage; but either way, cleaning up such a bug can be painful. You'll also notice how, often, you have to pull back objects just to see what information is actually stored, because depending upon when the object was inserted, it may have a different opinion on what the data it should be holding looks like. Of course, you can go back and dig through the code that puts the data in, but this assumes you know where that code lives. Maybe you could ask the guy who originally wrote that feature, but suppose he left the company six months ago? Oh well, grep is your friend.

So let's pretend you've taken the time to discuss what data you must hold up front, and someone keeps a diagram of that posted on the wall. . . awesome. Sadly, you must still implement it in a physical system. In all the hype about "schema free" databases, the fact that in order to get good performance you still have to make adjustments for physical layout, or build things like indexes into your system to make queries against the server fast enough, is often overlooked. Yes, even on NoSQL you still need to know about explain tools and indexing.

Think about what this means: Someone has to recognize that a certain piece of data is going to be requested a lot or notice a performance problem on the existing server. Once you figure out the right index to build, someone has to build it in production. This means locking, potentially, and certainly means an IO hit. Is your application developer going to be responsible for this? Does he know if your index build will require backfilling, like when you build a secondary index on existing data in Riak Don't get me wrong, it's not that they aren't capable of doing this work, it's just important to realize that this type of work must be done.

OK, you have a data model, and are managing the physical implementation; that's good. But did you know that your NoSQL system still must interact with the disks on your server? It does. The better question is, do you know *how* it interacts with the disks? Actually, before we talk about the disks, do you know how it interacts with RAM? Some NoSQL systems absolutely fall over when they hit thresholds larger than RAM. For some, it's total data set size, for others it might be the size of all index pointers in the system. Of course, maybe you have a system that doesn't fall over, it just becomes slower, perhaps unacceptably slower. In either case, you need to be aware of these limitations. Have monitors in place for them, and then perform capacity planning accordingly. Now, let's get back to disks. How crash-safe is your NoSQL server? Does it give you single node durability? Are writes automatically synced to disk with each put, or are they batched up and pushed out occasionally? Maybe it's configurable; do you know how your systems are set up? If you were using Postgres, you could tune the durability guarantees for all of these cases. Your DBA knows this, and whoever is in charge of your NoSQL system needs to know this. Even if you think you are storing data you can afford to lose, chances are your business model must be aware of just how much exposure it has. Oh, but yours is a start-up, so you don't have business concerns yet. . .? Still, the level of durability is going to have significant impact on your IO needs, and that, in turn, will impact your performance--and you can't post your devops data-porn if you can't get decent systems performance.

Of course, disks are kind of unimportant these days, given that everyone runs multiple nodes, and you can have a distributed hash table running across multiple nodes with just a handful of Chef commands. That said, have you ever managed a complex distributed system? You know who probably has? Your DBA. By far the most common answer to the failover problem is to stick up a replicated database slave. It's also common to see people putting up slaves for horizontal read scaling. DBAs understand the tradeoffs in consistency guarantees that come with these types of systems--not just at the node level, but from the applications point of view as well. You'll need solid understanding of this on your dev team if you are going to build apps against a distributed data system. In addition, someone has to manage all of these servers and make sure they perform well. If your NoSQL system uses master-slave replication, someone with experience in this area might be handy. If you've ever built a Master-Master pair with individual Slave systems, you probably know what I am talking about. Oh, do you think running a clustered hash table system is easier? Just because you can add a new node to the ring doesn't mean it's free. You need both server level and cluster level monitoring in place. You need to make sure you can afford the IO and network strains as data is copied around, and you need to know under what circumstances locking will be involved. These things really do happen.

I remember when the MySQL documentation had a section devoted to explaining why foreign keys weren't needed. Of course, once MySQL finally implemented foreign keys, it became a major headline for their release announcements. This is what happens as systems mature. Most NoSQL systems can cut down on overhead by eliminating (or more accurately, not implementing) many of the features people have come to expect from an RDBMS. Of course, which features are eliminated differs across systems.

Did you know you can write triggers for Riak in either Javascript or Erlang? Exactly which language you can use when differs depending upon the type of trigger.

To wrap your head around this, you need to have a good understanding of how triggers work, how asynchronous calls affect transaction semantics (or the lack thereof), and what types of work you might want to do on the server side. Some triggers are used to enforce data integrity or do data manipulation at the server level; these are the types I think work fine within a vertically scaled system. Others really are extensions of the application, and while they are sometimes frowned upon for adding overhead into a centralized resource like a typical RDBMS, in a decentralized system that scales out the arguments against them aren't as clear cut. One thing I do know though: this is probably not something your SA wants to be involved with at all.

If all of this isn't enough to make you think twice, let me mention one more thing. While you may not have a query language in your NoSQL system, that doesn't mean you don't query against it. Whether you are writing distributed map-reduce queries, trying to balance link-walking vs. secondary indexes, or trying to figure out whether the code you've written to pull back every key in the system is going to be a problem; there are going to be times when you will have to make these queries more efficient. This is probably going to be a more application-centric type of tuning than the traditional RDBMS, but watch as someone in your dev team becomes known as "the go to guy" for making your map/reduce query run more efficiently. And incidentally, you should also be aware that many of today's NoSQL systems are trying to bolt on SQL and SQL-like interfaces into their systems. Who fails to get excited when thinking about rewriting queries with subselects into joins clauses against a Hadoop cluster?

If you think that managing all of this sounds like an impossible task, you're welcome. This is the job that DBAs have been doing for years. . .and yes, it can be incredibly challenging. Of course, it doesn't have to work this way. You can draw the lines of responsibility differently right now. Make the application developers manage the data model, design the schema, and tune the queries. Let your ops people be in charge of building new nodes, managing replication, and ensuring you have valid backups. Maybe 10 years ago, you had to have a DBA to work with Oracle, but nowadays just about anyone doing software engineering can put up a Postgres database and tune their way to usability with about three wiki links; you don't need a DBA to design good operational habits.

Also, some of you might think of this as some type of doom and gloom piece against NoSQL; it is not meant that way at all. It's not that switching to NoSQL is a bad idea necessarily; there are some things that RDBMS software can't do as well as a more dedicated solution. But, if you think that switching to NoSQL will just let you hand-wave away all of the challenges of running a database, you are terribly misguided. If you're a DBA and you are worried about a future with NoSQL, take heart; study your product less and focus on these key architectural design points more. Those skills are critical now, and they will remain so in the future, NoSQL or not.

]]> http://omniti.com/seeds/infrastructure-cost-reduction http://omniti.com/seeds/infrastructure-cost-reduction Fri, 02 Dec 2011 20:48:33 GMT Everyone would like to spend less money on their server infrastructure, but it can be difficult to figure out where money can be saved, and whether reducing the amount that you spend on infrastructure will result in a revenue drop, caused by outages and reduced service quality. By looking at your costs, revenue, system metrics and task/time management system together, it's possible to avoid these pitfalls, and reduce the amount of money that you spend on your infrastructure while simultaneously improving the quality of the services that you provide to your customers.

One of the first steps to reducing your infrastructure cost is to make certain that you have a monitoring solution in place that can help you identify underutilized parts of your infrastructure, as well as what parts of your infrastructure are costing you money.  

Most shops already have metrics in their monitoring system that can be used to identify underutilized equipment, but if you do not: CPU usage, disk IO, network IO, and memory usage are all good places to start looking--and they are all easy to monitor. When you find underutilized equipment, a money saving solution is usually pretty obvious. If the system is older, and is a good candidate for virtualization, then it should probably be virtualized. If an expensive network link is not being heavily utilized, or it's being over utilized, reconsider whether there are cheaper (and better) options available from another vendor. Reducing costs by virtualizing, moving off of old hardware and getting network connections that are suited to your business should be second nature at this point, but it can still be difficult to isolate exactly where you can do this. Simple systems monitoring can help.

Figuring out which parts of your infrastructure are costing you money can be more difficult. Most businesses are not monitoring the number of customers they lose as a result of unplanned downtime, or what the cost of support on old hardware is over time. By integrating business information (like number of customer sign ups over a period of time, amount of money refunded or number of support cases opened) into your technical monitoring, error detection or trending system, you can immediately see what the results of an outage or change are; and, you'll know how much money should be spent to fix a problem, or to build a more fault-tolerant infrastructure.

There are some things that are difficult to automate in monitoring, but should still be reviewed on a regular basis. Support contracts, rack space/colocation bills, bandwidth overages (or underutilized contracts for bandwidth) and power bills all fall into this category. As equipment and environments age, fixed costs become taken for granted. When this happens, you'll frequently forget that you are paying money for rack space that is no longer in use, bandwidth that is no longer necessary, and expensive support contracts on systems that could be virtualized onto an under-utilized new system. A newer system is often already in place, but the legacy system is left running for months, if not years, in case the new system fails. When you replace something and keep the old system as a backup, set up a reminder to revisit the decision to keep the old system after a month or two. If you don't, the legacy equipment can end up staying in use for years before someone remembers it.

Finally, review your own time tracking system to see how you are spending your time. It's easy to get into the rut of documenting a manual way of taking care of a task, and then doing it that way every time. If you can automate a process (or even parts of a process), or make the documentation simple enough for anyone else to follow, you can reduce the amount of time you spend on things and have more time for setting up new clients, investigating new software and helping your users.

One of the things to look for in your time tracking system would be who is spending time on tasks, and what those tasks are; if senior people are using their time to do the same task over and over again, it can be a sign that the task should be better documented, so that more people in your group can take care of it (and the more expensive time of senior administrators can be used for more difficult work).

To summarize:

1. Monitor everything and look for under- or over-utilized resources.
2. Track your time; if you are spending lots of time on the same procedures; automate them. If you are responding to the same problems over and over again, find a way to permanently fix them.
3. Watch your invoices. It's easy to pay support, bandwidth and power bills month after month, or year after year, without reviewing them to see if you can get a better deal elsewhere, or to see if non-critical infrastructure is costing you more money than you would like.

These ideas are all simple, but by considering them during your day-to-day operations, as well as during periodic reviews, you may find yourself spending less money, and using less time, on keeping your infrastructure running well. Reducing the cost of your existing infrastructure gives you more time and money to spend on improvements and new projects, rather than merely on maintaining what is already in place.

]]> http://omniti.com/seeds/sometimes-sexy-can-be-the-right-choice http://omniti.com/seeds/sometimes-sexy-can-be-the-right-choice Wed, 23 Nov 2011 00:00:00 GMT Hardly a month goes by these days without some exciting new technology hitting the blogosphere, filling the imagination of CTOs all over. At OmniTI, we are often approached by people asking about the "razor's edge" technology of the week. Frequently, they are convinced that this is the technology that they need for their business, and often will try to shoe-horn their requirements to fit the new toy. We typically have to convince people of things like their dusty old relational database actually handling their data needs just fine, even if it isn't web scale. Tried and true typically works better than shiny and new.

Sometimes however, a client's requirements really do lend themselves nicely to the newer technologies and we are justified in playing with them during business hours instead of at home! We love our jobs at OmniTI.

The request we'll review here was fairly simple. The client needed a highly scalable and fast web service to provide geo-location data, based upon the ip address of the requester. They also had to serve a small static file. The service would run for a few months and then would be discontinued. It didn't require true high availability, but we had to be able to fix it quickly if something went wrong.

Using technologies we were already employing on the project, we wrote a simple Mungo-based perl script to look up the information in the MaxMind city level database and return the data inside of a JSON object. Once placed on the existing Apache httpd servers along with the static document, we had a working prototype for their third-party to develop against, while we looked at the more complex issues in the request.

In this case, there were two immediate concerns:

• The service had to be fast and handle a lot of requests.
• This component should not endanger the availability of the rest of the web services.

The web farm already deployed to handle the client's business used the Apache httpd server and, leveraging the platform flexibility, it grew to support a number of legacy web services. As this setup was already tweaked for these particular needs, we didn't really want to reconfigure it. However, we needed to know where we stood from a performance point of view, to find out how much traffic we could handle. A quick Apache bench testing revealed:

Document Length:        188 bytes
Requests per second:    1306.85 [#/sec] (mean)
Time per request:       38.260 [ms] (mean)
Time per request:       0.765 [ms] (mean, across all concurrent requests)
Transfer rate:          520.79 [Kbytes/sec] received

Our goal was to be able to safely handle about 5,000 requests per second. While we could sustain that traffic by scaling out across our client's multiple web servers, when traffic volume would reach worst-case expectations, there would be an unsafe likelihood of the servers becoming saturated, followed by service degradation. Or worse yet, all of the web services could become completely unavailable. Needless-to-say, either case would be unacceptable. We had to isolate this service from the rest, however isolating on similar hardware as we were currently using for the web-farm would have been a prohibitively expensive solution, especially considering the transient nature of a project designed to last only a few months.

With such requirements, a cloud deployment was the obvious choice. While there are plenty of reasons to stay away from the cloud, there are some really good reasons to use it, as well. The cloud would let us use exactly as much CPU and bandwidth as we needed, and provide an easy and quick way to get more if we required it. Our service did not store persistent data, even at a session level, so if a cloud instance went "poof," there was nothing we couldn't afford to lose. When no longer needed, we could just shut down or scale back the servers, without worrying about excess hardware--the exact benefit cloud supporters always want. EC2, here we come!

With the move to EC2, we had the option to deploy the prototype code that we had written already. However, as that code leveraged an existing ecosystem designed to service a much wider spectrum of needs, duplicating the environment would have been overkill, and attempting to strip it down to the minimum necessary would have been a rather daunting challenge with little long-term benefit. With the luxury of exploring a green field approach, we turned our attention to Node.js. At OmniTI, we had the advantage of having seen Node.js used already a few times for production services, and we had even incorporated it into a few solutions we had developed, so we knew that the type of light-weight, fast response code that we were looking to develop for this project was very well suited for Node.js. Through a bit of serendipity, Theo Schlossnagle had just recently branched, and then finished, a new version of node-geoip that was capable of reading the MaxMind City database. Add to that my personal joy for getting the chance to use Node.js in production, for a customer project, and the decision was clear.

Plan in hand, the perl script was quickly converted to Node.js and placed on a small Apache EC2 instance for load testing (thanks to Zach Malone for assistance with all of the cloud benchmarking work). The entire code follows.

var http = require('http'),
    sys  = require('sys'),
    geoip= require('geoip');

var con = new geoip.Connection('/www/geodata/GeoIPCity.dat', 0, function(){});

http.createServer(function (req, res) {
    if( req.url == '/get_city' ) {
        res.writeHead(200, {'Content-Type': 'text/plain'});
        var ip = req.headers['x-forwarded-for'] || 
                     req.connection.remoteAddress;
        con.query( ip, function(result) {
            var obj = new Object();
            if(!result){ obj.city = 'Unknown';   }
            else       { obj.city = result.city; }
            res.end(JSON.stringify(obj) + "\n");
        });
    } else if( req.url == '/crossdomain.xml' ) {
          res.writeHead(200, {'Content-Type': 'text/xml'});
          res.end("\n\n\n\n\n");
    } else {
          res.writeHead(404, {'Content-Type': 'text/plain'});
          res.end("File not found.\n");
    }
}).listen(80);

Slightly more than twenty lines of code. This will return a JSON object with the name of the nearest city, based upon client IP address, or "Unknown" if it does not resolve. It will also serve a crossdomain.xml file to any flash objects that need one, and return a 404 to any other requests. Where's the web server, one may ask? Node.js takes care of all of that for you.

Simple Apache bench testing of this code gives between 300 and 600 requests/second on a Small EC2 instance with a single virtual CPU.

Document Length:        223 bytes
Requests per second:    344.75 [#/sec] (mean)
Time per request:       145.033 [ms] (mean)
Time per request:       2.901 [ms] (mean, across all concurrent requests)
Transfer rate:          96.62 [Kbytes/sec] received

Yes, much slower than what we were benchmarking on our web-farm, but much cheaper to scale out, not to mention that we were having the service separation we wanted. To scale out, we had to load balance our instances; Amazon's Elastic Load Balancing was used here.

It was expected that a small decline in service would occur due to the overhead, but we were pleasantly surprised to see slightly BETTER performance. Apparently, getting the IP address in Node.js from the request header is faster than getting it from the connection object, so having a load balancer in the middle actually improved performance.

Document Length:        223 bytes
Requests per second:    367.87 [#/sec] (mean)
Time per request:       135.918 [ms] (mean)
Time per request:       2.718 [ms] (mean, across all concurrent requests)
Transfer rate:          112.40 [Kbytes/sec] received

In repeating the test with two, and then three, server instances behind the load balancer, each new instance added continued to scale the volume of requests-per-second we could handle by another ~350-600 requests/second. So, with only three small EC2 instances, we were able to crank out between 1200-1500 requests/sec of GeoIP lookups.

350 to 600 requests/second is a pretty large window, and it means that some of our EC2 instances do much more work then others. This is something that you have to deal with when you are deploying a cloud-based solution. Thankfully, EC2 gives you a lot of flexibility to rapidly create and destroy instances, so if you get an especially slow instance, it can be worth throwing the instance away and creating a new one. As a bonus, if needed, it takes fewer than 15 minutes to manually get a new instance provisioned, set up, and running, without using Amazon EBS. Not relying on EBS enabled us to dodge the infamous EC2 outage. Our service was unaffected despite running in the unfortunate Virginia data center cloud.

Now, just because we were using Node.js, and we were deploying to the cloud, doesn't mean we toss away due diligence. In order to make certain that the EC2 solution offered good performance for the money, we decided to benchmark the same code on a Joyent SmartMachine that we had available. A single Joyent system had the performance of ~3.5 small EC2 instances:

Document Length:        223 bytes
Requests per second:    1564.30 [#/sec] (mean)
Time per request:       31.963 [ms] (mean)
Time per request:       0.639 [ms] (mean, across all concurrent requests)
Transfer rate:          438.43 [Kbytes/sec] received

The cost of the Joyent system was, however, twice as much as three small EC2 instances, plus a Elastic Load Balancer. Joyent includes a generous amount of bandwidth with any instance (Amazon does not), but their large, fixed monthly cost meant that we would not have as much flexibility to scale up and down as we did with EC2, which has hourly billing.

So, we had a working solution at this point, but we still had to make sure it would continue to work; in short, it had to be monitored. Normal end-to-end monitors and request timing monitors were put in place on the load balancer, as well as checks on each individual server instance. But, we also wanted to know how much traffic we were serving without anymore fuss. Node.js could keep track of that for us as well. By simply adding:

var cities = 0, xmls = 0, fnf = 0, status = 0;

... some variable++'s in the appropriate spots, and ...

    } else if( req.url == '/status' ) {
        res.writeHead(200, {'Content-Type': 'text/plain'});
        status++;
        var obj = new Object();
        obj.cities = cities;
        obj.xmls   = xmls;
        obj.fnf    = fnf;
        obj.status = status;
        res.end(JSON.stringify(obj) + "\n");

. . .we could see exactly how much traffic, of each type, that each Node.js instance had served; along with whether any of them had crashed (as evidenced by a reset counter). This was set up to be pulled by Circonus which can consume the JSON data and graph the usage trends over time.

Perhaps also of interesting note, all of this was done almost a year ago. "A few months" turned into much longer. The client's required utilization has gone up and down with a corresponding number of EC2 nodes added or removed. But this simple script hasn't had to be modified or touched since. It has happily run a production service without any problems for a minimal amount of time invested.

To be fair, this was a rather simple problem that could have been solved in a number of different ways, perhaps even more effectively. But sometimes it behooves you to explore those sexy new technologies, learn their trade-offs and understand them better. In this way, you'll understand the trade-offs involved, and you can feel comfortable deploying them for critical components of an architecture. While it's essential to remember that sexy doesn't mean good, it's a pleasant reminder that sometimes good can be sexy.

]]> http://omniti.com/seeds/thoughts-on-web-application-deployment http://omniti.com/seeds/thoughts-on-web-application-deployment Tue, 01 Nov 2011 20:47:18 GMT Abstract: A quick overview of various strategies to ease deployment of web applications, and some common pitfalls and failure modes to avoid. Intended to be broadly technology-agnostic.

Introduction

Over the course of my career, I've worked in a number of different environments, each with their own particular processes and procedures for deploying systems, from development to production. Over time, a number of best practice patterns and common anti-patterns have emerged, which this article will attempt to enumerate and explain. I hope this information will give you pointers and direction to improve your processes so that that deployment is made both easier and less error-prone. As much as is possible, I will be technology agnostic, so of course your particular environment may vary or require additional steps, but following along the broader themes listed here should be helpful.

Step Zero: Intelligent Use of Source Control

This seems like something that is almost forehead-slap obvious, but you should be using source control during development. Working without it is like doing high-wire acrobatics without a safety net. There is no project so small, either in scope or staff, that cannot benefit from some sort of source configuration management (SCM) system in place. Selecting which SCM to use (e.g., Subversion, git, Perforce) depends upon your team's development style and environment choice , and is beyond the scope of this article. In general, the Pragmatic Programmers, O'Reilly and APress books covering particular systems tend to be good resources.

Further, source control must be used intelligently to be of much use. Common anti-patterns include things like having the development environment exist in only one shared space that uses one source control checkout so people are colliding over editing the same physical bits for any given file, or never branching/tagging so that trying to determine the exact state of your system at a given point in the past is an exercise in frustration. If you ever have to ask "is anyone else editing this file?," you are either using a supremely broken SCM or you are doing something gravely wrong.

Your SCM setup should enable you to work concurrently and in isolation with a minimum of hassles, allowing easy integration of work done concurrently on the same module or set of modules; easy reproduction of the system as it was at any point in time in the past; and, ideally, easy searching of commit history because all of these scenarios will come up repeatedly in any project of significant scale. For example, if you find yourself trying to diagnose a recurrence of an issue with a particular ticket number, your diagnosis will be vastly sped up if it is easy to find all of the commits related to that ticket number in the past. Similarly, being able to tell easily who exactly added a particular feature months or years ago might make it much faster to track down the organizational knowledge required to extend or fix it in the present.

The more isolation feasible in the environment, the less coordination overhead is required to work together as a team on a given workload, meaning that your productivity scales more linearly with additional developer resources. Good usage of a SCM can aid this by making it easy to keep individual development environments in sync; a common best practice is to make sure that each developer can run their own stand-alone copy of the execution environment based upon a frequently-updated SCM checkout. As a specific example, with the common LAMP (or similar) technology stack, it is easy for each developer to have an account on a shared machine, with a checkout in their home directory that is used as the root for a vhost/distinct port so that each developer may work in isolation talking to different ports on the same host.

Having a particular "branch" or "tag" devoted to staging/testing and production environments is a common best practice. The particulars of how this is done vary by SCM, but every modern SCM should have some facility corresponding to one or the other of the above if not both. Branching/merging tend to be slightly more complex operations in most SCM systems, but the effort expended in learning these operations will repay itself many times over as you begin to be able to take a more sophisticated approach to the various states and stages of your system's evolution over time. See steps two and three below for further discussion along this theme.

Step One: Deployment Planning

Take the time to write out a deployment plan, even if it's just a brief one. At a minimum, your deployment plan should include:

• Name and short purpose description of the project (seems obvious, but depending upon how widely this information is distributed and how big your organization is, your readers may not automatically know what you're working on in order to tie your authorship back to what this proposed production deployment is all about)
• Names of and contact information for the staff responsible for its development (particularly tech leads and project managers)
• Source location (e.g., links into the SCM's web interface or descriptions of how to retrieve the source for SCM's that don't have a web interface)
• List of affected systems/what resources will be used (i.e., which servers this code will be pushed to, are there any extra steps that have to take place, like running of database modification scripts or setting up of extra server software/new configuration for existing resources?)
• Deployment and Rollback procedures (this may reference standard operating procedures in other documents if there is nothing out of the ordinary for the given deployment)

A wiki is a common tool for this, but even an email to the right people or a mailing list can suffice. One advantage of a persistent document is that it can "grow" over time as the project evolves rather than being reconstructed from scratch on each deployment (i.e., things like project staff or the particular branch of the project's source being used may change only infrequently, but dated/versioned logs might be kept of which revisions were rolled out when, or which revisions required an extra step, etc.). This is particularly important for continuous deployment environments (see next section). It is also easy to maintain a template like this, so that each one follows a common layout and helps staff remember frequently overlooked steps (e.g., helping developers remember to mention system software config changes required or database schema changes).

This may seem like unnecessary overhead (developers who truly enjoy writing documentation are not common), but retaining organizational knowledge about what is deployed where, when and why is crucial to keeping non-trivial systems running over time. Even if the original staff who deployed a system are still with the company (staff turnover being a fact of life for any organization), remembering precisely what was done and why potentially years after the fact is not an easy feat. The effort invested in this now will repay itself in the future.

Step Two: Continuous Deployment
or Phased Deployment?

There are two common modes of deployment for web applications. The first models traditional software engineering by using phased deployment, where phases of release correspond to planned or scheduled bundles of additional features and new bug fixes. A variant of this is the "boxcar" or "feature train" model that ships a defined release on some set schedule ("If it's ready in this six week window, it goes on the shipment train, if not, it waits for the next one."). This is common for environments that have rigorous quality assurance or change control requirements, as it allows a built in time period prior to each phase's release for those processes to execute in a regular, repeatable fashion on a known schedule. In these environments it is common to "snapshot" the particular phase for deployment in some fashion, via something like a "branch" or "tag" as discussed in step zero. For example, a Q3 phased release for a system might have "prod-Q3-2011" as its source branch/tag. The state of the system so denoted might then further be used as the base for issue remediation hotfixes that must go live in between regular phased releases. For an automated deployment environment (see next section), the system would need to be aware of the correct current branch/tag to use as its deployment source (or perhaps offer the option of the currently available sources that match a given pattern to the user making the deployment).

The second, and more recent development, is continuous deployment. With continuous deployment, new features or bug fixes may go live at any time. Some environments push live to every user at the same time, and others use a "feature flag" approach where a given user must have a given flag or set of flags in their active session or profile to be exposed to the new code. Care must be taken for "feature flag" setups to ensure that tests of the system (see monitoring and verification section below) are using the correct flag or sets of flags to accurately capture the state of the system as the end users see it.

What I will say next has proven to be one of the more contentious parts of this essay in internal discussion, so I freely admit that this is a point worthy of further thought particularly as time provides more evidence on the ways that continuous deployment works or fails. I do not believe that continuous deployment systems should be configured such that the source for pushes to production machines (e.g. a branch or trunk or whichever nomenclature is appropriate for your environment) is the same space that developers initially check code into. I am willing to stipulate that things like developer mindset and discipline in concert with automated checking scripts within the commit process may eliminate many sources of error that could be introduced in such an "insta-live" system, but I'm also a big believer in the power of Murphy. Having some separation here, however low friction, should help prevent many errors (maybe something like a code review queue or holding pen that things go through before going to production, or development in branches with deploys drawn from trunk with many and small merges vs fewer large ones). In my mind continuous deployment is more about release automation, scope of work per released quanta, and democratization of the release process combining to empower individuals to release quickly than any particular SCM configuration litmus test.

Which of these approaches works best for your team may be dictated by the business/regulatory needs your application must satisfy, or may be limited only by the consensus of personal preferences involved. Generally speaking more conservative environments will trend toward phased deployment out of necessity.

Step Three: Deployment Automation

The easier you can make it to do the "right things" for your environment, the more likely people are to do them. What these steps might be will of course vary widely, but common examples may include things like moving files into place (static assets, interpreted code), compilation and movement/packing of resultant binaries (for compiled language environments), application of database changes, and so forth. Almost all of these various steps may be automated via some mechanism (e.g., via scripting languages either directly on a command line, or perhaps in more sophisticated environments an actual deployment manager standalone application). Particularly large, multi-server systems may choose to roll deployments out in stages to increasingly larger subsets of their total infrastructure, effectively using A/B testing with progressively larger portions of their active user base to check for any problems as scale increases or negative user experience feedback; this is obviously much better done in as automated as possible fashion as the chance for simple errors increases dramatically as the number of manual interactions increases.

As a quick example of a simple implementation of this kind of setup, several years ago I worked in a PHP-based environment which used the "qa" and "prod" CVS tags to tag particular revisions of various files as being suitable for deployment to a particular environment. When a developer (with the right access privileges) accessed the deployment manager web application, he could select which tag to deploy; the system would do a CVS checkout of that tag to a scratch area and then do an rsync command to move all of the code (and associated static assets) to the appropriate server(s). This radically reduced the overhead of deploying code, although it was not perfect in the sense that database and other system config changes still required the involvement of the relevant systems teams. A variant of this in use at another organization similarly depended on conventional branches for "qa" and "prod", but instead of using rsync would use ssh to invoke svn up commands directly on each affected machine (Apache was configured to deny access to .svn directories).

One tool that seems often overlooked in this area is use of OS-native software packaging mechanisms to distribute content and execute scripts required for the given change set. These scripts may be either tailored to the particular release, or may be general standard scripts that by convention draw data from named portions of the source being deployed (e.g. a "db/001.sql ... N.sql" file set might be iteratively applied in order if they exist, or a "etc/001.patch ... N.patch" set of patch files might be applied in a similar fashion). Use of this sort of packaging system will make it much quicker to verify that a given app is installed, what files are associated with it, whether any of those files have been modified, and so forth, and also makes installation/upgrade/removal far more automated. Another example for a Java-based system might be an OS-level package that contains the compiled WAR file and pre-/post-install scripts to invoke the correct application server steps to install or update the application.

Step Three: Monitoring and Verification

Being able to keep a real-time watch on your system's performance and user behavior is extremely important during and after a deployment. If server errors surge after a push, clearly the deployment will need to roll back, but other more subtle failure modes may also be important (a change that leads to increased latency on the site might hurt conversion/activity rates of users, for example). Having a system in place to collect and monitor these technical and business metrics will go a long way toward increasing your assurance that a given deployment has not introduced any issues.

In a related vein, having a suite of integration tests that you can run on production to quickly verify that all expected functionality is working at any given point in time can be extremely handy (so that you don't have to wait for a user to stumble on the one out of the way use case that happens to now throw an error). This becomes particularly powerful in systems large enough that manual testing of the entire API/UI is inefficient. These integration tests must be distinguished from unit tests which are likely also part of your testing and deployment strategy, albeit at a more granular source-code level. In all cases, designing for modularity and testability will make your life much easier when it comes to verifying the behavior of your software, but that is a matter for another article.

The resources and further reading section below has links to a few different tools for both areas listed above. There are many other options, of course, so finding the best fit for your environment would be a matter of further research.

Conclusion

I hope this article has given you some insight into how to improve your deployment processes, with the goal being reduction in complexity and uncertainty related to making your system evolve to fit ever-changing business needs. The steps outlined above may be adopted/adapted to your organization in stages, but the more fully you adopt them the more synergistic benefits you will see. In all cases, the guiding principle should be to make it easier to do the right things for your environment and minimize end-user complexity. No matter what technology stack you are using, and no matter what type of application you are writing, getting deployment right can make the difference between going crazy from stress and having a happy, productive work day.

Resources and Further Reading

Source Configuration Management Systems

• Subversion -- A common centralized SCM used by many organizations; free software. Quality books covering "svn" are available from several publishers, and some are available online freely as well, e.g. the red bean svn book
• Git -- An increasingly popular distributed SCM, used by large projects such as the Linux kernel; free software. As with svn above, git has several good texts in print and some are available online e.g. Pro Git
• trac -- A web interface to several common SCMs (svn, git, etc.); integrates a ticket management system and wiki as well as source browser, free.
• mtrack -- Similar to trac but with several enhancements e.g. native ability to handle multiple projects per single install (trac as shipped is intented to have one instance per managed project)

Deployment Planning

• dokuWiki -- A common and full-featured wiki; free. PHP based so anything supporting that (apache or similar on Unix, IIS on Windows, etc.) should at least have a good chance of running it.

Deployment Automation

• Scripting Languages -- This will greatly depend on your environment, but almost any enterprise computing platform these days will have some sort of scripting mechanism, e.g. perl, python, ruby, etc. (Windows versions in particular of things like perl and python may be obtained from ActiveState both freely and with support contracts.)
• rsync -- an intelligent method of syncing files between two computers, free software.
• Vlad the Deployer -- a free, ruby-based deployment automation system. I've seen this used in-house in concert with additional development in ruby to produce Solaris and CentOS packages automatically as well as rolling them out to the target systems.
• Ruby on Rails as a system deserves credit for thinking about deployment automation more than many other frameworks, e.g. database migrations and deployment managers like Capistrano/Bundler.
• Your chosen operating system's package management documentation; generally speaking any enterprise grade server operating system will have some sort of package management and documentation/guides will exist for how to make/maintain packages for that system.
• Cloud-based deployments are another special case, as many "cloud" infrastructures are themselves scriptable to allocate/deallocate additional resources, making another level of potential automation as well as simply managing the deployment of code and config changes. An example of this is Nimbul from the New York Times (centered around Amazon's set of elastic/cloud services).

Monitoring and Verification

• Selenium -- Selenium is a way to record and then play back web application interactions via browser, and is useful when constructing behavioral/integration tests to verify a site's functioning.
• Nagios -- Commonly used infrastructure monitoring tool, can be a bit of a bear to set up the first time; free.
• Cacti -- A graphing and trending application, free.
• Circonus -- Circonus takes the setup and maintenance hassles out of monitoring and trending, available as a service.
]]> http://omniti.com/seeds/your-orm-sucks http://omniti.com/seeds/your-orm-sucks Fri, 21 Oct 2011 21:02:58 GMT I don't like frameworks. Web application frameworks, ORMs, whatever.

I don't mean that as harshly as it probably sounds. It's something like saying, "I don't like cooking with microwaves." They have their uses, certainly - I'm not going to scrub out a pan in the morning because I want to make oatmeal, for example - but there are limits to what they can do, and I think there's a reluctance or inability to recognize that. I'm certainly not above nuking a pile of Bagel Bites, but I don't tell myself that it's haute cuisine.

Granted, like any framework, ORMs certainly have their uses, and most projects will benefit from using them in some capacity. No one likes writing the same boring INSERT, UPDATE and DELETE statements for every table. They enforce consistency - you essentially don't have a choice about naming conventions or class structure anymore, so you can't screw them up. They usually maintain relationships from the database as part of the code. Some have their own internal query cache. It's usually easy to extend them. Unfortunately, based upon my cursory research, there appears to be at least one attempt at a MongoDB ORM, so I can't use "don't support nosql" as a point in their favor anymore.

So by all means, use an ORM every time. By which I mean that every repository should probably have one, and emphatically not that it should be used for every query. Because whatever the tool at hand: Zend Framework, Class::ReluctantORM, a microwave--there always ends up being a place where it doesn't work, or doesn't work very well, and you're forced to do things the old-fashioned way. Sometimes the simple solution really is the best. Why would you bootstrap Zend and load up a bunch of model classes just to import some records? You can do that with a DBI handle and a perl script. Or flat files and sed/awk, probably. To some extent this is just a matter of opinion, and that's fair, but there are situations where my way - the ugly, hacky way - is objectively and demonstrably better. Not always. But sometimes.

By Way of Example

• In what might be the canonical case of "Why Would You Do This", a listing of articles with headlines and perhaps publication dates, with the titles linking to individual article pages containing the full text. To render this list, the ORM-written query was selecting all fields from the table. Because that's all it knew how to do; you ask for a list of articles and you get those articles, with no thought put into why you want them, or which fields you need, because it's a generic tool and that's all it knows how to do. Sometimes I think it helps to think of ORMs as the dumbest programmer you've ever worked with. Think of the query that guy would write, and that's probably similar to the inefficient unreadable gloop you're getting from the machine generation.
• A three-layer navigation menu, with almost all the items on it determined by what was, or wasn't in the database. After spending a few hours untangling what the thing was doing, it was something like this:

  
  select('e.event_id, e.name, e.url_name, i.url, 
   i.title, tr.title,  a.article_id, a.title, 
   ae.article_event_id, rg.title, rv.title,
   rm1.related_media_id, rm2.related_media_id, i.sort_order')
  ->from(CLASS . ' e')
  ->leftJoin('e.info_page i ON e.event_id = i.event_id 
   AND i.is_deployed IS TRUE AND i.pub_date <= NOW()')
  ->leftJoin('e.tour_results tr')
  ->leftJoin('e.articles ae')
  ->leftJoin('ae.article a ON a.article_id = ae.article_id 
   AND a.pub_date <= NOW() AND a.is_highlight IS TRUE 
   AND a.is_deployed IS TRUE')
  ->leftJoin('e.related_media rm1')
  ->leftJoin('rm1.photo_galleries rg 
   ON rg.photo_gallery_id = rm1.media_id 
   AND rm1.media_type = 'photo_gallery' 
   AND rg.is_highlight IS TRUE 
   AND rg.status = 1 AND rg.pub_date <= NOW()')
  ->leftJoin('e.related_media rm2')
  ->leftJoin('rm2.videos rv ON rv.video_id = rm2.media_id 
   AND rm2.media_type = 'video' AND rv.is_highlight IS TRUE 
   AND rv.status = 1 AND rv.pub_date <= NOW()')
  ->where('e.instance_id = ?', array(...))
  ->andWhere('e.deployed IS TRUE')
  ->orderBy('e.start_date ASC');
  
  

  This thing took around a second and a half to build and run the query, and returned 250 or so rows from the database. Then it took 30 more seconds to parse it all into a nested structure of PHP objects. And for all that, the developer still had to write most of the SQL themselves. Given that it made an entire section of the site unusable, and that the replacement, hand-wrought query (for all of it's faults) didn't, I'm content to throw our ORM under the bus here.
• A page to view poll results in a CMS admin. Either the ORM didn't support anything as simple as "SELECT COUNT(*) FROM answers GROUP BY answer_id", or the person who wrote it didn't think it was a problem to select 80,000 rows and then have PHP parse them into objects. Frankly I'm not thrilled by either alternative, and as you can probably guess, this thing ran out of memory and barfed on a pretty regular basis.

The root of the problem (as with most problems) is not thinking critically, not being aware that all this magical query dust doesn't come cheap.

You have to use the right tool for the job. It's not uncommon for the balance between generic, easy to use and quick to develop, and bespoke, laborious and highly performant, to tilt sharply towards the latter. The pain in the ass here is that it's not unusual for this sort of problem to lie dormant on a dev dataset (a dozen rows per table and just enough information to test out edge cases), and then one day rear up and slow your pages to a crawl or blow them up entirely, as soon as you hit real data. It's up to the developer to have some notion about seeing this coming. Even then, everyone gets bitten by this from time to time.

What it boils down to is that if you write a bad query, one that does "SELECT * FROM tbl_huge LEFT OUTER JOIN tbl_big_mclarge" and returns an unnecessarily wide data set full of BLOBs, or that joins across a dozen tables when it only really needs 3, or that has a big stupid slow "SELECT ... FROM ... WHERE NOT IN (SELECT ...)", or that tries to run a SQL "COUNT" in PHP, and it becomes a problem, it is your fault. I don't care if you wrote the thing yourself, or if you used an ORM and it wrote the query, it's still your fault, and you are going to have to fix it. "But that's the way the product does it" is not an acceptable response. Ever. For anything. Code is running on your servers. You are responsible for it. A microwave makes wretched chicken, so I guess it's time you learned how to work the stove, because I'm not eating that crap.

So by all means, use ORMs for your trivial cases, for basic stuff or where performance isn't an issue. But it's eventually going to hit a wall and you'll have to do your own dirty work. And when that happens, you can't say you weren't warned.

]]> http://omniti.com/seeds/the-opportunity-of-crises http://omniti.com/seeds/the-opportunity-of-crises Tue, 13 Sep 2011 19:57:35 GMT Nobody likes a crisis; they are difficult, troubling and sometimes dangerous. For most of us in web operations, the chances are slim that a crisis will be truly life-threatening, but when millions of dollars are on the line it can feel like a pressure cooker and have a negative impact on lifestyle, relationships and mental health. Most companies go to extreme lengths to avoid crises, and even when one does occur, the typical response is to first deal with it, and subsequently pretend it never happened. As if the memories are too painful to discuss, we avoid the topic all together; talking to your customers about it is quite risky. It is probably best not to mention it at all; you should just move on. Unfortunately, for most organizations, reacting like this means missing a grand opportunity to make your company better.

Like any organization, we are always on the lookout for new talent. Of course, you want people who are "smart and get things done", but beyond that, I have found one particular personality trait to be critical to long term success at OmniTI; the ability to stay calm in a crisis. While I tend to think OmniTI does well in avoiding them, we do have a tendency to attract customers with a lot on the line and, apparently, with a critical mass of customers, so we are no strangers to crises. While it is pretty clear to me that composure under stress is a fundamental requirement in high-stakes jobs (like large-scale web operations), I think it is generally helpful in any situation. Contingency planning can only get you so far, and when your packets are spilling all over the floor you need to keep a clear head about you to make sure you can assess and remediate as if you've done so since kindergarten. If you can't remain calm, the situation can deteriorate quickly. Turning to the blame game before solving the problem at hand is a sure sign of such deterioration. You must fight that urge. If you can't, your team can't be as open with communications as you need them to be, and your recovery time will suffer. Be upfront about how you want your teams to respond, ideally before problems arise. There are real crises in the world where people die at the hands of companies; walking though one of these exercises can be humbling and enlightening; James Lukaszewski takes us through a "Death by Burger" scenario in his Seven Dimensions of Crisis Communication Management, and outlines positive and negative ways that a company can respond to such an incident.

That said, resolving a crisis should not only be about solving the problem at hand. When calamities occur, it's important to recognize that your company has an opportunity for introspection. What is it about your processes that led to the crisis you've just survived? Do your process and tool chains do everything they need to do? Don't just determine if they work, but do they do the job in they way you would like the job to be done. Seldom will you arrive at good answers to these questions through the normal course of business. Even if you think failure is human (perhaps especially when you think so), it's important to understand what processes failed or what information was unavailable that led to this human error. That information is crucial because, in most cases, the people on your team are acting in a manner they think is safe and appropriate -- and in the best interest of the company. The knee-jerk reaction in these cases is often a summary dismissal, but that will often leave you with the core issue unaddressed: they thought their actions were acceptable. If you fail to gain an understanding of the underlying causes, this bleak episode is likely to become a rerun; either with a new employee, or perhaps with an existing team member who also doesn't understand where the appropriate lines need to be drawn.

One thing I believe is very helpful is to look at how others handle these things. The Internet is new, Web Operations even newer; but crisis management and postmortem analysis are not. Quite often I see people lay blame at either the wrong people or processes in times of (and even after) failure. Ideally you should not be trying to lay blame at all, but instead figure out where improvements are needed. Many people mistakenly assume that crises are born out of mistakes; often they are not. As businesses grow over time, it's easy for plans that were once appropriate to become inadequate. You need to look at your systems holistically. For folks in Web Operations, a healthy understanding of why complex systems fail can help you gain a better perspective.

If you are running a team, you owe it to yourself to turn crises into dialogue. If your customers were affected, be honest with them about where things went wrong, and why what you did was the appropriate thing or how you plan to adjust course going forward. Be careful not to overreact; the goal should not be to add more process, but rather to improve process. Your next crisis is your next great opportunity to learn more about your organization and to strengthen it for the future. Don't miss it!

]]> http://omniti.com/seeds/security-is-not-a-feature-its-a-state-of-mind http://omniti.com/seeds/security-is-not-a-feature-its-a-state-of-mind Thu, 11 Aug 2011 18:01:05 GMT "Is our site secure?"

That is never a question you want to hear when launching a new website. And it is also an impossible question to answer. The technical definition of security is "the state of being free from danger or injury"; but you can never protect anything perfectly all the time. So the question should be, "Is our site secure enough?"

And it is never a question that should be asked at launch time. Security must be part of the planning, part of the programming, part of the testing, part of the deployment process and, finally, part of the monitoring and upkeep of a site. It should be part of every stage of development, however, "security at every level" doesn't have to cost extra time or money, and in fact it shouldn't. If "site security" means millions of dollars or weeks of extra work, then the problem lies with the website development process and its creators. When you start treating security as a "feature" instead of a necessary part of your development process, it becomes a resource eating monster.

Far too many IT professionals, managers and others involved in creating new sites view security as a last-minute feature in the push to "get it out fast." In fact, generally the argument for not attending to it is: "We can't waste time or money on security features; we have to get this site launched!"

You won't have a site, or a business, for long if someone manages to retrieve your entire database of personal information or credit cards. Like most things in life, a balancing act is required. Defining what security is exactly, and what it means for a site is always the hard part. What does "security" mean in a website or a web application? It means being defensive.

"Defensive driving" is a term thrown at every student in drivers training. It means to drive as if every other person on the road were an idiot trying to hit you, because the majority of them are less-than-fantastic drivers and being aware of the danger is half of the solution.

Any developer working on a website should be thinking in the same manner: Every user is an idiot trying to break the site. However, the reaction to that constant danger should be equal to the needs of the website. When driving on a sunny, dry road in broad daylight, a driver can be far less diligent than when driving on a wet road, in a blizzard or in the dark. The conditions of the road are going to affect stopping distance, maneuverability and the ability to avoid hazards. The amount of diligence needed for a website should be equally tailored to environmental conditions. An e-commerce site has far different needs than a social networking site, or a fan site for an author or artist.

Having a plan--from the beginning--for the important issues with the site is a necessary first step. Implementing the plan as part of your general process shouldn't be the end of the line, however. The other critical piece of the puzzle is ongoing maintenance. Sites and audiences change, and those changes will mean new challenges. Proper monitoring and maintenance of a site is part of the process of security.

Knowing a site's operating environment and type of users will help to define what security measures are needed up front, eliminating the problems inherent with trying to "bolt on" security after the fact. Even a general overview of what kind of information a site is going to collect and distribute is enough to have an idea of what kind of audience that site will attract. It is far easier to leave room for future security enhancements than to try to plug holes in an existing system.

So take the time to sit down before you start creating the site and answer some of the following questions. Record the answers in a document and put it with your code so you can refer back to the answers.

1. What kind of data am I going to be collecting and storing?
1. Basic Information (Names and email addresses)
2. Personal Information (Phone numbers, physical addresses)
3. Asset Information (Credit Card numbers, bank information)
4. Identifying information (SS#, Drivers license numbers)
5. Business Information
6. Medical Information
2. What kind of physical system am I going to be using and who has access? This includes backups.
3. What kind of software am I going to be using and how will it be maintained?
4. What kind of ongoing system will be put in place to maintain the system and data?

These questions will give you an excellent idea of how much concern for security your site will warrant. The higher the level of information collected, the greater security you'll need. The less control you have over the physical systems in place, the more diligent your security measures need to be. The less control you have over the software in place, the more security measures you may need to put in place. If you have little budget for ongoing monitoring, you'll need to invest more in automating more security measures up front.

Remember that no matter what kind of site you are creating, the basics can never be ignored.

1. Keep your software up to date with security fixes
2. Validate all input
3. Escape all output
4. If you're dealing with something sensitive - use SSL for logins (the industry is showing signs of adopting SSL for everything).
5. Use sftp or scp or at the very least ftps for transferring files from your server
6. Regenerate a user's session when access permissions change
7. Validation should always be done server side, even if you have javascript checks

If security becomes part of your state of mind at every step along the way, instead of a last-minute, add-on feature, you'll never have to answer the question "is our site secure?" because you'll always be aware that it is secure enough.

]]> http://omniti.com/seeds/when-things-go-wrong-a-case-study http://omniti.com/seeds/when-things-go-wrong-a-case-study Wed, 22 Jun 2011 14:59:24 GMT Theo Schlossnagle is very fond of pointing out that in operations, you can never succeed in fulfilling expectations.

"Operations crews are responsible for the impossible: it must be up and functioning all the time. This is an expectation that one can never exceed." (Instrumentation and observability)

So, this article is about a time when things went wrong. It's not about an emergency situation where services were down, but more a subtle issue that almost went unnoticed. We will review how the issue was detected, how it was fixed—and most importantly—how a root cause was determined.

This issue affected a production website for one of OmniTI's clients. They had three web servers, all connected through a front-end load balancer appliance. (There were database servers as well, but they aren't relevant for this story.) Like any good (or even half decent) load balancer, it checks often (every few seconds) to ensure that that the web servers are up and serving web pages. If the web server appears down or isn't responding, then the load balancer stops directing traffic to that web server. This gives you a measure of redundancy in addition to load balancing, if you have enough web servers to cover the incoming requests even with some out of commission.

The problem was uncovered by chance when working on the load balancer. We spotted that the load balancer was misdetecting that a server was down, taking it out of service, and a few seconds later on the next check, it would bring the server back into the rotation. This cycle repeated over and over, with each of the web servers being taken out of service for a short period. During this time, the site was still available: at least one of the servers was continually in service. In addition, our monitoring showed that everything appeared to be OK, both the external checks against the main website, and the checks against the individual web servers.

Having discovered the issue, the troubleshooting began. One of the first things to look at with any issue are log files. When set up properly, logs go a long way in telling you what is going on; the hard part is figuring out which logs have the information you need.

The first log file we checked was the load balancer log. It had entries that looked like the following, that corresponded with the service failures:

Monitor_http302_of_foowidgets-www1:http(192.168.1.51:80): DOWN; Last response: Failure - TCP syn sent bad ack received with fin

So, according to the load balancer, the reason for the failure is 'TCP syn sent bad ack received with fin'. The error message is highly technical and not particularly helpful.

Here's a quick (and incomplete) overview of TCP to explain what that means:

When you open a connection, packets are sent back and forth with various flags set - the relevant flags here are SYN, ACK, and FIN. The opening sequence goes something like:



And to close the connection:



The explanation for the "syn sent bad ack received with fin" error is likely to be:



At this point, the load balancer gets very upset and sulks in the corner (well, it prints the weird log message). I'm guessing that the bad ACK/FIN is probably from a previous connection, but at a high level: "Something weird is going on with networking".

When things are screwy with the networking, you need to look in detail at what is going on across the network and try to work out what's going wrong. The tools to do this are tcpdump and wireshark.

The load balancer is an appliance with its own custom software, but underneath it's just Unix. You can get a shell and run tcpdump to see what is going across the network. Wireshark is essentially a graphical version of tcpdump and is used here to analyze the network traffic.

I grabbed all of the traffic, opened it with wireshark, and limited the view to just the traffic going to the web servers exhibiting the problem. The wireshark filter is:

ip.addr == 192.168.1.254 &&
    ( ip.addr == 192.168.1.51 ||
      ip.addr == 192.168.1.52 ||
      ip.addr == 192.168.1.53 ) &&
    tcp.port = 80

The 192.168.1.254 IP is the load balancer, and 51-53 are the web servers.

The following is the output of two complete HTTP transactions on the monitors:



There is no SYN sent with bad FIN/ACK; everything is green and looks just fine. So I rechecked the load balancer console, and it showed eight checks went out in the same time frame as the tcpdump. However, I saw only two in the previous tcpdump. Something wasn't right.

At the same time that this was going on, I was in touch with support for the load balancer vendor. They were very helpful (in the sense that they did everything they could without actually getting to the bottom of the problem), asking for several tcpdump traces, and even escalating to their engineering team. At this point we were convinced that the monitor that checks whether the server is down was broken, and reporting the server down, when it wasn't. Unfortunately, none of this shed any light on the underlying issue.

We had also checked the usual culprits:

• Checked that the backend servers were up
• Checked the physical cables. Other services were transiting the same physical path and they were fine.
• Ran tcpdump on the server's network interface that linked it to the load balancer, this showed the same thing as the dump from the load balancer.
• Checked the configuration of the monitor on the load balancer. Other services were using an identical configuration without issues.

Then, the crucial discovery:

• Not enough checks were being sent out (we spotted this before)
• The support representative casually mentioned seeing traffic going out of the load balancer through another MIP. He thought maybe some of the checks were going out of this other IP.
• None of us realized the significance of this at the time, and a couple of days went by—with support convinced there was an issue with the backend servers, and me running as many checks as I could to try to prove/disprove that the backend servers were an issue.

Here's a bit of explanation of what was going on:

The load balancer has 3 different types of IPs (simplifying a little):

• VIP - Virtual IP
• MIP - Mapped IP
• SNIP - Subnet IP

A VIP is an IP upon which you run your virtual servers. These are what client traffic hits.

A MIP is described as: "You use MIP addresses to connect to the backend servers." (from the vendor's knowledge base).

A SNIP is described as: ". . .an IP address that enables you to access a load balancer appliance from an external host that exists on another subnet." (from the vendor's knowledge base).

From the explanation above, it makes sense that you would configure a MIP to connect to the backend server. This is what we did when originally setting up the load balancer, and it turned out to be completely the wrong thing to do, although it did work for a while.

Some more explanation - Multihomed networking 101:

1. A server has two interfaces on two different subnets - 192.168.1.0/24 and 192.168.2.0/24.
2. The server wants to send a packet out to 192.168.2.10. To do this, it looks up the address in the routing table and sees that it should send the packet out of the second interface.
3. Also, the source IP of the packet sent out is the IP address that is associated with the second interface. For example: 192.168.2.2.


If you're familiar with networking, the above explanation should sound pretty straightforward. The problem is, the MIPs on the load balancer don't work like that. Things work the same for steps 1 and 2 above, but instead of matching the source IP address of the packet to the interface it's sending out on, it just picks one IP from the list of available IP addresses:



Now, to be fair to the load balancer vendor, this is the correct behavior for MIPs when you read more into what they are for. They're 'last resort' source IPs when nothing else is suitable, (i.e. you don't have a matching IP on the same subnet). Because it's a 'last resort' IP, it just picks one.

We had 3 different MIPs at the time, one for an external network, one for the client network, and one for our internal network. This meant that fully two-thirds of traffic from the load balancer was getting sent out from the wrong IP:

• 192.168.1.254
• 192.168.2.254 - wrong network
• 1.2.3.254 - external network

Believe it or not, this shouldn't have mattered, and in fact didn't matter for most of our services. The default route of the server was through the load balancer - it had to be to answer client requests, which came from external IP addresses.

However, by a horrible quirk of routing, on the backend web servers, 192.168.2.X was set to go out on a different interface, and traffic wasn't getting sent back to the load balancer, meaning 1 in 3 monitor responses weren't getting sent back.

This also meant that each web server was not serving traffic 33 percent of the time and we were effectively running off of two web servers. If the right combination of monitors went off, all three servers could be taken out of the rotation.

The temporary fix was to make sure that 192.168.2.0/24 went out via the load balancer. A single command fix gave a 50 percent capacity boost. The fix was just in the nick of time, too — just four days after I made the fix, the site was featured on the front page of msn.com and we got the biggest traffic spike ever in its history.

This fix was only temporary, as we were still having checks originate from the wrong IP, and the real fix was to use the Subnet IPs which, as their name suggests, actually respect subnetting.

As with all complex systems, the problem was caused by a number of different issues combined:

• The documentation was misleading, which lead to the wrong IP type being configured on the load balancer.
• Multiple networks on the backend server combined with virtualization, which led to incorrect routing when combined with the check originating from the wrong address.
• Fault tolerant systems combined with the tiny outage duration for individual web servers masking the issue and greatly increasing time to detection.

Experience is always earned the hard way. Having things go wrong leads to a deeper understanding of how complex systems work, as shown in the example above—at the end, we had a much better grasp of the inner workings of the system than before. Gaining this understanding is essential to prepare you for new technologies and production troubleshooting.

When a technology "just works," it is pretty much guaranteed that you don't know "how" it works (at least deeply). The real challenge is building architectures where usual, run-of-the-mill mistakes cause no disruption of service. That is an art and the artwork is an invaluable resource to the organization: it provides a canvas for learning and gaining hard-won experience.

]]> http://omniti.com/seeds/write-a-better-fm http://omniti.com/seeds/write-a-better-fm Fri, 22 Apr 2011 04:25:53 GMT If you've been around software for any time at all, you've encountered the type. You ask what seems to you as a reasonable question, and the belligerent sorts fall over themselves to be unhelpful; calling you lazy, an idiot, or worse, and telling you to RTFM. If you're lucky, they'll tell you where the FM is. If you're not, they'll tell you to STFW for it.

For those not up on the acronyms, RTFM and STFW stand for Read The Manual, and Search The Web, respectively.

The trouble is that there's a direct correlation between the probability that you'll be told to RTFM, and the probability that the FM is rubbish. That's because humility, patience and a willingness to help beginners go hand-in-hand with producing good documentation.

The burden falls on those of us within the software community to write a better FM.

For the last ten years or so, I've been involved in several efforts to write better documentation on several Open Source projects. I've noticed some trends in documentation. While some projects have stuck to the tried-and-not-so-true, RTFM, "the newbie is a loser" style of customer support, an increasing number of projects have moved to customer-centered documentation and customer-centered support.

If we want customers to RTFM, we are obliged to write a better FM.

I'm Not a "User"

Respect is very important. If you are unable to treat beginners with patience and respect, you shouldn't be doing customer support, or writing documentation. While there are places for banter and inside jokes, technical documentation is not one of them.

Thinking of your audience as customers, rather than as "users", or, worse yet, "lusers", will greatly influence how you write.

Listen To the Questions

While this may seem blindingly obvious, it's clear from the documentation of many products, and in particular their so-called 'Frequently Asked Questions', that they have no idea whatsoever what questions actual users of their product are asking.

To know what questions are being asked, you should frequent the places where these frustrated users hang out after not finding the answers in your documentation. These places tend to be web forums full of bad advice and broken examples. When you feel your irritation building, remember that they exist because your documentation wasn't filling the need, and so these third-party sites sprung up to fill it.

You will quickly find that they tend to be full of people asking the same questions, again and again, and getting a variety of answers of varying quality. It is now your job to make sure that the official documentation answers these questions correctly, showing best-practice solutions to the real-world problems, and makes them easy to find. Failure to do so simply drives people to these question-and-answer sites where they will continue to get bad advice.

Ask Smart Questions

Several years ago, Eric Raymond wrote a document about how to ask smart questions. While this seemed like a good idea at the time, it has since become a lengthy tome that no beginner will ever actually read, and which drips with condescension.

The document states three things:

1. Try to find the answer yourself before asking.
2. Provide all relevant supporting data with your question.
3. If you don't understand the answer, it's probably because you're too stupid to live.

Points 1 and 2 are good, right and important. Unfortunately, point 3 colors the tone of the whole document. Indeed, the word "idiots" appears first in the second paragraph. Although it seems that Eric thinks he's being funny, instead he insults everyone who doesn't know as much as him.

Down at the very bottom of the document is a section titled "How To Answer Questions in a Helpful Way." This is the most (and perhaps the only) useful part of the document, and well worth reading.

While it is indeed important to ask smart questions, and not expect that someone is going to hold your hand every step of the way, as documentation authors, it's important to cast our minds back to when we first started--how lost we felt, and how we didn't even know what questions to ask. If that was too long ago, you can readily refresh your mind by picking a software product you're unfamiliar with, in a language you don't know, and trying to install it and get it running. It will all come back to you.

As Donald Rumsfeld famously remarked, there are also unknown unknowns, the ones we don't know we don't know. Start by assuming that your customer isn't an imbecile, but that they may not know what questions they should be asking.

Help your customers know what questions to ask by structuring your documentation in terms of how they are going to use it. Segment the documentation by audience (Developer, User, Administrator), and then further by task (Installation, Reporting, Upgrading) rather than in seemingly arbitrary groupings like "How-To" and "Other Topics".

No Stupid Questions

You've often heard it said that there are no stupid questions. While this is obviously false, much documentation seems to start with the assumption that all questions are stupid. There is a middle ground.

You must start by assuming that questions are smart and useful.

Frequently when I'm watching in an IRC channel, someone will ask "How do I do X?", and the immediate response is "You don't do X, you idiot! That's a stupid thing to do! Did your mother ever drop you on your head?"

Rather than treating them like a teenager, try to imagine that the person asking the question is a professional, like yourself, working on a project that might not have been their idea, but that nevertheless they need to get working.

Likewise, when writing documentation, keep in mind the real-world problems that are faced when using your product. Remember that not everyone has the in-depth knowledge of the inner workings that you do. And, most importantly, remember to treat your customers with respect, all the time.

In practical terms, this means avoiding words like "obviously", "simply", and "just", while providing many immediately usable examples with detailed explanations of each point.

Laziness, Impatience and Hubris

Larry Wall once declared that the virtues of a programmer are laziness, impatience and hubris. This is often taken as license to be a jerk. I would assert, to the contrary, that the virtues of a documentation writer are laziness, patience and humility.

Yes, it's important to be lazy. When someone asks a good question, answer them thoroughly, in exhaustive detail, and then publish the response so that the next time the question is asked, you can answer with a URL. Doing something well once beats doing it poorly, again and again.

You must be patient. Being impatient with a customer implies that you think they are either being intentionally obtuse, or that they are just too stupid to understand what you are saying. Being patient with them shows respect. The patient, respectful answer will stick with them, while the impatient rude answer will be remembered only as an unpleasant experience to put behind them.

And you should be humble. I find it useful to remember that the person I'm talking to is probably an expert on something of which I am completely ignorant. I also find it helpful to remember the first time I was asking questions, and the way that I was treated at the time.

]]> http://omniti.com/seeds/integrating-search-into-your-application http://omniti.com/seeds/integrating-search-into-your-application Tue, 12 Apr 2011 18:12:17 GMT

Why Do It?

This may seem like silly a question at first, as search seems to be everywhere and its usefulness is apparent in so many contexts. The context you should consider though is your application. Ask yourself how search might benefit your application and your users. Keep in mind that search is not a replacement for a good user interface. A good user interface should make it easy for the user to locate what they're looking for on your site, without typing anything into a search box.

There are many browse-dominant users out there that prefer to click rather than put their hands to the keyboard, no matter how prominently you display a search box or how well your search performs.

It shouldn't be a chore for the user to find the contact information on your web site. I've looked at thousands of crappy flash restaurant web sites. If you have a search input box on your website and I have to type "address" into it because I can't find it on the site, there is a problem.

A bad search implementation can hurt the usability of your website. If your search functionality is very unfriendly or unaccommodating when it comes to search terms, people will become frustrated and potentially give up on what they wanted to do at your site. This is especially true of search-dominant users of your site. Your search should be able to handle commas, apostrophes, hyphens and other punctuation.

Deciding What you Need

When implementing search capabilities for a web application, many developers might rush into integrating a known solution without asking a few key questions first. While these questions may seem a given and lead you to the same tool, many people don't ponder too deeply about them and the result is an adequate, but not optimal, search functionality for the application. Let's take a look at some initial considerations.

1. What should be searchable in my application?

Let's say I have an online shopping cart. It seems reasonable that I would want my users to search the products. Which products exactly? Not products that haven't been published from admin yet or discontinued products; and perhaps I don't want to show products that are out of stock. Lay it all out. Everything that you expect to be searchable and the conditions that have to be met.

2. How do I want my users to interact with the search functionality?

How flexible will the search be with what user's type in? Will it show automatic suggestions as the user types? Will it attempt to fix misspellings?

3. Direct linking?

Should particular search terms go straight to a particular page instead of showing results? If there was only one result, it seems obvious to advance the user directly to their goal. Sometimes, in the case of certain search terms, you may still want to lead the user on a very specific journey.

4. How current do search results have to be?

Is it imperative that the product/blog post I just added be immediately available in the search results? If not, how much lag time is acceptable?

5. What kind of search options do I want to provide the user?

Are there advanced search options, including date ranges, sort order and relevancy? Note that you shouldn't overload the user with advanced search options from the start. There should be a simplified version of search that is the default, however the advanced options should be easily accessible.

6. Do certain results rank or weigh higher than others?

For example, If I search for tomato, does your blog post about your grandma's spaghetti recipe come up before the result for the contact page that has my address as 123 Tomato St?

Initial Setup

Once you've decided what is best for your application and its users with regard to the application's search functionality, you can start looking around at the available tools to implement it. You will want to map your search functionality needs to the capabilities provided for by the search tools. See how well these tools perform. Users expect search to be fast, they really don't care how much information you have to go through to find what they want. More than likely, the information you want to search is stored in a database. Ideally, one does not want to do full text searches against the database. It is expensive for the database and if you have a high-traffic site where searches are going to be performed fairly often, steer clear of it.

Instead, many popular search tools bring search outside of the database scope by indexing the data you want searched in your database. This usually means that I will choose the table columns that have the information I want available for search, such as product name and product description. I would then create a script that pulls this data and adds it to my index (note that index size is usually 20-30% of the initial data being indexed, depending on the search tool). You will more than likely want to run this script from a cron job to refresh your index on an interval that is dependent upon your needs. Note that you can add and delete from the index as items are inserted and deleted from your database. This means that your need to refresh the entire index may change if you use this approach.

A big chunk of the work that you will face is the initial index setup work required to provide for the features and conditions you want in place for your search. Features such as wild-card queries, sorting, field weighting, multiple merged indexes, multi-faceting, ranking, result clustering and date ranges are some examples.

Presenting the Results

Depending upon your application and how advanced you want your search to be, you can make assumptions and educated guesses about the intended results. Amazon is a good example. As of the date of this article, typing in "Black Swan" into the search box returns results for "The King's Speech", "The Fighter", and "True Grit"--all within the top 10 search results. Amazon is making the assumption that I may be interested in other Academy Award winning movies. Their end goal is that I will purchase those, as well. This makes sense for Amazon, does it make sense for your application? How search-centered is your application? Factoring in these types of assumptions makes for a lot more complexity in your search application. How long before those Academy Award related search results fade away and I am left with only "Black Swan" results?

Most results are displayed by relevancy, which makes sense; sometimes it makes sense to sort the results by date and can be a helpful option to those perusing the result set. Providing match context in your results can also be helpful. Consider users whose search term matches an exact phrase in an article on the site, but in the search results, only the article title is displayed. While it may be the very article the user was searching for, the user may not realize that they found what they were looking for in that search. Showing match context for the search terms in your display results, where applicable, can be helpful to users.

Often, a user will know meta information about what they want. Information such as, "I know it was in a blog post" or "I know it posted around Christmas time." While we are not mind readers, we can still make it easier for our users to find what they want, based upon the meta data they know.

One way to do this is by clustering your search results into relevant groups. When displaying results, instead of showing the relevant results for everything mixed together, show the relevant results grouped by a strong meta identifier, some example relevancy groups being "Articles", "Products", and "Users", or perhaps by year 2009, 2010, 2011.

Is there a mobile version of your application? How do your search results look there?

Many times, site searches are implemented with Google site search. This is not a bad idea depending upon your site content and search requirements, however keeping those search results contained on your site as opposed to sending the user to a google site search result page, keeps the user engaged on your site and is less confusing than being redirected. Google site search provides for this functionality.

Avoid presenting your search results to look like something from a Google text ad. Many people will think that is what it is and it will be ignored.

Post Implementation

What are people actually searching for? Are you monitoring the queries that are coming through your search form? What are some of the top queries? What results are being given to the user for these queries? Are they what is expected? Most developers will implement search functionality and make sure that it functions, but fail to monitor or provide tests to ensure the search is actually useful after being implemented.

Search is an important part of the web, and the technology behind it is becoming smarter and faster. Take advantage of it, but first take the time to discover your application needs and how you can best serve your users.

]]> http://omniti.com/seeds/the-web-developers-guide-to-writing-native-ios-apps http://omniti.com/seeds/the-web-developers-guide-to-writing-native-ios-apps Wed, 06 Apr 2011 17:04:14 GMT Ever since the release of the first iPhone and the first official iPhone/iOS SDK, mobile computing has taken a huge leap into the handheld realm. Where only a few years ago, it was normal to see hipsters hunched over laptops while sitting at Starbucks sipping their lattes, and surfing the web on the free wifi; nowadays they've broken free from their local coffee franchises (although still addicted to the caffeine) and roam the outside world, still surfing websites but now on their iPhones over 3G connections.

So how do you, as a veteran web developer, take advantage of this phenomena to write really cool mobile apps that engage the user on a whole new level? Well, there are several ways. You can bite the bullet and learn Objective-C, CocoaTouch, iOS SDK, and spend several months writing a true, native iPhone app. You can stay in your comfort-zone and write mobile-sized HTML5 web apps that sit, hands-tied to your servers with very limited device-centric capabilities. You can cheat a little bit and use PhoneGap or some other library to write HTML5 mobile web pages that can be submitted as apps to the App store. Or, you can cheat a lot and use Appcelerator to write your code in a language you're very familiar with, on a cross-platform API with practically all the device capabilities available to you--all resulting in native apps that can be submitted to the App Store and the Android Market.

What is Appcelerator?

Appcelerator is hard to define. It's an open source framework that acts partially like a compiler and partially like a runtime. Without going into the nitty-gritty about how it does its thing, let's just say Appcelerator will take Javascript code built on top of its API and turn it into a native application for the iOS and the Android platforms. For the iOS platform, Appcelerator's Javascript API maps to the Objective-C/CocoaTouch equivalents; and for the Android platform it maps to the Android Java framework. After the code is compiled, you end up with the respective platform's native binary app. Also, Appcelerator has a third mode which lets you write cross-platform WebKit-based desktop apps. We'll only be concentrating on the mobile side of things in this article.

Appcelerator comes in two parts, Titanium Mobile SDK and Titanium Developer. The Titanium Mobile SDK is the heart of the mobile app writing framework. Titanium Developer is a fancy, front-end GUI that lets you set up various environment settings, run the code in a simulator, compile it for a real mobile device and even package it for distribution to the Apple Store. As a side note, Titanium Developer itself is written using the Appcelerator Desktop SDK.

Installing Appcelerator

Required Prerequisites:

• Very good working knowledge of Javascript (knowledge of closures are a plus)
• An Intel-based Mac of some sort
• Xcode with the iOS SDK installed (http://developer.apple.com/ios/)
• Android SDK installed (http://developer.android.com/sdk)

Prerequisites if you want to run on an actual iOS device:

• Pay Apple the $99/yr fee and register as an iOS developer
• Create/download all the keys/certificates (just follow the directions they provide on http://developer.apple.com/devcenter/ios/ under the iOS Provisioning Portal)
• Add your mobile devices to the developer account
• Create an AppID for your app
• Create a Provisioning Profile that binds your AppID to the various devices you registered (this is the way to get adhoc distribution to developer iPhones for testing)
• Download all those certificates and provisioning profiles, install them and configure your Appcelerator app to use them when compiling, so the code can be signed correctly

Running the gauntlet of getting the environment setup to test on an actual iOS device requires another article entirely. However, once you get through it the first time, it gets easier. Just follow the instructions on Apple's site. Here, we will just stick with running things in the iOS Simulator.

You'll notice that an Android SDK must be installed even though we're only doing iOS development. This seems to be a nagging requirement in order for Titanium Developer to get going (things may get fixed in later releases so try it out without it first, but if it complains, go ahead and install the Android SDK).

Making a New Project:

1. Create a New Project
1. Open up Titanium Developer
2. Select New Project
3. Select Mobile as the Project Type
4. Fill in the other fields (make sure App Id exactly matches the AppID you registered in Apple's developer website)
5. Click "Create Project"
2. You'll be taken to the Project Settings window (the Edit tab)
1. Just click "Save Changes"
3. Now click the "Test & Package" tab, you'll see 3 sub tabs:
1. Run Emulator - Runs your program in the emulator
2. Run on Device - Runs it on a developer iPhone/iPod Touch (you'll need provisioning profiles setup)
3. Distribute - Packages up the app for submission to the App Store (or adhoc distribution to non-developer profiled iPhones/iPod Touches)
4. Click on the "Run Emulator" tab and then click on the "iPhone" subtab Ensure that the SDK is the latest iOS SDK version and click "Launch"

If all goes well you'll see a bunch of messages scroll past while the project code is compiling, and then the iOS Simulator launches.

Congratulations, you've successfully run the skeleton code!

Modifying the Code

Open up the directory you told Titanium Developer to create, all your code and assets will be stored in the Resources/ subdirectory. You'll notice that it already contains the app.js skeletal code; this is essentially the entry point into your app (ie. your main() routine). You'll also notice iphone/ and android/ subfolders. This is where you keep assets that will override the default assets you have in your Resources folder for when you need to target the particular platform.

Now let's actually write some code, rename/move the existing app.js file to app.js.old (be aware that any file with the .js extension will get compiled by Titanium Developer by default even if it isn't used in your end product). Create a new file called app.js and enter the following code:

Ti.API.info('Creating a new root window');

var w = Titanium.UI.createWindow({ backgroundColor: 'white' });
w.open();


Ti.API.info('Creating Label');

var label1 = Titanium.UI.createLabel({
    text: "Name Please",
    backgroundColor: 'gray',
    color: '#000000',
    top: 20,
    height: 'auto'
});
w.add(label1);   // add the label to the window


Ti.API.info('Creating Text Input Field');

var textfield1 = Titanium.UI.createTextField({
    backgroundColor: 'green',
    hintText: "Type Here",
    height: 35,
    top: 100,
    left: 10,
    right: 10,
    borderStyle: Titanium.UI.INPUT_BORDERSTYLE_ROUNDED
}); 
w.add(textfield1);  // add the textfield to the window


Ti.API.info('Creating Button');

var button1 = Titanium.UI.createButton({
    title: 'Click Me',
    width: 150,
    height: 30
});
w.add(button1);   // add the button to the window


Ti.API.info('Adding an eventListener to the button');


// Here's the button 'click' event listener, 
// notice the second parameter is an anonymous function with
// a parameter 'e', this is the event dictionary.

button1.addEventListener('click', function(e) {
    Ti.API.info('Button was clicked, e is ');
    Ti.API.info(e);
    Ti.API.info('Text field has value ' + textfield1.value);

    if(textfield1.value.length <= 0) {
        alert("Enter your name please");
    }
    else {
        label1.text = "Hello " + textfield1.value;
    }
});

Click on the Simulator tab and launch the app. If everything works, you'll be shown a white window with a gray background label that says "Name Please", a text field and a button that says "Click Me". Go ahead and try clicking the button. You'll see an alert popup message telling you to enter a name. If you look through the button1.addEventListener callback function, you'll see where that alert() is coming from. Now enter your name in the text field and click the button again, this time you'll notice the gray label up top change to say "Hello xxxx" (where xxxx is what you entered in the text field).

You'll notice that we don't have any main() functions or event loops; all that stuff is handled by the underlying iOS SDK. We're developing on essentially an asynchronous event-driven model (similar to the Javascript model in browsers).

Congratulations on writing your first app! Your next steps should be to browse through and experiment with the various API methods available in the Titanium Mobile SDK found at http://developer.appcelerator.com/documentation. Also be sure to read the Getting Started guides: http://wiki.appcelerator.org/display/guides/Getting+Started+with+Titanium as well as the Getting Started with the KitchenSink demo app: http://wiki.appcelerator.org/display/guides/Getting+Started+with+Kitchen+Sink

Programming Notes

Old-School Debugging

The best way to debug is to throw Titanium.API.info('some debugging message') all over your code. You can also use the name Ti rather than Titanium (ie. Ti.API.info('some message') ) to save you from having to type so much. It's also worth noting that you can usually pass arbitrary objects/variables by themselves to Ti.API.info and it will try to print out the string equivalent of the object (if available). Ti.API.warn() and Ti.API.error() are also available for logging purposes (they'll show up in the Titanium Developer console in different colors).

Subtleties of createWindow()

You can pass in a javascript filename to createWindow() using the dictionary key url, this way you can modularize your code into smaller chunks. The createWindow() method will essentially fire off the new window in a separate thread and you won't really have access to any variables or functions within it after its launched. You can send initial data into the new window by adding arbitrary key/values to the dictionary parameter of createWindow. From within the new window script you can get access to those intial variables via the Titanium.UI.currentWindow variable. For example if you had code like:

  var w = Titanium.UI.createWindow({ 
    url: 'newWindow.js',
    foo: 'bar', baz: ['zab', 'rab'] 
});

then, within the newWindow.js file you'll be able to get access to foo and baz like so:

Titanium.UI.currentWindow.foo      // gives us 'bar'
Titanium.UI.currentWindow.baz     // gives us ['zab','rab']

The only way to get data back out of the window is to use the event handling facilities of the framework. And on that note...

Event Handling is powerful asynchronous communications stuff

Make liberal use of the event mechanisms of the framework to communicate between various threads and subsystems in your app. In addition to the built-in events, addEventListener can listen to any arbitrary event name you want. The complementary function, fireEvent allows you to fire any arbitrary event name you like, with any parameters you like, in a dictionary that gets passed on to the event callback function. Practically every object in the Titanium SDK has the ability to fire or listen to events. The most globally accessible object is the Titanium.App object: you can write an event listener to Titanium.App.addEventListener and do a Titanium.App.fireEvent from completely different areas of your app (see the example I provided above). This allows you a great deal of power for inter-thread communications (i.e., sending messages between windows and the like). Any number of event callback functions can be added via the addEventListener, and they'll all be called in turn when the event is fired. You can also remove a call back function from an event listener by using removeEventListener. If you fire an event that isn't being listened to, it will just be ignored, no harm no foul. Events aren't queued forever so if you aren't listening on an event when it's fired and you missed it too bad.

Here's a quick example (we'll attach the event listener to the global Titanium.App object):

Titanium.App.addEventListener('fooEvent', function(e) {
     // e will hold { bar: 'rab', zab: 'baz', 
     // and some other core event fields }
     Ti.API.info('fooEvent was called and ' +
                      'we got this event dictionary:');
     Ti.API.info(e);   
});

// ... meanwhile somewhere else ...
Titanium.App.fireEvent('fooEvent', { bar: 'rab', zab: 'baz' });

Closures will be your best friends

As your applications get more complicated, you'll be writing a lot of addEventListener type code throughout. It is essential that you keep in mind in what context that event callback function will get called, and what variables are available in that scope. If you create/assign an event listener function within a scope and make use of a variable that is only available in that scope, then when the listener callback function is actually called (some time later way outside of the scope), you may not have access to that variable any more as it has long since been destroyed. This is a common enough thing in Javascript and the workaround is using Javascript Closures. There are plenty of articles out there that explain it in great detail but let's just say it's a nice way to bind a scope to a function so that when the function actually does get called the scope is available along with the variables. Here's a way of writing that button1 'click' event listener more robustly using a closure:

button1.addEventListener('click', (function (lbl1, txt1) {
    return function(e) {
         Ti.API.info('Button was clicked, e is ');
         Ti.API.info(e);
         Ti.API.info('Text field has value ' + txt1.value);

         if(txt1.value.length <= 0) {
             alert("Enter your name please");
         }
         else {
             lbl1.text = "Hello " + txt1.value;
         }
    };
}) (label1, textfield1) );

I realize that it looks a little weird, but if you study it carefully you'll see that we are actually creating an anonymous function that takes two parameters (lbl1 and txt1). Then, we're immediately executing that anonymous function, passing in the variables (label1 and textfield1) as the parameters. We do all that in one shot. Now within the anonymous function, all we do is return another anonymous function (this will be the actual event callback function that the addEventListener will get). Notice the inner anonymous function has the event 'e' parameter. Remember that Functions are first-class citizens in Javascript and are treated like any other object (there actually is a Function class in Javascript) and that's why we're able to do this. Notice within the inner anonymous function we're using txt1 and lbl1 as the variables instead of the textfield1 and label1 we used before. What this accomplishes is the outer function creates a scope having variables lbl1 and txt1 and the inner function binds to those variables; now no matter where the button 'click' event callback function is called the scope of txt1 and lbl1 will be correct. It's tricky at first, but if you get into the habit of using closures for such event-handling assignments it could save you a lot of painstaking debugging later. On a somewhat important side note, given that there is still a reference to that memory/object the garbage collector won't be repossessing the memory space until the reference count goes down to zero (ie. the anonymous function with the closure goes away); this may not seem like a big deal until you realize you're developing for a small embedded system with very tight memory constraints.

Need More Help?

The API documentation can be found at http://developer.appcelerator.com/apidoc/mobile/latest . Be forewarned that the documentation is usually a few steps behind the actual code base, and quite possibly may be incomplete or incorrect. Experimentation and educated guesses are usually required when trying out new functionality. Alternatively you can search in the "Q&A" section of the developer site to see if anyone else encountered the same problems http://developer.appcelerator.com/questions. If you get really desperate you can start digging through the actual framework Objective-C code located in YourApplicationDirectory/build/iphone/Classes/.

Here's a big hint that took me a very long time to figure out. Almost every class that takes the form Titanium.UI.*View is based off of the Titanium.UI.View class so they inherit all of View's properties and methods. They can be used interchangeably whenever a View is required. Views are the workhorses of the Titanium GUI framework, they provide the rectangular regions upon which the GUI widgets are drawn.

One of the best places to get working examples for the Titanium Mobile API is the KitchenSink demo app. As its name implies, it has demo code for practically every feature offered by the API. You should be able to download it from the same servers you downloaded SDK from (http://developer.appcelerator.com/get_started) . Simply uncompress the downloaded archive and import the KitchenSink directory containing the tiapp.xml file (the Titanium project configuration file) into Titanium Developer. Then, launch it in the Simulator. Browse through the KitchenSink/Resources/examples/ subfolder to find the acual javascript code. There's quite a bit of undocumented code/features in there.

Troubleshooting Tips

If you ever come across an annoying bug in code (and you will) that you are fully certain should work (especially obscure low-level Exceptions that are thrown before crashing), and you can't seem to get past it, try clearing out the build directory. For iPhones, the build directory you want to delete will be YourAppsDirectory/build/iphone/build/ . Do NOT delete the outer build directory or you'll have to create the project from scratch again.

Worst case scenario: You may have to create a new project from scratch and copy the Resources and all the assets into it. I've had this happen a few times, especially when the Mobile SDK was upgraded.

iPhone App Development--Only Faster

Appcelerator is a fickle beast that's definitely rough around the edges and ever-evolving, but when it works (and when you get used to its quirky ways), it'll help you throw together a working iPhone app much faster than if you had to write it all from scratch in Objective-C. It's especially useful for the lazy web developer.

]]> http://omniti.com/seeds/aimless-social-media-your-brand-deserves-better http://omniti.com/seeds/aimless-social-media-your-brand-deserves-better Tue, 22 Mar 2011 17:45:06 GMT

The state of social media integration today can be likened to the early stages of "Web 2.0", eight years ago. At that time many confused the phrase, assuming it meant either: a formalized change to the technology powering the Word Wide Web, or a visual design movement that focused on glossy headers, cute icons, crayola colors and gradients galore (cringe). Lucky for us, it represented a paradigm shift in how we (marketers, developers, designers and end-users) viewed interaction on the web—a shift that ultimately spawned the first social networks. Fast forward eight years: the organic development of user-generated media through these social networks and emerging mobile technologies spawned Web 2.0's progeny—Social Media. HOORAY! But, like many of those misguided early adopters in 2005 (who added RSS icons to sites containing no feeds) similar mistakes are being made integrating social media into sites today.

Curious about social media's "RSS icon"? Well, I'm sure you have seen it on sites before: the ubiquitous "Like" or "Follow Us" button. It seems like a great feature; it suggests, "these guys are cool, they are on Facebook or Twitter!" But what do these features do for your site? Do they spread the word? Sure. But to what gain? What happens when users "follow" your brand, or "like" your product and then find little substance once back on your site—finding only more "like" buttons? Odds are they will quickly move on. The web is chock-full of options, and you are not the only person dabbling in social media. So, how can your brand stand out over the "other guys"? How can you avoid the pitfalls of aimless social media integration?

The trick is to create a user experience that speaks to your audience; leveraging the best that social media has to offer, while avoiding any inherited flaws. The modern web is an ever-changing beast, mercurial in its habits and powered by the "ADD generation." Social media is a marketer's weapon for targeting users in this fickle medium. With a deeper understanding of the technologies behind social media applications and a little creativity from your team, you can create a rich, unique experience that will not only keep your users coming back, but will pave the way for a growing audience. Using these sites simply as viral launch pads is a near-sighted use of their true power...The users who give you their time—and your brand, in which you invest so much—deserve something more interesting.

How can you get started?

Step 1: Understand your user base, then create features for them.

Often social features are added to sites as a Hail Mary pass, fueled by notions like, "it's what they did, and they increased conversions by 8%". But they are not you, and their users are not necessarily the same as your users. So, when starting any social media project, you must first ask yourself: Are my users into social media, and if not, why not/could they be?

What if your user base is not filled with savvy social media junkies? Expecting a novice web user to leverage the full stack of social networking sites, means tough sledding. For most businesses, the safe bet is to start with Facebook. The core functionality behind the massive social network, has proven to appeal to a wide demographic. They allow businesses to setup a branded Facebook Page, with little risk or investment. Once your account is set up, users can subscribe to your page by "liking" it. This is where the "like" button is good: It's an easy-to-adopt viral feature that users understand. This is why it's so widely used. However, you still must have something to offer—something enticing to engage your users enough for them to share with others. If you don't, the virus dies. You must first create a compelling experience on your own site before you add in the "like" button and share content with the world; otherwise, the effect is minimal. It is as if you are yelling into a giant wind tunnel. With social media, you have to yell with a purpose in order to make an impact.

How do you yell with purpose? How do you take your branding and turn it into social media features? Here's where you must understand the value proposition for your users. Explore which products/services/experiences you offer that could be repurposed as a social media vehicle. For example, if you operate a travel site, media (photos/videos) or shared trip diaries may be an approach to take. If you operate a product site, sales and offers would provide an enticing reason to visit. The beauty of the medium is that social networks are as diverse as the sites they serve. Knowing who you are as a brand, and what your users want, will help to solidify your social media strategy and ultimately secure your place in the space.

Step 2: The technology exists. Understand it, then use it.

Under the hood, social network platforms, such as Facebook, Twitter, Flickr, Foursquare and many more, are powerful web applications. As they grew over the later part of this past decade, so did the widespread adoption of using Web Services, particularly REST APIs in web application development. Through the use of these Web APIs, the modern web has become an integrated platform of shared functionality, services and data. Applications have emerged that use a multiple external API approach to create a more robust feature set. In development terms, this approach is referred to as a web application hybrid, or "mashup." These mashups make up today's most popular applications including, surprisingly enough, Facebook: great for the social network. Even better for your site.

With a mashup of your own, you have the ability to leverage the power of multiple APIs when designing a rich user experience on your own site. It is an experience open to functionality that is as diverse as the web itself and includes applications like: Google Maps, Last.fm, BBC, Netflix, Amazon, Etsy, USAToday, WeatherBug, SalesForce, YouTube, Instagram, eBay, and the aforementioned social networks, to name a few. With all these options, we can be kids in a functionality candy store. In order to gain any traction, you must find the service that best suits your needs.

Let's start with Facebook:

First, design features that take full advantage of the API to create an integrated, enhanced experience on your site with segmented features and functionality. Sort of a: "Hey, if you clicked 'like' on our Facebook page, wait until you visit our site now—it's cooler—more intuitive and better because you are now in the community. Cool, huh?". This approach not only promotes conversion from Facebook back to your site, it also promotes sharing—viral sharing—and we all know how much fun that can be.

One way you can achieve this is by designing a community feature for your site. Use the objects available via the API to leverage Facebook features such as wall posts, likes, notes, comments and user profile information. You can then create an experience where users will be able to communicate on your site—about your products—while seamlessly sharing back on Facebook. This is how you gain the power of Facebook's social sharing features without affecting the user experience on your site. This community experience is one that you control; one that you can test, modify and enhance to better suit your own user base. Your users will appreciate the extra effort and care—something you can't achieve with a Facebook page alone.

This community would have to use Facebook's authentication methods in order to gain access to user data. By allowing users to sign-in using Facebook credentials, you take out the need for a registration process, making adoption easier for new users. In addition, you control the data collected, improving your business intelligence and paving the way for tightly targeted campaigns.

Now, say you've implemented this Facebook mashup; what happens if Facebook goes down? No worries! If you (or your engineers) have designed this feature intelligently, all objects will be generated and saved in your architecture, then pushed back to Facebook. In case Facebook goes down, the users on your site will remain unaware. Breaking your dependency on Facebook to handle operations avoids a potentially embarrassing situation for your brand.

I have tried Facebook and I need more. What about other social networks?

That answer really depends upon the user knowledge you gained through the discovery process done before you designed your strategy. For example, if you are a media company, the content/brand exposure vehicles, such as Twitter and Flickr, could be appealing options. However, if you are restaurateur, then the geo-location features in Foursquare are a logical choice. But not all businesses can make best use of the same solution. If you keep that in mind, then this exercise becomes vastly easier.

Retail Store Case Study:

Let's say you are a boutique retailer with seven physical storefronts in the northeast corridor:

In the past, your web-site has been used as a brochure site to introduce new product lines, present new sales and promotions, and sell your "high-end" brand image to prospective customers. It's sexy and well designed, but it doesn't "work" to achieve your business goals. Your promotions are falling flat and your users don't know they are there. How do you spread the word, and ultimately get more feet in the stores? Luckily for you, this is the core behind some of today's most interesting and powerful applications, like Foursquare.

Using Foursquare's Merchant Platform, you can create your own badges, campaigns and offers for users who check into your physical locations. This allows you to incentivize customers to come into your stores, with exclusive, geo-targeted offers.

You could create a "Super Shopper Badge," using the Merchant Platform. Set the unlock requirements to: "Check in at all seven of our locations, or at one location 20 times." Then, you would create a series of location-specific offers, that are only available to users with this achievement unlocked. If I was shopping in a mall where one of your stores is located, I would see a call to action pointing out an offer nearby, noting that this offer is for anyone with a "Super Shopper Badge," I would want to see what type of product I could get if I qualified for this offer. You can modify the terms of these offers to target first-time shoppers (first check-in), or the "Mayor" (most check-ins at one location). These features elicit buying behavior almost immediately. It would not only prompt return visits from current customers (in hopes of unlocking achievements), it gives a powerful incentive for potential customers and rewards your most devoted customers.

Once you have customers in the store, buying your product, the next step is to enable users to share with others—spread the word to non-Foursquare users. How do you go about enabling users to share? Achieving this would, once again, require creative implementation of social media within your own architecture. Possibly a leader-board style user-tracking feature on your site, or a mobile mashup application that is a branded tool to engage new users. Then, you could design campaigns and other social media initiatives around this "Super Shopper" campaign. Once you have a stable platform in place, the "like," "share" and "follow" buttons are back in the conversation. Now they have a job—a purpose that will give your campaign "legs" in the bigger social media market.

Okay, I have some Ideas, but what's next?

Unlike the "like" button (no pun intended) more robust social features are not a 20-minute task. Ultimately, this is all a pipe dream without the technical know-how for implementation. These social platform APIs can evolve rapidly and vary from incremental changes to underlying technologies For example, Facebook changing from REST to Graph API, or in the case of Twitter's recent announcement, a drastic change to the terms of service for API use. To stay ahead, you need an experienced team (in-house or vendor) that understands the functionality available, as well as the landscape for third-party applications on these social platforms.

To be successful, you must have creative thinking on both the code and the design side. You need it from marketing (sets the company marketing goal) to the designer (creates the experience to realize the goal); and from the designer to the developer (tech know-how to implement). Marketers have to engage their designers and developers to be successful. At the end of the day, you are adding social media to your site to drive new business. Without that goal in mind there is no ROI and the whole initiative falls flat. It's not a cakewalk, but the technology is out there. In this article, we have only skimmed the surface of what can be done. Marketers have to get educated about everything social media has to offer before beginning their three-way conversations with designers and developers.

Once you have done your homework, the fun can begin!

]]> http://omniti.com/seeds/breaking-social-dependency http://omniti.com/seeds/breaking-social-dependency Mon, 14 Mar 2011 17:23:05 GMT "OMG, Facebook is DOWN!" That was the cry from millions when Facebook was unavailable for about three hours because of network issues. Given the nature of Facebook's service, the downtime did not have any long-lasting effects on its user base. In fact, some say that the productivity significantly increased during the three-hour window without access to Facebook. The bottom line is: the unavailability of the social networking service doesn't negatively impact its users (ego and reputation of the service aside). Does this also hold true for the companies leveraging Facebook, or other social networks, like Twitter, Flickr, FourSquare in their daily operations?

Today, more and more companies operating online businesses try to break into the social media realm by leveraging existing services to increase visibility and loyalty to their brand and bring more people to their sites (and consequently, increase the conversions, visits, purchases or participation). I've seen many incarnations of social networking implementations, from the basic, simplified authentication with Facebook Connect augmenting the regular process (for ease of registration/login), to full-blown applications relying heavily upon multiple features available from these services' APIs. Now, personally, I am all for having these services available and used strategically throughout the applications. It provides a tremendous benefit not only in brand familiarity and content, but also in cost saving -- you're leveraging years of someone else's work for your gain. Consider Flickr. The storage, CDN and REST APIs to present the assets have all been developed and tested for you by a number of smart engineers for a number of years; all you need to do is to integrate the functionality within the content of your site. The same services are available to everyone, and you make the business decision about which features would be beneficial to your company's strategy. The implementation of the features, however, varies significantly.

One of the major risks when implementing a third-party service is the reliance upon the availability of that service -- one that you have no control over. And, no matter how large or successful that service is, it will go down at one point or another. Twitter, as an example, is well-known for intermittent service degradation, often followed by noticeable outages. Now, imagine what happens during the Twitter downtime if your site's content heavily relies only on Twitter API.

Let's examine a situation where a large online media company decided to switch to Facebook Connect as the exclusive authentication method for their site. (To prevent the discussion about the viability of this choice, let me just note that there were legitimate business reasons for choosing this approach). This is where the fun starts. The graph below represents HTTP load time for the pages on the site at every stage of the process. Even without the captions on the graph, everyone should be able to pin-point the exact time when the new code was deployed, and the load time of the pages tripled. The project owners were notified, but since the load times were extremely low to begin with (thank you, properly implemented caching) the load speed was deemed acceptable, and the changes remained in production. Time passed. And then some more time passed. And then the dark day came - the day when Facebook went down. And the page load times on the media site tripled again, for a very brief period of time (while Facebook servers were just lagging), and then dropped to 0, i.e. "users are unable to see the site." Just like that-Facebook's problem became the company's problem.



Upon closer code investigation the problem was identified and resolved quickly, also reducing the page load time to it's original threshold as a byproduct of the change, but it shows how dependent your site can become upon third-party service availability if the features are not implemented correctly.

How can these issues be avoided? There are a few common sense rules that, for some reason, are often ignored during development, which should help with the integration of external services without affecting your site's performance.

1. Only connect to a third-party service where needed.

   Don't try to connect to Facebook on every page load to validate that the user is still the user to whom you displayed the previous page. Cache the results locally.

2. Don't make connections to a third-party service in the critical path of the page load.

   Don't load Google Analytics as the first thing on your page, you will delay the display of the content that actually matters. Make the connections after your content is loaded, or better yet, connect asynchronously.

3. Trap time-outs and errors.

   You do it with your database connections, why would you treat external connections differently?

4. Create a fallback plan.

   You have no control over external services, but you do have control over the content presented to your users. If Flickr feed is the essential feature of your site -- store the displayed history locally, so you can fall back to the latest available content in case Flickr is unavailable. Remember, sometimes stale content is better than no content at all.

To make a blanket statement -- don't jump into using social media features without identifying a need for them and use them to support your primary business model. At the end of the day, when integrating any third party service, you are trying to leverage the benefits of the available functionality to enliven the experience for your own users, not to inherit the services' availability problems. Integrate smartly, not blindly.

]]> http://omniti.com/seeds/from-making-software-to-running-saas http://omniti.com/seeds/from-making-software-to-running-saas Tue, 01 Mar 2011 00:00:00 GMT Software has been around for a long time in various forms: open and closed, commercial and non-commercial. The one thing that holds true about software products is that you, as a consumer, have to acquire them, install them and operate them.  For the past several years, there has been an industry movement away from providing software in this traditional sense and instead providing the use of the software as a service (SaaS). SaaS has been around in many forms. Many companies (and investors) have recognized the opportunities that SaaS provides as a business model, but transitioning to it from a standard software development model requires a lot more than an executive decision. Herein I’ll try to lend some insight into what’s in store for you as you transition from a software company into a SaaS company.

1. A customer of one.

Typical software engineering processes are well-evolved and quite rigorous. They are designed to ensure that the product you release and ship around the world will boast minimal defects and incur as little as possible in the way of defect handling via patching or upgrading. While it may not be extraordinarily difficult to package the next version of your product, you must deal with making the installation/upgrade process as fool-proof as possible or you risk leaving customers stranded mid-upgrade. Getting the entire customer-base to upgrade to the latest version in a reasonable fashion is intractable and the more rapidly you release your product, the more frustrated customers become and the more unique versions you have to support “in the wild”.

SaaS engeering couldn’t be more different. Why? The typical software product driving a SaaS architecture has exactly one customer: you. You have one version of the product in production and it has to work all the time. An upgrade process, for example, is an entirely different beast. Making it robust and repeatable is far less important than making it quick and reversible. This is because the upgrade only ever happens once: on your install. Also, it only ever has to work right in one, exact variant of the environment: yours. And while typical customers of software can schedule an outage to perform an upgrade, scheduling downtime in SaaS is nearly impossible. So, you must be able to deploy new releases quickly, if not entirely seamlessly — and in the event of failure, rollback just as rapidly.

You will find that your needs in operating the product will have a tremendous impact on the the engineering roadmap. Interestingly, you will likely find that the features incorporated into the product should have been on the roadmap in the first place, but you lacked the insight or foresight, because you were not responsible for operating the product in a production setting. From here on out, while you build the service for your users, you build the underlying software products for a customer of one.

2. You aren’t a software company anymore.



You aren’t a software company anymore, you are an operations company. Software as a Service is much more about service than software. In fact, the users of your service will be just as satisfied thinking that magic pixies power the service they use as some complex software system. With this change comes a rather intimidating shift in expectations. Users expect software to have bugs, they expect to schedule downtime to upgrade, install, backup or otherwise manage the software product they are operating. With a service, however, there is a strong predisposition of users to expect things to be “always on.” As a simple analogy, if you sell a user a diesel generator, they will expect it to need maintenance, needs refueling and have the occasional service issue. Sell them electrical service and watch them come with pitchforks demanding refunds if you have an outage of any sort.

While this may seem silly at first, the expectation isn’t out of line. It’s a simple bit of economies of scale. Your job as a SaaS company is to operate the software, so logically you should do a better job than they would. Additionally, you are operating it for a large set of users, so it is a reasonable expectation that you have refined your operational techniques. Lastly, they pay you for one thing: to operate the service — so you had better get it right.

Working as an engineering company with an operations focus rather than a product focus can be a significant challenge for traditional software engineering companies.  You should expect to see roles removed, roles introduced and organizational structure changed to add accountability for operating your service as your users expect.

3. Continuous Deployment

One of the greatest advantages of being a customer of one for your software is that you don’t have to worry about the oddball deployment or “that guy” that refuses to upgrade.  It means that once you’ve deployed the latest version of code into production, you have no legacy copies, no troubleshooting of version differences and a definitively less complicated error reporting process. This, however, can cause a paradigm shift in development and deployment processes. It means that you can have a bug report at 8 a.m., a fix by 8:15 a.m. and a deployment by 8:20 a.m. Traditional software engineering companies have no other word to describe this but “insane.” It might seem reasonable to simply elect not to subscribe to that pattern of behavior due to the risks involved, but there is weakness in that stance.

In the era of SaaS, companies have engineered processes to successfully manage the risks of rapid deployment schedules (OmniTI, IMVU). What was once a patch release every two weeks can now be managed as hundreds of patch releases per day (in the extreme case). By carefully engineering risk out of the deployment process, a SaaS company gains agility to launch fixes, improvements and features into production at any time. If your competition can do this and you cannot, you are disadvantaged.

While it may take considerable effort to redefine your engineering processes to adequately limit risk and allow for continuous deployments, the advantages are significant. Due to the velocity of deployments that must be supported, the process of deploying itself must be engineered to be non-disruptive to services. This alone has the side effect of enabling feature launches, upgrades and triage without consequential downtime. It is the first step toward an “always on” architecture.

4. Quality Assurance is now a continuous process.

Quality assurance has a strong role in software engineering. While there is much effort expended automating QA, an automated QA process is sufficient if your service is used only by automated systems. If humans consume your service, you must also have human-driven QA processes. So, while much of the QA process can be automated and performed rapidly, it will never replace human usage of the application to detect both errors and perceived errors. In SaaS systems, the velocity of user-facing change is (at least) an order of magnitude higher than in traditional software engineering. It is inevitable that bugs will not only appear, but that they will reappear. Performing a full QA regression prior to each release is often unfeasible.

Your users are just as much members of your QA team as your employees. By making your users aware of that, by treating their feedback, complaints, bug reports and feature requests as first-class items, you enable them to improve your QA process and, more importantly, increase their tolerance for your mistakes. John Martin has a short, but enlightening, diatribe about the quintessential difference between QA in traditional environments and SaaS.

Perhaps the single most significant change to embrace is that of QA’s place. What was once a engineering phase, a deliverable, or a series of bars on a project manager’s Gantt chart (ultimately leading to a celebratory day of shipping a product release) is now a continuous and critical operational role within continually delivered and continually used service.

5. Multi-tenancy design.



So far, we’ve discussed mostly process changes that enable transforming from a builder of software to an operator of software. The last paradigm shift is perhaps the hardest as it relates to design philosophy and design goals rather than design processes.

When traditional software is designed, it runs on a system or set of systems for a single user. While a “user” in this sense can be an individual, or a business unit or perhaps even a whole organization, it is clearly not “all users.” It is the difference between engineering a car and engineering a complete metropolitan transit system. It is an issue of designing at scale.



Not only does this mean designing and building software that can handle thousands of times the load that your previous design enabled, but also engineering the solution to malfunction elegantly. Malfunction elegantly? Yes. All human engineered products will malfunction, it is a simple fact of life. In a SaaS, it is essential that when this happens that the malfunction is isolated to the smallest possible component of the service or to a specific customer. Back to our transit metaphor: the failure of a single bus, subway train or taxi must adversely affect as few users as possible; ideally, only those physically on the failed unit. This consideration is simply (and obviously) not present in the design of a single car.

The engineering paradigm shift from a single-user product to a multi-tenancy product is the most challenging metamorphosis required by a software company that intends to adapt and survive in the SaaS era. Two books that talk about the underlying mechanics of these challenges are Scalable Internet Architectures (written by me) and The Art of Scalability by Abbott and Fisher

Making good on a promise

While you may not have made a promise about what your SaaS offering will provide, the industry has set some undeniable expectations about what SaaS generally delivers.  At a minimum, you must meet these expectations or your users will abandon you. These expectations are naturally derived from the key drivers for adopting SaaS: no maintenance, no upgrades, always current, always available, no commitment (or the desire for operational expenses over capital expenses). It is imperative that you understand where the bar is set for those wishing to shift into a SaaS delivery model.

With the exception of software companies that produce software that powers SaaS, most traditional software companies must evolve into a SaaS delivery model or suffer death at the hands of competition.  Evolving into a SaaS delivery model without addressing the above key points will lead to substandard service, artificially high operating costs, user attrition and eventual collapse.  You have to do it. You have to do it right. Are you ready?

]]> http://omniti.com/seeds/maintainable-stylesheets-can-css-be-object-oriented http://omniti.com/seeds/maintainable-stylesheets-can-css-be-object-oriented Thu, 24 Feb 2011 18:10:24 GMT How can CSS be object-oriented? In short: it can’t. But if we stopped there, then this would be a pretty short article! Let’s take a look at what is meant by the term “Object-Oriented CSS” and how it can help improve your stylesheets.

What’s the Problem?

First, we should take a step back and look at the problems that lead to CSS being difficult to maintain. Most projects start out easily enough. We write some styles for page structure, some default content styles, then some specialty content styles, right? We always ensure that our styles are separate from our markup and that our markup is clean and semantic, thanks to the web standards movement. When the project launches, we feel good about our work and don’t think much about code maintainability. After all, it’ll be easy to maintain because we wrote the code.

The first round of updates comes and it’s not too difficult. The stylesheets are still fairly small and easy to navigate, so the appropriate styles are found and updated. Some new styles are added because the new content is a little different than the existing content. Soon, a second round of updates is made, then a third. Each time, a few more styles are added to the stylesheets. Then we’re taken off the project for one of a hundred reasons, and someone else begins maintaining the site. Or perhaps we simply don’t touch the site for a year and then have to come back and make some updates; after such a long interval, we don’t remember as much about the code as we think we do. In either case, it’s difficult to understand the mix of ids, classes and element names, so more style declarations are added, overriding previous styles where necessary to get the desired effect.

Before long, some styles become defunct and other styles become unnecessarily complex as the stylesheets become overly complicated. We’ve ended up with spaghetti code, and the site’s stylesheets now take longer to download as they get more bloated. How do we get out of this mess? Is there a way to create truly maintainable style sheets?

Yes, there is a way, and it’s recently come to be known by the term, “ Object-Oriented CSS,” thanks to Nicole Sullivan. Style declarations are not objects in the programming sense of the word, but the term “Object-Oriented” refers to a mindset of thinking about how styles should be written and applied.

Keep Your Styles Where They Belong

Before we begin, we need to lay down a ground rule for keeping our styles maintainable: keep styles in the stylesheets. This means two things:

1. Styles don’t belong in markup.

   Nowadays, we usually are pretty good about this, but it’s always a good reminder. Don’t succumb to the temptation to insert a quick style attribute here and there. It seems innocuous now, but we may regret it later.

2. Styles don’t belong in Javascript.

   This may or may not be up to us depending upon how the division of labor happens where we work. But if JS is on our plate along with CSS…great! That means we have the power to ensure that each site’s behavior (JS) is separated properly from its styling. Javascript should almost never be used to manually modify styles on an element—the lone exception to this is when a style value has to be calculated based upon other information. If we set styles via JS, what happens when we need to change those styles in the future? We’ll have to go dig through the JS and find every place those styles are being changed, instead of just making some simple stylesheet edits.

If we can’t edit styles with Javascript, how should JS interact with styles? The best method is using JS to modify elements’ classes. As we’ll see below, classes are the keys to elements’ identities and run-time states. By using Javascript only for modifying class names, all of the actual styling is handled by the stylesheets, making styles easy to find and modify.

And Now For Something Completely Different…Sort of

When we discuss this new, “Object-Oriented” mindset, there’s one key word that will be our mantra: “reusability.” Ok, there are really two key words: “reusability” and “patterns,” so I guess you could say we’re looking for “reusable patterns” here.

If we start thinking about creating “reusable patterns” at the beginning of a project, it’s a lot easier than going back and fixing a site that has already launched. But it still may be worthwhile to revisit existing sites’ code. Putting in the work now to make our stylesheets maintainable will not only save us future headaches, but it will also speed up our sites since the CSS files will be smaller and will download more quickly.

Before writing any code, we should pull out our layout comps and review them, looking for patterns. The patterns could be large and easy to spot like column arrangements or page block arrangements, or the patterns could be small like a box for entering login information. The patterns could be structural like column or block layouts, or the patterns could be stylistic like font and color choices. But finding the patterns is only the first step. When thinking about how to style these patterns, we need a paradigm shift in our approach. Instead of writing style selectors using IDs and element names, we will use classes to describe the elements’ identities and states. As a simple example of this, we’ll look at a small toolbar of zoom controls.

When thinking on how to style a zoom toolbar, our first inclination might be to style the entire toolbar using its ID #zoom_bar and then style the individual controls using their IDs or element names, either #zoom_bar #zoom_out or #zoom_bar button. But we should stop and think about this for a moment. If we use IDs, then these styles are only applicable to this particular instance of the toolbar. If we want to reuse the styles we have to add another set of selectors, giving us #zoom_bar button, #another_bar button. Now we have two sets of selectors and will probably need to add more in the future. And what if we’re styling link states? #zoom_bar a:hover, #zoom_bar a:active would turn into #zoom_bar a:hover, #zoom_bar a:active, #another_bar a:hover, #another_bar a:active. Writing styles this way quickly results in bloated stylesheets with way too many selectors.

Decouple Your Styles

Instead of styling elements according to “what we call them,” let’s shift our thinking to “reusable patterns” and style elements according to two things: their identities and their run-time states. Now, what is the core identity of the zoom toolbar? Zooming is what it “does,” not what it “is.” Its core identity is that of a toolbar, or perhaps more generically, “a group of buttons.” So what if we give it the class .button_set? This refers to its generic identity and is very reusable. As for the controls inside, we don’t use their element names either (like “button”), but instead we create a class based on their identity, which could be as simple as .button. This decouples an element’s styling from its markup; now we can use .button_set .button as our selector, which avoids the specificity of a particular instance ID, which is not really reusable. This also gives us the flexibility to use buttons in our zoom toolbar or to change the elements in other (or future) instances of .button_set. What if we encounter an issue that makes us change all the buttons to anchors? By using an “identity class” and decoupling the styles from the markup, all we need to do is change the markup, not the stylesheets.

The other type of classes we will create for our .button_set is for the elements’ run-time states. In this instance we may have three different states: the default state, an active state and an inactive state. Creating classes for these states is as easy as creating .active and .inactive classes. Notice that these class names only describe the elements’ states, not what the elements look like or where they are positioned. Again, this decouples the markup from the styles, making the class selectors generic and extremely reusable.

When making this paradigm shift, we need to be careful to only use classes when defining our patterns—don’t fall back into the trap of using IDs for styling. It’s easy to relapse, but because of IDs’ high specificity, we should be especially vigilant about not using them for styling except to define occasional exceptions. And even then we should always ask ourselves, “Can I take this exception and make it reusable?” If we have to make an exception once, chances are good that we'll have to make it again later, and then we'll have two exceptions instead of one reusable pattern.

Please note that one ramification of decoupling our styles is that we may have to use multiple classes on a single element, making Internet Explorer 6 tricky to work with. In general, we should be able to work around this issue by adding concurrent classes onto different elements. But if we must support IE6 with multi-class selectors, we have two options: we can either use a Javascript fix (like IE7-JS), or we can create "joining classes" which join together the styles from the individual declarations (e.g. create .one_two instead of using .one.two).

Our Future Selves Will Thank Us

So to recap: Object-Oriented CSS is a phenomenal approach to writing stylesheets, even if it has very little to do with traditional Object-Oriented-ness. However, one thing that all CSS does have in common with object-oriented programming is inheritance. Always remember to take advantage of inheritance! If we find ourselves wanting to add the same class or style to several elements, we should make sure it is absolutely necessary; we may be able to get the same results by adding the class or style to a parent element.

Taking advantage of inheritance…creating reusable patterns…decoupling our styles…it's pretty easy to see the benefits of using this approach on an entire site. Gone are the layered styles, on top of styles, on top of styles. Gone is the need to overwrite some previous style to get the effect we need. With a little careful planning, we all can have lean, fast stylesheets that make maintenance a breeze.

]]> http://omniti.com/seeds/instrumentation-and-observability http://omniti.com/seeds/instrumentation-and-observability Wed, 08 Sep 2010 00:00:00 GMT

There has been considerable momentum established behind a movement called devops. This momentum is good. There does not appear to be anyone coming out and saying "this whole devops movement is bad and ignorant." So, as one can assume with no notable adversaries, it stands to reason that the movement is a "good thing."

The devops movement is often thought of as an effort to bring the operations world into the development (software engineering) world. Statements like "A into B" are vague. Let's be clear on the concept: introduce the wisdom and experience from software engineering into the operations realm. The software engineering world, over its brief history, has established many excellent paradigms including testing, version control, release management, quality control, quality assurance and code review (just to name a few). These concepts, while they exist in good operations groups, are admittedly far less formalized and could stand some rigor.

So, while one might think we'd discuss the merits of software engineering principles in operations in this seed, we're happy to disappoint. There are plenty of people talking about this already; they are making excellent points and getting their points across.

We're here to speak to the other side of the coin. This is Theo at DevOps Days 2010-US, hosted by LinkedIn in Mountain View, California:

Operations is not, and has never been, a janitorial service. Operations crews are responsible for the impossible: it must be up and functioning all the time. This is an expectation that one can never exceed. One can argue that we establish SLAs (service level agreements) to bring these expectations within reason, but SLAs are legal terms that articulate allowable downtime, not desired down time. Users want services available all the time. As a result, operations is faced with an impossible task and, amazingly, makes good on unpromised availability more often than not. Let's talk about the not.

Operations is, by definition, the group that operates things. These "things" encompass the entire technology stack: networking and systems hardware, operating systems, COTS (commercial, off-the-shelf) and open source application software and in-house tools. Consider the following statement, which seems obvious, but is commonly overlooked: "It is easy to operate software and hardware that is operable." Many common components in the information technology stack are simply inoperable by our definition. This is where we get into the meat of things: how does one define operable?

Defining a component as operable is quite simple. Inevitably, things go wrong. When things go wrong, they must be understood to be repaired. Troubleshooting is a zetetic process. To progress, one must ask questions. These questions must be answered. This should be plain and obvious to anyone who has ever experienced an unexpected outcome to a situation (technical or not). So, why is this complicated? To be effective, one must not change the situation during the course of the question. This caveat is where things get complicated and fortunes are made.

To observe a situation without changing it is the ultimate achievement. While Heisenberg believed this to be impossible (and we agree), one can achieve a reasonably small disturbance during observation. An excellent example is the classic philosophical question, "If a tree falls and no one is there to hear it, does it make a sound?" Let's think about that question for a moment to better understand impact and side-effect. Is the sound of the tree falling more or less likely to affect the overall situation than the actual destruction and subsequent felling of the tree? The problem with many observation systems is that, in order to observe the sound of the tree, they must hew a tree during every instance of observation. We suggest a different approach.

Many systems have critical metrics, which are diverse and specific to the business in question. For the purposes of this discussion, consider a system where advertisements are shown. We, of course, track every advertisement displayed in the system and that information is available for query. Herein the problem lies. Most systems put that information in a data store that is designed to answer marketing-oriented information: who clicked on what, what was shown where, etc. Answering the question, "How many were shown?" is possible but is not particularly efficient. In order to answer the question, one must hew the tree and wait to hear the sound of its fall.

Instead of asking analytic questions, applications should expose this information as a consequence of normal behavior. Just as the sound of the tree falling is a natural consequence of the act of hewing, the ad serving system is responsible for tallying the total impressions and exposing that information to those that care. No significant work need be performed by the application to answer this question, just a pre-calculated response to a simple question. This enables a new way of application observation where witnessing metrics and their changes requires no substantial work by the application. This paves the way to new types of application monitors (for example high-frequency monitors) that need not worry about altering the situation by observing it.

Not all questions can be asked before a problem occurs. This is where observation ends and instrumentation begins. Instrumenting code allows new questions to be asked and subsequently answered in a running environment. A system admin or developer may look at a malfunctioning system and think, "How do I recreate this situation in a test environment?" The reason we ask that question is because debugging in production is taboo. If a developer instruments code well, profound knowledge of the problem may be derived without the risk of altering its state. DTrace is the king of these systems and its adoption across various operating environments is growing. Nevertheless, no one should argue that they should throw in the towel just because they don't have DTrace available to them. While powerful instrumentation might elude those without DTrace, we've found that we can get most of the way there with careful logging (a poor-man's instrumentation) and continuously exploring critical metrics to expose for observation.

Many architectural components today provide an HTTP interface, primarily via a REST API. Use it! Extend the HTTP server to expose critical component metrics via HTTP. Use JSON, or use the Resmon XML DTD. In Java, expose metrics via a Bean accessible via JMX. This can be a bit frustrating because Java-centric tools must be used to observe it, so instead, just expose those metrics via a servlet. There is even some free code for that: Resmon Java Servlet (see Resmon.java and ResmonResult.java). Exposed metrics can be tracked, trended and alerted on easily using tools like Reconnoiter or Circonus.

Making applications operable means that never again should operations personnel be stuck on the question, "The application appears hung, I wonder what it is doing?" All production code should be prepared to answer questions such as these at any time. "What are you doing?" and "How long is it taking?" are perfectly reasonable questions to ask of any piece of production code and you should demand a prompt and accurate answer. The resulting metric data is consumable by both dev and ops teams, and even by those teams' managers. After all, trending metrics is not just about detecting problems. It is also fundamental to quantifying success. This is what it means to be operable. Software engineers everywhere, please make your software operable!

]]> http://omniti.com/seeds/fast-by-default http://omniti.com/seeds/fast-by-default Tue, 13 Jul 2010 00:00:00 GMT I recently attended the O'Reilly Velocity 2010 conference in Santa Clara, CA. For the past two years this conference attracted some of the smartest minds in web performance and web operations; this year did not disappoint.


~ James Duncan Davidson (original)

Several exciting things debuted including Opscode's hosted platform, Yahoo!'s Boomerang and our very own Circonus Enterprise Platform.

Each year, the conference has adopted the mantra: "fast by default." This statement, largely applying to the web operations track, is an excellent theme. The concept is that speed is feature number one and that your success as a online company is intrinsically tied to how users perceive the performance of your online presence. This is true, the numbers tell us so.

The interesting part about web performance is that user-perceived performance comes from three separate elements: computation done by the service, computation done by the user and act of getting data between the two. Velocity really focuses on the latter two: how do I optimize how content is delivered to my users and optimize how it performs once they've got it? This perspective is incomplete. Should Velocity change to address all three elements? I say no. The audiences are different, the problems are different and there is no need to mess with a good thing. Surge, on the other hand, only concentrates on the first element: server side performance and scalability.

Let's face it, the server-side architectures that power today's web services are as unique as the services they power. Each site has its own unique challenges that come with its size, technologies, audience, offering and promises. Not to trivialize the web performance challenge, but the techniques used to increased user-perceived importance in transit and on the client side are largely the same from site to site (clean, small and effective DOM, CSS and Javascript, correct caching, image sprites, HTTP compression, etc.). However, on the server side is where the unique magic happens.

Do you really think that the technology powering Google's new Caffeine search indexer could be leveraged easily to help your internal service delivery platform? No. For a user to use your service, in an over-simplified form, they provide some input and receive from output. Each time they ask a question and expect a result, you must "do some work." Herein lies the challenge.

In a previous installment "YSlow to YFast in 45 Minutes", I explored reaping low-hanging fruit to achieve user-perceived speed-ups on this very site. The main effort there was to shorten the event horizon to render by removing, shortening and/or parallelizing various assets on which a page depends. The obvious, but often ignored, part is that it all starts with a single request: "the page."

On the OmniTI web site, there is very little going on and as such, you'd expect that very little time is spent on our end "doing work" to give you the page content. If you look at the details, you can see that to be true: 44 milliseconds waiting for data to start and 25 additional milliseconds waiting for the data to come down the pipe. This is relatively fast. This is not always the case; in fact, it often is not the case.

I was quite interested in this division of time and asked my helpful friends at Keynote for some aggregate information. That information paints a rather interesting picture. The average speed of a "web page load" comes in at over 2 seconds. Obviously, these 2 seconds are split in some fashion amongst our three buckets. What may be quite surprising is that, on average, 290ms seconds is spent server-side. I speculate this is due to one of two reasons. Most commonly, it is due to a lack of attention to how the architecture internally operates resulting in sloppy code and data architecture. To me, this is the better of the two reasons. The other reason is a focus on "scale-out" with a blatant disregard for a maximum acceptable service time.

One web performance company, who shall remain nameless, actually spends as much as 2 seconds "thinking" before sending data to the client, producing an awful waterfall. Note that the client-side performance is quite excellent, but still the user waits uncomfortably long.



To put this in some perspective, a processor today can operate around 2.9GHz (that's 2.9 billion instructions per second). 290ms sans a conservative 90ms of round-trip latency is 200ms of operating time or 580 million CPU instructions. The disturbing part of this is that most of what Keynote monitors is landing pages or specific hot paths, so many other pages on these websites are slower. We all know that most websites today are more complicated than a single machine serving information, so a direct correlation of service time to CPU cycles is deeply flawed; however, I still believe it is illustrative, useful and compelling.

Furthermore, if your system is spending 200ms servicing a single request, you can do the simple math to find that even on an 8-way system, you can still only serve 40 requests/second. As your demand increases, you must add more and more machines. While provisioning these machines used to be challenging, the cloud has played on general performance-optimization delinquency and made this approach seem acceptable by making massive machine provisioning easy. I'm here to tell you it is not acceptable. Not only is it environmentally wasteful (using power and generating unnecessary heat), it is also wasteful of shareholder investment. Faster sites running more optimally generate shareholder value.

The pervasive focus on front-end performance is explained by the easy gains that can be seen from relatively little investment. However, as the numbers show, for most sites this simply isn't enough to compete. Shopzilla recently completed a 12 month engineering effort to rearchitect their application because the server-side was too slow (pushing 8 seconds). Now that it is blazingly fast, they have less infrastructure to maintain per dollar of revenue and an increase in revenue of 7-12%.

Attention to internal performance in fundamental to the success of online businesses. Many of the larger web-based companies have smart people on staff that take performance seriously. If you need help, this is what we work on for our clients everyday at OmniTI.

]]> http://omniti.com/seeds/the-cloud-is-great-stop-the-hype http://omniti.com/seeds/the-cloud-is-great-stop-the-hype Tue, 23 Mar 2010 00:29:27 GMT Cloud computing isn't new, though I'm sure you've heard more about it in the last few months than you did previously. The cloud is an amazing thing, but one that is poorly understood. I believe this lack of understanding stems from technology confusion which is trumpeted by corporations that have identified "the cloud" as a medium for expansion and profit. Don't get me wrong, the cloud is useful — but I hear some of the dumbest reasons why.

Before I launch my rant, I'll qualify that SaaS existed before "the cloud," yet in many defintions (like the link above) it is considered a cloud service. I consider the cloud to be only the infrastructure because the software and the platform has been provided by a third-party successfully before the term "cloud" arrived. It isn't fair to legitimize your concept by repackaging two successfully proven technologies under your brand.

The Cloud

The cloud... what is it? A cloud is an infrastructure in which I can provision computing systems. What makes this different from a rack of servers? Very little, actually. The most important difference is that provisioning of these systems is made convenient. When a system is needed, the requester can programmatically start a new one and needs not be concerned with network infrastructure, machine specifications, power, cooling, etc. The cloud is built by someone who cares about all of those things, but then it is packaged in an easily consumable fashion. How does this happen? Well, this is where people get confused.

Virtualization

This simple provisioning is empowered by some sort of virtualization technology like Xen (likely one of the commercial implementations), VMWare, Solaris Containers (Zones), Virtuozzo/OpenVZ, etc. Why is this confusing? Beats me, but I see people listing the advantages of virtualization as advantages of the cloud. As with most technologies, you inherit the advantages of your foundation. Virtualization brings a lot to the table, but you don't need "the cloud" to get it. Period.

The concept of private and public clouds is also poorly defined. Some people hate the two terms, while others define them in useless ways. I'll define them in a very practical way in which the differences have deep business meaning.

Public Clouds

The public cloud is Amazon's EC2 and other similar "cloud providers" where the owner of the underlying physical infrastructure and the owner of the services running on the provisioned systems are not the same. In this environment, your services run on someone else's equipment. What does this mean?

If they don't pay their bills, the equipment can be seized. Other companies may be running virtual environments on the same hardware, same disks, same network. This means bugs in virtualization and data isolation could result in information disclosure — the really bad kind. At this point in time, I can't envision a way to make public cloud infrastructure PCI-DSS compliant — and even if you could, I believe it increases the possibility of compromise.

No virtualization is perfect (yet) in resource provisioning. This means that defining a reliable performance expectation for a node in the cloud can be very challenging.

It's not all negative though. Because public clouds are popular, they tend to have ample resources, which means more room for growth, and a provisioning request is "less likely" to result in an message that says "better luck next time, I'm flat out of horses."

Private Clouds

Private clouds are not shared. A private cloud is deployed by an organization that wants the benefits of a cloud, but wants the processes and premise controls over the infrastructure that powers it. The key differences between a private cloud and public one are control and size.

In a private cloud, you have fine-grained control over geographic location. This can be important for meeting data availability and/or redundancy guarantees made to clients. It can also be useful for ensuring that at least part of your infrastructure is in a country whose laws more closely align with your business needs.

There are clearly enormous advantages in the private cloud, in that data security exists and the design and operation of the private cloud can be congruent with business requirements providing more aligned availability and consistent performance. The downside is that it is likely to have more limited resources — provisioning 1000 new instances is far more likely to result in a failure due to insufficient resources.

So how big is big? In my experience, when you hit a run rate of 40 instances, build yourself a private cloud. That's the point at which it becomes undeniably cheaper.

Resources

One distinct and measurable difference between how private and public clouds can be run is seen in the choice of virtualization technology. Public clouds, by their nature, must isolate resources between customers as extensively as possible to achieve acceptable quality of service. There is no trust or cooperation between virtualized customers.

No virtualization technology does this perfectly, but some do a better job than others. Xen-based, and VMware-like solutions are some of the more capable in this arena. Because both implementations run completely separate operating system environments from a hypervisor, they tend to segregate the guests more thoroughly by sharing less resources.

This is good for guests, but bad for resource utilization. If I need as much as 16GB of RAM for my instance and I'd like to run 8 of them, that means I need 128GB of RAM in my host machine — that's an expensive box. On the other hand, if I need very little RAM (say 256MB on average of which 128MB is kernel and OS related processes) the hypervised virtualization becomes quite bulky.

On the other side of the virtualization field are technologies like OpenVZ and Solaris Containers (a.k.a. Zones). These technologies share a kernel (and usually a filesystem buffer cache) across guests. CPU resources can be sliced up, but memory (as it is shared) is a challenge to dedicate cleanly to individual guests. While this is clearly a bad (or at least challenging) thing for public cloud providers, it is often completely acceptable for private cloud needs.

The advantage of this "lightweight" virtualization is that you can pack more guests onto a single host. We regularly run 40 Solaris Zones on a single commodity server without issue. It is particularly useful for applications that are low-powered, but in need of multiple instances to meet their availability commitments.

Burning the Straw Man

Now that we know what clouds are, what's the problem? The hype. The hype is the problem. With hype come straw man arguments that delay or hold back the healthy evolution and incorporation of this technological paradigm.

Argument 1

I need the cloud. In the cloud, if I need to deploy 50 machines, I can just do it. Without the cloud, I have to buy servers and wait weeks for install and spend hours installing them.

Deploying 50 new instances in a cloud is easier than 50 new physical machines. But just because you can, doesn't mean you should. If it takes hours to install new machines, then you are doing your job wrong. If it takes weeks to get your machines, then you are using the wrong vendor. And most importantly if you suddenly realize that you need 50 new machines, then you simply didn't do your job well. The cloud is not an excuse to avoid a business model. A business model includes a budget and a solid, implementable plan for growth based on thorough capacity planning. With that, you should see it coming.

There are two reasons I hear when people justify the need to deploy a large number of new machines, and both arguments fall apart when you take a closer look.

Argument 1a

Holy cow! Look at that traffic! I need fifty new instances. Now!

I know a bit about sudden traffic spikes. If you need 50 machines suddenly to handle a traffic spike, then, in all likelihood, you have built something wrong and no amount of provisioning will help. I've had the privilege of working with some of the largest sites on the planet. I've seen traffic spikes of 10000% happen inside 30 seconds, but then again I've also seen more than a gigabit of production traffic served to the masses off two $3k USD boxes. If you are in that situation, you need a plan — and it likely shouldn't include "Oh shit! Bring 50 more instances online!"

If you are providing a service that is unavoidably computationally intensive, you actually have a solid argument. This is rare and I'll touch on that later.

Argument 1b

I have a lot of developers and they each need their own instance quickly and easily.

This is actually an awesome argument for the cloud. However, since these are development instances, they don't consume resources in the same way that production instances do. We give out instances like candy at OmniTI and typically can sustain about 40 instances on a single $3k USD box using lightweight virtualization. CapEx and OpEx on that are basically non-existent compared to an EC2 bill for the same. As you can see, this is an argument for virtualization, not the cloud.

Argument 2

I want to use the cloud because that way I don't have to worry about networking and hardware management.

Network management has to happen. Hardware management has to happen. You pay for it one way or you pay for it another. I've heard people say that it takes countless hours per month to run 40 systems including servers, switching equipment, routing, firewalls, etc. We manage around 1000 servers at OmniTI and from our immaculately maintained time tracking system I can tell you that less than 35 hours per month are spent on hardware provisioning, systems installation and concerns of space/power/cooling. That comes out to about 2 minutes per machine per month. Furthermore, I don't have any reason to believe that a cloud provider can do a significantly better job.

So, if so little time is spent on hardware and infrastructure management, why does OmniTI have a busy ops team? Because we're doing all the other stuff. Configuring software, performance tuning, and monitoring systems; monitoring systems to an egregious and offensive level. I'm not speaking of CPU temperature and disk failures (everyone monitors those). I'm talking about realized I/O ops per spindle, network packets per interface, HTTP response times, SSH keys, ICMP response latency, DNS, database health, application-level correctness and, most importantly, business level metrics. If you find this intimidating, look at Circonus as an enablement platform. If you like the cloud and/or SaaS, you'll love this service.

The operations team is the one place with access to data and traffic that is "real-time enough" to detect business issues before they manifest in significant monetary loss. Traffic anomalies, chargeback rates, visitor retention... all these translate into money. This is what ops does; they make things work; they make the business work. And they spend a lot more time trending, investigating and analyzing than they do replacing hard drives and network cards.

Argument 3

I can provision quickly in the cloud.

Yes. Yes you can. This is due to virtualization, not the cloud. Download a virtualization technology and provision quickly outside the cloud. I suppose that if my OS natively supports Virtualization (like all modern OSs do), and my operations team leverages that to deploy new instances quickly and easily, then we've created a cloud whether we like it or not. Damn terminology. While it is now called a "private cloud," I tend to just call it infrastructure operations.

Argument 4

Operating in the cloud makes your environment more resilient because you have to accommodate unexpected failures.

What? This has to be the most back-assward statement I've heard on cloud computing. Eagerly adopting an environment with a higher failure rate because it forces you to be a better engineer? Well, that's not an engineer I'd hire. Good engineers have always known that things can fail and have always had to design to accommodate that truth — incessant reinforcement by some public cloud providers is unwelcome and unneeded in this case. Assuming a well engineered system (which should be an expected outcome of any engineering group) the goal should always be to minimize the likelihood of failure within budget.

What the Cloud Lacks

In addition to dismantling poorly constructed arguments for the cloud, I thought I'd detail some of the things I find completely missing in the cloud.

Generalization is the root of all evil when it comes to performance. Just because you know how to use MySQL or PostgreSQL doesn't mean it is the right tool for every data storage need. People have learned this lesson fairly well. In cloud infrastructures, there is a goal to make systems alike to improve price points for capital expenditure, reduce operation expenditure (slightly) by learning one type of system well, and make the provisioning system simplistic. This leads to the abomination that is "small," "large," and "huge" instance sizes at some cloud providers.

As an engineer, when I have to build a system for a purpose I specify as much as possible. AMD vs. Intel vs. Sparc? How many gigs of RAM? What speed should the the RAM be? How much storage do I need? How many I/O operations per second are required? Should I use SSDs? How many networks must the system be on? Should we be using link aggregation or not? VLANs? No VLANs? These are all important things. If you need these things sometimes and everything has to be the same, then you get these things all the time — paying for it when you don't need it.

It is a reality that when systems are specified, compromises are made due to vendor relationships and part availability. However, the requirements that drive these specifications still exist and are at the root of the decisions: for instance I need 16GB of non disk-buffer memory for working sets and 10,000 I/O operations per second. That simply doesn't translate to three cookie-cutter sizes.

Data is a big issue. There are a lot of companies out there working on solving the data security issues that exist in public clouds — let's assume for a second that this is no longer an issue. A follow-on issue is that the cloud is "out there" and the only way to get data into and out of it is via the drinking-straw that is its uplink. Drinking-straw you ask? Yes. The internet is, even today, not as fast as a tractor-trailer full of tapes. If I have 10 TB of data (which is extremely reasonable for any business intelligence system these days), how do I back it up? I need a copy of that data off-site and secure. We have some creative solutions around this using ZFS, but still — I am contractually obligated to have my tapes (or some other off-site and off-line storage medium). Private clouds do not have this issue.

Scaling Out or Scaling Up

So many people talk about scaling out. Scaling out. Scaling out. Scaling out. Scaling out is an excellent approach to tackling requirements that cannot be easily accomplished on today's hardware. Not everything needs to be scaled out. I hear people say "I'm going to have millions of records, I need to make sure my design can operate on many machines." Millions? You're going to go through the effort of tackling distributed systems problems for a million rows? You have priority issues. A single machine (with failover) is enough to do most jobs. People lose sight of this too often. Making things redundant (hot failover) is a lot easier than making them actively distributed. So, if you can get away with scaling something vertically, do it.

There are many cases where the growth of a specific system component simply outpaces the availability of reasonably priced hardware to scale it vertically. In these cases, you should make your problems smaller. (You'd be surprised what can be accomplished over beers with an expert in the field). If that fails, then you roll up your sleeves and design your system to scale horizontally. Very few systems require horizontal scalability from soup to nuts.

Where It Works

I said before that if you need to spin up 50 instances you clearly didn't do a good job planning. I'll recant that and better qualify where that is acceptable. That is acceptable when that is your well thought-out plan. When would you need to spin up 50 new instances? Let's say you need to transcode a ton of video, let's say you need to sequence some DNA, let's say you need to use a lot of computational resources for a brief period of time and that is essential to your business model. This is where the cloud shines like a super-star.

For computationally intensive tasks that are irregular, the idea of batching work into a cloud of compute nodes is an excellent one. Here, the advantages are clear. Given that each job can really gobble up CPU resources, you can't leverage the consolidation that virtualization offers. At this point, the disadvantages are purely the outcome of an equation of economics. How much does a CPU-second cost and how much does it cost me to move the input for my job into the cloud and extract the output from the cloud: instance costs and bandwidth costs.

The Honest Truth

While it may appear that I hate the cloud, it simply isn't so. I hate the half-baked arguments for it. I hate the hype. It is a perfectly legitimate tool in the already large arsenal of engineering tools. Use the cloud where it makes sense, but please stop bludgeoning me with the hype.

]]> http://omniti.com/seeds/online-application-deployment-reducing-risk http://omniti.com/seeds/online-application-deployment-reducing-risk Wed, 17 Mar 2010 13:39:08 GMT Version control systems are nothing new to the world of software development. I'll take the time now to unapologetically call you an idiot if you don't already have all your code and configurations in a version control system. Once you start using version control, there are several approaches available and, interestingly, online applications work turns out to be profoundly different than shrink-wrapped software.

Traditional Software Development

With shrink-wrapped software, you have features and fixes that are integrated into the product and effectively queued up into what is called a release. Development of features is performed in version control on what is commonly called a "branch" that allows isolation of the developed feature until it is in an acceptable state to be "integrated" back into a main line of development (also a branch) that is used for integration testing. Eventually, the features are merged into a release branch and find their way to clients. Bug fixes and security related issues are addressed in a similar fashion (sometime backwards in the process to fast-track their release to consumers of the product). This is a fly-by, over-simplified description of the typical software development life-cycle.

A lot of people believe that how one manages to a release can profoundly affect the product; two common strategies being "agile" and "waterfall." I'll argue that both are valid, both have their place, and both work in traditional software development. The end goal is the same: ship a quality product within the bounds of expectations set by product management with the clients. Typically, product releases are made available to clients on regular intervals. I've commonly seen three, six, twelve and even 18 month release cycles. Bug fixes, security updates, patches, hot-fixes (they have many names) are released more frequently (monthly, or for problematic products weekly). The client is responsible for upgrading their systems and, if either feature or fix releases happen too frequently, the process can become overly burdensome.

I'll come out and make a rather unconventional claim: the approach described thus far only works well when the number of clients using the software is larger than one. The larger the user-base, the better this model works. It might seem at first that large web sites that have millions of users would be able to use this model to develop their service, but now we've just exposed the crux of the issue. Millions of users use their service, not their product. In fact, in most cases, the only user of the actual software product is that single web site. This alone shakes the foundation of traditional development paradigms. Online environments have many parameters that make this approach untenable.

Get it on(line)

Developing software for an online service, and developing traditional software, have some fundamental differences.

In the online world, a software product drives a service to which users have access. There is, most often, a single copy of the actual software product in use. There is one consumer of the software: you. The users are consumers of the service built atop the software.

Most online services have thousands, if not millions, of users, and as such the tolerance for disruptive upgrades is reduced (often eliminated). You are forced into an environment where each production upgrade happens only once, there are no practice runs and it simply has to work.

In a traditional software model, new features can be distributed to clients that are less risk-averse as a part of an early-adopter program (a.k.a. beta program or tech preview program). This approach allows varied real-world tests of the new features so that when they are made generally available in the product, the confidence in their correctness and performance is sufficiently high. This simply doesn't work when the software you write for your service is only used by one client.

Perhaps most challenging is the pace at which competition moves. In the online world, I can have an idea this morning, an implementation this afternoon and every client of my service that shows up tomorrow will see it. In fact, things can and do happen much faster than that. You might think that rapid concept-to-availability push is reckless. You might be right. But, your competition is doing it.

The question is, how to you maintain a competitive pace and conquer all these challenges, when the odds are stacked against you? The real problem here is that the traditional software model bundles many changes into a release and even the tiniest mistake can result in a failure of the entire release (one mistake can break the whole product). Each change should always be accompanied with a reversion plan. Sometimes those plans are as easy as redeploying the product sans the change, sometimes they are more involved. When hundreds of changes are combined into a single release, the reversion to a previous release becomes the intricate mess of hundreds of change reversion procedures. When posed in these terms, the answer becomes a bit more clear.

Each change could contain a mistake that could cripple the product. However, if we make each change its own release, then the failure is isolated to a micro-release that can be reverted with much less disruption.

This leads to the very controversial technique of "deploying from trunk." Trunk (or HEAD or tip) is a version control term describing the bleeding edge of the product. As people work on fixing regressions and other bugs, as well as add new features into the product, they are adding, modifying and removing code and configuration from version control. If these changes are applied continuously and micro-releases are done continuously, when the inevitable mistake occurs the reversion process is isolated and prevents rollback casualties.

What's a rollback casualty? If I make change A and you make change B and they make their way into a single release, we have a casualty if either (but not both) change has a bug requiring reversion. Due to my mistake in A, we need to downgrade the product to the previous release inducing a rollback of your perfectly functioning work on B. What's worse is that you could have put a lot of work into B ensuring that it was done perfectly because you know that rolling it back would be painful, but I knew that rolling back A would not be disruptive so I was much less careful. This is just a nasty mess all around.

Big changes are scary, there's a lot to test and a lot to plan. By making micro-releases you amortize the risk by investing in deployment efforts in a highly granular fashion.

So the real question is: How do you make this safe? Online applications are not just a piece of code being run. They consist of many moving parts that each change (often independently), but all depend upon each other for correct operation; this is what makes rolling back certain failed deployments so challenging. It might be challenging, but success is sweet: eBay, Etsy, and flickr. It's a tricky balance that combines various philosophies:

• "devops": engineering and operations are married and need to collaborate
• micro-releases: releases must never get too large, instead amortize risk with small, controlled releases
• dark launching features: building the feature out over time in a deployed and operational form to be simply "turned on" when properly qualified
• wired off: the approach that features should have on/off switches to provide an alternative to rolling back deployments
• fail forward: when things go wrong, have a solid plan to work forward to success (within your SLAs) instead of rolling back and trying again later.

Each of these techniques require their own in depth despcription, so we'll leave that for future Seeds articles. For now, just consider that a traditional software engineering mindset can put you at a desperate disadvantage in the world of online software engineering.

]]> http://omniti.com/seeds/marketing-malware http://omniti.com/seeds/marketing-malware Wed, 23 Dec 2009 01:40:21 GMT Internet registrar GoDaddy.com is notorious for two things: domain names and risque super bowl commercials. The infamy began in 2005, when GoDaddy paid $4.8 Million for two 30 second spots during Superbowl XXXIX. The commercial featured WWE wrestler Candice Michelle experiencing a “Wardrobe Malfunction,” a clear parody of the half-time show fiasco involving Janet Jackson the previous year. After having 16 variations of the storyboard filmed and rejected by the Fox Network, version number 17 was pre-approved for broadcast during the first and fourth quarters of the game. The 30 second spot in the first quarter drove a web site traffic increase of 1,600% to the GoDaddy site, and then a strange thing happened: the commercial never played a second time. NFL executives purportedly pressured Fox to pull the commercial due to its “inappropriate” nature, despite the fact that it had already been paid for, pre-approved by Fox, and initially aired. Led by GoDaddy CEO Bob Parsons, the blogosphere screamed censorship like bloody murder, which only served to fuel additional publicity. In the end, GoDaddy deemed the event so successful that they now define their brand around the “GoDaddy Girls,” airing annual Super Bowl commercials that tiptoe along the edge of broadcast acceptability.

At the center of the GoDaddy controversy is the correlation between advertising and brand identity. As a content provider, the Fox Network understands that their brand will be held responsible for the quality of both the program content and the advertising they deliver. If either aspect of the broadcast is sufficiently offensive or inept, they risk losing viewers to other stations. Consequently, Fox must provide content that is tame enough to avoid outrage from extremely conservative viewers while remaining provocative enough to satisfy the desire of third party companies seeking to push the envelope with their ad strategy. Content providers in any medium will be judged by the quality of the content they provide, and that includes the use and placement of advertising. This can be a tough balancing act, and Fox isn’t alone in walking the tightrope.

When it comes to the Web, the importance of managing advertising content takes on a new dimension. Hackers are increasingly using fraud and social engineering tactics to infiltrate advertising networks, and then utilizing their position within this circle of trust to inject malware, browser redirects, and cross site scripting attacks on unsuspecting visitors. If these attacks are successfully executed, hackers may steal credit cards, social security numbers, banking information, personal photos and anything else that has been digitized on the victim’s computer. Alternatively, a visitor’s computer may be turned into one of many “sleeper cell agents” in a botnet, ready to respond to a few keystrokes at any time and become an active participant in a worldwide Internet attack. It isn’t just web site visitors who are vulnerable. This same strategy can be used by black hat marketing consultants to siphon traffic from one web site to a competing web site or even to blacklist an entire site from the Google search index. The worst part? Hackers are able to target their attacks with profound granularity, making it extremely difficult for anyone within the targeted organization to know the attack is even happening.

The risk posed by vulnerable advertising mediums is not merely theoretical. In 2009, I documented two separate exploits that successfully penetrated highly trafficked and popular web sites. Both sites had full time IT teams who were previously unaware that the exploit was occurring. Furthermore, in a now highly publicized event in September of 2009, the New York Times was the victim of a malware advertiser who legitimately purchased ad space from the Times while pretending to be a representative of Vonage. IT departments and technical staff are trained to watch site visitors for malicious activity, but painfully few are watching the advertisers.

Deconstructing the Hack

Client-side arbitrary code execution is the primary culprit behind advertising based attacks. Attackers first gain trust with the target by posing as a legitimate advertiser. This process may be as simple as paying for space in an automated advertising system, or as involved as calling a major corporation while posing as a sales rep or marketing executive from another company. After the attacker has been approved as an advertiser, they develop a custom script to exploit the medium being used. They may start off displaying advertising that looks legitimate, but inevitably they switch to the malicious ad that begins to infect or otherwise manipulate site visitors.

Depending on the amount of freedom granted to an advertiser, a variety of techniques may be used to deliver the hacker’s payload. Many setups allow advertisers to automatically submit a combination of HTML, CSS and JavaScript code to be embedded within the layout of the publisher’s site. In this scenario, hackers can easily embed malicious scripts by using JavaScript or by including an external Flash SWF in the markup. In the off chance that this code is reviewed at all before publication, it is likely reviewed by someone in marketing or sales who is only examining the submission based on the content currently displayed and is unable to analyze the underlying code for potential security vulnerabilities.

Allowing advertisers to place custom Javascript or Flash files inline as part of a web site’s markup is especially dangerous as the advertiser is no longer restricted by cross-domain access policies. This leaves the advertiser with the power to do virtually anything the web site developers can do, such as posting AJAX requests or altering any element of the DOM. In an attempt to prevent this, some web sites have opted for an alternative setup that links an iframe or traditional frame to a server belonging to the advertiser. However, this approach is also flawed because changes to the advertising code may be made at anytime and the publishing site is powerless to implement a pre-approval process or apply any automated content filtering.

Regardless of the method utilized, if an attacker gains the ability to execute custom code on the target site, Pandora’s box has been opened and virtually all the evil of the Internet may be unleashed. A few fictional yet plausible exploit scenarios include the following:

You Won (Malware)!

Johann is a regular visitor to Finance Magazine’s web site. Like most Finance Magazine visitors, he is an investor with a diversified portfolio in mutual funds, bonds, and individual stocks. While browsing the latest financial news, a popup branded with the magazine logo suddenly appears and announces that he won a free 2 Year subscription to FM and a chance to win lunch with Warren Buffett. Lunches with Buffett are normally valued in the millions, and Johann has been wanting a print subscription to the magazine for some time. He clicks “Accept” and is asked to download the registration form. At first he is a bit suspicious and doesn’t understand why the registration form is a downloadable .exe file, but it is a company promotional from the official web site, so he reasons it must be okay. After downloading the application, he launches it and is asked several questions about his financial net worth. He is then asked for his full name, phone number, address, e-mail address and social security number. He is again a bit suspicious and doesn’t understand why he needs to enter his social security number, but the form does looks very professional. He clicks the info button next to the Social Security Number field and is shown this dialogue message:

You must enter your social security number in order to ensure that “Lunch With Warren” contest participants are limited to one entry per person.

With visions of the “Sage of Omaha” in his head and the promise of the next printed issue of Finance Magazine at his door step, Johann fills out the form and clicks submit.

Several months later, Johann’s financial life is in ruins. His personal information was used to register for several credit cards and a bank loan was issued for a Mercedes SLK in California. The executable file he downloaded included a custom Trojan Horse virus that allowed the attackers to login to his personal machine. They used this access to acquire his banking information and passwords, which they used four months later to wire over $10,000 to an account in the Cayman Islands. Although Johann is now suing for criminal negligence, his life (and his credit) will never be the same.

The Black Hat Who Stole Christmas

Brianna is a successful small business owner with an online niche retail store that averages $25,000 in sales and 500,000 visitors per month. In addition to a product catalog, her site also contains high quality articles and videos that pull traffic from Google and allow her to further monetize her online presence by selling custom advertising. Advertising is sold on a month-by-month contract basis, and advertising providers are given an account that they paste HTML, CSS, and JavaScript in so it can be included directly in the site markup. Usually, Brianna’s sales skyrocket for the entire month of December as Christmas approaches. This year, however, she has seen a 60% reduction in traffic and sales are plummeting. After closely analyzing her site analytics, she realizes that traffic from Google has drastically been reduced. She went from being in the top 10 results for her product niche to not appearing anywhere within the top 5 pages, and she can’t figure out why. What Brianna doesn’t realize is that she sold an advertising slot for December to a Black Hat SEO optimizer working for a competitor. As part of his overall strategy for propelling his client to the top, he decided to cut the legs out from underneath the competitors by running advertising templates on their sites that carefully and skillfully violated nearly every technical SEO guideline required by Google. Brianna made a few hundred dollars for the advertising space, but unknowingly violating Google’s SEO rules would cost her tens of thousands of dollars in the year ahead.

Defending Against Malicious Advertisers

Unfortunately, a foolproof technical solution to malicious advertisement doesn’t exist. However, this fact is not an excuse for apathy. The risk to both organizations and web site visitors can be mitigated by applying these technical steps:

1. Preview Ad Submission

   Within your advertising process, a system should be setup to preview all advertising before it is published live. This process should minimally include the ability to preview the way an ad looks and functions, but ideally it will also involve a quick scan of the actual advertising code to check for any unusual pieces of content (e.g. a few lines of encrypted JavaScript). If ad provider’s can automatically update their content, they should be sure that each new version is also approved before publication.

   This tactic can not be used if ad placement is achieved by embedding frames or iframes that link directly to a third party server.

2. Restrict Dynamic Advertising

   The single most effective method of preventing advertising abuse is to ban your advertisers from executing any dynamic advertising code. This process is as simple as stripping any scripting tags from advertising content and is extremely effective. However, the potency of this tactic comes at the cost of advertising flexibility as it limits all advertising to stylized images and text. In a medium where small games and other interactive content is often used to garner clicks and entertain eyeballs, this is often not a viable tactic. When scripting code is permitted, certain dynamic content should still be banned. For example, anything that involves alert() or document.location() calls in JavaScript could be stripped while leaving other code in place.

   This tactic can not be used if ad placement is achieved by embedding frames or iframes that link directly to a third party server.

3. Sandbox Dynamic Advertising

   In scenarios where dynamic content is permitted, it is useful to place ads within frames or iframes to take advantage of cross-domain safety restrictions. Yet as we have seen, using these constructs to link directly to a third party server is a security vulnerability as advertisers can easily change the content displayed without providing the publishers an opportunity to review the changes or strip malicious code. To obtain the best of both worlds, create an advertising sandbox by designating a domain or subdomain specifically to serve advertising content. Then place frames or iframes on your main site that link directly to the new domain. Because you control the ad domain, you will be able to preview ad submission, restrict dynamic advertising, and benefit from the added security that cross-domain security restrictions provide.

Even when implementing the safeguards above, the decision to grant an advertiser space on your web site should not be taken lightly. Doing so inherently confers a degree of trust upon a third party. Ensure that trust is well placed by implementing a screening policy for all new advertising sign ups. Such a policy could be as simple as calling companies directly to verify that the representative is authorized to sell advertising on their behalf or as involved as requiring all advertisers to provide copies of their business incorporation license or other government issued identification. Regardless, the threat can not be delegated to the IT staff and forgotten. Marketing and sales play an equally important role, and the safest organizations are those who view security as the shared responsibility of all the members within it.

There was a time when good advertising meant entertaining video or amusing copy. It could be judged purely on face value and the ability to generate ROI. That time has passed. In an increasingly interactive world, it is now more important than ever for organizations and individuals to understand that advertising consists of both form and function. Ignoring this fact can result in something far worse than assaulting the sensibilities of your audience; it can devastate their lives. Let content providers and audiences beware: the age of badvertising has begun.

]]> http://omniti.com/seeds/business-metrics-too http://omniti.com/seeds/business-metrics-too Wed, 16 Dec 2009 00:00:00 GMT When I began tinkering around with web services as a hobby, it was common to fiddle with an application for days. I would curse and grind and sputter with Apache and hobbled-together programs. This would frequently unearth new challenges: setting up a mail service, creating a database to store user accounts and perhaps pulling content from a third party. Inevitably these minor distractions would monopolize my attention and the original application would be left to gather dust, without any documentation or monitoring in place.

This seems to be a common problem with many professional software development shops. Project managers help to keep the development teams focused, but their goals are still feature-driven with an eye on the next release cycle. The IT and Operations teams are painfully undermanned, left to maintain their systems and services without any training on the care and feeding of their new pet. For the hobbyist or Open Source project, this becomes an annoyance. If you’re running a business, operational neglect can have a dire impact on your bottom line.

Systems Administrators are unconsciously trained to look at everything through a boolean filter. Hosts are up or down. Services are on or off. Their understanding of the application stack is often superficial, limited to the same perspective as that of a typical user. Does the website load? Can I ping the servers? This is a completely logical approach. And yet, it fails to consider those "corner" cases where activity looks normal, but an internal component suffers an unexpected condition. Stealthy failures like these can be missed for months and result in significant lost revenue or wasted overhead.

Monitoring systems have improved over the years with "advanced" features like automatic discovery of hosts and services. Scanning a network, they can identify hosts and differentiate web servers from workstations. Resources are grouped logically. It’s a very turnkey way to add monitoring to your infrastructure. Unfortunately, for many companies, this is where the story ends (and the pain begins). Attentions are subsequently focused elsewhere. Priorities are reestablished. One of the most important resources, the one that makes sure everything else is operating smoothly, becomes forgotten and orphaned. It’s easy to forget about something that doesn’t make your job easier or offer intrinsic value to your bottom line.

A poor economy and high unemployment levels remind us how important it is to optimize our existing architecture. The current trend towards Cloud Computing and Virtualization makes this even more challenging. These technologies are useful for creating highly elastic platforms on a budget, but they complicate engineering by outsourcing data storage and processing to an external black box. In turn, we’re forced to add resiliency in the form of additional processing nodes and redundant storage. This added complexity introduces countless opportunities for disaster. It’s a vicious cycle.

As the Web has become the obvious target for fresh product development, additional layers of abstraction are introduced into the application stack. New technologies and components offer exciting ways to communicate with the end user and from one business to another. The higher we go, the more these layers are decoupled from traditional monitoring proficiencies. The resulting programs are overly intricate and opaque. We need new ways to increase visibility and derive useful data from modern business systems.

Gaining visibility over business operations is probably the easiest improvement any company can make. Quality analytics require a solid understanding of your IT operations and business processes, which come from transparency into your systems. Once these have been established we should be equipped with the tools to streamline and simplify any infrastructure.

1. Key Performance Indictators

   First and foremost, identify the external business metrics that directly affect your revenue. Establish thresholds and put fault-detection monitors into place, just like you would for any server or application. Alerts on business operations (e.g. new user registrations, orders per hour) are more important than the systems that support them. Remember that revenue is an asset, and hardware is a cost. Not the other way around.

2. Review IT Monitors

   Evaluate your existing IT monitoring systems. Ensure that metrics are being gathered for every single host and service. The breadth and depth of data collected now will directly influence the quality of the information that can be extracted later on. It’s paramount to have the metrics to support your decisions, but you won’t know which they are until you can juxtapose them later.

3. Stockpile Data

   Collect as many metrics as possible, for as long as possible. There are no good excuses for not storing metrics indefinitely. Storage is inexpensive, and a variety of technologies allow us to scale capacity with ease. In three years we should be able to look back on data with as much granularity as the information that was collected just yesterday.

4. Highlight Deficiencies

   Graph your metrics. Study their trends and formulate a plan to address the immediate capacity limitations. When deploying new resources, look for hints in the trends that reveal hidden relationships in your network. But remember that this goes beyond planning for the future; this data has inherent value in supporting your ongoing decisions.

5. Build Relationships

   Correlate graphs in ways that represent your business systems. Pinpoint metrics that relate towards a common goal (sales per visit, length of visit, average page size, network latency and webserver load). You might be shocked at the patterns revealed. If your trending application doesn’t allow you to correlate incongruent data easily, find a new one.

6. Empower Stakeholders

   Distribute the accumulated knowledge with individuals and teams within your organization that can take action towards positive change. If possible, give them access to all of the information, not just the data that directly affects them. For large architectures, there is rarely a single person with a holistic view of the entire stack. Trust your partners and there’s a good chance they’ll unearth something you missed previously.

Fault-detection and Trending solutions should return more on investment than high uptime or speedy notifications. They should prepare an organization to increase capacity before limits are reached, realign resources to meet unexpected traffic spikes, help Development & Design teams to better understand your customers, and decrease the maintenance and staffing necessary for normal IT operations. Feedback should be real-time and tuned to the needs of your organization. In a nutshell, it should pay for itself and then some.

]]> http://omniti.com/seeds/transcending-the-medium http://omniti.com/seeds/transcending-the-medium Thu, 15 Oct 2009 16:30:02 GMT I grew up in Boyd County, Kentucky, where John Deere tractors are practically an indigenous species. This now famous brand, started by none other than Mr. John Deere himself, began life with one sole product: the steel plow. Fortunately for lawn care companies and husbands everywhere, Mr. Deere was a man who understood his market. He understood that his company existed to serve the needs of the agriculture industry, and not purely to cut furrows in soil. Because of his focus on the needs of his consumers, Deere & Co. now manufactures hundreds of products and parts in multiple agricultural categories, perhaps the most iconic of which is the quintessential riding lawn mower. Business leaders would do well to take a page from the Deere & Co. play book, especially when it comes to their technology strategy. If John Deere were alive today, I think he might offer the following advice: place your focus on the consumer, and then find ways for technology to serve them. I’ll say it even more forcefully: For just a moment, completely forget about the specific technology you’re using to serve your market. Instead, just think about the lives of those who use it. Why do they use it? What does it help them achieve? Is there a better way for them to reach their goals, even if it means completely stepping outside of the medium you have created?

Consider Facebook, YouTube, and Digg. These are not “web” companies. They are, respectively, communications, entertainment, and news companies. If tomorrow a new technology emerged (and inevitably it will) that allowed consumers to better achieve their goals than the services offered by these so-called “web” companies, all three organizations would be forced to either embrace that new technology or risk significant loss of market share. The kicker is that they aren’t alone: what is true for Facebook, YouTube, and Digg is also true for you.

Now for the secret: tomorrow may have come a day early. In April of 2009, Apple announced that the iPhone App Store had surpassed one billion downloads in just 9 months¹. Consider that for a moment. It took Firefox, the most used web browser in the world², over 4 years to reach one billion downloads³, and VOIP/IM service Skype served its one billionth download after 5 years⁴. Apple did it in 10 months, and they did it in an emerging market with an entirely unique distribution model. As impressive as that was, Apple has recently outdone themselves once again, reaching the 2 Billion download mark in September of 2009⁵, doubling the total number of downloads served in just 5 months.

This kind of exponential growth is unlikely to slow down anytime soon, and Apple’s success with both the iPhone hardware as well as the iPhone App Store has revolutionized the pace of innovation in the smart phone market. Verizon is projected to carry 18 different smart phones powered by the Google Android OS by the end of 2009⁶, and Blackberry now has an “App World” to provide consumers of their smart phone line with third party applications.

What does this impressive growth mean for business owners, executives, and IT managers? It means that consumers are becoming increasingly mobile, connected, empowered, and, ultimately, accessible. As large segments of the consumer market begin to adopt mobile technology, the organizations who benefit the most will be those who evolve to service them, finding new distribution or advertising channels for existing products and in some cases utilizing this technological shift to create entirely new offerings.

Yet the most profitable organizations will capitalize on the opportunities created by the mobile market without becoming lost within them. As smart phones are beginning to reach critical mass, it is perhaps now more important than ever to realize that much of the excitement in the mobile space is fueled by HTTP and the Web, and Internet usage certainly isn’t going anywhere but up in the foreseeable future. In the United States alone, Internet Service Providers have reached a point of market saturation with 71% of all Americans reporting consistent Internet and Web availability at home or work⁷, and 85% of iPhone users report using mobile Safari to browse the web on a regular basis⁸. Savvy managers will apply many of the lessons learned in previous online ventures to future expansion in the mobile space while continuing to build lasting value wherever their consumers are.



In the past 10 years, I’ve had the pleasure of working on a variety of projects that span multiple mediums. In varying capacities, I’ve directly contributed to television productions, radio broadcasts, print publications, trade show exhibits, stage shows and online advertising campaigns. I’ve also served as the lead engineer on a variety of desktop, web, and mobile software projects. I consider myself to be a graduate of the school of hard knocks, and when you dance with lady experience long enough, you begin to realize that the principles of success transcend all mediums. The following are just a few “Transcendent Themes” that I believe must be applied regardless of the medium an organization chooses to operate within:

1. Launching is Not Enough

   “If you build it, they will come.” It worked great for Kevin Costner’s character in Field of Dreams, but in the real world, merely bringing a product or service to market is rarely enough for it to succeed. Good products require good marketing initiatives behind them, and what is true in conventional mediums is also true on the web and the emerging mobile market.

   From banner ads to social media, today’s marketers have more tools at their disposal than ever before. Finding the right mix can be a challenge, but the solution will likely involve a cross-comparison of proven conversion rates against perceived market presence. In English: find out where your market is digitally congregating, look at the proven effectiveness of the tools at your disposal that are able to reach them, and then start a small initial campaign to test your analysis.

2. Sample It to Sell It

   Shopping mall fast food chains face some of the fiercest competition around. Sales quantity is king in a confined space with direct competition, low profit margins, and a product lifespan measured in hours instead of days or months. The presence of free samples in food courts isn’t primarily motivated by desperation but by survival. Fast food chains continue to offer free samples month after month in shopping centers across America because they understand that the cost of giving away small amounts of their product is adequately covered by the sales those samples generate.

   This same principle is applied in virtually every industry. Clothing stores often allow customers to use in store fitting rooms to try on outfits before purchasing. Automobile dealers allow qualified leads the famed “Test Drive,” and the movie industry produces previews of all new releases to entice the viewer to later enjoy the full experience.

   Samples sell, and this age old axiom may be even more effective when applied to digital products than to physical goods.

   Consider the case study of iCombat, a top 100 iPhone application. The month after the makers of iCombat released a free, “lite” version of their paid application, they were rewarded with an 8.73% conversion rate. The most impressive part? A conversion rate of 8.73% resulted in a monthly sales revenue increase of 496%⁹. From the iCombat creator:

   we had waited months longer than we should have to launch a lite version. There was no point to waiting and sacrificing the initial new release buzz.

   When it comes to profiting from the power of sampling, iCombat isn’t alone. Mobile analytics company Flurry released a study of the impact sample applications had on paid application sales and found that there is on average an 85% “free-to-paid” sales lift generated by application sampling¹⁰. From the Flurry report:

   Among your strongest marketing plays in the App Store is to offer a free trial of your game or application. Not only is the App Store designed for this, but also it’s the best way to reduce consumer risk in trying your application, with the goal of eventually getting that user to purchase the full version. Think: money. And from our data, it’s among the most effective moves you can make.

3. Make a Good First Impression

   1/20th of a second. That’s how long you have to make a good impression on the typical web user according to Dr. Gitte Lindgaard of Carleton University in Ontario, Canada. Dr. Lindgaard published an article in Behaviour and Information Technology in which he describes that a visitor not only forms a cognitive bias in the first 1/20th of a second after visiting a web site, but that this cognitive bias, formally known in psychology as the “halo effect,” would also significantly influence the user’s opinions on the reliability and usability of the web site.

   In the words of Dr. Lindgaard:

   ...the strong impact of the visual appeal of the site seemed to draw attention away from usability problems. This suggests that aesthetics, or visual appeal, factors may be detected first and that these could influence how users judge subsequent experience.... Hence, even if a website is highly usable and provides very useful information presented in a logical arrangement, this may fail to impress a user whose first impression of the site was negative.

   While no studies have been published to measure the impact of the "halo effect" in mobile applications, it is undoubtedly an important aspect of the user experience. While a thorough discussion of design principles is beyond the scope of this article, consider the fact that proper use of color alone has been proven to increase brand recognition by up to 80%¹¹. In visual mediums, color certainly matters. Give your users the right impression by abiding by the principles of color psychology shown below:



Success is not limited to any single medium, and neither are your consumers. The companies that realize and apply that to their operations today will be the companies taking consumers into the frontiers of tomorrow. Be among them by placing the needs of your consumers first and then utilizing whatever technology is necessary to serve them.

Footnotes

1. http://www.apple.com/pr/library/2009/04/24appstore.html
2. http://www.w3schools.com/browsers/browsers_stats.asp
3. http://en.wikipedia.org/wiki/Firefox#Market_adoption
4. http://share.skype.com/sites/en/2008/09/celebrating_1_billion_download.html
5. http://www.apple.com/pr/library/2009/09/28appstore.html
6. http://bits.blogs.nytimes.com/2009/05/27/google-expect-18-android-phones-by-years-end/
7. http://www.census.gov/compendia/statab/tables/09s1118.pdf
8. http://macdailynews.com/index.php/weblog/comments/16715/
9. http://www.theapplicationfarm.com/2009/07/just-how-much-does-a-lite-version-help-boost-sales/
10. http://blog.flurry.com/bid/19375/iPhone-App-Store-Marketing-Give-it-Away-to-Get-Paid
11. http://www.colormatters.com/market_whycolor.html
]]> http://omniti.com/seeds/when-commodity-makes-sense http://omniti.com/seeds/when-commodity-makes-sense Tue, 29 Sep 2009 13:34:20 GMT We’d all like to spend as little money as possible to get the performance we desire from our computing hardware. When the term “commodity” is used in relation to computing, it typically refers to products that are mass-produced and widely available, with little to distinguish them other than price. This is in contrast to “enterprise” hardware — specialized, vertically-integrated product lines such as Sun SPARC and IBM POWER that target a narrower slice of the computing market and differentiate themselves much more on features than on price. When I say “commodity”, I don’t simply mean standardized hardware, I mean the cheapest, lowest-common-denominator gear that gets the job done. There are legitimate use cases for both types of hardware, but as with all computing solutions, there are tradeoffs that must be understood to make the wisest choices.


The choice to design an architecture with commodity hardware in mind comes with some enticing benefits. First, instead of one expensive widget, I can afford a bunch of cheaper widgets and spread out my work load among them, which also helps isolate failures and improves the overall continuity of service to my customers. Second, it allows me to scale my solution as the demands of the business grow. Third, money saved by avoiding pricey hardware is freed to be spent in other areas.

Data storage is one market where there is a stark difference between enterprise and commodity hardware. The first time I heard the term RAID, I learned that the “I” stood for “Inexpensive”. Later, I discovered that it is often given as “Independent”. Both make sense in context, but it seems now that the former meaning has been lost when 15K-rpm drives are de rigueur and sit at the top end of the price range. Lowly 7200-rpm or even 5400-rpm SATA drives occupy the low end. This is but one area of computing where commodity drives is not seen as capable of matching more expensive, enterprise drives. However, a holistic systems approach reveals that there are plenty of places where commodity drives makes sense in terms of delivering on business goals and ensuring a high quality of service.

At the high end, dedicated storage arrays with custom hardware controllers filled with 15K-rpm drives define enterprise storage. These are speed demons; the high spindle speed means low latency (around 2ms, compared to 4-5ms for 7200 rpm drives). To drive latency even lower, some arrays use a technique called “short-stroking,” which utilizes only the innermost area of each disk platter, minimizing the distance that the heads must move. Such luxury comes with a steep price. Short-stroking reduces the usable space of each drive, requiring more drives for a given amount of storage. Not only that, but 15K drives max out at 300GB, so obtaining the kind of storage sizes required by large enterprises requires entire cabinets full of disk shelves. 15K drives are power-hungry, hot-running monsters which, at a time when electricity and carbon footprint concerns are becoming increasingly important, means that the luxury of high performance is ever more expensive, outstripping the performance gains of adding more spindles.

By contrast, a commodity storage approach seeks to maximize storage space and minimize costs, both initial and ongoing. The same budget can buy more spindles to keep latencies down and IOPS up. These additional spindles can fit within the same (or less) space and power budget as well.

The commodity storage picture is not all rosy. Enterprise drives are manufactured to withstand the higher level of vibration that comes with putting a lot of drives into the same chassis. They also have lower bit-error rates and higher MTBF than their commodity cousins. Simply put, commodity drives fail more often. “Failure” could be anything from silent data corruption to outright mechanical malfunction. Commodity drives also don’t spin as fast — 7200 rpm at most. That increases latencies, especially on random read loads. Any solution that employs commodity drives must account for these realities and work around them.

Thanks to some excellent, disruptive technology from Sun, namely ZFS, I can design a system around inexpensive, 7200 rpm drives, bolstered by a few SSDs, which provides more capacity with fewer spindles and improves read/write latencies far beyond the capabilities of short-stroked 15K drives. ZFS features such as end-to-end checksums, guaranteed on-disk consistency, intelligent prefetch, and immense scalability make it a good fit for my motley array of cheap disks. If I run low on space, adding more disks is extremely simple and cost-effective. Likewise, features like self-healing data and top-down, metadata-driven resilvering mean that silent data corruption and device failures don't have to be the ulcer-inducing events they once were.

Storage is just one example of how some careful thought during the design process can yield significant savings during implementation. The same theory applies to entire server farms when deploying web applications. An application that is designed to scale horizontally can be run on a large number of cheap servers rather than a few very expensive ones.

Sometimes it doesn’t make sense to go the commodity route. It is as important to know when you can scale horizontally as it is to know when you should not. An application may require more engineering to rearchitect it to scale out than it would cost to buy a larger single machine to scale it up. I'm not just talking about high-end RISC gear either — some relatively large x86-64 configurations are possible. For example, the upper end of IBM’s System x line can be configured with up to 4 4U chassis linked together in an 8-socket, 48-core, 128-DIMM system. That’s a monster box (cue the Tim Allen grunts). If you can run your app today on one machine, and you can plan its growth to fit into a monster box, then the cost-effective approach may just be to use the larger machine.

]]> http://omniti.com/seeds/stacking-the-deck-for-publishers http://omniti.com/seeds/stacking-the-deck-for-publishers Tue, 22 Sep 2009 18:03:36 GMT

Newspapers and magazines have a unique opportunity with online publishing. They have the best content. They have the most talented writers, editors, and managers. The industry has survived everything the world has thrown at it since newspapers first emerged in the 16th century. But, making the most of the Web requires specialist help. Whether publications have a dedicated digital team, or integrate print and web together, there is a distinct difference in how editorial, design, and production need to operate. On the one hand, the recent furor about “deep linking” — where the legality of sites linking to individual pages was finally put to rest — showed the fragility of trying to apply standards that are reasonable in print to the Web. On the other hand The Telegraph has shown with its in-house video and audio studios, evolutionary news room and integrated advertising solution, that a radical content strategy can pay dividends. The whole approach for print media on the Web is evolving. It’s definitely a brave new world, but not everything has changed: content is still king!

Doom, gloom, flourish!

As some commentators, fueled by the news of falling readerships and economic woes, prematurely sound the death knell for print, The New Yorker published a beautifully researched article by Jill Lepore (in print and pixel) entitled Back Issues: The Day The Newspaper Died. In it, we relive the last time the death of print loomed large in America: November 1st, 1765. On that day the Stamp Act came into force. It required printers to affix a stamp to each of their pages, pay a halfpenny tax on each half sheet of paper, and a two shilling tax on each advertisement. Ostensibly, the tax was imposed in the “colonies” by the British Parliament to fund the French and Indian wars, but the backlash was severe. Printers were better placed than most to vent their frustrations. It has been argued that the Stamp Tax was one of the sparks that provoked revolution, bringing the issue of taxation without consent sharply into focus, with the ire and vitriol of printers like Benjamin Edes of The Boston Gazette, and Benjamin Franklin thrown in for good measure. They survived that calamity, and some newspapers like The Hartford Courant are still published today both in print and pixel.

Jill Lapore’s article in January coincided neatly with the month the credit crunch came home to roost. It was looming, it was imminent, then suddenly it was here. About the same time we also heard that the Pulitzer Prize-winning Christian Science Monitor was going online-only. The 27-year-old PC Magazine followed suit soon after. Various teen magazines previously had made the switch, including Missbehave, Cosmogirl, Ellegirl, and Teen. In December and January other publications also went online-only, including Hoy Nueva York, AsiaWeek (now redirecting to Time), and the Kansas City Kansan.

Far from “closing”, as some sources seem to suggest, publications are evolving just as they always have. The shift is perhaps a little faster and more pronounced than previous changes, but only a few titles are swapping print production for online-only. The vast majority of newspapers and magazines are augmenting print with online production. How well the transition occurs is the rub. Many will struggle with a half-hearted approach; others, like The Telegraph, are embracing the change, investing and flourishing. It’s also worth affirming that a publication doesn’t have to appear on paper to be worthy, a fact validated by the Pulitzer Prize Board when they announced in December 2008 that they were broadening the competition to allow online-only publications.

Content is almost enough

Print publishers still have the most important advantage: great content. The Web sometimes can seem packed with titillating dross, but people don’t thirst just for a quick bit of light-hearted refreshment; they also hunger for the substantial. Satisfying both with a content strategy specifically for the Web, putting the right infrastructure in place to support it, and understanding the behavior of the Web’s audience are the keys to success.

The Telegraph is investing in user experience design and using it to sell ad space. Get the user experience right and people will become subscribers and readers more readily than they ever have for print. The potential audience is global. The advertising revenue stream is global. The publishers who grasp the nuances of online publishing first will have the competitive edge to evolve into the first truly international news and feature sites. Those who invested early are already ahead: The Guardian launched in America in 2007. The paper predicts its podcasts will be profitable by April this year. Today, it has an almost equal split of readers, with a third from the UK, a third from the U.S., and a third from the rest of the world.

International audiences have very different requirements from content and advertising. Designing production and technical infrastructure that can deliver location-specific material is the future of truly international publications. The same digital content delivery channels (web site, RSS feeds, email, podcasts, and video) are still relevant. The material and style will respond to the context and the audience receiving it. Hot topics emerge much faster in the new era, and publications need the right technology in place to know what they are and be able to react. Some publishers are already providing location-specific content, and being very sophisticated in how they understand and serve their global audience. Advertisers are taking notice.

“Web 2.0” and all that jazz

Print publications provide an enacted narrative. Readers start at the cover, then flip, and read. They observe the story told by the publication in a pre-determined sequence, with only the words and images from the staff to tell the tale. In contrast, the narrative of the Web can be both enacted and emergent. The narrative emerges from both the published material of the professionals and the audience contributions. These contributions can be within the web site, on personal blogs aggregated by services like Technorati, or on social networks like Twitter. When Tim O’Reilly coined the term “Web 2.0”, user-generated content was at the core of his thoughts. Whatever we call it — user-generated content, Web 2.0, emergent narratives, or reader contributions — it requires an approach that is more sophisticated than just opening up a site to comments. There are many ways in which users can and should be able to reuse content, add their own, and participate. There are also many ways in which publciations can pull in content from around the Web to help them tell the tale. In fact, editing such content is a valuable service. You only have to look at the success of sites like Newsvine and Ffffound to see how citizen journalism can contribute to the industry. It’s not like the idea is new. James Franklin, editor of The New England Courant, had this to say about his editorial policy just before American independence:

I hereby invite all Men, who have leisure, Inclination and Ability, to speak their Minds with Freedom, Sense and Moderation, and their Pieces shall be welcome to a Place in my Paper.

Connecting the dots

The overall objective for newspapers or magazines remains the same as it’s ever been: a large audience to attract and retain high-paying advertisers. Meeting that objective online means combining the content with the best user experience. User experience is not a term found in the print world. Paper is a static medium. It’s a passive experience. Letters to the editor have been the traditional interaction of the audience with print publications. The Web is neither static or passive. It’s dynamic, with content being syndicated, read, and shared in new ways. It’s interactive, with content being augmented, reused and commented on as it’s published. Reader behavior has changed. Expectations have changed. People still want to passively read, but they also want to interact, republish, share, and comment — and they want these on demand. They want a different experience. It requires a different kind of strategy that understands the audience expectations and technology, and describes exactly how publications can use both to be successful.

User experience design

The first step is to have a clear set of business objectives in a reasonable timescale. Publications have to invest. How they invest is the big question. All the solutions are already available to bridge the gap between business objectives and audience behavior. Understanding audience behavior is the first step on the path to profitability. User experience design delivers just that. Rather than asking questions, it observes behavior. From that we can build a clear picture of how the audience experience can be optimized. That may involve more than just tweaking the design of the interface. It can encompass elements like content strategy: what is published, how it’s delivered to people, and how it’s written to achieve the business objectives.

Web application development

User experience design is nothing without the right applications to support publishing operations and deliver the content through the various channels. They should actively help journalists, editors, and managers do their job. Applications should make interaction for readers quick, easy and fun — an adventure. The software has to be secure. It has to scale well as (hopefully) increasing numbers of visitors find the great content, and keep coming back. The experience has to be fast, safe, and helpful. Anything less is a disservice to the reader in much the same way that badly printed text or art would be in print.

Internet architecture and infrastructure

Applications need machines to run on. That means intelligent technical architectures and infrastructure. If the audience is international then the infrastructure needs to be. That means multiple locations. It means twenty-four hour monitoring, often using tools written specifically for the task. It means rapid reaction times to fluctuations in traffic.

After our recent work with the award-winning National Geographic, their readership went up by 500%. The applications and infrastructure behind the site also had to handle massive traffic spikes as stories spread virally around the Web. Having infrastructure that can perform when the spikes arrive is almost an art. It can get expensive, and as we all become more conscious about environmental impact, performance is the answer. Friendster’s situation gave us a chance to show how to scale a site properly. They were about to launch in China and predicted they would need twice as many servers to do so. Page loading times were already slow at 9 seconds. With a little help from us, they launched in China with the same number of servers they already had. They doubled the number of users to 60 million, but pages loaded more than twice as fast at 3.5 seconds.

Stacking the deck

As good as the content might be, or the perception of the brand in the real world, when infrastructure or applications fail, or are hacked or threatened, the brand can be irreparably harmed in the eyes of readers. How this happens is often straightforward: using different vendors for design, application development, and infrastructure turns the gaps between them into critical fault lines. Under the pressure of success or failure, the fault lines are amplified. For example, vendors have to communicate with each other, often having very different processes, and contractual obligations. Trying to fix a problem, innovate, or improve performance becomes expensive because each vendor only understands their specialist area. No single vendor can see the whole story to find the most efficient solution. While all this is going on, the experience fails and the audience falls away in frustration.

The business case for separating different production areas no longer exists. Business objectives are best met with a holistic approach to design, development, and infrastructure. They all fundamentally affect the user experience, which is the single most important factor affecting visitor numbers on the Web. Combining great content with holistic technology will save money and encourage innovation. Publishers who do it early and do it right will give their readers the best possible experience, and stack the deck in their favor for years to come.

]]> http://omniti.com/seeds/concepts-of-cloud-ish-storage http://omniti.com/seeds/concepts-of-cloud-ish-storage Tue, 22 Sep 2009 17:49:37 GMT It’s rare that I write an article simply to educate. Most of the time I am attempting to articulate or justify a position, or simply rebutting someone’s nonsensical yammering. For a refreshing change, I thought I would take some time to educate you on the fundamentals of large-scale data storage. Many people think of “storage as a service” (now being called “cloud storage”) as a magic black box. At the end of the day, it is just bits on disks. And like all things, if you use enough of it, you can more than cover the cost of managing it yourself by simply eliminating your vendor’s margin (insourcing).

There are more and more services providing outsourced storage. The concept is simple: you upload a digital asset to the vendor (via some sort of API or tool), they return an identifying key of some sort (sometimes this key is provided by you, the uploader) and they store the asset for you. To retrieve the asset, you use a similar method to the one used to upload. In the simplest terms, you can think of it as a mapped network drive to which you can save assets, and later reconnect to retrieve them.

By no means is this new technology. However, the idea of managing one’s own storage, combined with growing space requirements and fear of loss due to lack of redundancy, have driven people to want to make this particular problem someone else’s. Making this choice — to solve the problem yourself or to outsource — is always the outcome of several factors: cost, convenience, and safety.

Redundancy: The basics

Let’s take a look at the fundamentals of data storage. We all want our data to be safe. It’s pretty obvious that storing exactly one copy of the data isn’t safe, but it’s actually more complex than you would think — storing two copies doesn’t buy you much without taking a few extra steps.

Before we dive in and explore methods for keeping data safe across systems, we need to realize that one of our fundamental assumptions is invalid. We assume that when we write data to a disk, it will have no errors when we read it back. This assumption is fundamentally wrong. There’s this little evil thing called a bit error (basically, one of the zeros or ones that was written came back inverted). How often this type of error occurs is a probability called bit error rate (BER). The BER on modern spinning disks is usually around 10^-13 or 10^-14. Basically, for every 1 to 10 terabytes you write, one of the bits — when read — won’t equal what was written. A single erroneous bit might not matter for some types of data, but for others, such an error could be disastrous. We write a lot of data these days, and bit errors are silent, so the lesson here is: write checksums with your data.

The classic method of ensuring that data is safe is to store multiple copies on different physical media. Inside a single system, this can be accomplished with RAID1 (mirroring), which makes sure all data is on two physically separate disks. With a bit of (somewhat) clever math, we can take that same data, split it into a few pieces and store each piece on a different drive. We can then calculate a block of parity data, and store that on an additional drive. Retracing the same math backwards shows that we can lose any single disk in the set, and we’ll still be able to reconstruct our data. This is the basis for RAID5. Sometimes systems need to be resistant to multiple concurrent disk failures (hence the introduction of RAID6, which uses an erasure code such as Reed-Solomon).

None of these scenarios are designed to reduce the risk of data corruption. Rather, they were designed to prevent data loss due to hardware failure of one or more underlying disks. One issue with using RAID is that you are storing files on a set of drives, those files consist of chunks of data (blocks) which map to physical blocks of bits on the drives, and somewhere along that path we could lose our way. If a specific physical block goes bad, or somehow becomes unreadable, we can’t easily map it back to a logical object, such as a file. We only find out that there’s a problem when we try to read the object. Another problem with this general technique is that all of these disks live in a single system and if that system fails, all of the data is unavailable (or worse, lost).

So, RAID is designed to keep our data somewhat safe within a single system, but it doesn’t address system failures. The most obvious design is to put all of our information on two systems. There are pros and cons with this approach. On the positive side, once we’ve identified which system holds a copy of our asset, we only need to communicate with that single system to retrieve a copy of the asset — simplicity. The downside here is that we’ve used half of our storage as redundancy, and yet if two of our nodes fail, we’ve necessarily made unavailable (or permanently lost) 1/(N*(N-1)) of our assets. With two nodes, this works out to 100% (of course), and with 10 nodes, it’s around 2%.

Taking a different approach altogether allows us to use half of our storage for redundancy, while maintaining dramatically greater availability.

Erasure codes

High availability of assets in light of system failures is achieved by today’s peer-to-peer systems. Their technical description is clear-cut, yet extremely detailed. By using erasure codes, these systems are able to split data into many pieces (similar to RAID5), but instead of calculating simple parity, they calculate unique erasure codes.

Imagine we split our data into 5 pieces, and then calculate 5 additional pieces of data, any of which could be used to reconstruct any of the original 5 pieces were they found to be unavailable — these are erasure codes. So, with the data in 5 fragments + 5 erasure fragments, we’ve consumed twice the space but can now stand to lose any five pieces before the data becomes unavailable and/or lost. The main drawbacks to such a system are that calculating and distributing erasure codes is much more complicated than simply storing two copies of the same data, and that retrieving data requires contacting at least 5 machines to serve an asset.

This erasure code approach assumes a slightly larger network of servers. With two copies and 100 machines we see 99.8% availability with 5 machine faiures. With a 10 fragment (5 data + 5 coded) scenario, if 5 nodes fail, we maintain 100% availability. In the pathological case where 50 of our 100 nodes fail, the two-copy method would result in an availability of approximately 75.3%, whereas the erasure code method would achieve approximatately 98.7% asset availability.

Back to reality

In peer-to-peer systems, where clients enter and leave the network rapidly, the use of erasure codes for high redundancy is quite necessary. However, in a datacenter environment, with redundancy on each system and maintenance windows that we control, the situation is entirely different. Controlling the servers, their configuration and their region of deployment gives us a landscape on which we can build a sufficiently redundant system with all sorts of advantages.

Reduced system complexity and simple distributed processing are significant advantages that result from having whole data objects like images or documents present on a single node. With this model, we can offload some computational processing to the nodes that hold the data and they can act without consuming additional resources such as the CPU time and network bandwidth required to reconstitute whole objects from their distributed pieces.

At the end of the day, a hybrid/adaptive approach between the two would yield the best outcome. I see that being the next thing in distributed storage. Most of us that are faced with storing large amounts of data have already thrown traditional filesystems and POSIX-compliance to the wind and are looking for fresh, more appropriate solutions to our specific problems.

For now, until these merge, the approach of redundantly storing whole assets makes the most sense. It is simple and easy to build, deploy and administer. It is also trivially easy to understand and troubleshoot.

]]> http://omniti.com/seeds/what-is-web-operations http://omniti.com/seeds/what-is-web-operations Tue, 22 Sep 2009 17:47:35 GMT The field of web operations is one with which I am intimately familiar. For the last twelve years, I have immersed myself in this field and have had the distinct privilege in helping define it. Even now, writing a job description for a web operations specialist is nearly impossible and when I speak with colleagues about what web operations truly is, we all seem to articulate things differently. I wrote an article a little over a year ago after attending the first O’Reilly Velocity Summit. I now sit in a hotel room preparing my workshop for delivery at the second annual Velocity conference and realize very little has changed. While I still believe the definition of web operations is in flux, I truly appreciate a forum in which it can be explored further. I strongly encourage anyone in the bay area to swing by and partake.

While attending the summit that helps plan this conference, I had two epiphanies:

1. a realization of the lack of a career path for people who do what we do (no standard titles, no standard roles and responsibilities and certainly a lack of sex appeal);
2. a clear lack of terminology for the technology requirements that are so common in these environments.

Terminology is easy, in my opinion — you just argue until someone wins. Of course, arguing is a hobby of mine, so I have bias. On the other hand, defining a career path that is an industry accepted path is hard.

The Career: Web Operations

The term Web Operations was used a lot during this event. While it is not awful, I really do not like this term. The hard part is that the captains, superstars, or heroes in these roles are multidisciplinary experts. They have a deep understanding of networks, routing, switching, firewalls, load-balancing, high availability, disaster recovery, TCP & UDP services, NOC management, hardware specifications, several different flavors of UNIX, several web server technologies, caching technologies, several databases, storage infrastructure, cryptography, algorithms, trending and capacity planning. The issue: how can we expect to find good candidates that have fluency in such a nimiety of technologies? In the traditional enterprise, you have architects which are broad and shallow and their team of experts which are focused and deep. However, the expectation is that your “web operations” engineer be both broad and deep: fix your gigabit switch, optimize your MySQL database and guide the overall architecture design to meet scalability requirements.

I struggle with this. Not everyone can be a superstar. More importantly, no one can really start as a superstar. If we use an apprentice model (which is common in industries without institutional support) we limit the total number of able workers in this field. So, how do we (re)define the requirements for a junior web operations person?

We have to have a plan for hiring on people and progressing them through a career path to make this a legitimate discipline. During conversation, one of my colleagues said they just hire people that they think are agile — “If I tell them to know IOS well enough to configure a router and troubleshoot a problem, I expect them to show up tomorrow with a basic understanding of IOS and ready to start typing in commands at a console.” I agree this sort of “no boundaries” attitude is required for the job, but where do you start?

Another person mentioned that the reason for the lack of sex appeal in the position was due to popular attitude. Many people apply for development positions and “don’t quite make the cut” and are instead offered system administration positions. I personally don't subscribe to this philosophy and we certainly do not operate like that at OmniTI, but I have seen it in other companies — I hope it is not prevalent.

Basically, this is one of the few positions in the organization that has no boundaries of responsibility. If something breaks, it is your problem. Why isn’t this the case throughout the organization — why is it that even the most junior of developers doesn't wake up to fix their code when it breaks and causes service degradation in the middle of the night? It is uncommon that this level of responsibility is expected of developers, while it is a quite common expectation of the operations crew.

Circling back, I really do not like the term “web ops.” I realize it is not far off, but it isn’t sexy. Google has a few different roles with this level of responsibility. One I like is called: “Site Reliability Engineer.” However, I would like a set of job titles and a progression through them that makes this an appealing career path for young, ambitious geeks.

In order to define these roles, we should think about what they are responsible for. In our organization I see this as a few things:

Junior

On the junior level, they are responsible for learning. They are responsible for deploying new services and documenting such deployments. They are responsible for instrumenting deployments to make sure that faults are detected and trending is possible.

Mid-level

On the mid-level, they are responsible for all of the above, and more. Effective and complete troubleshooting of failures. Making sense of trending information. Understanding work loads that exist. Tuning systems to better accommodate current workloads and proactive tuning to handle known future workloads. One of the key differences between mid-level and junior is the ability to correctly prioritize remediation of issues during incident response. Staying calm, collected and executing with clarity of thought during an emergency.

What does “complete troubleshooting” mean? I mean troubleshooting without boundaries. I want no shyness in cracking open developer code and telling them what they did wrong and why. Finger pointing at people simply doesn’t work, you have to point your finger at implementation problems, not people. To do that requires the skill to track a performance problem or reliability issue down to a specific line of code or approach.

Senior

On the senior side, technology research and selection is a must. Additionally, they are responsible for incorporating new technologies in the architecture to improve availability and reduce costs, constantly analyzing systems to improve efficiency and capacity planning to understand growth well enough to ensure provisioning and deployment outpace need. Donald Knuth long said that premature optimization is the root of all evil; I've long said that the ability to accurately determine what is premature separates senior from junior.

One of the core responsibilities that all engineering disciplines share is assessing the appropriateness of the technologies at hand. For example, a “Web Architect” must ensure that technology selection as well as development and deployment strategy match the business need. This is “hard.”

Above and Beyond

Web operations is a special role. This role is in no way fitting for failed developers, it is for developers/engineers that have outpaced their career path. One that has a deep understanding of how things work: “a complete systemic view of general site architecture.” However, they want more responsibility, they want to make sure that all of it works all of the time: the app, the stack, the hardware, the network. Whatever technology the business needs, it must work, it must performs and it must be able to meet demand. Lastly, in their heart of hearts, they must believe that all problems are equal in their need for resolution and problem prioritization is dictated by business impact and not by flights of fancy (how cool or interesting the problem is).

It is an impossible job requirement: “Knows everything about all technologies deployed in Internet architectures.” While no one fills this requirement, what I want is someone whose career goal is to find out how close they can get.

]]> http://omniti.com/seeds/orms-done-right http://omniti.com/seeds/orms-done-right Tue, 22 Sep 2009 15:47:47 GMT Object-Relational Mapper (ORM) systems are one of the most contentious topics in database application development. Creating an ORM is notoriously perilous, but using them has pitfalls as well. Most ORMs provide little protection against misuse; the inexperienced developer can easily create an application that unilaterally imposes awkward database design constraints, hammers the database with innumerable queries, and is very difficult to optimize.

ORMs provide an automated link between the application object model and the database model. Practically speaking, this generally means that tables become classes and rows become objects. At the most basic level, this provides per-object persistence services. Most ORMs also handle relationships between objects, turning compositional relationships between objects into foreign key relationships in the database.

In this entry, we present the benefits and pitfalls of ORMs and introduce a new Perl ORM implementation, Class::ReluctantORM. Class::ReluctantORM is “a reluctant ORM for reluctant people.” Its design goals are to create a framework that is unambitious, scalable, and easily circumvented. These goals are not so much technological as philosophical; an approach that has value, as we shall see.

Benefits of ORMs

Why developers love them

Developers see a tremendous productivity gain: persistence no longer has to be hand-crafted into each class. Since all model classes are using the same techniques, there is a large consistency gain as well. Developers also get to keep their head in application-model space, without having to shift gears into database-model space. Context-switching is expensive, and switching between implementation languages can be especially jarring.

Consider getting a list of pirates on a ship:

  my $dbh = DBI->connect(...);
  my $sth = $dbh->prepare(>>EOSQL);
  SELECT p.* 
    FROM highseas.pirates p
      INNER JOIN highseas.ships s ON s.ship_id = p.ship_id
    WHERE s.name = ?
EOSQL
  $sth->execute('Golden Hind');
  my @pirates;
  while (my $row = $sth->fetchrow_hashref()) {
     push @pirates, Pirate->new($row);
  }
  $sth->finish();
  foreach my $pirate (@pirates) {
     # Do something with $pirate
  }

Compared to:

  my $ship = Ship->fetch_by_name('Golden Hind');
  my @pirates = Pirate->search_by_ship($ship);
  foreach my $pirate (@pirates) {
     # Do something with $pirate
  }

The second example is much more legible, and remains entirely in the application model — you don't have to think about how the database is set up, or how the tables interact. You don’t even need to know SQL.

Additionally, most modern ORMs can shield the business logic from limited changes in the database schema (such as table or column renames). While this sort of change is usually better hidden at the database logical layer using a view, organizations that have restrictive database change policies may appreciate the added flexibility.

Why leads love them

Team leads find ORMs to be very useful for several reasons. The most obvious is the reduced amount of time spent wiring up persistence layers; instead, developers can stay focused on the business problems. This enables new capabilities. For example, using an ORM, it’s much easier to knock out a quick prototype in response to an RFP, or explore an alternative design. Additionally, since the amount of SQL is dramatically reduced, developers need not have SQL skills to be productive.

Why project managers love them

Project managers like ORMs for many of the same reasons that team leads do. Because productivity is increased, bids can be lower, or more features can be delivered for the same schedule. This may lead to more contracts. The reduced skillset needs of an ORM-based project can also help solve staffing problems.

Pitfalls

DBA Gripe #1: ORMS that dictate DB design

Some ORMs dictate database design. These constraints typically center around keys. Commonly, primary keys are required to be be single-column, integer, and auto-incrementing. Foreign keys are often under the same constraints. This leads to the proliferation of artificial keys.

Naming conventions are another sore point. Some ORMs require primary keys to be named id, others require them to be named pirate_id, or even pirate. Tables may be required to be named in the plural, and one ORM’s notion of pluralization may not match that of another ORM (e.g. staff vs staffs vs staves).

These constraints are annoying but tolerable if the database design is new and the ORM-based application is the only client. But in most real-world situations, the ORM-based application is only one consumer of the database. It may be a pre-existing design with several legacy apps already using the schema. It is possible to use views and rules to appease the ORM's requirements, but that trades the developer's productivity gain with a busywork task for the DBA.

Finally, there is the issue of schema ownership. Both the ORM and the DB know about the database structure. When a change is needed, where do you make the change? Some ORMs “own” the schema, and will execute DDL to modify the database to match changes in the object model. Others don’t own the schema but instead mirror the database schema in a configuration file. Better ORMs read the database at startup, and configure the object model accordingly (though this has problems of its own, especially related to startup speed).

Class::ReluctantORM — stay agnostic

Class::ReluctantORM is firmly in the “read the schema from the database on startup” camp. Some configuration is still needed — to set up connection handles, declare classes, and to create relationships that cannot be auto-detected. Pushing more of the configuration into auto-configurators, while maintaining overridablility, is an active area of development.

DBA Gripe #2: Opaque, Baroque Query generators

An ORM, by its very nature, must contain some kind of query generation mechanism. SQL is an easy language to generate, but a very difficult language to generate well. There are many dialects. An ORM may choose to generate standards-compliant (but slow) queries, or it may attempt to optimize for the particular database engine. As the optimization increases, the query complexity often increases. Some ORMs choose to punt, generating many simple queries (see DBA gripe #3); others may generate one massive, multi-JOIN query. In either case, at some point you will get the classic complaint that “the database is slow.” A DBA wants to be able to tune and replace these queries with hand-crafted versions. This may or may not be possible. Even if it is, the query generator is buried in the ORM code itself, in developer-land, and often requires both developer and DBA to invest time to optimize a query.

Class::ReluctantORM — query monitors

It is important that the SQL generation process be as transparent as possible. To this end, Class::ReluctantORM provides a unique monitoring facility that provides hooks for several key events in the life of a query, including initiation, SQL generation, execution with bound parameters, result fetching, and teardown of the query.

Monitors may execute arbitrary Perl code at any or all of the events. A monitor may abort a query if needed, or simply log statistics or debugging data. Monitors may be attached at compile time or runtime, and may be attached to a particular class, or all classes in the model.

Class::ReluctantORM ships with six canned monitors, including those for join count, column count, data volume, timing, diagnostic, and one which executes the query under EXPLAIN ANALYZE to predict performance. The developer is free to add new monitors.

DBA Gripe #3: hidden expensive actions

Consider this expression:

  my $jewels = $ship->pirates->first->hideaways->find_by_name('Skull Island')->treasures->first->jewel_count();

While it won’t win any awards for formatting, it is fairly clear: get the number of jewels that the first pirate on my ship has stashed away on Skull Island. It’s easy to imagine a junior programmer writing this, or a journeyman programmer writing a less contrived example.

Does this code, at first glance, look like it is hammering the database? How many database queries will this result in? Depending on the ORM, it may range from 1 to 6. And depending on the database, the 1 query might be better or worse than the 6. In almost every case, the queries involved will pull back more information than they need from the database, so even when the ORM gets the queries right, it’s still likely to have unnecessary overhead.

Or this common case:

  foreach my $ship (@fleet) {
     foreach my $pirate ($ship->pirates()) {
        foreach my $hideaway ($pirate->hideaways()) {
           foreach my $loot ($hideaway->treasures()) {
              tithe_to_queen($loot);
           }
        }
     }
  }

That should have scared you.

Class::ReluctantORM — mandatory prefetching

Looking back at this example:

  # 1-6 queries
  my $jewels = $ship->pirates
                    ->first
                    ->hideaways
                    ->find_by_name('Skull Island')
                    ->treasures
                    ->first
                    ->jewel_count();

This usage is problematic because:

• There is no indication that queries are occurring.
• Any performance issues will be detected in production, not in development.

Class::ReluctantORM does not allow accessors to directly execute queries. Instead, each accessor looks for a cached value, and returns it if found. A cache miss throws a FetchRequired exception. A full-featured prefetching facility is available:

  # One query
  my $ship = Ship->fetch_deep(
    where => 'name' => 'Golden Hind',
    with => {
      pirates => {
        hideaways => {
          treasures => {}
        }
      }
    }
  );
  # Zero queries
  my $jewels = $ship->pirates
                    ->first
                    ->hideaways
                    ->find_by_name('Skull Island')
                    ->treasures
                    ->first
                    ->jewel_count();

This fetch_deep call executes exactly one SELECT SQL statement, JOINing against the related tables. The results are then processed to create one ship object, which has a collection of pirates, each of which has a collection of hideaways, each of which has a collection of treasures. This data is now prefetched, and a long, deep chain of method calls like above is now permissible.

Importantly, if a programmer adds a method call (say, $pirate->parrots) that is not prefetched, an exception will be thrown the first time it is executed. The developer will see this immediately in testing, and add the required clause to the prefetch. This integrates scalability directly into the development process. This feature, unique to Class::ReluctantORM, is what provides its name: it is reluctant to do database fetches.

Software engineering: impedance mismatch rabbit hole

We have a fundamental problem with ORMs: relations aren’t classes, and tuples aren’t objects. This problem is called the “impedance mismatch” between the database model and the application model, and is discussed in detail in several places on the Internet. Some of the more troubling issues include:

• Identity — Two objects referring to the same record are distinct, though there is only one record.
• Partial fetches — Most OOP languages do not have a notion of an object that is only partially populated, but it is perfectly valid (and desirable) to select only a subset of columns from a table.
• Inheritance — There is no clear analogue of inheritance in the database world. Several approaches exist, but they all have severe drawbacks.
• Caching — The object model will be out of date as soon as it leaves the database. Should results be cached? How long?

ORM developers are faced with a few nasty choices. Keep it simple, and let the user of the ORM know that the ORM is an unsynchronized, approximate model of the database. Or gradually add complexity, attempting to patch over the impedance mismatch. The latter path gets into diminishing returns quickly.

Class::ReluctantORM — 90% rule

Some ambitious ORMs try to solve 100% of the object-database problem. Class::ReluctantORM tries to solve the easiest 90%. That means it makes the choice that the impedance mismatch is a very hard problem, and the ORM will do its best, but you still need to be aware of its limitations. This scope limit helps exclude features that would dramatically increase complexity (for example, there is very little support for aggregates).

Skill atrophy

One of the big advantages of ORMs is also a major disadvantage: no, or little, use of SQL. We learn skills through exposure and experience, and if we are never exposed to SQL, we’ll never learn it. Or, if we know some SQL, then use ORMs exclusively, our skills will likely atrophy. In almost every application, the ORM will need to be bypassed at some point, and then SQL skills will be sorely missed.

Class::ReluctantORM — SQL pass-thru

For the remaining 10% of problems outside the scope of Class::ReluctantORM, several avenues are provided to bypass the query generator and use SQL directly. Because it was developed in a shop with a heavy mistrust of ORMs, Class::ReluctantORM is designed to make this bypass as easy as possible. The documentation mentions how to bypass the ORM early.

Avenues of SQL support, ranging from SQL-centric to object-centric:

1. Ask the ORM-managed object or class for a database handle, and execute statements on it. Results are in raw values, not part of the object model.
2. As above, but wrap this into a method call on an ORM object or class, thus integrating SQL into the object model. This is handy for aggregate functions.
3. Future releases aim to provide the ability to override specific ORM-generated queries with your own SQL.
4. Ask Class::ReluctantORM to intepret the SQL into its own representation, and execute. If the translation was successful, return values will be ORM-based objects. This is a Class::ReluctantORM-exclusive feature.
5. Write a query directly using Class::ReluctantORM’s abstract SQL engine. You’re no longer writing SQL directly, but performing method calls on FromClause objects, for example. This is guaranteed to return ORM objects.
6. Use ORM methods and pass SQL fragments as arguments (e.g., a WHERE clause for a search() method).

In all six cases, the developer must use SQL or SQL concepts. This may help reduce SQL atrophy. In many cases, because the SQL can be just “dropped in,” you can have a DBA or SQL expert develop SQL for a specific query with no contact with the ORM.

Framework lock-in

Like any framework, using one is often irreversible. It is very difficult to adapt an application to use a different ORM — even if the interfaces are similar, often times there will be differences among the query specifications, the DB requirements, or the semantics of operations (e.g. do inserts cascade?). ORMs are especially susceptible to lock-in because their footprint is so ubiquitous in the application code. Every time you deal with a relationship between your model objects, you interact with the ORM.

Class::ReluctantORM — unsuprising interface

While there is little that can be done to fight lock-in, Class::ReluctantORM tries to reduce the pain of switching to another ORM, or dropping ORM support altogether, by using common conventions for method names (accessors are named directly after the property, for example). When a new feature is added, the interfaces of other ORMs are studied, and similar conventions are adopted if possible.

Sometimes it’s the wrong tool

ORMs are not good for everything. ORMs by their nature are weaker at these tasks:

• Reporting and summarization — ORMs are good at treating rows as objects. What happens when a column is an aggregate? In this case, raw SQL is much more convenient. Aggregate APIs are often inflexible and complex.
• Anything involving fast startup — If the ORM queries the database for its schema at startup, there will be a lag before the ORM is ready. This isn't a problem for long-running processes like web servers, but it can be a burden for command-line scripts.

Class::ReluctantORM — lots of fish in the sea

If an ORM isn’t right for your project, Class::ReluctantORM won’t help you. Even if an ORM is a good fit, keep in mind it is a slow-startup ORM. There are others that are fast-startup, large-configuration ORMs, and even some that can cache their configuration.

Conclusion

For all their pitfalls, the tremendous productivity advantages of ORMs will continue to tempt developers to use them. Like any productivity booster, ORMs seem to draw a lot of hype, and it’s important to see through the hype to the realities and shortcomings of the technology. Once those shortcomings have been addressed, however, ORMs can be used conscientiously.

Class::ReluctantORM is a new ORM implementation that seeks to make it harder to fall into the traps. It avoids some “impedance mismatch” issues by narrowing its scope to the most common 90% of use cases. For the more complex situations, numerous SQL bypass avenues are available. Whether queries are ORM-generated or customized, they all pass through the query monitoring system, providing an early warning system for scalability problems. Finally, mandatory prefetching can reduce bad coding practices early in the development cycle.

]]> http://omniti.com/seeds/virtualization-zfs-and-zetaback http://omniti.com/seeds/virtualization-zfs-and-zetaback Tue, 22 Sep 2009 03:52:50 GMT It used to be the case that when you wanted to deploy a new application you would need to buy new server hardware to host it on. Today however, there are many different virtualization technologies to choose from, each allowing you to have more than one virtual server per physical machine. Virtualization has a number of benefits — lower cost, power, space, and cooling. Of course, you need to have a machine powerful enough, but many services, especially internal ones such as company wikis and instant messaging servers, do not require the full resources of a physical server, and it makes sense to combine these using virtualization.

In many cases, web applications can be combined on a single server using virtual hosting facilities in Apache, but this is an imperfect solution. Inevitably the situation arises where you have an application that doesn't play well in a virtual hosting situation, be it badly written, or requiring specific versions of libraries or modules that conflict with another application. There are also administrative concerns — anybody who has access to one application has access to them all.

The virtual hosting method also eliminates one of the biggest benefits of using virtualization on entire servers. Many virtualization technologies provide some method of transferring a virtual machine between physical hardware — if a particular server is behaving badly, just transfer all the virtual machines onto replacement hardware with little to no loss of service and without having to reinstall the operating system/applications.

Here at OmniTI many of our servers run Solaris, giving us two very powerful features on which we heavily rely when it comes to making use of virtualization: Solaris containers (Zones), and ZFS.

Virtualization using Zones

Zones provide lightweight virtualization for Solaris. Unlike many other virtualization solutions such as VMWare or VirtualBox, Solaris zones don't emulate physical hardware on which several complete operating systems run; rather, there is one kernel running in the system with multiple partitions (the zones) in which user programs run.

This type of virtualization doesn't force you to pick a set amount of RAM for each virtual machine, or set up virtual disk images (although resource limits can be set for each zone). Because there is no hardware emulation going on, zones are also incredibly fast — fast enough that we are able to run multiple production services on a single machine without any perceptible slowdown. Even for high traffic sites that can saturate an entire (physical) server, we are still able to make use of zones (with just one non-global zone per server) without any significant performance hit. This allows us to benefit from the ease of moving a zone from one machine to another, either in the event of hardware failure, or to migrate to a more powerful machine.

Zones can also ease administration of multiple servers by centralizing package management. By default, any package installed on the global zone is automatically installed to all non-global zones. You can also specify that certain paths are inherited from the global zone, reducing disk space requirements per zone. The inherited paths become read only, forcing them to be the same across all zones. If all packages are installed from the global zone, and you make use of inherited paths, then you can be assured that every zone has the same software configuration.

However, this doesn't have to be the case — you also have the option of installing packages in the zones themselves if different zones need different packages installed. To do this, don't inherit any directories. This creates a 'large' or 'whole root' zone, and you are free to install whatever is needed inside the zone itself.

ZFS and Zones

ZFS has many useful features that put it far ahead of most other filesystems that are available. Several of them are of particular interest in that they make virtualization better: a pooled storage model, snapshots, and the ability to transfer filesystems via the zfs send command.

Pooled storage does away with the idea of having filesystems on individual partitions, and having to guess how much space will be occupied by individual filesystems. You just create one pool across the entire disk (or set of disks) that you want to store your data on. Any filesystems you then create in that pool will only use up as much space as needed to hold the data.

In practice, this means we can create individual filesystems for each of our zones without having to worry about how much space to assign to each. Having each zone on its own filesystem is required to be able to snapshot, backup, restore and transfer zones individually.

Snapshots give you almost instant point-in-time copies of your filesystem, each of which only take up enough space to hold what has changed since the snapshot was taken. The benefits of this are numerous including the ability to roll back to an earlier time and consistent backups (take a backup from the snapshot, and you won't have files being modified while the backup is in progress). From the point of view of virtualization however, one of the biggest benefits of snapshots is in combination with zfs send.

The zfs send command allows you to send a snapshot of a ZFS filesystem from one machine to another (or on the same machine, if you so desire):

# zfs send data/zones/myzone@somesnapshot | \
    ssh remote_machine zfs receive data/zones/myzone

This allows you to quickly move (or copy) a zone from one machine to another: detach your zone, zfs send the filesystem to another machine, attach the zone, and you have your zone up and running on a completely different machine.

You can also make use of incremental snapshots to minimize the amount of time the zone is down (a zone has to be halted in order to detach it): snapshot the zone's filesystem and send it across while the zone is still running, shut the zone down, detach it, snapshot the zone's filesystem once more and send the incremental snapshot across.

Until recently there were issues with upgrading zones that live on a zfs filesystem, but this has been fixed in Solaris 10/08 (u6), and Live Upgrade is now supported. There is now little reason not to use ZFS as the filesystem for zones.

Backing it all up with Zetaback

It doesn't take much thought to realize that the snapshot/zfs send tools can also be used to take backups of systems, especially when you make use of incremental snapshots. At OmniTI we have developed Zetaback, a backup tool based on zfs that automates much of the work of taking and managing backups.

With Zetaback, you specify a list of hosts, the retention policy, and how often to take a full/incremental backup. Then you just let it go. It connects to each host via ssh, scans the host for filesystems to back up, and by default will back up everything, automatically picking up new filesystems. You can filter the list using regular expressions if you want to limit what is backed up.

In addition to taking backups themselves, Zetaback provides tools to quickly restore zfs filesystems, view the status of backups and generate reports showing which filesystems violate the backup policy (e.g. those that have not had a successful backup in 1 week).

The choice to make use of virtualization is often an easy one, the choice of which solution to go with is somewhat harder. If Solaris meets the needs of your applications, then it is worth considering Zones. Combined with the features of ZFS and Zetaback, they provide a flexible and powerful solution.

Some real-world numbers

We're a web infrastructure and development shop, so we run a lot of development servers. Each environment needs the flexibility of its own software selection including version. To accommodate that, we run 37 zones on 2 development servers. Each development server has 8GB of RAM and two dual-core 64-bit AMD processors — in financial terms: about $2300 each. Our production boxes, that serve corporate mail, document management, version control, instant messaging, directory services, etc., all run in zones also. For that we have two boxes (just like the development ones) on which 17 zones happily reside. All of our important services run on a rather small set of machines -- easy to manage, cheap to power and cool. And for our purposes, it is far more efficient than heavy-weight virtualization like VMWare ESX.

We've been running this type of light-weight virtualization for over two years now. We're pretty happy with it. I suggest you give it a whirl.

]]> http://omniti.com/seeds/dissecting-todays-internet-traffic-spikes http://omniti.com/seeds/dissecting-todays-internet-traffic-spikes Tue, 22 Sep 2009 03:52:12 GMT Today's Internet has changed quite a bit from the Internet I used to know. The Internet has always been successful because of net neutrality. What's net neutrality? It's complicated, but essentially it means that anyone anywhere can publish with equal rights. These aren't the kind of rights people usually talk about… I'm not speaking of freedom of speech. Instead, I'm talking about content being simply bits. It doesn't matter if it comes from CNN or my personal blog, you as a reader can download the bits that make up the pages you see without bias or preferential treatment. This makes it darn easy to be a publisher and leads to a fabulous ecosystem with an overwhelming amount of varied content. However, with more content it is easy to recognize that much of it is utter trash. Yes. Yes. I know that one man's trash is another man's treasure. However, it presents opportunities for sites that help you navigate the wasteland.

Many popular sites today are popular because they link to articles and news items and photographs and movies all over the Internet; they are "interest aggregation services." And while the Internet has (for now) a decent preservation of net neutrality when it comes to simple web content, not all publishers are on equal footing. Not long ago, anyone could run a server anywhere (their basement) with DSL or cable or (gasp) dial-up -- now, the challenge is coping with unexpected attention.

Years ago, the site slashdot coined a term "slashdotted" which meant that a site received so much sudden traffic that service degraded beyond an acceptable point and the site was effectively unavailable. This often happened to sites that were at the end of small pipes (DSL, T1, etc.) and occasionally (though rarely) due to bad engineering. While slashdot might have coined the term, they simply don't have the viewership numbers that other large sites today have.

At OmniTI, I work on sites that aren't on the end of T1 lines. Sites with gigabits or tens of gigabits of connectivity. Sites with 50 million or more users. Sites powered by thousands of machines. I also work on sites that service millions of people from just a handful of machines (efficiency certainly has its advantages sometimes). I find it particularly interesting that already popular sites (with significant baseline bandwidth) are seeing these unexpected surges. For a long time, my blog has been on this same machine which is a vhost for several other web sites. I've had traffic spikes from places like slashdot, reddit, digg, etc. And, no surprise, I couldn't actually see the bandwidth jump on the graphs… 10Mbits to 11Mbs? That's not a spike.

Things are changing. Sites like Digg are becoming ever more popular and people are drawn to them as a means of sifting the waste of the Internet. This means as more people rely on Digg and Reddit and other similar sites, the number of unexpected viewers of your content can rise more sharply.

What does all of this mean? It means that the old rule of thumb that your infrastructure should see 70% resource utilization at peak is starting to falter. The typical trends used to look like this (this is last week's graph from a retail client with a user base of 3 million):



We see a nice peak, a nice valley. Thursday afternoon, we see a nice traffic spike. Well, this used to be what I called a traffic spike. Now, different services have different spike signatures. It resembles traffic model of classic Internet advertising, except that there is genuine interest and thus dramatically higher conversion rates. It's a simple combination of placement, frequency and exposure. Because content, unlike ad banners, exists for an extended period of time (sometimes forever), the frequency is very high. Digg and Reddit have excellent placement with very little exposure (things move out quickly). A site like CNN or NYTimes usually provides mediocre placement (unless you are on the front page) and excellent exposure.

Lately, I see more sudden eyeballs and what used to be an established trend seems to fall into a more chaotic pattern that is the aggregate of different spike signatures around a smooth curve. This graph is from two consecutive days where we have a beautiful comparison of a relatively uneventful day followed by long-exposure spike (nytimes.com) compounded by a short-exposure spike (digg.com):



The disturbing part is that this occurs even on larger sites now due to the sheer magnitude of eyeballs looking at today's already popular sites. Long story short, this makes planning a real bitch.

And the interesting thing is perspective on what is large… People think Digg is popular -- it is. The New York Times is too, as is CNN and most other major news networks -- if they link to your site, you can expect to see a dramatic and very sudden increase in traffic. And this is just in the United States (and some other English speaking countries)… there are others… and they're kinda big.

What isn't entirely obvious in the above graphs? These spikes happen inside 60 seconds. The idea of provisioning more servers (virtual or not) is unrealistic. Even in a cloud computing system, getting new system images up and integrated in 60 seconds is pushing the envelope and that would assume a zero second response time. This means it is about time to adjust what our systems architecture should support. The old rule of 70% utilization accommodating an unexpected 40% increase in traffic is unraveling. At least eight times in the past month, we've experienced from 100% to 1000% sudden increases in traffic across many of our clients.

I talk about scalability a lot. It's my job. It's my passion. I regularly emphasize that scalability and performance are truly different beasts. One key to scalability is that a "systems design" scales. Architectures are built to be able to scale, they are not built "at scale." It's just too expensive to build a system to serve a billion people (until you have a billion people). It's cheap to design a system to serve a billion people. Once you have a billion people accessing your site, you can likely justify executing on your design. Google is successful for this reason: their ideas scale and they can build into them as demand rises. On the flip side, traffic anomalies in the form of spikes are unexpected (by their definition) and scaling a system out to meet the unexpected demand is almost unreasonable. I would even argue that it is more of a performance-centric issue. I want every asset I serve to be as cheap to serve as possible allowing me to handle larger and larger spikes.

The reason I find all of this stuff interesting is that understanding performance and scalability, understanding the principles of scalable systems design and having sound and efficient processes for handling performance issues is becoming crucial for sites regardless of their size. This takes insight and practice and it reminds me of Knuth's famous saying:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

That's all well and good, but which 97% of the time? My response to Knuth's statement (with which I completely agree) is:

Understanding what is and isn't "premature" is what separates senior engineers from junior engineers.

Let's add perspective on the word "sudden." Most network monitoring systems poll SNMP devices (like switches, load-balancers, and hosts) once every five minutes (we do this every 30 seconds in some environments). Some people say, "my site scales! bring it on." We see these spikes happen inside 60 seconds and they occasionally induce a ten-fold increase over trended peaks. Often times, this spike can be well underway for several minutes before your graphing tools even pick up on it. Then, before you have time to analyze, diagnose and remediate… poof… it's gone. Be careful what you wish for.

This, in many ways, is like a tornado. Our ability to predict them sucks. Our responses are crude and they are quite damaging. However, predicting these Internet traffic events isn't even possible -- there are no building weather patterns or early warning signs. Instead we are forced to focus on different techniques for stability and safety. The idea of a DoS, a DDoS or the sometimes similar signature of a sudden popularity spike doesn't increase my heart rate anymore -- it's just another day on the job. However, I thought I'd share the four guidelines that I believe are key to my sanity in these situations:

1. Be Alert: build automated systems to detect and pinpoint the cause of these issues quickly (in less than 60 seconds).
2. Be Prepared: understand the bottlenecks of your service systemically. Understanding your site inside and out. Contemplate how you would respond if a specific feature or set of features on your site were to get "suddenly popular."
3. Perform Triage: understand the importance of the various services that make up your site. If you find yourself in a position to sacrifice one part to ensure continued service of another, you should already know their relative importance and not hesitate in the decision.
4. Be Calm: any action that is not analytically driven is a waste of time and energy. Be quick, not rash.

Back to those other countries… Enter China and their recently lessened censorship and we have a looming tidal wave for smaller sites that achieve sudden popularity. Spikes of several hundred megabits per second are difficult to account for when your normal trend is around twenty megabits per second. The following graph is traffic induced from a link from a popular foreign news site (that I can't read). I call it: "ouch:"


]]> http://omniti.com/seeds/yslow-to-yfast-in-45-minutes http://omniti.com/seeds/yslow-to-yfast-in-45-minutes Tue, 22 Sep 2009 03:45:11 GMT The web is a complex beast. There are many moving parts involved in delivering a complete web application today. For a significant portion of my career, I have focused primarily on the architecture and implementation of the parts that an end-user never sees. Racks, servers, databases, switches, routers and load-balancers; the list goes on, but you get the point. The goal of such an architecture, of course, is to receive a user’s HTTP request and construct and return a complete result as quickly as possible. To say that there are “a lot of moving parts” in today’s web architectures is an understatement — they are beasts.

What makes this even more complicated is that once you spew forth the result to the end-user you have a daunting set of user-perceptible performance issues remaining to be addressed. This performance challenge happens in a hostile environment: one we do not control (the user’s computer) over a long-haul network we do not control driven via a browser we did not select — a daunting challenge indeed.

We are very fortunate to have excellent tools at our disposal with which to tackle this challenge. Two of my favorites are Yahoo’s YSlow! and Google’s Page Speed tools. Both are extensions to the most excellent FireBug add-on for Mozilla’s FireFox web browser. Both tools will help you dissect the various aspects of the content you deliver to end-users and understand how each bit will contribute to perceived slowness. In the web (and most other things in life) perception is king. A user’s perception drives their response.

Irony: the not-so-delicious kind.

I recently attended the Velocity conference and the first workshop I attended was Steve Souders’ excellent presentation on Website Performance Analysis. Steve Souders is the original author of YSlow! which I use on a daily basis. Steve used YSlow! to show how to analyze website performance (as one might have assumed from his workshop title). I popped open YSlow! on our corporate website and... horror!

While OmniTI has enormous breadth in the Internet space, we are primarily known as an Internet performance and scalability company. This made the fact that we received an F on YSlow! all the more embarrassing. This was a case of the right hand not knowing what the left hand was doing — something we evangelize against. I decided that I would fix that and aim to do it by the end of Steve's presentation. A play-by-play follows.

No Expires Headers.

It turns out that our images, javascript, and CSS didn’t have expires headers. Our CSS is in a directory /c/, our javascript is located in /js/, and all our images are in /i/. I could do this by content type, but a location-based approach gives me the flexibility of serving dynamic/uncacheable content with those content types if I choose to later:

<Directory "/www/sites/omniti.com/www/i">
    ExpiresActive On
    ExpiresDefault "access plus 1 month"
</Directory>
<Directory "/www/sites/omniti.com/www/c">
    ExpiresActive On
    ExpiresDefault "access plus 1 month"
</Directory>
<Directory "/www/sites/omniti.com/www/js">
    ExpiresActive On
    ExpiresDefault "access plus 1 month"
</Directory>

Using Etags.

Etags are on. This isn’t really a problem in and of itself, but since some of our static content can be served by multiple machines and the Etag in Apache is based off inode, it will be different from machine to machine and cause issues:

<FilesMatch "\.(js|css|gif|png|jpe?g)$">
  FileETag None
</FilesMatch>

Uncompressed content.

This is even easier. We run Apache 2.2, so:

AddOutputFilterByType DEFLATE \
       text/html text/plain text/xml \
       application/javascript text/css

No CDN.

We have a fast CDN-like caching layer residing at s.omniti.net that we can leverage... so I flipped all the images over to that. Technically, this is cheating because you have to add s.omniti.net to the YSlow! configuration to be recognized as a CDN. I was pleased to learn that even without formally moving the images to a known CDN, we still moved to an A rating in YSlow!

Assets served from a domain with cookies.

The move of all static assets to s.omniti.net resolved this issue. This goes to show that even if you don’t have a CDN, simply putting your static assets in a different domain (that has no cookies) can considerably speed performance in two ways: (1) it allows for more concurrency on the network layer and (2) it reduces the upstream payload for quicker requests.

The result?

A noticeably faster web site in under 45 minutes.

Before I fixed things up, it took 486ms to render (over the conference Internet connection).



After, a bit of work, I was able to drop the time-to-render to 315ms over the same link. That's a 35% reduction and it almost drops the page load time down into the “so fast it doesn’t matter” arena.



There are several things I’d like to do that would further improve page load/render times. The javascript used could be consolidated into a single js file (aside from the web analytics parts). The CSS could also be consolidated from two files to one. On our about page we have thumbnail photos of all our staff, they are all the same size and we could easily turn this into a single image and use CSS sprites; that would dramatically improve the perceived performance of that page.

Some things we did right? Our search is wicked fast as we pull the results in AJAX and make a single DOM manipulation to visualize them.

Next steps.

Go fix your site. Make it faster. Make the web a better place. It took me 45 minutes to make significant positive impact. Granted, if I didn’t know your web application or it was more complicated than our corporate site (which I believe all are), it will take a bit longer. It’s worth it. Do it. Or hire us to do it.

]]> http://omniti.com/seeds/rubyrep-yet-anoth-tool-postgresql-replic http://omniti.com/seeds/rubyrep-yet-anoth-tool-postgresql-replic Wed, 19 Aug 2009 03:02:26 GMT One of the key features any enterprise considers when choosing a database technology for their architecture solution stack is that of replication. Oracle and MySQL both have built in replication solutions, but as of yet PostgreSQL doesn't support a built in replication solution. There are many replication solutions available however, and different companies are using different solutions customized for their needs.

Among all of the solutions, Slony is probably the most widely tested and deployed within organizations, although it does have the following limitations:

• Replicated tables must have a unique or primary key
• It does not support replication of large objects
• Schema changes are not propagated (though they can be coordinated)
• It does not support synchronizing databases outside of replication
• There are limitations on version compatability; you can not replicate from PostgreSQL 8.2 to PostgreSQL 8.4 for example
• It is more difficult to set up than many other replication solutions

One new alternative to Slony is a project known as RubyRep, which is designed to avoid some of the limitations of Slony. RubyRep provides both master-slave and master-master replication, and it works for PostgreSQL as well as MySQL. It is currently developed by Arndt Lehmann, a German who has been living since 2001 in Tokyo, Japan. He also provides great support to the RubyRep mailing list, especially for adding new features or fixing bugs.

RubyRep always operates on two databases. To make it simple to understand, the databases are referred to as "left" and "right" database respectively.

RubyRep's key features includes:

• Simple configuration, complete setup can be done via single configuration file.
• Simple Installation, if you have a JVM installed, then you just have to download and extract the files.
• Platform Independent, it runs on Unix and Windows platform.
• Table Design Independent, meaning that all commands work on tables no matter if they have a simple primary key (all data types acceptable), a combined primary key, or no primary key at all. It successfully processes multi-byte texts and "big" data types

In addition to the above, RubyRep actually provides three tools in one; a Compare, Sync, and Replication tools.

Compare

This tool scans corresponding tables of left and right database, looking for diverging data. Key features of the comparison tool are:

• Different output modes, from a count of differences to full row dumps.
• Low bandwidth mode available, reducing the number of round-trips so only actual differences go through the network.
• A progress bar with estimated remaining amount of work.
• Server load is targeted toward only the "right" database server.

In one test we ran, we compared two 50 million row tables in around 3 hours, without affecting production server load. This is accomplished by comparing rows in batches, and you can adjust the batch size in the configuration file.

Sync

The sync tool is used to synchronize data in corresponding tables of a left and right pair of databases. Key features of the sync tool are:

• All features of the Compare tool also apply to syncs
• Automatically orders table syncs to avoid foreign key conflicts.
• You can configure the Sync policy to ignore deletes in left database, or to ignore creating records in right database, and other such combinations
• Provides two prebuilt conflict resolution methods, either left db wins or right db wins
• Custom conflict resolution methods specifiable via ruby code snippets
• Merge decisions can optionally be logged in the rubyrep event log table.

Replicate

Of course RubyRep also provides a replication tool. Some of the key features of the replication tool include:

• Automatically sets up all necessary triggers, log tables, etc.
• Automatically discovers newly added tables and synchronizes the table content
• Automatically reconfigures sequences to avoid duplicate key conflicts
• Tracks changes to primary key columns
• Can implement either master-slave or master-master replication
• Prebuilt conflict resolution methods available include left or right wins, or earlier, later change wins
• Custom conflict resolution specifiable via ruby code snippets
• Replication decisions can optionally be logged in the rubyrep event log table

One of the problems common to replication solutions is that of setting up new nodes. With Slony, there are always some headaches caused by high load on master database server, as a result of the TRUNCATE/COPY cycle Slony goes through. In the case of RubyRep, most of the CPU load is on the slave server, and you can use the Sync command in advance before you start replicating database. RubyRep also provides some flexibility to ignore the Sync commands if you don't want to sync the database again.

RubyRep in action...

• Help

 
; ./rubyrep --help

Usage: ./bin/rubyrep [general options] command [parameters, ...]

Asynchronous master-master replication of relational databases.

Available options:
        --verbose                    Show errors with full stack trace
    -v, --version                    Show version information.
        --help                       Show this message

Available commands:
  generate        Generates a configuration file template
  help            Shows detailed help for the specified command
  proxy           Proxies connections from rubyrep commands to the database
  replicate       Starts a replication process
  scan            Scans for differing records between databases
  sync            Syncs records between databases
  uninstall       Removes all rubyrep tables, triggers, etc. from "left" and "right" database

 

• Generate configuration file

 
; ./rubyrep generate pagila.conf
 

• Compare/Sync Example:

 
; cat pagila.conf
  RR::Initializer::run do |config|
  config.left = {
    :adapter  => 'postgresql', # or 'mysql'
    :database => 'pagila,
    :username => 'rubyrep',
    :password => 'rubyrep',
    :host     => '192.168.0.1',
    :port =>'5432'
  }

  config.right = {
    :adapter  => 'postgresql',
    :database => 'pagila',
    :username => 'rubyrep',
    :password => 'rubyrep',
    :host     => '127,0.0.1',
    :port     => '5483'
  }

  config.include_tables 'users'
  # config.include_tables /^e/ # regexp matching all tables starting with e
  # config.include_tables /./ # regexp matching all tables in the database
end

; ./rubyrep scan -d=keys -b -c pagila.conf  > users_diff.log

; cat users_diff.log
users             users .........................  1
--- 
:conflict: 
- lastname: patel
  zipcode: "2096"
  ipaddress: 192.168.0.126
 userid: 48212620
 address: columbia

; ./rubyrep sync -c pagila.conf
 

• Replication example:

By default, RubyRep runs in master-master replication mode, but you can adjust the following configuration setting to make it master-slave replication:

; cat pagila_replicate.conf
 RR::Initializer::run do |config|
  config.left = {
    :adapter  => 'postgresql', 
    :database => 'pagila,
    :username => 'rubyrep',
    :password => 'rubyrep',
    :host     => '192.168.0.1',
    :port =>'5432',
    :schema_search_path => 'public,pagila' 
  }

  config.right = {
    :adapter  => 'postgresql',
    :database => 'pagila',
    :username => 'rubyrep',
    :password => 'rubyrep',
    :host     => '127,0.0.1',
    :port     => '5483',
    :schema_search_path => 'public,pagila' 
  }

   config.include_tables /./ # regexp matching all tables in the database
   config.options[:auto_key_limit] = 60
   config.options[:adjust_sequences] = false 
   config.options[:sequence_increment] = 1 
   #Sync Policy: Changes in the right database will not be applied to the left database. 
   config.options[:right_record_handling] = :ignore 
   config.options[:sync_conflict_handling] = :left_wins 
  # Additional logging 
  config.options[:logged_replication_events] = [
                                         :ignored_changes,
                                         :ignored_conflicts
                                                ]
#ignore history tables
 config.exclude_tables /_history/
 config.exclude_tables 'test1'
 config.exclude_tables 'pagila'
end

Detailed information for the each configuration setting can be found in the RubyRep documentation. There are also sample configuration files and a tutorial available too for getting familiar with each of the features RubyRep offers. Based on our initial testing, it should also be possible to upgrade some older PostgreSQL databases from 8.2 to 8.4; We found a problem with tsvector support, but that will not affect everyone, and hopefully it can be fixed soon.

]]> http://omniti.com/seeds/under-the-hood http://omniti.com/seeds/under-the-hood Mon, 08 Jun 2009 15:26:44 GMT My perspective on the evolution of OmniTI is somewhat like that of a mechanic on a team of race car designers. As the company changes and becomes more sophisticated, my job has been and still is to ensure we have all the necessary parts to accommodate those changes and that they are incorporated into the new design. So while the techies are customizing the machine (the glamorous part of the job), I am busy working under the hood. So what changes have taken place over the years to keep the OmniTI racing machine on the track and way out in front?

In the beginning our headquarters was located in Theo's house. It was very nice but small. We built out a secure data room there in order to satisfy the security requirements of one of our clients. It had plexiglass windows, rack space and a 200-pound metal door. We personally did the build out which was an onerous job, to say the least! It seems mind boggling to think that we now manage thousands of machines in datacenters all around the globe. From that office we upgraded to a suite of three executive offices in Calverton, Maryland. While staying there we located unfinished office space in Columbia and designed it to accommodate our particular work requirements. We were growing fast and needed office space for our additional staff as well as for meeting with clients.

We started with two people and within three years grew to a staff of four. Our clients required services and support 24/7 so the days were long, as were the nights. Holidays, weekends and vacations were commonly workdays, and working into the wee hours of the morning after a full workday was the norm. The work scaled with the increase of staff from 2 to 4 people, so the workload remained the same across the board. Then we reached a point where we were in complete overload. At that point the engine needed overhauling and refitting to stay in the race. So we interviewed and hired 4 new staff - 3 developers and 1 system administrator. This was a major redesign.

The addition of new staff meant we had to make some serious changes to the shop. We needed an HR department and benefits package that would be both attractive and competitive with other companies -- yet another upgrade. We also had to have a more comprehensive employment contract and that meant having a labor lawyer to advise us. This was in addition to the corporate counsel who helped us craft our client contracts. The pit crew was growing!

After the move to the new site in Columbia, we immediately increased the size of the team. Before we knew it we were fifteen strong and still growing. During this time we formally defined our product initiatives Ecelerity, Postal Engine and MultiVIP and then proceeded to trademark them. We also began to do business as a separate entity called Message Systems. Now we had two race cars on the track!

As the staff grew we began to understand how instrumental our culture was to our success and started to truly nurture it. This was and remains a unique work atmosphere that permeates the operations of the entire workforce. What exactly is the OmniTI culture? Ask ten people on staff and you probably will get ten different answers. I, on the other hand, have been part of that culture from the get-go and can tell you that, however it is defined, it is the heart and soul of OmniTI and the fuel that feeds the machine. The culture is derived from work principals that were instilled at OmniTI's inception. These include providing quality services to meet clients' needs as the number one priority; standing behind our work and being accountable for our mistakes; being passionate about our work; and using mindshare and brainstorming as working tools. Our office is designed specifically to enable and encourage this sort of interactive environment. As a result, the OmniTI culture has attracted some of the best and the brightest in the industry.

After the Columbia office came a second larger Columbia office, and an office down under the Manhattan Bridge (DUMBO) in Brooklyn, New York. We currently have a staff of 7 working in that office.

What about the crew you may ask? Over the years we have been fortunate to have a diverse staff representing a collection of ethnic and cultural backgrounds for which we are all richer. And our team has had many sponsors including trade associations, private industry, not-for-profits, political organizations, and government to name a few. As we zoom around the race track we also take time for the occasional pit stop by having pizza every Thursday, spring cookouts (with serious volley ball games), summer picnics and awesome holiday parties!

So as our race cars become more and more sophisticated they continue to require constant attention to maintain all the working parts. Each day brings new adjustments to the engines and frameworks to keep the motors fine-tuned and the goings smooth.

I'll take this opportunity to share some helpful hints with the other mechanics out there:

1. Stay organized and, despite this digital age, keep hard copies of everything.
2. Retain legal counsel that understands your business and earns your trust.
3. Set deadlines for everything.
4. If you are going to do something, always take the time to understand how to do it right. If you don't have time to execute it right, take the time to document the corners you cut and how that is likely to bite you later.

Zoom-zoom!

]]> http://omniti.com/seeds/increasing-the-aperture-on-security http://omniti.com/seeds/increasing-the-aperture-on-security Fri, 13 Feb 2009 13:05:17 GMT Security is good. Security is necessary. Security is someone else's concern. Security is for our CISSP engineers to focus on in their dimly lit rooms with their lava lamps and empty Red Bulls stacked to the heavens. It's an ugly business, and making sense of it requires professionals with years of experience and dog-eared certificates. It has become an industry unto itself and frightens politicians to adopt far-reaching policies that stoke the smoldering furnace of paranoiac innovation. We attend Hacker conferences and Security summits to behold the latest zero-day vulnerability and poke fingers at the developers who fail to create secure software.

We are Systems Administrators. We are Web Developers. We are Database Engineers and Storage Architects and Pointy-Haired Bosses with hard copies of Schneier's blog littering our desks. We've been told before that Security is our mandate, but to what end? We have heeded the vendor patch announcements and updated our servers. We comply with PCI requirements and pass the Nessus scans. We've studied the latest OWASP Top 10 list and applied its principles to combat the cross-site scripting attacks and SQL injections. Haven't we? Sure we have! Or have we? Have I, as a Systems Administrator, considered the implications of the AJAX interface written by the dev team? Has the DBA considered the impact of an exploit in the chrooted webserver? Have our PHP developers been in touch with the SAN administrator to ensure that he has the capacity to withstand unlimited file uploads, and what effect that might have on our encrypted volume?

A secure infrastructure is not a zero-sum game. While applications on the Internet have become more complex and availability has increased, so have the attack vectors and their impact on the rest of the application stack. We've long recognized that an OSI-centric approach to security is a losing proposition. Firewalls are no longer considered a panacea for network attacks. Modern intrusions exercise multiple layers of the defense perimeter. Engineering secure applications becomes akin to a round of Jenga; how many pieces can we lose before the structure collapses? A cohesive approach to Web Application Security mandates a holistic approach across the entire engineering organization. But how many of us think beyond the edges of the envelope that defines our professional skills and aptitude?

Most of us gain our skills through a function-oriented approach. We learn repeatable steps that culminate in the desired result. Often this includes a period of trial-and-error where we assimilate aspects of the misstep and adapt to avoid the failure condition. This is only natural and is reinforced by our trainers or educators in order to attain the prescribed goals. However, these tactics can also be used to seek out the stress points within a system. When you increase your knowledge of the entire application stack, you reveal "opportunities for efficiency" by understanding the relationship of each component to the whole.

Unfortunately, the churn of modern software development fosters a "feature-first" mentality. We're all familiar with the process. Marketing or Project Management determines the feature set that will drive purchases and upgrades for the client base. Deadlines are aligned with revenue cycles rather than product maturity. In the end, Developers feel the squeeze from Project Managers focused on their calendars and from Customers, frustrated by another release as unwitting QA test subjects. Engineers are forced into specialization, becoming a cog with a narrowed focus. Our aperture begins to close, decreasing our exposure to the application stack and impeding interoperability with other project teams and departments. Although we've entered the Information Age, popular software engineering practices are still rooted in the assembly line mentality.



The birth of secure software requires more than a commitment to correct code and elegant program design. A sort of Renaissance man is needed, a polymath who familiarizes himself (or herself) with the belts and pulleys of neighboring components. Someone who has a passion for the whole system. Orthogonal studies should become a desired trait, not a distraction. Database Administrators, Network Engineers, Java Developers and Systems Administrators working in harmony.

Hackers, in the acceptable sense, are a strange breed. By definition we enjoy the art of deconstruction. We want to know what makes things tick. Perhaps it's something in our biological blueprint that drives our thirst for knowledge. We have an insatiable curiosity for understanding the misunderstood or unknown. An expensive hobby, for certain. Whether that price comes in the form of relationships, money or spare parts, it matters not. The knowledge of how that system works is compelling enough to hold our attention for hours and days and weeks and years.

Or maybe we just like breaking stuff.

After an attack, it's only natural to wonder what motivated the attacker to focus on us. We comb through the evidence looking for the exposed stress points. But to focus on the bugs is to ignore the problem and merely serves to reinforce the broken processes. The same vulnerability will crawl out of a different hole next time. It will wait for the next hacker. And they will come. The hacker has already created new tools to make it easier next time. The hacker is generous. He likes to share his toys with his hacker friends.

Engineering teams with a foundation in "whole system" design stand a better chance of resisting and recovering from attacks. By studying different layers of the application stack they gain an understanding of the operational complexities and attack vectors. They can predict vulnerabilities in the design and planning phases. They can isolate exploits faster and pinpoint failures in unfamiliar regions. New code becomes inoculated by the changes in philosophy. Junior programmers and administrators pass these principles on to their peers. A new Renaissance begins.

]]> http://omniti.com/seeds/embracing-failure-to-rise-above-enterprise-class-enterpris-class-thinking http://omniti.com/seeds/embracing-failure-to-rise-above-enterprise-class-enterpris-class-thinking Fri, 13 Feb 2009 12:59:05 GMT Failures in technical systems are inevitable. Drives die, network interfaces wink out, backhoes take out cross-country backbones, data rooms flood. The effects of such failures range from minor inconveniences to crippling outages, but thoughtful planning can greatly increase the possibility that the next failure will be the former instead of the latter. The fact is that 100% uptime for 100% of users is an unrealistic goal. Creating an information technology infrastructure that expects failures and minimizes user exposure to those failures is critical to preserving continuity of service to the majority of users. This is the point of transcendence into carrier-class thinking.

Military planners always factor in casualties when deciding on a plan of action. The failure of an individual component (a soldier, a tank, an airplane) is expected, but the overall goal will still be achieved. Many enterprises do excellent risk management in their business operations but fail to apply those same principles to their IT infrastructure. The mantra of smart investing is diversification; likewise, in the insurance industry, the goal is to spread the company's risk among a wide population, only a few of whom will actually make a claim in a given year. Yet when it comes to IT planning, all the eggs go into one large (often expensive) basket. No amount of money can ensure that a single point of failure will never fail. That money would be better spent on engineering around failures, to design systems that fail gracefully, or that at least fail only partially, limiting the damage to some subset of users. Put another way, plan for the failure of the most critical piece of infrastructure and engineer service continuity despite that failure.

I like stories. As a systems administrator for more than 10 years, I have my fair share of them, both good and bad, and the really memorable ones have valuable lessons to teach us about how to construct systems that allow most users to see little or no interruption to their service.

In my ISP (Internet Service Provider) days, the company I worked for had one physical server hosting email. It was a relatively large, expensive UNIX server, but it got the job done and had impressive reliability compared to PC hardware of the day. As the ISP business expanded, the demand for email grew beyond what the server could handle, and when the inevitable outage occurred, it affected every single mail user in the system. The solution was, of course, to get more servers. It was not cost-effective to grow with more big UNIX servers, due to a number of factors such as rack space and power, not to mention the capital investment. We needed more (and smaller) servers to store the mail, to both absorb our growing capacity of users and to reduce the impact of an individual server going down. The system we came up with decoupled mail routing from mail delivery and mailbox access. This enabled us to deploy lightweight MX (Mail Exchanger) servers that didn't need much in the way of local storage, as all incoming mail was delivered to some other host. The MX servers were behind a load balancer, so we could scale them horizontally as required to keep up with demand. The mail storage hosts had more local storage, utilizing RAID (Redundant Array of Inexpensive Disks) to survive disk failures, and had standby hosts to which all mailbox data was replicated in case of host failure. Gluing it all together was a set of proxy hosts backed by LDAP (Lightweight Directory Access Protocol) to locate users' mail storage host and handle mailbox access. The directory service was also used by the MX hosts for inbound delivery, to locate the appropriate storage host. Users connected to the proxies instead of directly to their mail storage host. We could do quick maintenance or handle short outages without most customers ever realizing there was a problem. For example, POP (Post Office Protocol) clients checking for new mail would be given a "no new mail" response when the backing store was unavailable. This architecture was much more resilient to failures, and in the event of a failure (or even a maintenance event), the existence of the proxy between users and the actual server allowed us to reduce the users' exposure to the problem.

The next illustrative story comes from a client who operates a large email infrastructure supporting millions of users. Their mail storage sits on a SAN (Storage Area Network), implemented on three expensive, vertically-integrated systems from a major vendor and interconnected on a costly Fibre Channel switching fabric (which is, as ZFS author Jeff Bonwick puts it, "a network designed by disk firmware writers. God help you.") The result is a very high ratio of spindles to control units, so when there is a problem with one unit, that problem affects one-third of their customers, which could run well over several million users. That's a lot of eggs in one basket. The price of the basket does not guarantee an absence of problems-- the redundant control heads must run the same firmware version, so a firmware bug will wipe out both of them. The cost of the storage platform is sufficiently high that scaling horizontally becomes prohibitively expensive, and doesn't go very far to address the spindle-to-control-unit ratio. What they need is a fundamental shift in storage planning. More and cheaper baskets to hold fewer eggs each, so fewer eggs are lost when a basket fails. In this case the baskets are commodity servers and direct-attach storage running free software and exporting block devices over iSCSI (Internet Small Computer Systems Interface) to the servers handling client connections. For the cost of one of the vendor-supplied storage systems, we get nine new storage nodes, each with redundant control heads and data storage. These nine nodes provide the same amount of usable space as the three old units, and have capacity to spare. The cost savings enables more nodes to be purchased, and facilitates horizontal scaling to meet demand. Additional cost savings are realized on the interconnects, which can be standard 10Gb Ethernet. The larger number of nodes means a three-fold decrease in the number of users exposed to a node failure, and future scaling only decreases this number further.

Turning away from email, my final story covers data warehousing for a large, web-focused marketing company. Their OLTP (Online Transaction Processing) database that backs the website runs on Oracle. They need a separate place to run intensive data-mining queries and transformations that are not appropriate for the OLTP system, for which the typical solution is an Operational Data Store (ODS), a type of data warehouse. Initially this was another Oracle instance on a single server. When the size of the dataset grew beyond the capacity of the server, a decision had to be made. A server with enough memory and CPU power to handle the load would have exceeded the Oracle product license, but purchasing additional licenses was cost-prohibitive. The solution was two-fold: convert the ODS to the open source PostgreSQL server, and put it on two systems instead of one. The conversion to PostgreSQL is outside the scope of this article, but the decision to use two servers provides several distinct advantages. First, they are not set up as master/slave, which keeps the setup simple. They both replicate from Oracle in parallel, having no awareness of one another. This works fine since the data-mining queries are essentially read-only (some jobs do data transformations, but they operate on temporary tables.) Second, both systems are fast enough to handle the entire operational load, so if one system is down, all its jobs can be shifted to the other with no degradation of service to users. Third, upgrades to PostgreSQL can be tested with live data without disrupting service, as jobs can again be shifted away from the instance being upgraded. Under normal circumstances, both servers are used for production work, yielding the best return on investment.

These stories illustrate the advantages of expecting failure and engineering around it to create robust internet architectures. Failure is inevitable, but dire consequences need not be.

]]> http://omniti.com/seeds/the-irony-of-sun-database-technology http://omniti.com/seeds/the-irony-of-sun-database-technology Fri, 13 Feb 2009 12:49:07 GMT It's been just over a year since Sun announced it had agreed to purchase MySQL, the ever popular open source database technology. At the time most people saw the move as a way for Sun to make it's way into the internet space, where MySQL has made a lot of in-roads. There was also a thought that Sun might be able to help MySQL overcome some of the technical hurdles it faced when moving into enterprise level usage.

So after one year, what has this marriage produced? Well, they finally pushed out the 5.1 release which had been stuck in development for a couple years, although it didn't include any major changes from what was on the road-map before the Sun purchase. They also pushed out a fork of the code base, which took the interesting step of removing features, rather than adding the enterprise technology many people were looking for, leaving Sun a bill for $1 Billion dollars but without an "Enterprise" database to call it's own. The irony of all of this is that, even before the MySQL purchase, Sun already had a product containing technologies similar to today's leading commercial database, it's just that the technology lives in a file system, specifically ZFS (Zettabyte Filesystem).

One of the basic tenets of a database system is that you can guarantee that data is safe on disk, and generally that any database will give you a chance to throw away changes if you need to. In the database world, you know this as COMMIT and ROLLBACK, common operations to most people, although missing from the MYISAM technology that Sun purchased from MySQL. In the ZFS world, while not implemented the same way as in a database, these ideas are embodied in the commands zfs snapshot and zfs rollback. Both of the commands work with active data partitions, and work so well that you can use them as protection in large batch command style operations against MYISAM; simply zfs snapshot your system before hand, run your large MYISAM command, and then zfs rollback afterwards if you find you need to go back.

Of course what good is a system, database or any other, if you cannot back it up? The back-up process for MySQL is straightforward, although it's use of LOCK TABLES makes it a second-rate solution at best. Consider with any sufficiently large system, LOCK TABLES will keep you from providing five nines uptime almost by definition. ZFS on the other hand gives you the ability to make backups with ease. Once you have a snapshot of your system, the ability to clone, promote, or send a snapshot gives you quite a bit of flexibility for backing up your system, and it can all be done on-line.

But it gets even better really. One of the things that many databases deal with is caching data files from the file system, using some algorithm to determine what should be kept in memory. In MySQL, the database only caches index files (data files themselves are left to the OS to handle), and it does so using a simple LRU (least recently used) cache; a caching mechanism where the least recently used data is purged whenever new data requests are made. Again, ZFS contains something more sophisticated. ZFS uses something known as an ARC (adaptive replacement cache), which improves upon the LRU idea by keeping track of not just how recently something is used, but also how frequently it is used. Again the nature of work being done makes for different specifics in implementation, but other database have looked at and implemented ARC systems and seen significant improvements over the LRU method.

And still there are other examples, take the ZFS intent log. The ZFS intent log is used by ZFS to gather systems calls in memory and log them, both for purposes of performance; system calls can be aggregated together before execution; and crash recovery; in the event of a crash, ZFS can examine the log and replay any system calls that did not finish execution. Of course those familiar with databases will recognize this approach, as it is commonly implemented as a transaction log within database systems, for much the same reasons; commits to the database can be aggregated for performance, and in the event of a system crash, the commit log can be replayed to ensure all committed transaction made it to disk. Unfortunately, MYISAM, the storage engine owned by Sun, does not get these benefits.

Now, we must say that MySQL has been around for some time, so it's users have gone through the trouble of finding workarounds to the lack of functionality we've been seeing in ZFS. Luckily Oracle provides a storage engine for MySQL, known as InnoDB, which implements much of the features discussed above. Also MySQL has simplified replication support built in, which allows for users to set up multiple copies of the database without significant effort. In fact, these techniques are encouraged, as you can use the slave database system for taking backups or for crash recovery in case of loss of the primary node. What we think is often overlooked is that here, the database, which should be a model of data integrity and robustness, gives you workarounds and tools like CHECK and REPAIR, while the filesystem, what you typically expect your database to protect you from, is so carefully designed in ZFS to ensure data integrity, that CHECK/REPAIR are unnecessary.

Unfortunately the cynic in us has to wonder if we will ever see some of the more sophisticated ideas from ZFS make their way into MySQL. After all, since the current workarounds tend to require running multiple instances, and Sun is in the business of selling hardware (either multiple servers, or servers large enough to house multiple virtual servers, take your pick), keeping things status quo creates a nice relationship between these two divisions of the company. Given that, maybe there is no irony at after all.

]]> http://omniti.com/seeds/custom-trending-and-the-benefits-of-source-code-availability http://omniti.com/seeds/custom-trending-and-the-benefits-of-source-code-availability Fri, 13 Feb 2009 12:40:33 GMT One of the self evident truths about system administration is that you need to know what is going on with your systems. Monitoring - knowing that your systems are working as expected and, more importantly, knowing when they aren't - is the thing most people consider first when they realize that fact. Equally important however, is trending - knowing what your systems were doing in the past. Trending allows you to determine if the current state of your system is normal, or if something has changed that could signify a problem. Trending also allows you to predict when your current systems will become unable to handle what is expected of them. When the amount of traffic to your website is about to outgrow your systems, you can see this and add more capacity, either by adding servers or replacing them with something more powerful, before the capacity problems start to occur.

At OmniTI, we use a number of systems for trending, including Cacti and our very own Reconnoiter. Each system comes with a large number of monitors built in, allowing you to trend anything from network traffic, to system load to disk space. Sometimes however, you need metrics for which there is nothing currently available. This is where these systems' extensibility comes into play.

The following example shows a situation where we needed information that our current monitoring/trending systems were not able to provide, and we needed to extend them with a custom trending solution:

One of our clients had a website that had become very popular, and was suffering performance issues as a result. We suspected that at least part of the system was I/O bound, and so we wanted to gather metrics on the I/O performance of the system over time. The systems in question were running Solaris 10 with the data on ZFS. The normal iostat command, for which a number of monitors exist, does not give true values for reads and writes performed by ZFS. Iostat can only see read/write requests from filesystems. True I/O statistics can be obtained on the command line by running the zpool iostat command. This works in a similar way, producing output similar to the following:

# zpool iostat rpool 10 5
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
rpool       16.6G  57.7G      1      0  43.5K  2.06K
rpool       16.6G  57.7G      0      0      0      0
rpool       16.6G  57.7G      0      0      0      0
rpool       16.6G  57.7G      0      0      0      0
rpool       16.6G  57.7G      0      0      0      0

The statement above that our monitoring systems were not able to provide the information that we needed isn't quite true. Elsewhere, we had a monitor that obtained zpool I/O statistics using a long running zpool iostat process, taking the values out and entering them into a database, with a custom script that fetched the values from the database and entered them into cacti. The system in question was a database server, so this method, while clunky, worked well enough for its purpose. For monitoring the web servers however, using the same method just wasn't practical and we needed something better. We needed something that didn't require running a long running process and running a database server on the machine just for trending information.

The obvious choice here was to use SNMP. Cacti (as well as pretty much every monitoring/trending package) has built-in support for obtaining data over SNMP, and net-snmp (the snmp agent in use on the server) has various ways of extending functionality to get custom metrics.

Having chosen SNMP, the next decision was how to get the data we needed and present it over SNMP. The seemingly obvious choice would be to run zpool iostat and parse the output as was done previously, presenting those values over SNMP. However, that either requires the long running zpool iostat process, or running it once for a few seconds at a time to get a snapshot of the I/O over that period, which will lead to inaccurate results (it won't tell us anything about the performance of the system during the time between checks). One of the things that Cacti (or rather rrdtool, which cacti makes use of) is very good at is taking raw data and generating meaningful statistics from it. If we could somehow get raw I/O values rather than already aggregated values such as 'n KB over the past m seconds read' and pass those to cacti, then cacti could do the work and we would get accurate values.

Enter open source. The source code to OpenSolaris is available, including the source code to the zpool command, which it possible to see how the zpool iostat command itself worked. Once you trace the various calls made by that function, it turns out that underneath, the zpool iostat command uses libzfs to fetch the exact raw values we are looking for. It was then a relatively simple matter to take that code and print out the raw values:

#include <stdio.h>
#include <sys/fs/zfs.h>
#include <libzfs.h>

/*
 * Sample code to demonstrate printing of raw zpool io stats.
 * Compile with: cc -lzfs -lnvpair zpoolio.c -o zpoolio
 */

int print_stats(zpool_handle_t *zhp, void *data) {
    uint_t c;
    boolean_t missing;

    nvlist_t *nv, *config;
    vdev_stat_t *vs;

    if (zpool_refresh_stats(zhp, &missing) != 0)
        return (1);

    config = zpool_get_config(zhp, NULL);

    if (nvlist_lookup_nvlist(config, 
        ZPOOL_CONFIG_VDEV_TREE, &nv) != 0) {
        return 2;
    }

    if (nvlist_lookup_uint64_array(nv, 
        ZPOOL_CONFIG_STATS, (uint64_t **)&vs, &c) != 0) {
        return 3;
    }

    printf(
        "pool:%s read_ops:%llu write_ops:%llu " \
            "read_bps:%llu write_bps:%llu\n",
        zpool_get_name(zhp),
        vs->vs_ops[ZIO_TYPE_READ],
        vs->vs_ops[ZIO_TYPE_WRITE],
        vs->vs_bytes[ZIO_TYPE_READ],
        vs->vs_bytes[ZIO_TYPE_WRITE]
    );
    return 0;
}

int main() {
    libzfs_handle_t *g_zfs;
    g_zfs = libzfs_init();
    return(zpool_iter(g_zfs, print_stats, NULL));
}

Once this was done, the next step was to get the values exported over SNMP so that cacti could view them. Net-SNMP has a pass directive that allows you to delegate an OID to an external program, and have that program print out the results it needs. These values are then exported over SNMP, available for any of the above monitoring tools to make use of.

In Cacti, it was then just a matter of creating an appropriate SNMP Data Query, adding some Graph Templates, and wait for the pretty pictures to come flowing in.

This example shows how you might approach developing code to obtain custom metrics, and shows the benefits of having the source code available so that you can learn from tools that do most, but not all of what you are trying to achieve. Sometimes, what you need just isn't available and you just have to build a solution from the available pieces.

]]> http://omniti.com/seeds/using-less-is-green http://omniti.com/seeds/using-less-is-green Fri, 13 Feb 2009 12:23:00 GMT Every time I hear about green computing I feel like there is a gap — an enormous gap. The same thing is true in most conservation efforts I witness:
• I see grand plans to make it easier and cheaper to produce foods, but no trend to actually eat less;
• companies make hybrid vehicles and the consumers flock to them without regard for the environmental manufacturing costs and even as these issues are solved and hybrids have less per-mile environmental impact people will still drive too much;
• the fast adoption of florescent light bulbs (and now LED) and yet people still leave their lights on when they don't need them.

I tend to argue a point where I believe my point is right and the alternative is wrong. In this unique case, I find that the alternative is right, but just not "right enough." We could do better. We should do better. Think complete.

Green Computing through Hardware Optimization

So much focus is placed on making equipment (processors, ram, storage) more energy efficient that people are losing sight of the bigger picture. Energy efficient equipment is certainly one piece of the puzzle. Unfortunately, too many people see that one piece as a fait accompli in their energy conservation efforts.

At OmniTI we're always careful about fully understanding the power profile of the hardware we install. We are conservative and look for the most power efficient machines we can find that still meet our architectural requirements (which can vary wildly from component to component). Everyone should do this. IBM and HP and Intel are all telling you that you should do it and that they can help. Do it. Let them. But please, don't stop there.

Green Computing through Virtualization

The next step that is popular in the efforts to save your wallet (and the planet) is consolidation. This is the philosophy that one of today's machine is powerful enough to accomplish the goals of many of yesteryear's machines. So, virtualize! Take the old machines, turn them into virtual servers and run them on one machine today. Virtualization (of one type or another) has many advantages including: ease of management, simplistic disaster recovery, flexibility in technology selection, shorter provisioning times and the opportunity for consolidation.

Many of our engineers run VirtualBox or VMWare to quickly launch the platform of their choice. They are allocated one machine each, so they only have the opportunity to use a certain number of watts. Virtualization makes their job a bit faster and a bit easier despite the user experience being ever-so-slightly slower than running native. This use of virtualization does not reduce energy consumption in any significant way though it does increase individual productivity.

We have development environments that are managed by the operations team here that must resemble (as closely as is economically feasible) the production environment to which they deploy. We have many of these and they are all distinct, but not heavily loaded. It is feasible that consolidation could be used in this approach. Our actual situation is that we have to operate 40 isolated development environment. We do this on…

• Two $2300 1U machines
• Solaris Containers (Zones) as the lightweight virtualization technology
• at about 200W run rate, which results in about 3.5 MW-hours per year

If you considered the alternative naive implementation:

• 40 1U machines
• at about 180W run rate, which results in about 63.1 MW-hours per year.

We realize a savings of 59.6 megawatts. Wow! Now, that is an utterly naive method. Instead, let's look at a popular method like VMWare ESX:

To run 40 VMWare instances…

• I need some substantially bigger hardware at 2GB of RAM per instance (Solaris containers and other similar technologies have some memory sharing efficiencies).
• We only have 40 instances here, so going the blade center route seems less compelling.
• An IBM x3650 should be able to manage about six instances (which aren't peak and can afford some occasional performance degradation).
• Seven of these at 230W each we burn 14.1 MW-hours per year.
• This assumes you use local storage. If you need a SAN, you'll have to add that into the power profile too.

One can say they burn 14 megawatts per year instead of 63! But to me, burning 3.5MW is even better. Now, for those financially responsible types, I've only spoken to recurring operational costs. If you run the numbers on initial capital investment you'll see an even more significant savings by simply choosing the right tool for the job (between $80k and $100k by our internal calculations).

This isn't to say that you should never use VMWare or a similar heavy-weight virtualization technology. Those technologies afford you specific advantages (like the ability to run entirely different operating systems in each instance). You could also consider something slightly lighter-weight like Xen. But, if you find that your virtualization requirements on Solaris will fit in the Containers model (or your Linux needs would be satisfied by OpenVZ) you stand to gain a lot. We only have 40 instances, and the choice saved us 10 megawatts over the next best virtualization solution. Imagine if you had 1000.

These concepts are not likely to be foreign to any reader. Most people have considered virtualization approaches along with hardware replacement to reduce energy costs. But please, don't stop there.

Green Computing through Performance Optimization

When I look to virtualization technologies for consolidation, there is one requirement — a single machine has enough horsepower to power more than a single virtual instance. At OmniTI we deal with some large Internet architectures that serve millions upon millions of people. The bottom line is, I can completely saturate any piece of hardware you give me. There is no opportunity for consolidation in many of these architectures. The awful thing is that I see people choose hardware that is more energy efficient and simply leave it at that. The logical conclusion everyone has arrived at is: "if I can get the same CPU cycles and I/O operations for less watts, I win!" Yes, you win. No, this is not the conclusion of anything. It is the beginning. I hope your ultimate goal is not to spend CPU cycles, it is to service users. The obvious progression from here is: "if I can serve the same number of users with less CPU cycles and I/O operations, I win!" Now we're getting some where. That statement starts with the end in mind. This is the land of performance optimization.

I usually try to explain concept through metaphors and analogies, but this multi-resolutioned efficiency concept was a hard one to translate. So hard, that I'm at a loss. Those who know me well, will say: "Theo without a clever analogy at hand?! That's like Denis Leary without a vulgar rant." Alas, I'll just give some examples.

• We increased both the functionality and the performance in core XSLT technologies for Friendster and we enabled them increase system performance by a factor of over 2.5. That translates to 60% less hardware or 2.5 times as many users. Armed with that, they chose to enter China.
• We developed a purpose-built content publishing system for National Geographic Magazine and were able to deploy an infrastructure of less than half the size (less than half the power) of the leading competitive offering. This architecture was able to sustain several prolonged front-page exposures on msn.com — delivering, at peak, as many as 3000 new visitors per second.
• We developed the Message Systems MTA that helps the largest of the large ISPs handle incoming mail volume with as much as 80% reduction in infrastructure when replacing competing commercial incumbents and as much as 95% reduction when replacing open source incumbents.

The goal is to get where you are going while spending less. Less of what? Less money, less power, less heat, less CPU cycles, less, less, less. Less of everything. Not only is it better for our planet, it's simply cheaper. Don't excessively or wastefully use resources. Be responsible: conserve.

]]> http://omniti.com/seeds/oracle-flashback-versions-query http://omniti.com/seeds/oracle-flashback-versions-query Tue, 27 Jan 2009 00:00:00 GMT In database system, some times flashback for the older versions of the data or query is important for some applications. In Oracle9i, there is a feature called "time machine" manifested a form of Flashback query. This feature allows DBA to see the value of the column as of a specific time, as long as the before-image copy is available in UNDO segment. However, this feature provides a fixed snapshot of the data as of a time, not a running representation of the changed data between two time points.




During thinking process, I came cross an application , which involves the management of foreign currency , may need to see the value of data changes in a period, not just at a specific time. In Oracle 10g and onwards, you can perform this task easily and efficiently.




For Example:




This table shows the exchange rate of US$ against various other currencies. In financial industry, exchange rates are not updated when changes rather they are recorded in history. Suppose, if we do transaction at 1:00PM but it is effective at 11:00AM, the applicable rate is that at 11:00 AM, not now.




SQL> DESC RATES;

Name                Null? Type

---------------------- -------- ------------------ 

CURRENCY             VARCHAR2(4)

RATE                      NUMBER(15,10)




Up until now, the only option was to create rate history table and store rate changes and query table to see the history. Alternative option is to create additional columns in RATES table  to track the changes using trigger.




In Oracle10g, the Flashback Versions Query feature eliminate the need to maintain history table or store start and end times.




These are the operations we performed on the RATES table:




SQL> insert into rates values('CAD',1.23183); 

1 row created. 

SQL> commit; 

Commit complete. 

SQL> update rates set rate=1.23184; 

1 row updated.

SQL> commit; 

Commit complete. 

SQL> update rates set rate=1.23000; 

1 row updated. 

SQL> commit;

Commit complete. 

SQL> delete rates; 

1 row deleted. 

SQL> commit; 

Commit complete. 

SQL> insert into rates values('CAD',1.23111); 

1 row created. 

SQL> commit; 

Commit complete. 

SQL> update rates set rate=1.23112; 

1 row updated. 

SQL> commit;

Commit complete.




The following query shows the changes made to the table:




SQL> select versions_starttime as v_starttime, versions_endtime as v_endtime,

versions_xid as v_xid, versions_operation as v, rate

from rates versions between timestamp minvalue and maxvalue

order by VERSIONS_STARTTIME ;




V_STARTTIME              V_ENDTIME                       V_XID                  V   RATE

-----------------------------        -----------------------------            -----------------------     ---  ---------------

26-JAN-09 11.58.37 AM   26-JAN-09 11.58.39 AM   0700190030030000    I  1.23183 

26-JAN-09 11.58.37 AM   26-JAN-09 11.58.52 AM   02000F0039030000   U  1.23184

26-JAN-09 11.58.52 AM   26-JAN-09 11.59.01 AM   0700110031030000   U  1.23

26-JAN-09 11.59.01 AM                                         09000B00F2030000  D  1.23

26-JAN-09 11.59.16 AM   26-JAN-09 11.59.33 AM   04000E0041030000   I  1.23111 

26-JAN-09 11.59.33 AM                                         100190041030000     U  1.23112




Note that for the table all the changes to the row are shown here when the row was delete and reinserted. The VERIONS_OPERATION column shows which DML operator was performed on the row. This was done without any need of the history tables or additional columns.




More details about the actual query can be found by querying FLASHBACK_TRANSACTION_QUERY table using XID or transaction id.




SQL > SELECT UNDO_SQL FROM FLASHBACK_TRANSACTION_QUERY

WHERE XID = '04000E0041030000';

UNDO_SQL

-----------------------------------------------------------------

insert into "DENISH"."RATES"("CURRENCY","RATE") values ('CAD','1.23111');







It is extremely important to remember that the maximum available versions of the query dependent on the UNDO_RETENTION parameter.




]]>