Seeds2013

In this article, we share some of our experiences in cookbook authorship with an eye toward informing and inspiring you to create your own cookbook design guidelines. We especially focus on solid practices for attributes and roles. This is part two in an ongoing series on our experiences with Chef. Read Part One.

Intended Audience

This is not a Chef tutorial, nor a journal of one person’s experience with Chef over a month. You will not learn what a resource is or enough Ruby for Chef. Instead, we’re assuming you have used Chef enough to know what it is, what it does, and at least one way to do what you want to do. You may have authored several cookbooks for your organization’s internal use, and taken slightly different approaches as you learned what works and what doesn’t over time.

Understanding Cookbook Maturity

As cookbooks develop over time, they go through several stages. It can be helpful to understand where a cookbook resides on the development path, to know when investing more time would increase its usefulness. Not every improvement will provide a return.

Early

• May be written for a specific project: no initial intent to re-use across the company or release as open-source
• Values hardcoded, no attributes used; or attributes peculiar to the project
• Depending upon the skill level of the author, may not be aware of existing Resources that can do the job (e.g., might use a “bash” Resource to create a directory)
• No readme, or simply boilerplate

Mid-Life

• Generalized to be a stand alone, non-project-specific cookbook, with bundled templates
• Provides a basic readme and metadata file; may have to read source for details
• Well-defined use of attributes in its own namespaced tree; common cases covered, though rare cases may not be parameterized yet
• Provides defaults for its attributes in a cookbook attribute file
• Provides an attribute-walking default recipe
• Some test scripts, though incomplete; not likely to be portable
• Released company-wide

Mature

• Readme contains sections on recipes, attributes, use cases, resources
• Standardized interaction with data bags, via databag-walking recipe
• Related functionality provided in auxiliary recipes
• Provides ability to operate either via attributes or via databags, with configurable databag names and well documented databag item formats.
• Push software action into LWRPs so recipe authors can use resources without tampering with attributes
• Push functionality into libraries if needed for DRY
• Extensive testing, including for portability across platforms
• Open-sourced

Of course, different organizations will see variations on this path. For example, your cookbook may begin its life on github from day one; you may use TDD and write tests before you even write the first recipe.

Define Attribute Trees Carefully

As adoption of Chef spread within our organization, we found that most users fell into the “black box” category - those who did not have time to learn any Ruby, and wanted to limit their need to learn Chef terminology and concepts. By exposing as much functionality as possible via attributes, we were able to empower this group of users.

That means that the attribute trees effectively become the user interface for many people. The Ruby learning requirement has thus been reduced to editing the literals in the attribute definitions in the role files. Syntactically, this is a low barrier; but there are some massive conceptual barriers to using attributes that still remain. Since the attribute tree is wide open - you can put anything anywhere, for any reason - we tried a variety of approaches, and developed some conventions that helped reduce needless confusion and complexity.

Attribute Trees Should Be Namespaced with the Cookbook Name

For example, if we have a cookbook that installs packages named “packages,” we would use the attribute tree to store lists of packages to install, under the “packages” branch. In a role file, this might look like:

Each cookbook is thus free to define whatever structures are needed for that individual cookbook.

Like any naming convention, it is most powerful when applied consistently. We encountered numerous bugs and moments of confusion because our sudo recipe used a nonstandard namespace ( node[:authorization][:sudo] ). Users created roles that populated values under the node[:sudo] tree, which is what you would expect; but because the cookbook was looking in node[:authorization][:sudo], the attributes did not have any impact. Because the majority of users only make occasional changes to the attributes, they cannot be expected to remember arbitrary differences in namespaces.

Each Project Should Have a Bespoke Cookbook with its own Namespace

Every project seems to be a little different: this one has memcached, mysql and PHP; that one has Java app container, with a complex vanity hostname setup. Each project will thus need to break out of the attributes defined by the stock, shared cookbooks, to define special values that are used in the project’s custom recipes.

Without guidelines, we found that each project tended to define multiple top-level trees, often with very generic names (:cdn_info, :load_balancing, :users, etc). This quickly became a problem, causing collisions with new shared cookbooks (:users, especially). When we moved staff between projects, it was difficult to tell what was a company-standard attribute tree, and what was unique to the new project. This made it difficult to leverage experience gained on past projects.

We found that giving each project its own top-level namespace solved many of these problems.

Prefer Hashes of Attributes to Arrays of Attributes

When one has a list of similar objects, an array is the natural choice for the data structure. Examples might include operating system packages, or database users. However, due to the precedence merging algorithm Chef uses, Arrays are very difficult to work with. You cannot replace an individual element from an array; you can only append to the array (same level of precedence) or overwrite the array entirely (higher level of precedence).

Neither option is attractive. If you’re overriding you almost certainly want to replace an element; and if you recreate the entire array, you now have duplicated configuration data, which is a common cause of errors (as one list gets updated, but not the other).

We have found that in most cases, using hashes is more readable (the keys are self-documenting) and much more friendly to using overrides, because each element is addressable by name within the attribute tree. You can thus override the setting(s) in a higher-precedence call, or even in a later, same-precedence call.

There are some tradeoffs here:

• You still can’t delete individual elements, but you can override their child values. Here, an :enable flag is being set to disable the entry (presumably, the ponies cookbook would look for and respect that value).
• Hashes are unordered. If order of the elements is important, you may need to add an :order key-value pair to the child attributes.
• It’s not as concise as the array form - each entry must have a name and some kind of value, which in its degenerate form may be an empty hash. That’s more verbose, but it allows more room for future functionality - adding a :color attribute to the ponies, for example. Additionally, the keys make the code more self-documenting - which is ideal for people who are only occasional users of Chef.

Because we can now “knock out” individual elements while still remaining at the ‘default’ precedence level, we have found that we very rarely need to use the ‘normal’ or ‘override’ levels. This is a big win for our black-box users; the precedence merging rules are complex and hard to remember. By keeping things at one level of precedence, we can simplify the learning curve for our largest group of users.

Provide Defaults for Multi-Element Attribute Trees

In the above example, you may have observed that if we are disabling Rarity by setting :enable to false, there must be an implicit assumption that :enable is true by default. Implicit assumptions are a big source of errors; how can we make the defaults explicit?

We can’t use the normal attribute precedence mechanisms, because they operate when the attribute path is known in advance. If you have an attribute tree that is expected to have a dynamically changing population of child elements, we can’t construct a default that will match it using the normal methods.

To solve this problem, we wrote a short library[a] that performs dynamic merging in the attribute tree at compile-time (just prior to convergence). It is exposed as a method call on Chef::Recipe, merge_attribute_tree().

More typically, we would place the call to merge_attribute_tree in a special recipe, fixup_attributes, that performs the merge(s) and may also do things alike attribute validation. This fixup_attributes recipe would then get pulled in via include_recipe in each recipe in a cookbook.

This approach has proven to be a powerful tool in the effort to de-duplicate configuration data, while still being intuitive to the occasional Chef user.

Keeping the Roles Sane

Role files are wonderful. They are simple, reusable, and easily understood in isolation. Unfortunately, role files end up being our only tool in many situations, and we end up pounding a lot of nails with them.

One of our client’s project has about 45 different roles. A wide variety of technologies are in play, but there are a number of commonalities; so each node typically uses 4-7 role files, some recursively. This can quickly become unwieldy, especially for the on-call ops staff trying to track down an incorrect attribute in the wee hours of the morning.

Consider using Ruby Nodefiles

Because we use chef-solo, we are dependent on our JSON nodefiles. This is a point of frustration for several reasons:

• The node files are stored apart from the role files; you have to look in two different places to trace an attribute or runlist
• node files must be in JSON, but role files can be in Ruby; if they are different, you can’t cut-paste to move attributes between roles and nodes.
• JSON is somewhat user-hostile. While the same can be said for Ruby literals, at least you can have comments, your choice of quotes, and in some cases, trailing commas and “reasonable” error messages. Preferences vary as to which is more awful.

One solution we have found is to have a nearly-empty JSON nodefile: {“runlist”:[“role[node-foomachine]”]}

The corresponding role file acts exactly as the nodefile would have, but now enjoys the (perceived) benefits of Ruby.

Name Your Role Files Along Several Dimensions

We found that people are using role files for several common, cross-cutting needs. We then leveraged that fact to devise a naming scheme that reduces surprises about which roles a machine would have.

We discovered the following dimensions - you may find you need more or fewer:

• dc: Data Center, the physical location and hosting environment. Often involves things like settings for LDAP servers, routes, etc.
• env: Deployment environment - Different DSNs, fixture files for database testing, etc. Should only include attributes, empty runlist.
• task: Application task - Usually has the “real” runlists. The classic chef role notion.
• vm: Hypervisor peculiarities. We use this especially for Vagrant, controlling memory allocations.
• os: which platform we’re on. Chef has other mechanisms for this which can be used in concert.

We then chose to name each role as follows:

Looking in the roles directory for the OmniTI Website project, you might thus see:

This simplifies answering a lot of questions:

• What roles would I expect a production database server in Chicago to have? (ows-dc-chicago, ows-env-prod, ows-task-db)
• I need all the webservers to also do a geoip database update. Where should I put that? ows-task-www
• Does this project care what hypervisor it runs under? Apparently not - no ows-vm-* files.
• Where should I start tracing for a problem on corpweb1? Start with node-corpweb1, and if you need to make a snowflake change, it’s safe to make it there.
• Our dev environment is EXACTLY like prod, right? (That’s never true - but by comparing the two files, you have explicit knowledge)

This has another important effect: many settings are driven out of “hiding” in the cookbook attributes file, and into roles. We like to make the cookbook attribute files intentionally broken (i.e., providing a “CHANGEME” dummy value) in some cases - so that you MUST include an env role explicitly.

The above naming scheme works well enough for us, but we are constantly refining it and finding new things that don’t fit. Having some kind of organization is better than none, however.

A Plea: Better Attribute Tracing Tools

Most frustrations I hear from both developers and operations engineers concern the difficulty of tracing an attribute. It’s easy to add debugging statements: Chef::Log.info(“My foo is: “ + node[:foo].inspect())

But that will only tell you the final value of node[:foo] - which is usually not actionable information by itself. Instead, what we want to know is:

• what file first set that value?
• at what precedence was it set?
• which files are overwriting - or overriding - the value?
• if I change the value in this role, which nodes are affected?
• what version control revision introduced this change?
• what other values exist in the attribute tree nearby (can help you detect misplaced subtrees)?

Imagine a tree browser tool - something that lets you start at a node, and descend into the attribute tree, optionally viewing the “history” of each key in the tree as it was merged. Such a tool would have tremendous value, both as a diagnostic tool and as an educational tool.

Conclusion

While recipes get a lot of the glory, attributes are truly the workhorses of Chef. Most interaction with chef - in terms of making changes to a deployment - can be handled through attributes if the cookbook is well-designed. Since most users are using the attributes - and not seeing the lower-level pieces like recipes, LWRPs, and the like - it’s important to present a attribute space that can be lived with over time.

In our next installment, we’ll explore applying software engineering techniques to Chef deployments, and vice versa.