My Journey to integrate Chef on AIX is still going on and I’m working more than ever on these topics. I know that using such tools is not something widely adopted by AIX customers. But what I also know is that whatever happens you will in a near -or distant- future use an automation tool. These tools are so widely used in the Linux world that you just can’t ignore it. The way you were managing your AIX ten years ago is not the same as what you are doing today, and what you do today will not be what you’ll do in the future. The AIX world needs a facelift to survive, a huge step has already be done (and is still ongoing) with PowerVC thanks to a fantastic team composed by very smart people at IBM (@amarteyp; @drewthorst, @jwcroppe, and all the other persons in this team!) The AIX world is now compatible with Openstack and with this other things are coming … such as automation. When all of these things will be ready AIX we will be able to offer something comparable to Linux. Openstack and automation are the first brick to what we call today “devops” (to be more specific it’s the ops part of the devops word).
I will today focus on how to manage your AIX machines using Chef. By using the word “how” I mean what are the best practices and infrastructures to build to start using Chef on AIX. If you remember my session about Chef on AIX at the IBM Technical University in Cannes I was saying that by using Chef your infrastructure will be testable, repeatable, and versionnable. We will focus on this blog post on how to do that. To test your AIX Chef cookbooks you will need to understand what is the test kitchen (we will use the test kitchen to drive PowerVC to build virtual machines on the fly and run the chef recipes on it). To repeat this over and over to be sure everything is working (code review, be sure that your cookbook is converging) ok without having to do anything we will use Jenkins to automate these tests. Then to version your cookbooks development we will use gitlab.
To better understand why I’m doing such a thing there is nothing better than a concrete example. My goal is to do all my AIX post-installation tasks using Chef (motd configuration, dns, devices attributes, fileset installation, enabling services … everything that you are today doing using korn shells scripts). Who has never experienced someone changing one of these scripts (most of the time without warning the other members of the team) resulting in a syntax error then resulting in an outage for all your new builds. Doing this is possible if you are in a little team creating one machine per month but is inconceivable in an environment driven by PowerVC where sysadmin are not doing anything “by hand”. In such an environment if someone is doing this kind of error all the new builds are failing …. even worse you’ll probably not be aware of this until someone who is connecting on the machine will say that there is an error (most of the time the final customer). By using continuous integration your AIX build will be tested at every change, all this changes will be stored in a git repository and even better you will not be able to put a change in production without passing all these tests. Even if using this is just mandatory to do that for people using PowerVC today people who are not can still do the same thing. By doing that you’ll have a clean and proper AIX build (post-install) and no errors will be possible anymore, so I highly encourage you to do this even if you are not adopting the Openstack way or even if today you don’t see the benefits. In the future this effort will pay. Trust me.
The test-kitchen
What is the kitchen
The test-kitchen is a tool that allows you to run your AIX Chef cookbooks and recipes in a quick way without having to do manual task. During the development of your recipes if you don’t use the test kitchen you’ll have many tasks to do manually. Build a virtual machine, install the chef client, copy the cookbook and the recipes, run it, check everything is in the state that you want. Imagine doing that on different AIX version (6.1, 7.1, 7.2) everytime you are changing something in your post-installation recipes (I was doing that before and I can assure you that creating and destroy machine over and over and over is just a waste of time). The test kitchen is here to do the job for you. It will build the machine for you (using the PowerVC kitchen driver), install the chef-client (using an omnibus server), copy the content of your cookbook (the files), run a bunch of recipe (described in what we call suites) and then test it (using bats, or serverspec). You can configure your kitchen to test different kind of images (6.1, 7.1, 7.2) and differents suites (cookbooks, recipes) depending on the environment you want to test. By default the test kitchen is using a Linux tool called Vagrant to build your VM. Obsiouvly Vagrant is not able to build an AIX machine, that’s why we will use a modified version of the kitchen-openstack driver (modified by my self) called kitchen-powervc to build the virtual machines:
Installing the kitchen and the PowerVC driver
If you have an access to an enterprise proxy you can directly download and install the gem files from your host (in my case this is a Linux on Power … so Linux on Power is working great for this).
- Install the test kitchen :
# gem install --http-proxy http://bcreau:mypasswd@proxy:8080 test-kitchen Successfully installed test-kitchen-1.7.2 Parsing documentation for test-kitchen-1.7.2 1 gem installed
# gem install --http-proxy http://bcreau:mypasswd@proxy:8080 kitchen-powervc Successfully installed kitchen-powervc-0.1.0 Parsing documentation for kitchen-powervc-0.1.0 1 gem installed
# gem install --http-proxy http://bcreau:mypasswd@proxy:8080 kitchen-openstack Successfully installed kitchen-openstack-3.0.0 Fetching: fog-core-1.38.0.gem (100%) Successfully installed fog-core-1.38.0 Fetching: fuzzyurl-0.8.0.gem (100%) Successfully installed fuzzyurl-0.8.0 Parsing documentation for kitchen-openstack-3.0.0 Installing ri documentation for kitchen-openstack-3.0.0 Parsing documentation for fog-core-1.38.0 Installing ri documentation for fog-core-1.38.0 Parsing documentation for fuzzyurl-0.8.0 Installing ri documentation for fuzzyurl-0.8.0 3 gems installed
If you don’t have the access to an enterprise proxy you can still download the gems from home and install it on your work machine:
# gem install test-kitchen kitchen-powervc kitchen-openstack -i repo --no-ri --no-rdoc # # copy the files (repo directory) on your destination machine # gem install *.gem
Setup the kitchen (.kitchen.yml file)
The kitchen configuration file is the .kitchen.yml, when you’ll run the kitchen command, the kitchen will look at this file. You have to put it in the chef-repo (where the cookbook directory is, the kitchen will copy the file from the cookbook to the test machine that’s why it’s important to put this file at the root of the chef-repo.) This file is separated in different sections:
- The driver section. In this section you will configure howto created virtual machines. In our case how to connect to PowerVC (credentials, region). You’ll also tell in this section which image you want to use (PowerVC images), which flavor (PowerVC template) and which network will be used at the VM creation (please note that you can put some driver_config in the platform section, to tell which image or which ip you want to use for each specific platform.:
- name: the name of the driver (here powervc).
- openstack*: the PowerVC url, user, password, region, domain.
- image_ref: the name of the image (we will put this in driver_config in the platform section).
- flavor_ref: the name of the PowerVC template used at the VM creation.
- fixed_ip: the ip_address used for the virtual machine creation.
- server_name_prefix: each vm created by the kitchen will be prefixed by this parameter.
- network_ref: the name of the PowerVC vlan to be used at the machine creation.
- public_key_path: The kitchen needs to connect to the machine with ssh, you need to provide the public key used.
- private_key_path: Same but for the private key.
- username: The ssh username (we will use root, but you can use another user and then tell the kitchen to use sudo)
- user_data: The activation input used by cloud-init we will in this one put the public key to be sure you can access the machine without password (it’s the PowerVC activation input).
driver: name: powervc server_wait: 100 openstack_username: "root" openstack_api_key: "root" openstack_auth_url: "https://mypowervc:5000/v3/auth/tokens" openstack_region: "RegionOne" openstack_project_domain: "Default" openstack_user_domain: "Default" openstack_project_name: "ibm-default" flavor_ref: "mytemplate" server_name_prefix: "chefkitchen" network_ref: "vlan666" public_key_path: "/home/chef/.ssh/id_dsa.pub" private_key_path: "/home/chef/.ssh/id_dsa" username: "root" user_data: userdata.txt
#cloud-config ssh_authorized_keys: - ssh-dss AAAAB3NzaC1kc3MAAACBAIVZx6Pic+FyUisoNrm6Znxd48DQ/YGNRgsed+fc+yL1BVESyTU5kqnupS8GXG2I0VPMWN7ZiPnbT1Fe2D[..]
provisioner: name: chef_solo chef_omnibus_url: "http://myomnibusserver:8080/chefclient/install.sh" sudo: false
platforms:
- name: aix72
driver_config:
image_ref: "kitchen-aix72"
fixed_ip: "10.66.33.234"
- name: aix71
driver_config:
image_ref: "kitchen-aix71"
fixed_ip: "10.66.33.235"
- name: aix61
driver_config:
image_ref: "kitchen-aix61"
fixed_ip: "10.66.33.236"
suites:
- name: aixcookbook
run_list:
- recipe[aix::root_authorized_keys]
- recipe[aix::gem_source]
attributes: { gem_source: { add_urls: [ "http://10.14.66.100:8808" ], delete_urls: [ "https://rubygems.org/" ] } }
):busser: sudo: false
After configuring the kitchen you can check the yml file is ok by listing what’s configured on the kitchen:
# kitchen list Instance Driver Provisioner Verifier Transport Last Action aixcookbook-aix72 Powervc ChefSolo Busser Sshaixcookbook-aix71 Powervc ChefSolo Busser Ssh aixcookbook-aix61 Powervc ChefSolo Busser Ssh
Anatomy of a kitchen run
A kitchen run is divided into five steps. At first we are creating a virtual machine (the create action), then we are installing the chef-client (using an omnibus url) and running some recipes (converge), then we are installing testing tools on the virtual machine (in my case serverspec) (setup) and we are running the tests (verify). Finally if everything was ok we are deleting the virtual machines (destroy). Instead of running all theses steps one by one you can use the “test” option. This one will do destroy,create,converge,setup,verify,destroy in on single “pass”. Let’s check in details each steps:
- Create: This will create the virtual machine using PowerVC. If you choose to use the “fixed_ip” option in the .kitchen.yml file this ip will be choose at the machine creation time. If you prefer to pick an ip from the network (in the pool) don’t set the “fixed_ip”. You’ll see the details in the picture below. You can at the end test the connectivity (transport) (ssh) to the machine using “kitchen login”. The ssh public key was automatically added using the userdata.txt file used by cloud-init at the machine creation time. After the machine is created you can use the “kitchen list” command to check the machine was successfully created:
# kitchen create
- Converge: This will converge the kitchen (on more time converge = chef-client installation and running chef-solo with the suite configuration describing which recipe will be launched). The converge action will download the chef client and install it on the machine (using the omnibus url) and run the recipe specified in the suite stanza of the .kitchen.yml file. Here is the script I use for the omnibus installation this script is “served” by an http server:
# cat install.sh
#!/usr/bin/ksh
echo "[omnibus] [start] starting omnibus install"
echo "[omnibus] downloading chef client http://chefomnibus:8080/chefclient/lastest"
perl -le 'use LWP::Simple;getstore("http://chefomnibus:8080/chefclient/latest", "/tmp/chef.bff")'
echo "[omnibus] installing chef client"
installp -aXYgd /tmp/ chef
echo "[omnibus] [end] ending omnibus install"
# ls -l /apps/chef/chefclient
total 647896
-rw-r--r-- 1 apache apache 87033856 Dec 16 17:15 chef-12.1.2-1.powerpc.bff
-rwxr-xr-x 1 apache apache 91922944 Nov 25 00:24 chef-12.5.1-1.powerpc.bff
-rw------- 2 apache apache 76375040 Jan 6 11:23 chef-12.6.0-1.powerpc.bff
-rwxr-xr-x 1 apache apache 364 Apr 15 10:23 install.sh
-rw------- 2 apache apache 76375040 Jan 6 11:23 latest
# cat httpd.conf
[..]
Alias /chefclient/ "/apps/chef/chefclient/"
Options Indexes FollowSymlinks MultiViews
AllowOverride None
Require all granted
# kitchen converge
- Setup and verify: these actions will run a bunch of tests to verify the machine is in the state you want. The test I am writing are checking that the root home directory was created and the key was successfully created in the .ssh directory. In a few words you need to write tests checking that your recipes are working well (in chef words: “check that the machine is in the correct state”). In my case I’m using serverspec to describe my tests (there are different tools using for testing, you can also use bats). To describe the tests suite just create serverspec files (describing the tests) in the chef-repo directory (in ~/test/integration/
/serverspec in my case ~/test/integration/aixcookbook/serverspec). All the serverspec test files are suffixed by _spec:
# ls test/integration/aixcookbook/serverspec/ root_authorized_keys_spec.rb spec_helper.rb
# ls test/integration/aixcookbook/serverspec/ root_authorized_keys_spec.rb spec_helper.rb
# cat spec_helper.rb
require 'serverspec'
set :backend, :exec
# cat root_authorized_keys_spec.rb
require 'spec_helper'
describe file('/root/.ssh') do
it { should exist }
it { should be_directory }
it { should be_owned_by 'root' }
end
describe file('/root/.ssh/authorized_keys') do
it { should exist }
it { should be_owned_by 'root' }
it { should contain 'from="1[..]" ssh-rsa AAAAB3NzaC1[..]' }
end
# gem list | grep -E "busser|serverspec" busser (0.7.1) busser-bats (0.3.0) busser-serverspec (0.5.9) serverspec (2.31.1) # gem server Server started at http://0.0.0.0:8808
# cat gem_source.rb
ruby_block 'Changing gem source' do
block do
node['gem_source']['add_urls'].each do |url|
current_sources = Mixlib::ShellOut.new('/opt/chef/embedded/bin/gem source')
current_sources.run_command
next if current_sources.stdout.include?(url)
add = Mixlib::ShellOut.new("/opt/chef/embedded/bin/gem source --add #{url}")
add.run_command
Chef::Application.fatal!("Adding gem source #{url} failed #{add.status}") unless add.status == 0
Chef::Log.info("Add gem source #{url}")
end
node['gem_source']['delete_urls'].each do |url|
current_sources = Mixlib::ShellOut.new('/opt/chef/embedded/bin/gem source')
current_sources.run_command
next unless current_sources.stdout.include?(url)
del = Mixlib::ShellOut.new("/opt/chef/embedded/bin/gem source --remove #{url}")
del.run_command
Chef::Application.fatal!("Removing gem source #{url} failed #{del.status}") unless del.status == 0
Chef::Log.info("Remove gem source #{url}")
end
end
action :run
end
# kitchen setup # kitchen verify
- Destroy: This will destroy the virtual machine on PowerVC.
# kitchen destroy
Now that you understand how the kitchen is working and that you are now able to run it to create and test AIX machines you are ready to use the kitchen to develop and create the chef cookbook that will fit your infrastructure. To run the all the steps “create,converge,setup,verify,destroy”, just use the “kitchen test” command:
# kitchen test
As you are going to change a lot of things in your cookbook you’ll need to version the code you are creating, for this we will use a gitlab server.
Gitlab: version your AIX cookbook
Unfortunately for you and for me I didn’t had the time to run gitlab on a Linux on Power machine. I’m sure it is possible (if you find a way to do this please mail me). Anyway my version of gitlab is running on an x86 box. The goal here is to allow the chef workstation (in my environment this user is “chef”) user to push all the new developments (providers, recipes) to the git development branch for this we will:
- Allow the chef user to push its source to the git server trough ssh (we are creating a chefworkstation user and adding the key to authorize this user to push the changes to the git repository with ssh).
- Create a new repository called aix-cookbook.
- Push your current work to the master branch. The master branch will be the production branch.
# git config --global user.name "chefworkstation" # git config --global user.email "chef@myworkstation.chmod666.org" # git init # git add -A . # git commit -m "first commit" # git remote add origin git@gitlabserver:chefworkstation/aix-cookbook.git # git push origin master
# git checkout -b dev # git commit -a # git push origin dev
The git server is ready: we have a repository accessible by the chef user. Two branch created the dev one (the one we are working on used for all our development) and the master branch used for production that will be never touched by us and will be only updated (by jenkins) if all the tests (foodcritic, rubocop and the test-kitchen) are ok
Automating the continous integration with Jenkins
What is Jenkins
The goal of Jenkins is to automate all tests and run them over and over again every time a change is applied onto the cookbook you are developing. By using Jenkins you will be sure that every change will be tested and you will never push something that is not working or not passing the tests you have defined in your production environment. To be sure the cookbook is working as desired we will use three different tools. foodcritic will check the will check your chef cookbook for common problems by checking rules that are defined within the tools (this rules will check that everything is ok for the chef execution, so you will be sure that there is no syntax error, and that all the coding convention will be respected), rubocop will check the ruby syntax, and then we will run a kitchen test to be sure that the developement branch is working with the kitchen and that all our serverspec tests are ok. Jenkins will automate the following steps:
- Pull the dev branch from git server (gitlab) if anything has changed on this branch.
- Run foodcritic on the code.
- If foodcritic tests are ok this will trigger the next step.
- Pull the dev branch again
- Run rubocop on the code.
- If rubocop tests are ok this will trigger the next step.
- Run the test-kitchen
- This will build a new machine on PowerVC and test the cookbook against it (kitchen test).
- If the test kitchen is ok push the dev branch to the master branch.
- You are ready for production
![:-)]()
First: Foodcritic
The first test we are running is foodcritic. Better than trying to do my own explanation of this with my weird english I prefer to quote the chef website:
Foodcritic is a static linting tool that analyzes all of the Ruby code that is authored in a cookbook against a number of rules, and then returns a list of violations. Because Foodcritic is a static linting tool, using it is fast. The code in a cookbook is read, broken down, and then compared to Foodcritic rules. The code is not run (a chef-client run does not occur). Foodcritic does not validate the intention of a recipe, rather it evaluates the structure of the code, and helps enforce specific behavior, detect portability of recipes, identify potential run-time failures, and spot common anti-patterns.
# foodcritic -f correctness ./cookbooks/ FC014: Consider extracting long ruby_block to library: ./cookbooks/aix/recipes/gem_source.rb:1
In Jenkins here are the steps to create a foodcritic test:
- Pull dev branch from gitlab:
- Check for changes (the Jenkins test will be triggered only if there was a change in the git repository):
- Run foodcritic
- After the build parse the code (to archive and record the evolution of the foodcritic errors) and run the rubocop project if the build is stable (passed without any errors):
- To configure the parser go in the Jenkins configuration and add the foodcritic compiler warnings:
Second: Rubocop
The second test we are running is rubocop it’s a Ruby static code analyzer, based on the community Ruby style guide. Here is an example below
# rubocop . Inspecting 71 files ..CCCCWWCWC.WC..CC........C.....CC.........C.C.....C..................C Offenses: cookbooks/aix/providers/fixes.rb:31:1: C: Assignment Branch Condition size for load_current_resource is too high. [20.15/15] def load_current_resource ^^^ cookbooks/aix/providers/fixes.rb:31:1: C: Method has too many lines. [19/10] def load_current_resource ... ^^^^^^^^^^^^^^^^^^^^^^^^^ cookbooks/aix/providers/sysdump.rb:11:1: C: Assignment Branch Condition size for load_current_resource is too high. [25.16/15] def load_current_resource
In Jenkins here are the steps to create a rubocop test:
- Do the same thing as foodcritic except for the build and post-build action steps:
- Run rubocop:
- After the build parse the code and run the test-kitchen project even if the build is fails (rubocop will generate tons of things to correct … once you are ok with rubocop change this to “trigger only if the build is stable”) :
Third: test-kitchen
I don’t have to explain again what is the test-kitchen . It is the third test we are creating with Jenkins and if this one is ok we are pushing the changes in production:
- Do the same thing as foodcritic except for the build and post-build action steps:
- Run the test-kitchen:
- If the test kitchen is ok push dev branch to master branch (dev to production):
More about Jenkins
The three tests are now linked together. On the Jenkins home page you can check the current state of your tests. Here are a couple of screenshots:
Conclusion
I know that for most of you working this way is something totally new. As AIX sysadmins we are used to our ksh and bash scripts and we like the way it is today. But as the world is changing and as you are going to manage more and more machines with less and less admins you will understand how powerful it is to use automation and how powerful it is to work in a “continuous integration” way. Even if you don’t like this “concept” or this new work habit … give it a try and you’ll see that working this way is worth the effort. First for you … you’ll discover a lot of new interesting things, second for your boss that will discover that working this way is safer and more productive. Trust me AIX needs to face Linux today and we are not going anywhere without having a proper fight versus the Linux guys 
(yep it’s a joke).