The world is changing fast, especially regarding sysadmin jobs and skills. Everybody has noticed that being a good sysadmin now implies two things. The first one is “being a good dev”. Not just someone who knows how to write a ksh or bash script but someone who is able to write in one of these three languages: python, ruby, go. Most of my team members does not understand that. This is now almost mandatory. I truly believe what I’m saying here. The second one is to have strong skills in automation tools. On this part I’m almost ok being good at Chef, Saltstack and Ansible. Unfortunately for the world the best tool is never the one who wins and that’s why Ansible is almost winning everywhere in the battle of automation. It is simple to understand why: Ansible is simple to use and to understand and it is based on ssh. My opinion is that Ansible is ok for administration stuff but not ok when scaling. Being based on ssh make it being based on a “push” model and in my humble opinion push models are bad and pull models is the future. (One thing to say about that: this is just my opinion. I don’t want this to end in never ending trolls on Twitter. Please do blog posts if you want to express yourself) (I’m saying this because twitter is becoming a place to troll and not anymore a place to share skills and knowledge, like it was before). This is said.

The first part of this blog post will talk about a tool I am coding called pvcctl. This tool is a python tool allowing you to use PowerVC in command line. It was also for me the opportunity to be better at python and to improve my skills developing in this language. Keep in mind that I’m not a developer but I’m here going to give you simple tips and tricks to use python to write your own tools to query and interact with Openstack. I must admit that I’ve tried everything: httplib, librequest, Chef, Ansible, Salstack. None of this solution was ok for me. I finally ended in using Openstack python api to write this tool. It’s not that hard and it now allows me to write my own programs to interact with PowerVC. Once again keep in mind that this tool fit my needs and it will probably not fit yours. This is an example of how to write a tool based on python Openstack api. Not an official tool or anything else.

The second part of this blog post will talk about an Ansible playbook I’ve written to take care of PowerVC installation and NovaLink post installation. The more machine I deploy the more PowerVC I need (yeah yeah, I was forced to) and the more NovaLink I need too. Instead of doing the same thing over and over again the best solution was to use an automation tool and as it is now the most common one used on Linux the one I choose was ansible.

Using python Openstack api to code a PowerVC command line

This part of the post will show you how to use the python Openstack api to create scripts to query and interact with PowerVC. First of all I know that their are other ways to use the APIs but (it’s my opinion) I think that using service-specific clients is the simplest way to understand and to work with the API. This part of the blog post will only talk about service-specific clients (ie. novaclient, cinderclient, and so on …). I wanted to thanks Matthew Edmonds from the PowerVC team. He helped me to better understand the api and gave me good advises. So a big shout out to you Matthew :-). Thank you.

Initialize you script

Sessions

Almost all Openstack tools are using “rc” files to load authentication credentials and endpoints. As I wanted my tool to work like this (ie. sourcing an rc file containing my credentials) I have found that the best way to do this was to use session. By using session you don’t have to manage or work with any tokens or to be worried about that. Sessions is taking care of that for you and you have nothing to do. As you can see on the code below the “OS_*” environment variables are used here. So before running the tool all you have to do is to export these variables. It’s as simple as that:

• An example “rc” file filled with the OS_* values (note that crt file must be copied from the PowerVC host to the host running the tool (/etc/pki/tls/certs/powervc.crt)):

# cat powervrc
export OS_AUTH_URL=https://mypowervc:5000/v3/
export OS_USERNAME=root
export OS_PASSWORD=root
export OS_TENANT_NAME=ibm-default
export OS_REGION_NAME=RegionOne
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_CACERT=~/powervc_labp8.crt
export OS_IMAGE_ENDPOINT=https://mypowervc:9292/
export NOVACLIENT_DEBUG=0
# source powervcrc

from keystoneauth1.identity import v3
from keystoneauth1 import session

auth = v3.Password(auth_url=env['OS_AUTH_URL'],
                   username=env['OS_USERNAME'],
                   password=env['OS_PASSWORD'],
                   project_name=env['OS_TENANT_NAME'],
                   user_domain_id=env['OS_USER_DOMAIN_NAME'],
                   project_domain_id=env['OS_PROJECT_DOMAIN_NAME'])

sess = session.Session(auth=auth, verify=env['OS_CACERT'])

The logger

Instead of using the print statement each time I need to debug my script I have found that most of Openstack API can be used with a python logger object. By using a logger you’ll be able to see all your http calls to the Openstack API (your post, put, get, delete with their json body, their response and their url). It is super useful to debug your scripts and it’s super simple to use. The piece of code below will create a logger object writing to my log directory. You’ll see after how to use a logger when creating a client object (a nova, cinder, or neutron object):

import logging

logger = logging.getLogger('pvcctl')
hdlr = logging.FileHandler(BASE_DIR + "/logs/pvcctl.log")
logger.addHandler(hdlr)
logger.setLevel(logging.DEBUG)

Here is an exemple of the output created by the logger with a novaclient (the novaclient was created specifying a logger object):

REQ: curl -g -i --cacert "/data/tools/ditools/pvcctl/conf/powervc.crt" -X POST https://mypowervc/powervc/openstack/compute/v2.1/51488ae7be7e4ec59759ccab496c8793/servers/a3cea5b8-33b4-432e-88ec-e11e47941846/os-volume_attachments -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.19" -H "X-Auth-Token: {SHA1}9b8becc425d25fdfb98d5e4f055c71498d2e744f" -d '{"volumeAttachment": {"volumeId": "a90f04ce-feb2-4163-9b36-23765777c6a0"}}' 
RESP: [200] Date: Fri, 28 Oct 2016 13:05:24 GMT Server: Apache Content-Length: 194 Content-Type: application/json X-Openstack-Nova-Api-Version: 2.19 Vary: X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-58addd19-7421-4c9f-a712-9c386f46b6cb Cache-control: max-age=0, no-cache, no-store, must-revalidate Pragma: no-cache Keep-Alive: timeout=5, max=93 Connection: Keep-Alive 
RESP BODY: {"volumeAttachment": {"device": "/dev/sdb", "serverId": "a3cea5b8-33b4-432e-88ec-e11e47941846", "id": "a90f04ce-feb2-4163-9b36-23765777c6a0", "volumeId": "a90f04ce-feb2-4163-9b36-23765777c6a0"}} 

The clients

Each Openstack service (nova, glance, neutron, swift, cinder, …) is provided with a python Openstack API. I’m -in my tool- only using novaclient, cinderclient and neutronclient but it will be the exact same thing if you want to use ceilomeiter or glance. Before doing anything else you have to install the clients you want to use (using your package manager (yum on my side) or using pip)

# yum install python2-novaclient.noarch
# pip install python-neutronclient

Initializing the clients

After the clients are installed you can use them in your python scripts, import them and create the objects. Use the previously created session object to create the clients objects (session=sess in my example below):

from novaclient import client as client_nova
from neutronclient.v2_0 import client as client_neutron
from cinderclient import client as client_cinder

nova = client_nova.Client(2.19, session=sess)
neutron = client_neutron.Client(session=sess)
cinder = client_cinder.Client(2.0, service_type="volume", session=sess)

If your client can be created using a logger object you can specify this at the time of the object creation. Here is an example with novaclient:

nova = client_nova.Client(2.19, session=sess, http_log_debug=True, logger=logger)

Using the clients

After the objects are created using them is super simple. I’ll give you here a couple of examples below:

• Searching a vm (this will return a server object):

server = nova.servers.find(name=machine)

server = nova.servers.find(name=machine)
server.update(name=new_name)

server = nova.servers.find(name=machine)
server.start()

server = nova.servers.find(name=machine)
server.stop()

for server in nova.servers.list():
  name = getattr(server, 'OS-EXT-SRV-ATTR:hostname')
  print name

vlan = neutron.list_networks(name=vlan)

cinder.volumes.create(name=volume_name, size=size, volume_type=storage_template)

And so on. Each client type has it’s own method, the best way to find which methods are available for each object is to check in the official Openstack API documentation:

• novaclient: http://docs.openstack.org/developer/python-novaclient/api.html
• neutronclient: http://docs.openstack.org/developer/python-neutronclient/usage/library.html
• cinderclient:http://docs.openstack.org/developer/python-cinderclient/

What about PowerVC extensions ? (using get,put,delete …)

If you have already read my blog posts about PowerVC you will probably already know that PowerVC add some extensions to OpenStack. That means that for the PowerVC extension using the Openstack method shipped with the API will not work. To be more specific the methods used to query or interact with the PowerVC extensions will simply not exists at all. The good part of these API is that they are also shipped with the http common methods. This means that for each Openstack api object, let’s say nova, you’ll be able to directly use the put, post, get, delete and so on method. By doing that you’ll be able to use the same object to use all api method (let’s say create or rename a server) and to use the PowerVC extensions. For instance the “host-sea” is a PowerVC added extension (link here). You can simply use a novaclient to query or post something to the extension (the example below shows you both post and a get on the PowerVC extension “host-seas”:

resp, host_seas = nova.client.get("/host-seas?network_id=" + net_id + "&vlan_id" + vlan_id)
resp, body = nova.client.post("/host-network-mapping", body=mapping_json)

Here is another example for onboarding or unmanaging volume (which is a PowerVC extension to Openstack):

resp, body = cinder.client.post("/os-hosts/" + oshost['host_name'] + "/onboard", body=onboard_json)
resp, body = cinder.client.post("/os-hosts/" + oshost['host_name'] + "/unmanage", body=onboard_json)

Working with json

Last part for this tips and tricks on how to write your own python code using Openstack api: you’ll probably see that you’ll need to work with json. What is cool with python is that json can be managed as a dict object. It’s super simple to use:

• Importing json:

import json

json_load = json.loads('{ "ibm-extend": { "new_size": 0 } }')

json_load['ibm-extend']['new_size'] = 200

resp, body = cinder.client.post("/volumes/" + volume.id + "/action", body=json_grow)

The pvcctl tool

Now that you have understand this I can now say to you that I’ve written a tool called pvcctl based on the Openstack python api. This tool is freely available on github. As I said before this tools fit my needs and is an example of what can be done using the Openstack API in python. Keep in mind that I’m not a developer and the code can probably be better. But this tool is used by my whole team on PowerVC so … it will probably be good enough to create shells scripts on top of it or for daily PowerVC administration. The tool can be found a this address: https://github.com/chmod666org/pvcctl. Give it a try and tell me what you think a about it. I give you below a couple of example of how to use the tools. You’ll see it’s super simple:

• Create a network:

# pvcctl network create name=newvlan id=666 cidr='10.10.20.20/24' dns1='8.8.8.8' dns2='8.8.9.9' gw='10.10.20.254'

# pvcctl vm set_description vm=deckard desc="We call it Voight-Kampff"

# pvcctl vm migrate vm=tyrell host=21AFF8V

# pvcctl vm attach_vol vm=tyrell vol=myvol

# pvcctl vm create ip='10.14.20.240' ec_max=1 ec_min=0.1 ec=0.1 vp=1 vp_min=1 vp_max=4 mem=4096 mem_min=1024 mem_max=8192 weight=240 name=bcubcu disks="death" scg=ssp vlan=vlan-1331 image=kitchen-aix72 aggregate=hg2 user_data=testvm srr=yes

# pvcctl volume create provider=mystorageprovider name=volume_test size=10

# pvcctl volume grow vol=test_volume size=50

Automating PowerVC and NovaLink installation and post-installation with ansible

At the same time I have released my pvcctl tool I also had the idea that releasing my PowerVC and NovaLink playbook for Ansible will be a good thing. This playbook is not so huge and is not doing a lot of things but I was using it a lot when deploying all my NovaLink hosts (I now have 16 MME managed by NovaLink) and when creating PowerVC servers for different kind of project. That’s a shame that everybody in my company has not yet understood why having multiple PowerVC is just a wrong idea and a waste of time (I’m not surprised that between a good and a bad idea they prefer to choose the bad one , obvious when you never touch to production at all but when you still have the power of deciding things in your hands). Anyway this playbook is used for two things, first one is preparing my novalink hosts (being sure I’m at the latest version of NovaLink, being sure that everything is configured as I want to (ntp, dns, rsct)), second one is installing PowerVC hosts (installing PowerVC is just super boring you always have to install tons of rpms needed for dependencies and if like me, you do not have a satellite connection or access to the internet it can be a real pain). The only thing you have to do is to configure the inventories files and the group_vars files located in the playbook directory. The playbook can be founded at this address https://github.com/chmod666org/ansible-powervc-novalink.

• Put the name of your NovaLink hosts in the hosts.novalink file:

# cat inventories/hosts.novalink
nl1.lab.chmod666.org
nl2.lab.chmod666.org
[..]

# cat inventories/hosts.powervc
pvc1.lab.chmod666.org
pvc2.lab.chmod666.org
[..]

ntpservers:
  - myntp1
  - myntp2
  - myntp2
dnsservers:
  - 8.8.8.8
  - 8.8.9.9
dnssearch:
  - lab.chmod666.org
vepa_iface: ibmveth6
repo: novalinkrepo

ntpservers:
  - myntp1
  - myntp2
  - myntpd3
dnsservers:
  - 8.8.8.8
  - 8.8.9.9
dnssearch:
  - lab.chmod666.org
repo_rhel: http://myrepo.lab.chmod666.org/rhel72le/
repo_ibmtools: http://myrepo.lab.chmod666.org/ibmpowertools71le/
repo_powervc: http://myrepo.lab.chmod666.org/powervc
powervc_base: PowerVC_V1.3.1_for_Power_Linux_LE_RHEL_7.1_062016.tar.gz
powervc_upd: powervc-update-ppcle-1.3.1.2.tgz
powervc_rpm: [ 'python-dns-1.12.0-1.20150617git465785f.el7.noarch.rpm', 'selinux-policy-3.13.1-60.el7.noarch.rpm', 'selinux-policy-targeted-3.13.1-60.el7.noarch.rpm', 'python-fpconst-0.7.3-12.el7.noarch.rpm', 'python-pyasn1-0.1.6-2.el7.noarch.rpm', 'python-pyasn1-modules-0.1.6-2.el7.noarch.rpm', 'python-twisted-web-12.1.0-5.el7_2.ppc64le.rpm', 'sysfsutils-2.1.0-16.el7.ppc64le.rpm', 'SOAPpy-0.11.6-17.el7.noarch.rpm', 'SOAPpy-0.11.6-17.el7.noarch.rpm', 'python-twisted-core-12.2.0-4.el7.ppc64le.rpm', 'python-zope-interface-4.0.5-4.el7.ppc64le.rpm', 'pyserial-2.6-5.el7.noarch.rpm' ]
powervc_base_version: 1.3.1.0
powervc_upd_version: 1.3.1.2
powervc_edition: cloud_powervm

You then just have to run the playbook for Novalink and PowerVC hosts to run the installation and post-installation:

• Novalink post-install:

# ansible-playbook -i inventories/hosts.novalink site.yml

# ansible-playbook -i inventories/hosts.powervc site.yml

Just to give you an example of one the the tasks of this playbook here is the task in charge to install PowerVC. Pretty simple :

## install powervc
- name: check previous installation
  command: bash -c "rpm -qa | grep ibmpowervc-"
  register: check_base
  ignore_errors: True
- debug: var=check_base

- name: install powervc binaires
  command: chdir=/tmp/powervc-{{ powervc_base_version }} /tmp/powervc-{{ powervc_base_version }}/install -s cloud_powervm
  environment:
    HOST_INTERFACE: "{{ ansible_default_ipv4.interface }}"
    EGO_ENABLE_SUPPORT_IPV6: N
    PATH: $PATH:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-3.b17.el7.ppc64le/jre/bin/:/usr/sbin:/usr/bin
    ERL_EPMD_ADDRESS: "::ffff:127.0.1.1"
  when: check_base.rc == 1

- name: check previous update
  command: rpm -q ibmpowervc-{{ powervc_upd_version }}-1.noarch
  register: check_upd
  ignore_errors: True
- debug: var=check_upd
  
- name: updating powervc
  command: chdir=/tmp/powervc-{{ powervc_upd_version }} /tmp/powervc-{{ powervc_upd_version }}/update -s 
  when: check_upd.rc == 1

The goal here is not to explain how Ansible is working but to show you a simple example of what I’m doing with Ansible on my Linux boxes (all of this related to Power). If you want to check further have a look in the playbook itself on github

Conclusion

This blog post is just a way to show you my work on both pvcctl and Ansible playbook for NovaLink and PowerVC. It’s not a detailed blog post about deep technical stuffs. I hope you’ll give a try to the tools and tell me what can be improved or changed. As always … I hope it helps.