First of all to avoid any ambiguity I first have to say that none of this work is the result of something that have been asked to me on my work place. This piece of work was done during my after hours work (this means mostly at home, and mostly on my home server (couple of things were done on my work lab for some obvious reasons)).
After working on Docker and Swarm I realized something: just having a Swarm cluster (with mesh routing) is not enough to provide to customers a simple and viable way to use Docker. That’s why I wanted to create a “Swarm as a Service” infrastructure providing enterprise class products to my customers. Then after finding all those products and after knowing how they work I wanted to automate the creation of the Swarm Cluster and all the services associated to it.
I warn you that this blog post is very long. Take your time to read it part by part and even if you do not understand what I’m trying to say at the beginning you will finally see my point of view at the end of the post. It may looks a little bit unclear but I want you to glean the information you want and to trash the ones that you don’t care about.. Not everything will fit your needs or match your environment that’s certain. Last advice: while reading the post, check the Ansible playbook on my github page, you will see in this one all the steps described here.
On my previous blog post the storage part was achieved by using NFS. After looking on the web and searching over and over about the best solution to share data among a Docker Swarm Cluster I finally come to the conclusion that the RedHat GlusterFS was the best solution to have a reliable way to have a replicated storage across all the nodes of my Swarm Cluster. GlusterFS just combine two things I love: Simplicity and Reliablity. Unfortunately for us you’ll see below when I’ll talk about details that I didn’t found an up to date version of GlusterFS running on the Power Systems (ie. any ppc64le system). The conclusion today if you want to do that is to use an old version of GlusterFS. No worries this version does the job perfectly.
![antmascot]()
The second thing I realized is that this infrastructure has to be dynamic. By this I mean being able to reconfigure some services (a proxy) on the fly when you decide to stop or create a service (a bunch of containers) in the Swarm cluster. To achieve this you need an intelligent proxy capable of listening the Swarm Discovery Service and modify your proxy rules at the deletion/creation time. I was already a Traefik user on x86 but I decided to check every possible solution and realized that the only second viable to achieve that was to use Dockerflow (based on haproxy). Unfortunately (or fortunately) for me I never succeeded in compiling Dockerflow on ppc64le, the second solution was to use Traefik. The job was easier on this one and after struggling a few days I had a working version of Traefik running on ppc64le. The image was ready. Next I had to found a way to be autonomous to avoid working with my network team. A solution is to ask them to create a DNS “wildcard” on a domain (for instance *.docker.chmod666.org). As Traefik is a wonderful tool it allows you to configure the reverse proxy with what I call context path (for instance http://www.chmod666.org/mynewservice). By doing this I’m able to be fully autonomous even if the network team does not allow me a DNS “wildcard”.
![traefik.logo]()
Next I wanted to be able to recreate the infrastructure again and again and again. Not just being able to recreate it from scratch but also to make it “scalable” by adding or removing Swarm nodes on the fly (to be honest here I have just coded the “scale up” part and not the “scale down” one ;-)). This part is achieved by two products: obviously Ansible to setup the software part inside the vm and Hashicrop Terraform to be able to drive my Openstack cloud (obviously PowerVC on PowerSystems).
Being able to run any configuration tools is a good thing, being able to track down every changes and store them somewhere is even better. To be able to have such a functionality I’m using a tool called Openstack ARA (Ansible Runbook Analysis) which is a good way to keep a trace of every playbook run in you infrastructure. Having ARA in an Ansible infrastructure is now from me something mandatory. Even if you are an Ansible Tower user, having ARA is a good way to track every playbook run. The web interface provided by ARA is simple and efficient. I like the way it is. Just a single tool doing a single job at it best.
![ansible]()
![terraform]()
![ara]()
Finally to be able to let everybody (even noobs, non super tech guys, or just people who does not have the time to learn everything) I wanted to have a web ui capable of driving my Docker infrastructure. A fantastic tool called Portainer was created by a couple of folks. I -one more time- had to recompile the entire product for ppc64le and recreate my own Docker images.
![portainer]()
Before telling and explaining you all the details if you do not want to do all these steps by hand you can use my Docker images for ppc64le available on the official dockerhub, for the Ansible playbooks all of my work is available on Github. Last words for the introduction I know this post will be huge, take you time to read it (for instance on part per day). I just wanted to let you know this post is the end of a two months working on the project. Let me also thank the fantastic guys of the Opensource Community who gave me help without counting the time they have to spend on it (for ARA: @dmsimard , for Portainer: @Tony_Lapenna, for Traefik: timoreimann).
GlusterFS for data sharing among all the cluster nodes
In my previous blog post about Docker NFS was used to share the data among all the nodes of the Swarm cluster. I decided to give a try to GlusterFS. The first reason is because I think the solution is pretty simple to setup. Second reason is that GlusterFS is a replicated storage solution (ie. each disks are local to each machines, you then create files with a chosen number of copy (for instance two)) allowing to avoid sharing any disks across all the cluster nodes (much simpler for the Ansible automation that we will use to automate all of this). Neat. Here is what I have do to run GlusterFS on my ppc64le systems:
The only version I found for ppc64le is available here https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.20/RHEL/epel-7.3/ppc64le/. You can retrieve all the rpm file using a “wget –mirror” command. I have personnally created my own repository for GlusterFS. (I assume here you are smart enough to do this by yourself):
# cat /etc/yum/yum.repos.d/gluster.repo
[gluster-ppc64le]
name=gluster-ppc64le
baseurl=http://respository.chmod666.org:8080/gluster-ppc64le/
enabled=1
gpgcheck=0
Install the needed packages on all the nodes.
# yum -y install glusterfs-server userspace-rcu parted
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Package userspace-rcu-0.7.16-1.el7.ppc64le already installed and latest version
Package parted-3.1-28.el7.ppc64le already installed and latest version
Resolving Dependencies
[..]
(1/4): glusterfs-api-3.7.20-1.el7.ppc64le.rpm | 87 kB 00:00:00
(2/4): glusterfs-3.7.20-1.el7.ppc64le.rpm | 460 kB 00:00:00
(3/4): glusterfs-fuse-3.7.20-1.el7.ppc64le.rpm | 127 kB 00:00:00
(4/4): glusterfs-server-3.7.20-1.el7.ppc64le.rpm | 1.3 MB 00:00:00
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 14 MB/s | 2.0 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : glusterfs-3.7.20-1.el7.ppc64le 1/4
Installing : glusterfs-fuse-3.7.20-1.el7.ppc64le 2/4
Installing : glusterfs-api-3.7.20-1.el7.ppc64le 3/4
Installing : glusterfs-server-3.7.20-1.el7.ppc64le 4/4
Verifying : glusterfs-server-3.7.20-1.el7.ppc64le 1/4
Verifying : glusterfs-fuse-3.7.20-1.el7.ppc64le 2/4
Verifying : glusterfs-api-3.7.20-1.el7.ppc64le 3/4
Verifying : glusterfs-3.7.20-1.el7.ppc64le 4/4
Installed:
glusterfs-server.ppc64le 0:3.7.20-1.el7
Dependency Installed:
glusterfs.ppc64le 0:3.7.20-1.el7 glusterfs-api.ppc64le 0:3.7.20-1.el7 glusterfs-fuse.ppc64le 0:3.7.20-1.el7
I’m creating local file-systems on each local disks on each cluster node (each cluster node have a local disk attached to it (in my example 600Gb disks), with a filesystem called /data/brick/gvol_docker created on it) (I’m personally using ext4 filesystems (yes, xfs is just a good old Linux joke)):
node1# df | grep brick
/dev/mapper/mpathc1 591G 73M 561G 1% /data/brick/gvol_docker
node1# grep brick /etc/fstab
/dev/mapper/mpathc1 /data/brick/gvol_docker ext4 defaults 0 0
node2# df | grep brick
/dev/mapper/mpathb1 591G 73M 561G 1% /data/brick/gvol_docker
node2# df | grep brick
/dev/mapper/mpathb1 /data/brick/gvol_docker ext4 defaults 0 0
node1# systemctl start glusterd
node2# systemctl start glusterd
From my first node I’m declaring the second node of my cluster. This is what we call “peer probing”. After the probing is done from one node you can check from the other that he “sees” the other node:
node1# gluster peer status
Number of Peers: 0
node1# gluster peer probe node2
peer probe: success.
node1# gluster peer status
Number of Peers: 1
Hostname: node2
Uuid: 3ab38945-c736-4dd2-87dc-1ed25f997608
State: Accepted peer request (Connected)
node2# gluster peer status
Number of Peers: 1
Hostname: node1
Uuid: 9c4d1c7a-de39-4239-a8e1-a35bbe26e25b
State: Accepted peer request (Connected)
From one of the node create a volume for you docker data. I’m here telling I want 2 copies (ie. 2 replicas, one of each node of my cluster). The volume then have to be “started”:
# gluster volume create gvol_docker replica 2 transport tcp node1.chmod66.org:/data/brick/gvol_docker node2.chmod666.org:/data/brick/gvol_docker force
volume create: gvol_docker: success: please start the volume to access data
# gluster volume start gvol_docker
volume start: gvol_docker: success
# gluster volume list
gvol_docker
Modify the /etc/fstab file to be sure mounting GlusterFS volume will be persistent after a reboot. Then manually mount the file-system and check that everything is working as intended:
node1# grep gluster /etc/fstab
node1:gvol_docker /data/docker/ glusterfs defaults,_netdev 0 0
node2# grep gluster /etc/fstab
node1:gvol_docker /data/docker/ glusterfs defaults,_netdev 0 0
node1# mount /data/docker
node1# df -h |grep /data/docker
node1:gvol_docker 591G 73M 561G 1% /data/docker
node1# echo "test" > /data/docker/test
node2# mount /data/docker
node2# cat /data/docker/test
test
Keep in mind that this is just an example . You can do much more using GlusterFS.
Traefik, an intelligent proxy designed for Docker
As I was saying before we will use Traefik as a reverse proxy in our Swarm cluster. I’ll not detail here how to use Traefik but just how to build it on Power. There will be another part in this blog post telling you how to use Traefik inside our cluster. This part is just here to tell you how to build your own Traefik docker image running on the ppc64le architecture.
Building Traefik from sources
To compile Traefik for the ppc64le architecture you first need go. Instead of recompiling once again everything for ppc64le I took the descision to directly install this (go) from a rpm file provided by the EPEL project (that’s why you can see fc for Fedora Core):
# rpm -qa | grep -i golang
golang-src-1.8-2.fc26.noarch
golang-1.8-2.fc26.ppc64le
golang-bin-1.8-2.fc26.ppc64le
Clone the Traefik git repository (using git) (from the internet):
# git clone https://github.com/containous/traefik
Cloning into 'traefik'...
remote: Counting objects: 13246, done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 13246 (delta 2), reused 1 (delta 1), pack-reused 13230
Receiving objects: 100% (13246/13246), 16.25 MiB | 2.09 MiB/s, done.
Resolving deltas: 100% (5429/5429), done.
# ls -ld traefik
drwxr-xr-x 26 root root 4096 Apr 14 14:38 traefik
Traefik needs node and npm, this time I have recompiled node and npm from source (I’ll not explain here how to do this) (but it takes a lot of time, so take a coffee on this one):
# node -v
v8.0.0-pre
# npm --version
4.2.0
# which npm node
/usr/local/bin/npm
/usr/local/bin/node
Set your GOPATH and PATH to be able to run the go binaries. Most of the time to GOPATH is set in a go directory in your home directory:
# export GOPATH=~/go
# export PATH=$PATH:$GOPATH/bin
Download glide and glide-vc (both are prerequisites to build Traefik):
# go get github.com/Masterminds/glide
# go get github.com/sgotti/glide-vc
# go get github.com/jteeuwen/go-bindata/...
# which glide glide-vc
/root/go/bin/glide
/root/go/bin/glide-vc
Install all go prerequisites:
# cd traefik
# script/glide.sh install
[..]
For some unknow reasons or because there is something I do not understand with glide the glide.sh script was not working for me and was not installing any dependencies. I had to use this command to install all the needed go dependencies (based from the glide.yml file were we have a list of all the dependencies needed by Traefik). Then build the Traefik binary:
# cat glide.yaml | awk '$1 ~ /^-/ && $2 ~ /package/ { print "http_proxy=\"http://pinocchio:443\" https_proxy=\"http://pinocchio:443\" go get "$NF }' | sh
# go generate
# go build
# ./traefik version
Version: dev
Codename: cheddar
Go version: go1.8
Built: I don't remember exactly
OS/Arch: linux/ppc64le
# ls -l traefik
-rwxr-xr-x 1 root root 71718807 Apr 14 15:41 traefik
# file traefik
traefik: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked (uses shared libs), not stripped
Building the Traefik image
With the existing binary create a Docker image based on the Docker file provided in the Traefik repository:
# mkdir dist
# cp traefik dist
# docker build -t traefik .
Sending build context to Docker daemon 216.7 MB
Step 1/5 : FROM scratch
--->
Step 2/5 : COPY script/ca-certificates.crt /etc/ssl/certs/
---> Using cache
---> 691efa795e96
Step 3/5 : COPY dist/traefik /
---> 220efc04bd2e
Removing intermediate container 859940376bdb
Step 4/5 : EXPOSE 80
---> Running in badf7d2111d6
---> 4b113142428d
Removing intermediate container badf7d2111d6
Step 5/5 : ENTRYPOINT /traefik
---> Running in cbaa60108171
---> 442a90ed7e7f
Removing intermediate container cbaa60108171
Successfully built 442a90ed7e7f
I’m testing the image is working by running it one time (example toml configuration file can be found in the repository):
# docker run -p 8080:8080 -p 80:80 -v /data/docker/traefik/traefik.toml:/etc/traefik/traefik.toml -v /var/run/docker.sock:/var/run/docker.sock traefik
time="2017-04-14T15:20:55Z" level=error msg="Error opening fileopen log/access.log: no such file or directory"
time="2017-04-14T15:20:55Z" level=error msg="Error opening fileopen log/traefik.log: no such file or directory"
time="2017-04-14T15:20:55Z" level=info msg="Traefik version dev built on I don't remember exactly"
time="2017-04-14T15:20:55Z" level=info msg="Using TOML configuration file /etc/traefik/traefik.toml"
The Docker image is ready to be pushed in your private repository if you want to. Normally Traefik comes with a webUI allowing you to check the configuration of the reverse proxy and the services running on it. Unfortunately for me I didn’t succeed in the compilation of this webUI on ppc64le. I was stuck with a node-saas dependencies note running on ppc64le :
Portainer a free and Open webUI to drive you Swarm cluster
Portainer is a web interface for Docker. It is capable of managing single Docker engines or Swarm clusters. I really like the way it works and all the capabilities. I think this is a mandatory things that you need to provide to your customers if you want them to start on Docker, especially developers who are not all aware of how Docker is working and do not want to be worried knowing every single command line arguments ;-). You can with portainer, manage containers (deletion, creation), manage service across the Swarm cluster (deletion, creation, scaling), work with the volumes, with the networks and so on. It’s easy and beautiful, and it’s OpenSource :-). The only single problem was that I had to recompile it from scratch to be able to run it on the ppc64le arch. Neat !. I also to wanted to thanks the main Portainer developper (Anthony Lapenna). First of all, it a french guy ! (why country pride should be reserved to American people). The second reason is that Anthony is kind, smart, and ready to help. This is a rare thing nowadays I had to it. Once again thank you Anthony. So … here is how to compile Portainer for ppc64le (I’ll also show you a couple of example of how to use … just screenshots):
Building the golang-builder image
The “compilation” of the Portainer binary is based on a docker image running the go language (golang) (It’as a Docker image running a go compilation). As I’m building on a ppc64le archirecture I had to first recreate this image (they have a cross-compiler image, but running on x86, and I do not have any x86 machines). As I didn’t want to recompile go from scratch I decided to use and existing golang image already published on the official docker hub (ppc64le/golang). I then just clone the Portainer sources for their golang-builder image and use those sources to create my own builder image :
# git clone https://github.com/portainer/golang-builder
Cloning into 'golang-builder'...
remote: Counting objects: 186, done.
remote: Total 186 (delta 0), reused 0 (delta 0), pack-reused 186
Receiving objects: 100% (186/186), 47.06 KiB | 0 bytes/s, done.
Then I just had to rewrite to Dockerfile and base it from the ppc64le/golang image. I also obviously had to copy the files from the Portainer repository into the directory in which I’m building the image (if you do not have an internet access, pull the golang Docker image first (docker pull ppc64le/golang:1.7.1)) :
# cat ~/golang-builder-ppc64le/dockerfile
FROM ppc64le/golang:1.7.1
VOLUME /src
WORKDIR /src
COPY build_environment.sh /
COPY build.sh /
ENTRYPOINT ["/build.sh"]
# cp ~/golang-builder/builder-cross/*.sh ~/golang-builder-ppc64le/
# docker pull ppc64le/golang:1.7.1
# docker build -t chmod666/golang-builder:cross-platform .
Sending build context to Docker daemon 6.144 kB
Step 1/6 : FROM ppc64le/golang:1.7.1
---> 25dc29440507
Step 2/6 : VOLUME /src
---> Running in c83ddc8536cf
---> 8c3124eb1bfc
Removing intermediate container c83ddc8536cf
Step 3/6 : WORKDIR /src
---> 6c9f090aa96e
Removing intermediate container 1efc388dc274
Step 4/6 : COPY build_environment.sh /
---> d433a1d71f9b
Removing intermediate container 5233122d6c39
Step 5/6 : COPY build.sh /
---> 325d5d1677f9
Removing intermediate container 147dea39f0fe
Step 6/6 : ENTRYPOINT /build.sh
---> Running in a247e55d9491
---> f059b5f2eb0d
Successfully built f059b5f2eb0d
Building the portainer image
Now that my image is ready I can now prepare everything to build the Portainer container. The web interface is based on node and the build process is done trough a grunt file. I had to change a couple of thing inside this grunt file to be able to compile Portainer for (and on) ppc64le architecture. I had to point to my golang-builder image and add a task “release ppc64le”. The changes can be checked in this github pull request I’ll do as soon as I have the time to do it ![;-)]()
:
- First I had to install node for ppc64le (I’ll not detail here how to do that I just had to once again recompile it from official sources).
- Using npm I had to install bower, run bower to get the dependencies, and install all the npm needed modules:
# npm install -g bower
/usr/local/bin/bower -> /usr/local/lib/node_modules/bower/bin/bower
/usr/local/lib
└── bower@1.8.0
# npm install
npm WARN deprecated grunt-recess@0.3.5: Deprecated as RECESS is unmaintained
npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
[..]
└─┬ grunt-usemin@3.1.1
├─┬ debug@2.6.3
│ └── ms@0.7.2
├── lodash@3.10.1
└── path-exists@1.0.0
npm WARN portainer@1.12.4 No license field.
npm WARN You are using a pre-release version of node and things may not work as expected
# cat ~/.bowerrc
{ "strict-ssl": false }
# bower-install --allow-root
bower filesize#~3.3.0 not-cached https://github.com/avoidwork/filesize.js.git#~3.3.0
bower filesize#~3.3.0 resolve https://github.com/avoidwork/filesize.js.git#~3.3.0
[..]
jquery#1.11.1 bower_components/jquery
moment#2.14.2 bower_components/moment
font-awesome#4.7.0 bower_components/font-awesome
bootstrap#3.3.7 bower_components/bootstrap
└── jquery#1.11.1
Chart.js#1.0.2 bower_components/Chart.js
To be able to run all the grunt tasks I also had to install grunt and grunt-cli:
# npm install grunt
npm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
portainer@1.12.4 /root/portainer
└── grunt@0.4.5
npm WARN portainer@1.12.4 No license field.
npm WARN You are using a pre-release version of node and things may not work as expected
# npm install -g grunt-cli
/usr/local/bin/grunt -> /usr/local/lib/node_modules/grunt-cli/bin/grunt
/usr/local/lib
└─┬ grunt-cli@1.2.0
├─┬ findup-sync@0.3.0
│ └─┬ glob@5.0.15
│ ├─┬ inflight@1.0.6
│ │ └── wrappy@1.0.2
│ ├── inherits@2.0.3
│ ├─┬ minimatch@3.0.3
│ │ └─┬ brace-expansion@1.1.7
│ │ ├── balanced-match@0.4.2
│ │ └── concat-map@0.0.1
│ ├── once@1.4.0
│ └── path-is-absolute@1.0.1
├── grunt-known-options@1.1.0
├─┬ nopt@3.0.6
│ └── abbrev@1.1.0
└── resolve@1.1.7
I also had to install shasum :perl-Digest-SHA-5.85-3.el7.ppc64le):
# yum install perl-Digest-SHA-5.85-3.el7.ppc64le
Run grunt release-ppc64le to build the Portainer binary (Portainer can then by run outside of Docker in the dist directory)
# grunt release-ppc64le
Running "clean:all" (clean) task
Running "if:unixPpc64leBinaryNotExist" (if) task
[if] tests evaluated false: dist/portainer not found!
[if] running tasks:
shell:buildUnixPpc64leBinary
Running "shell:buildUnixPpc64leBinary" (shell) task
[..]
Running "clean:tmp" (clean) task
Cleaning "dist/js/angular.4932b769.js"...OK
Cleaning "dist/js/portainer.e7a82e29.js"...OK
Cleaning "dist/js/vendor.2a5420ef.js"...OK
Cleaning "dist/css/portainer.82f0a61d.css"...OK
Cleaning "dist/css/vendor.20e93620.css"...OK
Running "replace:dist" (replace) task
>> 1 replacement in 1 file.
Done, without errors.
![portainerbuild]()
Finally I had to build the Docker image to be able to run Portainer inside a Docker container:
# grunt run-dev
Running "if:unixBinaryNotExist" (if) task
[if] tests evaluated true:
[if] running tasks:
Running "shell:buildImage" (shell) task
Sending build context to Docker daemon 10.1 MB
Step 1/6 : FROM centurylink/ca-certs
---> ec29b98d130f
Step 2/6 : COPY dist /
---> 200245c49d3a
Removing intermediate container 558cc1ad7bef
Step 3/6 : VOLUME /data
---> Running in 5397202734af
---> be421eb41ebf
Removing intermediate container 5397202734af
Step 4/6 : WORKDIR /
---> 6d63a16558f6
Removing intermediate container e557d646cd95
Step 5/6 : EXPOSE 9000
---> Running in d69ec6302996
---> 4fea90ff91be
Removing intermediate container d69ec6302996
Step 6/6 : ENTRYPOINT /portainer
---> Running in cf0fad52eb9b
---> dd0e2261e1de
Removing intermediate container cf0fad52eb9b
Successfully built dd0e2261e1de
Running "shell:run" (shell) task
Error response from daemon: No such container: portainer
Error response from daemon: No such container: portainer
8f34683455a5115cb3b9c1da6df2776b5747e22169edde4c6ebe30a73c074743
Running "watch:build" (watch) task
Waiting...
# grunt lint
Running "jshint:files" (jshint) task
>> 107 files lint free.
The last step is to tag and push this image on my Docker hub repository:
# docker tag portainer chmod666/portainer_ppc64le:1.12.4
# docker push chmod666/portainer_ppc64le:1.12.4
The push refers to a repository [docker.io/chmod666/portainer_ppc64le]
2ff68fafd090: Pushed
0cfde93eba7d: Mounted from centurylink/ca-certs
5f70bf18a086: Mounted from centurylink/ca-certs
1.12.4: digest: sha256:88a8e365e9ad506b0ad580a634dc13093b6781c102b323d787f5b2d48e90c27c size: 944
![portaindockerhub]()
Run the image
Run the image and check you can access the Portainer gui from you browser:
# docker run -d -p 80:9000 -v /var/run/docker.sock:/var/run/docker.sock --name portainer chmod666/portainer_ppc64le:1.12.4
485b44fb1eabba5d14359a2f0b62c76aeafb8b4e48d7d035c2bb13ccffce0233
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
485b44fb1eab chmod666/portainer_ppc64le:1.12.4 "/portainer" 4 seconds ago Up 3 seconds 0.0.0.0:80->9000/tcp portainer
# uname -a
Linux mydockerhost 3.10.0-514.el7.ppc64le #1 SMP Wed Oct 19 11:27:06 EDT 2016 ppc64le ppc64le ppc64le GNU/Linux
![portainer1]()
![portainer2]()
You can see on the images above that Portainer is running ok. Once again I’ll not explain here how to use Portainer. The tool is so good that I do not have to explain this. It’s just super simple ![;-)]()
.
Use Ansible and Terraform to create this infrastucture
I’m changing my tooling system every couple of months regarding how to “drive” Openstack. I’ve move from the curl way, to the python one, to the Ansible, to finally come to the conclusion that for this purpose (Swarm as a Service) Terraform was the best one. I’ll explain here how to use both Ansible and Terrafrom to drive your Openstack (PowerVC). You have a couple of things to change on the PowerVC host itself. Keep in my that this is not supported at all. ![:-)]()
To use Ansible or Terraform you first have to “prepare” the PowerVC host. Because Ansible use Shade (a python library) and Terafrom it’s own code you have to create some endpoints, availability zones to be able to use those tools. My understanding is that these tools are based on Openstack standards … but everybody knows that standards may differ depending to who you are talking to… so let’s say that PowerVC Openstack standards are not the same as the ones of the people from Shade or Terraform ;-).
Something I’m sure about is that almost nobody is using Openstack “host-aggregate” to isolate group of server (most of the time theses host aggregates are used to isolate type of architecture and not geographic zones, or resiliency zones). That’s why most of the tools that allows you to create Openstack servers are able to set an availibity zone and not a host aggregate. The first thing to do is to set an availaiblity zone per host aggregate you have on your PowerVC host:
# source /opt/ib/powervc/powervcrc
# openstack
(openstack) aggregate list
+----+---------------+-------------------+
| Id | Name | Availability Zone |
+----+---------------+-------------------+
| 21 | myaz1 | - |
| 1 | Default Group | - |
| 22 | myaz2 | - |
+----+---------------+-------------------+
(openstack) aggregate set --zone myaz1 21
(openstack) aggregate list
+----+---------------+-------------------+
| ID | Name | Availability Zone |
+----+---------------+-------------------+
| 21 | myaz1 | myaz1 |
| 1 | Default Group | None |
| 22 | MarneSud | None |
+----+---------------+-------------------+
(openstack) aggregate set --zone myaz2 22
(openstack) aggregate list
+----+---------------+-------------------+
| ID | Name | Availability Zone |
+----+---------------+-------------------+
| 21 | myaz1 | myaz1 |
| 1 | Default Group | None |
| 22 | myaz2 | myaz2 |
+----+---------------+-------------------+
Once again if you want to create volumes using those tools you have to create service and enpoint for volumev2. Both Ansible and Teraform are using volumev2 and PowerVC is using volumev1. To do this you first have to change the keystone policies allowing you to create modify and delete endpoint and services:
# vim /opt/ibm/powervc/policy/keystone/policy.json
[..]
"identity:create_endpoint": "role:service or role:admin",
"identity:delete_endpoint": "role:service or role:admin",
[..]
"identity:create_service": "role:service or role:admin",
"identity:delete_service": "role:service or role:admin",
[..]
Then create the endpoint and service for volumev2:
# source /opt/ibm/powervc/powervcrc
# openstack
(openstack) service create --name cinderv2 --description "Openstack block storage v2" volumev2
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | Openstack block storage v2 |
| enabled | True |
| id | 62865ae6bc1149238fff31b65f34d8ea |
| name | cinderv2 |
| type | volumev2 |
+-------------+----------------------------------+
(openstack) endpoint create --region RegionOne volumev2 public https://deckard.chmod666.org:9000/v2/%(tenant_id)s
+--------------+---------------------------------------------------------+
| Field | Value |
+--------------+---------------------------------------------------------+
| enabled | True |
| id | 904ad762cf4945ad8b57bd4f8e1e8bdc |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 62865ae6bc1149238fff31b65f34d8ea |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | https://deckard.chmod666.org:9000/v2/%(tenant_id)s |
+--------------+---------------------------------------------------------+
(openstack) endpoint create --region RegionOne volumev2 admin https://deckard.chmod666.org:9000/v2/%(tenant_id)s
+--------------+---------------------------------------------------------+
| Field | Value |
+--------------+---------------------------------------------------------+
| enabled | True |
| id | f81509119bdb4338adf0c4a0b30c9415 |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 62865ae6bc1149238fff31b65f34d8ea |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | https://deckard.chmod666.org:9000/v2/%(tenant_id)s |
+--------------+---------------------------------------------------------+
(openstack) endpoint create --region RegionOne volumev2 internal https://127.0.0.1:9000/v2/%(tenant_id)s
+--------------+-----------------------------------------+
| Field | Value |
+--------------+-----------------------------------------+
| enabled | True |
| id | de730a950aa34a4bb1a8fd921183e169 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 62865ae6bc1149238fff31b65f34d8ea |
| service_name | cinderv2 |
| service_type | volumev2 |
| url | https://127.0.0.1:9000/v2/%(tenant_id)s |
+--------------+-----------------------------------------+
(openstack) endpoint list --service volumev2
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+
| 904ad762cf4945ad8b57bd4f8e1e8bdc | RegionOne | cinderv2 | volumev2 | True | public | https://deckard.chmod666.org:9000/v2/%(tenant_id)s |
| de730a950aa34a4bb1a8fd921183e169 | RegionOne | cinderv2 | volumev2 | True | internal | https://127.0.0.1:9000/v2/%(tenant_id)s |
| f81509119bdb4338adf0c4a0b30c9415 | RegionOne | cinderv2 | volumev2 | True | admin | https://deckard.chmod666.org:9000/v2/%(tenant_id)s |
+----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------------------+
For Terraform to be able to create machines with fixed ip addresses you once again have to modify the policy.json file for nova:
[..]
"os_compute_api:os-tenant-networks": "role:service or role:admin",
"os_compute_api:os-tenant-networks:discoverable": "role:service or role:admin",
[..]
At this time you will be able to use both Ansible and Terraform to drive you PowerVC cloud:
Installing Terraform
There are no Terraform packages available for ppc64le, we -one more time- have to build the tool from sources:
# export GOPATH=/root/.go
# export PATH=$PATH:$GOPATH/bin
# go get -u golang.org/x/tools/cmd/stringer
# go get -u github.com/hashicorp/terraform
# cd $GOPATH/src/github.com/hashicorp/terraform/
# make fmt
# make dev
==> Checking that code complies with gofmt requirements...
go generate $(go list ./... | grep -v /terraform/vendor/)
2017/04/20 19:10:21 Generated command/internal_plugin_list.go
==> Removing old directory...
==> Installing gox...
==> Building...
Number of parallel builds: 7
--> linux/ppc64: github.com/hashicorp/terraform
==> Results:
total 198M
-rwxr-xr-x 1 root root 198M Apr 20 19:11 terraform
# file bin/terraform
bin/terraform: ELF 64-bit MSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, not stripped
./bin/terraform --help
Usage: terraform [--version] [--help] [args]
The available commands for execution are listed below.
The most common, useful commands are shown first, followed by
less common or more advanced commands. If you're just getting
started with Terraform, stick with the common commands. For the
other commands, please read the help and docs before usage.
Common commands:
apply Builds or changes infrastructure
console Interactive console for Terraform interpolations
destroy Destroy Terraform-managed infrastructure
env Environment management
fmt Rewrites config files to canonical format
get Download and install modules for the configuration
I’m then creating a Terraform plan allowing me to create three docker machines. Each of the machine will have a 500Gb disk for its dockervg volume group and a 600Gb disk for it’s Gluster disks. The Terraform plan deploy.tf below shows you how to do this. There are a couple of things to node. The “ssh_key_file” variable is the public key of an Openstack key/pair I’ve create on PowerVC. After the machines are installed I’m running an Ansible playbook allowing me to install the rmc packages (this is the local-exec in the “openstack_compute_instance_v2″). After this I’m attaching both disks for each machines. Finally I’m running two local-exec providers one discovering the new disks attached before, and the second one running my Ansible playbook installing and configuring the Swarm cluster. (The one available in my github account) :
# cat deploy.tf
variable "volumetype" {
default = "myvoltype"
}
variable "imagename" {
default = "redhat72ppc64le"
}
variable "network_name" {
default = "myvlan"
}
variable "ssh_key_file" {
default = "~/.ssh/id_rsa.terraform"
}
variable "docker_machines_names" {
default = [ "docker1", "docker2", "docker3" ]
}
variable "docker_machines_ips" {
default = [ "10.19.10.101", "10.19.10.102", "10.19.10.103" ]
}
resource "openstack_compute_keypair_v2" "terraform_key" {
name = "terraform_key"
public_key = "${file("${var.ssh_key_file}.pub")}"
}
resource "openstack_blockstorage_volume_v2" "dockervg_volumes" {
count = "${length(var.docker_machines_names)}"
name = "${element(var.docker_machines_names, count.index)}_docker"
size = "500"
volume_type = "${var.volumetype}"
}
resource "openstack_blockstorage_volume_v2" "gluster_volumes" {
count = "${length(var.docker_machines_names)}"
name = "${element(var.docker_machines_names, count.index)}_gluster"
size = "600"
volume_type = "${var.volumetype}"
}
resource "openstack_compute_instance_v2" "docker_machines" {
count = "${length(var.docker_machines_names)}"
name = "${element(var.docker_machines_names, count.index)}"
image_name = "${var.imagename}"
key_pair = "${openstack_compute_keypair_v2.terraform_key.name}"
flavor_name = "docker"
network {
name = "${var.network_name}"
fixed_ip_v4 = "${element(var.docker_machines_ips, count.index)}"
}
provisioner "local-exec" {
command = "sleep 600 ; ansible-playbook -i /srv/ansible/inventories/ansible_inventory -l -l ${element(var.docker_machines_ips, count.index)} -e ansible_role_path=/srv/ansible /srv/ansible/rmc_linux_on_p.yml\" ; sleep 120"
}
}
resource "openstack_compute_volume_attach_v2" "dockervg_volumes_attach" {
count = "${length(var.docker_machines_names)}"
volume_id = "${element(openstack_blockstorage_volume_v2.dockervg_volumes.*.id, count.index)}"
instance_id = "${element(openstack_compute_instance_v2.docker_machines.*.id, count.index)}"
}
resource "openstack_compute_volume_attach_v2" "gluster_volumes_attach" {
count = "${length(var.docker_machines_names)}"
volume_id = "${element(openstack_blockstorage_volume_v2.gluster_volumes.*.id, count.index)}"
instance_id = "${element(openstack_compute_instance_v2.docker_machines.*.id, count.index)}"
}
resource "null_resource" "discover_disks" {
count = "${length(var.docker_machines_names)}"
provisioner "local-exec" {
command = "ansible-playbook -i /srv/ansible/inventories/ansible_inventory -l ${element(var.docker_machines_ips,count.index)} -e ansible_role_path=/srv/ansible /srv/ansible/rescan_scsi_bus.yml\""
}
depends_on = ["openstack_compute_volume_attach_v2.gluster_volumes_attach"]
}
resource "null_resource" "ansible_run" {
provisionner "local-exec" {
command = "ansible-playbook -i /srv/ansible/inventories/ansible_inventory -e ansible_role_path=/srv/ansible /srv/ansible/docker.yml\""
}
depends_on = ["null_resource.discover_disks"]
}
I’m then just running Terrafrom to deploy the whole stack (machine creation, and Swarm cluster creation):
# ls -l deploy.tf
-rw-rw-r-- 1 root root 3335 Apr 20 19:23 /root/terraform_docker/deploy.tf
# terraform deploy
![tfblog1]()
![tf4]()
Use Ansible to drive your PowerVC Openstack infrastructure
The exact same thing can be done using Ansible. I’ll not go here into deep details but here are playbook samples of machine creation using Ansible:
# cat /tmp/myserver.json
{ "vm_name": "myserver", "fixed_ip": "10.10.10.115", "image_name": "aix72", "ec": "0.20", "vp": "2", "mem": "20480", "vlan_name": "vlan12", "availz": "myaz1" }
# cat create_server.yml
- name: PowerVC | Set Fact
run_once: True
set_fact:
vm_file: "{{ lookup('file', '/tmp/{{ item }}/{{ item }}.json') | from_json }}"
with_items:
- "{{ servers }}"
register: res
delegate_to: localhost
- name: PowerVC | Create instance
run_once: True
os_server:
state: present
timeout: 600
name: "{{ item.ansible_facts.vm_file.vm_name }}"
#boot_volume: "{{ vm_name }}_boot1"
image: "{{ item.ansible_facts.vm_file.image_name }}"
flavor: "cloud"
availability_zone: "{{ item.ansible_facts.vm_file.availz }}"
nics:
- port-name: "{{ item.ansible_facts.vm_file.vm_name }}-port"
with_items:
- "{{ res.results }}"
delegate_to: localhost
Volume attachments can be done too, here is an extract of the playbook:
[..]
- name: PowerVC | Attaching volumes
run_once: True
os_server_volume:
state: present
server: "{{ item[0] }}"
volume: "{{ item[1] }}"
when: (( item[0]+"_" in item[1]) and ( "-boot" not in item[1]))
with_nested:
- "{{ servers }}"
- "{{ volumes.stdout_lines }}"
delegate_to: localhost
tags: move
Log every Ansible run using Openstack ARA
the goal Openstack ARA is to provide to the final user a way to trace and debug their Ansible run. It’s a simple web interface storing and tracing every action that you have done with Ansible. I like it a lot. It respects the KISS principle (Keep it Soft and Simple). I’m my opinion ARA is ten time better than the Ansible tower for the reporting part. Here is my simple how to to run ARA in Docker containers:
Building the Docker Container for the ARA webapp
I’m going to build the ARA web application based on the “official” image of CentOS. I’m doing this because I do not want to use my custom images because I want to publish this image on the hub and obviously you don’t care about any of my customization. So let start:
# docker search ppc64le
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
[..]
ppc64le/centos 0
hitomitak/centos72-ppc64le 0
# docker pull ppc64le/centos
Using default tag: latest
latest: Pulling from ppc64le/centos
d2664d978557: Pull complete
Digest: sha256:997bd3301d0032ba1f04be925e006b9fdabc254be7dfbf8b0fc7bbab21108efc
Status: Downloaded newer image for ppc64le/centos:latest
I’m then writing a dockerfile in charge of installing ARA and the needed dependencies. A script called run.sh is copied into the container to run the ara-manager. I’ll later need to have an environment variable called ARA_DATABASE to tell the ara web application where the database is. I’m not putting this into the Dockerfile to allow the user to use whatever he wants to.
- The image is build from the docker “ppc64le/centos” image.
- I’m installing the needed dependencies for ARA.
- I’m installing pip and ARA.
- I’m creating a volume, copying the run.sh into the container and make it the entry point.
- I’m buidling the container with the –build-args arguments because I’m using a proxy to access the internet.
# cat docker file
FROM ppc64le/centos
RUN yum -y update && yum -y install gcc python-devel libffi-devel openssl-devel python libxml2-devel libxslt-devel MySQL-python curl mailcap
RUN curl -k https://bootstrap.pypa.io/get-pip.py -o /get-pip.py && python /get-pip.py && pip install ara
EXPOSE 8080
COPY run.sh /
RUN chmod +x run.sh
ENTRYPOINT ["/run.sh"]
# docker build --build-arg http_proxy="http://myproxy:443" --build-arg https_proxy="http://myproxy:443" -t chmod666/ara_webapp .
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM ppc64le/centos
---> 3d0ded8c6f42
Step 2 : RUN yum -y update && yum -y install gcc python-devel libffi-devel openssl-devel python libxml2-devel libxslt-devel MySQL-python
---> Using cache
---> 3d3e45af237b
Step 3 : RUN curl -k https://bootstrap.pypa.io/get-pip.py -o /get-pip.py && python /get-pip.py && pip install ara
---> Running in d84924b00812
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1558k 100 1558k 0 0 1451k 0 0:00:01 0:00:01 --:--:-- 1450k
[..]
The run.sh script just run the ara-manage command, we will see at the execution time that we pass the ARA_DATABASE as an environment variable to allow the web application to query the database:
# cat run.sh
#!/bin/bash -e
ara-manage runserver -h 0.0.0.0 -p 8080
Building the Docker Container for the ARA databse
I now need a database to store the ara records. I’m using mariadb. In this image I’m creating the ara database, and an ara user with all the privileges needed to write in the database, here is the Dockerfile and the build execution:
# docker build --build-arg http_proxy="http://myproxy:443" --build-arg https_proxy="http://myproxy:443" -t chmod666/ara_db .
Sending build context to Docker daemon 2.56 kB
Step 1 : FROM ppc64le/centos
---> 3d0ded8c6f42
Step 2 : RUN yum -y update && yum -y install mariadb-server && echo "mysql_install_db --user=mysql" > /tmp/config && echo "mysqld_safe &" >> /tmp/config && echo "mysqladmin --silent --wait=30 ping || exit 1" >> /tmp/config && echo "mysql -e 'CREATE DATABASE ara;'" >> /tmp/config && echo "mysql -e \"CREATE USER ara@'%'IDENTIFIED BY 'password';\"" >> /tmp/config && echo "mysql -e 'GRANT ALL PRIVILEGES ON ara.* TO ara@\"%\";'" >> /tmp/config && echo "mysql -e 'FLUSH PRIVILEGES;'" >> /tmp/config && bash /tmp/config && rm -f /tmp/config
---> Running in 083ab7adc7fc
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
[..]
Step 3 : VOLUME /etc/mysql /var/lib/mysql
---> Running in 50230896793d
---> d77628675cf2
Removing intermediate container 50230896793d
Step 4 : CMD mysqld_safe
---> Running in 5a997c29b5c1
---> 9bbb54c36d41
Removing intermediate container 5a997c29b5c1
Step 5 : EXPOSE 3306
---> Running in 6340f6f9b975
---> 85fcb3f6436e
Removing intermediate container 6340f6f9b975
Successfully built 85fcb3f6436e
FROM ppc64le/centos
RUN \
yum -y update && \
yum -y install mariadb-server && \
echo "mysql_install_db --user=mysql" > /tmp/config && \
echo "mysqld_safe &" >> /tmp/config && \
echo "mysqladmin --silent --wait=30 ping || exit 1" >> /tmp/config && \
echo "mysql -e 'CREATE DATABASE ara;'" >> /tmp/config && \
echo "mysql -e \"CREATE USER ara@'%'IDENTIFIED BY 'password';\"" >> /tmp/config && \
echo "mysql -e 'GRANT ALL PRIVILEGES ON ara.* TO ara@\"%\";'" >> /tmp/config && \
echo "mysql -e 'FLUSH PRIVILEGES;'" >> /tmp/config && \
bash /tmp/config && \
rm -f /tmp/config
VOLUME ["/etc/mysql", "/var/lib/mysql"]
CMD ["mysqld_safe"]
EXPOSE 3306
I’m tagging and uploading my images into my private repository. I’ll not show here how to run ARA we will see this in the next part.:
# docker tag chmod666/ara_webapp registry.chmod666.org:5000/chmod666/ara_webapp
# docker tag chmod666/ara_db registry.chmod666.org:5000/chmod666/ara_db
# docker push registry.chmod666.org:5000/chmod666/ara_webapp
The push refers to a repository [registry.chmod666.org:5000/chmod666/ara_webapp]
e6daa00bf7b0: Pushed
66ff8c22376f: Pushed
6327608d1321: Pushed
4cdf21a0f079: Pushed
8feae09cb245: Pushed
latest: digest: sha256:2d097fee31170eb2b3fbbe099477d345d1beabbbd83788ca9d35b35cec353498 size: 1367
# docker push registry.chmod666.org:5000/chmod666/ara_webapp
[..]
Installing the callback on the Ansible master
After the service is started and created (look at the next part for this) you have to install the ARA call back on the Ansible master host. Use pip to do this. Then you just have to export some variable to tell Ansible where the call back is and where is the ARA database. I have personnally set this in the /etc/bashrc, by doing this every Ansible run is recorded into ARA:
# ARA env var
export ara_location=$(python -c "import os,ara; print(os.path.dirname(ara.__file__))")
export ANSIBLE_CALLBACK_PLUGINS=$ara_location/plugins/callbacks
export ANSIBLE_ACTION_PLUGINS=$ara_location/plugins/actions
export ANSIBLE_LIBRARY=$ara_location/plugins/modules
export ARA_DATABASE="mysql+mysqldb://ara:password@node1.chmod666.org/ara"
Verify everything is working as attended (access the ARA webapp by a web brower)
Run a few Ansible playbooks and verify that you know have some data in the ARA database, here are some example after a few day running Ansible in production:
![arab1]()
![arab2]()
![arab3]()
![aradb4]()
![arab5]()
![arab6]()
![arab7]()
Working with Traefik
Now that we have all our Docker images ready to build our Docker As a Service platform let’s see how to run those image behind the Traefik reserve proxy. To do so we will create “services” on our Swarm cluster. With the latest version of Swarm (Running in Docker 17.03) we now have the possibility to create compose files to run those services, theses services can be run using the “docker stack” command, let’s see how this is working. The first thing to do is to create a service to run Traefik itself. The yaml file below shows you how to create a service running Traefik, publish port 80 443 and 8080.
# cd stack_files
# cat traefik.yml
version: "3"
services:
traefik:
image: chmod666/traefik
command: --docker --docker.swarmmode --docker.watch --web
networks:
- swarm_overlay
ports:
- "80:80"
- "443:443"
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /data/docker/traefik/traefik.toml:/etc/traefik/traefik.toml
networks:
swarm_overlay:
external:
name: mynet
# docker stack deploy --compose-file=traefik.yml traefik
Creating service traefik_traefik
# docker service ls
ID NAME MODE REPLICAS IMAGE
ycrymxbsif77 traefik_traefik replicated 1/1 chmod666/traefik
You can then check that Traefik is running on one of the node of our cluster:
# docker stack ls
NAME SERVICES
traefik 1
# docker stack ps traefik
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
dldc45i1j2w8 traefik_traefik.1 chmod666/traefik node1 Running Running 1 second ago
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c64d7869bbfa chmod666/traefik:latest "/traefik" 13 seconds ago Up 8 seconds 80/tcp traefik_traefik.1.dldc45i1j2w8wfvipm80u6nco
693fc9cd91db registry "./entrypoint.sh /..." 2 weeks ago Up 5 days 0.0.0.0:5000->5000/tcp registry
Now that Traefik is running we will now run our Portainer service. Note here that I’m not exposing any port. This is the role of Traefik to do the job of a reverse proxy. So you can see that no port are exposed. So how does Traefik knows how to access Portainer ? This is achieved by the labels you see in the stack file (into the deploy section). The “traefik.port” label tells Traefik that this service is running on port 9000 (Traefik will here redirect all the traffic coming from the url having a ContextPath named /portainer into the Portainer container running on port 9000 (on the overlay network)). The “traefik.frontend.rule=PrefixPathStrip:/portainer” tells Traefik this service will be accessible using the context path “/portainer” (ie. for instance docker.chmod666.org/portainer). Most of the time this part is achieved using the “Host” rule (ie. “traefik.frontend.rule=Host:portainer.docker.chmod666.org), this is much simpler and work everytimes (some web application needs complex rewrite rules to work with a context path, and sometime the Traefik rules (Path, PathPrefix, PathStrip, PathPrefixStrip and AddPrefix) are not enough (I had some difficulties for with WordPress for instance). Problem is you have to create a new CNAME entry every time a new service is created …. not that scalable …. The solution to this problem is to create a DNS willcard entry (ie. *.docker.chmod666.org will tell a.docker.chmod666.org or randomname.docker.chmod666.org will be resolved with the same address). Unfortunately some dns providers does not allows doing this or if like me you can’t talk with you network team … you’re stuck and have to play with the other rules (Path ones). The example below shows you the utilization of “PathPrefixStrip”. Please also note that I’m running this service in global mode, telling I want this service to run one every node of the cluster:
version: "3"
services:
portainer:
image: chmod666/portainer
command: -H unix:///var/run/docker.sock
deploy:
labels:
- traefik.port=9000
- traefik.frontend.rule=PathPrefixStrip:/portainer
mode: global
networks:
- swarm_overlay
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
swarm_overlay:
external:
name: mynet
# docker stack deploy --compose-file=portainer.yml portainer
Creating service portainer_portainer
# docker stack ls
NAME SERVICES
portainer 1
traefik 1
# docker stack ps portainer
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
ndmx4bvolrfn portainer_portainer.636fmsoqyspcu9bbzs12r8fs8 chmod666/portainer node1.chmod666.org Running Running 5 seconds ago
m2jtuz4hzcwf portainer_portainer.0piqhrpzkjfynakyvek7r53mw chmod666/portainer node2.chmod666.org Running Running 2 seconds ago
# docker service ls
ID NAME MODE REPLICAS IMAGE
gsmn18is9whg portainer_portainer global 2/2 chmod666/portainer
i7uucqflzewt traefik_traefik replicated 1/1 chmod666/traefik
If we have a look at the Traefik logs we can see that Traefik sees that new docker images were created allowing us to access Portainer using the context path “/portainer”. You can check the picture below prooving this. I repeat this but as the service is replicated if we stop one node of the swarm cluster portainer will still be accessible:
time="2017-04-15T21:08:29Z" level=debug msg="Last docker config received more than 2s, OK"
time="2017-04-15T21:08:29Z" level=debug msg="Creating frontend frontend-PathPrefixStrip-portainer"
time="2017-04-15T21:08:29Z" level=debug msg="Wiring frontend frontend-PathPrefixStrip-portainer to entryPoint http"
time="2017-04-15T21:08:29Z" level=debug msg="Creating route route-frontend-PathPrefixStrip-portainer PathPrefixStrip:/portainer"
time="2017-04-15T21:08:29Z" level=debug msg="Creating backend backend-portainer-portainer"
time="2017-04-15T21:08:29Z" level=debug msg="Creating load-balancer wrr"
time="2017-04-15T21:08:29Z" level=debug msg="Creating server server-portainer_portainer-7y0ujh71y4nkvt5ynng7il6bv at http://10.0.0.6:9000 with weight 0"
time="2017-04-15T21:08:29Z" level=debug msg="Creating server server-portainer_portainer-oz7b9ev1js17wgvktd6uuqdww at http://10.0.0.5:9000 with weight 0"
time="2017-04-15T21:08:29Z" level=info msg="Server configuration reloaded on :80"
![portainerbehindtraefik]()
For ARA I had tried different configuration of Traefik using the Path* rules but none of them was working. I had to create a new CNAME alias for my service and then add to use the “Host” rule. This is also a good way to show you rule with Host:
# cat ara.yml
version: "3"
services:
ara:
image: chmod666/ara_webapp
environment:
- ARA_DATABASE="mysql+mysqldb://ara:password@aradb/ara"
deploy:
labels:
- traefik.port=80
- traefik.frontend.rule=Host:unixara.chmod666.org
mode: global
networks:
- aranet
aradb:
image: chmod666/ara_db
ports:
- "3306:3306"
volumes:
- /var/lib/mysql/:/data/docker/ara_db
networks:
- aranet
networks:
aranet:
external:
name: mynet
# docker stack deploy --compose-file=ara.yml ara
Creating service ara_ara
Creating service ara_aradb
# docker logs 7484945344c4
time="2017-04-15T22:27:59Z" level=debug msg="Last docker config received more than 2s, OK"
time="2017-04-15T22:27:59Z" level=debug msg="Creating frontend frontend-Host-unixara-fr-net-intra"
time="2017-04-15T22:27:59Z" level=debug msg="Wiring frontend frontend-Host-unixara-fr-net-intra to entryPoint http"
time="2017-04-15T22:27:59Z" level=debug msg="Creating route route-frontend-Host-unixara-chmod666-org Host:unixara.chmod666.org"
time="2017-04-15T22:27:59Z" level=debug msg="Creating backend backend-ara-ara"
![aratraefik]()
Mixing everything together
We now have every pieces of our puzzle ready to be assembled. The automation is realized by an Ansible playbook that I named “swarm_as_a_service_for_ppc64le”. This playbook is available on my github account at this address: https://github.com/chmod666org/swarm_as_a_service_ppc64le. The playbook is split in different roles:
- Docker engine: This role install the Docker engine on the machines. It creates a dockervg (on a 500GB disk) and setup the device mapper using the RedHat Atomic host project “docker-storage-setup”. It also install docker-compose. Both tools are cloned from the official docker repository (cloned on the Ansible master, then copied on the nodes). After this step you have a ready to be used “docker machine”
- Registry server: This role deploys a private registry on the first node of your managers node. This registry is secured, this means that self signed certificate are created during the process. All our images are pushed into this registry (ara, traefik, portainer). The registry is also authenticated by a user and a password.
- Registry client: This role tells every nodes of the Swarm cluster to log on the private registry. Needed certificates are copied from the host hosting the registry to every node of the cluster. Once this is done images are pulled locally (we will need this later).
- Swarm: This role creates the docker swarm. In the inventory every node should be “tagged” as a manager ([docker_swarm_managers]) or a worker ([docker_swarm_worker]). You can have as many nodes as you want.
- Gluster: This role is in charge to create a gluster filesystem in /data/docker/[hostname_of_the_first_manager] available on every node of the cluster. Each node of the gluster cluster must have a 600GB disk. The default number of replicas is two.
- Docker service: This role run our services (ie. traefik, ara, and portainer) in the cluster using the new “docker stack” command. The services are described in “docker-compose.yml” for each services. At this time after services are created a random password is generated for the Portainer admin user (extract of the playbook below) and a mail is send to the user telling him the Swarm cluster is ready to be used.
- name: docker_services | Setting Portainer root password
run_once: True
uri:
url: 'http://{{ groups.docker_swarm_managers[0] }}/portainer/api/users/admin/init'
method: POST
validate_certs: no
HEADER_Content-Type: "application/json"
HEADER_Accept: "application/json"
return_content: yes
body_format: json
body: "{{ lookup('template', 'portainer_admin.json.j2') }}"
status_code: 200
timeout: 240
delegate_to: localhost
when: '"portainer" not in docker_stacks.stdout'
Check the inventory is ok, and run the playbook. (Normally this is run by Terraform after the machine are created but for the convenience of this blog post I’m showing here how to launch the playbook by hand).
# cat inventories/hosts.docker
[docker]
node1.chmod666.org
node2.chmod66.org
[docker_registry_servers]
node1.chmod666.org
[docker_registry_clients]
node2.chmod666.org
[docker_swarm_managers]
node1.chmod666.org
[docker_swarm_workers]
node2.chmod666.org
[gluster_main]
node1.chmod666.org
[gluster_nodes]
node1.chmod666.org
node2.chmod666.org
# ansibe-playbook -i inventories/hosts.docker docker.yml
![aansible11]()
![ansible3]()
When everything is finish the final customer got an email telling him his Swarm Cluster is ready to be used:
![aansible3]()
Conclusion
You now do have no excuse anymore to tell your boss PowerSystems are ready for DevOps. Spread the word and tell everybody that we can do the exact same thing on Power than they do on x86. It’s a fact ! Finally think about the advantages of doing this on Power. You beat the moore law. You have a simple Openstack infrastructure ready to be used on Power. You can mix teams and OS (Linux and AIX) on the same hardware. Finally … you’ll be different by design. It is not good to be different ?
). Here are a couple of things and best practices to know before doing this.
)
