In the previous blog post, we looked into Docker’s single-host networking for MySQL containers. This time, we are going to look into the basics of multi-host networking and Docker swarm mode, a built-in orchestration tool to manage containers across multiple hosts.

Docker Engine - Swarm Mode

Running MySQL containers on multiple hosts can get a bit more complex depending on the clustering technology you choose.

Before we try to run MySQL on containers + multi-host networking, we have to understand how the image works, how much resources to allocate (disk,memory,CPU), networking (the overlay network drivers - default, flannel, weave, etc) and fault tolerance (how is the container relocated, failed over and load balanced). Because all these will impact the overall operations, uptime and performance of the database. It is recommended to use an orchestration tool to get more manageability and scalability on top of your Docker engine cluster. The latest Docker Engine (version 1.12, released on July 14th, 2016) includes swarm mode for natively managing a cluster of Docker Engines called a Swarm. Take note that Docker Engine Swarm mode and Docker Swarm are two different projects, with different installation steps despite they both work in a similar way.

Some of the noteworthy parts that you should know before entering the swarm world:

• The following ports must be opened:
  • 2377 (TCP) - Cluster management
  • 7946 (TCP and UDP) - Nodes communication
  • 4789 (TCP and UDP) - Overlay network traffic
• There are 2 types of nodes:
  • Manager - Manager nodes perform the orchestration and cluster management functions required to maintain the desired state of the swarm. Manager nodes elect a single leader to conduct orchestration tasks.
  • Worker - Worker nodes receive and execute tasks dispatched from manager nodes. By default, manager nodes are also worker nodes, but you can configure managers to be manager-only nodes.

More details in the Docker Engine Swarm documentation.

In this blog, we are going to deploy application containers on top of a load-balanced Galera Cluster on 3 Docker hosts (docker1, docker2 and docker3), connected through an overlay network as a proof of concept for MySQL clustering in multiple Docker hosts environment. We will use Docker Engine Swarm mode as the orchestration tool.

“Swarming” Up

Let’s cluster our Docker nodes into a Swarm. Swarm mode requires an odd number of managers (obviously more than one) to maintain quorum for fault tolerance. So, we are going to use all the physical hosts as manager nodes. Note that by default, manager nodes are also worker nodes.

1. Firstly, initialize Swarm mode on docker1. This will make the node as manager and leader:

   [root@docker1]$ docker swarm init --advertise-addr 192.168.55.111
   Swarm initialized: current node (6r22rd71wi59ejaeh7gmq3rge) is now a manager.
   
   To add a worker to this swarm, run the following command:
   
       docker swarm join \
       --token SWMTKN-1-16kit6dksvrqilgptjg5pvu0tvo5qfs8uczjq458lf9mul41hc-dzvgu0h3qngfgihz4fv0855bo \
       192.168.55.111:2377
   
   To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

2. We are going to add two more nodes as manager. Generate the join command for other nodes to register as manager:

   [docker1]$ docker swarm join-token manager
   To add a manager to this swarm, run the following command:
   
       docker swarm join \
       --token SWMTKN-1-16kit6dksvrqilgptjg5pvu0tvo5qfs8uczjq458lf9mul41hc-7fd1an5iucy4poa4g1bnav0pt \
       192.168.55.111:2377

3. On docker2 and docker3, run the following command to register the node:

   $ docker swarm join \
       --token SWMTKN-1-16kit6dksvrqilgptjg5pvu0tvo5qfs8uczjq458lf9mul41hc-7fd1an5iucy4poa4g1bnav0pt \
       192.168.55.111:2377

4. Verify if all nodes are added correctly:

   [docker1]$ docker node ls
   ID                           HOSTNAME       STATUS  AVAILABILITY  MANAGER STATUS
   5w9kycb046p9aj6yk8l365esh    docker3.local  Ready   Active        Reachable
   6r22rd71wi59ejaeh7gmq3rge *  docker1.local  Ready   Active        Leader
   awlh9cduvbdo58znra7uyuq1n    docker2.local  Ready   Active        Reachable

   At the moment, we have docker1.local as the leader. 

Overlay Network

The only way to let containers running on different hosts connect to each other is by using an overlay network. It can be thought of as a container network that is built on top of another network (in this case, the physical hosts network). Docker Swarm mode comes with a default overlay network which implements a VxLAN-based solution with the help of libnetwork and libkv. You can however choose another overlay network driver like Flannel, Calico or Weave, where extra installation steps are necessary. We are going to cover more on that later in an upcoming blog post.

In Docker Engine Swarm mode, you can create an overlay network only from a manager node and it doesn’t need an external key-value store like etcd, consul or Zookeeper.

The swarm makes the overlay network available only to nodes in the swarm that require it for a service. When you create a service that uses an overlay network, the manager node automatically extends the overlay network to nodes that run service tasks.

Let’s create an overlay network for our containers. We are going to deploy Percona XtraDB Cluster and application containers on separate Docker hosts to achieve fault tolerance. These containers must be running on the same overlay network so they can communicate with each other.

We are going to name our network “mynet”. You can only create this on the manager node:

[docker1]$ docker network create --driver overlay mynet

Let’s see what networks we have now:

[docker1]$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
213ec94de6c9        bridge              bridge              local
bac2a639e835        docker_gwbridge     bridge              local
5b3ba00f72c7        host                host                local
03wvlqw41e9g        ingress             overlay             swarm
9iy6k0gqs35b        mynet               overlay             swarm
12835e9e75b9        none                null                local

There are now 2 overlay networks with a Swarm scope. The “mynet” network is what we are going to use today when deploying our containers. The ingress overlay network comes by default. The swarm manager uses ingress load balancing to expose the services you want externally to the swarm.

Deployment using Services and Tasks

We are going to deploy the Galera Cluster containers through services and tasks. When you create a service, you specify which container image to use and which commands to execute inside running containers. There are two type of services:

• Replicated services - Distributes a specific number of replica tasks among the nodes based upon the scale you set in the desired state, for examples “--replicas 3”.
• Global services - One task for the service on every available node in the cluster, for example “--mode global”. If you have 7 Docker nodes in the Swarm, there will be one container on each of them.

Docker Swarm mode has a limitation in managing persistent data storage. When a node fails, the manager will get rid of the containers and create new containers in place of the old ones to meet the desired replica state. Since a container is discarded when it goes down, we would lose the corresponding data volume as well. Fortunately for Galera Cluster, the MySQL container can be automatically provisioned with state/data when joining.

Deploying Key-Value Store

The docker image that we are going to use is from Percona-Lab. This image requires the MySQL containers to access a key-value store (supports etcd only) for IP address discovery during cluster initialization and bootstrap. The containers will look for other IP addresses in etcd, if there are any, start the MySQL with a proper wsrep_cluster_address. Otherwise, the first container will start with the bootstrap address, gcomm://.

1. Let’s deploy our etcd service. We will use etcd image available here. It requires us to have a discovery URL on the number of etcd node that we are going to deploy. In this case, we are going to setup a standalone etcd container, so the command is:

   [docker1]$ curl -w "\n"'https://discovery.etcd.io/new?size=1'
   https://discovery.etcd.io/a293d6cc552a66e68f4b5e52ef163d68

2. Then, use the generated URL as “-discovery” value when creating the service for etcd:

   [docker1]$ docker service create \
   --name etcd \
   --replicas 1 \
   --network mynet \
   -p 2379:2379 \
   -p 2380:2380 \
   -p 4001:4001 \
   -p 7001:7001 \
   elcolio/etcd:latest \
   -name etcd \
   -discovery=https://discovery.etcd.io/a293d6cc552a66e68f4b5e52ef163d68

   At this point, Docker swarm mode will orchestrate the deployment of the container on one of the Docker hosts.

3. Retrieve the etcd service virtual IP address. We are going to use that in the next step when deploying the cluster:

   [docker1]$ docker service inspect etcd -f "{{ .Endpoint.VirtualIPs }}"
   [{03wvlqw41e9go8li34z2u1t4p 10.255.0.5/16} {9iy6k0gqs35bn541pr31mly59 10.0.0.2/24}]

   At this point, our architecture looks like this:

Deploying Database Cluster

1. Specify the virtual IP address for etcd in the following command to deploy Galera (Percona XtraDB Cluster) containers:

   [docker1]$ docker service create \
   --name mysql-galera \
   --replicas 3 \
   -p 3306:3306 \
   --network mynet \
   --env MYSQL_ROOT_PASSWORD=mypassword \
   --env DISCOVERY_SERVICE=10.0.0.2:2379 \
   --env XTRABACKUP_PASSWORD=mypassword \
   --env CLUSTER_NAME=galera \
   perconalab/percona-xtradb-cluster:5.6

2. It takes some time for the deployment where the image will be downloaded on the assigned worker/manager node. You can verify the status with the following command:

   [docker1]$ docker service ps mysql-galera
   ID                         NAME                IMAGE                                  NODE           DESIRED STATE  CURRENT STATE            ERROR
   8wbyzwr2x5buxrhslvrlp2uy7  mysql-galera.1      perconalab/percona-xtradb-cluster:5.6  docker1.local  Running        Running 3 minutes ago
   0xhddwx5jzgw8fxrpj2lhcqeq  mysql-galera.2      perconalab/percona-xtradb-cluster:5.6  docker3.local  Running        Running 2 minutes ago
   f2ma6enkb8xi26f9mo06oj2fh  mysql-galera.3      perconalab/percona-xtradb-cluster:5.6  docker2.local  Running        Running 2 minutes ago

3. We can see that the mysql-galera service is now running. Let’s list out all services we have now:

   [docker1]$ docker service ls
   ID            NAME          REPLICAS  IMAGE                                  COMMAND
   1m9ygovv9zui  mysql-galera  3/3       perconalab/percona-xtradb-cluster:5.6
   au1w5qkez9d4  etcd          1/1       elcolio/etcd:latest                    -name etcd -discovery=https://discovery.etcd.io/a293d6cc552a66e68f4b5e52ef163d68

4. Swarm mode has an internal DNS component that automatically assigns each service in the swarm a DNS entry. So you use the service name to resolve to the virtual IP address:

   [docker2]$ docker exec -it $(docker ps | grep etcd | awk {'print $1'}) ping mysql-galera
   PING mysql-galera (10.0.0.4): 56 data bytes
   64 bytes from 10.0.0.4: seq=0 ttl=64 time=0.078 ms
   64 bytes from 10.0.0.4: seq=1 ttl=64 time=0.179 ms

   Or, retrieve the virtual IP address through the “docker service inspect” command:

   [docker1]# docker service inspect mysql-galera -f "{{ .Endpoint.VirtualIPs }}"
   [{03wvlqw41e9go8li34z2u1t4p 10.255.0.7/16} {9iy6k0gqs35bn541pr31mly59 10.0.0.4/24}]

   Our architecture now can be illustrated as below:

Deploying Applications

Finally, you can create the application service and pass the MySQL service name (mysql-galera) as the database host value:

[docker1]$ docker service create \
--name wordpress \
--replicas 2 \
-p 80:80 \
--network mynet \
--env WORDPRESS_DB_HOST=mysql-galera \
--env WORDPRESS_DB_USER=root \
--env WORDPRESS_DB_PASSWORD=mypassword \
wordpress

Once deployed, we can then retrieve the virtual IP address for wordpress service through the “docker service inspect” command:

[docker1]# docker service inspect wordpress -f "{{ .Endpoint.VirtualIPs }}"
[{p3wvtyw12e9ro8jz34t9u1t4w 10.255.0.11/16} {kpv8e0fqs95by541pr31jly48 10.0.0.8/24}]

At this point, this is what we have:

Our distributed application and database setup is now deployed by Docker containers.

Connecting to the Services and Load Balancing

At this point, the following ports are published (based on the -p flag on each “docker service create” command) on all Docker nodes in the cluster, whether or not the node is currently running the task for the service:

• etcd - 2380, 2379, 7001, 4001
• MySQL - 3306
• HTTP - 80

If we connect directly to the PublishedPort, with a simple loop, we can see that the MySQL service is load balanced among containers:

[docker1]$ while true; do mysql -uroot -pmypassword -h127.0.0.1 -P3306 -NBe 'select @@wsrep_node_address'; sleep 1; done
10.255.0.10
10.255.0.8
10.255.0.9
10.255.0.10
10.255.0.8
10.255.0.9
10.255.0.10
10.255.0.8
10.255.0.9
10.255.0.10
^C

 At the moment, Swarm manager manages the load balancing internally and there is no way to configure the load balancing algorithm. We can then use external load balancers to route outside traffic to these Docker nodes. In case of any of the Docker nodes goes down, the service will be relocated to the other available nodes.

**Warning: The setup shown in this page is just a proof of concept. It may be incomplete for production use and does not cover a full single point of failure (SPOF) setup.

That’s all for now. In the next blog post, we’ll take a deeper look at Docker overlay network drivers for MySQL containers.

Welcome to this week’s Planets9s, covering all the latest resources and technologies we create around automation and management of open source database infrastructures.

	MySQL Query Tuning: Process & Tools - watch the replay You can now watch the replay of this week’s webinar on MySQL Query Tuning: Process & Tools. Many thanks to everyone who participated. We looked at the query tuning process for MySQL from building, collecting, analysing, through to tuning and testing. And we discussed some of the main tools involved in this process, such as tcpdump and Pt-query-digest. Watch the replay
MySQL on Docker: Introduction to Docker Swarm Mode and Multi-Host Networking While we recently looked into Docker’s single-host networking for MySQL containers, we are now discussing the basics of multi-host networking and Docker swarm mode, a built-in orchestration tool to manage containers across multiple hosts. It is recommended to use an orchestration tool to get more manageability and scalability on top of your Docker engine cluster, and this blog post provides insight and pointers as to how to proceed with that. Read the blog
	Severalnines talks & tutorials at Percona Live Amsterdam If you’re in Europe early October, then you may want to drop in to the Percona Live conference in Amsterdam, which this year covers a wide range of specialist topics for MySQL, MongoDB & PostgreSQL users. We’ll be there as well with a number of talks and tutorials, as well as a presence in the exhibition hall and we’d love to meet you there. We’ll be talking about how to become a MySQL and MongoDB DBA, load balancers and more. There’s still time to sign up! Find out more

That’s it for this week! Feel free to share these resources with your colleagues and follow us in our social media channels.

Have a good end of the week,

Jean-Jérôme Schmidt
Planets9s Editor
Severalnines AB

In the previous post, we looked into the basics of running MySQL containers on multiple Docker hosts managed by Swarm Mode, a native orchestration tool comes with Docker 1.12. However, at the time of writing, Docker Engine Swarm Mode does not support other networking plugins like Calico, Weave or Flannel. If we’d like to run any of these, we must run it outside of Docker Swarm mode and use other tools for orchestration e.g, Kubernetes, Mesos or Docker Swarm.

In this blog post, we are going to look into other networking drivers that support multi-host networking to best fit our MySQL setups. We are going to deploy MySQL Replication on top of three Docker hosts via Calico’s driver on multi-host networking. Weave and Flannel will be covered in the upcoming blog posts.

Calico cannot be treated as an “overlay network” - which means it does not encapsulate one packet inside another packet. It uses pure Layer 3 approach and avoids the packet encapsulation associated with the Layer 2 solution which simplifies diagnostics, reduces transport overhead and improves performance. Calico also implements BGP protocol for routing combined with a pure IP network, thus allows internet scaling for virtual networks.

Consider having 3 physical hosts installed with Docker Engine v1.12.1. All hosts are running on CentOS 7.1. The following is the output of /etc/hosts on each host:

192.168.55.111    docker1.local docker1
192.168.55.112    docker2.local docker2
192.168.55.113    docker3.local docker3

Key-Value Store (etcd)

Etcd is a popular open-source distributed key value store that provides shared configuration and service discovery. A simple use-case is to store database connection details or feature flags in etcd as key value pairs.

Calico requires etcd to operate. Etcd can be clustered with many instances. In this example, we are going to install etcd on each of the Docker host and form a three-node etcd cluster for better availability.

1. Install etcd packages:

   $ yum install etcd

2. Modify the configuration file accordingly depending on the Docker hosts:

   $ vim /etc/etcd/etcd.conf

   For docker1 with IP address 192.168.55.111:

   ETCD_NAME=etcd1
   ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
   ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
   ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
   ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.55.111:2380"
   ETCD_INITIAL_CLUSTER="etcd1=http://192.168.55.111:2380,etcd2=http://192.168.55.112:2380,etcd3=http://192.168.55.113:2380"
   ETCD_INITIAL_CLUSTER_STATE="new"
   ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-1"
   ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

   For docker2 with IP address 192.168.55.112:

   ETCD_NAME=etcd2
   ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
   ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
   ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
   ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.55.112:2380"
   ETCD_INITIAL_CLUSTER="etcd1=http://192.168.55.111:2380,etcd2=http://192.168.55.112:2380,etcd3=http://192.168.55.113:2380"
   ETCD_INITIAL_CLUSTER_STATE="new"
   ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-1"
   ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

   For docker3 with IP address 192.168.55.113:

   ETCD_NAME=etcd3
   ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
   ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
   ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
   ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.55.113:2380"
   ETCD_INITIAL_CLUSTER="etcd1=http://192.168.55.111:2380,etcd2=http://192.168.55.112:2380,etcd3=http://192.168.55.113:2380"
   ETCD_INITIAL_CLUSTER_STATE="new"
   ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-1"
   ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

3. Start the service on docker1, followed by docker2 and docker3:

   $ systemctl start etcd

4. Verify our cluster status:

   [docker3 ]$ etcdctl cluster-health
   member 2f8ec0a21c11c189 is healthy: got healthy result from http://0.0.0.0:2379
   member 589a7883a7ee56ec is healthy: got healthy result from http://0.0.0.0:2379
   member fcacfa3f23575abe is healthy: got healthy result from http://0.0.0.0:2379
   cluster is healthy

That’s it. Our etcd is now running as a cluster on three nodes. Our setup now looks like this:

Calico Installation

Ensure etcd is installed as mentioned in the “Key-Value Store (etcd)” section. The following commands should be performed on each Docker host unless specified otherwise:

1. Download Calico and make it executable:

   $ wget http://www.projectcalico.org/builds/calicoctl -P /usr/local/bin
   $ chmod +x /usr/local/bin/calicoctl

2. Create the calico node for docker1. Specify the Docker host’s IP address as below:

   [root@docker1 ~]$ calicoctl node --ip=192.168.55.111 --libnetwork
   Running Docker container with the following command:
   
   docker run -d --restart=always --net=host --privileged --name=calico-node -e HOSTNAME=docker1.local -e IP=192.168.55.111 -e IP6= -e CALICO_NETWORKING_BACKEND=bird -e AS= -e NO_DEFAULT_POOLS= -e ETCD_AUTHORITY=127.0.0.1:2379 -e ETCD_SCHEME=http -v /var/log/calico:/var/log/calico -v /lib/modules:/lib/modules -v /var/run/calico:/var/run/calico calico/node:latest
   
   Calico node is running with id: 37cf85e600e3fc04b16cdad7dc1c9f3bb7e8ace80b5fef4dbc21155d60440a78
   Waiting for successful startup
   Waiting for etcd connection...
   Calico node started successfully
   Calico libnetwork driver is running with id: 4c2c4b3fd5b8155622a656440513680c8da051ed6881a94a33fbdc1e8748c060

3. Same goes to docker2, where the host IP address is 192.168.55.112:

   [root@docker2 ~]$ calicoctl node --ip=192.168.55.112 --libnetwork
   Running Docker container with the following command:
   
   docker run -d --restart=always --net=host --privileged --name=calico-node -e HOSTNAME=docker2.local -e IP=192.168.55.112 -e IP6= -e CALICO_NETWORKING_BACKEND=bird -e AS= -e NO_DEFAULT_POOLS= -e ETCD_AUTHORITY=127.0.0.1:2379 -e ETCD_SCHEME=http -v /var/log/calico:/var/log/calico -v /lib/modules:/lib/modules -v /var/run/calico:/var/run/calico calico/node:latest
   
   Calico node is running with id: 37cf85e600e3fc04b16cdad7dc1c9f3bb7e8ace80b5fef4dbc21155d60440a78
   Waiting for successful startup
   Waiting for etcd connection...
   Calico node started successfully
   Calico libnetwork driver is running with id: 4c2c4b3fd5b8155622a656440513680c8da051ed6881a94a33fbdc1e8748c060

4. Then, on docker3:

   [root@docker3 ~]$ calicoctl node --ip=192.168.55.113 --libnetwork
   Running Docker container with the following command:
   
   docker run -d --restart=always --net=host --privileged --name=calico-node -e HOSTNAME=docker3.local -e IP=192.168.55.113 -e IP6= -e CALICO_NETWORKING_BACKEND=bird -e AS= -e NO_DEFAULT_POOLS= -e ETCD_AUTHORITY=127.0.0.1:2379 -e ETCD_SCHEME=http -v /var/log/calico:/var/log/calico -v /lib/modules:/lib/modules -v /var/run/calico:/var/run/calico calico/node:latest
   
   Calico node is running with id: 37cf85e600e3fc04b16cdad7dc1c9f3bb7e8ace80b5fef4dbc21155d60440a78
   Waiting for successful startup
   Waiting for etcd connection...
   Calico node started successfully
   Calico libnetwork driver is running with id: 4c2c4b3fd5b8155622a656440513680c8da051ed6881a94a33fbdc1e8748c060

   A calico-node service (“calico/node:latest”) runs on each Docker host which handles all of the necessary IP routing, installation of policy rules, and distribution of routes across the cluster of nodes.

5. Verify the Calico status:

   [root@docker1 ~]$ calicoctl node show
   +---------------+----------------+-----------+-------------------+--------------+--------------+
   |    Hostname   |   Bird IPv4    | Bird IPv6 |       AS Num      | BGP Peers v4 | BGP Peers v6 |
   +---------------+----------------+-----------+-------------------+--------------+--------------+
   | docker1.local | 192.168.55.111 |           | 64511 (inherited) |              |              |
   | docker2.local | 192.168.55.112 |           | 64511 (inherited) |              |              |
   | docker3.local | 192.168.55.113 |           | 64511 (inherited) |              |              |
   +---------------+----------------+-----------+-------------------+--------------+--------------+

6. Configure an IP pool for our Calico network:

   [root@docker1 ~]$ calicoctl pool add 192.168.0.0/16

   Verify the address pool is there:

   [root@docker1 ~]$ calicoctl pool show --ipv4
   +----------------+---------+
   |   IPv4 CIDR    | Options |
   +----------------+---------+
   | 192.168.0.0/16 |         |
   +----------------+---------+

7. Create a new profile so we can group all MySQL Replication containers under the same roof:

   [root@docker1 ~]$ calicoctl profile add mysql-replication
   Created profile mysql-replication

With the profile created, we can illustrate our architecture as per below:

Deploying Multi-Host MySQL Replication Containers with Calico

Calico can be configured without having to use the Docker networking commands. Rather than have Docker configure the network, we are going use the “calicoctl” command line tool to add a container into a Calico network - this adds the required interface and routes into the container, and configures Calico with the correct endpoint information.

Calico use profiles to manage container isolation. You can create profiles and append containers with Calico network into different profiles. Only containers in the same profile are able to talk to each other. Containers from different profiles cannot access each other even though they are in the same CIDR subnet. As shown in step #7 in the previous section, we are going to append all containers under the same profile called mysql-replication.

Host docker1 will be running mysql-master container, while mysql-slave1 and mysql-slave2 containers on host docker2 and docker3 respectively.

MySQL Master

The following commands should be performed on docker1.

1. Firstly, create a directory to be used by the master:

   [root@docker1 ~]$ mkdir -p /opt/Docker/mysql-master/data

2. Create the MySQL master container:

   [root@docker1 ~]$ docker run -d \
   --name mysql-master \
   -v /opt/Docker/mysql-master/data:/var/lib/mysql \
   -e MYSQL_ROOT_PASSWORD=mypassword \
    mysql \
   --gtid_mode=ON \
   --log-bin \
   --log-slave-updates \
   --enforce-gtid-consistency \
   --server-id=1

3. Create the slave user to be used by our slaves:

   [root@docker1 ~]$ docker exec -ti mysql-master 'mysql' -uroot -pmypassword -vvv -e "GRANT REPLICATION SLAVE ON *.* TO repl@'%' IDENTIFIED BY 'slavepass'

4. Add Calico’s network interface into the container and assign an IP address in the range of Calico pool:

   [root@docker1 ~]$ calicoctl container add mysql-master 192.168.0.50
   IP 192.168.0.50 added to mysql-master

5. Add the container into the profile:

   [root@docker1 ~]$ calicoctl container mysql-master profile append mysql-replication
   Profile(s) mysql-replication appended.

When each container is added to Calico, an "endpoint" is registered for each container's interface. Containers are only allowed to communicate with one another when both of their endpoints are assigned the same profile.

MySQL Slave #1

The following commands should be performed on docker2.

1. Create a directory to be used by the slave1:

   [root@docker2 ~]$ mkdir -p /opt/Docker/mysql-slave1/data

2. Create the MySQL slave1 container:

   [root@docker2 ~]$ docker run -d \
   --name mysql-slave1 \
   -v /opt/Docker/mysql-slave1/data:/var/lib/mysql \
   -e MYSQL_ROOT_PASSWORD=mypassword \
   mysql \
   --gtid_mode=ON \
   --log-bin \
   --log-slave-updates \
   --enforce-gtid-consistency \
   --server-id=101

3. Add Calico’s network interface into the container and assign an IP address in the range of Calico pool:

   [root@docker2 ~]$ calicoctl container add mysql-slave1 192.168.0.51
   IP 192.168.0.51 added to mysql-slave1

4. Add the container into the profile:

   [root@docker2 ~]$ calicoctl container mysql-slave1 profile append mysql-replication
   Profile(s) mysql-replication appended.

5. Change the master and supply the slave credentials as created in mysql-master:

   [root@docker2 ~]$ docker exec -ti mysql-slave1 'mysql' -uroot -pmypassword -e 'CHANGE MASTER TO master_host="192.168.0.50", master_user="repl", master_password="slavepass", master_auto_position=1;' -vvv

6. Start the replication:

   [root@docker2 ~]$ docker exec -ti mysql-slave1 'mysql' -uroot -pmypassword -e "START SLAVE;" -vvv

7. Verify if mysql-slave1 is running:

   [root@docker2 ~]$ docker exec -ti mysql-slave1 'mysql' -uroot -pmypassword -e "SHOW SLAVE STATUS \G"
   ...
                Slave_IO_Running: Yes
               Slave_SQL_Running: Yes
   ...

MySQL Slave #2

The following commands should be performed on docker3.

1. Create a directory to be used by the slave2:

   [root@docker3 ~]$ mkdir -p /opt/Docker/mysql-slave2/data

2. Create the MySQL slave2 container:

   [root@docker3 ~]$ docker run -d \
   --name mysql-slave2 \
   -v /opt/Docker/mysql-slave2/data:/var/lib/mysql \
   -e MYSQL_ROOT_PASSWORD=mypassword \
   mysql \
   --gtid_mode=ON \
   --log-bin \
   --log-slave-updates \
   --enforce-gtid-consistency \
   --server-id=102

3. Add Calico’s network interface into the container and assign an IP address in the range of Calico pool:

   [root@docker3 ~]$ calicoctl container add mysql-slave2 192.168.0.52
   IP 192.168.0.52 added to mysql-slave2

4. Add the container into the profile:

   [root@docker3 ~]$ calicoctl container mysql-slave2 profile append mysql-replication
   Profile(s) mysql-replication appended.

5. Change the master and supply the slave credentials as created in mysql-master:

   [root@docker3 ~]$ docker exec -ti mysql-slave2 'mysql' -uroot -pmypassword -e 'CHANGE MASTER TO master_host="192.168.0.50", master_user="repl", master_password="slavepass", master_auto_position=1;'

6. Start the replication:

   [root@docker3 ~]$ docker exec -ti mysql-slave2 'mysql' -uroot -pmypassword -e "START SLAVE;"

7. Verify if mysql-slave2 is running:

   [root@docker3 ~]$ docker exec -ti mysql-slave2 'mysql' -uroot -pmypassword -e "SHOW SLAVE STATUS \G"
   ...
                Slave_IO_Running: Yes
               Slave_SQL_Running: Yes
   ...

   We can get a summary of all endpoints created by Calico with “--detailed” flag:

   [root@docker1 ~]$ calicoctl endpoint show --detailed
   +---------------+-----------------+------------------------------------------------------------------+----------------------------------+-----------------+-------------------+-------------------+--------+
   |    Hostname   | Orchestrator ID |                           Workload ID                            |           Endpoint ID            |    Addresses    |        MAC        |      Profiles     | State  |
   +---------------+-----------------+------------------------------------------------------------------+----------------------------------+-----------------+-------------------+-------------------+--------+
   | docker1.local |      docker     | 89d2ef40918100037e250911e782f71129dd38d7253df274c70d4a31b281de0f | bb3b9fb4870e11e6a88a000c29d498bb | 192.168.0.50/32 | 6a:d1:42:30:05:9a | mysql-replication | active |
   | docker2.local |      docker     | ef379f4d46f957165513f86e7859613be9971e82364dc81f1641fd4faae1ec5d | 96aef76c870f11e6b31a000c2903c574 | 192.168.0.51/32 | 3a:d5:08:0c:d9:4a | mysql-replication | active |
   | docker3.local |      docker     | 61e9f51a6fb5b36a30e35d2779e1064267604fad0ee28562567cf166a2d90727 | 387c06c0886411e6b2f1000c29bb2913 | 192.168.0.52/32 | da:18:9f:42:68:9c | mysql-replication | active |
   +---------------+-----------------+------------------------------------------------------------------+----------------------------------+-----------------+-------------------+-------------------+--------+

Our architecture is now looking like this:

Exposing to the Public

Now we are ready to expose our MySQL Replication to the outside world. To do this, add an inbound rule to port 3306:

[root@docker1 ~]$ calicoctl profile mysql-replication rule add inbound allow tcp to ports 3306

Verify the profile’s inbound and outbound rules:

[root@docker1 ~]$ calicoctl profile mysql-replication rule show
Inbound rules:
   1 allow from tag mysql-replication
   2 allow tcp to ports 3306
Outbound rules:
   1 allow

Then, on each of the Docker host, add the DNAT iptables rules respectively. Consider the public IP is listening on interface eth0 of the Docker host.

Docker1 (mysql-master):

[root@docker1 ~]$ iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 3306 -j DNAT --to 192.168.0.50:3306
[root@docker1 ~]$ iptables -A OUTPUT -t nat -p tcp -o lo --dport 3306 -j DNAT --to-destination 192.168.0.50:3306

Docker2 (mysql-slave1):

[root@docker2 ~]$ iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 3306 -j DNAT --to 192.168.0.51:3306
[root@docker2 ~]$ iptables -A OUTPUT -t nat -p tcp -o lo --dport 3306 -j DNAT --to-destination 192.168.0.51:3306

Docker3 (mysql-slave2):

[root@docker3 ~]$ iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 3306 -j DNAT --to 192.168.0.52:3306
[root@docker3 ~]$ iptables -A OUTPUT -t nat -p tcp -o lo --dport 3306 -j DNAT --to-destination 192.168.0.52:3306

Verify that the outside world can reach the MySQL master on docker1:

[host-in-another-galaxy]$ mysql -uroot -pmypassword -h192.168.55.111 -P3306
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 5.7.15-log MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> show master status\G
*************************** 1. row ***************************
             File: 89d2ef409181-bin.000001
         Position: 429
     Binlog_Do_DB:
 Binlog_Ignore_DB:
Executed_Gtid_Set: 4f25482e-870d-11e6-bd5b-0242ac110002:1

With port 3306 exposed, our final architecture can be illustrated as per below:

That sums up our blog post now. Calico is well-known for its performance. If you wonder how good Calico performs in multi-host containers, you can read this blog post written by Percona’s Vadim Tkachenko. In the next blog post, we are going to look into Weave.

Welcome to this week’s Planets9s, covering all the latest resources and technologies we create around automation and management of open source database infrastructures.

	MySQL on Docker: Multi-Host Networking for MySQL Containers (Part 2 - Calico) We’re continuing with our popular blogs covering MySQL on Docker and this time we’re looking at deploying MySQL Replication on top of three Docker hosts via Calico’s driver on multi-host networking. Having previously looked at Docker’s single-host networking for MySQL containers as well as multi-host networking and Docker swarm mode, we’re now casting our eyes on other networking drivers, starting with Calico. Read the blog
Still time to sign up: 9 DevOps Tips for going in production with Galera Cluster We’re live next Tuesday, October 11th, with Johan Andersson, CTO at Severalnines, who will be sharing his 9 DevOps tips with you for a successful Galera Cluster for MySQL / MariaDB production environment. Monitoring, managing schema changes and pushing them in production, performance optimizations, configurations, version upgrades, backups: Johan will guide you through all aspects to consider before going live with Galera Cluster. Sign up for the webinar
	We had plenty of 9s on deck at Percona Live Amsterdam this week If you didn’t get the chance to attend this year’s Percona Live Europe, here’s our recap of the conference with soundbites, live broadcast replays from our booth where we discussed ClusterControl and conference highlights with Severalnines CEO Vinay Joosery … we’re also putting our team in the spotlight, so that you can see some of the faces behind the 9s! Read the conference blog
Sharding MySQL with MySQL Fabric and ProxySQL There are numerous ways to migrate into MySQL Fabric, which can all be pretty challenging. The solution to this complex challenge consists of several elements: 1. MySQL Fabric with its sharding system of high availability groups and tools around it. 2. MySQL Router - it allows regular MySQL clients to connect to different high availability groups created in MySQL Fabric. 3. ProxySQL, which allows users to perform a flawless failover (from the old database to the sharded setup). This blog post and related whitepaper show you how to set up a sharded MySQL setup based on MySQL Fabric and ProxySQL. Read the blog

That’s it for this week! Feel free to share these resources with your colleagues and follow us in our social media channels.

Have a good end of the week,

Jean-Jérôme Schmidt
Planets9s Editor
Severalnines AB

In the previous blog post, we have looked into the multi-host networking capabilities with Docker with native network and Calico. In this blog post, our journey to make Galera Cluster run smoothly on Docker containers continues. Deploying Galera Cluster on Docker is tricky when using orchestration tools. Due to the nature of the scheduler in container orchestration tools and the assumption of homogenous images, the scheduler will just fire the respective containers according to the run command and leave the bootstrapping process to the container’s entrypoint logic when starting up. And you do not want to do that for Galera - starting all nodes at once means each node will form a “1-node cluster” and you’ll end up with a disjointed system.

“Homogeneousing” Galera Cluster

That might be a new word, but it holds true for stateful services like MySQL Replication and Galera Cluster. As one might know, the bootstrapping process for Galera Cluster usually requires manual intervention, where you usually have to decide which node is the most advanced node to start bootstrapping from. There is nothing wrong with this step, you need to be aware of the state of each database node before deciding on the sequence of how to start them up. Galera Cluster is a distributed system, and its redundancy model works like that.

However, container orchestration tools like Docker Engine Swarm Mode and Kubernetes are not aware of the redundancy model of Galera. The orchestration tool presumes containers are independent from each other. If they are dependent, then you have to have an external service that monitors the state. The best way to achieve this is to use a key/value store as a reference point for other containers when starting up.

This is where service discovery like etcd comes into the picture. The basic idea is, each node should report its state periodically to the service. This simplifies the decision process when starting up. For Galera Cluster, the node that has wsrep_local_state_comment equal to Synced shall be used as a reference node when constructing the Galera communication address (gcomm) during joining. Otherwise, the most updated node has to get bootstrapped first.

Etcd has a very nice feature called TTL, where you can expire a key after a certain amount of time. This is useful to determine the state of a node, where the key/value entry only exists if an alive node reports to it. As a result, the node won’t have to connect to each other to determine state (which is very troublesome in a dynamic environment) when forming a cluster. For example, consider the following keys:

    {"createdIndex": 10074,"expiration": "2016-11-29T10:55:35.218496083Z","key": "/galera/my_wsrep_cluster/10.255.0.7/wsrep_last_committed","modifiedIndex": 10074,"ttl": 10,"value": "2881"
    },
    {"createdIndex": 10072,"expiration": "2016-11-29T10:55:34.650574629Z","key": "/galera/my_wsrep_cluster/10.255.0.7/wsrep_local_state_comment","modifiedIndex": 10072,"ttl": 10,"value": "Synced"
    }

After 10 seconds (ttl value), those keys will be removed from the entry. Basically, all nodes should report to etcd periodically with an expiring key. Container should report every N seconds when it's alive (wsrep_cluster_state_comment=Synced and wsrep_last_committed=#value) via a background process. If a container is down, it will no longer send the update to etcd, thus the keys are removed after expiration. This simply indicates that the node was registered but is no longer synced with the cluster. It will be skipped when constructing the Galera communication address at a later point.

The overall flow of joining procedure is illustrated in the following flow chart:

We have built a Docker image that follows the above. It is specifically built for running Galera Cluster using Docker’s orchestration tool. It is available at Docker Hub and our Github repository. It requires an etcd cluster as the discovery service (supports multiple etcd hosts) and based on Percona XtraDB Cluster 5.6. The image includes Percona Xtrabackup, jq (JSON processor) and also a shell script tailored for Galera health check called report_status.sh.

You are welcome to fork or contribute to the project. Any bugs can be reported via Github or via our support page.

Deploying etcd Cluster

etcd is a distributed key value store that provides a simple and efficient way to store data across a cluster of machines. It’s open-source and available on GitHub. It provides shared configuration and service discovery. A simple use-case is to store database connection details or feature flags in etcd as key value pairs. It gracefully handles leader elections during network partitions and will tolerate machine failures, including the leader.

Since etcd is the brain of the setup, we are going to deploy it as a cluster daemon, on three nodes, instead of using containers. In this example, we are going to install etcd on each of the Docker hosts and form a three-node etcd cluster for better availability.

We used CentOS 7 as the operating system, with Docker v1.12.3, build 6b644ec. The deployment steps in this blog post are basically similar to the one used in our previous blog post.

1. Install etcd packages:

   $ yum install etcd

2. Modify the configuration file accordingly depending on the Docker hosts:

   $ vim /etc/etcd/etcd.conf

   For docker1 with IP address 192.168.55.111:

   ETCD_NAME=etcd1
   ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
   ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
   ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
   ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.55.111:2380"
   ETCD_INITIAL_CLUSTER="etcd1=http://192.168.55.111:2380,etcd2=http://192.168.55.112:2380,etcd3=http://192.168.55.113:2380"
   ETCD_INITIAL_CLUSTER_STATE="new"
   ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-1"
   ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

   For docker2 with IP address 192.168.55.112:

   ETCD_NAME=etcd2
   ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
   ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
   ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
   ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.55.112:2380"
   ETCD_INITIAL_CLUSTER="etcd1=http://192.168.55.111:2380,etcd2=http://192.168.55.112:2380,etcd3=http://192.168.55.113:2380"
   ETCD_INITIAL_CLUSTER_STATE="new"
   ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-1"
   ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

   For docker3 with IP address 192.168.55.113:

   ETCD_NAME=etcd3
   ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
   ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
   ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
   ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.55.113:2380"
   ETCD_INITIAL_CLUSTER="etcd1=http://192.168.55.111:2380,etcd2=http://192.168.55.112:2380,etcd3=http://192.168.55.113:2380"
   ETCD_INITIAL_CLUSTER_STATE="new"
   ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-1"
   ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

3. Start the service on docker1, followed by docker2 and docker3:

   $ systemctl enable etcd
   $ systemctl start etcd

4. Verify our cluster status using etcdctl:

   [docker3 ]$ etcdctl cluster-health
   member 2f8ec0a21c11c189 is healthy: got healthy result from http://0.0.0.0:2379
   member 589a7883a7ee56ec is healthy: got healthy result from http://0.0.0.0:2379
   member fcacfa3f23575abe is healthy: got healthy result from http://0.0.0.0:2379
   cluster is healthy

That’s it. Our etcd is now running as a cluster on three nodes. The below illustrates our architecture:

Deploying Galera Cluster

Minimum of 3 containers is recommended for high availability setup. Thus, we are going to create 3 replicas to start with, it can be scaled up and down afterwards. Running standalone is also possible with standard "docker run" command as shown further down.

Before we start, it’s a good idea to remove any sort of keys related to our cluster name in etcd:

$ etcdctl rm /galera/my_wsrep_cluster --recursive

Ephemeral Storage

This is a recommended way if you plan on scaling the cluster out on more nodes (or scale back by removing nodes). To create a three-node Galera Cluster with ephemeral storage (MySQL datadir will be lost if the container is removed), you can use the following command:

$ docker service create \
--name mysql-galera \
--replicas 3 \
-p 3306:3306 \
--network galera-net \
--env MYSQL_ROOT_PASSWORD=mypassword \
--env DISCOVERY_SERVICE=192.168.55.111:2379,192.168.55.112:2379,192.168.55.113:2379 \
--env XTRABACKUP_PASSWORD=mypassword \
--env CLUSTER_NAME=my_wsrep_cluster \
severalnines/pxc56

Persistent Storage

To create a three-node Galera Cluster with persistent storage (MySQL datadir persists if the container is removed), add the mount option with type=volume:

$ docker service create \
--name mysql-galera \
--replicas 3 \
-p 3306:3306 \
--network galera-net \
--mount type=volume,source=galera-vol,destination=/var/lib/mysql \
--env MYSQL_ROOT_PASSWORD=mypassword \
--env DISCOVERY_SERVICE=192.168.55.111:2379,192.168.55.112:2379,192.168.55.113:2379 \
--env XTRABACKUP_PASSWORD=mypassword \
--env CLUSTER_NAME=my_wsrep_cluster \
severalnines/pxc56

Custom my.cnf

If you would like to include a customized MySQL configuration file, create a directory on the physical host beforehand:

$ mkdir /mnt/docker/mysql-config # repeat on all Docker hosts

Then, use the mount option with “type=bind” to map the path into the container. In the following example, the custom my.cnf is located at /mnt/docker/mysql-config/my-custom.cnf on each Docker host:

$ docker service create \
--name mysql-galera \
--replicas 3 \
-p 3306:3306 \
--network galera-net \
--mount type=volume,source=galera-vol,destination=/var/lib/mysql \
--mount type=bind,src=/mnt/docker/mysql-config,dst=/etc/my.cnf.d \
--env MYSQL_ROOT_PASSWORD=mypassword \
--env DISCOVERY_SERVICE=192.168.55.111:2379,192.168.55.112:2379,192.168.55.113:2379 \
--env XTRABACKUP_PASSWORD=mypassword \
--env CLUSTER_NAME=my_wsrep_cluster \
severalnines/pxc56

Wait for a couple of minutes and verify the service is running (CURRENT STATE = Running):

$ docker service ls mysql-galera
ID                         NAME            IMAGE               NODE           DESIRED STATE  CURRENT STATE           ERROR
2vw40cavru9w4crr4d2fg83j4  mysql-galera.1  severalnines/pxc56  docker1.local  Running        Running 5 minutes ago
1cw6jeyb966326xu68lsjqoe1  mysql-galera.2  severalnines/pxc56  docker3.local  Running        Running 12 seconds ago
753x1edjlspqxmte96f7pzxs1  mysql-galera.3  severalnines/pxc56  docker2.local  Running        Running 5 seconds ago

External applications/clients can connect to any Docker host IP address or hostname on port 3306, requests will be load balanced between the Galera containers. The connection gets NATed to a Virtual IP address for each service "task" (container, in this case) using the Linux kernel's built-in load balancing functionality, IPVS. If the application containers reside in the same overlay network (galera-net), then use the assigned virtual IP address instead. You can retrieve it using the inspect option:

$ docker service inspect mysql-galera -f "{{ .Endpoint.VirtualIPs }}"
[{89n5idmdcswqqha7wcswbn6pw 10.255.0.2/16} {1ufbr56pyhhbkbgtgsfy9xkww 10.0.0.2/24}]

Our architecture is now looking like this:

As a side note, you can also run Galera in standalone mode. This is probably useful for testing purposes like backup and restore, testing the impact of queries and so on. To run it just like a standalone MySQL container, use the standard docker run command:

$ docker run -d \
-p 3306 \
--name=galera-single \
-e MYSQL_ROOT_PASSWORD=mypassword \
-e DISCOVERY_SERVICE=192.168.55.111:2379,192.168.55.112:2379,192.168.55.113:2379 \
-e CLUSTER_NAME=my_wsrep_cluster \
-e XTRABACKUP_PASSWORD=mypassword \
severalnines/pxc56

Scaling the Cluster

There are two ways you can do scaling:

1. Use “docker service scale” command.
2. Create a new service with same CLUSTER_NAME using “docker service create” command.

Docker’s “scale” Command

The scale command enables you to scale one or more services either up or down to the desired number of replicas. The command will return immediately, but the actual scaling of the service may take some time. Galera needs to be run an odd number of nodes to avoid network partitioning.

So a good number to scale to would be 5 and so on:

$ docker service scale mysql-galera=5

Wait for a couple of minutes to let the new containers reach the desired state. Then, verify the running service:

$ docker service ls
ID            NAME          REPLICAS  IMAGE               COMMAND
bwvwjg248i9u  mysql-galera  5/5       severalnines/pxc56

One drawback of using this method is that you have to use ephemeral storage because Docker will likely schedule the new containers on a Docker host that already has a Galera container running. If this happens, the volume will overlap the existing Galera containers’ volume. If you would like to use persistent storage and scale in Docker Swarm mode, you should create another new service with a couple of different options, as described in the next section.

At this point, our architecture looks like this:

Another Service with Same Cluster Name

Another way to scale is to create another service with the same CLUSTER_NAME and network. However, you can’t really use the exact same command as the first one due to the following reasons:

• The service name should be unique.
• The port mapping must be other than 3306, since this port has been assigned to the mysql-galera service.
• The volume name should be different to distinguish them from the existing Galera containers.

A benefit of doing this is you will got another virtual IP address assigned to the “scaled” service. This allows you to have an additional option for your application or client to connect to the “scaled” IP address for various tasks, e.g. perform a full backup in desync mode, database consistency check or server auditing.

The following example shows the command to add two more nodes to the cluster in a new service called mysql-galera-scale:

$ docker service create \
--name mysql-galera-scale \
--replicas 2 \
-p 3307:3306 \
--network galera-net \
--mount type=volume,source=galera-scale-vol,destination=/var/lib/mysql \
--env MYSQL_ROOT_PASSWORD=mypassword \
--env DISCOVERY_SERVICE=192.168.55.111:2379,192.168.55.112:2379,192.168.55.113:2379 \
--env XTRABACKUP_PASSWORD=mypassword \
--env CLUSTER_NAME=my_wsrep_cluster \
severalnines/pxc56

If we look into the service list, here is what we see:

$ docker service ls
ID            NAME                REPLICAS  IMAGE               COMMAND
0ii5bedv15dh  mysql-galera-scale  2/2       severalnines/pxc56
71pyjdhfg9js  mysql-galera        3/3       severalnines/pxc56

And when you look into the cluster size on one of the container, you should get 5:

[root@docker1 ~]# docker exec -it $(docker ps | grep mysql-galera | awk {'print $1'}) mysql -uroot -pmypassword -e 'show status like "wsrep_cluster_size"'
Warning: Using a password on the command line interface can be insecure.
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 5     |
+--------------------+-------+

At this point, our architecture looks like this:

To get a clearer view of the process, we can simply look at the MySQL error log file (located under Docker’s data volume) on one of the running containers, for example:

$ tail -f /var/lib/docker/volumes/galera-vol/_data/error.log

Scale Down

Scaling down is simple. Just reduce the number of replicas or remove the service that holds the minority number of containers to ensure that Galera is still in quorum. For example, if you have fired two groups of nodes with 3 + 2 containers and reach total of 5, the majority need to survive thus you can only remove the second group with 2 containers. If you have three groups with 3 + 2 + 2 containers, you can lose a maximum of 3 containers. This is due to the fact that the Docker Swarm scheduler simply terminates and removes the containers corresponding to the service. This makes Galera think that there are nodes failing, as they are not shut down in a graceful way.

If you scaled up using “docker service scale” command, you should scale down using the same method by reducing the number of replicas. To scale it down, simply do:

$ docker service scale mysql-galera=3

Otherwise, if you chose to create another service to scale up, then simply remove the respective service to scale down:

$ docker service rm mysql-galera-scale

Known Limitations

There will be no automatic recovery if a split-brain happens (where all nodes are in Non-Primary state). This is because the MySQL service is still running, yet it will refuse to serve any data and will return error to the client. Docker has no capability to detect this since what it cares about is the foreground MySQL process which is not terminated, killed or stopped. Automating this process is risky, especially if the service discovery is co-located with the Docker host (etcd would also lose contact with other members). Although if the service discovery is healthy somewhere else, it is probably unreachable from the Galera containers perspective, preventing each other to see the container’s status correctly during the glitch.

In this case, you will need to intervene manually.

Choose the most advanced node to bootstrap and then run the following command to promote the node as Primary (other nodes shall then rejoin automatically if the network recovers):

$ docker exec -it [container ID] mysql -uroot -pyoursecret -e 'set global wsrep_provider_option="pc.bootstrap=1"'

Also, there is no automatic cleanup for the discovery service registry. You can remove all entries using either the following command (assuming the CLUSTER_NAME is my_wsrep_cluster):

$ curl http://192.168.55.111:2379/v2/keys/galera/my_wsrep_cluster?recursive=true -XDELETE # or
$ etcdctl rm /galera/my_wsrep_cluster --recursive

Conclusion

This combination of technologies opens a door for a more reliable database setup in the Docker ecosystem. Working with service discovery to store state makes it possible to have stateful containers to achieve a homogeneous setup.

In the next blog post, we are going to look into how to manage Galera Cluster on Docker.

Welcome to this week’s Planets9s, covering all the latest resources and technologies we create around automation and management of open source database infrastructures.

	Watch the replay: how to build a stable MySQL Replication environment Thanks to everyone who participated in this week’s webinar on building production-ready MySQL Replication environments. Krzysztof Książek, Senior Support Engineer at Severalnines, shared his top 9 tips on that topic with sanity checks before migrating into MySQL replication setup, operating system configuration, replication, backup, provisioning, performance, schema changes, reporting and disaster recovery. If you'd like to learn how to build a stable environment with MySQL replication, then watch this webinar replay. Watch the replay
MySQL on Docker: Deploy a Homogeneous Galera Cluster with etcd As you might know, we’ve been on a journey in the past months exploring how to make Galera Cluster run smoothly on Docker containers and that journey continues. Deploying Galera Cluster on Docker is tricky when using orchestration tools. Due to the nature of the scheduler in container orchestration tools and the assumption of homogenous images, the scheduler will just fire the respective containers according to the run command and leave the bootstrapping process to the container’s entrypoint logic when starting up. And you do not want to do that for Galera … This blog post discusses why and provides insight into how to deploy a homogeneous Galera Cluster with etcd. Read the blog
	About cloud lock-in and open source databases If you haven’t read it yet, do check out the editorial our CEO Vinay Joosery recently published on the importance of avoiding cloud database lock-in. The cloud is no longer a question of if, but of when. Many IT leaders, however, find that one consistent barrier to their adoption of the cloud is vendor lock-in. What do you do when you are forced to stay with a provider that no longer meets your needs? And is cloud lock-in a problem? Find out what Vinay’s thoughts are on these topics. Read the editorial

That’s it for this week! Feel free to share these resources with your colleagues and follow us in our social media channels.

Have a good end of the week,

Jean-Jérôme Schmidt
Planets9s Editor
Severalnines AB

Welcome to this week’s Planets9s, which is also the last edition of 2016!

Throughout the year, we’ve covered here all of the latest resources and technologies we create around automation and management of open source database infrastructures for MySQL, MongoDB and PostgreSQL. Thank you for following us and for your feedback on these resources. We look forward to continuing to interact with you all in 2017 and will strive to publish more information that’s helpful and useful in the new year.

But for now, and in no particular order, here are this year’s top 9 most popular s9s resources:

	MySQL on Docker blog series - Part 1 In the first blog post of this series, we cover some basics around running MySQL in a container. The term ‘Docker’ as the container platform is used throughout the series, which also covers topics such as Docker Swarm Mode and Multi-Host Networking and more. Read the blog(s)
MySQL Load Balancing with HAProxy - Tutorial This tutorial walks you through how to deploy, configure and manage MySQL load balancing with HAProxy using ClusterControl. Read the tutorial
	MySQL Replication for High Availability - Tutorial Learn about a smarter Replication setup that uses a combination of advanced replication techniques including mixed binary replication logging, auto-increment offset seeding, semi-sync replication, automatic fail-over/resynchronization and one-click addition of read slaves. Read the tutorial
The Holy Grail Webinar: Become a MySQL DBA - Database Performance Tuning Our most popular webinar this year discusses some of the settings that are most often tweaked and which can bring you significant improvement in the performance of your MySQL database. Performance tuning is not easy, but you can go a surprisingly long way with a few basic guidelines. Watch the replay
	Become a ClusterControl DBA - Blog Series Follow our colleague Art van Scheppingen, Senior Support Engineer, as he covers all the basic operations of ClusterControl for MySQL, MongoDB & PostgreSQL with examples on how to do this and make most of your setup, including a deep dive per subject to save you time. Read the series
Top 9 Tips for building a stable MySQL Replication environment - Webinar Replay Get all the tips & tricks needed to build a stable environment using MySQL replication, as shared by Krzysztof Ksiazek, Senior Support Engineer at Severalnines. Watch the replay
	Top 9 DevOps Tips for Going in Production with Galera Cluster for MySQL / MariaDB Galera Cluster for MySQL / MariaDB is easy to deploy, but how does it behave under real workload, scale, and during long term operation? Find out more with this popular webinar. Watch the replay
Migrating to MySQL 5.7 - The Database Upgrade Guide In this whitepaper, we look at the important new changes in MySQL 5.7 and show you how to plan the test process and do a live system upgrade without downtime. For those who want to avoid connection failures during slave restarts and switchover, this document goes even further and shows you how to leverage ProxySQL to achieve a graceful upgrade process. Download the whitepaper
	Become a MongoDB DBA - Blog Series If you are a MySQL DBA you may ask yourself: “why would I install MongoDB”? This blog series provides you an excellent starting point to get yourself prepared for MongoDB. From deployment, monitoring & trending, through to scale reading and sharding, we’ve got you covered. Read the series

As always, feel free to share these resources with your colleagues and follow us in our social media channels.

And that’s it for this year! Here’s to happy clustering in 2017!

“See” you all then,

Jean-Jérôme Schmidt
Planets9s Editor
Severalnines AB

Note: This post updated on 28th April 2015 with updated installation steps.

Kubernetes is a system for managing containerized applications in a clustered environment. It provides basic mechanisms for deployment, maintenance and scaling of applications on public, private or hybrid setups. It also comes with self-healing features where containers can be auto provisioned, restarted or even replicated. 

Kubernetes is still at an early stage, please expect design and API changes over the coming year. In this blog post, we’ll show you how to install a Kubernetes cluster with three minions on CentOS 7, with an example on how to manage pods and services. 

Kubernetes Components

Kubernetes works in server-client setup, where it has a master providing centralized control for a number of minions. We will be deploying a Kubernetes master with three minions, as illustrated in the diagram further below.

Kubernetes has several components:

• etcd - A highly available key-value store for shared configuration and service discovery.
• flannel - An etcd backed network fabric for containers.
• kube-apiserver - Provides the API for Kubernetes orchestration.
• kube-controller-manager - Enforces Kubernetes services.
• kube-scheduler - Schedules containers on hosts.
• kubelet - Processes a container manifest so the containers are launched according to how they are described.
• kube-proxy - Provides network proxy services.

Deployment on CentOS 7

We will need 4 servers, running on CentOS 7.1 64 bit with minimal install. All components are available directly from the CentOS extras repository which is enabled by default. The following architecture diagram illustrates where the Kubernetes components should reside:

Prerequisites

1. Disable iptables on each node to avoid conflicts with Docker iptables rules:

$ systemctl stop firewalld
$ systemctl disable firewalld

2. Install NTP and make sure it is enabled and running:

$ yum -y install ntp
$ systemctl start ntpd
$ systemctl enable ntpd

Setting up the Kubernetes Master

The following steps should be performed on the master.

1. Install etcd and Kubernetes through yum:

$ yum -y install etcd kubernetes

2. Configure etcd to listen to all IP addresses inside /etc/etcd/etcd.conf. Ensure the following lines are uncommented, and assign the following values:

ETCD_NAME=default
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"

3. Configure Kubernetes API server inside /etc/kubernetes/apiserver. Ensure the following lines are uncommented, and assign the following values:

KUBE_API_ADDRESS="--address=0.0.0.0"
KUBE_API_PORT="--port=8080"
KUBELET_PORT="--kubelet_port=10250"
KUBE_ETCD_SERVERS="--etcd_servers=http://127.0.0.1:2379"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
KUBE_ADMISSION_CONTROL="--admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"
KUBE_API_ARGS=""

4. Start and enable etcd, kube-apiserver, kube-controller-manager and kube-scheduler:

$ for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler; do
    systemctl restart $SERVICES
    systemctl enable $SERVICES
    systemctl status $SERVICES
done

5. Define flannel network configuration in etcd. This configuration will be pulled by flannel service on minions:

$ etcdctl mk /atomic.io/network/config '{"Network":"172.17.0.0/16"}'

6. At this point, we should notice that nodes' status returns nothing because we haven’t started any of them yet:

$ kubectl get nodes
NAME             LABELS              STATUS

Setting up Kubernetes Minions (Nodes)

The following steps should be performed on minion1, minion2 and minion3 unless specified otherwise.

1. Install flannel and Kubernetes using yum:

$ yum -y install flannel kubernetes

2. Configure etcd server for flannel service. Update the following line inside /etc/sysconfig/flanneld to connect to the respective master:

FLANNEL_ETCD="http://192.168.50.130:2379"

3. Configure Kubernetes default config at /etc/kubernetes/config, ensure you update the KUBE_MASTER value to connect to the Kubernetes master API server:

KUBE_MASTER="--master=http://192.168.50.130:8080"

4. Configure kubelet service inside /etc/kubernetes/kubelet as below:
minion1:

KUBELET_ADDRESS="--address=0.0.0.0"
KUBELET_PORT="--port=10250"
# change the hostname to this host’s IP address
KUBELET_HOSTNAME="--hostname_override=192.168.50.131"
KUBELET_API_SERVER="--api_servers=http://192.168.50.130:8080"
KUBELET_ARGS=""

minion2:

KUBELET_ADDRESS="--address=0.0.0.0"
KUBELET_PORT="--port=10250"
# change the hostname to this host’s IP address
KUBELET_HOSTNAME="--hostname_override=192.168.50.132"
KUBELET_API_SERVER="--api_servers=http://192.168.50.130:8080"
KUBELET_ARGS=""

minion3:

KUBELET_ADDRESS="--address=0.0.0.0"
KUBELET_PORT="--port=10250"
# change the hostname to this host’s IP address
KUBELET_HOSTNAME="--hostname_override=192.168.50.133"
KUBELET_API_SERVER="--api_servers=http://192.168.50.130:8080"
KUBELET_ARGS=""

5. Start and enable kube-proxy, kubelet, docker and flanneld services:

$ for SERVICES in kube-proxy kubelet docker flanneld; do
    systemctl restart $SERVICES
    systemctl enable $SERVICES
    systemctl status $SERVICES
done

6. On each minion, you should notice that you will have two new interfaces added, docker0 and flannel0. You should get different range of IP addresses on flannel0 interface on each minion, similar to below:
minion1:

$ ip a | grep flannel | grep inet
inet 172.17.45.0/16 scope global flannel0

minion2:

$ ip a | grep flannel | grep inet
inet 172.17.38.0/16 scope global flannel0

minion3:

$ ip a | grep flannel | grep inet
inet 172.17.93.0/16 scope global flannel0

6. Now login to Kubernetes master node and verify the minions’ status:

$ kubectl get nodes
NAME             LABELS                                  STATUS
192.168.50.131   kubernetes.io/hostname=192.168.50.131   Ready
192.168.50.132   kubernetes.io/hostname=192.168.50.132   Ready
192.168.50.133   kubernetes.io/hostname=192.168.50.133   Ready

You are now set. The Kubernetes cluster is now configured and running. We can start to play around with pods.

Creating Pods (Containers)

To create a pod, we need to define a yaml file in the Kubernetes master, and use the kubectl command to create it based on the definition. Create a mysql.yaml file: 

$ mkdir pods
$ cd pods
$ vim mysql.yaml

And add the following lines:

apiVersion: v1
kind: Pod
metadata:
  name: mysql
  labels:
    name: mysql
spec:
  containers:
    - resources:
        limits :
          cpu: 1
      image: mysql
      name: mysql
      env:
        - name: MYSQL_ROOT_PASSWORD
          # change this
          value: yourpassword
      ports:
        - containerPort: 3306
          name: mysql

Create the pod:

$ kubectl create -f mysql.yaml

It may take a short period before the new pod reaches the Running state. Verify the pod is created and running:

$ kubectl get pods
POD       IP            CONTAINER(S)   IMAGE(S)   HOST                            LABELS       STATUS    CREATED
mysql     172.17.38.2   mysql          mysql      192.168.50.132/192.168.50.132   name=mysql   Running   3 hours

So, Kubernetes just created a Docker container on 192.168.50.132. We now need to create a Service that lets other pods access the mysql database on a known port and host.

Creating Service

At this point, we have a MySQL pod inside 192.168.50.132. Define a mysql-service.yaml as below:

apiVersion: v1
kind: Service
metadata:
  labels:
    name: mysql
  name: mysql
spec:
  externalIPs:
    - 192.168.50.132
  ports:
    # the port that this service should serve on
    - port: 3306
  # label keys and values that must match in order to receive traffic for this service
  selector:
    name: mysql

Start the service:

$ kubectl create -f mysql-service.yaml

You should get a 10.254.x.x IP range assigned to the mysql service. This is the Kubernetes internal IP address defined in /etc/kubernetes/apiserver. This IP is not routable outside, so we defined the public IP instead (the interface that connected to external network for that minion):

$ kubectl get services
NAME            LABELS                                    SELECTOR     IP               PORT(S)
kubernetes      component=apiserver,provider=kubernetes   <none>       10.254.0.2       443/TCP
kubernetes-ro   component=apiserver,provider=kubernetes   <none>       10.254.0.1       80/TCP
mysql           name=mysql                                name=mysql   10.254.13.156    3306/TCP
                                                                       192.168.50.132

Let’s connect to our database server from outside (we used MariaDB client on CentOS 7):

$ mysql -uroot -p -h192.168.50.132
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.6.24 MySQL Community Server (GPL)

Copyright (c) 2000, 2014, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> show variables like '%version%';
+-------------------------+------------------------------+
| Variable_name           | Value                        |
+-------------------------+------------------------------+
| innodb_version          | 5.6.24                       |
| protocol_version        | 10                           |
| slave_type_conversions  |                              |
| version                 | 5.6.24                       |
| version_comment         | MySQL Community Server (GPL) |
| version_compile_machine | x86_64                       |
| version_compile_os      | Linux                        |
+-------------------------+------------------------------+
7 rows in set (0.01 sec)

That’s it! You should now be able to connect to the MySQL container that resides on minion2. 

Check out the Kubernetes guestbook example on how to build a simple, multi-tier web application with Redis in master-slave setup. In a follow-up blog post, we are going to play around with Galera cluster containers on Kubernetes. Stay tuned!

References

• Creating a Kubernetes Cluster to Run Docker Formatted Container Images - https://access.redhat.com/articles/1353773
• Kubernetes Github - https://github.com/googlecloudplatform/kubernetes
• Persistent Installation of MySQL and WordPress on Kubernetes - https://github.com/GoogleCloudPlatform/kubernetes/tree/master/examples/mysql-wordpress-pd

The Percona Live Data Performance Conference (for MySQL and MongoDB users and more) is coming up in just a few months and talk submissions have been going strong judging by the social media activity.

As you might have seen from various communications, Percona are asking participants to vote upfront for the tutorials and talks that have been submitted for consideration in the conference programme.

We’ve been busy ourselves with submissions and we’d like to ask you to have a look at the content we submitted. If you like what you see and would like to find out more in Santa Clara, then please vote for your prefered Severalnines talks and/or tutorials below!

Thank you and we look forward to seeing you in April!

Tutorials

Become a MySQL DBA

This hands-on tutorial is intended to help you navigate your way through the steps that lead to becoming a MySQL DBA. We are going to talk about the most important aspects of managing MySQL infrastructure and we will be sharing best practices and tips on how to perform the most common activities.
Vote for this tutorial!

Become a Polyglot Persistence DBA with ClusterControl

This tutorial will cover the four major operational challenges when extending your infrastructure from only MySQL to deploying other storage backends: deployment, management, monitoring and scaling … and how to deal with them using Severalnines’ ClusterControl software. We will cover MySQL, PostgreSQL and MongoDB storage backends and provide a setup using virtuals where you can freely test upon.
Vote for this tutorial!

Talks

Docker and Galera: Stateful in a stateless World!

Docker is becoming more mainstream and adopted by users as a method to package and deploy self-sufficient applications in primarily stateless Linux containers. It's a great toolset on top of OS level virtualization (LXC a.k.a containers) and plays well in the world of micro services. There are a number ways to provide persistent storage in docker containers and in this presentation we will talk about how to setup a persistence data service with docker that can be teared down and brought up across hosts and containers.
Vote for this talk!

Load Balancers for MySQL: an overview and comparison of options

This session aims to give a solid grounding in load balancer technologies for MySQL and MariaDB. We will review the wide variety of open-source options available: from application connectors (php-mysqlnd, jdbc), TCP reverse proxies (HAproxy, Keepalived, Nginx) and SQL-aware load balancers (MaxScale, ProxySQL, MySQL Router), and look at what considerations you should make when assessing their suitability for your environment.
Vote for this talk!

MySQL (NDB) Cluster - Best Practices

In this session we will talk about Core architecture and Design principles of NDB Cluster APIs for data access (SQL and NoSQL interfaces) Important configuration parameters Best practices: indexing and schema We will also compare performance between MySQL Cluster 7.4 and Galera (MySQL 5.6), and how to best make use of the feature set of MySQL Cluster 7.4.
Vote for for this talk!

Performance Analysis and Auto-tuning of MySQL using DBA-Minions

In this session, we will introduce you to a new tool that can be used to develop DBA-Minions. The tool is available freely to the wider MySQL community, and a number of DBA-Minions for analysing database performance are already available to download from GitHub. At the end of the session, attendees will learn how to use these DBA-Minions, modify them, or create new ones using an integrated IDE.
Vote for this talk!

How to automate, monitor and manage your MongoDB servers

The business model of the company behind MongoDB is to sell premium support and administration tools to maintain and monitor MongoDB. As an alternative there are (open source) solutions that could make your life as a DBA easier. In this session, we will go beyond the deployment phase and show you how you can automate tasks, how to monitor a cluster and how to manage MongoDB overall.
Vote for this talk!

Polyglot Persistence for the MySQL & MongoDB DBA

The introduction of DevOps in organisations has changed the development process, and perhaps introduced some challenges. Developers, in addition to their own preferred programming languages, also have their own preference for backend storage. Extending your infrastructure from only MySQL, to deploying other storage backends like MongoDB and PostgreSQL, implies you have to also monitor, manage and scale them. This session, we will show you how!
Vote for this talk!

About the speakers

	Alex Yu, VP Products, Severalnines Alex is the VP of Products at Severalnines, responsible for all product related strategy and operations. Prior to Severalnines, Alex was Master Principal Sales Consultant at MySQL/Sun Microsystems/Oracle in the APAC region, where he worked with some of the region's largest telecoms service providers and network equipment manufactures to build massively scalable database infrastructures. He previously held key development roles in various startups, and was part of the original MySQL Cluster development team at Ericsson Alzato, which MySQL acquired 2003.
	Johan Andersson, CTO, Severalnines Johan is CTO at Severalnines, a company that enables developers to easily deploy, manage, monitor and scale highly-available MySQL clusters in the data center, in hosted environments and on the cloud. Prior to Severalnines, Johan worked at MySQL/Sun/Oracle and was the Principal Consultant and lead of the MySQL Clustering and High Availability consulting group, where he designed and implemented large-scale MySQL systems at key customers.
	Art van Scheppingen, Senior Support Engineer, Severalnines Art is a pragmatic MySQL and Database expert with over 15 years experience in web development. He previously worked at Spil Games as Head of Database Engineering, where he kept a broad vision upon the whole database environment: from MySQL to Couchbase, Vertica to Hadoop and from Sphinx Search to SOLR. He regularly presents his work and projects at various conferences (Percona Live, FOSDEM) and related meetups.
	Ashraf Sharif, System Support Engineer, Severalnines Ashraf is System Support Engineer at Severalnines. He was previously involved in hosting world and LAMP stack, where he worked as principal consultant and head of support team and delivered clustering solutions for large websites in the South East Asia region. His professional interests are on system scalability and high availability.

See you all in Santa Clara!

ClusterControl integrates with a number of popular third-party tools for deployment and monitoring - from Docker images to Puppet modules, Chef cookbooks, Vagrant images, and Nagios/Zabbix/Syslog/PagerDuty plugins. In this blog, we’ll have a quick look at the latest available resources.

Docker

From v1.2.11, ClusterControl does not have to be deployed on the same OS as the database server’s operating system. Previously, ClusterControl on a Redhat system could only monitor database nodes running on a Redhat system, the same goes for Debian/Ubuntu. This is no longer the case.

We have rebuilt our Docker container image under a single operating system, CentOS 6 with latest minor version. The deployment instruction is now way simpler as you can see in the Docker Hub page. The remaining containers are removed from the list. Deploying ClusterControl on Docker is now as simple as:

$ docker pull severalnines/clustercontrol
$ docker run -d --name clustercontrol -p 5000:80 severalnines/clustercontrol

Next, setup SSH keys from ClusterControl to all target containers/nodes and start adding your existing DB infrastructure into ClusterControl. For details, check out the Docker Registry page.

Puppet Module & Chef Cookbooks

Previously, the Puppet module and Chef Cookbooks for ClusterControl were built with “add existing database cluster” in mind. The idea was that you already had a running database cluster deployed using Puppet/Chef, and wanted to manage it using ClusterControl.

We’ve now updated the cookbooks so that you install ClusterControl first and then use it to deploy your databases and clusters.

Note that, once you have deployed ClusterControl, it is also possible from the UI to manage an existing database setup.

We have also streamlined the installation method of these tools to follow our installer script, install-cc (as shown in the Getting Started page). The module/cookbooks will setup /etc/cmon.cnf with minimal configuration, and subsequent cluster configuration files will reside under /etc/cmon.d/ directory. This new version also introduces support for PostgreSQL instances.

Example Puppet host definition for ClusterControl node is now as simple as below:

node "clustercontrol.local" {
   class { 'clustercontrol':
           is_controller => true,
           api_token => 'b7e515255db703c659677a66c4a17952515dbaf5'
   }
}

Same goes to Chef Cookbooks, the simplest databag for ClusterControl node is now:

{"id" : "config","clustercontrol_api_token" : "92ee6e2368df29ae52fba0e2aad2b28240e6908b"
}

For more details, please refer to Puppet Forge or Chef Supermarket.

Vagrant

Our Vagrantfile deploys 4 instances on VirtualBox platform, three for DB nodes plus one for ClusterControl. It installs the latest released ClusterControl version on Centos/Redhat or Debian/Ubuntu based distributions.

Compared to our previous version of Vagrantfile as described in this blog post, we are no longer depending on the custom Vagrant boxes that we have built. The new Vagrantfile will perform the following actions when firing up new instances:

1. Download a base image from HashiCorp.
2. Fire up 4 virtual machines from the downloaded base image.
3. Add port forwarding for ClusterControl UI.
4. Install ClusterControl latest version on 10.10.10.10.
5. Set up SSH key on all virtual machines (ClusterControl + DB nodes).

The default memory allocated for each instance is 768MB. The default MySQL root user password on the ClusterControl host is 'root123'. The default IP address for each virtual machine would be:

• 10.10.10.10 - ClusterControl
• 10.10.10.11 - DB #1
• 10.10.10.12 - DB #2
• 10.10.10.13 - DB #3

You can then use the available DB nodes to create a database cluster or single DB instances.

CentOS

Launches a VirtualBox’s instance for ClusterControl and three instances for DB Nodes, running on CentOS 7.1 base image:

$ git clone https://github.com/severalnines/vagrant
$ vagrant box add bento/centos-7.1
$ cd clustercontrol/centos
$ vagrant up

Ubuntu

Launches a VirtualBox’s instance for ClusterControl and three instances for DB Nodes, running on Ubuntu 14.04 base image:

$ git clone https://github.com/severalnines/vagrant
$ vagrant box add ubuntu/trusty64
$ cd clustercontrol/ubuntu
$ vagrant up

Once the virtual machines are up, setup password-less SSH access from the ClusterControl node to all DB nodes:

$ vagrant ssh vm1

Copy over the root's public ssh key (default password for the vagrant user is 'vagrant'):

[vagrant@n1 ~]$ for h in 10.10.10.11 10.10.10.12 10.10.10.13; do sudo cat /root/.ssh/id_rsa.pub | ssh $h "sudo mkdir -p /root/.ssh && sudo tee -a /root/.ssh/authorized_keys && sudo chmod 700 /root/.ssh && sudo chmod 600 /root/.ssh/authorized_keys"; done;

Open your web browser to http://localhost:8080/clustercontrol and create a ClusterControl default admin user. You can now create your databases and clusters on the available DB nodes by clicking on 'Create Database Cluster' or ‘Create Database Node’. Specify root as SSH user and /root/.ssh/id_rsa as the SSH key in the dialog and you are good to go.

Database Advisors - s9s-advisor-bundle

Advisors in ClusterControl are powerful constructs; they provide specific advice on how to address issues in areas such as performance, security, log management, configuration, storage space, etc. They can be anything from simple configuration advice, warning on thresholds or more complex rules for predictions or cluster-wide automation tasks based on the state of your servers or databases. In general, advisors perform more detailed analysis, and produce more comprehensive recommendations than alerts.

With ClusterControl, we ship a set of basic advisors that are open source. These include rules and alerts on security settings, system checks (NUMA, Disk, CPU), queries, innodb, connections, performance schema, Galera configuration, NDB memory usage, and so on. The advisors can be downloaded from Github. Through ClusterControl Developer Studio, you can create new ones. It is also easy to import the bundles written by our partners or users, or export your own for others to try out.

To build a bundle, navigate to the directory and run create_bundle.sh script:

$ cd /home/user
$ ./create_bundle.sh dirname

This will create a file: dirname.tar.gz. To import into Developer Studio, go to ClusterControl > Manage > Developer Studio > Import. You will be prompted if you want to overwrite the existing files.

For more information, check out the Github repository here. Full documentation on ClusterControl DSL is available at documentation page.

clustercheck-iptables

Are you using a TCP load balancer in front of Galera cluster, and it has limited healthcheck capabilities? This healthcheck script allows your favorite TCP load balancer like F5, nginx, balance, pen and others to correctly distribute database connections to the different nodes in a Galera cluster.
The idea is simple; If the Galera node is healthy, it is reachable on the mirror port (a redirection port to the real port) on the DB nodes. The TCP load balancer will then just need to forward the incoming MySQL connections to any available mirror port on the backend.

The simplest way to get it running on each DB node:

$ git clone https://github.com/ashraf-s9s/clustercheck-iptables
$ cp clustercheck-iptables/mysqlchk_iptables /usr/local/sbin/
$ mysqlchk_iptables -d --user=check --password=checkpassword

Details on this are explained in this Github repository page and this blog post. We also recorded an asciinema screencast on how the script works in action.

ClusterControl Plugins

We have made a number of improvements on the ClusterControl plugins. These now support the new CMON RPC interface, which was made available from ClusterControl v1.2.10. All of the plugins are compatible with 1.2.11 as well.

The following ClusterControl plugins are available under s9s-admin repository:

• nagios - pull database cluster status and alarms from ClusterControl
• pagerduty - push ClusterControl alarms to Pagerduty
• syslog - push ClusterControl alarms to syslog
• zabbix - pull database cluster status, backup status and alarms from ClusterControl

Note that Nagios and Zabbix plugins are only tested on the latest version of the respective products.

You can find a list of active tools on Severalnines’ Tools page. We highly recommend users to keep up-to-date with the latest release of ClusterControl to get the best out of these tools/plugins.

Docker is quickly becoming mainstream, as a method to package and deploy self-sufficient applications in primarily stateless Linux containers. But for a stateful service like a database, this might be bit of a headache. How do we best configure MySQL in a container environment? What can go wrong? Should we even run our databases in a container environment? How does performance compare with e.g. running on virtual machines or bare-metal servers? How do we manage replicated or clustered setups, where multiple containers need to be created, upgraded and made highly available?

So, welcome to our new blog series - “MySQL on Docker”. We will touch upon swarms, shared volumes, data-only-containers, security and configuration management, multi-host networking, service discovery and implications on monitoring when we move from host-centric to role-centric services with shorter life cycles.

In our first blog post, we are going to cover some basics around running MySQL in a container. We are going to use the term ‘Docker’ as the container platform throughout the blog series.

MySQL Docker Containers

Think about a container as a “lightweight virtual machine”. Unlike virtual machines though, containers do not require an entire operating system, all required libraries and the actual application binaries. The same Linux kernel and libraries can be shared between multiple containers running on the host. Docker makes it easy to package Linux software in self-contained images, where all software dependencies are bundled and deployed in a repeatable manner. An image will have exactly the same software installed, whether we run it on a laptop or on a server. The key benefit of Docker is that it allows users to package an application with all of its dependencies into a standardized unit (container). Running many containers allows each one to focus on a specific task; multiple containers then work in concert to implement a distributed system.

The traditional way to run a MySQL database is to install the MySQL packages on a host (bare-metal, virtual machine, cloud instance), and applications would just have to connect to the listening port. Most of the management tasks, for example, configuration tuning, backup, restore, database upgrade, performance tweaking, troubleshooting and so on have to be executed on the database host itself. You would expect to have several ports accessible for connection, for example port TCP 22 for SSH, TCP 3306 for MySQL or UDP 514 for syslog.

In a container, think of MySQL as one single unit that only serve MySQL related stuff on port 3306. Most of the operation should be performed under this single channel. Docker works great in packaging your application/software into one single unit, which you can then deploy anywhere as long as Docker engine is installed. It expects the package, or image to be run as a single process per container. With Docker, the flow would be you (or someone) build a MySQL image using a specific version and vendor, package the image and distribute to anybody who wants to quickly fire a MySQL instance.

Let’s get it running

Let’s familiarize ourselves with a MySQL container running on Docker. We’ll take a ‘break/fix’ approach, so expect to see some errors pop up here and there. We’ll look at the errors and see why they happen. We are going to use the official MySQL image created and maintained by Docker.

To begin with, we must have a host. It can be any type of hosts (physical or virtual) running on Linux, Mac OS X or Windows. Please refer to Docker’s installation guide for details. You can also use docker-machine to provision hosts on a supported cloud provider like DigitalOcean and AWS EC2 Container Service, but we will cover that in another blog post. Here, we are going to use Ubuntu 14.04 as our machine host and use standard command line for deployment and management.

Next, find a container image that you want from the Docker registry. It can be a public registry like Docker Hub or a private registry, where you host the containers’ image on-premises, within your own network. If you can’t find the image that fits you, you can build your own.

There are many MySQL container images available in the Docker Hub registry. The following screenshot shows some examples:

Firing up a MySQL container

First, you have to install Docker. In the Linux box:

$ sudo apt-get install docker.io #Ubuntu/Debian
$ yum install docker # RedHat/CentOS

Then, use the following basic command to run a MySQL container:

$ docker run --name=test-mysql mysql

Yeap, that’s it. Just two steps. Here is what the second command line does:

• run - Run a command in a new container.
• --name - Give a name to the container. If you don’t specify this, Docker will generate a random name.
• mysql - The image name as stated on the Docker Hub page. This is the simplest image name. The standard is “username/image_name:tag”, for example “severalnines/mysql:5.6”. In this case, we specified “mysql”, which means it has no username (the image is built and maintained by Docker, therefore no username), the image name is “mysql” and the tag is latest (default). If the image does not exist, it will pull it first from Docker Hub into the host, and then run the container.

You should then see the following lines:

Status: Downloaded newer image for mysql:latest
error: database is uninitialized and password option is not specified
  You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_ALLOW_EMPTY_PASSWORD and MYSQL_RANDOM_ROOT_PASSWORD

It looks like the container deployment failed. Let’s verify with the following command if there is any running container:

$ docker ps

There is no running container. Let’s show all containers (including the non-running ones):

$ docker ps -a
CONTAINER ID        IMAGE                                     COMMAND                CREATED             STATUS                     PORTS               NAMES
80b4914976a2        mysql:latest                              "docker-entrypoint.s   6 minutes ago       Exited (1) 6 minutes ago                       test-mysql

Under the ‘STATUS’ column, you can see the status was “Exited (1) 6 minutes ago”. If a program ended while returning a non-zero value, it means that the program was terminated with some kind of error. So, what happened? The MySQL image was successfully downloaded but Docker failed to run it as container because the environment is not properly set up. This is stated in the error lines.

Let’s try to fix this by specifying one of the environment variables:

$ docker run --name=test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" mysql
FATA[0000] Error response from daemon: Conflict. The name "test-mysql" is already in use by container 80b4914976a2. You have to delete (or rename) that container to be able to reuse that name.

Oops, another error occurred. We were trying to run a new container with the same name as an existing container. Let’s remove the created container and run the command again:

$ docker rm test-mysql
$ docker run --name=test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" mysql

You should see lots of lines appear. That’s MySQL initialization when starting up as newly installed software. You should see MySQL is ready to accept connections:

2016-06-01T12:06:59.543352Z 0 [Note] mysqld: ready for connections.
Version: '5.7.12'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)

Looks good. Our MySQL container is now running. However, you are now stuck in the terminal and can’t do anything because the container is running in attach mode (running in foreground). This is so inconvenient. We would expect MySQL to run as a service instead. Let’s consider this as a failed deployment and stop the current container. In another terminal, stop the running container and run it again in detach mode (running as background):

$ docker stop test-mysql
$ docker rm test-mysql
$ docker run --detach --name=test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" mysql
a6b09a8d332a16e008cb3035ffd36bcd664886b79c9d2533c3dc1d47026a33a0

You will get an output of the container ID, indicating the container is successfully running in the background. Let’s verify the status of the container:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
83285aa548ba        mysql:latest        "docker-entrypoint.s   4 minutes ago       Up 4 minutes        3306/tcp            test-mysql

MySQL container is now running and accessible on port 3306 of that container. Since it was running in the background, we could not see what was happening during the MySQL startup. Use the following command to see what happened during the container startup:

$ docker logs test-mysql

Connecting to the Container

Next, we retrieve the IP address of that container in order to access it. Run the inspect command:

$ docker inspect test-mysql

We can see lots of low-level information of that container. Lookup the “IPAddress” line:

"IPAddress": "172.17.0.20",

From the physical host, we can now access the MySQL server. Ensure the MySQL client package is installed beforehand:

$ apt-get install mysql-client
$ mysql -uroot -pmypassword -h 172.17.0.20 -P 3306
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.12 MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Voila! We now have a MySQL instance running in a container. However, this port is only accessible within the Docker network. If you have another Docker container for your application, you can connect with them directly via IP address 172.17.0.20 on port 3306, as illustrated in the following diagram:

Docker allocates a dynamic IP address on every running container. Whenever a container is restarted, you will get a new IP address. You can get the IP address range from the Docker network interface in the Linux box. Run the following command to see Docker’s network range:

$ ip a | grep docker | grep inet
    inet 172.17.42.1/16 scope global docker0

Our container’s IP address is 172.17.0.20 which is in the range of 172.17.42.1/16. Let’s restart the container, and you should get a new IP address being assigned by Docker:

$ docker stop test-mysql
$ docker start test-mysql
$ docker inspect test-mysql | grep IPAddress"IPAddress": "172.17.0.21",

Our IP address just changed to 172.17.0.21. If you had an application that connects to this container via the old IP address, the application would not get connected anymore. Docker introduces another way to link your container with another container, to ensure whatever IP address assigned to it will get updated in the linked container. Let’s say we deploy a Wordpress application (which has no MySQL installed on that image), and want to link with our existing MySQL container, test-mysql. Here is what you should do:

$ docker run --detach --name test-wordpress --link test-mysql:mysql wordpress

In a couple of minutes, the container “test-wordpress” will be up and running and linked to our test-mysql container:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
0cb9f4152022        wordpress:latest    "/entrypoint.sh apac   15 seconds ago      Up 15 seconds       80/tcp              test-wordpress
0a7aa1cf196e        mysql:latest        "docker-entrypoint.s   16 minutes ago      Up 16 minutes       3306/tcp            test-mysql

To verify if it’s linked correctly, enter the test-wordpress container and look at the content of /etc/hosts:

$ docker exec -it test-wordpress bash
root@0cb9f4152022:/var/www/html# cat /etc/hosts
172.17.0.22    0cb9f4152022
127.0.0.1    localhost
::1    localhost ip6-localhost ip6-loopback
fe00::0    ip6-localnet
ff00::0    ip6-mcastprefix
ff02::1    ip6-allnodes
ff02::2    ip6-allrouters
172.17.0.21    mysql 0a7aa1cf196e test-mysql

The application can now see an entry with IP address and hostname related to the linked MySQL container. If you restart the MySQL container and get another IP address, the entry will be updated by Docker accordingly.

You can also expose the MySQL container to the outside world by mapping the container’s MySQL port to the host machine port using the publish flag (as illustrated in the above diagram). Let’s re-initiate our container and run it again with an exposed port:

$ docker rm -f test-mysql
$ docker run --detach --name=test-mysql --env="MYSQL_ROOT_PASSWORD=mypassword" --publish 6603:3306 mysql

Verify if the container is correctly mapped:

CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                    NAMES
8d97b70658a9        mysql:latest        "docker-entrypoint.s   3 seconds ago       Up 3 seconds        0.0.0.0:6603->3306/tcp   test-mysql
0cb9f4152022        wordpress:latest    "/entrypoint.sh apac   15 minutes ago      Up 15 minutes       80/tcp                   test-wordpress

At this point, we can now access the MySQL container directly from the machine’s port 6603.

Configuration management

The container comes with a standard MySQL 5.7 configuration options inside /etc/mysql/my.cnf. Let’s say our application that connects to this MySQL server requires more max_connections (default is 151) during startup, so we need to update the MySQL configuration file. The best way to do this in a container is to create alternative configuration files in a directory on the host machine and then mount that directory location as /etc/mysql/conf.d inside the mysql container.

On the host machine, create a directory and a custom MySQL configuration file:

$ mkdir -p /root/container/test-mysql/conf.d
$ vim /root/container/test-mysql/conf.d/my-custom.cnf

And add the following lines:

[mysqld]
max_connections=200

Then, we have to re-initiate the MySQL container (remove and run) by mapping the volume path as shown in the following command (the long command is trimmed to make it more readable):

$ docker run \
--detach \
--name=test-mysql \
--env="MYSQL_ROOT_PASSWORD=mypassword" \
--publish 6603:3306 \
--volume=/root/docker/test-mysql/conf.d:/etc/mysql/conf.d \
mysql

This will start a new container test-mysql where the MySQL instance uses the combined startup settings from the default /etc/mysql/my.cnf and /etc/mysql/conf.d/my-custom.cnf, with settings from the latter taking precedence.

Verify the new setting is loaded from the machine host:

$ mysql -uroot -pmypassword -h127.0.0.1 -P6603 -e 'show global variables like "max_connections"';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 200   |
+-----------------+-------+

Many configuration options can also be passed as flags to mysqld. This will give you the flexibility to customize the container without needing a custom .cnf file. For example, if you want to change the max connections similar to the above and collation for all tables to use UTF-8 (utf8mb4), just run the following:

$ docker run \
--detach \
--name=test-mysql \
--env="MYSQL_ROOT_PASSWORD=mypassword" \
--publish 6603:3306 \
mysql \
--max-connections=200 \
--character-set-server=utf8mb4 \
--collation-server=utf8mb4_unicode_ci

Data Storage

There are several ways to store data used by MySQL that run in Docker containers. Docker can manage the storage of your database’s data by writing the database files to disk on the host system, using its own internal volume management. If you run the inspect command, look at the “Volumes” directive and you should notice by default MySQL data directory (/var/lib/mysql) is mounted into Docker’s internal volume:

$ docker inspect test-mysql
    ..."Volumes": {"/etc/mysql/conf.d": "/root/docker/test-mysql/conf.d","/var/lib/mysql": "/var/lib/docker/vfs/dir/4d437e2fe5ab2f71ffeea2590d72a417a9ca88607c130b46f5ad819d0a5b68cd"
    }

This is the easiest way and fairly transparent to the user. The downside is that the files may be hard to locate for tools and applications that run directly on the host system, i.e. outside containers.

The other way is to create a data directory on the host system (outside the container) and mount this to a directory visible from inside the container. This places the database files in a known location on the host system, and makes it easy for tools and applications on the host system to access the files. The downside is that the user needs to make sure that the directory exists, and that e.g. directory permissions and other security mechanisms on the host system are correctly set up.

Create a data directory on a suitable volume on your host system, e.g. /storage/docker/mysql-datadir:

$ mkdir -p /storage/docker/mysql-datadir

Start your mysql container like this:

$ docker run \
--detach \
--name=test-mysql \
--env="MYSQL_ROOT_PASSWORD=mypassword" \
--publish 6603:3306 \
--volume=/root/docker/test-mysql/conf.d:/etc/mysql/conf.d \
--volume=/storage/docker/mysql-datadir:/var/lib/mysql \
mysql

The ”--volume=/storage/docker/mysql-datadir:/var/lib/mysql“ part of the command mounts the /storage/docker/mysql-datadir directory from the underlying host system as /var/lib/mysql inside the container, where MySQL by default will write its data files, as illustrated in the following diagram:

When inspecting the container, you should see the following lines:

$ docker inspect test-mysql
    ..."Volumes": {"/etc/mysql/conf.d": "/root/docker/test-mysql/conf.d","/var/lib/mysql": "/storage/docker/mysql-datadir"
    }

Which is now clearer for you to see the directory and files on the machine host created by this container:

$ ls -al /storage/docker/mysql-datadir/
total 188452
drwxr-xr-x 5  999  999     4096 Jun  3 10:13 .
drwxr-xr-x 3 root root     4096 Jun  3 10:06 ..
-rw-r----- 1  999  999       56 Jun  3 10:13 auto.cnf
-rw-r----- 1  999  999     1329 Jun  3 10:13 ib_buffer_pool
-rw-r----- 1  999  999 79691776 Jun  3 10:13 ibdata1
-rw-r----- 1  999  999 50331648 Jun  3 10:13 ib_logfile0
-rw-r----- 1  999  999 50331648 Jun  3 10:13 ib_logfile1
-rw-r----- 1  999  999 12582912 Jun  3 10:13 ibtmp1
drwxr-x--- 2  999  999     4096 Jun  3 10:13 mysql
drwxr-x--- 2  999  999     4096 Jun  3 10:13 performance_schema
drwxr-x--- 2  999  999    12288 Jun  3 10:13 sys

Note that restarting or removing the container does not remove the MySQL data directory. When you restart a MySQL container by using “stop” and “start” command, it would be similar to restarting the MySQL service in a standard installation:

$ docker stop test-mysql
$ docker start test-mysql

If you remove the MySQL container, the data in the mounted volumes will still be intact and you can run a new instance, mounting the same volume as data directory:

$ docker rm -f test-mysql
$ docker run -d --name=new-mysql -p 6604:3306 -v /storage/docker/mysql-datadir:/var/lib/mysql mysql

*If a MySQL container runs on top of an existing MySQL datadir, the $MYSQL_ROOT_PASSWORD variable should be omitted from the “run” command line; it will in any case be ignored, and the pre-existing database will not be changed in any way.

However, only one running (active) container is allowed to access the MySQL data directory at a time. Running another container mounting the same datadir volume will cause MySQL startup error on the later container:

$ docker run -d --name=another-new-mysql -p 6605:3306 -v /storage/docker/mysql-datadir:/var/lib/mysql mysql
$ docker logs another-new-mysql
2016-06-10T07:16:59.538635Z 0 [Note] InnoDB: Completed initialization of buffer pool
2016-06-10T07:16:59.540373Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2016-06-10T07:16:59.551646Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2016-06-10T07:16:59.551656Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2016-06-10T07:16:59.551659Z 0 [Note] InnoDB: Retrying to lock the first data file
2016-06-10T07:17:00.552294Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2016-06-10T07:17:00.552364Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.

This is expected since a MySQL process must exclusively own the MySQL data directory to avoid any potential conflicts. Having more than one mysqld process connecting to the same data directory is just not feasible. That’s why MySQL horizontal scaling can only be achieved via replication.

That concludes this blog. Do share your experience in managing MySQL containers in the comments section below.

Our journey in adopting MySQL and MariaDB in containerized environments continues, with ClusterControl coming into the picture to facilitate deployment and management. We already have our ClusterControl image hosted in Docker Hub, where it can deploy different replication/cluster topologies on multiple containers. With the introduction of Docker Swarm, a native orchestration tools embedded inside Docker Engine, scaling and provisioning containers has become much easier. It also has high availability covered by running services on multiple Docker hosts.

In this blog post, we’ll be experimenting with automatic provisioning of Galera Cluster on Docker Swarm with ClusterControl. ClusterControl would usually deploy database clusters on bare-metal, virtual machines and cloud instances. ClusterControl relies on SSH (through libssh) as core communication module to connect to the managed hosts, so these would not require any agents. The same rule can be applied to containers, and that’s what we are going to show in this blog post.

ClusterControl as Docker Swarm Service

We have built a Docker image with extended logic to handle deployment in container environments in a semi-automatic way. The image is now available on Docker Hub and the code is hosted in our Github repository. Please note that only this image is capable of deploying on containers, and is not available in the standard ClusterControl installation packages.

The extended logic is inside deploy-container.sh, a script that monitors a custom table inside CMON database called “cmon.containers”. The created database container shall report and register itself into this table and this script will look for new entries and perform the necessary action using a ClusterControl CLI. The deployment is automatic, and you can monitor the progress directly from the ClusterControl UI or using “docker logs” command.

Before we go further, take note of some prerequisites for running ClusterControl and Galera Cluster on Docker Swarm:

• Docker Engine version 1.12 and later.
• Docker Swarm Mode is initialized.
• ClusterControl must be connected to the same overlay network as the database containers.

To run ClusterControl as a service using “docker stack”, the following definition should be enough:

 clustercontrol:
    deploy:
      replicas: 1
    image: severalnines/clustercontrol
    ports:
      - 5000:80
    networks:
      - galera_cc

Or, you can use the “docker service” command as per below:

$ docker service create --name cc_clustercontrol -p 5000:80 --replicas 1 severalnines/clustercontrol

Or, you can combine the ClusterControl service together with the database container service and form a “stack” in a compose file as shown in the next section.

Base Containers as Docker Swarm Service

The base container’s image called “centos-ssh” is based on CentOS 6 image. It comes with a couple of basic packages like SSH server, clients, curl and mysql client. The entrypoint script will download ClusterControl’s public key for passwordless SSH during startup. It will also register itself to the ClusterControl’s CMON database for automatic deployment.

Running this container requires a couple of environment variables to be set:

• CC_HOST - Mandatory. By default it will try to connect to “cc_clustercontrol” service name. Otherwise, define its value in IP address, hostname or service name format. This container will download the SSH public key from ClusterControl node automatically for passwordless SSH.
• CLUSTER_TYPE - Mandatory. Default to “galera”.
• CLUSTER_NAME - Mandatory. This name distinguishes the cluster with others from ClusterControl perspective. No space allowed and it must be unique.
• VENDOR - Default is “percona”. Other supported values are “mariadb”, “codership”.
• DB_ROOT_PASSWORD - Mandatory. The database root password for the database server. In this case, it should be MySQL root password.
• PROVIDER_VERSION - Default is 5.6. The database version by the chosen vendor.
• INITIAL_CLUSTER_SIZE - Default is 3. This indicates how ClusterControl should treat newly registered containers, whether they are for new deployments or for scaling out. For example, if the value is 3, ClusterControl will wait for 3 containers to be running and registered into the CMON database before starting the cluster deployment job. Otherwise, it waits 30 seconds for the next cycle and retries. The next containers (4th, 5th and Nth) will fall under the “Add Node” job instead.

To run the container, simply use the following stack definition in a compose file:

  galera:
    deploy:
      replicas: 3
    image: severalnines/centos-ssh
    ports:
      - 3306:3306
    environment:
      CLUSTER_TYPE: "galera"
      CLUSTER_NAME: "PXC_Docker"
      INITIAL_CLUSTER_SIZE: 3
      DB_ROOT_PASSWORD: "mypassword123"
    networks:
      - galera_cc

By combining them both (ClusterControl and database base containers), we can just deploy them under a single stack as per below:

version: '3'

services:

  galera:
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure
        delay: 10s
    image: severalnines/centos-ssh
    ports:
      - 3306:3306
    environment:
      CLUSTER_TYPE: "galera"
      CLUSTER_NAME: "Galera_Docker"
      INITIAL_CLUSTER_SIZE: 3
      DB_ROOT_PASSWORD: "mypassword123"
    networks:
      - galera_cc

  clustercontrol:
    deploy:
      replicas: 1
    image: severalnines/clustercontrol
    ports:
      - 5000:80
    networks:
      - galera_cc

networks:
  galera_cc:
    driver: overlay

Save the above lines into a file, for example docker-compose.yml in the current directory. Then, start the deployment:

$ docker stack deploy --compose-file=docker-compose.yml cc
Creating network cc_galera_cc
Creating service cc_clustercontrol
Creating service cc_galera

Docker Swarm will deploy one container for ClusterControl (replicas:1) and another 3 containers for the database cluster containers (replicas:3). The database container will then register itself into the CMON database for deployment.

Wait for a Galera Cluster to be ready

The deployment will be automatically picked up by the ClusterControl CLI. So you basically don’t have to do anything but wait. The deployment usually takes around 10 to 20 minutes depending on the network connection.

Open the ClusterControl UI at http://{any_Docker_host}:5000/clustercontrol, fill in the default administrator user details and log in. Monitor the deployment progress under Activity -> Jobs, as shown in the following screenshot:

Or, you can look at the progress directly from the docker logs command of the ClusterControl container:

$ docker logs -f $(docker ps | grep clustercontrol | awk {'print $1'})>> Found the following cluster(s) is yet to deploy:
Galera_Docker>> Number of containers for Galera_Docker is lower than its initial size (3).>> Nothing to do. Will check again on the next loop.>> Found the following cluster(s) is yet to deploy:
Galera_Docker>> Found a new set of containers awaiting for deployment. Sending deployment command to CMON.>> Cluster name         : Galera_Docker>> Cluster type         : galera>> Vendor               : percona>> Provider version     : 5.7>> Nodes discovered     : 10.0.0.6 10.0.0.7 10.0.0.5>> Initial cluster size : 3>> Nodes to deploy      : 10.0.0.6;10.0.0.7;10.0.0.5>> Deploying Galera_Docker.. It's gonna take some time..>> You shall see a progress bar in a moment. You can also monitor>> the progress under Activity (top menu) on ClusterControl UI.
Create Galera Cluster
- Job  1 RUNNING    [██▊       ]  26% Installing MySQL on 10.0.0.6

That’s it. Wait until the deployment completes and you will then be all set with a three-node Galera Cluster running on Docker Swarm, as shown in the following screenshot:

In ClusterControl, it has the same look and feel as what you have seen with Galera running on standard hosts (non-container) environment.

Management

Managing database containers is a bit different with Docker Swarm. This section provides an overview of how the database containers should be managed through ClusterControl.

Connecting to the Cluster

To verify the status of the replicas and service name, run the following command:

$ docker service ls
ID            NAME               MODE        REPLICAS  IMAGE
eb1izph5stt5  cc_clustercontrol  replicated  1/1       severalnines/clustercontrol:latest
ref1gbgne6my  cc_galera          replicated  3/3       severalnines/centos-ssh:latest

If the application/client is running on the same Swarm network space, you can connect to it directly via the service name endpoint. If not, use the routing mesh by connecting to the published port (3306) on any of the Docker Swarm nodes. The connection to these endpoints will be load balanced automatically by Docker Swarm in a round-robin fashion.

Scale up/down

Typically, when adding a new database, we need to prepare a new host with base operating system together with passwordless SSH. In Docker Swarm, you just need to scale out the service using the following command to the number of replicas that you desire:

$ docker service scale cc_galera=5
cc_galera scaled to 5

ClusterControl will then pick up the new containers registered inside cmon.containers table and trigger add node jobs, for one container at a time. You can look at the progress under Activity -> Jobs:

Scaling down is similar, by using the “service scale” command. However, ClusterControl doesn’t know whether the containers that have been removed by Docker Swarm were part of the auto-scheduling or just a scale down (which indicates that we deliberately wanted the containers to be removed). Thus, to scale down from 5 nodes to 3 nodes, one would:

$ docker service scale cc_galera=3
cc_galera scaled to 3

Then, remove the stopped hosts from the ClusterControl UI by going to Nodes -> rollover the removed container -> click on the ‘X’ icon on the top right -> Confirm & Remove Node:

ClusterControl will then execute a remove node job and bring back the cluster to the expected size.

Failover

In case of container failure, Docker Swarm automatic rescheduling will kick in and there will be a new replacement container with the same IP address as the old one (with different container ID). ClusterControl will then start to provision this node from scratch, by performing the installation process, configuration and getting it to rejoin the cluster. The old container will be removed automatically from ClusterControl before the deployment starts.

Go ahead and try to kill one of the database containers:

$ docker kill [container ID]

You’ll see the new containers that Swarm created will be provisioned automatically by ClusterControl.

Creating a new cluster

To create a new cluster, just create another service or stack with a different CLUSTER_NAME and service name. The following is an example that we want to create another Galera Cluster running on MariaDB 10.1 (some extra environment variables are required for MariaDB 10.1):

version: '3'
services:
  galera2:
    deploy:
      replicas: 3
    image: severalnines/centos-ssh
    ports:
      - 3306
    environment:
      CLUSTER_TYPE: "galera"
      CLUSTER_NAME: "MariaDB_Galera"
      VENDOR: "mariadb"
      PROVIDER_VERSION: "10.1"
      INITIAL_CLUSTER_SIZE: 3
      DB_ROOT_PASSWORD: "mypassword123"
    networks:
      - cc_galera_cc

networks:
  cc_galera_cc:
    external: true

Then, create the service:

$ docker stack deploy --compose-file=docker-compose.yml db2

Go back to ClusterControl UI -> Activity -> Jobs and you should see a new deployment has started. After a couple of minutes, you will see the new cluster will be listed inside ClusterControl dashboard:

Destroying everything

To remove everything (including the ClusterControl container), you just need to remove the stack created by Docker Swarm:

$ docker stack rm cc
Removing service cc_clustercontrol
Removing service cc_galera
Removing network cc_galera_cc

That’s it, the whole stack has been removed. Pretty neat huh? You can start all over again by running the “docker stack deploy” command and everything will be ready after a couple of minutes.

Summary

The flexibility you get by running a single command to deploy or destroy a whole environment can be useful in different types of use cases such as backup verification, DDL procedure testing, query performance tweaking, experimenting for proof-of-concepts and also for staging temporary data. These use cases are closer to developer environment. With this approach, you can now treat a stateful service “statelessly”.

Would you like to see ClusterControl manage the whole database container stack through the UI via point and click? Let us know your thoughts in the comments section below. In the next blog post, we are going to look at how to perform automatic backup verification on Galera Cluster using containers.

In the last couple of blog posts on Docker, we have looked into understanding and running Galera Cluster on Docker Swarm. It scales and fails over pretty well, but there are still some limitations that prevent it from running smoothly in a production environment. We will be discussing about these limitations, and see how we can overcome them. Hopefully, this will clear some of the questions that might be circling around in your head.

Docker Swarm Mode Limitations

Docker Swarm Mode is tremendous at orchestrating and handling stateless applications. However, since our focus is on trying to make Galera Cluster (a stateful service) to run smoothly on Docker Swarm, we have to make some adaptations to bring the two together. Running Galera Cluster in containers in production requires at least:

• Health check - Each of the stateful containers must pass the Docker health checks, to ensure it achieves the correct state before being included into the active load balancing set.
• Data persistency - Whenever a container is replaced, it has to be started from the last known good configuration. Else you might lose data.
• Load balancing algorithm - Since Galera Cluster can handle read/write simultaneously, each node can be treated equally. A recommended balancing algorithm for Galera Cluster is least connection. This algorithm takes into consideration the number of current connections each server has. When a client attempts to connect, the load balancer will try to determine which server has the least number of connections and then assign the new connection to that server.

We are going to discuss all the points mentioned above with a great detail, plus possible workarounds on how to tackle those problems.

Health Check

HEALTHCHECK is a command to tell Docker how to test a container, to check that it is still working. In Galera, the fact that mysqld is running does not mean it is healthy and ready to serve. Without a proper health check, Galera could be wrongly diagnosed when something goes wrong, and by default, Docker Swarm’s ingress network will include the “STARTED” container into the load balancing set regardless of the Galera state. On the other hand, you have to manually attach to a MySQL container to check for various MySQL statuses to determine if the container is healthy.

With HEALTHCHECK configured, container healthiness can be retrieved directly from the standard “docker ps” command:

$ docker ps
CONTAINER ID        IMAGE                       COMMAND             CREATED             STATUS                    PORTS
42f98c8e0934        severalnines/mariadb:10.1   "/entrypoint.sh "   13 minutes ago      Up 13 minutes (healthy)   3306/tcp, 4567-4568/tcp

Plus, Docker Swarm’s ingress network will include only the healthy container right after the health check output starts to return 0 after startup. The following table shows the comparison of these two behaviours:

The only problem with Docker health check is it only supports two exit codes - either 1 (unhealthy) or 0 (healthy). This is enough for a stateless application, where containers can come and go without caring much about the state itself and other containers. With a stateful service like Galera Cluster or MySQL Replication, another exit code is required to represent a staging phase. For example, when a joiner node comes into the picture, syncing is required from a donor node (by SST or IST). This process is automatically started by Galera and probably requires minutes or hours to complete, and the current workaround for this is to configure [--update-delay] and [--health-interval * --health-retires] to higher than the SST/IST time.

For a clearer perspective, consider the following “service create” command example:

$ docker service create \
--replicas=3 \
--health-interval=30s \
--health-retries=20 \
--update-delay=600s \
--name=galera \
--network=galera_net \
severalnines/mariadb:10.1

The container will be destroyed if the SST process has taken more than 600 seconds. While in this state, the health check script will return “exit 1 (unhealthy)” in both joiner and donor containers because both are not supposed to be included by Docker Swarm’s load balancer since they are in syncing stage. After failures for 20 consecutive times at every 30 seconds (equal to 600 seconds), the joiner and donor containers will be removed by Docker Swarm and will be replaced by new containers.

It would be perfect if Docker’s HEALTHCHECK could accept more than exit "0" or "1" to signal Swarm’s load balancer. For example:

• exit 0 => healthy => load balanced and running
• exit 1 => unhealthy => no balancing and failed
• exit 2 => unhealthy but ignore => no balancing but running

Thus, we don’t have to determine SST time for containers to survive the Galera Cluster startup operation, because:

• Joiner/Joined/Donor/Desynced == exit 2
• Synced == exit 0
• Others == exit 1

Another workaround apart setting up [--update-delay] and [--health-interval * --health-retires] to be higher than SST time is you could use HAProxy as the load balancer endpoints, instead of relying on Docker Swarm’s load balancer. More discussion further on.

Data Persistency

Stateless doesn’t really care about persistency. It shows up, serves and get destroyed if the job is done or it is unhealthy. The problem with this behaviour is there is a chance of a total data loss in Galera Cluster, which is something that cannot be afforded by a database service. Take a look at the following example:

$ docker service create \
--replicas=3 \
--health-interval=30s \
--health-retries=20 \
--update-delay=600s \
--name=galera \
--network=galera_net \
severalnines/mariadb:10.1

So, what happens if the switch connecting the three Docker Swarm nodes goes down? A network partition, which will split a three-node Galera Cluster into 'single-node' components. The cluster state will get demoted into Non-Primary and the Galera node state will turn to Initialized. This situation turns the containers into an unhealthy state according to the health check. After a period 600 seconds if the network is still down, those database containers will be destroyed and replaced with new containers by Docker Swarm according to the "docker service create" command. You will end up having a new cluster starting from scratch, and the existing data is removed.

There is a workaround to protect from this, by using mode global with placement constraints. This is the preferred way when you are running your database containers on Docker Swarm with persistent storage in mind. Consider the following example:

$ docker service create \
--mode=global \
--constraints='node.labels.type == galera' \
--health-interval=30s \
--health-retries=20 \
--update-delay=600s \
--name=galera \
--network=galera_net \
severalnines/mariadb:10.1

The cluster size is limited to the number of available Docker Swarm node labelled with "type=galera". Dynamic scaling is not an option here. Scaling up or down is only possible if you introduce or remove a Swarm node with the correct label. The following diagram shows a 3-node Galera Cluster container with persistent volumes, constrained by custom node label "type=galera":

It would also be great if Docker Swarm supported more options to handle container failures:

• Don’t delete the last X failed containers, for troubleshooting purposes.
• Don’t delete the last X volumes, for recovery purposes.
• Notify users if a container is recreated, deleted, rescheduled.

Load Balancing Algorithm

Docker Swarm comes with a load balancer, based on IPVS module in Linux kernel, to distribute traffic to all containers in round-robin fashion. It lacks several useful configurable options to handle routing of stateful applications, for example persistent connection (so source will always reach the same destination) and support for other balancing algorithm, like least connection, weighted round-robin or random. Despite IPVS being capable of handling persistent connections via option "-p", it doesn’t seem to be configurable in Docker Swarm.

In MySQL, some connections might take a bit longer time to process before it returns the output back to the clients. Thus, Galera Cluster load distribution should use "least connection" algorithm, so the load is equally distributed to all database containers. The load balancer would ideally monitor the number of open connections for each server, and sends to the least busy server. Kubernetes defaults to least connection when distributing traffic to the backend containers.

As a workaround, relying on other load balancers in front of the service is still the recommended way. HAProxy, ProxySQL and MaxScale excel in this area. However, you have to make sure these load balancers are aware of the dynamic changes of the backend database containers especially during scaling and failover.

Summary

Galera Cluster on Docker Swarm fits well in development, test and staging environments, but it needs some more work when running in production. The technology still needs some time to mature, but as we saw in this blog, there are ways to work around the current limitations.

In the last couple of blogs, we covered how to run a Galera Cluster on Docker, whether on standalone Docker or on multi-host Docker Swarm with overlay network. In this blog post, we’ll look into running Galera Cluster on Kubernetes, an orchestration tool to run containers at scale. Some parts are different, such as how the application should connect to the cluster, how Kubernetes handles failover and how the load balancing works in Kubernetes.

Kubernetes vs Docker Swarm

Our ultimate target is to ensure Galera Cluster runs reliably in a container environment. We previously covered Docker Swarm, and it turned out that running Galera Cluster on it has a number of blockers, which prevent it from being production ready. Our journey now continues with Kubernetes, a production-grade container orchestration tool. Let’s see which level of “production-readiness” it can support when running a stateful service like Galera Cluster.

Before we move further, let us highlight some of key differences between Kubernetes (1.6) and Docker Swarm (17.03) when running Galera Cluster on containers:

• Kubernetes supports two health check probes - liveness and readiness. This is important when running a Galera Cluster on containers, because a live Galera container does not mean it is ready to serve and should be included in the load balancing set (think of a joiner/donor state). Docker Swarm only supports one health check probe similar to Kubernetes’ liveness, a container is either healthy and keeps running or unhealthy and gets rescheduled. Read here for details.
• Kubernetes has a UI dashboard accessible via “kubectl proxy”.
• Docker Swarm only supports round-robin load balancing (ingress), while Kubernetes uses least connection.
• Docker Swarm supports routing mesh to publish a service to the external network, while Kubernetes supports something similar called NodePort, as well as external load balancers (GCE GLB/AWS ELB) and external DNS names (as for v1.7)

Installing Kubernetes using kubeadm

We are going to use kubeadm to install a 3-node Kubernetes cluster on CentOS 7. It consists of 1 master and 2 nodes (minions). Our physical architecture looks like this:

1. Install kubelet and Docker on all nodes:

$ ARCH=x86_64
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-${ARCH}
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
$ setenforce 0
$ yum install -y docker kubelet kubeadm kubectl kubernetes-cni
$ systemctl enable docker && systemctl start docker
$ systemctl enable kubelet && systemctl start kubelet

2. On the master, initialize the master, copy the configuration file, setup the Pod network using Weave and install Kubernetes Dashboard:

$ kubeadm init
$ cp /etc/kubernetes/admin.conf $HOME/
$ export KUBECONFIG=$HOME/admin.conf
$ kubectl apply -f https://git.io/weave-kube-1.6
$ kubectl create -f https://git.io/kube-dashboard

3. Then on the other remaining nodes:

$ kubeadm join --token 091d2a.e4862a6224454fd6 192.168.55.140:6443

4. Verify the nodes are ready:

$ kubectl get nodes
NAME          STATUS    AGE       VERSION
kube1.local   Ready     1h        v1.6.3
kube2.local   Ready     1h        v1.6.3
kube3.local   Ready     1h        v1.6.3

We now have a Kubernetes cluster for Galera Cluster deployment.

Galera Cluster on Kubernetes

In this example, we are going to deploy a MariaDB Galera Cluster 10.1 using Docker image pulled from our DockerHub repository. The YAML definition files used in this deployment can be found under example-kubernetes directory in the Github repository.

Kubernetes supports a number of deployment controllers. To deploy a Galera Cluster, one can use:

• ReplicaSet
• StatefulSet

Each of them has their own pro and cons. We are going to look into each one of them and see what’s the difference.

Prerequisites

The image that we built requires an etcd (standalone or cluster) for service discovery. To run an etcd cluster requires each etcd instance to be running with different commands so we are going to use Pods controller instead of Deployment and create a service called “etcd-client” as endpoint to etcd Pods. The etcd-cluster.yaml definition file tells it all.

To deploy a 3-pod etcd cluster, simply run:

$ kubectl create -f etcd-cluster.yaml

Verify if the etcd cluster is ready:

$ kubectl get po,svc
NAME                        READY     STATUS    RESTARTS   AGE
po/etcd0                    1/1       Running   0          1d
po/etcd1                    1/1       Running   0          1d
po/etcd2                    1/1       Running   0          1d

NAME              CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
svc/etcd-client   10.104.244.200   <none>        2379/TCP            1d
svc/etcd0         10.100.24.171    <none>        2379/TCP,2380/TCP   1d
svc/etcd1         10.108.207.7     <none>        2379/TCP,2380/TCP   1d
svc/etcd2         10.101.9.115     <none>        2379/TCP,2380/TCP   1d

Our architecture is now looking something like this:

Using ReplicaSet

A ReplicaSet ensures that a specified number of pod “replicas” are running at any given time. However, a Deployment is a higher-level concept that manages ReplicaSets and provides declarative updates to pods along with a lot of other useful features. Therefore, it’s recommended to use Deployments instead of directly using ReplicaSets, unless you require custom update orchestration or don’t require updates at all. When you use Deployments, you don’t have to worry about managing the ReplicaSets that they create. Deployments own and manage their ReplicaSets.

In our case, we are going to use Deployment as the workload controller, as shown in this YAML definition. We can directly create the Galera Cluster ReplicaSet and Service by running the following command:

$ kubectl create -f mariadb-rs.yml

Verify if the cluster is ready by looking at the ReplicaSet (rs), pods (po) and services (svc):

$ kubectl get rs,po,svc
NAME                  DESIRED   CURRENT   READY     AGE
rs/galera-251551564   3         3         3         5h

NAME                        READY     STATUS    RESTARTS   AGE
po/etcd0                    1/1       Running   0          1d
po/etcd1                    1/1       Running   0          1d
po/etcd2                    1/1       Running   0          1d
po/galera-251551564-8c238   1/1       Running   0          5h
po/galera-251551564-swjjl   1/1       Running   1          5h
po/galera-251551564-z4sgx   1/1       Running   1          5h

NAME              CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
svc/etcd-client   10.104.244.200   <none>        2379/TCP            1d
svc/etcd0         10.100.24.171    <none>        2379/TCP,2380/TCP   1d
svc/etcd1         10.108.207.7     <none>        2379/TCP,2380/TCP   1d
svc/etcd2         10.101.9.115     <none>        2379/TCP,2380/TCP   1d
svc/galera-rs     10.107.89.109    <nodes>       3306:30000/TCP      5h
svc/kubernetes    10.96.0.1        <none>        443/TCP             1d

From the output above, we can illustrate our Pods and Service as below:

Running Galera Cluster on ReplicaSet is similar to treating it as a stateless application. It orchestrates pod creation, deletion and updates and can be targeted for Horizontal Pod Autoscales (HPA), i.e. a ReplicaSet can be auto-scaled if it meets certain thresholds or targets (CPU usage, packets-per-second, request-per-second etc).

If one of the Kubernetes nodes goes down, new Pods will be scheduled on an available node to meet the desired replicas. Volumes associated with the Pod will be deleted, if the Pod is deleted or rescheduled. The Pod hostname will be randomly generated, making it harder to track where the container belongs by simply looking at the hostname.

All this works pretty well in test and staging environments, where you can perform a full container lifecycle like deploy, scale, update and destroy without any dependencies. Scaling up and down is straightforward, by updating the YAML file and posting it to Kubernetes cluster or by using the scale command:

$ kubectl scale replicaset galera-rs --replicas=5

Using StatefulSet

Known as PetSet on pre 1.6 version, StatefulSet is the best way to deploy Galera Cluster in production, because:

• Deleting and/or scaling down a StatefulSet will not delete the volumes associated with the StatefulSet. This is done to ensure data safety, which is generally more valuable than an automatic purge of all related StatefulSet resources.
• For a StatefulSet with N replicas, when Pods are being deployed, they are created sequentially, in order from {0 .. N-1}.
• When Pods are being deleted, they are terminated in reverse order, from {N-1 .. 0}.
• Before a scaling operation is applied to a Pod, all of its predecessors must be Running and Ready.
• Before a Pod is terminated, all of its successors must be completely shut down.

StatefulSet provides first-class support for stateful containers. It provides a deployment and scaling guarantee. When a three-node Galera Cluster is created, three Pods will be deployed in the order db-0, db-1, db-2. db-1 will not be deployed before db-0 is “Running and Ready”, and db-2 will not be deployed until db-1 is “Running and Ready”. If db-0 should fail, after db-1 is “Running and Ready”, but before db-2 is launched, db-2 will not be launched until db-0 is successfully relaunched and becomes “Running and Ready”.

We are going to use Kubernetes implementation of persistent storage called PersistentVolume and PersistentVolumeClaim. This to ensure data persistency if the pod got rescheduled to the other node. Even though Galera Cluster provides the exact copy of data on each replica, having the data persistent in every pod is good for troubleshooting and recovery purposes.

To create a persistent storage, first we have to create PersistentVolume for every pod. PVs are volume plugins like Volumes in Docker, but have a lifecycle independent of any individual pod that uses the PV. Since we are going to deploy a 3-node Galera Cluster, we need to create 3 PVs:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-galera-0
  labels:
    app: galera-ss
    podindex: "0"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  hostPath:
    path: /data/pods/galera-0/datadir
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-galera-1
  labels:
    app: galera-ss
    podindex: "1"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  hostPath:
    path: /data/pods/galera-1/datadir
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-galera-2
  labels:
    app: galera-ss
    podindex: "2"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  hostPath:
    path: /data/pods/galera-2/datadir

The above definition shows we are going to create 3 PV, mapped to the Kubernetes nodes’ physical path with 10GB of storage space. We defined ReadWriteOnce, which means the volume can be mounted as read-write by only a single node. Save the above lines into mariadb-pv.yml and post it to Kubernetes:

$ kubectl create -f mariadb-pv.yml
persistentvolume "datadir-galera-0" created
persistentvolume "datadir-galera-1" created
persistentvolume "datadir-galera-2" created

Next, create define the PersistentVolumeClaim:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-datadir-galera-ss-0
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  selector:
    matchLabels:
      app: galera-ss
      podindex: "0"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-datadir-galera-ss-1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  selector:
    matchLabels:
      app: galera-ss
      podindex: "1"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-datadir-galera-ss-2
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  selector:
    matchLabels:
      app: galera-ss
      podindex: "2"

The above definition shows that we would like to claim the PV resources and use the spec.selector.matchLabels to look for our PV (metadata.labels.app: galera-ss) based on the respective pod index (metadata.labels.podindex) assigned by Kubernetes. The metadata.name resource must use the format “{volumeMounts.name}-{pod}-{ordinal index}” defined under the spec.templates.containers so Kubernetes knows which mount point to map the claim into the pod.

Save the above lines into mariadb-pvc.yml and post it to Kubernetes:

$ kubectl create -f mariadb-pvc.yml
persistentvolumeclaim "mysql-datadir-galera-ss-0" created
persistentvolumeclaim "mysql-datadir-galera-ss-1" created
persistentvolumeclaim "mysql-datadir-galera-ss-2" created

Our persistent storage is now ready. We can then start the Galera Cluster deployment by creating a StatefulSet resource together with Headless service resource as shown in mariadb-ss.yml:

$ kubectl create -f mariadb-ss.yml
service "galera-ss" created
statefulset "galera-ss" created

Now, retrieve the summary of our StatefulSet deployment:

$ kubectl get statefulsets,po,pv,pvc -o wide
NAME                     DESIRED   CURRENT   AGE
statefulsets/galera-ss   3         3         1d        galera    severalnines/mariadb:10.1   app=galera-ss

NAME                        READY     STATUS    RESTARTS   AGE       IP          NODE
po/etcd0                    1/1       Running   0          7d        10.36.0.1   kube3.local
po/etcd1                    1/1       Running   0          7d        10.44.0.2   kube2.local
po/etcd2                    1/1       Running   0          7d        10.36.0.2   kube3.local
po/galera-ss-0              1/1       Running   0          1d        10.44.0.4   kube2.local
po/galera-ss-1              1/1       Running   1          1d        10.36.0.5   kube3.local
po/galera-ss-2              1/1       Running   0          1d        10.44.0.5   kube2.local

NAME                  CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                               STORAGECLASS   REASON    AGE
pv/datadir-galera-0   10Gi       RWO           Retain          Bound     default/mysql-datadir-galera-ss-0                            4d
pv/datadir-galera-1   10Gi       RWO           Retain          Bound     default/mysql-datadir-galera-ss-1                            4d
pv/datadir-galera-2   10Gi       RWO           Retain          Bound     default/mysql-datadir-galera-ss-2                            4d

NAME                            STATUS    VOLUME             CAPACITY   ACCESSMODES   STORAGECLASS   AGE
pvc/mysql-datadir-galera-ss-0   Bound     datadir-galera-0   10Gi       RWO                          4d
pvc/mysql-datadir-galera-ss-1   Bound     datadir-galera-1   10Gi       RWO                          4d
pvc/mysql-datadir-galera-ss-2   Bound     datadir-galera-2   10Gi       RWO                          4d

At this point, our Galera Cluster running on StatefulSet can be illustrated as in the following diagram:

Running on StatefuleSet guarantees consistent identifiers like hostname, IP address, network ID, cluster domain, Pod DNS and storage. This allows the Pod to easily distinguish itself from others in a group of Pods. The volume will be retained on the host and will not get deleted if the Pod is deleted or rescheduled onto another node. This allows for data recovery and reduces the risk of total data loss.

On the negative side, the deployment time will be N-1 times (N = replicas) longer because Kubernetes will obey the ordinal sequence when deploying, rescheduling or deleting the resources. It would be a bit of a hassle to prepare the PV and claims before thinking about scaling your cluster. Take note that updating an existing StatefulSet is currently a manual process, where you can only update spec.replicas at the moment.

Connecting to Galera Cluster Service and Pods

There are a couple of ways you can connect to the database cluster. You can connect directly to the port. In the “galera-rs” service example, we use NodePort, exposing the service on each Node’s IP at a static port (the NodePort). A ClusterIP service, to which the NodePort service will route, is automatically created. You’ll be able to contact the NodePort service, from outside the cluster, by requesting {NodeIP}:{NodePort}.

Example to connect to the Galera Cluster externally:

(external)$ mysql -udb_user -ppassword -h192.168.55.141 -P30000
(external)$ mysql -udb_user -ppassword -h192.168.55.142 -P30000
(external)$ mysql -udb_user -ppassword -h192.168.55.143 -P30000

Within the Kubernetes network space, Pods can connect via cluster IP or service name internally which is retrievable by using the following command:

$ kubectl get services -o wide
NAME          CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE       SELECTOR
etcd-client   10.104.244.200   <none>        2379/TCP            1d        app=etcd
etcd0         10.100.24.171    <none>        2379/TCP,2380/TCP   1d        etcd_node=etcd0
etcd1         10.108.207.7     <none>        2379/TCP,2380/TCP   1d        etcd_node=etcd1
etcd2         10.101.9.115     <none>        2379/TCP,2380/TCP   1d        etcd_node=etcd2
galera-rs     10.107.89.109    <nodes>       3306:30000/TCP      4h        app=galera-rs
galera-ss     None             <none>        3306/TCP            3m        app=galera-ss
kubernetes    10.96.0.1        <none>        443/TCP             1d        <none>

From the service list, we can see that the Galera Cluster ReplicaSet Cluster-IP is 10.107.89.109. Internally, another pod can access the database through this IP address or service name using the exposed port, 3306:

(etcd0 pod)$ mysql -udb_user -ppassword -hgalera-rs -P3306 -e 'select @@hostname'
+------------------------+
| @@hostname             |
+------------------------+
| galera-251551564-z4sgx |
+------------------------+

Internally, you can also connect to the external NodePort from inside the pod on port 30000:

(etcd0 pod)$ mysql -udb_user -ppassword -h192.168.55.143 -P30000 -e 'select @@hostname'
+------------------------+
| @@hostname             |
+------------------------+
| galera-251551564-z4sgx |
+------------------------+

Connection to the backend Pods will be load balanced accordingly using least connection.

Summary

At this point, running Galera Cluster on Kubernetes in production seems much more promising as compared to Docker Swarm. As discussed in the last blog post, the concerns raised are tackled differently with the way Kubernetes orchestrates containers in StatefulSet, (although it’s still a beta feature in v1.6). We do hope that the suggested approach is going to help run Galera Cluster on containers at scale in production.

(This blog was updated on June 20, 2017)

We’re excited to announce our first step towards dockerizing our products. Please welcome the official ClusterControl Docker image, available on Docker Hub. This will allow you to evaluate ClusterControl with a couple of commands:

$ docker pull severalnines/clustercontrol

The Docker image comes with ClusterControl installed and configured with all of its components, so you can immediately use it to manage and monitor your existing databases. Supported database servers/clusters:

• Galera Cluster for MySQL
• Percona XtraDB Cluster
• MariaDB Galera Cluster
• MySQL/MariaDB Replication
• MySQL/MariaDB single instance
• MongoDB Replica Set
• PostgreSQL single instance

As more and more people will know, Docker is based on the concept of so called application containers and is much faster or lightweight than full stack virtual machines such as VMWare or VirtualBox. It's a very nice way to isolate applications and services to run in a completely isolated environment, which a user can launch and tear down the whole stack within seconds.

Having a Docker image for ClusterControl at the moment is convenient in terms of how quickly it is to get it up and running and it's 100% reproducible. Docker users can now start testing ClusterControl, since we have an image that everyone can pull down and then launch the tool.

ClusterControl Docker Image

The image is available on Docker Hub and the code is hosted in our Github repository. Please refer to the Docker Hub page or our Github repository for the latest instructions.

The image consists of ClusterControl and all of its components:

• ClusterControl controller, CLI, cmonapi, UI and NodeJS packages installed via Severalnines repository.
• Percona Server 5.6 installed via Percona repository.
• Apache 2.4 (mod_ssl and mod_rewrite configured)
• PHP 5.4 (gd, mysqli, ldap, cli, common, curl)
• SSH key for root user.
• Deployment script: deploy-container.sh

The core of automatic deployment is inside a shell script called “deploy-container.sh”. This script monitors a custom table inside CMON database called “cmon.containers”. The database container created by run command or some orchestration tool shall report and register itself into this table. The script will look for new entries and perform the necessary actions using s9s, the ClusterControl CLI. You can monitor the progress directly from the ClusterControl UI or using “docker logs” command.

New Deployment - ClusterControl and Galera Cluster on Docker

There are several ways to deploy a new database cluster stack on containers, especially with respect to the container orchestration platform used. The currently supported methods are:

• Docker (standalone)
• Docker Swarm
• Kubernetes

A new database cluster stack would usually consist of a ClusterControl server with a three-node Galera Cluster. From there, you can scale up or down according to your desired setup. We have built another Docker image to serve as database base image called “centos-ssh”, which can be used with ClusterControl for automatic deployment. This image comes with a simple bootstrapping logic to download the public key from ClusterControl container (automatic passwordless SSH setup), register itself to ClusterControl and pass the environment variables for the chosen cluster setup.

Docker (Standalone)

To run on standalone Docker host, one would do the following:

1. Run the ClusterControl container:

   $ docker run -d --name clustercontrol -p 5000:80 severalnines/clustercontrol

2. Then, run the DB containers. Define the CC_HOST (the ClusterControl container's IP) environment variable or simply use container linking. Assuming the ClusterControl container name is ‘clustercontrol’:

   $ docker run -d --name galera1 -p 6661:3306 --link clustercontrol:clustercontrol -e CLUSTER_TYPE=galera -e CLUSTER_NAME=mygalera -e INITIAL_CLUSTER_SIZE=3 severalnines/centos-ssh

   $ docker run -d --name galera2 -p 6662:3306 --link clustercontrol:clustercontrol -e CLUSTER_TYPE=galera -e CLUSTER_NAME=mygalera -e INITIAL_CLUSTER_SIZE=3 severalnines/centos-ssh

   $ docker run -d --name galera3 -p 6663:3306 --link clustercontrol:clustercontrol -e CLUSTER_TYPE=galera -e CLUSTER_NAME=mygalera -e INITIAL_CLUSTER_SIZE=3 severalnines/centos-ssh

ClusterControl will automatically pick the new containers to deploy. If it finds the number of containers is equal or greater than INITIAL_CLUSTER_SIZE, the cluster deployment shall begin. You can verify that with:

$ docker logs -f clustercontrol

Or, open the ClusterControl UI at http://{Docker_host}:5000/clustercontrol and look under Activity -> Jobs.

To scale up, just run new containers and ClusterControl will add them into the cluster automatically:

$ docker run -d --name galera4 -p 6664:3306 --link clustercontrol:clustercontrol -e CLUSTER_TYPE=galera -e CLUSTER_NAME=mygalera -e INITIAL_CLUSTER_SIZE=3 severalnines/centos-ssh

Docker Swarm

Docker Swarm allows container deployment on multiple hosts. It manages a cluster of Docker Engines called a swarm.

Take note of some prerequisites for running ClusterControl and Galera Cluster on Docker Swarm:

• Docker Engine version 1.12 and later.
• Docker Swarm Mode is initialized.
• ClusterControl must be connected to the same overlay network as the database containers.

In Docker Swarm mode, centos-ssh will default to look for 'cc_clustercontrol' as the CC_HOST. If you create the ClusterControl container with 'cc_clustercontrol' as the service name, you can skip defining CC_HOST variable.

We have explained the deployment steps in this blog post.

Kubernetes

Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, and provides a container-centric infrastructure. In Kubernetes, centos-ssh will default to look for “clustercontrol” service as the CC_HOST. If you use other name, please specify this variable accordingly.

Example YAML definition of ClusterControl deployment on Kubernetes is available in our Github repository under Kubernetes directory. To deploy a ClusterControl pod using a ReplicaSet, the recommended way is:

$ kubectl create -f cc-pv-pvc.yml
$ kubectl create -f cc-rs.yml

Kubernetes will then create the necessary PersistentVolume (PV) and PersistentVolumeClaim (PVC) by connecting to the NFS server. The deployment definition (cc-rs.yml) will then run a pod and use the created PV resources to map the datadir and cmon configuration directory for persistency. This allows the ClusterControl pod to be relocated to other Kubernetes nodes if it is rescheduled by the scheduler.

Import Existing DB Containers into ClusterControl

If you already have a Galera Cluster running on Docker and you would like to have ClusterControl manage it, you can simply run the ClusterControl container in the same Docker network as the database containers. The only requirement is to ensure the target containers have SSH related packages installed (openssh-server, openssh-clients). Then allow passwordless SSH from ClusterControl to the database containers. Once done, use “Add Existing Server/Cluster” feature and the cluster should be imported into ClusterControl.

Example of importing the existing DB containers

We have a physical host, 192.168.50.130 installed with Docker, and assume that you already have a three-node Galera Cluster in containers, running under the standard Docker bridge network. We are going to import the cluster into ClusterControl, which is running in another container. The following is the high-level architecture diagram:

Adding into ClusterControl

1. Install OpenSSH related packages on the database containers, allow the root login, start it up and set the root password:

   $ docker exec -ti [db-container] apt-get update
   $ docker exec -ti [db-container] apt-get install -y openssh-server openssh-client
   $ docker exec -it [db-container] sed -i 's|^PermitRootLogin.*|PermitRootLogin yes|g' /etc/ssh/sshd_config
   $ docker exec -ti [db-container] service ssh start
   $ docker exec -it [db-container] passwd

2. Start the ClusterControl container as daemon and forward port 80 on the container to port 5000 on the host:

   $ docker run -d --name clustercontrol -p 5000:80 severalnines/clustercontrol

3. Verify the ClusterControl container is up:

   $ docker ps | grep clustercontrol
   59134c17fe5a        severalnines/clustercontrol   "/entrypoint.sh"       2 minutes ago       Up 2 minutes        22/tcp, 3306/tcp, 9500/tcp, 9600/tcp, 9999/tcp, 0.0.0.0:5000->80/tcp   clustercontrol

4. Open a browser, go to http://{docker_host}:5000/clustercontrol and create a default admin user and password. You should now see the ClusterControl landing page.

5. The last step is setting up the passwordless SSH to all database containers. Attach to ClusterControl container interactive console:

   $ docker exec -it clustercontrol /bin/bash

6. Copy the SSH key to all database containers:

   $ ssh-copy-id 172.17.0.2
   $ ssh-copy-id 172.17.0.3
   $ ssh-copy-id 172.17.0.4

7. Start importing the cluster into ClusterControl. Open a web browser and go to Docker’s physical host IP address with the mapped port e.g, http://192.168.50.130:5000/clustercontrol and click “Add Existing Cluster/Server” and specify following information:

   Ensure you got the green tick when entering the hostname or IP address, indicating ClusterControl is able to communicate with the node. Then, click Import button and wait until ClusterControl finishes its job. The database cluster will be listed under the ClusterControl dashboard once imported.

Happy containerizing!

While the idea of containers have been around since the early days of Unix, Docker made waves in 2013 when it hit the market with its innovative solution. What began as an open source project, Docker allows you to add your stacks and applications to containers where they share a common operating system kernel. This lets you have a lightweight virtualized system with almost zero overhead. Docker also lets you bring up or down containers in seconds, making for rapid deployment of your stack.

Severalnines, like many other companies, got excited early on about Docker and began experimenting and developing ways to deploy advanced open source database configurations using Docker containers. We also released, early on, a docker image of ClusterControl that lets you utilize the management and monitoring functionalities of ClusterControl with your existing database deployments.

Here are just some of the great resources we’ve developed for Docker over the last few years...

Severalnines on Docker Hub

In addition to the ClusterControl Docker Image, we have also provided a series of images to help you get started on Docker with other open source database technologies like Percona XtraDB Cluster and MariaDB.

Check Out the Docker Images

ClusterControl on Docker Documentation

For detailed instructions on how to install ClusterControl utilizing the Docker Image click on the link below.

Read More

Top Blogs

MySQL on Docker: Running Galera Cluster on Kubernetes

In our previous posts, we showed how one can run Galera Cluster on Docker Swarm, and discussed some of the limitations with regards to production environments. Kubernetes is widely used as orchestration tool, and we’ll see whether we can leverage it to achieve production-grade Galera Cluster on Docker.

Read More

MySQL on Docker: Swarm Mode Limitations for Galera Cluster in Production Setups

This blog post explains some of the Docker Swarm Mode limitations in handling Galera Cluster natively in production environments.

Read More

MySQL on Docker: Composing the Stack

Docker 1.13 introduces a long-awaited feature called Compose-file support. Compose-file defines everything about an application - services, databases, volumes, networks, and dependencies can all be defined in one place. In this blog, we’ll show you how to use Compose-file to simplify the Docker deployment of MySQL containers.

Read More

MySQL on Docker: Deploy a Homogeneous Galera Cluster with etcd

Our journey to make Galera Cluster run smoothly on Docker containers continues. Deploying Galera Cluster on Docker is tricky when using orchestration tools. With this blog, find out how to deploy a homogeneous Galera Cluster with etcd.

Read More

MySQL on Docker: Introduction to Docker Swarm Mode and Multi-Host Networking

This blog post covers the basics of managing MySQL containers on top of Docker swarm mode and overlay network.

Read More

MySQL on Docker: Single Host Networking for MySQL Containers

This blog covers the basics of how Docker handles single-host networking, and how MySQL containers can leverage that.

Read More

MySQL on Docker: Building the Container Image

In this post, we will show you two ways how to build a MySQL Docker image - changing a base image and committing, or using Dockerfile. We’ll show you how to extend the Docker team’s MySQL image, and add Percona XtraBackup to it.

Read More

MySQL Docker Containers: Understanding the basics

In this post, we will cover some basics around running MySQL in a Docker container. It walks you through how to properly fire up a MySQL container, change configuration parameters, how to connect to the container, and how the data is stored.

Read More

ClusterControl on Docker

ClusterControl provides advanced management and monitoring functionality to get your MySQL replication and clustered instances up-and-running using proven methodologies that you can depend on to work. Used in conjunction with other orchestration tools for deployment to the containers, ClusterControl makes managing your open source databases easy with point-and-click interfaces and no need to have specialized knowledge about the technology.

ClusterControl delivers on an array of features to help manage and monitor your open source database environments:

• Management & Monitoring: ClusterControl provides management features to repair and recover broken nodes, as well as test and automate MySQL upgrades.
• Advanced Monitoring: ClusterControl provides a unified view of all MySQL nodes and clusters across all your data centers and lets you drill down into individual nodes for more detailed statistics.
• Automatic Failure Detection and Handling: ClusterControl takes care of your replication cluster’s health. If a master failure is detected, ClusterControl automatically promotes one of the available slaves to ensure your cluster is always up.

Learn more about how ClusterControl can enhance performance here or pull the Docker Image here.

We hope that these resources prove useful!

Happy Clustering!

In our “MySQL on Docker” blog series, we continue our quest to make Galera Cluster run smoothly in different container environments. One of the most important things when running a database service, whether in containers or bare-metal, is to eliminate the risk of data loss. We will see how we can leverage a promising feature in Kubernetes called StatefulSet, which orchestrates container deployment in a more predictable and controllable fashion.

In our previous blog post, we showed how one can deploy a Galera Cluster within Docker with the help of Kubernetes as orchestration tool. However, it is only about deployment. Running a database in production requires more than just deployment - we need to think about monitoring, backups, upgrades, recovery from failures, topology changes and so on. This is where ClusterControl comes into the picture, as it completes the stack and makes it production ready. In simple words, Kubernetes takes care of database deployment and scaling while ClusterControl fills in the missing components including configuration, monitoring and ongoing management.

ClusterControl on Docker

This blog post describes how ClusterControl runs in a Docker environment. The Docker image has been updated, and now comes with the standard ClusterControl packages in the latest stable branch with additional support for container orchestration platforms like Docker Swarm and Kubernetes, we’ll describe this further below. You can also use the image to deploy a database cluster on a standalone Docker host.

Details at the Github repository or Docker Hub page.

ClusterControl on Kubernetes

The updated Docker image now supports automatic deployment of database containers scheduled by Kubernetes. The steps are similar to the Docker Swarm implementation, where the user decides the specs of the database cluster and ClusterControl automates the actual deployment.

ClusterControl can be deployed as ReplicaSet or StatefulSet. Since it’s a single instance, either way works. The only significant difference is the container identification would be easier with StatefulSet, since it provides consistent identifiers like the container hostname, IP address, DNS and storage. ClusterControl also provides service discovery for the new cluster deployment.

To deploy ClusterControl on Kubernetes, the following setup is recommended:

• Use centralized persistent volumes supported by Kubernetes plugins (e.g. NFS, iSCSI) for the following paths:
  • /etc/cmon.d - ClusterControl configuration directory
  • /var/lib/mysql - ClusterControl cmon and dcps databases
• Create 2 services for this pod:
  • One for internal communication between pods (expose port 80 and 3306)
  • One for external communication to outside world (expose port 80, 443 using NodePort or LoadBalancer)

In this example, we are going to use simple NFS. Make sure you have an NFS server ready. For the sake of simplicity, we are going to demonstrate this deployment on a 3-host Kubernetes cluster (1 master + 2 Kubernetes nodes). For production use, please use at least 3 Kubernetes nodes to minimize the risk of losing quorum.

With that in place, we can deploy the ClusterControl as something like this:

On the NFS server (kube1.local), install NFS server and client packages and export the following paths:

• /storage/pods/cc/cmon.d - to be mapped with /etc/cmon.d
• /storage/pods/cc/datadir - to be mapped with /var/lib/mysql

Ensure to restart NFS service to apply the changes. Then create PVs and PVCs, as shown in cc-pv-pvc.yml:

$ kubectl create -f cc-pv-pvc.yml

We are now ready to start a replica of the ClusterControl pod. Send cc-rs.yml to Kubernetes master:

$ kubectl create -f cc-rs.yml

ClusterControl is now accessible on port 30080 on any of the Kubernetes nodes, for example, http://kube1.local:30080/clustercontrol. With this approach (ReplicaSet + PV + PVC), the ClusterControl pod would survive if the physical host goes down. Kubernetes will automatically schedule the pod to the other available hosts and ClusterControl will be bootstrapped from the last existing dataset which is available through NFS.

Galera Cluster on Kubernetes

If you would like to use the ClusterControl automatic deployment feature, simply send the following YAML files to the Kubernetes master:

$ kubectl create -f cc-galera-pv-pvc.yml
$ kubectl create -f cc-galera-ss.yml

Details on the definition files can be found here - cc-galera-pv-pvc.yml and cc-galera-ss.yml.

The above commands tell Kubernetes to create 3 PVs, 3 PVCs and 3 pods running as StatefulSet using a generic base image called “centos-ssh”. In this example, the database cluster that we are going to deploy is MariaDB 10.1. Once the containers are started, they will register themselves to ClusterControl CMON database. ClusterControl will then pick up the containers’ hostname and start the deployment based on the variables that have been passed.

You can check the progress directly from the ClusterControl UI. Once the deployment has finished, our architecture will look something like this:

HAProxy as Load Balancer

Kubernetes comes with an internal load balancing capability via the Service component when distributing traffic to the backend pods. This is good enough if the balancing (least connections) fits your workload. In some cases, where your application needs to send queries to a single master due to deadlock or strict read-after-write semantics, you have to create another Kubernetes service with a proper selector to redirect the incoming connection to one and only one pod. If this single pod goes down, there is a chance of service interruption when Kubernetes schedules it again to another available node. What we are trying to highlight here is that if you’d want better control on what’s being sent to the backend Galera Cluster, something like HAProxy (or even ProxySQL) is pretty good at that.

You can deploy HAProxy as a two-pod ReplicaSet and use ClusterControl to deploy, configure and manage it. Simply post this YAML definition to Kubernetes:

$ kubectl create -f cc-haproxy-rs.yml

The above definition instructs Kubernetes to create a service called cc-haproxy and run two replicas of “severalnines/centos-ssh” image without automatic deployment (AUTO_DEPLOYMENT=0). These pods will then connect to the ClusterControl pod and perform automatic passwordless SSH setup. What you need to do now is to log into ClusterControl UI and start the deployment of HAProxy.

Firstly, retrieve the IP address of HAProxy pods:

$ kubectl describe pods -l app=cc-haproxy | grep IP
IP:        10.44.0.6
IP:        10.36.0.5

Then use the address as the HAProxy Address under ClusterControl -> choose the DB cluster -> Manage -> Load Balancer -> HAProxy -> Deploy HAProxy, as shown below:

**Repeat the above step for the second HAProxy instance.

Once done, our Galera Cluster can be accessible through the “cc-haproxy” service on port 3307 internally (within Kubernetes network space) or port 30016 externally (outside world). The connection will be load balanced between these HAProxy instances. At this point, our architecture can be illustrated as the following:

With this setup, you have maximum control of your load-balanced Galera Cluster running on Docker. Kubernetes brings something good to the table by supporting stateful service orchestration.

Do give it a try. We would love to hear how you get along.

We’re excited to announce the live version of our blog ‘MySQL on Docker: Understanding the Basics’, which will be presented by its author, our colleague Ashraf Sharif, on September 27th during this new webinar.

With 100K+ views to date, ‘MySQL on Docker: Understanding the Basics’ has become a popular go-to resource for MySQL users worldwide who are looking to get an initial understanding of and start experimenting with Docker.

So if you’re looking at taking your first steps with Docker for MySQL then this webinar is made for you :-)

Docker is quickly becoming mainstream as a method to package and deploy self-sufficient applications in primarily stateless Linux containers. This could be a challenge though for a stateful service like a database. As a database user you might be asking yourself: How do I best configure MySQL in a container environment? What can go wrong? Should I even run my databases in a container environment? How does performance compare with e.g. running on virtual machines or bare-metal servers? How do I manage replicated or clustered setups, where multiple containers need to be created, upgraded and made highly available? We’ll look at helping you answer these questions with our new Docker webinar series.

Check out the agenda and sign up for its first installment below, we look forward to “seeing” you there.

Date, Time & Registration

Europe/MEA/APAC

Wednesday, September 27th at 09:00 BST / 10:00 CEST (Germany, France, Sweden)

Register Now

North America/LatAm

Wednesday, September 27th at 09:00 PST (US) / 12:00 EST (US)

Register Now

Agenda

This webinar is for MySQL users who are Docker beginners and who would like to understand the basics of running a MySQL container on Docker. We are going to cover:

• Docker and its components
• Concept and terminology
• How a Docker container works
• Advantages and disadvantages
• Stateless vs stateful
• Docker images for MySQL
• Running a simple MySQL container
• The ClusterControl Docker image
• Severalnines on Docker Hub

Speaker

Ashraf Sharif is System Support Engineer at Severalnines. He was previously involved in hosting world and LAMP stack, where he worked as principal consultant and head of support team and delivered clustering solutions for large websites in the South East Asia region. His professional interests are on system scalability and high availability.

Yes, it’s that time of year again: the Percona Live Europe Conference is just around the corner.

And we’ll be broadcasting live from the Emerald Isle!

Quite literally, since we’re planning to be on our Twitter and Facebook channels during the course of the conference, so do make sure to tune in (if you’re not attending in person). And this year’s location is indeed Dublin, Ireland, so expect lots of banter and a bit of craic.

More specifically though, the Severalnines team will be well represented with three talks and a booth in the exhibition area. So do come and meet us there. If you haven’t registered yet, there’s still time to do so on the conference website: https://www.percona.com/live/e17/

Our talks at the conference:

Ashraf Sharif, Senior Support Engineer will be talking about MySQL on Docker and containerised dolphins (which only sounds like a good idea in the database world).

Krzysztof Książek, Senior Support Engineer, will share his knowledge and experience on all things MySQL load balancing.

And Ruairí Newman, also Senior Support Engineer, will be discussing what some of the main considerations are to think through when looking at automating and managing MongoDB - including a closer look at MongoDB Ops Manager and ClusterControl.

And since we don’t solely employ Senior Support Engineers at Severalnines, you’ll be pleased to know that Andrada Enache, Sales Manager, and myself will also be present to talk open source database management with ClusterControl at our booth.

This year’s conference agenda looks pretty exciting overall with a wide enough range of topics, so there’ll be something of interest for every open source database aficionado out there.

See you in Dublin ;-)

Severalnines was pleased to yet again sponsor Percona Live Europe, the Open Source Database Conference which was held this year in Dublin, Ireland.

At the Conference

Severalnines team members flew in from around the world to partner up with our two local Dubliners to demo ClusterControl in the exhibit hall and present three sessions (see below).

On our Twitter feed we live tweeted both of the keynote sessions to help keep those who weren’t able to attend up-to-speed on the latest happenings in the database world.

We were also able to sit down with René Cannaò, creator of ProxySQL to talk about what’s new with the exciting load balancing technology.

Our Sessions

Members of the Severalnines team presented three technical sessions, all of which were widely attended… some with standing room only!

MySQL Load Balancers - MaxScale, ProxySQL, HAProxy, MySQL Router & nginx - A Close Up Look

Session Details: Load balancing MySQL connections and queries using HAProxy has been popular in the past years. Recently however, we have seen the arrival of MaxScale, MySQL Router, ProxySQL and now also Nginx as a reverse proxy.

For which use cases do you use them and how well do they integrate in your environment? This session aims to give a solid grounding in load balancer technologies for MySQL and MariaDB.

We review the main open-source options available: from application connectors (php-mysqlnd, jdbc), TCP reverse proxies (HAproxy, Keepalived, Nginx) and SQL-aware load balancers (MaxScale, ProxySQL, MySQL Router).

We also look into the best practices for backend health checks to ensure load balanced connections are routed to the correct nodes in several MySQL clustering topologies. You'll gain a good understanding of how the different options compare, and enough knowledge to decide which ones to explore further.

MySQL on Docker - Containerizing the Dolphin

Session Details: Docker is becoming more mainstream and adopted by users as a method to package and deploy self-sufficient applications in primarily stateless Linux containers. It's a great toolset on top of OS-level virtualization (LXC, a.k.a containers) and plays well in the world of micro services.

However, Docker containers are transient by default. If a container is destroyed, all data created is also lost. For a stateful service like a database, this is a major headache to say the least.

There are a number ways to provide persistent storage in Docker containers. In this presentation, we will talk about how to setup a persistence data service with Docker that can be torn down and brought up across hosts and containers.

We touch upon orchestration tools, shared volumes, data-only-containers, security and configuration management, multi-host networking, service discovery and implications on monitoring when we move from host-centric to role-centric services with shorter life cycles.

Automating and Managing MongoDB: An Analysis of Ops Manager vs. ClusterControl

Session Details: In any busy operations environment, there are countless tasks to perform - some monthly, or weekly, some daily or more frequently, and some on an ad-hoc basis. And automation is key to performing fast, efficient and consistently repeatable software deployments and recovery.

There are many generic tools available, both commercial and open source, to aid with the automation of operational tasks. Some of these tools are even deployed in the database world. However, there are a small number of specialist domain-specific automation tools available also, and we are going to compare two of these products: MongoDB?s own Ops Manager, and ClusterControl from Severalnines.

We cover Installation and maintenance, Complexity of architecture, Options for redundancy, Comparative functionality, Monitoring, Dashboard, Alerting, Backing up and restoring, Automated deployment of advanced configurations, and Upgrading existing deployments

Thanks to the Percona Team for organising another great conference and to everyone who participated from near and afar! We hope to see you again soon!