Tech Blog

While I was working on the active/active iSCSI cluster, I began wondering if the same technology and concepts could be used to create a pair of active/active Apache/MySQL servers.  Turns out it can.  The primary issue to keep in mind is that in this setup, the potential exists for running 2 instances of both Apache and MySQL (actually MariaDB which is a drop-in replacement) on the same system.  So long as you have unique sockets (IP address + port) and the resources on each system to do so, that shouldn't be a problem.

Let's start with some details on my setup:

- node3 - CentOS 7 virtual machine with two NICs, one with an IP of 192.168.1.190 and one with 172.16.0.3
- node4 - CentOS 7 virtual machine with two NICs, one with an IP of 192.168.1.191 and one with 172.16.0.4
- VIP1 - A floating virtual IP of 192.168.1.192
- VIP2 - A floating virtual IP of 192.168.1.193
Each machine has two 10 GB partition (in my case /dev/xvdb and /dev/xvdc) that will be used for the clustered data drives. Note also that the IP addresses in the 172.16.0 subnet will be used for cluster communications while the 192.168.1 subnet is intended for access to Apache.

We will create 2 LVM volume groups with 2 logical volumes per group.  I found it useful to keep track of what each volume on each system would contain so here is my cheat sheet:
/dev/mapper/vg01-lv1 - Apache files for instance running on VIP1
/dev/mapper/vg01-lv2 - MySQL databases for instance running on VIP1
/dev/mapper/vg02-lv1 - Apache files for instance running on VIP2
/dev/mapper/vg02-lv2 - MySQL databases for instance running on VIP2

So the idea is that one instance of Apache will run on port 80/443 on VIP1 and the other Apache instance will run on port 80/443 on VIP2.  One instance of MySQL will run on port 3306 on VIP1 and the other MySQL instance will run on port 3306 on VIP2.  When both nodes are healthy, one instance of each will reside on each node.  If one node fails or needs to be taken down for maintenance, both instances of Apache and MySQL will be able to run on the remaining node.

Let's start by making sure that everything resolves as it should by running the following command on both nodes and note you may have to hit <ENTER> after the EOF below and make sure to change your names/address as needed:
cat <<'EOF' >> /etc/hosts
192.168.1.190 node3 node3.theharrishome.lan
192.168.1.191 node4 node4.theharrishome.lan
172.16.0.3 node3-ha node3-ha.theharrishome.lan
172.16.0.4 node4-ha node4-ha.theharrishome.lan
192.168.1.192 vip1 vip1.theharrishome.lan
192.168.1.193 vip2 vip2.theharrishome.lan
EOF

Before moving forward, verify that each can ping everything by name. Here is what my /etc/hosts file now looks like on both nodes:
hosts file

And now to install some software on both nodes:
# yum -y install corosync pacemaker pcs ntp policycoreutils-python mariadb-server httpd php

Let’s make sure we have accurate time on each system. We’ll configure NTP to take care of that as follows:
# systemctl enable ntpd
# ntpdate pool.ntp.org
# systemctl start ntpd

I am leaving SELinux enabled so we need to tell it that DRBD is legit with the following command on both nodes:
# semanage permissive -a drbd_t

We are also leaving the firewall (firewalld) on so we need to allow our cluster traffic through. We could allow them one at a time but the following command will take care of many of the services so let's make use of it:
# firewall-cmd --permanent --add-service=high-availability

And for Apache http and https:
# firewall-cmd --permanent --add-service=http
# firewall-cmd --permanent --add-service=https

And for MySQL:
# firewall-cmd --permanent --add-service=mysql

Now we need to allow DRBD traffic:
# firewall-cmd --permanent --add-port=7788-7799/tcp

Now let’s reload the firewall with our new settings:
# firewall-cmd --reload

Now we need to install a repository named ELrepo so we can install the tools necessary to work with DRBD. Actually DRBD is part of modern Linux kernels such as the one shipped with CentOS 7 but unfortunately the tools are not.
# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

Then install the repo:
# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

And install the necessary DRBD tools:
# yum -y install drbd84-utils kmod-drbd84

We are ready to configure DRBD. We are going to configure 2 DRBD resources so each node will have one when it is up and healthy. Copy the following config file to /etc/drbd.d/r0.res MAKING SURE to change the host names as well as volume info to match your setup. Note the use of IP addresses instead of names. That is a requirement. 

resource vg01 {
on node3.theharrishome.lan {
volume 0 {
device /dev/drbd0;
disk /dev/xvdb;
flexible-meta-disk internal;
}
address 172.16.0.3:7788;
}
on node4.theharrishome.lan {
volume 0 {
device /dev/drbd0;
disk /dev/xvdb;
flexible-meta-disk internal;
}
address 172.16.0.4:7788;
}
}

And for the second resource, we will use /etc/drbd.d/r1.res which contains the following and notice this resource uses a different port (7789) than the first one:

resource vg02 {
on node3.theharrishome.lan {
volume 0 {
device /dev/drbd1;
disk /dev/xvdc;
flexible-meta-disk internal;
}
address 172.16.0.3:7789;
}
on node4.theharrishome.lan {
volume 0 {
device /dev/drbd1;
disk /dev/xvdc;
flexible-meta-disk internal;
}
address 172.16.0.4:7789;
}
}

Now let’s initialize the DRBD metadata on both resources on both nodes:
# drbdadm create-md vg01
# drbdadm create-md vg02

Now we need to load the DRBD kernel module on both nodes:
# modprobe drbd

To set the module to load at boot:
# echo drbd >> /etc/modules-load.d/drbd.conf

And bring both resources up on both nodes:
# drbdadm up vg01
# drbdadm up vg02

Now we need to set one of the nodes as the primary for both resources. To do that from node3:
# drbdadm primary --force vg01
# drbdadm primary --force vg02

It should now begin to synchronize and this will probably take a while. You can always check the status as follows:
# cat /proc/drbd

Here is what mine looks like while it is synchronizing:
drbd initial sync

When the sync is finished, Primary/Secondary will show UpToDate/UpToDate instead of UpToDate/Inconsistent as shown above.

Now we need to run corosync-keygen so that the communication between nodes is encrypted. We only need to do this on one node as we will then copy the key to the other node. The command to generate the key is as follows and note also that it may take a few minutes to complete:
# corosync-keygen

You will be asked to press keys on your keyboard to generate entropy. Then wait . . .

Time to create the corosync config file of /etc/corosync/corosync.conf. The installation comes with an example config file at /etc/corosync/corosync.conf.example that you could copy and modify but to get started, here is an example of the one I am using. Note that I am using rrp_mode of passive and two rings for redundancy with the primary being an address used specifically for cluster traffic. Make sure to change the entries as noted to match your environment.

totem {
version: 2
secauth: off
cluster_name: cluster1
transport: udpu
rrp_mode: passive
}
nodelist {
node {
ring0_addr: node3-ha.theharrishome.lan
ring1_addr: node3.theharrishome.lan
nodeid: 1
}
node {
ring0_addr: node4-ha.theharrishome.lan
ring1_addr: node4.theharrishome.lan
nodeid: 2
}
}
quorum {
provider: corosync_votequorum
two_node: 1
}
logging {
to_logfile: yes
logfile: /var/log/cluster/corosync.log
to_syslog: yes
}

You will need to copy this to each node along with the key generated above which should be at /etc/corosync/authkey. The command I used from node3 was as follows:
# scp /etc/corosync/corosync.conf /etc/corosync/authkey This email address is being protected from spambots. You need JavaScript enabled to view it.:/etc/corosync/

Now let's enable and start the pcsd service on both nodes:
# systemctl enable pcsd.service;systemctl start pcsd.service

Let's change the password for the hacluster user account that was created when pcs was installed. The standard passwd command can be used. It is a good idea to set the hacluster password the same on both nodes. So the following should do the trick:
# passwd hacluster

Now we need to authenticate pcs to both nodes using the hacluster user and the password we just set. We do this from a single node:
# pcs cluster auth node3-ha.theharrishome.lan node4-ha.theharrishome.lan

If you see any errors in the output, good places to begin looking would include the firewall and to verify pcsd is running.  Here is what mine looks like:
pcs cluster auth

At this point, we should have all we need for pcs such that any change we make via the pcs command will make the necessary changes on both nodes. So let’s set up our cluster from a single node as follows:
# pcs cluster setup --name cluster1 node3-ha.theharrishome.lan node4-ha.theharrishome.lan --force

And a view of mine:
pcs cluster setup

Now we should be able to start the cluster. The following command ran from one node should start all services on both nodes:
# pcs cluster start --all

Some people recommend NOT setting corosync and pacemaker to run on startup but I have always preferred they start automatically. The following command will set them both to run at startup:
# systemctl enable corosync;systemctl enable pacemaker

Now we should be able to see the status of our new cluster with the following:
# pcs status

Here is what mine looks like:
pcs initial status

Notice the warning at the top that says "WARNING: no stonith devices and stonith-enabled is not false". I have covered stonith in previous posts so for now, I will just disable it with the following command on either node:

# pcs property set stonith-enabled=false

While we are at it, we need to also set the no-quorum-policy to ignore since we only have two nodes and a failure in one will cause a loss of quorum:
# pcs property set no-quorum-policy=ignore

That should get rid of the warning. Now we can focus on setting up the logical volumes. As mentioned in the introduction, I am going to use one logical volume per DRBD resource so we will have a total of two with one being on each node when the cluster is healthy. You could add more as you see fit. Here are the commands I used to set mine up and note this is done from the node that shows as primary for the DRBD resources (issue command cat /proc/drbd to find out which that is):
# vgcreate vg01 /dev/drbd0
# vgcreate vg02 /dev/drbd1
# lvcreate -L 5G -n lv1 vg01
# lvcreate -L 4G -n lv2 vg01
# lvcreate -L 5G -n lv1 vg02
# lvcreate -L 4G -n lv2 vg02

Let's go ahead and format our two logical volumes:
# mkfs.ext4 /dev/mapper/vg01-lv1
# mkfs.ext4 /dev/mapper/vg01-lv2
# mkfs.ext4 /dev/mapper/vg02-lv1
# mkfs.ext4 /dev/mapper/vg02-lv2

We need to figure out what volume groups are configured for local storage. We can do that by issuing the following command from the node we just used to create the volumes:
# vgs --noheadings -o vg_name

Here is my output:vgs vg names

Let's start setting up MySQL.  Since we will have the potential of having 2 MySQL instances running on the same node at the same time, we need 2 different configuration files with each binding to a different socket and pointing to a different location for database storage.  Let's create the mount points first on both nodes.  Since they are each bound to a specific VIP, I am using that in the name so we can easily tell which is which:

# mkdir /var/lib/mysql_vip1
# mkdir /var/lib/mysql_vip2

We need to make sure the permissions are correct for both directories on both nodes:
# chown mysql.mysql /var/lib/mysql_vip1
# chown mysql.mysql /var/lib/mysql_vip2

Now we need to start MySQL so we can initialize it and create the default system tables. We need to do this on both nodes:
# systemctl start mariadb

Next, we need to temporarily mount the logical volumes that will hold the database information for each instance.  First lets vg01-lv2:
# mount /dev/mapper/vg01-lv2 /mnt

We need to set up the MySQL system tables as follows:
# mysql_install_db --datadir=/mnt --user=mysql

I want to secure the installation as well so I ran the following:
# mysql_secure_installation

Lets set up SELinux so it is OK with the new MySQL data directory we have mounted:
# semanage fcontext -a -t mysqld_db_t "/mnt(/.*)?"

Now we need to apply this to the running system:
# restorecon -R -v /mnt

Now we can unmount the first logical volume:
# umount /mnt

Next, we need to repeat the process for /dev/mapper/vg02-lv2.  Here are the command all together to do that:
# mount /dev/mapper/vg02-lv2 /mnt
# mysql_install_db --datadir=/mnt --user=mysql
# mysql_secure_installation
# umount /mnt
# semanage fcontext -a -t mysqld_db_t "/mnt(/.*)?"
# umount /mnt

We are done with the MySQL daemon for now so let's stop it AND make sure it is not enabled on BOTH nodes as we want the cluster to take care of that::
# systemctl stop mariadb
# systemctl disable mariadb

We need to create a separate config file for each MySQL instance on BOTH nodes.  We are binding the first one to the IP address of VIP1.  We do that with the following and make sure to hit enter after EOL:

cat << EOL > /etc/my_vip1.cnf
[mysqld]
symbolic-links=0
bind_address = 192.168.1.192
datadir = /var/lib/mysql_vip1
pid_file = /var/run/mariadb/mysqld_vip1.pid
socket = /var/run/mariadb/mysqld_vip1.sock

[mysqld_safe]
bind_address = 192.168.1.192
datadir = /var/lib/mysql_vip1
pid_file = /var/run/mariadb/mysqld_vip1.pid
socket = /var/run/mariadb/mysqld_vip1.sock

!includedir /etc/my.cnf.d
EOL

And here is the command for the second instance that runs on VIP2 and note the change in IP for bind_address.  Again this should be run on BOTH nodes:
cat << EOL > /etc/my_vip2.cnf
[mysqld]
symbolic-links=0
bind_address = 192.168.1.193
datadir = /var/lib/mysql_vip2
pid_file = /var/run/mariadb/mysqld_vip2.pid
socket = /var/run/mariadb/mysqld_vip2.sock

[mysqld_safe]
bind_address = 192.168.1.193
datadir = /var/lib/mysql_vip2
pid_file = /var/run/mariadb/mysqld_vip2.pid
socket = /var/run/mariadb/mysqld_vip2.sock

!includedir /etc/my.cnf.d
EOL

Let's go ahead and remove the default MySQL data directory on both nodes just so we don't later forget that we are not actually using it:
# rm /var/lib/mysql -rfv

That takes care of MySQL so we now turn our attention to Apache.  Let's start by copying the existing Apache configuration file to new configuration files, one for each instance on both nodes:
# cp /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd_vip1.conf
# cp /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd_vip2.conf

Now we can remove the original httpd.conf file from both nodes so as not to get confused later:
# rm /etc/httpd/conf/httpd.conf

We need to edit both of our new Apache config files.  Here is rundown of how the changes for /etc/httpd/conf/httpd_vip1.conf:
Change
Listen 80
To
Listen 192.168.1.192:80

Change
DocumentRoot "/var/www/html"
To
DocumentRoot "/var/www_vip1/html"

Change
IncludeOptional conf.d/*.conf
To
IncludeOptional /var/www_vip1/conf.d/*.conf

Change the first line of the following:
<Directory "/var/www">
   AllowOverride None
   # Allow open access:
  Require all granted
</Directory>

So it looks like this:
<Directory "/var/www_vip1/html">
   AllowOverride None
   # Allow open access:
  Require all granted
</Directory>

And we need to add the following line anywhere within the file:
PidFile /var/run/httpd_vip1.pid

Now we need to make changes to /etc/httpd/conf/httpd_vip2.conf:

Change
Listen 80
To
Listen 192.168.1.193:80

Change
DocumentRoot "/var/www/html"
To
DocumentRoot "/var/www_vip2/html"

Change
IncludeOptional conf.d/*.conf
To
IncludeOptional /var/www_vip2/conf.d/*.conf

Change the first line of the following:
<Directory "/var/www">
   AllowOverride None
   # Allow open access:
  Require all granted
</Directory>

So it looks like this:
<Directory "/var/www_vip1/html">
   AllowOverride None
   # Allow open access:
  Require all granted
</Directory>

And we need to add the following line anywhere within the file:
PidFile /var/run/httpd_vip2.pid

Finally we need to make the mount directories for both resources on both nodes:
# mkdir /var/www_vip1
# mkdir /var/www_vip2

Now we need to mount the logical volumes so we can create the directories for Apache:
# mount /dev/mapper/vg01-lv1 /mnt
# mkdir /mnt/html
# mkdir /mnt/conf.d
# umount /mnt

Now let's do the same thing for the next volume on both nodes:
# mount /dev/mapper/vg02-lv1 /mnt
# mkdir /mnt/html
# mkdir /mnt/conf.d
# umount /mnt

Now we can turn our attention to LVM.  We need to figure out what volume groups are configured for local storage. We can do that by issuing the following command on the node we configured LVM on:
# vgs --noheadings -o vg_name

Here is my output:
GRAPHIC vgs-vg_names.jpg

You can see that my local install of CentOS 7 makes use of a volume group named "centos". The other two listed are the volume groups I created above. We want the cluster to handle all volumes EXCEPT those for the local group. The following entry in /etc/lvm/lvm.conf will take care of that for me and note that if you had multiple volumes, you could add them with a comma between entries as shown in the examples within the file:

volume_list = [ "centos" ]

Now we need to tell LVM to read the physical volume signatures from the DRBD device and not the block devices that make it up. So we edit /etc/lvm/lvm.conf and modify the filter setting. All of my devices start with /dev/xvd so my entry is as follows on both nodes:
filter = [ "r|/dev/xvd.*|" ]

It is also recommended to disable the LVM write cache in /etc/lvm/lvm.conf which is done with the following setting:
write_cache = 0

Now issue the following command on both nodes to ensure that locking_type is set to 1 and that use_lvmetad is set to 0 in the /etc/lvm/lvm.conf file. This command also disables and stops any lvmetad processes immediately.
# lvmconf --enable-halvm --services --startstopservices

Finally, let's deactiave the two LVM volume groups:
# vgchange -an vg01
# vgchange -an vg02

And after all that setup, we can finally begin to create our cluster resources. I am not going into detail for each resource as I have covered most before and I think it is relatively straight forward to figure out what each does. First, we will create the physical DRBD resources:
# pcs resource create p_drbd_vg01 ocf:linbit:drbd drbd_resource=vg01 op monitor interval=30s
# pcs resource create p_drbd_vg02 ocf:linbit:drbd drbd_resource=vg02 op monitor interval=30s

Now we need to clone them since they must run on both nodes with only one being the master of each:
# pcs resource master ms_drbd_vg01 p_drbd_vg01 master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
# pcs resource master ms_drbd_vg02 p_drbd_vg02 master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

A cleanup of the resources using the following command should clean things up:
# pcs resource cleanup

Here is what mine now looks like after:
pcs status 1

At this point, we should be able to begin adding our cluster resources.  Note I am adding each resource to one of 2 group, Group1 and Group 2.  Let's start by adding our VIPs and note we add all our resources from a single node:
# pcs resource create vip1 IPaddr2 ip=192.168.1.192 cidr_netmask=24 --group Group1
# pcs resource create vip2 IPaddr2 ip=192.168.1.193 cidr_netmask=24 --group Group2

Since the previous 2 commands also set up our 2 groups, let's go ahead and add a colocation constraint:
# pcs constraint colocation add Group1 with ms_drbd_vg01 INFINITY with-rsc-role=Master
# pcs constraint colocation add Group2 with ms_drbd_vg02 INFINITY with-rsc-role=Master

And let's make sure our DRBD resources on each before our group resources start:
# pcs constraint order promote ms_drbd_vg01 then start Group1
# pcs constraint order promote ms_drbd_vg02 then start Group2

And next come our volume groups:
# pcs resource create vg01 LVM volgrpname=vg01 exclusive=true --group Group1
# pcs resource create vg02 LVM volgrpname=vg02 exclusive=true --group Group2

Now comes our file systems:
# pcs resource create http_fs_vip1 Filesystem device=/dev/mapper/vg01-lv1 directory=/var/www_vip1 fstype=ext4 --group Group1
# pcs resource create http_fs_vip2 Filesystem device=/dev/mapper/vg02-lv1 directory=/var/www_vip2 fstype=ext4 --group Group2
# pcs resource create mysql_fs_vip1 Filesystem device="/dev/mapper/vg01-lv2" directory="/var/lib/mysql_vip1" fstype="ext4" --group Group1
# pcs resource create mysql_fs_vip2 Filesystem device="/dev/mapper/vg02-lv2" directory="/var/lib/mysql_vip2" fstype="ext4" --group Group2

Next comes our Apache and MySQL instances:
# pcs resource create http_service_vip1 ocf:heartbeat:apache configfile=/etc/httpd/conf/httpd_vip1.conf op monitor interval=30s --group Group1
# pcs resource create http_service_vip2 ocf:heartbeat:apache configfile=/etc/httpd/conf/httpd_vip2.conf op monitor interval=30s --group Group2
pcs resource create mysql_service_vip1 ocf:heartbeat:mysql binary="/usr/bin/mysqld_safe" config="/etc/my_vip1.cnf" datadir="/var/lib/mysql_vip1" pid="/var/lib/mysql_vip1/mysql.pid" socket="/var/lib/mysql_vip1/mysql.sock" additional_parameters="--bind-address=192.168.1.192" op start timeout=60s op stop timeout=60s op monitor interval=20s timeout=30s --group Group1
pcs resource create mysql_service_vip2 ocf:heartbeat:mysql binary="/usr/bin/mysqld_safe" config="/etc/my_vip2.cnf" datadir="/var/lib/mysql_vip2" pid="/var/lib/mysql_vip2/mysql.pid" socket="/var/lib/mysql_vip2/mysql.sock" additional_parameters="--bind-address=192.168.1.193" op start timeout=60s op stop timeout=60s op monitor interval=20s timeout=30s --group Group2

And to finish up, let's set each group so it prefers a specific node to run on:
# pcs constraint location Group1 prefers node3-ha.theharrishome.lan=50
# pcs constraint location Group2 prefers node4-ha.theharrishome.lan=50

So here is a screen shot of what mine looks like with everything set up and working
pcs status final

And here is a screen shot of my constraints:
pcs constraint final

Now that we have everything set up, we should verify that MySQL and Apache are NOT set to start automatically as we want the cluster to take care of that:
# systemctl disable httpd
# systemctl disable mariadb

And there we have it.  An Active/Active Apache/MySQL cluster.  So those sites to be accessed on VIP1 (192.168.1.192) would have their html/php etc. files located under /var/www_vip1/html, and their config files would be under /var/www_vip1/conf.d.  They would connect to the MySQL instance running on 192.168.1.192 on port 3306.  Sites on VIP2 (192.168.1.193) would have their html/php etc. files located under /var/www_vip2/html, and their config files would be under /var/www_vip2/conf.d.  They would connect to the MySQL instance running on 192.168.1.193 on port 3306.

- Kyle H.