MySQL Replication on Ubuntu with DRBD

MySQL Replication on Ubuntu with DRBD


I’ve been looking around for some easy and open-source ways to handle database replication for a handful of small but important MySQL databases. A few options were viable but usually included too many config changes for things like creating a new database. DRDB on a Linux server seems to be one of the fastest and easiest methods to handle database synchronisation for DR purposes, so this is the subject of this post. The content is a combination of two main sources from Mark Schoonover and the Ubuntu server guide and the gotchas I found along the way.

This post will show you how to create two MySQL servers that automatically replicate all their databases using DRBD. With Heartbeat installed on a third machine you’ll have basic fail over protection as well (we’ll do this in another post). Only one of the database servers will be active at any one time.

To follow along exactly, you’ll need the following items set up or readily at hand:

* VMware Server or ESXi, etc to host the virtual machines we use. You could just as easily use physical machines if you have the hardware lying around.
* Ubuntu 9.04 server – other versions or distros should work just as well but you may have to make some changes to the steps as you go through.
* Some basic skills in cmd-line usage of Ubuntu and MySQL.
* Being Ubuntu we’re using a whole heap of sudo prefixed commands here. If you want to drop them out, you can switch to a root session with sudo -s.

Start things off by creating a virtual machine with 512MB RAM and two 2GB disks attached. One disk will be for the OS and one will be used as the data location for MySQL. You can change the RAM or disk sizes as you like but we will be duplicating this machine so don’t use up all your RAM and disk space with the first one.

Install Ubuntu 9.04 server onto this machine and name it node0 (that’s n-o-d-e-zero if the font isn’t clear). Just use one of the disks for the OS and don’t format or prepare the second one yet. You can also leave out the LAMP and other installation packages for now as we’ll grab what we need later.

Once the install is complete, reboot the node0 server, login, run a package update and install MySQL and DRBD with,

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install mysql-server mysql-client drbd8-utils heartbeat

You’ll have to download roughly 45MB of packages for those four items and the extras that come with them.

You now have one server configured with the software we need to do the rest. Depending on your luck, you now have two options for creating the node1 virtual machine.
1. Duplicate the Virtual machine in VMware

To make things easy, we’ll simply copy the node0 machine in VMware to create our node1. Make sure you shutdown the node0 machine in VMware and (assuming your VMware server host is Ubuntu too) open up a cmd prompt and run,

cd /var/lib/vmware/Virtual\ Machines
sudo cp -r node0 node1

Once that’s completed, you should be able to open the node0.vmx file that’s in the new node1 folder using VMware console. VMware will add the ‘new’ virtual machine to it’s inventory and you’ll probably have to rename it as VMware will have kept the same node0 name.

Start up your new node1 machine and watch for VMware to prompt you about the machine ID. You’ll need to say you copied the VM to create a new virtual machine ID in VMware.

Log in to node1 and change the server name in /etc/hostname and /etc/hosts files, so the two servers don’t get confused over their identities. Shutdown node1 for the changes to take effect. While the server is offline, edit it’s VMware configuration to remove the NIC and then re-add it. You need to do this to keep the virtual networking in VMware happy.

Depending on how you network is configured, it may be a good idea to set both servers to have static IP addresses. If you don’t want to do this, make sure you know their IPs and add them to the “other” server’s /etc/hosts file. This will mean that node0 can reach node1 by it’s DNS name and vice versa. Test it’s all working with a ping from each machine to the other.
2. Repeat the Ubuntu Installation

OK, so maybe your virtual machine just didn’t want to be duplicated like mine. If so, just create a new VMware guest config with the same settings as the first and install Ubuntu server the same way.
3. Disk Configuration

Now that we have our two node servers configured and the required apps installed, let’s practice safe-virtualisation by taking a VMware snapshot of both servers. Then if it all blows up in the next few steps, at least we’ll have something to go back to.

Let’s create a new disk partition on that extra virtual hard disk we added to both our machines. On both node0 and node1 use fdisk to add a Linux type 83 partition.

sudo fdisk /dev/sdb

Press p to make sure you have the right disk and there are no current partitions. Press n and create a primary partition with the ID of 1, accept the other default options. Press w to save the changes and exit fdisk.
4. Configure DRBD

We can now configure DRBD to use this empty partition space we’ve created. Like many other Linux apps we need to set up DRBD by editing a config file – /etc/drbd.conf

On node0, make your drbd.conf file look something like the file linked below. There are plenty a massive number of other settings and comments but these are the key ones – assuming you’ve used the same disk names as above and apply your own network addresses. My server IPs were node0:192.168.0.17 and node1:192.168.0.18 so replace those with your information. Take your time to read through at least some of this file as other things like disconnects on failures may come in handy.

drbd.conf Need to add the stuff

on node0 {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.0.17:7788;
flexible-meta-disk internal;

}

on node1 {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.0.18:7788;
meta-disk internal;
}


From node0, copy that drbd.conf file to your home directory (~) on node1 using the scp command,

sudo scp /etc/drbd.conf node1:~

You’ll have to accept the RSA key for node1 if you haven’t SSH’d between those servers yet.

And on node1 we need to move that file to the correct place,

sudo mv ~/drbd.conf /etc/

DRBD will now start, but we need to prepare the partition we created earlier so that DRBD can use it. Turn off node1 and on node0 do this,

sudo drbdadm create-md mysql

Restart node0 and log back in again. Typing ‘yes’ to DRBD’s timeout will speed things up. You’ll see some warnings about using the ‘drbd-peer-outdater’ in a mode where something else needs root privileges. Just ignore that for now and we’ll make those changes later.

Run cat /proc/drbd on node0 to have a look at the messages logged when drbd started up. We can see that node0 is in an unknown state as it can’t see what’s happening on node1 properly. So lets sort out node1.

On node1 create the drbd file system just like a couple of minutes ago,

sudo drbdadm create-md mysql

And restart DRBD on node1 to make sure it’s using that new data,

sudo /etc/init.d/drbd restart

Run cat /proc/drbd again to check the status on both node0 and node1 and you should see that they are both connected and set as secondary with an inconsistent status. If you don’t see that then give both nodes a reboot and try again. If you still don’t see that output, start back tracking and check the previous steps.

At this stage both nodes are essentially looking at each other but don’t know who is the primary so they’re not going to do anything. By running the following command on node0 you make that machine the primary and data will start to be replicated straight away.

sudo drbdadm -- --overwrite-data-of-peer primary mysql

You can either use cat /proc/drbd or watch -n1 cat /proc/drbd to view the status of the replication as it happens. This may take some time, even for the relatively small 2GB of empty space we are using, as DRBD will replicate every block in the device /dev/mysql. Between the VMs on an old P4 with a PATA disk I was seeing about 2.8 MB/s.

When you ran that command to set the primary, you probably got some messages about the heartbeat program requiring root privileges to run drbdsetup and drbdmeta. Do that now to get rid of those warnings.

sudo chgrp haclient /sbin/drbdsetup
sudo chmod o-x /sbin/drbdsetup
sudo chmod u+s /sbin/drbdsetup
sudo chgrp haclient /sbin/drbdmeta
sudo chmod o-x /sbin/drbdmeta
sudo chmod u+s /sbin/drbdmeta

Prepare, Mount & Test the DRBD Device

Now although we have our two nodes talking and set as primary and secondary, the disk space is not formatted or mounted for any apps (i.e. MySQL) to use. So we first format our /dev/drbd0 device with ext3 and a nice filesystem label.

sudo mkfs.ext3 -L mysql /dev/drbd0

Then we create the mount point on node0 and mount the new filesystem. Whatever gets dropped into /mnt/mysql will magically be synchronised from the primary node to the secondary.

sudo mkdir /mnt/mysql
sudo mount /dev/drbd0 /mnt/mysql

To test our new setup copy some files and folders into /mnt/mysql and unmount the DRBD device. We then switch the primary node0 to be the secondary and jump onto the node1 server.

sudo cp -r ~ /mnt/mysql
sudo umount /mnt/mysql
sudo drbdadm secondary mysql

Now move to the node1 server and create the same mount point, mount our DRBD device and we’ll see what’s inside.

sudo mkdir /mnt/mysql
sudo mount /dev/drbd0 /mnt/mysql

Like magic you should now see that same data you copied on node0, has appeared in the same device on node1.

Now switch the roles back again so node0 is the primary and node1 is the secondary, and delete the test data in /mnt/mysql.
Configure MySQL

The core settings for MySQL is in a file called /etc/mysql/my.cnf and it’s this file that we are going to edit to tell MySQL to use our DRBD device to store it’s databases, indexes and logs.

At this point it’s very important that you do not have any MySQL databases already running on node0 or node1 – none that you want to keep anyway. Let’s stop the MySQL server and backup our original MySQL settings so we can then make changes on node0.

sudo /etc/init.d/mysql stop
cd /etc/mysql
sudo cp my.cnf myold.cnf

Open up the my.cnf file in your favorite text editor like vi or nano, and change the datadir option as below. If you have logging enabled, or will at some point, you may want to synchronise the log files as well. For more information on other options in this file see the MySQL documentation.

datadir = /mnt/mysql/data

Save my.cnf and make the same changes on node1. We have to repeat this as these files aren’t in out replicating DRBD device.

Run the following commands to set the correct permissions on the new location and also update the AppArmor settings to allow access. This is important else the mysql user and group will not be able to access the new directory properly and you will get errors. Trying to restart the MySQL daemon at the moment will fail.

sudo chown -R mysql:mysql /mnt/mysql
sudo nano /etc/apparmor.d/usr.sbin.mysqld

On the bottom of the list of locations, add the following (within the curly closing bracket } ),

/mnt/mysql/data/ r,
/mnt/mysql/data/** rwk,

Restart AppArmor to pick up the new locations for MySQL,

sudo /etc/init.d/apparmor restart

Make the same changes to AppArmor on node1 – again these changes are server based and not automatically replicated.

We should try and move the previous databases into the new location with the following commands. Repeat the first copy command for any other databases you might have. (That mysql directory we’re copying first is the MySQL system database.)

sudo cp -r /var/lib/mysql/mysql /mnt/mysql/data/
...
sudo chown -R mysql:mysql /mnt/mysql/data/*

The last thing to do is restart MySQL and check it’s working.

sudo /etc/init.d/mysql start

If this gives permission errors about the debian-sys-maint user, check your permissions and/or restart the server. Restarting MySQL might not be enough.

Connect to mysql as the root user and let’s just create a new database and table to test things before we do another manual fail-over to node1.

mysql -u root -p
create database drbdtest;
use drbdtest;
create table users (id INTEGER(9), name VARCHAR(255));
exit;

You can now see your new database files in /mnt/mysql/data/drbdtest. So we fail-over from node0 to node1 again, just for practice. We must stop the mysql server before we unmount the DRBD device else very bad things may happen.

sudo /etc/init.d/mysql stop
sudo umount /mnt/mysql
sudo drbdadm secondary mysql

and over on node1 . . .

sudo drbdadm primary mysql

And check those database files are there with the mysql client or looking in /mnt/mysql/data/drbdtest.
Conclusion

So we can see that this will save us (with some manual intervention), when our MySQL storage falls over. But wouldn’t it be nice if there was an automated way to make this happen? See the next article on using Heartbeat to make this happen.
Potential Issues

If you try and mount /dev/drbd0 and get an error about the device being read-only, check that the DRBD node you’re on is in the correct mode i.e. primary or secondary.

For Better Clarity Please follow the below link
https://help.ubuntu.com/9.04/serverguide/C/drbd.html
http://www.drbd.org/
http://blog.irwan.name/?p=118