Step-By-Step: How to configure a SANless MySQL Linux failover cluster in Amazon EC2

In this step by step guide, I will take you through all steps required to configure a highly available, 2-node MySQL cluster (plus witness server) in Amazon’s Elastic Compute Cloud (Amazon EC2). The guide includes both screenshots, shell commands and code snippets as appropriate. I assume that you are somewhat familiar with Amazon EC2 and already have an account. If not, you can sign up today. I’m also going to assume that you have basic familiarity with Linux system administration and failover clustering concepts like Virtual IPs, etc.

Failover clustering has been around for many years. In a typical configuration, two or more nodes are configured with shared storage to ensure that in the event of a failover on the primary node, the secondary or target node(s) will access the most up-to-date data. Using shared storage not only enables a near-zero recovery point objective, it is a mandatory requirement for most clustering software. However, shared storage presents several challenges. First, it is a single point of failure risk. If shared storage – typically a SAN – fails, all nodes in the cluster fails. Second, SANs can be expensive and complex to purchase, setup and manage. Third, shared storage in public clouds, including Amazon EC2 is either not possible, or not practical for companies that want to maintain high availability (99.99% uptime), near-zero recovery time and recovery point objectives, and disaster recovery protection.

The following demonstrates how easy it is to create a SANless cluster in the clouds to eliminate these challenges while meeting stringent HA/DR SLAs. The steps below use MySQL database with Amazon EC2 but the same steps could be adapted to create a 2-node cluster in AWS to protect SQL, SAP, Oracle, or any other application.

NOTE: Your view of features, screens and buttons may vary slightly from screenshots presented below

1. Create a Virtual Private Cloud (VPC)
2. Create an Internet Gateway
3. Create Subnets (Availability Zones)
4. Configure Route Tables
5. Configure Security Group
6. Launch Instances
7. Create Elastic IP
8. Create Route Entry for the Virtual IP
9. Disable Source/Dest Checking for ENIs
10. Obtain Access Key ID and Secret Access Key
11. Linux OS Configuration
12. Install EC2 API Tools
13. Install and Configure MySQL
14. Install and Configure Cluster
15. Test Cluster Connectivity

Overview

This article will describe how to create a cluster within a single Amazon EC2 region. The cluster nodes (node1, node2 and the witness server) will reside different Availability Zones for maximum availability. This also means that the nodes will reside in different subnets.

The following IP addresses will be used:

• node1: 10.0.0.4
• node2: 10.0.1.4
• witness: 10.0.2.4
• virtual/”floating” IP: 10.1.0.10

Step 1: Create a Virtual Private Cloud (VPC)

First, create a Virtual Private Cloud (aka VPC). A VPC is an isolated network within the Amazon cloud that is dedicated to you. You have full control over things like IP address blocks and subnets, route tables, security groups (i.e. firewalls), and more. You will be launching your Azure Iaas virtual machines (VMs) into your Virtual Network.

From the main AWS dashboard, select “VPC”

Under “Your VPCs”, make sure you have selected the proper region at the top right of the screen. In this guide the “US West (Oregon)” region will be used, because it is a region that has 3 Availability Zones. For more information on Regions and Availability Zones, click here.

Give the VPC a name, and specify the IP block you wish to use. 10.0.0.0/16 will be used in this guide:

You should now see the newly created VPC on the “Your VPCs” screen:

Step 2: Create an Internet Gateway

Next, create an Internet Gateway. This is required if you want your Instances (VMs) to be able to communicate with the internet.

On the left menu, select Internet Gateways and click the Create Internet Gateway button. Give it a name, and create:

Next, attach the internet gateway to your VPC:

Select your VPC, and click Attach:

 

Step 3: Create Subnets (Availability Zones)

Next, create 3 subnets. Each subnet will reside in it’s own Availability Zone. The 3 Instances (VMs: node1, node2, witness) will be launched into separate subnets (and therefore Availability Zones) so that the failure of an Availability Zone won’t take out multiple nodes of the cluster.

The US West (Oregon) region, aka us-west-2, has 3 availability zones (us-west-2a, us-west-2b, us-west-2c). Create 3 subnets, one in each of the 3 availability zones.

Under VPC Dashboard, navigate to Subnets, and then Create Subnet:

Give the first subnet a name (“Subnet1)”, select the availability zone us-west-2a, and define the network block (10.0.0.0/24):

Repeat to create the second subnet availability zone us-west-2b:

Repeat to create the third subnet in availability zone us-west-2c:

Once complete, verify that the 3 subnets have been created, each with a different CIDR block, and in separate Availability Zones, as seen below:

Step 4: Configure Route Tables

Update the VPC’s route table so that traffic to the outside world is sent to the Internet Gateway created in a previous step. From the VPC Dashboard, select Route Tables. Go to the Routes tab, and by default only one route will exist which allows traffic only within the VPC.

Click Edit:

Add another route:

The Destination of the new route will be “0.0.0.0/0” (the internet) and for Target, select your Internet Gateway. Then click Save:

Next, associate the 3 subnets with the Route Table. Click the “Subnet Associations” tab, and Edit:

Check the boxes next to all 3 subnets, and Save:

Verify that the 3 subnets are associated with the main route table:

Later, we will come back and update the Route Table once more, defining a route that will allow traffic to communicate with the cluster’s Virtual IP, but this needs to be done AFTER the linux Instances (VMs) have been created.

Step 5: Configure Security Group

Edit the Security Group (a virtual firewall) to allow incoming SSH and VNC traffic. Both will later be used to configure the linux instances as well as installation/configuration of the cluster software.

On the left menu, select “Security Groups” and then click the “Inbound Rules” tab. Click Edit:

Add rules for both SSH (port 22) and VNC. VNC generally uses ports in the 5900, depending on how you configure it, so for the purposes of this guide, we will open the 5900-5910 port range. Configure accordingly based on your VNC setup:

Step 6: Launch Instances

We will be provisioning 3 Instances (Virtual Machines) in this guide. The first two VMs (called “node1” and “node2”) will function as cluster nodes with the ability to bring the MySQL database and it’s associated resources online. The 3rd VM will act as the cluster’s witness server for added protection against split-brain.

To ensure maximum availability, all 3 VMs will be deployed into different Availability Zones within a single region. This means each instance will reside in a different subnet.

Go to the main AWS dashboard, and select EC2:

 

Create “node1”

Create your first instance (“node1”). Click Launch Instance:

Select your linux distribution. The cluster software used later supports RHEL, SLES, CentOS and Oracle Linux. In this guide we will be using RHEL 7.X:

Size your instance accordingly. For the purposes of this guide and to minimize cost, t2.micro size was used because it’s free tier eligible. See here for more information on instance sizes and pricing.

Next, configure instance details. IMPORTANT: make sure to launch this first instance (VM) into “Subnet1“, and define an IP address valid for the subnet (10.0.0.0/24) – below 10.0.0.4 is selected because it’s the first free IP in the subnet.
NOTE: .1/.2/.3 in any given subnet in AWS is reserved and can’t be used.

Next, add an extra disk to the cluster nodes (this will be done on both “node1” and “node2”). This disk will store our MySQL databases and the later be replicated between nodes.

NOTE: You do NOT need to add an extra disk to the “witness” node. Only “node1” and “node2”. Add New Volume, and enter in the desired size:

Define a Tag for the instance, Node1:

Associate the instance with the existing security group, so the firewall rules created previous will be active:

Click Launch:

IMPORTANT: If this is the first instance in your AWS environment, you’ll need to create a new key pair. The private key file will need to be stored in a safe location as it will be required when you SSH into the linux instances.

Create “node2”

Repeat the steps above to create your second linux instance (node2). Configure it exactly like Node1. However, make sure that you deploy it into “Subnet2” (us-west-2b availability zone). The IP range for Subnet2 is 10.0.1.0/24, so an IP of 10.0.1.4 is used here:

Make sure to add a 2nd disk to Node2 as well. It should be the same exact size as the disk you added to Node1:

Give the second instance a tag…. “Node2”:

Create “witness”

Repeat the steps above to create your third linux instance (witness). Configure it exactly like Node1&Node2, EXCEPT you DON’T need to add a 2nd disk, since this instance will only act as a witness to the cluster, and won’t ever bring MySQL online.

Make sure that you deploy it into “Subnet3” (us-west-2c availability zone). The IP range for Subnet2 is 10.0.2.0/24, so an IP of 10.0.2.4 is used here:

NOTE: default disk configuration is fine for the witness node. A 2nd disk is NOT required:

Tag the witness node:

It may take a little while for your 3 instances to provision. Once complete, you’ll see then listed as running in your EC2 console:

Step 7: Create Elastic IP

Next, create an Elastic IP, which is a public IP address that will be used to connect into you instance from the outside world. Select Elastic IPs in the left menu, and then click “Allocate New Address”:

 

Select the newly created Elastic IP, right-click, and select “Associate Address”:

Associate this Elastic IP with Node1:

Repeat this for the other two instances if you want them to have internet access or be able to SSH/VNC into them directly.