replication Archives - SIOS SANless clusters

Maximise replication performance for Linux Clustering with Fusion-io

November 27, 2018 by Jason Aw Leave a Comment

Tips To Maximise Replication Performance For Linux Clustering With Fusion-io

When most people think about setting up a cluster, it usually involves two or more servers, and a SAN – or some other type of shared storage. SAN’s are typically very costly and complex to setup and maintain. Also, they technically represent a potential Single Point of Failure (SPOF) in your cluster architecture. These days, more and more people are turning to companies like Fusion-io, with their lightning fast ioDrives, to accelerate critical applications. These storage devices sit inside the server (i.e. aren’t “shared disks”). Therefore it can’t be used as cluster disks with many traditional clustering solutions. Fortunately, there are ways to Maximise replication performance for Linux Clustering with Fusion-io. Solutions that allow you to form a failover cluster when there is no shared storage involved – i.e. a “shared nothing” cluster.


Traditional Cluster	“Shared Nothing” Cluster

When leveraging data replication as part of a cluster configuration, it’s critical that you have enough bandwidth so that data can be replicated across the network just as fast as it’s written to disk. The following are tuning tips that will allow you to get the most out of your “shared nothing” cluster configuration, when high-speed storage is involved:

Network

Use a 10Gbps NIC: Flash-based storage devices from Fusion-io (or other similar products from OCZ, LSI, etc) are capable of writing data at speeds in the HUNDREDS (750 ) of MB/sec or more. A 1Gbps NIC can only push a theoretical maximum of ~125 MB/sec, so anyone taking advantage of an ioDrive’s potential can easily write data much faster than could be pushed through a 1 Gbps network connection. To ensure that you have sufficient bandwidth between servers to facilitate real-time data replication, a 10 Gbps NIC should always be used to carry replication traffic
Enable Jumbo Frames: Assuming that your Network Cards and Switches support it, enabling jumbo frames can greatly increase your network’s throughput while at the same time reducing CPU cycles. To enable jumbo frames, perform the following configuration (example from a RedHat/CentOS/OEL linux server)
- ifconfig <interface_name> mtu 9000
- Edit /etc/sysconfig/network-scripts/ifcfg-<interface_name> file and add “MTU=9000” so that the change persists across reboots
- To verify end-to-end jumbo frame operation, run this command: ping -s 8900 -M do <IP-of-other-server>
Change the NIC’s transmit queue length:
- /sbin/ifconfig <interface_name> txqueuelen 10000
- Add this to /etc/rc.local to preserve the setting across reboots

TCP/IP Tuning

Change the NIC’s netdev_max_backlog:
- Set “net.core.netdev_max_backlog = 100000” in /etc/sysctl.conf
Other TCP/IP tuning that has shown to increase replication performance:
- Note: these are example values and some might need to be adjusted based on your hardware configuration
- Edit /etc/sysctl.conf and add the following parameters:
  - net.core.rmem_default = 16777216
  - net.core.wmem_default = 16777216
  - net.core.rmem_max = 16777216
  - net.core.wmem_max = 16777216
  - net.ipv4.tcp_rmem = 4096 87380 16777216
  - net.ipv4.tcp_wmem = 4096 65536 16777216
  - net.ipv4.tcp_timestamps = 0
  - net.ipv4.tcp_sack = 0
  - net.core.optmem_max = 16777216
  - net.ipv4.tcp_congestion_control=htcp

Adjustments

Typically you will also need to make adjustments to your cluster configuration, which will vary based on the clustering and replication technology you decide to implement. In this example, I’m using the SteelEye Protection Suite for Linux (aka SPS, aka LifeKeeper), from SIOS Technologies. It allows users to form failover clusters leveraging just about any back-end storage type: Fiber Channel SAN, iSCSI, NAS, or, most relevant to this article, local disks that need to be synchronized/replicated in real time between cluster nodes. SPS for Linux includes integrated, block level data replication functionality that makes it very easy to setup a cluster when there is no shared storage involved.

Recommendations

In order to Maximise replication performance for Linux Clustering with Fusion-io, let’s try this. SteelEye Protection Suite (SPS) for Linux configuration recommendations:

Allocate a small (~100 MB) disk partition, located on the Fusion-io drive to place the bitmap file. Create a filesystem on this partition and mount it, for example, at /bitmap:
- # mount | grep /bitmap
- /dev/fioa1 on /bitmap type ext3 (rw)
Prior to creating your mirror, adjust the following parameters in /etc/default/LifeKeeper
- Insert: LKDR_CHUNK_SIZE=4096
  - Default value is 64
- Edit: LKDR_SPEED_LIMIT=1500000
  - (Default value is 50000)
  - LKDR_SPEED_LIMIT specifies the maximum bandwidth that a resync will ever take — this should be set high enough to allow resyncs to go at the maximum speed possible
- Edit: LKDR_SPEED_LIMIT_MIN=200000
  - (Default value is 20000)
  - LKDR_SPEED_LIMIT_MIN specifies how fast the resync should be allowed to go when there is other I/O going on at the same time — as a rule of thumb, this should be set to half or less of the drive’s maximum write throughput in order to avoid starving out normal I/O activity when a resync occurs

From here, go ahead and create your mirrors and configure the cluster as you normally would.

Interested to Maximise Replication Performance For Linux Clustering With Fusion-io, see what else SIOS can offer.
Reproduced with permission from LinuxClustering

Cloud Witness To Build Multi-Instance SQL Server Failover Cluster In Azure

September 10, 2018 by Jason Aw Leave a Comment

New Azure ILB Feature Allows You To Build A Multi-Instance SQL Server Failover Cluster In Azure

The new feature, Cloud Witness is my favourite at the moment. Before we look at the new quorum features in Windows Server 2016, I think it is important to know where we came from. In my previous post Understanding the Windows Server Failover Cluster Quorum in Windows Server 2012 R2 I went into some great detail regarding the history and evolution of the cluster quorum. I suggest you review that post to understand how the quorum works in Windows Server 2012 R2. Also, how the new features of Windows Server 2016 are going to make your cluster deployments even more resilient.

Cloud Witness

A Cloud Witness allows you to leverage Azure Blob Storage to act as a witness for your cluster. This witness would be in place of a Disk Witness or File Share Witness. The configuration of a Cloud Witness is extremely easy. From my experience costs next to nothing to host in Azure. The only downside is that the cluster nodes will need to be able to communicate over the internet to with your Azure Blob Storage. Very often cluster nodes are forbidden to communicate over to the public internet. So you will need to coordinate with your security team if you want to enable a Cloud Witness.

There are many compelling reasons for using a Cloud Witness to build the Multi-Instance SQL Server Failover Cluster In Azure. But for me it makes most sense in three very specific environments: Failover Cluster in Azure, Branch Office Clusters, and Multisite Clusters.

On A Closer Look

Let’s take a look at each of these scenarios to see how a Cloud Witness can help.

New ILB Feature For Multi-Instance SQL Server Failover Cluster In Azure — Figure 1 – When you’re trying to build Multi-Instance SQL Server Failover Cluster In Azure, the cloud witness storage account should always be configured Locally Redundant Storage (LRS)

Highly Available Deployments

If you are moving to Azure (or really any cloud provider), you will want to make sure your deployments are highly available. If you are taking about SQL Server, File Servers, SAP or other workloads traditionally clustered with Windows Server Failover Clustering, you will need to use either a File Share Witness or a Cloud Witness, since a Disk Witness is not possible in Azure. With Windows Server 2012 R2 or Windows Server 2008 R2, you will need to use a File Share Witness. Windows Server 2016 makes it possible to use a Cloud Witness. The advantage of a Cloud Witness is that you don’t have to maintain another Windows instance in Azure to host the File Share. Instead, Microsoft allows you to leverage Blob Storage. This gives you a less expensive solution, one that is much easier to manage, and more resilient.

Location

When looking at cluster deployments in branch offices, cost and maintenance is always a consideration. For a retail chain with hundreds or thousands of locations, having a SAN in each location can be cost prohibitive. Each location might to run a two node Hyper-V cluster on a S2D Hyper-converged configuration or a 3^rd party replication solution to host a number of virtual machines. Now what a Cloud Witness can do is to help the business avoid the cost of adding an additional physical server in each location to act as a File Share Witness or the cost of adding a SAN to each location.

Eliminates The Need For A 3^rd Data Center

And finally, when deploying a multisite cluster, the Cloud Witness eliminates the need for a 3^rd data center to host the File Share Witness. Before the introduction of the Cloud Witness, best practice would dictate that the File Share Witness reside in a 3^rd location. Access to a 3^rd datacenter just to host a file share witness was not always feasibly and certainly introduced another layer of complexity. By using a Cloud Witness you eliminate the need to maintain a 3^rd location and access to the witness is done over the public internet, minimizing the network requirements as well.

Site Awareness

When building a multisite cluster, there has always been another common problem. Controlling the failover to always prefer the local site was not possible. While you could specify Preferred Owners, the Preferred Owners setting is commonly misunderstood. Administrators may not have realized this. But do you know even if they didn’t list a server as Preferred Owner, the server is automatically appended to the end of the Preferred Owners list maintained by the cluster. The result of this misunderstanding is that although you may have only listed the local servers as Preferred Owners, you could potentially have a cluster resource failover to the DR site. And this is even when there is a perfectly good node available in the local site. Obviously this is not what you expect and using Site Awareness will eliminate this problem moving forward.

Site Awareness fixes this problem by always preferring the local site when deciding which node to bring online. So in a normal circumstance a clustered workload will always failover to a local node unless you have a complete site outage. In which case one of the DR nodes will come online. The same holds true once you are running in the DR site. The cluster will recover the workload on a server in the DR site if it was previously running on a node in the DR site. Site Awareness will always prefer a local node.

Fault Domains

Building upon site awareness is Fault Domains. Fault Domains goes a step further and lets you define Node, Chasse, and Rack locations in addition to Site. Fault Domains have three benefits: Storage Affinity in a Stretch Cluster, increases Storage Spaces resiliency. It enhances the Health Services alerts by including meta data about the location of the associated resources raising the alarm. Storage Affinity will help ensure that your cluster workloads and storage are running in the same location. You certainly wouldn’t want your VM reading and writing data that is sitting on a CSV in a different city.

However, I think the biggest winner here is the Storage Spaces Direct (S2D) scenario. SD2 will leverage the information you provide about your cluster nodes location (Site, Rack, Chassis) to ensure that the multiple copies of data that is written for redundancy all live in different Fault Domains. This helps ensure that data placement is optimized so that the failure of a single Node, Chassis, Rack or Site does not bring down your entire S2D deployment. Cosmos Darwin has an excellent video on Channel 9 that explains this concept in great detail.

Summary

Windows Server 2016 adds several new enhancements to the cluster quorum that will provide some immediate benefits to your cluster deployments. In addition, check out some of the other great new cluster enhancements like rolling system upgrade, Virtual Machine Resiliency, Workgroup and Multi-Domain Clusters and others.

To read about other tips such as building a new Multi-Instance SQL Server Failover Cluster In Azure with Cloud Witness, have a read at our posts.

Reproduced with permission from Clusteringformeremortals.com

SIOS Data Replication And Disaster Recovery Solution For Data Protection

May 27, 2018 by Jason Aw Leave a Comment

Software Company Serving Educational Institutions Uses SIOS’ Cost-Effective Data Replication And Disaster Recovery Solution For Continuous Data Protection

The much anticipated Windows Server 2008 R2 became available in late October. VISUCATE became one of many small businesses to deploy Microsoft Hyper-V and enjoy its new features such as live migration. The company required a data replication and disaster recovery solution. One that was reasonably priced and delivered first-class protection.

In an effort to complete its set-up, VISUCATE wanted a business continuity platform that met its small business expectations. They took advantage of the high availability features of Windows Server Failover Clustering. However, 5fhbvd VISUCATE needed additional assurances that a loss of critical data or downtime would not compromise its software sales. To address these specific data replication hurdles, VISUCATE turned to SIOS DataKeeper Cluster Edition.

The Challenge

They required an affordable, uncomplicated and robust data replication and disaster recovery solution to protect its new Hyper-V set-up. To prevent any downtime, the company needed its servers to replicate and maintain their operational capabilities. If one server fails, the other server is configured to take over to sustain operations, maximize uptime and assure user productivity. The joint solution of Microsoft Hyper-V with Windows Server Failover Clustering and SIOS DataKeeper Cluster Edition addressed those business requirements essential for VISUCATE as well as any organization intent on overcoming this challenge.

VISUCATE Maintains Hyper-V Availability, Business Continuity with SIOS DataKeeper®

VISUCATE deployed Windows Server 2008 R2 on two physical servers with the Hyper-V role enabled. The company uses Windows Server Failover Clustering and SIOS DataKeeper Cluster Edition to provide replication and failover of the virtual machines. With the Hyper-V deployment, VISUCATE’s five virtual machines were installed across both servers. Three in one server and two on the other server.

By keeping an operational Windows Server 2008 Hyper-V virtual machine synchronized between two physical servers, SIOS DataKeeper enables disaster recovery without the recovery and downtimes typically associated with traditional back-up and restore technology. Realtime continuous replication of active Windows Server 2008 Hyper-V virtual machines ensures that in the event of any downtime impacting VISUCATE set-up, the replicated virtual machine can be automatically brought into service with minimal or no data loss. VISUCATE considered several options for a failover cluster solution. The company dismissed the option of creating a cluster with either a lowcost SAN or NAS/file server. If the SAN in that configuration crashed, the entire set-up would fail.

SIOS DataKeeper Cluster Edition reduces the cost of deploying clusters by eliminating the need for a SAN. It also increases the availability of virtual machines and applications by eliminating the single point of a failure that the SAN represents in a traditional shared storage cluster.

Benefits

SIOS DataKeeper Cluster Edition allows companies such as VISUCATE to build “shared-nothing” and geographically dispersed Windows Server 2008 Hyper-V clusters. By eliminating the requirement for shared storage, companies can protect against both planned and unplanned downtime for servers and storage. Also, the use of SIOS DataKeeper with Windows Server 2008 Hyper-V virtual machines allows for non-disruptive disaster recovery testing. By simply accessing the replicated virtual machine in the disaster recovery site, VISUCATE and other companies can segment a virtual network separate from the production network. Also, it would be able to start the replicated virtual machine for disaster recovery testing. An administrator can perform complete data replication and disaster recovery solution testing without impacting the production site.

In addition to support for Hyper-V clusters, SIOS DataKeeper Cluster Edition enables multi-site clusters for all other Microsoft cluster resource types. This includes SQL Server, Exchange, File/Print and DHCP.

To find out more about SIOS products, go here
To read about how SIOS helped VISUCATE achieve data replication and disaster recovery solution, go here