| November 14, 2026 |
Fail Open vs. Fail Closed: Why It’s a False Dilemma (and How True HA Fixes It) |
| Appliance Type | Fail Open Behavior | Fail Closed Behavior | The Real-World Risk |
| Email Spam / Mail Security Gateways | Bypasses AV/phishing filters; sends raw inbound mail straight to exchange/inbox. | Refuses SMTP connections (4xx temporary error or 5xx reject). | In Fail Open, a wave of ransomware or spear-phishing hits inboxes. In Fail Closed, business communication freezes, and external senders get bounce-backs. |
| Web Application Firewalls (WAFs) | Reverts to a plain layer-4/7 reverse proxy, passing SQLi, XSS, and bot traffic through. | Returns 502 Bad Gateway / 504 Gateway Timeout or drops TCP sessions. | In Fail Open, web assets are completely exposed to zero-days. In Fail Closed, customer traffic stops entirely. |
| DDoS Scrubbing / IPS Nodes | Hard bypass via optical/bypass switches; passes unfiltered traffic upstream. | Interface shuts down; link goes dark. | In Fail Open, transit pipes saturate immediately. In Fail Closed, the service goes offline—achieving the attacker’s goal anyway. |
| Identity & Authentication Proxies | Grants fallback access or drops MFA enforcement. | Rejects all logins (Access Denied). | In Fail Open, unauthorized access spikes. In Fail Closed, zero staff or customers can authenticate. |
Why “Fail Open vs. Fail Closed” Is an Architecture Antipattern
Relying on an appliance’s native fallback mechanism assumes that failure must mean compromise or downtime. It accepts single points of failure at the architectural level.
A bypass switch or a static drop rule only addresses what to do after an instance dies. It does nothing to:
- Re-establish active inspection immediately.
- Maintain synchronization of active connection states or security session tables.
- Automatically remediate the underlying service fault.
Choosing between security and uptime is usually a sign that the appliance lacks true Application-Aware High Availability (HA).
Escaping the Binary: How something like SIOS LifeKeeper Solves the Dilemma
Rather than deciding whether your crashed WAF, proxy, or MTA should blindly drop or blindly bypass traffic, SIOS LifeKeeper eliminates the dead-end entirely by keeping the inspection layer highly available.
1. Deep Application-Aware Health Monitoring
Standard hypervisors or cloud orchestrators only look at whether a virtual machine is running. If the underlying Linux OS is alive but the spam filter’s amavisd, clamav, or custom WAF inspection worker hangs in a deadlock, hypervisors see a healthy VM.
SIOS LifeKeeper includes Application Recovery Kits (ARKs) for many applications but also includes the ability to make your own custom “GenApp ARK” to provide customized monitoring that query the application layer directly:
- Is the filtering daemon responding to probe requests?
- Are memory limits approaching threshold panics?
- Are the network sockets actively processing queues?
2. Local Self-Healing First, Failover Second
Before triggering a costly cross-node failover, LifeKeeper can attempt an automated, controlled restart of the misbehaving security service. If the process is irrecoverable, it orchestrates an automated failover.
3. Zero-Data-Loss Configuration & State Replication
Security appliances often depend on dynamic blocklists, rate-limiting caches, and session states. In tandem with SIOS DataKeeper, LifeKeeper provides block-level synchronous or asynchronous storage replication without requiring expensive, shared SAN hardware. The standby node has access to the exact policy sets, certificates, and IP reputation caches the moment it assumes control.
4. Automated Virtual IP and Network Redirection
When failover occurs, LifeKeeper reallocates the cluster Virtual IP (VIP), updates cloud routing tables (e.g., across AWS AZs, Azure Availability Zones, or on-premises subnets), and starts the security services on the secondary node.
The result:
- The system behaves like “Fail Closed” from a security perspective: uninspected packets are never let through blindly.
- The system behaves like “Fail Open” from an uptime perspective: the service recovers automatically in seconds, eliminating prolonged outages.
Real-world example… Protecting ModSecurity with SIOS LifeKeeper
As a widely adopted open-source web application firewall engine embedded within Apache HTTP Server, Nginx, or IIS, ModSecurity sits directly inline to evaluate incoming HTTP traffic against malicious signatures. Consequently, when an unhandled daemon lockup or process crash occurs, infrastructure teams face a stark operational dilemma: expose the application to immediate compromise or accept a total service outage. This makes ModSecurity an ideal scenario for leveraging SIOS LifeKeeper to resolve the trade-off through robust High Availability.
Custom GenApp ARK Monitoring for Web Server Processes
Using the SIOS LifeKeeper Generic Application Recovery Kit (GenApp ARK), administrators can define custom health-check scripts tailored to ModSecurity and its host web server (Apache or Nginx):
- Regularly probe web server worker processes and verify that the ModSecurity engine is actively parsing requests without deadlocks.
- Monitor rule engine memory usage and HTTP response codes to detect silent inspection failures.
Local Service Self-Healing
If the GenApp ARK detects that the Apache or Nginx process hosting ModSecurity has hung or crashed, LifeKeeper first attempts an automated local restart of the web server and WAF module. This resolves transient software faults immediately without interrupting cluster state or switching node ownership.
Automated Failover and Rule Synchronisation
If local recovery fails, LifeKeeper automatically initiates failover to a standby node. In conjunction with SIOS DataKeeper block-level replication, custom SecRule directives, audit logs, and IP reputation tables are continuously synchronised, ensuring the secondary node assumes active WAF inspection with up-to-date protection profiles.
The Verdict
“Fail open or fail closed” is a forced choice that stems from treating security nodes as isolated, standalone appliances.
For perimeter security where data breaches and downtime carry identical financial risk, the goal should be neither. By anchoring your email security clusters, WAFs, and proprietary Linux/Windows security servers to an active-passive or active-active framework via SIOS LifeKeeper, you take the human gamble out of the equation: security stays rigid, and operations stay online.
About the Author: Aaron West is a Solutions Engineer at SIOS Technology Corp, specializing in high availability, open-source solutions, and networking. With over a decade of experience, Aaron has held leadership roles, including Head of Solutions at Loadbalancer.org, where he built and led teams in technical support and pre-sales consulting. His expertise spans network engineering, Linux, and Microsoft systems, with a strong focus on team building and customer-centric solutions. Aaron is passionate about delivering robust open-source technologies and providing tailored solutions for clients across various industries.
Reproduced with permission from SIOS
The 5-Step Checklist for Validating Your Failover Setup
The 5-Step Checklist for Validating Your Failover Setup
Having a failover environment configured is only one part of completing the setup. Validating and verifying whether your configuration will work as expected is the next and most important step. Here are ways you can be confident your infrastructure can withstand a disaster.
Build a QA Cluster That Mirrors Production
A QA cluster is typically a replica of the production cluster that is used for testing and/or implementing any changes that need to be introduced into the cluster. With environments that contain multiple failover solutions, it’s necessary to have an environment that allows for testing to see how the environment behaves. When solutions work separately, there is sometimes an assumption that they will seamlessly work together. This may not always be the case. Finding out that solutions do not work well together could happen during a failover. If the order of operations is not tested and verified to work, you may experience a failure rather than a failover, possibly due to the order of operations. Having a QA environment is useful in mitigating and preventing issues arising when HA/DR response is needed most.
Test Failover and Switchover Scenarios
Testing is crucial in validating your failover setup. Testing helps ensure that an environment behaves properly when a failure scenario happens. Simultaneously, testing is important to verify if the behavior of the environment aligns with your expectations. There are circumstances where you may not want to begin a failover immediately, which is why it helps to understand the difference between switchover and failover testing.
Verify Logs Before Go-Live
Before going into production, always check the logs from your system and the failover solution, such as SIOS LifeKeeper for Linux. Ideally, you will want to verify that within your environment all needed systems, clusters, and infrastructure elements can communicate with each other to ensure smooth operation. Checking the logs for any anomalies is also helpful in understanding how your environment is behaving, especially during testing, should your expectations not be met.
Document Everything in a Runbook
Always have a Runbook to document procedures for the environment and to outline the tests that should be executed. Documenting the process of setting up the environment and testing the environment helps beyond having an understanding of what you are doing and the information that is being shared. Documentation makes sure processes are repeatable, but it also serves as a record that you can audit if needed. By keeping documentation of how procedures should be performed and the tests that should be verified, a paper trail can be maintained so any analyses performed after an issue opens up room for improvement.
During this phase of validating your configuration, it is not expected that everything will be 100% successful; failures are inevitable. When they do happen, a runbook provides documentation on the steps that led to the situation. Runbooks and documentation are important because they provide insight into the events that did – or did not – happen in the lead-up to an issue. With thorough documentation, you get the ability to adjust procedures and prevent these circumstances from creating an issue in your production environment.
Confirm Application Setup Meets Your Failover Solution’s Requirements
Certain failover solutions may differ in their requirements for application protection. Setting up the applications is sometimes not as simple as making sure the application itself works, but making sure the application and the failover solution work well together. Verify with the respective documentation on how applications need to be set up.
For example, the LifeKeeper for Linux documentation outlines a requirement to modify some default settings to allow proper protection of an SAP environment, as seen in the modify ASCS and ERS instance profile settings for SAP page.
Considerations like these are crucial to ensure that your HA/DR solution and protected applications work together cohesively and grant the most value from the solutions in use.
When deploying an environment, there are a lot of factors at play. With this checklist, rollouts can be much more efficient and manageable. This checklist will help ensure your environment is validated and ready for production, giving you confidence in the life of your deployment. Want an expert to check your setup? Schedule an HA Health Check with SIOS.
About the Author: Alexus Gore is a Customer Experience Software Engineer at SIOS Technology Corp. She focuses on helping customers optimize their high availability and disaster recovery environments, ensuring their systems are resilient and reliable.
Reproduced with permission from SIOS
Managing an Inherited DataKeeper Cluster: Challenges and Best Practices
Managing an Inherited DataKeeper Cluster: Challenges and Best Practices
How’s that inheritance coming along? Part 2 of our Inheriting DataKeeper series.
Inheriting an IT environment is rarely a one-and-done process, and inheriting a DataKeeper-enabled cluster is no exception. When we first discussed the idea of “inheritance” in the context of DataKeeper in our first post, Inheriting DataKeeper,, we emphasized the importance of getting your bearings—understanding the environment, asking the right questions, and collaborating across multiple teams. But once the “honeymoon” phase is over, the real work begins. So, how’s it going? Are you thriving in your new role as steward of your inherited cluster, or are you slowly discovering the quirks, gaps, and potential pitfalls lurking beneath the surface?
In this follow-up blog, let’s revisit some of the concepts from the original post, dig deeper into the challenges an administrator may now be facing, and explore strategies for ongoing success as you continue to manage your DataKeeper inheritance.
Key Takeaways
- Treat your DataKeeper runbook as a living document and update it after every change or test.
- Map the dependencies between DataKeeper mirrors, SQL Server, storage, and networking—a change in one can ripple through the others.
- Schedule failover tests regularly; an untested HA/DR configuration is an assumption, not a plan.
- Track licensing renewals, release notes, and patches to avoid surprise service interruptions.
- Use monitoring and alerts to catch replication delays before they impact workloads.
Reality Check: What You’ve Learned About Your Inherited Cluster
If you’re a few weeks or months into your role as the inheritor of a DataKeeper environment, you’ve likely gained some valuable insights—both about the technology and about the infrastructure-specific nuances of your cluster. Here are some common “aha moments” I’ve encountered in and throughout my career:
1. Runbooks Are Great—Until They’re Not
Remember how we stressed the importance of a well-documented runbook? By now, you’ve probably realized that runbooks can be hit-or-miss. If your predecessor was meticulous in their documentation, congratulations—you’re in great shape. But if you’ve inherited a runbook that’s outdated, incomplete, or nonexistent, you may be feeling the strain of piecing together critical workflows on your own.
Personal Tip: Make it a priority to update your runbook as you uncover gaps or introduce changes. Treat it as a living document, not a dusty binder in the corner.
2. Dependencies Are Sneaky
IT systems are like ecosystems—everything is interconnected. You may have discovered that changes to one component (e.g., SQL Server configurations, storage volumes, or network settings) can ripple through your cluster in unexpected ways. Identifying these dependencies is an ongoing process, and it requires constant communication with cross-functional teams.
Personal Tip: Build strong relationships with the SQL Server, Networking, and Storage teams, and establish regular touchpoints to share updates and flag potential issues early.
3. Failover Testing Is Not Optional
If you’ve already performed a failover test (or several), kudos to you. If you haven’t—what are you waiting for? Failover testing is critical for ensuring your high availability (HA) and disaster recovery (DR) configurations work as intended. Just because everything looks good on paper doesn’t mean it will behave as expected during an actual failover scenario.
Personal Tip: Schedule failover tests regularly, and involve key stakeholders in the process. Document the results, and update your runbook accordingly.
Common Challenges in the “Inheritance Phase”
Managing your inherited cluster is an ongoing process, and it’s not without its challenges. Here are a few common pain points that may be surfacing as you settle into your role:
1. Licensing and Account Management
Are you still tracking licensing renewals and account details? If not, you could be on a collision course with unexpected service interruptions. Stay vigilant about renewal deadlines, and ensure your credentials for the licensing portal are up to date.
2. Performance Bottlenecks
As workloads evolve, your cluster’s performance may be impacted. Have you noticed increased latency or replication delays? It may be time to revisit your storage bandwidth, mirror types, or network configurations, just to name a few.
Personal Tip: Collaborate with the Networking and Storage teams to identify and address bottlenecks before they escalate.
3. Software Updates and Patch Management
Keeping your DataKeeper environment up to date is essential for security, stability, and performance. But coordinating updates across multiple systems can be tricky. Are you proactively monitoring for new releases and patches?
Personal Tip: Establish a patch management strategy that includes regular updates, testing procedures, and rollback plans.
4. Knowledge Silos (Yes! It’s a thing at a Data Center, and that’s where I picked up the lingo)
If your predecessor didn’t do a thorough knowledge transfer—or if key team members have left—critical information about your cluster may be locked away in silos. Bridging these gaps requires patience, detective work, and collaboration.
Personal Tip: Host regular knowledge-sharing sessions with your team to surface hidden insights and ensure everyone is aligned.
Strategies for Longterm DataKeeper Cluster Health
So, what’s next? How do you ensure your inherited DataKeeper environment continues to run smoothly while avoiding burnout or unexpected surprises? Here are a few strategies to keep in mind:
1. Proactive Monitoring and Alerts
Set up monitoring tools and alerts to keep an eye on your cluster’s health. By proactively identifying issues—such as replication delays or resource contention—you can address them before they impact your workloads.
2. Continuous Learning
DataKeeper is a powerful tool, but it’s also nuanced. Take advantage of training resources, webinars, and documentation provided by SIOS to deepen your understanding of the technology.
Personal Tip: Bookmark the SIOS Resource Center for easy access to guides, whitepapers, and case studies.
3. Cross-Functional Collaboration
The success of your cluster depends on collaboration across teams. Build strong relationships with key stakeholders, and establish regular communication channels to keep everyone in the loop.
4. Prepare for Growth
Is your cluster environment expected to scale in the near future? Whether you’re adding new nodes, workloads, or mirrors, ensure you have a plan in place to support growth without compromising performance or availability.
The Final Question: How’s It Going?
Inheriting a DataKeeper environment is both a privilege and a responsibility. It’s an opportunity to build on the foundation laid by your predecessor while introducing improvements, efficiencies, and innovations of your own. So, how’s it going? Are you embracing the challenge, or are you still finding your footing? Wherever you are in your journey, know that you’re not alone. Lean on your team, your documentation, and resources like SIOS to guide you through the process.
Remember: your inheritance is only as valuable as the care and attention you give it. Keep learning, keep collaborating, and keep improving—and you’ll transform your inherited cluster into a thriving environment that supports your organization’s goals for years to come.
Frequently Asked Questions
Start by getting your bearings: confirm who owns the licensing and support accounts, review (or create) the runbook, inventory your mirrors and cluster resources, and identify the SQL Server, storage, and networking teams whose changes can affect the cluster. Then schedule a failover test to verify the environment works as documented.
Test on a regular schedule and after any significant change to SQL Server, storage, networking, or DataKeeper itself. [Greg: add recommended cadence.] Document the results of each test and update your runbook so everyone knows what was verified and when.
A DataKeeper runbook should list every mirror (source and target volumes, synchronous or asynchronous mode), the DataKeeper service account, license details and renewal dates, WSFC resource dependencies, switchover and failback procedures, failover test history, and escalation contacts for SIOS Support and your internal teams.
Replication delays usually trace back to limited network bandwidth or high latency between nodes, storage performance limits, a mirror type that doesn’t suit the connection (for example, synchronous mirroring over a long-distance link), or workloads that have outgrown the original design. Work with your networking and storage teams to pinpoint the bottleneck.
Monitor SIOS release notes, confirm compatibility with your Windows Server and SQL Server versions, test updates outside production where possible, and keep a rollback plan. Follow the documented DataKeeper Cluster Edition upgrade procedure, then update your runbook with the new version and any configuration changes.
Author: Greg Tucker, Senior Product Support Engineer at SIOS
Setting Up Nginx Load Balancer on AWS
Setting Up Nginx Load Balancer on AWS
Step 1: Launching EC2 Instances
- Navigate to the AWS Management Console and launch two EC2 instances, each in a different AZ within the same region.
- Choose an appropriate instance type based on your workload requirements and configure security groups to allow inbound traffic on ports 80 and 443.
- Ensure that the instances are provisioned with the latest version of your preferred Linux distribution.
Step 2: Installing Nginx
- SSH into each EC2 instance and install Nginx using the package manager (e.g., apt for Ubuntu, yum for CentOS).
- Configure Nginx as a load balancer by editing the Nginx configuration file (nginx.conf) to define upstream servers and load balancing algorithm (e.g., round-robin).
- Test the Nginx configuration and start the Nginx service on both instances.
Step 3: Configuring DNS and Health Checks
- Assign Elastic IP addresses to the EC2 instances for static IP resolution.
- Configure DNS records (e.g., A records) to point to the Elastic IPs of the Nginx instances.
- Implement health checks to monitor the availability of backend servers and ensure traffic is routed only to healthy instances.
- Ensure that any relevant firewalls or security group rules allow traffic on required ports (e.g., HTTP/HTTPS and health check ports).
Enhancing High Availability with SIOS LifeKeeper
Step 1: Understanding SIOS LifeKeeper
SIOS LifeKeeper is a comprehensive solution for achieving high availability (HA) and disaster recovery for critical applications in cloud and hybrid environments. It provides robust application-aware failover protection by combining continuous health monitoring, automated failover orchestration, and multi-layer recovery across physical, virtual, and cloud architectures. In AWS deployments, traditional failover mechanisms relying on Layer 2/3 broadcast or multicast protocols—such as VRRP or keepalived—do not operate natively due to cloud networking constraints. SIOS LifeKeeper solves these limitations by managing virtual overlay IP addresses, dynamic Route 53 DNS updates, or Elastic IP (EIP) reassociations, ensuring robust and seamless failover across Availability Zones without single points of failure.
Step 2: Integrating SIOS LifeKeeper with Nginx
- Deploy SIOS LifeKeeper agents on each EC2 instance running Nginx.
- Configure SIOS LifeKeeper communication paths between nodes to enable heartbeat monitoring across network interfaces.
- Set up a LifeKeeper resource hierarchy to manage the Nginx instances, IP addresses, and underlying dependencies as a highly available cluster.
- Configure a Quorum/Witness mechanism (such as an NFS file witness) to avoid split-brain scenarios in two-node cluster setups.
Step 3: Implementing EC2 Automatic Recovery Kit (ARK)
- Install the SIOS EC2 Application Recovery Kit (ARK) on both EC2 instances to facilitate automatic instance recovery in the event of failure.
- Configure the ARK to monitor EC2 instances and Nginx processes, triggering failover actions when a failure is detected.
- Test switchover and failover scenarios using the LifeKeeper administration interface to validate high availability and ensure seamless service continuity.
Conclusion
By deploying Nginx as a load balancer across multiple AWS Availability Zones and integrating it with SIOS LifeKeeper and the SIOS EC2 Application Recovery Kit (ARK), organizations can establish an enterprise-grade, highly available infrastructure tailored for mission-critical workloads. This architecture overcomes cloud networking constraints by automating active-passive failover and dynamic address reassociation, ensuring continuous uptime, data integrity, and seamless service continuity. Supported by application-aware monitoring and robust quorum management, this unified strategy minimizes downtime, optimizes performance, and provides a scalable, resilient foundation for modern digital operations.
Author: Aaron West, Sales Engineer at SIOS
Reproduced with permission from SIOS
With Great Power: Cluster Maintenance Best Practices
With Great Power: Cluster Maintenance Best Practices
In the world of IT infrastructure, uptime is everything. Keeping your users online and functional is the foundation on which most modern businesses run. In a world where downtime is the supervillain, clustering becomes the hero who can save your business from disaster. However, as the title states, “with great power comes great responsibility.” Similar to how heroes of fiction must control their powers to ensure they are not abused or misused, clusters (and the engineers working on them) must be controlled to ensure they are able to save the day when disaster does strike. In this blog, we will discuss some general best practices to keep your cluster prepared to battle a disaster situation. Just like a protagonist, it all starts with the training and maintenance.
Storage Changes in a Clustered Environment
Clusters are a powerful tool; with that power comes additional restrictions that a single system setup does not need to consider. When maintaining your clustered system, you must perform additional validation beyond what is needed for a basic system. For example, provisioning additional storage is a few clicks away through a cloud provider console; however, in a clustered environment, you do not want to jump the gun. A clustered environment with mirrored data will have much stricter requirements for storage than your typical server. Understanding this and how to work with clustered storage is key to ensuring your cluster stays at full strength 24/7. For a detailed overview of proper storage procedures, please feel free to review this technical document:
Beyond storage, there are further cluster considerations that you should arm your engineers with. Understanding the interplay between clustering and antivirus is key to preventing unwanted quarantine of critical clustering files/processes. Properly documenting system updates and ensuring clustered nodes remain consistent allows for all changes to be well understood by current and future engineers, and prevents problems that may stem from node inconsistencies.
Antivirus Software, Malware, etc. Exclusions for DataKeeper for Windows
Network Changes and Mirror Endpoints
Another major factor in clustering is networking. In order for a cluster to communicate, the network must function properly, and the configured addresses must stay consistent. In a single server environment, changing network infrastructure hardly comes into play, but when working with a clustered setup, all of the nodes must maintain communication on dedicated network paths, and changing those paths requires that you update the cluster configuration. SIOS does provide tools to update these values, but it is important to understand the procedure prior to making any changes.
Learn more about changing mirror end points, here.
To summarize, going from a standard single server setup to a “superhero” cluster is not an instant hands-free solution. Just like any superhero, it requires proper training and maintenance to perform at a “super” level. Careful considerations around any storage, networking, AV, and other system changes will allow for your cluster to stay in tip-top shape to save your clients and business from the wrath of downtime. For further information and training, please feel free to review more blogs, visit our documentation site, complete our online training, or reach out to support@us.sios.com if you are interested in procuring a more comprehensive synchronous training session with a dedicated SIOS engineer.
Author: Connor Toohey, Sr. Product Support Engineer at SIOS Technology
Reproduced with permission from SIOS