SIOS Recovery Kits provide a wealth of best practices for application-aware monitoring and recovery.  In general, each SIOS recovery kit provides a step-by-step programmatic approach to restoring the application, database, or service in accordance with High Availability (HA) best practices.  The SIOS Recovery Kits provide the intelligence needed to restore operation after a normal system shutdown, after an unexpected system failure or crash, and even in the case where the application, database, or service itself crashes or becomes unavailable.   In addition, each recovery includes experiential wisdom and improvements from over two decades in the field.

However, if a customer still needs to roll their own script for providing HA, SIOS LifeKeeper for Windows and SIOS LifeKeeper for Linux include an option for script integration via the Generic Application (Gen/App) Recovery Kit.

Best Practices for Writing Gen/App Recovery Scripts

1. Use Modern, Supported Scripting Languages for Gen/App Recovery

A common practice with existing solutions is to use the old existing scripts on new systems and architecture.  However, it is essential to make sure you are using a modern, supported scripting language.

2. Avoid Hardcoded Values in Gen/App Scripts

Using hardcoded values can cause portability issues, as well as challenges with long-term maintenance.  Avoid using hard-coded values that are subject to change in future deployments, for example, directory paths, user names, or similar.

3. Practice Code Reuse to Improve Gen/App Script Quality

Duplicate code is a common problem in customer-developed scripts.  Duplicate code creates quality, maintenance, and troubleshooting problems.  Practice code reuse, such as inheritance, functions, and subroutines.

4. Choose Meaningful Names for Functions and Variables

Descriptive variables are more helpful than single-character variables such as ‘n’ or ‘i.  When looking at code months or years later, will the variable ‘n’ mean as much as iReturnCode?

5. Remove Unused Functions and Variables to Prevent Code Bloat

While meaningful names for functions and variables are important, avoid cluttering the code with unused variables and functions.  Declaring variables and not using them can create confusion during future updates and troubleshooting.  While the days of 8 MB of memory are long gone, additional variables or functions that provided limited reuse or no additional value are still burdensome and create code bloat.

6. Verify All Input Parameters for Reliable Gen/App Execution

In the rush to get something working, don’t ignore input variable validation.  Verify all input to the script and to functions.  Don’t assume that if “we got here,” all of our inputs are valid.

7. Log Helpful and Actionable Messages

Consider what output needs to be logged for status/progress, error conditions, or troubleshooting.  Each message should be thoughtfully considered and appropriately worded to provide helpful feedback to operators and future developers.

8. Check Return Codes on All Method/Function/API Calls and Take Defensive Action

Commands that are executed within the body of the script or function will have return codes, explicitly pass, fail, or other.  Be sure to check, log, and properly handle both expected and unexpected return codes from methods, functions, and API calls.

9. Use Defensive Programming Techniques

Apply best practices for defensive programming, including least privilege access, input validation, error handling, etc.

10. Test Gen/App Recovery Scripts Beyond the Happy Path

Working code is not enough.  Develop a robust validation plan and test the code extensively, especially beyond the happy path when everything is expected to work.

11. Use Version Control for Script Management and Troubleshooting

Use version control and code management tools.  Version control is essential for troubleshooting, management, and tracking the inevitable fixes required for your scripts.

12. Catch Errors Early with Code Inspections and Peer Reviews

Use code inspections and peer reviews to increase the resilience and robustness of the code.  Code reviews help find problems early and reduce the cost, risk, and burden of late-stage failures and bugs.

13. Verify Permissions Required for Execution in Gen/App Recovery

Having well-organized, modern, reviewed, inspected, tested, and controlled code is an essential part of a well-crafted gen/app script.  However, the best-coded script will fail to execute if it does not have the right permissions.  Ensure that the script has the correct permissions to execute standalone as well as under the service/user accounts of the HA solution.

14. Comment Code Clearly to Explain Logic and Business Use Cases

Provide comments that help explain the business logic and use case, describe expected function inputs and returns, and contribute to overall understanding.  Well-written code still needs comments, especially if it is not obvious what business logic or requirement is being addressed.  An example comment block could look like:
Name:

Purpose:

Preconditions:

Postconditions:

Returns:

Ready to Simplify Gen/App Recovery with Confidence?

Don’t leave high availability to chance. With SIOS LifeKeeper and the Generic Application (Gen/App) Recovery Kit, you can safeguard critical applications, streamline recovery, and reduce downtime.

Request a demo today to see how SIOS can help you achieve reliable, cost-effective high availability and disaster recovery.

Author: Cassius Rhue, VP, Customer Experience at SIOS

Reproduced with permission from SIOS

Application Intelligence in the context of High Availability (HA) refers to the system’s ability to understand and respond intelligently to the behavior and health of applications in real time to maintain continuous service availability.

What is Application Intelligence?

So, what is Application Intelligence? Application intelligence involves monitoring, analyzing, and reacting to several factors. These can include application state, like whether the application is up or down? Performance metrics include response time, error rates, throughput, and memory usage. Application dependencies, such as databases or external services. Lastly, they look at user behavior or patterns. Using Application Intelligence takes a more holistic view of the application. It uses various data points to make educated decisions about the state of the application itself, not just the infrastructure. Let’s take the example of a web server; it’s not simply enough to know if the server is running, but is the site accessible without any errors? Is the response slow at all? Are users refreshing multiple times and trying to access it? Is the database the website relies on also up and running and accessible?  All the above are examples of the factors that application intelligence considers to be successful.

How LifeKeeper Uses Application Intelligence

So, how does LifeKeeper use application intelligence to enhance high availability for critical applications? Let’s break it down.  LifeKeeper uses application-specific recovery kits (ARKs) that contain knowledge for each application (SAP, SQL, PostgreSQL, Oracle, etc.). This allows LifeKeeper to handle the startup/shutdown procedures of each application, monitor the health and status of both the application and any dependencies, as well as orchestrate intelligent failover/failback operations without corrupting any data. Users can group together related resources in a hierarchical relationship within LifeKeeper, which allows LifeKeeper to understand the dependencies between different application components (when a service relies on an IP or database, for example). This ensures LifeKeeper failovers happen in the correct order and recovery actions don’t break the application or leave it in an inconsistent or broken state.

Additionally, LifeKeeper does deep health checks, not just determining if the server is up, but also more detailed checks, such as whether a database is accepting connections or if a web service is returning expected responses. It can even monitor if certain expected background processes are running. LifeKeeper also uses application-specific configuration files to ensure data configuration consistency across nodes and that application settings are preserved or restored correctly.  Lastly, LifeKeeper has the ability to use custom scripts to further fine-tune these deep checks to support less common or homegrown applications intelligently as well.

PostgreSQL ARK: A Real-World Example of Application Intelligence

To take a deeper dive, we can look at how PostgreSQL ARK uses Application Intelligence.  The PostgreSQL ARK uses specific logic to monitor, start, stop, and failover PostgreSQL via knowledge of the specific PostgreSQL startup and shutdown commands, awareness of critical config files like postgresql.conf and pg_hba.conf and understanding the data directory layout and lock file behavior.

Intelligent Monitoring and Ordered Failover for PostgreSQL

Additionally, it doesn’t just check that PostgreSQL is running, it also checks if the database is responding to queries, the correct data directory is accessible, and if there is any corruption in the transaction logs?  It uses dependency tracking to make sure that the resources PostgreSQL often depends on are available such as the Virtual IP for client connections and the mounted storage for its data directory.  This ensures that LifeKeeper can bring up the resources in the correct order in case of a failover, such as mounting the disk first, bringing up the IP, and then starting PostgreSQL before verifying the service health.

Preventing Split-Brain and Ensuring Data Integrity

Lastly, LifeKeeper uses application intelligence to avoid split-brain (a phenomenon where more than one node thinks it’s the ‘primary’ node) scenarios by avoiding starting two active PostgreSQL servers with the same data directory and avoiding data corruption by not failing over when writes are still in progress. These are examples of all the different ways LifeKeeper and the various ARKs have implemented application intelligence to make the combined product as resilient as possible.

Strengthen Application Resilience with Intelligent High Availability

In summary, LifeKeeper’s built-in application intelligence enables precise, fast, and reliable failover and recovery by understanding how applications behave and what they need to run correctly.

Ensure application resilience and uninterrupted service—request a demo or start your free trial today to experience how SIOS LifeKeeper uses application intelligence to protect your critical workloads.

Author: Cassy Hendricks-Sinke, Principal Software Engineer, Team Lead

Reproduced with permission from SIOS