SIOS SANless clusters

SIOS SANless clusters High-availability Machine Learning monitoring

  • Home
  • Products
    • SIOS DataKeeper for Windows
    • SIOS Protection Suite for Linux
  • News and Events
  • Clustering Simplified
  • Success Stories
  • Contact Us
  • English
  • 中文 (中国)
  • 中文 (台灣)
  • 한국어
  • Bahasa Indonesia
  • ไทย

How to Understand & Respond to Availability Alerts

January 29, 2021 by Jason Aw Leave a Comment

Understand & Respond to Availability Alerts

Houston We Have a Problem (or How to Understand & Respond to Availability Alerts)

A Successful Failure

Houston we have a problem!  It is an iconic line that reminds countless space buffs and movie fans about the great difficulty, potential disaster, and the perilous state of the Apollo 13 space mission – a mission NASA now calls “A Successful Failure.”  Ignoring your own application availability alerts may not go down in history as a defining moment, but can also wreak similar havoc

Now back to 1970:

“A routine stir of an oxygen tank ignited damaged wire insulation inside it, causing an explosion that vented the contents of both of the Service Module’s (SM) oxygen tanks to space. Without oxygen, needed for breathing and for generating electric power, the SM’s propulsion and life support systems could not operate. The Command Module’s (CM) systems had to be shut down to conserve its remaining resources for reentry, forcing the crew to transfer to the Lunar Module (LM) as a lifeboat. With the lunar landing canceled, mission controllers worked to bring the crew home alive.”

An explosion of oxygen tanks triggered alarms, warnings, pressure and voltage drops, interrupted communications, and then the now famous radio communication between the astronauts and Mission Control.  But what if, after the explosion, the crew did nothing? What if they never checked on the explosion, never responded to the warnings and gauges, and never informed Mission Control of there being an issue?  What if Mission Control, after being notified or alerted back at their dashboard in the control center, never attempted to provide any assistance?  What if the team buried their heads in the sand, or resigned themselves to fate and chance, never tried to learn, improvise, or improve from the failure they encountered?  The result would have been tragic!  It may have made it to a documentary, but hardly a blockbuster movie featuring an iconic line.

What Do You Do When an Alert is Triggered in Your Environment?

Space walks are a far cry from our own day to day activities, unless of course you work for NASA, but recent blogs on Apollo 13 do spark a question applicable to availability.  What do you do when there is an alert triggered in your environment? Do you just ignore it?  Do you downplay it, waiting to see if the alerts, log messages, or other indicators will just go away?  Do you contact your vendor support to understand how you can disable these alerts, warnings, and messages?  Or do you say, “We have a problem here and we need to work it out”?

As a VP of Customer Experience at SIOS Technology Corp. we have experienced both sides of alerts and indicators.  We have painstakingly walked with customers who chose to ignore warnings, turning off critical alerts that indicated issues, ranging from application thresholds to network instability to potential data inconsistency.  And we have also seen customers who have tuned into their alerts, investigated why their alarms were going off, uncovered the root cause and enjoyed the fruit of their labor.  This fruit is most often the sweet reward of improved stability, innovation and learning, or an averted disaster.

4 things you can do when you your availability product triggers an alert

1. Determine if the type and criticality of the availability alert.

Is the alert or error indicative of a warning, an error, or a critical issue? A good place to assist you and your team with understanding criticality is to consult with available documentation. Check the product documentation, online forums, knowledge base articles (KBA), and internal team data and process manuals.

2. Assess the immediacy of the alert. 

For warnings and errors, how likely are they to progress into a critical issue or event.  For critical issues and alerts, this may be obvious but an assessment, even of critical events will provide some guidance on your next steps; self-correction, issue isolation, or immediate escalation.

3. Consult additional sources. 

What other sources can you access to make a determination about the alert condition? For example, if the alert is storage related, are there other tools that can expose the health of your storage?  If the issue is a network alert, are there hypervisor tools, traffic tools, NIC statistics, or other specialized monitoring tools deployed to help with analysis.

4. Contact support.

In other words, if you are unsure, alert Mission Control. After determining the type, assessing the immediacy, and consulting additional sources, it is a good idea to contact your vendor for support.  A warning about a threshold for API calls may seem innocent. But if the API calls will fail once such a limit is reached, this could be cause for immediate action. Getting the authority of the specialist can be helpful in keeping peace of mind and avoiding disaster.

An experienced vendor like SIOS can help you quickly identify the causes of problems and recommend the best solution.

Repeatedly ignoring problems in your availability environment can lead to unexpected, but no less devastating results. Addressing the problems indicated by alerts, log messages, warning indicators, or other installed and configured indicators gives your customers, your business, your teams, and yourself the “opportunity to solve the problems,” before it becomes a disaster. And at the same time, strengthens your availability strategy and infrastructure.  Which will you choose?

–  Cassius Rhue, VP, Customer Experience

Reproduced from SIOS

Filed Under: Clustering Simplified Tagged With: Application availability, application monitoring, disaster recovery, High Availability, high availability - SAP, SQL Server High Availability

Six Reasons Not to Buy SIOS High Availability Software . . . If You Dare

October 25, 2020 by Jason Aw Leave a Comment

Six Reasons Not to Buy SIOS High Availability Software . . . If You Dare

Six Reasons Not to Buy SIOS High Availability Software . . . If You Dare

Six Reasons Not to Buy SIOS High Availability Software . . . If You Dare

You need SIOS Protection Suite (for Linux or Windows) or SIOS DataKeeper Cluster Edition for high availability protection for business critical applications.

UNLESS

1. You prefer free solutions only.

I get it. There are definitely times when I do the same thing when I need to learn a new skill, get a quick tip, drop a few pounds, or set up a quick demo. Rather than signing up for a subscription, purchasing a license, or investing in a combination of the two, I have gone the free route.

However, the saying often holds true, you get what you pay for. Free trials are fine. Permanently free high availability is like gas station sushi – is the risk really worth it? Be sure that free doesn’t prevent you from utilizing the fullness available for optimizing uptime and increasing availability. Make sure you aren’t passing over a reasonably priced high availability solution that is proven to protect your mission-critical applications.

2. Being a single solution shop solution is more important than meeting your HA needs.

We were a “Ford tough” family for decades. Seriously. I understand what it is like to be a one solution shop. My dad owned a Ford truck for work, a Ford Mustang for leisure, a Ford 3600 tractor for the farm, and a Ford minivan for family travel. There was even a season where we received model toy cars with the brandished blue oval as well.

But, when my wife and I were branching out on our own family needs, we broke away from the single solution to address needs that fell out of the Ford wheelhouse (at the time). You may be a single shop buyer, but if your needs have changed and the HA provider or solution hasn’t kept up, consider whether expanding the solution set will eliminate risks, improve success, or be worth the investment in a complementary solution for those new needs. When we needed a reliable, gas efficient, sleek, family friendly, and economical solution for our family, we supplemented Ford tough with a Honda Odyssey. If you are a single stop shop, and you are not worried about vendor lock-in best of luck.

3. You are more of a do it yourself-er coder.

You like coding. You like to write a lot of scripts, and don’t mind pulling out your bash, ksh, perl, python, powershell, batch or command tool kit and wiring things up yourself. You value the joy in flexibility and adding your own tweaks.

I love writing code as well, but there are times when the last thing I want to do is spend time writing a lot of code and scripts for a problem that is solved, proven, and off the shelf ready. For the do-it-yourself admin, off-the-shelf may not be your preference, but consider whether 20 years of expertise and experience should be rehashed and re-architected for your enterprise. But, if you have to get the code writing fix in, High Availability Software SIOS provides the Generic Application Recovery Kit for you to get in a coding fix.

4. You need Ubuntu support (or Solaris).

Your environment is unique. You have customers who’ve cut their teeth on Solaris and are hanging on to it for dear life. Or you’ve got those who have fully embraced the Linux realm and have moved to Ubuntu. In either case, you look at the SIOS products matrix and Ubuntu isn’t currently a match for your SIOS version. Bummer!

While this is true, consider the rich and vast features and flavors of support that are still available. While there are parts of your enterprise that have dug in on Solaris and others that have raced to embrace Ubuntu and newer variants of Linux, it is more likely that you need a solution capable of supporting RHEL, OEL, SuSE, CentOS and possibly Windows as well. Be sure not to single out a high availability solution by what it doesn’t provide and consider the depth of what it does.

5. You don’t run a hybrid of anything in your environment.

I heard it in the middle of a movie last week. The lead character commented on the idea of moving forward with some new idea of an overly excited owner. The classic line: “Sometimes the juice isn’t worth the squeeze.” In your mind you feel that you aren’t running a hybrid environment. Your applications are critical, but not complex. The moving parts are simple- a database, front end and a supporting application. It makes sense that you might not want to “complicate” things with additional processes, products, solutions or services, and you may feel like the juice isn’t worth the squeeze.

Before you make that final decision for a High Availability Software, assess whether a non-hybrid environment is the same as a simple environment. Consider whether or not the moving parts are as simple as you imagine or whether a solution with failover orchestration would be beneficial to reducing your overall RTO and increasing your RPO.

6. Endorsements from HA experts and experience don’t matter.

I bought a set of headphones online in mid-April. As I suspected, I discovered that anyone can do bluetooth headphones. But, not everyone can do them well. Ergonomically, the “new to market” headphones are a nightmare. Pairing was a breeze, but accidental unpairing is a constant battle. The sound quality is amazing, but that amplifies my annoyance when the headphones randomly chirp – loudly and clearly – for system sounds or at the end of a song.

You may believe that high availability and application monitoring can be done by anyone and that experience doesn’t matter. However, consider your own experiences and mine and ask if you’d really want to trust your enterprise environment to a group that just started thinking about the complexities of hybrid environments, or the dependencies and application-centric knowledge needed for the applications you use most frequently.

When deciding the right High Availability Software for your environment, consider carefully whether you want to go without the many best in class features, hardened and tested solutions, knowledgeable experts, broad swath of supported applications and environments, and industry leading experience and decades of insight. Then after careful consideration, choose wisely.

-Cassius Rhue, Vice President, Customer Experience, SIOS

Reproduced with permission from SIOS

Filed Under: Clustering Simplified Tagged With: Application availability, disaster recovery, High Availability, SQL Server

Test/QA Systems are a Critical Part of Enterprise Availability

July 8, 2020 by Jason Aw Leave a Comment

Test/QA Systems are a Critical Part of Enterprise Availability

Test/QA Systems are a Critical Part of Enterprise Availability

“I could kiss you,” that’s what a friend blurted out to me nearly three decades ago as she ran towards me. She had dropped her reeds for her saxophone on the way to one of the biggest band competitions in our region. I didn’t know whose they were, but when I saw the pack of reeds on the seat on the bus I picked them up and took them with me to the warm-up area. Three minutes into her warm-up, her 1st reed cracked and she panicked as she reached into empty pockets for replacements. When I piped up that I had found them, she blurted out, “I could kiss you right now.”

As the VP of Customer Experience at SIOS Technology Corp. I have the unique and distinct pleasure of working with a number of enterprise customers and partners at different phases of the availability spectrum. Sometimes I have the opportunity of working with end customers for issue resolution, mitigation, and improvements. At other times our teams are actively working with partners and customers to architect and implement enterprise availability to protect their systems from downtime. A recent customer experience reminded me of something that happened nearly 30 years ago when my friend blurted out, “I could kiss you.”

My team and I were on a customer call. The call began with the usual pleasantries, introductions, and an overview of the customer’s enterprise environment. Thirty minutes into the call, things were going so well. Their architecture was solid, thoughtful, and well documented. Their team was knowledgeable, technically sound, and experienced. But then, the customer intimated that due to cost savings they would not be planning to maintain a dedicated test/quality system. I took a deep breath.  Actually it was more of an exhale like the rush of air from a gut punch. I prepared to respond, but before I could a voice broke through.  “The number one cause of downtime is lack of process,” exclaimed the Partner Rep Architect on the call with us. After a brief banter, the customer agreed to maintain a test/QA system and I nearly blurted out, “I could kiss you!”

On the front lines of many Enterprise deployments (new systems, data center migrations, and system updates) my teams in Support and Services have seen dozens of issues that could have been mediated by utilizing a test system/cluster.

A test/quality system is an invaluable part of an HA strategy to avoid downtime. Common tasks associated with maintaining an enterprise deployment such as patches, updates, and configuration changes come with risk. Enormous risk.

Commonly identified risks of testing in production include several serious and potentially catastrophic issues: 

  • Corrupted or invalid data
  • Leaked protected data
  • Incorrect revenue recognition (canceled orders, etc.)
  • Overloaded systems
  • Unintended side effects or impacts on other production systems
  • High error rates that set off alerts and page people on-call
  • Skewed analytics (traffic funnels, A/B test results, etc.)
  • Inaccurate traffic logs full of script and bot activity (a)

If a customer attempts to apply risky changes in production, the result can be quite damaging. On top of those listed above, there is an increased risk of downtime, corruption of application installations, and in some cases irreversible damage. Take the case of Customer X (a high profile SAP Enterprise shop in the manufacturing industry).

After reading a critical notice from a reputable site, the OS Administrator quickly updated his production nodes to the latest kernel update available. Within hours the Production nodes began a series of uninitiated crashes and kernel panics. In his haste, he had installed a kernel that was incompatible with his configuration; the combination of existing application packages, devices, file systems, and related packages. This caused a production outage and several high priority escalations to multiple vendors.

When patches are applied to a test/QA or sandbox system, patches and critical fixes can be managed and verified to reduce loss of productivity and unplanned downtime. Testing applications in a production-like environment allows you to identify unforeseen problems and correct the issues before they adversely impact your operations. Pre-production design and testing eliminate costly business disruption, improve your customer experience and protect your brand.

Using a test QA System to Improve Production Availability and Processes

Here are the basics that using a test/QA system, can provide for improving your production availability and processes. A controlled environment, that is similar (it must resemble production as close as possible) to the production environment, provides the ability to:

  1. Test kernel updates and security updates
  2. Validate settings and configuration tuning
  3. Reproduce production issues and test software updates and patches
  4. Verify application version compatibility and reduce the risk of downtime due to incompatible changes
  5. Provide a safe space to practice and revise go-live, maintenance, outage, and other enterprise procedural activities
  6. Train new hires and team members without impacting enterprise clients

If you have a Test/QA environment for deploying your critical enterprise availability software, I could kiss you right now. Having this environment gives your team the ability “to test, validate and verify(2)” architecture, business requirements, user scenarios, and general integration with a system or set of systems that most closely resembles the production environment- you know the one that makes the money. Of course, you will still have to schedule windows to maintain your production systems and perform testing on them as well, but after a safe buffer step has been completed in between.

— Cassius Rhue, VP, Customer Experience

————-

References:

  1. https://opensource.com/article/19/5/dont-test-production Accessed 5/4/2020
  2. https://www.softwaretestingclass.com/system-testing-what-why-how/ Accessed 5/4/2020

Filed Under: Clustering Simplified Tagged With: disaster recovery, High Availability, Q&A, Risks, Testing

Enterprise Availability: Lessons from the Court

June 29, 2020 by Jason Aw Leave a Comment

Enterprise Availability: Lessons from the Court

Enterprise Availability: Lessons from the Court

I love basketball. I love to play it, watch it, and think through the cerebral aspects of the game; the thoughts and motivations, strategy and tactics. I like to look for the little things that work or fail, the screen set too soon or the roll that happened too late. I like defense and rotation. I like to know the coaches’ strategy for practice, walkthroughs, travel, and so on.  So naturally a few months ago, when I had a day off from the 24/7 world of availability, imagine that, I took my day off to watch basketball, and more specifically my daughter’s middle school basketball practice.

About a third of the way through watching, I couldn’t contain myself. I whistled to and “prodded” the young girl lollygagging and trotting up the court and yelled, “Run! Hustle!”  And she did, as did the teammates within earshot.  The next few minutes, plays, and drills were filled with energy, crisp cuts, smooth motions, and drive.  But, it didn’t last.  Instead, there were more whistles required, more emphatic pleas to move and run, to play hard, make sharp cuts, dive, pay attention, focus, learn, and correct. When the 2 hours were nearly over I took my last moment of attention to prophesy, “The way you practice will be the way you play!”

I can almost feel you channeling the spirit of AI, not Artificial Intelligence (AI), Allen Iverson (AI).  “Are we talking about, practice.  Practice!”  I thought this was about availability.  Well, my love for basketball met my passion for availability when I considered my daughter and her teammates. How?

Three Ways Basketball Strategies Are Like Availability Strategies:

  • In basketball, every team needs a plan, ditto for enterprise availability.
  • In basketball, every team needs to practice that plan, ditto for availability, disaster recovery, and especially planned maintenance.
  • In basketball, the plan when tested under fire will hold up only as good as those plans were practiced

Enterprise Availability Needs a Plan 

Your availability, specifically your disaster, planned maintenance, and outage recovery strategies, are only as good as those you create. Simply put, what is your plan for an outage (note clouds fail, servers crash, networks get saturated, and human error— enough said).  Do you have a documented plan?  Do you have identified owners and backups owners?  Do you know your architecture and topology (what server does what, where is it located, what team does it belong to, what function does it serve, what business priorities are tied to it, and what SLO/SLA does it require)?  Who are your key vendors, and what are their call down lists?  What are your checkpoints, data protection plans, and backup strategies?  And what are your test plans and validation plans for verification of this plan?

Enterprise Availability Needs Practice 

A good plan, check. Now what about practice.  Implementing disaster recovery steps and unplanned outage strategies are a necessary component of every, every enterprise configuration. But, a strategy that is not rehearsed is not really a strategy. In that case it is simply a possible and proposed approach.  It is more like a suggestion, rather than an actual plan of record. The second step is practice. Walk through the strategies of your plan. Rehearse maintenance timings. Restore backups and data. Validate assumptions and failure modes.

Enterprise Availability Requires Testing 

A plan and a walkthrough, check. Now that you have two of the three let me go back to my daughter’s team.  My parting words, as an “unofficial coach “ were as follows: “The way you practice will be the way you play!”  Fast forward three days. The game is down to the final minutes. The team they are playing is athletically mismatched, and outsized just as they were last year when that year’s game was over by halftime.  But this year, the undermanned and undersized team had clearly come in more prepared. What should have been an easy win now enters the final minute nearly tied. The home team, the opponent, begins a press— something my daughter’s team had prepared for, albeit haphazardly and lethargically, during that fateful practice.  What ensued wasn’t pretty. Four unforced turnovers, two critical fouls during three-pointer attempts, a four to nothing run, and a bevy of frustrations culminating in a devastating one-point loss as time expired.

My final point, how well are you practicing for your real outage, disaster, or planned maintenance?  Do you practice with real data, real clients, and with a real sense of urgency?  How often does your upper management check-in?  Trust me, the presence of a boss in pressure-packed moments makes people do strange and unwise things!  Does your sandbox and test system look like production?  In a past life, I once worked with a customer who had different hardware, storage and Linux OS versions between prod and QA. When they went into prod with application updates, disaster struck hard.  Do you have users and data, and jobs that run during your testing? What about actual disaster simulation?  It’s a hard pill to swallow, testing a hard crash with potentially destructive consequences, recovery from offsite, and even harder to simulate simultaneous multi-point, multiple systems failures, but the unpracticed is often the weak point that turns a 2 hour planned maintenance into an eight-hour multi-team corporate disaster.  The under-practiced or poorly practiced is the difference between a stunning victory for your strategy and team, or a crushing defeat and costly failure for team, vendors, enterprise, and customers.

In basketball, the plan under fire will hold up only as good as the plan was practiced.  When implementing a recovery and disaster plan a good plan and validation are key, but great practice is king.

Contact a rep at SIOS to learn how our availability experts and products can help you with the plan, procedures, and practice.

Visit back for a post on tests you should never avoid simulating.

— Cassius Rhue, VP, Customer Experience

Article reproduced from SIOS

Filed Under: Clustering Simplified Tagged With: Application availability, disaster recovery, Enterprise Availability

SQL Saturday: High Availability and Disaster Recovery for SQL Server in Azure IaaS

August 23, 2019 by Jason Aw Leave a Comment

High Availability and Disaster Recovery for SQL Server in Azure IaaS

Speaker: Jason Aw

Duration: 60 minutes

Level: Beginner

Track: Application, DBA and Database Development

The CEO just tasked you with moving all the SQL Server instances to Azure, or maybe you are deploying a brand new application and want to leverage Azure IaaS to host SQL Server. Beyond security and performance, your most pressing concern might be ensuring SQL Server running in Azure is highly available.

While on-prem high availability and disaster recovery options for SQL Server are well defined, moving those instances to Azure immediately presents some questions and challenges. Can I simply lift and shift my SQL Server Failover Cluster Instance to the cloud? Do I need to upgrade to SQL Server Enterprise Edition and us Always On Availability Groups? What about shared storage and failover clustering? What about disaster recovery, what are my options there? Load Balancers, Fault Domains, Availability Zones, Azure Site Recovery and Region Pairs, what are these things and why do they matter to me?

High Availability and Disaster Recovery professional Jason Aw with 20 years of experience explains all this and more in detail.

Accompanying Material

  • [Powerpoint] – PASS_Perth_SQLSaturday_2019_SIOS_Datakeeper_HA_DR.pdf

Filed Under: News and Events Tagged With: disaster recovery, High Availability

  • « Previous Page
  • 1
  • …
  • 16
  • 17
  • 18
  • 19
  • 20
  • …
  • 26
  • Next Page »

Recent Posts

  • Fail Open vs. Fail Closed: Why It’s a False Dilemma (and How True HA Fixes It)
  • The 5-Step Checklist for Validating Your Failover Setup
  • Managing an Inherited DataKeeper Cluster: Challenges and Best Practices
  • Setting Up Nginx Load Balancer on AWS
  • With Great Power: Cluster Maintenance Best Practices

Most Popular Posts

Maximise replication performance for Linux Clustering with Fusion-io
Failover Clustering with VMware High Availability
create A 2-Node MySQL Cluster Without Shared Storage
create A 2-Node MySQL Cluster Without Shared Storage
SAP for High Availability Solutions For Linux
Bandwidth To Support Real-Time Replication
The Availability Equation – High Availability Solutions.jpg
Choosing Platforms To Replicate Data - Host-Based Or Storage-Based?
Guide To Connect To An iSCSI Target Using Open-iSCSI Initiator Software
Best Practices to Eliminate SPoF In Cluster Architecture
Step-By-Step How To Configure A Linux Failover Cluster In Microsoft Azure IaaS Without Shared Storage azure sanless
Take Action Before SQL Server 20082008 R2 Support Expires
How To Cluster MaxDB On Windows In The Cloud

Join Our Mailing List

Copyright © 2026 · Enterprise Pro Theme on Genesis Framework · WordPress · Log in