disaster recovery | SIOS SANless clusters

Stages of IT Disaster Recovery Grief

March 8, 2021 by Jason Aw Leave a Comment

Stages of IT Disaster Recovery Grief

Disaster recovery grief can hit you out of nowhere if you haven’t implemented the right enterprise availability architecture. Meet our friend Dave in IT to walk us through the 5 stages of disaster grief.

Stage 1: Denial

Dave in IT: “Uh oh. What’s that alert? It’s just a little application crash, right? No big deal. I’ll have things up and running in no time.”

In the land of enterprise availability, there is no such thing as a little application crash or no big deal. Companies have SLA with real money on the line. Your selective reality is probably not the same perspective of your customers and stakeholders.

Stage 2: Anger

Dave in IT: “Are you kidding me. Of all the… [censored]...times, today the application won’t start. Ughh. I hate this[censored]...[censored]... application. Wait, what’s this new alert. Seriously, now, the datacenter is down!”

It gets messy really, really fast in the fast pace, and high stakes environments. When unchecked alerts and failures happen, problems can mount quickly along with pressure, frustration and anger.

State 3: Bargaining

Dave in IT: “Hey Ard in Applications, this is Dave in IT. Do you guys have any backups for the App1 environment? . . .Ard are you sure? Could you just check again? I know you’ve checked twice, but can you check one more time. I’ll buy drinks on Taco Tuesday!”

Dave in IT: “Hey Donna DBA, this is Dave in IT. Art in Applications said you might help me out. Did you by chance setup any database replication for that finance database or the inventory management system? . . . Are you sure? Umh, do you remember if we have any way to recover from a umh . . . datacenter crash?”

When my daughter gets in trouble, bargaining is her first go to. Okay, second. The first is to disappear, but you’re too smart to just walk away from the flames. But, Dave in IT isn’t the only one to realize that bargaining and begging is a poor substitute for a well defined strategy for high availability and disaster recovery. Skip the bargaining and begging about your disaster because “80% of the people don’t care, and 20% are glad it’s you (paraphrased from Les Brown).”

Stage 4: Sadness

Dave in IT: “This is just great. The application server crashed, the datacenter is down, and backups, if I can find them and if I can load them, will take hours to get restored. There is no way I’m getting out of this… where did I put that updated resume.”

Of course you have backups, and you’ve validated them. But there is an RTO and RPO impact of going back to those backups. Are you able to absorb this time? That is of course, after your data center recovers.

Step 5: Acceptance

Dave in IT: “It’s been two hours. I never knew we had this many Executive stakeholders before. No way I’m making it to my 2nd year anniversary after this. Well, I guess I’ll clean out my office tomorrow. No way I’m making it through this!”

Failures happen. Datacenters go down. Applications fail. There is no denying the possibility of losing a data center, having a server fail, or an application crash. This type of acceptance is normal, a part of improving your availability. Accepting that you may lose your job or worse because you failed to implement an availability strategy is something the experts at SIOS Technology Corp. want to make sure you avoid.

Don’t be like Dave in IT. Avoid the stages of disaster grief, and the hours of disaster recovery and downtime by architecting and implementing an enterprise availability architecture that includes the best of hybrid, on-premise, or cloud coupled with the best solution for monitoring, recovery, and system failover automation.

– Cassius Rhue, VP Customer Experience

Reproduced from SIOS

Why Does High Availability Have To Be So Complicated?

March 1, 2021 by Jason Aw Leave a Comment

Why Does High Availability Have To Be So Complicated?

Why Does love High Availability Have To Be So Complicated?

It’s the Hallmark movie season, I mean Christmas season, I mean Hallmark Christmas movie season… (don’t judge too harshly, I’m a father of six young ladies, a hopeless romantic, and married to an amazing spouse who enjoys a good holiday laugh and happy ending). If you are in the Hallmark movie season, you know that it is highly likely that you’ll hear the phrase, “Why is love so complicated?” It will be spoken just before the heartbroken young person has developed feelings for a new love interest, and is ready to dance the night away in their arms, just as the old flame walks into the party. If you aren’t into the Hallmark holiday romances, maybe it isn’t love that you are wondering about. Perhaps you want to know: “Why does high availability have to be so complex.

Ten Reasons That High Availability Is So ‘Gosh Darn’ Complicated:

The speed of innovation

Cloud computing, edge computing, hyper converged, multi-cloud, containers, and machine learning are changing the landscape of enterprise availability at a blistering pace. By conservative estimates, AWS currently has over 175 services, and “provides a highly reliable, scalable, low-cost infrastructure platform in the cloud that powers hundreds of thousands of businesses in 190 countries around the world.” Choosing an HA solution that allows consistent management across all of these environments, with infrastructure and application awareness is an important way to reduce complexity.
Randomness of disasters

Someone once said, “make your solution disaster proof, and the universe will build a better disaster.” Not only are we seeing innovations in the realm of technology, but also in the world of disasters. Resource starvation, cooling system disasters, natural disasters, power grid failures, and a host of new and random disasters often make it harder to insulate the entirety of your enterprise. Last year’s solutions will likely need updates to handle this year’s unprecedented outages. It’s important to work with a vendor that has focused on high availability for many years – who has firsthand experience with finding solutions to the randomness of disasters.
Application complexity

As technology moves head in the realm of virtualization and cloud computing, applications are following suite. As these application vendors add new options to take advantage of the cloud, they are also adding additional complexity. Your applications should be protected by solutions designed for higher availability and clustering in AWS, Azure, GCP or other environments. Look for vendors who provide greater application awareness, understanding of best practices, and who deliver availability solutions architected to taking account of how the application may have been architected and are able to optimize the application’s orchestration in the cloud.
Advances in threats

The threats to your enterprise also impact your availability. Systems have always had to handle the attacks from intruders, hackers, and even the self-inflicted. These attacks have become more sophisticated, and the solutions and methods to avoid being victimized often impact the layout, architecture, and software that is deployed within your organization. This software has to “play nice” with your availability solution and your applications. As VP of Customer Experience for SIOS Technology, I have seen how an overly aggressive virus scanner can impact your application and your availability solution. Ensure you understand the impact of your security systems on your HA/DR environment and choose a HA solution that works with, not against your security goals.
Regulatory requirements

Data breaches impact the architecture for your application, hypervisor and environment, but so too does the regulatory requirements. Businesses that have become global now have to make sure they are compliant with data handling regulations in multiple countries. This can impact what region your solutions can be deployed in, and how many zones you can use for redundancy. Additional, regulatory requirements can also impact the teams that can support your organization which may impact the choices for your availability software and support.
Shrinking windows

In the world of 24/7 searches, shopping, gaming, banking, and research the windows are shrinking. Queries must run faster and take less time. Responses have to be quicker and have better data. This means that the allowable downtime for your environment is shrinking faster than you previously imagined. It also means that maintenance windows are tighter, packed, and have to be optimized and highly coordinated. Work with an HA vendor that can provide guidance on optimizing your cluster configuration for both application performance and fast recovery time.
Increasing competitive pressure

I grew up in a small town. The hardware store had one competitor. The grocery store had one competitor. The bookstore, antique shop, car dealership, rental office, and bank all had one competitor. Today, you have thousands upon thousands of competitors who want nothing more than to see your customers in their checkout carts. This competition impacts the complexity of your entire business. It weighs heavily on what can and cannot be done in maintenance windows, with upgrades, and at what speed you innovate. Environments that may have been refreshed once every five years have moved to the cloud where optimizations and advancements in processor speed and memory can be had in seconds or minutes. Systems that once had a single run book covering a simple list of applications now look closer to “War and Peace” and cover the growing number of processes, products, services and intelligence being added to increase profits while simultaneously working to reduce risks and downtime.
High availability solution costs

We all wish we had an unlimited budget, but the reality between what you have available is sometimes somewhere between a little and not enough. Teams are often forced to balance consumption versus fixed cost, license costs for applications on the standby clusters, and associated costs for availability software. Enterprise licenses often add a ‘tough to swallow’ price tag for a standby server in an availability environment. Architecting an availability solution is never free, even if you are a hard core ‘DIY’ team. DIY comes with additional costs in maintenance, management, source control, testing, deployment, version management and version control, patches, and patch management. While your team of experts may be clearly up for the challenge, your business likely would prefer their highly valued talents be applied to creating more revenue opportunities.
Business growth

Growth of your business due to innovation means that your teams are now responsible for more critical applications, more sites, more offices, and more data that needs to be accessible and highly available. As your business grows and thrives the challenges that come with scaling up and scaling out add to the complexities mentioned previously, but also just expand what you have to prepare and plan for.
Team turnover

The complexity of the environments, speed of innovation, growth of your business, advances in the application tier, and growth in the competitive landscape brings with it the challenge of retaining top talent to keep your infrastructure running smoothly. Most companies understand that availability is a merger of people, process, product, and architecture among other things. So finding ways to reduce the complexity of clustering environments with automated configuration, documented run books, leveraging products with consistent HA strategies across the infrastructure is a key to both retaining the talent that installs and manages your infrastructure, and mitigating the risks and heavy lifting of those responsible for the key components of availability.

Let’s face it, love takes hard work, good communication, time, investment, skill and determination. There are no shortcuts to a successful relationship. The same can be said about achieving the best outcomes in an ever emerging, increasingly complex, and fluid technology space within your enterprise. Availability, clustering, disaster recovery and up time is so ‘gosh darn’ hard because it requires a serious, dedicated, non-stop top to bottom cultural shift accounting for the speed of innovation, the complexity of applications and orchestration, competition and growth, and the other components of keeping applications, databases, and critical infrastructure available to those who need them, when they need them.

-Cassius Rhue, Vice President, Customer Experience

How to Understand & Respond to Availability Alerts

January 29, 2021 by Jason Aw Leave a Comment

Houston We Have a Problem (or How to Understand & Respond to Availability Alerts)

A Successful Failure

Houston we have a problem! It is an iconic line that reminds countless space buffs and movie fans about the great difficulty, potential disaster, and the perilous state of the Apollo 13 space mission – a mission NASA now calls “A Successful Failure.” Ignoring your own application availability alerts may not go down in history as a defining moment, but can also wreak similar havoc

Now back to 1970:

“A routine stir of an oxygen tank ignited damaged wire insulation inside it, causing an explosion that vented the contents of both of the Service Module’s (SM) oxygen tanks to space. Without oxygen, needed for breathing and for generating electric power, the SM’s propulsion and life support systems could not operate. The Command Module’s (CM) systems had to be shut down to conserve its remaining resources for reentry, forcing the crew to transfer to the Lunar Module (LM) as a lifeboat. With the lunar landing canceled, mission controllers worked to bring the crew home alive.”

An explosion of oxygen tanks triggered alarms, warnings, pressure and voltage drops, interrupted communications, and then the now famous radio communication between the astronauts and Mission Control. But what if, after the explosion, the crew did nothing? What if they never checked on the explosion, never responded to the warnings and gauges, and never informed Mission Control of there being an issue? What if Mission Control, after being notified or alerted back at their dashboard in the control center, never attempted to provide any assistance? What if the team buried their heads in the sand, or resigned themselves to fate and chance, never tried to learn, improvise, or improve from the failure they encountered? The result would have been tragic! It may have made it to a documentary, but hardly a blockbuster movie featuring an iconic line.

What Do You Do When an Alert is Triggered in Your Environment?

Space walks are a far cry from our own day to day activities, unless of course you work for NASA, but recent blogs on Apollo 13 do spark a question applicable to availability. What do you do when there is an alert triggered in your environment? Do you just ignore it? Do you downplay it, waiting to see if the alerts, log messages, or other indicators will just go away? Do you contact your vendor support to understand how you can disable these alerts, warnings, and messages? Or do you say, “We have a problem here and we need to work it out”?

As a VP of Customer Experience at SIOS Technology Corp. we have experienced both sides of alerts and indicators. We have painstakingly walked with customers who chose to ignore warnings, turning off critical alerts that indicated issues, ranging from application thresholds to network instability to potential data inconsistency. And we have also seen customers who have tuned into their alerts, investigated why their alarms were going off, uncovered the root cause and enjoyed the fruit of their labor. This fruit is most often the sweet reward of improved stability, innovation and learning, or an averted disaster.

4 things you can do when you your availability product triggers an alert

1. Determine if the type and criticality of the availability alert.

Is the alert or error indicative of a warning, an error, or a critical issue? A good place to assist you and your team with understanding criticality is to consult with available documentation. Check the product documentation, online forums, knowledge base articles (KBA), and internal team data and process manuals.

2. Assess the immediacy of the alert.

For warnings and errors, how likely are they to progress into a critical issue or event. For critical issues and alerts, this may be obvious but an assessment, even of critical events will provide some guidance on your next steps; self-correction, issue isolation, or immediate escalation.

3. Consult additional sources.

What other sources can you access to make a determination about the alert condition? For example, if the alert is storage related, are there other tools that can expose the health of your storage? If the issue is a network alert, are there hypervisor tools, traffic tools, NIC statistics, or other specialized monitoring tools deployed to help with analysis.

4. Contact support.

In other words, if you are unsure, alert Mission Control. After determining the type, assessing the immediacy, and consulting additional sources, it is a good idea to contact your vendor for support. A warning about a threshold for API calls may seem innocent. But if the API calls will fail once such a limit is reached, this could be cause for immediate action. Getting the authority of the specialist can be helpful in keeping peace of mind and avoiding disaster.

An experienced vendor like SIOS can help you quickly identify the causes of problems and recommend the best solution.

Repeatedly ignoring problems in your availability environment can lead to unexpected, but no less devastating results. Addressing the problems indicated by alerts, log messages, warning indicators, or other installed and configured indicators gives your customers, your business, your teams, and yourself the “opportunity to solve the problems,” before it becomes a disaster. And at the same time, strengthens your availability strategy and infrastructure. Which will you choose?

– Cassius Rhue, VP, Customer Experience

Reproduced from SIOS

Six Reasons Not to Buy SIOS High Availability Software . . . If You Dare

October 25, 2020 by Jason Aw Leave a Comment

Six Reasons Not to Buy SIOS High Availability Software . . . If You Dare

You need SIOS Protection Suite (for Linux or Windows) or SIOS DataKeeper Cluster Edition for high availability protection for business critical applications.

UNLESS

1. You prefer free solutions only.

I get it. There are definitely times when I do the same thing when I need to learn a new skill, get a quick tip, drop a few pounds, or set up a quick demo. Rather than signing up for a subscription, purchasing a license, or investing in a combination of the two, I have gone the free route.

However, the saying often holds true, you get what you pay for. Free trials are fine. Permanently free high availability is like gas station sushi – is the risk really worth it? Be sure that free doesn’t prevent you from utilizing the fullness available for optimizing uptime and increasing availability. Make sure you aren’t passing over a reasonably priced high availability solution that is proven to protect your mission-critical applications.

2. Being a single solution shop solution is more important than meeting your HA needs.

We were a “Ford tough” family for decades. Seriously. I understand what it is like to be a one solution shop. My dad owned a Ford truck for work, a Ford Mustang for leisure, a Ford 3600 tractor for the farm, and a Ford minivan for family travel. There was even a season where we received model toy cars with the brandished blue oval as well.

But, when my wife and I were branching out on our own family needs, we broke away from the single solution to address needs that fell out of the Ford wheelhouse (at the time). You may be a single shop buyer, but if your needs have changed and the HA provider or solution hasn’t kept up, consider whether expanding the solution set will eliminate risks, improve success, or be worth the investment in a complementary solution for those new needs. When we needed a reliable, gas efficient, sleek, family friendly, and economical solution for our family, we supplemented Ford tough with a Honda Odyssey. If you are a single stop shop, and you are not worried about vendor lock-in best of luck.

3. You are more of a do it yourself-er coder.

You like coding. You like to write a lot of scripts, and don’t mind pulling out your bash, ksh, perl, python, powershell, batch or command tool kit and wiring things up yourself. You value the joy in flexibility and adding your own tweaks.

I love writing code as well, but there are times when the last thing I want to do is spend time writing a lot of code and scripts for a problem that is solved, proven, and off the shelf ready. For the do-it-yourself admin, off-the-shelf may not be your preference, but consider whether 20 years of expertise and experience should be rehashed and re-architected for your enterprise. But, if you have to get the code writing fix in, High Availability Software SIOS provides the Generic Application Recovery Kit for you to get in a coding fix.

4. You need Ubuntu support (or Solaris).

Your environment is unique. You have customers who’ve cut their teeth on Solaris and are hanging on to it for dear life. Or you’ve got those who have fully embraced the Linux realm and have moved to Ubuntu. In either case, you look at the SIOS products matrix and Ubuntu isn’t currently a match for your SIOS version. Bummer!

While this is true, consider the rich and vast features and flavors of support that are still available. While there are parts of your enterprise that have dug in on Solaris and others that have raced to embrace Ubuntu and newer variants of Linux, it is more likely that you need a solution capable of supporting RHEL, OEL, SuSE, CentOS and possibly Windows as well. Be sure not to single out a high availability solution by what it doesn’t provide and consider the depth of what it does.

5. You don’t run a hybrid of anything in your environment.

I heard it in the middle of a movie last week. The lead character commented on the idea of moving forward with some new idea of an overly excited owner. The classic line: “Sometimes the juice isn’t worth the squeeze.” In your mind you feel that you aren’t running a hybrid environment. Your applications are critical, but not complex. The moving parts are simple- a database, front end and a supporting application. It makes sense that you might not want to “complicate” things with additional processes, products, solutions or services, and you may feel like the juice isn’t worth the squeeze.

Before you make that final decision for a High Availability Software, assess whether a non-hybrid environment is the same as a simple environment. Consider whether or not the moving parts are as simple as you imagine or whether a solution with failover orchestration would be beneficial to reducing your overall RTO and increasing your RPO.

6. Endorsements from HA experts and experience don’t matter.

I bought a set of headphones online in mid-April. As I suspected, I discovered that anyone can do bluetooth headphones. But, not everyone can do them well. Ergonomically, the “new to market” headphones are a nightmare. Pairing was a breeze, but accidental unpairing is a constant battle. The sound quality is amazing, but that amplifies my annoyance when the headphones randomly chirp – loudly and clearly – for system sounds or at the end of a song.

You may believe that high availability and application monitoring can be done by anyone and that experience doesn’t matter. However, consider your own experiences and mine and ask if you’d really want to trust your enterprise environment to a group that just started thinking about the complexities of hybrid environments, or the dependencies and application-centric knowledge needed for the applications you use most frequently.

When deciding the right High Availability Software for your environment, consider carefully whether you want to go without the many best in class features, hardened and tested solutions, knowledgeable experts, broad swath of supported applications and environments, and industry leading experience and decades of insight. Then after careful consideration, choose wisely.

-Cassius Rhue, Vice President, Customer Experience, SIOS

Reproduced with permission from SIOS

Test/QA Systems are a Critical Part of Enterprise Availability

July 8, 2020 by Jason Aw Leave a Comment

Test/QA Systems are a Critical Part of Enterprise Availability

“I could kiss you,” that’s what a friend blurted out to me nearly three decades ago as she ran towards me. She had dropped her reeds for her saxophone on the way to one of the biggest band competitions in our region. I didn’t know whose they were, but when I saw the pack of reeds on the seat on the bus I picked them up and took them with me to the warm-up area. Three minutes into her warm-up, her 1st reed cracked and she panicked as she reached into empty pockets for replacements. When I piped up that I had found them, she blurted out, “I could kiss you right now.”

As the VP of Customer Experience at SIOS Technology Corp. I have the unique and distinct pleasure of working with a number of enterprise customers and partners at different phases of the availability spectrum. Sometimes I have the opportunity of working with end customers for issue resolution, mitigation, and improvements. At other times our teams are actively working with partners and customers to architect and implement enterprise availability to protect their systems from downtime. A recent customer experience reminded me of something that happened nearly 30 years ago when my friend blurted out, “I could kiss you.”

My team and I were on a customer call. The call began with the usual pleasantries, introductions, and an overview of the customer’s enterprise environment. Thirty minutes into the call, things were going so well. Their architecture was solid, thoughtful, and well documented. Their team was knowledgeable, technically sound, and experienced. But then, the customer intimated that due to cost savings they would not be planning to maintain a dedicated test/quality system. I took a deep breath. Actually it was more of an exhale like the rush of air from a gut punch. I prepared to respond, but before I could a voice broke through. “The number one cause of downtime is lack of process,” exclaimed the Partner Rep Architect on the call with us. After a brief banter, the customer agreed to maintain a test/QA system and I nearly blurted out, “I could kiss you!”

On the front lines of many Enterprise deployments (new systems, data center migrations, and system updates) my teams in Support and Services have seen dozens of issues that could have been mediated by utilizing a test system/cluster.

A test/quality system is an invaluable part of an HA strategy to avoid downtime. Common tasks associated with maintaining an enterprise deployment such as patches, updates, and configuration changes come with risk. Enormous risk.

Commonly identified risks of testing in production include several serious and potentially catastrophic issues:

Corrupted or invalid data
Leaked protected data
Incorrect revenue recognition (canceled orders, etc.)
Overloaded systems
Unintended side effects or impacts on other production systems
High error rates that set off alerts and page people on-call
Skewed analytics (traffic funnels, A/B test results, etc.)
Inaccurate traffic logs full of script and bot activity (a)

If a customer attempts to apply risky changes in production, the result can be quite damaging. On top of those listed above, there is an increased risk of downtime, corruption of application installations, and in some cases irreversible damage. Take the case of Customer X (a high profile SAP Enterprise shop in the manufacturing industry).

After reading a critical notice from a reputable site, the OS Administrator quickly updated his production nodes to the latest kernel update available. Within hours the Production nodes began a series of uninitiated crashes and kernel panics. In his haste, he had installed a kernel that was incompatible with his configuration; the combination of existing application packages, devices, file systems, and related packages. This caused a production outage and several high priority escalations to multiple vendors.

When patches are applied to a test/QA or sandbox system, patches and critical fixes can be managed and verified to reduce loss of productivity and unplanned downtime. Testing applications in a production-like environment allows you to identify unforeseen problems and correct the issues before they adversely impact your operations. Pre-production design and testing eliminate costly business disruption, improve your customer experience and protect your brand.

Using a test QA System to Improve Production Availability and Processes

Here are the basics that using a test/QA system, can provide for improving your production availability and processes. A controlled environment, that is similar (it must resemble production as close as possible) to the production environment, provides the ability to:

Test kernel updates and security updates
Validate settings and configuration tuning
Reproduce production issues and test software updates and patches
Verify application version compatibility and reduce the risk of downtime due to incompatible changes
Provide a safe space to practice and revise go-live, maintenance, outage, and other enterprise procedural activities
Train new hires and team members without impacting enterprise clients

If you have a Test/QA environment for deploying your critical enterprise availability software, I could kiss you right now. Having this environment gives your team the ability “to test, validate and verify(2)” architecture, business requirements, user scenarios, and general integration with a system or set of systems that most closely resembles the production environment- you know the one that makes the money. Of course, you will still have to schedule windows to maintain your production systems and perform testing on them as well, but after a safe buffer step has been completed in between.

— Cassius Rhue, VP, Customer Experience

————-

References: