SIOS SANless clusters

SIOS SANless clusters High-availability Machine Learning monitoring

Beginning Well is Great, But Maintaining Uptime Takes Vigilance

by Leave a Comment

 

 

Beginning Well is Great, But Maintaining Uptime Takes Vigilance

Author Isabella Poretsis states, “Starting something can be easy, it is finishing it that is the highest hurdle.” It is great to have a kickoff meeting.  It is invigorating, and exciting. Managers and leaders look out at the greenfield with excitement and optimism is high.  But, this moment of kickoff, and even the Champagne popping moment of a successful deployment are but just the beginning. Maintaining uptime requires ongoing vigilance.

High availability and the elusive four nines of uptime for your critical applications and databases aren’t momentary occurrences, but rather, a constant endeavor to end the little foxes that destroy the vineyard.  Staying abreast of threats, up-to-date on the updates, and properly trained and prepared is the work from which your team “is never entitled to take a vacation.”

For those who want to stay vigilant in maintaining uptime, here are five tips:

1. Monitor the Environment 

Very little in enterprise software still follows the “set it and forget it” mindset.  Everything, since the day you uncorked the grand opening champagne to now, has been moving toward a state of decline.  If you aren’t monitoring the servers, workloads, network traffic, and hardware (virtual or physical), you may lose uptime and stability.

2. Perform Maintenance

One thing that I have always noticed in over twenty plus years of software development and services is that all software comes with updates.  Apply them.  Remember to execute sound maintenance policies, including taking and verifying backups. One tech writer suggested the only update you regret is the one you failed to make.

3. Learn Continuously

My first introduction to high availability came when I unplugged one end of the Token Ring for a server in our lab as an intern, fresh from the CE-211 lab.  The administrator was in my face in minutes.  After an earful, he gave me an education.  Ideally, you and your team want to learn without taking down your network, but you do absolutely want to keep learning.  Look into paid courses on existing technology, new releases, emerging infrastructure.  Check your vendors for courses and items related to your process, environment, software deployments and company enterprise.  Free courses for many things also exist if money is an issue.

4. Multiply the learning

In addition to continuous learning, make a plan to multiply the learning.  As the VP of Customer Experience at SIOS we have seen the tremendous difference between teams who share their learning and those who don’t.  Teams that share their learning avoid gaps in knowledge that compromise downtime.  The best way to know that you learned something is to teach it to somebody else. As you learn, share the learning with team members to reduce the risk of downtime due to error, and for that matter vacation.

5. End well . . .before the next beginning

All projects, servers, and software have an ending.  End well.  Decommission correctly.  Begin the next phase, deployment, software relationship, etc well by closing up loose ends, documenting what went well, what did not, and what to do next.  Treat your existing vendors well.  You just may need them again later.  Understand the existing systems and high availability solutions before proceeding with a new deployment.  This proper ending helps you begin again from a better starting place headed towards a stronger outcome.

Keeping the system highly available is a continuous process.  Set it and forget it is a nice catch phrase, but the reality is that uptime takes vigilance, continual monitoring, proper maintenance, and constant.

-Cassius Rhue, VP, Customer Experience

Reproduced with permission from SIOS

Fifty Ways to Improve Your High Availability

by Leave a Comment

I love the start of another year.  Well, most of it.  I love the optimism, the mystery, the potential, and the hope that seems to usher its way into life as the calendar flips to another year.  But, there are some downsides with the turn of the calendar.  Every year the start of the New Year brings ‘____ ways to do_____.  My inbox is always filled with, “Twenty ways to lose weight.”  “Ten ways to build your portfolio.”  “Three tips for managing stress.”  “Nineteen ways to use your new iPhone.”  The onslaught of lists for self improvement, culture change, stress management, and weight loss abound, for nearly every area of life and work, including “Thirteen ways to improve your home office.”  But, what about high availability?  You only have so much time every week. So how do you make your HA solution more efficient and robust than ever.  Where is your list?  Here it is, fifty ways to make your high availability architecture and solution better:

  1. Get more information from the cluster faster
  2. Set up alerts for key monitoring metrics
  3. Add analytics.  Multiply your knowledge
  4. Establish a succinct architecture from an authoritative perspective
  5. Connect more resources. Link up with similar partners and other HA professionals
  6. Hire a consultant who specializes in high availability
  7. 100x existing coverage. Expand what you protect
  8. Centralize your log and management platforms
  9. Remove busywork
  10. Remove hacks and workarounds
  11. Create solid repeatable solution architectures
  12. Utilize your platforms: Public, private, hybrid or multi-cloud
  13. Discover your gaps
  14. Search for Single Points of Failure (SPOFs)
  15. Refuse to implement incomplete solutions
  16. Crowdsource ideas and enhancements
  17. Go commercial and purpose built
  18. Establish a clear strategy for each life cycle phase
  19. Clarify decision making process
  20. Document your processes
  21. Document your operational playbook
  22. Document your architecture
  23. Plan staffing rotation
  24. Plan maintenance
  25. Perform regular maintenance (patches, updates, security fixes)
  26. Define and refine on-boarding strategies
  27. Clarify responsibility
  28. Improve your lines of communication
  29. Over communicate with stakeholders
  30. Implement crisis resolution before a crisis
  31. Upgrade your infrastructure
  32. Upsize your VM; CPU, memory, and IOPs
  33. Add redundancy at the zone or region level
  34. Add data replication and disaster recovery
  35. Go OS and Cloud agnostic
  36. Get training for the team (cloud, OS, HA solution, etc)
  37. Keep training the team
  38. Explore chaos testing
  39. Imitate the best in class architectures
  40. Be creative.  Innovation expands what you can protect and automate.
  41. Increase your automation
  42. Tune your systems
  43. Listen more
  44. Implement strict change management
  45. Deploy QA clusters.  Test everything before updating/upgrading production
  46. Conduct root cause analysis exercises on any failures
  47. Address RCA and Closed Loop Corrective Action reports
  48. Learn your lesson the first time.  Reuse key learnings.
  49. Declutter.  Don’t run unnecessary services or applications on production clusters
  50. Be persistent.  Keep working at it.

So, what are the ideas and ways that you have learned to increase and improve your enterprise availability? Let us know!

-Cassius Rhue, VP, Customer Experience

Reproduced from SIOS

The New Normal Will Still Include High Availability

by Leave a Comment

The New Normal Will Still Include High Availability

The Importance Of Uptime In A Post Pandemic World

As vaccines roll into production and roll out to facilities and communities, and companies are beginning to prepare for reentry to normal.  Many articles and writers, both in technical and non-technical spheres, are predicting that ‘normal’ in the post-pandemic era will look a lot different from the ‘normal’ we were used to  in 2020.  Experts vary. But genuinely agree that every business and type of industry will see a change in what was ‘normal’.  This change will affect everything from academics to manufacturing plants to financial institutions to houses of worship. While the new normal will potentially look different than it did when we abruptly left these places in 2020, some things will still be a part of the new normal.

Four Reasons High Availability Will Still Be Included In The New Normal of 2021. 

  • New database and application systems

Predictions abound that home delivery, home schooling, home entertainment, and even home gyms will be a booming part of the future.  This boom will lead to new businesses. It means that these new businesses will deploy cloud services and applications which need to be highly available to handle the additional online shopping. Not to mention the growth in shipping, and related services from manufacturing to accounting.  For these new businesses and services, downtime is not an option that they will be willing to accept. Cloud availability SLAs – which only address infrastructure availability – will need to be subsidized with application-level HA.  New databases and application systems will require 99.99% availability as an essential requirement.

  • Existing database and application systems

The predicted boom of everything “At Home” will definitely lead to a spike in new businesses with their new databases, applications and services.  But, the rise in these new businesses and upstarts will not mean that existing companies will fold their tents and vacate the space.  Instead, the boom of new competitors to the various “At Home” spaces will drive an even greater urgency for existing businesses to fortify their databases and applications.  The businesses that exist in this space will need to expand to keep up with competition and growth.  As they expand, maintaining high availability will be a key focus for their existing database and application systems as they transition to cloud, hybrid cloud, or a multitude of hosting solutions.  These existing applications will neither abandon HA in the current locations, nor consider facing a disaster in any new permutation without high availability and disaster recovery.

  • IoT Management systems

New at home businesses will spark growth requiring higher availability. The shipping industry will continue to expand and generate new systems requiring availability.  In fact, whether the prediction of continued work-at-home boom pan out or not, the IoT boom will almost certainly come to fruition.  As more customers shop online, ship what was once hand delivered, they would require more assistance with their products. These eager and anxious customers will want more ways to track and get updates on the location of the package.  Additional checkpoints, check in hubs, and IoT devices would probably be added as mainstays of the new normal. At the same time, the plans will need to include making sure the additional systems are  reliable and available. It is even more important to generate the currency of trust for the customer and data for the enterprise.

  • New Causes for Downtime 

As the whole world looks forward to a return to “normal”, this return will generate new challenges and opportunities.  Alongside these new technologies, and expanded deployments of database and application 2021 will experience new causes for downtime, old nemesis causing disasters, and other unexpected outages. Applications – even those with more robust monitoring and capabilities – will still experience the old nemesis of coding bugs that lead to crashes or integration issues that lead to instability or hangs.  Systems, cloud or on-premise, will still be susceptible to hardware faults, human faults, the occasional simple maintenance that isn’t so simple, and forces like mother nature or the new guy with elevated privileges and reduced knowledge.  Ushered in with opportunity will be new disasters that require thoughtful approaches to highly available clusters, solutions and services.

Yes, a lot will have changed for companies since things shut down in early 2020. However, the new normal that any business returns to will definitely require high availability.

Writers Note: 

For many families, the post pandemic world will be drastically different with chairs that sit empty, beds that are no longer filled, and laughter and memories that ended in heart-rending pain. To all of these families, SIOS Technology Corp. extends our deepest and most heartfelt sympathies for your loss and our prayers for your comfort and grief.  

– Cassius Rhue, Vice President, Customer Experience

Reproduced from SIOS

How To Build A Highly Available Server Solution?

by Leave a Comment

How To Build A Highly Available Server Solution?

How To Build A Highly Available Server Solution?

A key component to any high availability solution is figuring out how to redirect the client traffic. Almost every user-based application needs to connect to the server. Redirecting the client traffic will allow users to connect without having to know where the application or the database actually resides.

Most solutions recommend network-based IP redirection or network based DNS redirection. This works. However, the best solution for a high availability server in our experience is the use of a virtual IP address that can be switched from one server to another. The server is listening to connections from the virtual IP address, where it’s hosted on one server today and switched to another on another day.

To take it one step further, you can automate the failover. This is where the system makes decisions and switches the application when there is a failure detected. Bear in mind this step is key to building a highly available solution.

Benefits of Buy vs. Build High Availability Solution

This can be implemented using scripts and logic to check the status of processes and virtual IP addresses from one server to another. But one of the challenges we face in a buy vs build high availability solution is how much time we really have to spend in build. This includes time for script coding, API development such as cloudwatch API or lambda functions. Let’s not forget testing, and maintenance.

When I was younger, I was eager to write that code. But after  working for large Fortune 100 companies, and getting yelled at by a high level manager, when one of my scripts didn’t work at 3 am in the morning, I feel differently. This issue was exacerbated when I discovered an issue for a code I wrote a year ago. My managers wanted the highly available solution to work 100%. If it didn’t work, time to call up someone and yell at them.

SIOS Automates High Availability

Isn’t it cheaper in the long run to buy the solution and spend a little time to tweak it to fit into our setting? This is where SIOS high availability (HA) solutions come in, whatever the application or database. SIOS has the code to switch the stack of the processes from one server to another. This gives users and managers the peace of mind that comes from automating the failover orchestration and high availability.

There are two things that I love about the SIOS HA umbrella are. One, the code for the virtual IP where the IP address is added to the server and the application is restarted to listen to the connections. The second is enabled through the use of the application agnostic API set that SIOS provides. This allows anyone to protect any application by the use of plugins. Contact SIOS today to learn more about high availability solutions specific to your environment.

– Edmond Melkomian, PMP, MCSD, consultant, SIOS technology, Inc.

Reproduced from SIOS

Stages of IT Disaster Recovery Grief

by Leave a Comment

Stages of IT Disaster Recovery Grief

Stages of IT Disaster Recovery Grief

Disaster recovery grief can hit you out of nowhere if you haven’t implemented the right enterprise availability architecture. Meet our friend Dave in IT to walk us through the 5 stages of disaster grief.

Stage 1: Denial

Dave in IT: “Uh oh.  What’s that alert?  It’s just a little application crash, right?  No big deal.  I’ll have things up and running in no time.”

In the land of enterprise availability, there is no such thing as a little application crash or no big deal.  Companies have SLA with real money on the line.  Your selective reality is probably not the same perspective of your customers and stakeholders.

Stage 2: Anger

Dave in IT: “Are you kidding me.  Of all the… [censored]...times, today the application won’t start.  Ughh.  I hate this[censored]...[censored]... application.  Wait, what’s this new alert.  Seriously, now, the datacenter is down!”

It gets messy really, really fast in the fast pace, and high stakes environments. When unchecked alerts and failures happen, problems can mount quickly along with pressure, frustration and anger.

State 3: Bargaining

Dave in IT: “Hey Ard in Applications, this is Dave in IT.  Do you guys have any backups for the App1 environment? . . .Ard are you sure?  Could you just check again?  I know you’ve checked twice, but can you check one more time.  I’ll buy drinks on Taco Tuesday!”

Dave in IT: “Hey Donna DBA, this is Dave in IT. Art in Applications said you might help me out.  Did you by chance setup any database replication for that finance database or the inventory management system? . . . Are you sure?  Umh, do you remember if we have any way to recover from a umh . . . datacenter crash?”

When my daughter gets in trouble, bargaining is her first go to.  Okay, second.  The first is to disappear, but you’re too smart to just walk away from the flames.  But, Dave in IT isn’t the only one to realize that bargaining and begging is a poor substitute for a well defined strategy for high availability and disaster recovery.  Skip the bargaining and begging about your disaster because “80% of the people don’t care, and 20% are glad it’s you (paraphrased from Les Brown).”

Stage 4: Sadness

Dave in IT: “This is just great.  The application server crashed, the datacenter is down, and backups, if I can find them and if I can load them, will take hours to get restored.  There is no way I’m getting out of this… where did I put that updated resume.”

Of course you have backups, and you’ve validated them.  But there is an RTO and RPO impact of going back to those backups.  Are you able to absorb this time?  That is of course, after your data center recovers.

Step 5: Acceptance

Dave in IT: “It’s been two hours.  I never knew we had this many Executive stakeholders before.  No way I’m making it to my 2nd year anniversary after this.  Well, I guess I’ll clean out my office tomorrow.  No way I’m making it through this!”

Failures happen.  Datacenters go down.  Applications fail.  There is no denying the possibility of losing a data center, having a server fail, or an application crash.  This type of acceptance is normal, a part of improving your availability.  Accepting that you may lose your job or worse because you failed to implement an availability strategy is something the experts at SIOS Technology Corp. want to make sure you avoid.

Don’t be like Dave in IT.  Avoid the stages of disaster grief, and the hours of disaster recovery and downtime by architecting and implementing an enterprise availability architecture that includes the best of hybrid, on-premise, or cloud coupled with the best solution for monitoring, recovery, and system failover automation.

– Cassius Rhue, VP Customer Experience

Reproduced from SIOS