drj logo
drj logo

Welcome to DRJ

Already registered user? Please login here

Create new account
(it's completely free). Subscribe

x

Closing the Delta Between RTO and RTC

Business Continuity ManagementCyber Resilience & IT Disaster RecoveryData Protection: Backup & Recovery

The ability to recover critical systems in a timely manner during an event is the cornerstone of any effective disaster recovery program. All too often large and small organizations fail to recover their critical infrastructure and applications because of the inability to recognize the variance between the time they would like to recover, known as the recovery time objective (RTO), and the time they can actually recover, or the recovery time capability (RTC). This recovery variance, or “delta,” may be attributed to several factors including policies and standards, outdated technology, unrealistic business needs, operational issues, and management issues. The presence of any or all of these factors will prevent your disaster recovery program from achieving the desired recovery outcomes.
 
Using your BIA to align RTO values between the Business and IT
A business impact analysis (BIA) should be the logical step when seeking to identify business processes and the effect a possible business disruption might have on them. It helps distinguishing the mission-critical versus the not-so-critical. Business management should also be able to understand interdependencies in terms of other processes and applications and, with the help of their information technology counterparts, link those to the infrastructure that support them.

It’s no secret that everybody wants job security, so what better way to provide the importance of what we do than with a BIA. One of the main results of the BIA is the recovery time objective (RTO). As defined by the Disaster Recovery Journal Business Continuity Glossary, RTO is “the period of time within which systems, applications, or functions must be recovered after an outage.” It is a key parameter when defining recovery strategies, since it allows business and IT to agree, design, document, and validate how the business processes will recover and meet the business’s continuity goals.

Unrealistic business needs get put to the test when designing recovery strategies. The more time-sensitive a business process is the more expensive and complicated usually it is to design a strategy. Business management should work along with their IT counterparts in order to design a recovery strategy that fits business needs with IT recovery capability. Business management then needs to meet in the middle with IT, considering the organization’s risk appetite or “the amount and type of risk that an organization is willing to retain,” as defined by ISO 22301.

DR Testing to Uncover the Delta
Disaster recovery (DR) testing is a quick way to determine how large the delta is between your RTO and the RTC, especially for larger enterprises with several hundreds to thousands of business processes in place. DR testing provides the true recovery capabilities for both infrastructure and applications and should be process-driven versus component-driven. This allows the applications to be truly tested from end-to-end and prevents cherry picking from the easiest to recover component parts. 

Bryant1

Recovery Time Capability is defined in the Disaster Recovery Journal’s Business Continuity Glossary as the “demonstrated amount of time in which systems, applications, and/or functions have been recovered during an exercise or actual event.” It can be argued that a more accurate definition of this term is the demonstrated amount of time in which business-critical systems, applications, and functions are recovered to a state-of-business readiness and usage, including validation, during an exercise or actual event. Below is an example of this variance concept and the impact on business process recovery.
Table 1 and Table 2 break out the variance between RTO and RTC by the components mentioned in the DRJ Continuity Glossary definition of RTC. Application validation is included as a part of the Application RTC for this example, and the function area would include any third-party connectivity or additional services that support the critical business processes. As illustrated, each process component contributes to the total variance between the business process RTO and the business process RTC, with the infrastructure variance providing the bulk of the overall disparity. A good approach to initially evaluating the process variance is to take a look at the largest variance in the component parts in your environment and begin there. Keep in mind the very nature of validating and confirming that all component parts of a critical business process are ready to be utilized may extend the business process RTC beyond previously anticipated or determined values. This task is sometimes glossed over in an effort to get the business running again as quickly as possible, so be certain you include validation efforts in your total variance equation.

Based on the example from Table 1 and Table 2, it is clear there is a problem recovering the business processes in the requested timeframe. The disaster recovery is going to be a minimum of eight hours overdue for each critical business process we are required to recover because of the culmination of variances we see. Infrastructure is the biggest offender with a variance of six hours across all critical processes, with applications and functions contributing an hour each to the overall total.

Factors Affecting the Recovery Variance
The recovery variance between RTO and RTC must be addressed quickly because of the most important business-critical function that exists in your enterprise: making money. The loss of the ability to make money for even a short amount of time can send your business into a death spiral from which it may never fully recover. So how do policies and standards, outdated technology, and operational and management issues really impact the recovery variance for an enterprise?  They all work to artificially inflate the RTC values of the recovery operation before it has a chance to succeed.

The first question to look at is how your enterprise defines DR. What does your policy or standard define as a true disaster, or is it the piece you’re missing? Is the scope so narrow that your organization will never face a true disaster, or is it so broad that you face a never-ending stream of disaster declarations and recoveries. It is essential to put some genuine thought into this question and make sure you define a disaster based on the impact to your people, facilities, technology, and your bottom line. Alignment with business continuity during this process will also ensure the definition of a disaster doesn’t run counter to any of their policies or standards.  This will need to be communicated up the management chain so there is no confusion about what a true disaster means and how your enterprise needs to recover from it.  The time to decide what a true disaster really looks like is before you’re in the middle of an actual event.

Bryant2

Technology will make or break your efforts to recover in a reasonable time and meet the business critical RTOs your company requires. Does your technical infrastructure support the RTOs of these business-critical processes? If the answer is no, there’s your problem, Houston! Your enterprise will never be able to close the variance gap between RTO and RTC unless the infrastructure is in place to support it. Staying current with technology has always been a slippery slope for companies with low margins, often resulting in a “pay now or pay later” model. The cost of local area storage and cloud-based recovery solutions continue to go down while the reliability of virtual tape libraries and disk-based backups continues to improve. Implementing improvements in technology may prove to be too daunting of a task to tackle during tough economic times, so an alternative is to address the RTO requirements with the business in an effort to get them changed. A change in RTO values should be considered a short-term solution because of the increased impact on profitability this change can create.

Operations and management issues can make a large recovery variance go from bad to worse.  A lack of disaster recovery training and expertise, poorly defined roles and responsibilities concerning disaster recovery activities, unrealistic business expectations, and poor operational or management support of disaster recovery objectives are just a few of the issues that can cause the recovery variance to grow. A core team of technical recovery experts who are trained to recover specific infrastructure components should be in place to manage recovery operations for the enterprise. A disaster recovery coordinator role can manage the recovery team’s activities and provide status updates to their progress at pre-defined intervals. Nothing kills a recovery team’s momentum quicker than having to stop all recovery efforts to answer 20 phone calls requesting the current status. Operations and management support of regularly scheduled and unannounced disaster recovery testing is also crucial in closing the variance between RTO and RTC. A recovery team’s performance typically increases when they consistently train to meet a variety of disaster recovery scenarios ranging from minor impacts to full datacenter failovers.

Closing the Recovery Delta
The best method to determine where your disaster recovery program could close the delta between RTO and RTC is to do a self-assessment of the current state-of-the program. A quick checklist to review your current state of recoverability can include current on-call schedule (you do have one of these, right?), current technical employee training records, updated contact list with roles and responsibilities, updated disaster recovery plans by application and infrastructure with requested RTO, current application to infrastructure pairing report, report of last disaster recovery test with issues log and lessons learned, and current schedule of disaster recovery tests.  If you find any of these checklist items missing or outdated, start the process of updating or creating them immediately, as all of the listed items contain information that would be critical to have during an actual recovery.

Ultimately the process to close the recovery variance between RTO and RTC boils down to resources. Does your enterprise have the correct technical and human resources in place? Is there management agreement and support on how to utilize these resources during a disaster recovery event? If you can answer “yes” to these questions then you are well on your way to closing the disaster recovery delta.

Robby J. Bryant, MSEM, CHPP, ABCP, ITILv3 is a senior disaster recovery consultant who has gained more than 20 years of experience in business continuity, disaster recovery, and operational security while serving in various technical and leadership roles in the manufacturing, healthcare, emergency services, and consulting industries. He has a master’s of science degree in emergency management with a focus on preparedness and resiliency. He can be reached at rbryant3@triad.rr.com.

Bryant-Robby

Hiram Barbosa, MBA, MBCI, CBCP, is a business continuity professional who provides leadership and expertise in the consulting, technology, insurance, and financial services industries. He has been performing BC/DR functions since 2007, mainly focusing on the development, oversight, and compliance of the business continuity program, lifecycle, and methodologies. He also completed the Crisis Management & Business Continuity Professional Education Short Program from MIT and obtained an MBA in operational management. Barbosa-Hiram

 

Latest News
DRJ HOT ITEMS
Webinar Spotlight
Fetching Upcoming Webinars...
Journal Categories

AI: Automation & Innovation

Business Continuity Management

Crisis Management & Emergency Response

Cyber Resilience & IT Disaster Recovery

Leadership: Culture & Workforce Resilience

Operational Resilience

Risk Management & Quantification

Sector-Specific & Critical Infrastructure Resilience

Supply Chain & Third-Party Resilience

Governance: Compliance & Regulatory Readiness

Incident Management & Response Coordination

Resilience Strategy & Program Maturity

Data Protection: Backup & Recovery

Exercises: Testing & Scenario Planning

Emerging Threats: Geopolitical & Climate Risk

Contact Us

Newsletter

The Journal, right in your inbox.