DRP, i.e. Disaster Recovery Plan and Environmental Recovery – Case study

Currently, servers are a fundamental part of the IT infrastructure in most enterprises and organizations. The reliability and availability of data, applications, and the machines they host play a key role in everyday business operations. Unpredictable failures of this infrastructure can cause disruptions in the smooth operation of the environment and other undesirable consequences, including financial and reputational damage. The most common...

The most popular solution to disaster recovery is the use of regular backups, which are discussed in detail in our blog article. These can be stored:

• Locally – on an external USB drive, magnetic tapes, or a NAS (Network Attached Storage) device, often including additional data loss protection in the form of a RAID array, which prevents data loss in the event of a disk failure.
• Online – data is stored on an external server – using the IaaS (Infrastructure as a Service) model, ensuring protection against local failures, and backups can be accessed independently of the company's internal network.

What is a Disaster Recovery Plan? When is it needed?

A Disaster Recovery Plan is a document containing procedures and guidelines for the efficient restoration of proper operation, including IT infrastructure, in the event of an unexpected failure. A Disaster Recovery Plan is a component of Business Continuity Planning, a strategy for ensuring the continuity of an organization's key processes.
A Disaster Recovery Plan is defined by, among other things, the following metrics:

• RPO (Recovery Point Objective) – specifies the acceptable amount of data loss. This is the time between the occurrence of a disaster and the last data backup. Assuming a company accepts an RPO of 3 hours, data backup or replication should occur at least every 3 hours.
• RTO (Recovery Time Objective) – specifies the maximum time to restore a backup, service, or environment after a disaster.

A DRP is important when the risk of data loss or loss of access to the IT infrastructure is assessed as high. Threats posing this risk include:

• Natural disasters – floods, fires, earthquakes.
• Technological threats – utility disruptions, hazardous material leaks, IT infrastructure failures.
• Human errors – data leaks or human-induced threats, such as cyberattacks.

Disaster Recovery – Case Study

In today's technological world, where companies and organizations increasingly rely on IT infrastructure, the ability to efficiently recover an environment after a disaster is essential for maintaining productivity and data security.

One of Support Online's clients is a company that has implemented a disaster recovery service. This service involves replicating key client machines located in a local server room to a geographically remote data center.

The case study described in this article describes the test process for recovering an environment after a controlled failure of the local IT infrastructure.

Environment Configuration – Key Elements for Recovery

A Support Online client's office is configured with two key virtual local area networks (VLANs): One is the DMZ (demilitarized zone) – a network area separated from the internal network, containing servers hosting websites, among other things, which are more susceptible to cyberattacks. This prevents attackers from accessing the local network. The second virtual network is the internal network, containing the remaining IT infrastructure. Both networks must be configured identically in the replicated environment to mirror the network configuration at the client's office.
Servers from the client's office are replicated to the Support Online servers at one-hour intervals, allowing the environment to be restored in a state consistent with the servers' state during replication in the event of a failure in the local server room.

Configuring a VPN connection is also crucial, allowing connection to the running backup environment. In this case, company employees connect remotely to the office via VPN using the FortiClient application. For testing purposes, we automatically distributed a second connection profile to all employees. This allows remote employees to easily connect to the backup environment and work on the servers running there.

The environment mentioned above, consisting of ten servers and the network configuration, will be crucial for us to recover from an unexpected failure.

The test described in this article simulates an event in which all servers in the office are damaged, for example, in the event of a fire in the server room.

Replicated servers view

Environment Overview

Disaster Recovery Process

Receiving Information and Diagnosing the Failure

The first step before recovery is receiving information about an unexpected IT infrastructure failure. Online Support monitoring is the first to notify about any interruption in server operation or access, which is described in detail in a separate article on our blog. Our 24/7 engineers confirm the failure with the person responsible for authorizing such reports on the client's side. Once all persons involved in the procedures have been informed of the initiation of DRP and/or BCP procedures, we can begin the DR environment recovery procedure.

A view during a failure from the dashboard of the application monitoring client servers

Stopping Server Replication

Disabling or stopping replication in the backup software is the next step, which is important for a smooth, test recovery of the environment. This allows us to revert to the original state after the test.

Creating an Additional Machine Checkpoint

Creating an additional restore point is an additional option for securing the environment. If key data is deleted or lost in the recovered environment, it can be recovered by restoring the checkpoint.

A new server restore point has been created in "Checkpoints."

Changing Virtual Network Adapters

Network reconfiguration occurs after replication is disabled. All replicated servers within the Hyper-V virtualization software are assigned to the appropriate virtual local area network, and the machines' network adapters are changed to reflect the infrastructure environment at the company before the failure. This allows the machines to run on the appropriate and assigned VLANs, and services exposed to the world remain exposed as they were in the production environment (any changes must be made on the DNS server side, taking into account the new/different WAN addressing).

Above: Network adapter configured for replication.
Below: Changed network adapter and configured VLAN

Power-on Servers

Power-on machines and verification of proper operation are the next steps in environment recovery. This process verifies installed software, application communication functionality between servers, databases, and websites hosted on the servers.
When starting servers, the order in which machines are started is important. If one machine is acting as a DHCP server, it should be started first; otherwise, servers that should be assigned IP addresses may change their network configuration, requiring manual configuration.

Each employee can connect to the backup environment via the backup VPN profile on their computer and work on the shared resources there.

Informing about the Recovery of the Environment

The final step is to inform those participating in the DRP procedures about the recovery and verify the proper functioning of the environment. The DRP procedure test described above allowed the company to continue operations for 30 minutes after the failure in the local server room.

Summary:

DRP is a key element to consider when creating a Business Continuity Planning plan to maintain business continuity and minimize downtime following an unexpected disaster. To ensure the effectiveness of your disaster recovery plan, regular audits, tests, and procedure updates are essential.

Support Online has many years of experience implementing disaster recovery (DR) solutions. During this time, we have conducted dozens of tests in various environments and developed the best methods. Today, they allow us to implement similar projects for companies where security and business continuity are a priority.