A question that came to mind from the previous blog "A New Kind of Data Protection: Cyber-Recovery":
Where do all these pieces fit ??
How do the Data Protection (DP) based products fit into Disaster Recovery (DR) and Cyber Recovery (CR) architectures we'd drawn and discussed during the last blog ?
I began listing products we have at DELL within the DP portfolio. I trawled and scraped sites and documents and then began listing categories and models within Figure 1, ....and Boy there are alot of products !!!
Figure 1 : Data Protection Products listed
I will in future blogs write dedicated articles which cover particular products in full detail, but this is not the objective here. I want to try and achieve an aeriel view of it all before attempting to place these pieces into DR and CR solutions so lets first go through some very brief descriptions of the products from Figure 1.
Lets start with PowerProtect Data Manager (PPDM). This organises, manages and enables or disables various protection options within target devices. Because of PPDM's central function in control, it does require Target storage such as PowerProtect DD (formally Data Domain). PPDM can then organise and manage flows of data/backups to and from that target including adding any additional functions onto that data for how it's stored and protected. PPDM can either be deployed easily from an OVA onto a hypervisor, be installed and deployed from AWS, Azure and Google Cloud Market places or it also comes as applianec based. . PPDM is complimentary to the other DP products listed within this category, it not a replacement.
Then we have the more traditional Backup and Recovery products such as Avamar and Netwoker. Avamar also comes in a virtual addition and both products support backup and recovery operations on premise and within AWS, Microsoft Azure and Google Cloud.
RecoverPoint comes as appliance based and connects over Fibre Channel and TCPIP. It offers something called Continous Data Protection (CDP) locally , and Continous Remote Replication (CRR) which are more applicable to DR type scenarios so configuration over multiple sites. It supports various other methods of protection and snapshots, has the capability to roll back on recovery to a specific point in time. Recoverpoint also has integration points with a number of other products, including Site Recovery Manager (SRM) which I can use to create Automated Recovery Plans.
Data Protection Advisor is a reporting tool which lets one track backups, SLAs, check compliance requirements and performance of implmented DP solutions.
The appliance based models of PPDM include the new Data Manager Appliance, the DM5500 which has all the PPDM based capabilties including backup and recovery and orchestration capabilities with target devices, as well as the new transparant snapshot capability which radicaly reduces times associated with snapshot creation, recovery.
The Integrated Appliances include the DP4400. The DM5500 is positioned at the same target market as the DP4400, with its increased functionality. Next size up within the Integrated Appliance is the DP5900 which offers more capacity than the DP4400.
The Target Appliances or PowerProtect DD Series, previously known as Data Domain (DD) are purpose built appliances which prioritise data invulnerability in order to focus on data integrity for the puposes of protecting backup and archive data. These appliances are not just storage systems built on cheap disks. DD is the preferred target storage for PPDM. It's built on PowerEdge hardware and the appliance models range in functionality and capacity starting with the DD3300 which goes upto 96TB, right to the DD9900 with a maximum capacity of 1.5PB Usable capacity and upto 4.5PB useable capacity with Cloud Tier. As well as the appliances there is also PowerProtect DD Virtual Edition which can be deployed either on-premise or in the cloud. DD features include retention lock, HA, hardware based compression.
This is new release at time of writing. SmartScale simplfies capacity management across multiple PowerProtect DD systems. Smart Scale is enabled from PowerProtect DD Management Center (DDMC) and gives analytics on all DD systems. It in addition enables the pooling of DD capacity into pools of storage, rather than relying on seperate capacity with each DD system. The DD systems are grouped together by Smart Scale and then managed by DDMC
Later in Figure 2 you will notice this piece sits at the centre of it all, controls and manages the various flows and pieces. The Cyber Recovery operations include the Replication set up on the DD source and destination systems using MTree replication; Creation and maintenance of Point in Time Copies (PITs); The ability to Lock and therefore secure the PITs by using features based on DD Retention lock features; Analysis and checking the protected data for signatures of attack, malware via the use of CyberSense; Recovery operations which can be performed and scheduled on demand. We'll discuss this cyber recovery component further below as we discuss a CR based design.
Cyber Recovery Architecture
So now lets start with the cyber recovery solution. We'd previously looked at the below diagram and described 9 steps flowing from left to right, showing what is involved setting up a solution to protect and recover from cyber-attacks. I'm not going to go over the workflow as we covered that in detail during the last blog.
So how do the actual products fit into the big picture ? Figure 2 below shows the Cyber Protection & Recovery workflow now with products from Figure 1 added. Note, not all the products shown below would be used at the same time, a number offer similar functionalities. For example, one can either use software to backup such as Avamar, or an appliance such as the DM5500 which has the capability of backing up data in the form of assets. Never the less I've added all products below to show examples of where they would fit into a cyber related workflow. We'll discuss the scenario in relation to the products below.
Figure 2 : Cyber Recovery blueprint
The Cyber Recovery design is seperated into two distinct areas. The Production Environment and the Vault area. Products will sit in either one of those areas or in some cases, both.
Cyber Recovery Servers
We'll see the following Server types as part of a Cyber Recovery solution:
Cyber Recovery Management Server
Application Analytics Server
Backup aAplication Recovery Server
Application Recovery Server
Recovery Metrics
Another aspect to consider are the Recovery metrics, which contribute to the overall design and hence CR Solution will differ from standard DR design and concepts. For DR we design a solution based on a customer's Recovery Point Objective (RPO) and Recovery Time Objective (RTO). Within a CR solution we account for the following:
Destruction Detection Objective (DDO)
When the attack occured and when it was then detected, so the time between the two.. which from past cases could be weeks.
Destruction Assessment Objective (DAO)
The time allocated to the cyber security team to access the attack and then next steps.
Cyber Recovery Point (CRP)
CRP is simlar to RPO so the last point you can return to. For example if using Point in Time Snap Shots, which one to fall back to. This helps determine how much has been lost.
Cyber Recovery Time (CRT)
How long it will take to recover.
Cyber Recovery Synchronisation Interval (CRSI)
How often data is copied between the Production Network and Vault Area. In Figure 2 this would determine how often data is replicated between the source DD and Vault DD systems. This is similar to the syncronisation levels set within DR scenarios based on an RPO value.
Cyber Recovery Data Count Copy (CRDCC)
The number of copies to hold onto within the Cyber Recovery Vault. As an example one can have a CRSI of 24 hours, so a copy created once every 24hours. In addition have a CRDCC of 7 which means hold onto 7 copies / covering 7 days before permanantly removing a copy.
Now where do the products fit ? Reference Figure 2 from Left to right :
Production Environment
Backup
We need something that can backup and recover our data and there are a few ways to do this. We can use some form of backup software or we can even use the new PPDM Data Manager Appliance (DM5500) which is itself fully capapable of backing up objects using the form of 'assets'. The Cyber Recovery software will also need to support the se pieces as it will be responsible for overall management and configurations. The software does in addition support DD integrations with NetWorker, Avamar, and PowerProtect Data Manager applications, and 3rd party backup and recovery applications, which we will disuss below.
3rd party Backup & Replication
I have spoken with a number of customers who are using a third party application to backup and so they will ask whether they can still build a CR based solution using the Dell products ?
Customers using software such as Veeam backup & Replication, can use PowerProtect DD (DD) (previously known as Data Domain) as the target protected storage. Communication from Veeam to DD can be either over the TCP/IP or Fibre Channel protocols. Veeam uses Data Mover (DM) services to communicate with DD. The services are responsible for the data transfer and because these services are not hosted on DD, an additional Backup Proxy and Gateway Server are deployed to enable communication between Veeam and DD.
Figure 3 : Backup and Replication to DD uing Veeam
During a Veeam job the backup repository is addressed and then the data movers running on both the Proxy and the Gateway servers establish a connection. The DD appliance is presented to Veeam as DDBoost, CIFS, NFS or DD VTL storage target.
Production Source Storage
The production environment must have at least one PowerProtect DD system that is configured for replication to a DD system within the Cyber Recovery Vault area as seen within Figure 2.
A further Question that does come up:
Why do I need 2 PowerProtect DD appliances, one within production and a 2nd within the Vault Area rather than just a single DD system within the Vault ?
MTree Replication
PowerProtect DD uses a system of replication known as DD Series MTree Replication between at least one other PowerProtectDD appliance. MTree is a logical partition of a filesystem and within our scenario in Figure 2 replication is of the entire MTree including all subfolders. It allows granuar management of snapshots, quotas and DD Retention locks. The Data is sent from the Production DD system to the Air-Gapped Cyber Recovery Vault DD system securly between the appliances. Once the initial full replication from Production into Vault has completed, subsequent replications consist of only changes (deduplication technology). The cyber recovery software controls data synchronisation between Production and Vault. The data is writable within the Source DD system at the Production network-end until reaching the Vault Area where it is then protected and Read-only via immutabilitaty settings.
Similaly, Integrated Appliances can sit within Production and Vault areas so for example the DP4400. The replication target in the Cyber Recovery vault must now also be a DP4400 Integrated Data Protection Appliance so the geometry of the production and target servers must match. Other than DD OS and Avamar Virtual Edition (AVE), the Cyber Recovery software supports a limited number of functions on the Integrated Data Protection Appliances within the Cyber Recovery vault.
Another option is with the PPDM Data Manager Appliance DM5500, I mentioned earlier it is capbable of Protection and Recovery itself using assets, which are the basic units that PPDM Appliance protects and then stores as Backups. Assets which can be protected include virtual machines, Microsoft Exchange Server databases, Microsoft SQL Server databases, Oracle databases, SAP HANA databases, file systems, or Kubernetes namespaces.
Vault Area
The Vault can house a number of hosts including a Cyber Recovery Management host which runs Cyber Recovery software. The DD target storage also sits within the vault. PPDM can in addition sit inside the Vault as well as outside.
Cyber Recovery Software & Host
We can add software which can manage the flow and operations of the data through the workflows shown in Figure 2 and Figure 4. Dell PowerProtect Cyber Recovery software which is installed on the management host, can oversee and organise backup operations as well as the flows of data to the source and target DD storage. The Cyber Recovery Management Host can either be a physical host or virtual. The virtual appliance is a preconfigured virtual machine (VM) running SUSE Linux Enterprise Server that can be deployed onto a VMware hypervisor.
This sets up the replication between the Production DD system and target DD system within the Vault area. The host can configure additional functions if supported, so as an example during replication from Source DD to Target DD within the Vault, a DD based Point in Time Copy (PIT) of the last replication is created. This serves as a restore point. Multiple PITs can be created and these can be used to recover into a sandbox area within the Vault. The PITs can in addition be retention locked at various points and then scanned by some form of analytics, running via the cyber recovery software (see CyberSense below). This means the data cannot then be modified by an attack. The DD retention lock functionality is additionally set up on a perMTree basis.
Running these components inside the Vault enables recovery and restore functions within the Vault/Sandbox area, which if successful can then be sent to the Production area once tested successfully within the Vault. You would not want to recover using a backup or PIT which had been attacked /contained malware. A vCenter will need to be added into the Vault Area as a Server Asset else any recovery will fail. A Timing server is also recommended such as an atomic clock, Assets within the Cyber Recovery vault are represented as storage, Application, and vCenter server objects.
Air-Gap
The Cyber Recovery software has the responsibility for opening and closing the connection (AirGap) between the Production and Vault Areas, else the replication between the Production DD storage and Vault based Target DD storage will not be successful. The Connection between the two areas is a dedicated connection, and is only enabled for short periods of time to limit access and possibility of exposing the Vault to attacks. A Firewall can be installed on the replication path between the Production Network and Vault area. Replication between the Source DD system and Target DD system withim the Vault area use MTree Replication, as discussed previously.
CyberSense
By the time data is copied ino the Vault, it is possible the data may already already have been attacked. The Cyber Recovery software initiates running analytics on data stored within the Target DD system within the Vault using the optional component CyberSense. This is not mandatory but if used, must be installed within the Vault. It is not supported if installed within the Production area. CyberSense gives the ability to analyze saved copies, for example Point in Times (PITs) incase there has been some form of compromise. It behaves similar to a virus checker but instead looks to identify attack type signatures, so copy and deletes, encryptions etc . Once malware has been identified, The PIT in question would then be marked "not for recovery". The PITs can be used as sources for recovery hence the importance to scan and confirm whether clean and usable.
Disaster Recovery Architecture
So how do the DP based products fit into a DR type architecture ?
There are more options available when considering DP architectures to protect from traditional types of disasters. Unlike a CR based solution, we do have more Replication based options available, minus a Vault, although a Vault can be used. Traditionally within a DR based solution there doesn't tend to be a Vault Area although this is beginning to change now due to the high visibility of Cyber Crime. What we are srarting to see are hybrid type designs which combine elements of both DR and CR into a solution.
Earlier we'd looked at key metrics and measurements which contribute to the design of the CR based solution. The metrics that contribute to how DP based product(s) are used within a DR solution as well as the overall DR design, are a customer's:
Recovery Point Objective (RPO)
Basically How much data can you afford to loose so therefore how old the recovery can be for a restore or failover.
Recovery Time Objective (RTO)
How fast recovery has to be or how long systems can remain unavailable.
Many replication based operations are built within the operating systems being used so for example if using VMware and ESXi servers, then we have vsphere based replication (asyncronous based replication) between sites so from a Site A to Site B. Another option would be to configure a stretch cluster (synchronous replication) between Site A and Site B which again is a function of vSphere, as long as the sites are within the maximum latency ranges specified. One can use appliance based replication and protection, fo example using products such as Recoverpoint. Storage Array based replication over a Storage Area Network (SAN) is a well used and proven method of replication for a DR based setup.
Regarding the DP products, Power Protect Data Manager System Protection enables you to protect the data of PPDM from loss so protect the actual PPDM product as seen within Figure 4.
Figure 4: Disaster Recovery blueprint
This functionality for protection and recovery also exists within the Appliance based models so the Data Manager Appliance DM5500, and the Integrated Appliances DP4400 and DP 5900. The PowerProtect Data Manager Appliance system protection service protects the persistent data of a an Appliance system from catastrophic loss by the use of a series of server disaster recovery (DR) backups.
DR for PPDM including aplliance based requires the consideration of two scenarios:
The loss of the PowerProtect Data Manager Appliance Server itself
Secondly the loss of the entire site
With PPDM and the Appliances, three methods of DR are available which protect for the above use cases:
"System Recovery" covers loss of the PPDM server. The PPDM Databases are backed up by automated creations of Point in Time Snapshots (PITs). These can be used to recover a lost server. This is also known as Appliance Recovery .
Known as "Quick Recovery". The loss of the entire site is accounted for here. Under the Disaster Recovery Option within PPDM, one can create a Server DR backup Target and add another as a Replication target within the UI, this could be a DD system. One can either specify DD Boost or an NFS path. The replication Target can not be the same as the Backup Destination. For the Appliances such as the DM5500, this option makes a remote PPDM destination Appliance aware and creates a Recovery View so enables the restore of assets from the replicated backups.
Cloud Disaster Recovery: During a DR activity, One can restore services to a Cloud DR service and recover workloads in the cloud. We will look at this in more detail within the next blog.
With the above in mind, PPDM supports recovery using a number of connectivity toplogies, for example replication of one to many or one to one, or many to one. An example topology is shown within Figure 4
Well...this article ended up being longer than I'd originally intended.... Hopefully it's helped give a much clearer inderstanding on how all the pieces fit together within within a CR or DR based design.
In Part 2 we'll discuss Cloud based Cyber Recovery & Disaster Recovery options available
Opinions expressed in this article are entirely our own and may not be representative of the views of DELL Technologies/VMware Inc.
This article has been written by Faisal Choudry from DELL
Faisal Choudry has experience and a real interest in implementations of Data Protection and Recovery, whether protecting from Disaster, Cyber attack, or both. He has worked on a number of large implementations of DR and failover (audited) testing and has written a number of papers on the topic. Faisal also worked on the engineering, development, and early implementations of a number of products including VxRail and VCF on VxRail, vBlock, SAP HANA.
#cyberrevovery #disasterrecovery #cyberattack #disaster #recoveryplan #cyber #dataprotection #data #protect #recover #cloud #draas #dell #dellemc #vmware
#backup #replication #cyberwar #dataprotection #cyberattack #cyberrecovery #disasterrecovery #disaster #ransomware