Can’t Judge a Book by its Recover! Cyber Recovery
Updated: Mar 2
This year, an increasing number of customers began asking us the same question relating to their infrastructure, VxRail and VCF on VxRail:
“How do we recover from a cyber-attack...?”
They elaborated further:
“We have to assume our existing data protection and Disaster Recovery implementations have also been compromised by the attack!”
The next requirement further complicated matters:
“Additionally, the existing servers may not be safe due to the cyber-attack, A forensic team may isolate the hardware for the purposes of investigation and analysis. The recovery may therefore have to take place on new servers/hardware.”
Well…that threw most of the options previously assumed relevant for recovery, out of the Window! Reading on the topic and brainstorming with colleagues on the use case and subject matter, surprisingly it become evident just how unprepared most are.
The objective of this blog series is to discuss cyber-attacks and recovery in the context of the infrastructure and the above use case. The wider topic surrounding Cyber Recovery, particularly the common misconception that traditional Disaster Recovery (DR) and Backups are adequate, will be covered in a future post.
Statistically your infrastructure is more likely to suffer a cyber-attack than a disaster such as earth-quake, flood or explosion, yet the majority are only prepared for these forms of traditional disaster. Cyber-attacks should not be viewed in the same lens as disasters. They require a vastly different approach for protection and recovery.
Let’s look at some recent examples of attacks. Preferred targets tend to be large companies including manufacturing, government agencies, financial institutions and their infrastructure. That is because they're more likely to pay ransom rather than suffer larger monetary loss.
A common theme among the below use cases: large companies/organisations, have a mix of IT systems ranging from legacy, to very new systems. The attacks were initiated via proxy from inside, for example an email or an attachment was used. Once component(s) installed, the attackers then gained further access over weeks or months but nothing prevented or detected that. By the time the actual attack took place, it was too late. Complete shutdown followed as an attempt to stem the attack. Recovery took months and involved entire rebuilds. A lot of the DR systems in place, could not be used due to their current design of being connected to the production systems
Cyber Attack on the healthcare system in Ireland (HSE)
On Fr 14th May 2021, The IT systems of the entire health service of Ireland, the Health Service Executive (HSE) were subjected to a cyber-attack. Once the HSE IT security teams collectively realised what was happening, the HSE declared a Critical Incident and began a sequence of events which led to the only option available to them: Switch off all their IT systems and disconnect all networks to avoid any further penetration or damage from the attackers.
Independent Report by PricewaterhouseCoopers (PWC)
PricewaterhouseCoopers were brought in to analyse and review the HSE cyber-attack. A 150-page report was published (see link above) which examined all events including the aftermath and the implications for the health service when all systems were shut off. Every aspect of the health service from patient record access, appointments, referrals, surgery, to Covid vaccine deployment were affected. Staff were forced to use pen and paper and produce manual systems. All recovery and business continuity systems in place were designed to recover from traditional type of disasters, hence were of no use in this situation.
Norsk Hydro Attack
Norsk Hydro is one of the world’s largest aluminium and renewable energy companies, headquartered in Oslo Norway. During March 2019 at 4am a call came into the CEO. “We are under a severe cyber-attack; This is not a drill.”
IT had no choice but to shut everything down including all servers and networks. They later found the ransomware was LockerGoga. The hackers had weaponized a customer email string and inserted a Trojan horse which infected an employee desktop. LockerGoga eventually encrypted files across thousands of servers, desktops and laptops and posted the message in the figure above.
The attack cost the company $71 million. They had to completely shutdown all IT. Operations and production had to run manually with pen and paper even to the point of recalling retired employees, more experienced with manual procedures for production.
Norsk Hydro did however take a different approach. They refused to pay ransom. Instead, they went public with the news, including letting the media onsite to watch Microsoft’s Detection and Response Team (DART) and Norsk Hydro’s IT teams work through recovery. Norsk’s share price increased and much of the industry learnt valuable lessons from their openness. Recovery took months and is still on-going.
Colonial Pipeline: The largest publicly disclosed U.S. Cyber Attack
This is the largest disclosed attack against critical infrastructure within the USA.
Colonial Pipeline is the largest pipeline operator in the US and provides approximately 45% of the East coast’s fuel supply. The attack took place April 2021 and resulted in the shutdown of operations.
Following the attack, President Joe Biden declared a U.S Federal State of Emergency, and executive order was later signed by the president relating to cyber-security and included a Bill of materials definition requirement. That gives a sense of perspective of the scale of threat posed by cyber-attacks to society and world economies.
The attack disrupted systems which control supply lines and so caused fuel shortages including at gasoline stations, which resulted in panic buying, Fuel price fluctuations, flight cancellations with major airlines resulted, even disruption of fuel supplies to the military. Supplies and operations were run manually to resume services. An example below of just one of the measures taken to counter the far reaching effects of the attack:
“To keep supplies flowing, the USDOT Federal Motor Carrier Safety Administration (FMCSA) issued a Regional Emergency Declaration on Sunday 9, easing standard restrictions on the land transport of fuel and the permissible working hours of drivers. “
Source: Federal Motor Carrier Safety Administration (FMCA)
The below link is a broadcast on how the group responsible for the attack are being tracked down:
In the next part of this series we will discuss the misconception that Disaster Recovery is sufficient for recovery from a Cyber-attack. We will also look at our use case and begin describing how we can protect and recover VCF and VCF VxRail from an attack.
Opinions expressed in this article are entirely our own and may not be representative of the views of DELL Technologies/VMware Inc.
This article has jointly been written by Faisal Choudry from DELL and Shreyash Nalamwar
Faisal Choudry has experience and an interest on implementations of DR including audits and live testing, as well as papers written on the topic. Faisal also worked on the engineering, development and early implementations of VxRail and VCF on VxRail.
Shreyash Nalamwar has 18+ years of experience in cloud and datacenter technologies. Currently working as a Staff Solution Architect at VMware and is one of the founding members of the VCF team, focused on providing design and deployment guidance for VCF customers across the APJ region.