The first signs of the ransomware attack at data storage vendor Spectra Logic were reports from a number of IT staffers about little things going wrong at the beginning of the day. Matters steadily worsened within a very short time and signs of a breach became apparent. Screens then started to display a ransom demand, which said files had been encrypted by the NetWalker ransomware virus. The ransom demand was $3.6 million, to be paid in bitcoin within five days.
Tony Mendoza, Senior Director of Enterprise Business Solutions at Spectra Logic, laid out the details of the attack at the annual Fujifilm Recording Media USA (FRMA) Conference in San Diego late last month.
“We unplugged systems, as the virus was spreading faster than we could investigate,” Mendoza told conference attendees. “As we didn’t have a comprehensive cybersecurity plan in place, the attack brought the entire business to its knees.”
Cyber Insurer Provides HelpThe IT team spent the day assessing systems to see which ones were virus-free. Most had been infected. As Spectra Logic had the foresight to take out cyber insurance, Chubb representatives were professional and helpful, according to Mendoza. Chubb set the company up with Ankura, a cybersecurity recovery specialist that has also trained the FBI on cybersecurity.
Ankura immediately provided security operations center (SOC) services to stop the virus from spreading, protect against further damage, and to begin the process of removing it. Forensic analysis of the breach came to a quick conclusion – a phishing attempt had tricked a user with privileged access into clicking on a malicious link.
“The guys in the SOC discovered that the virus came in via a remote user, had spread over the VPN and then began to look for security flaws,” said Mendoza.
Fortunately, the email system escaped the virus. The team brought that system back online the next morning to enable the company to commence limited operations and to inform employees what had occurred. The IT team worked through the night for many days on intensive remediation efforts.
Also read:
Backups Wiped Out But Tape, Snapshots SurviveAs the backup account had been compromised and the backup server wiped out, online backups were useless. A detailed check revealed that no data had left the premises, although the criminals behind the hack had been stealing passwords. Instructions were issued to change passwords immediately.
Meanwhile, Spectra Logic got in touch with the FBI. The agency said the virus itself could not easily be undone in the systems that had been infected. At that point, it became a race against time to see if there was some way to recover systems before the ransom deadline arrived.
Although the backup server was useless, the company had retained a copy of all its data on tape. These tape cartridges were not impacted by the hack. Those stored on premises had been kept offline. Further tape copies were available at an offsite location.
“The air gap provided by offline tape cartridges gave us hope that we could actually get the company fully operational again in a reasonable time,” said Mendoza.
IT got to work restoring things from tape. Initial estimates put time to recovery from tape at about 30 days for Tier 1 production and up to six weeks for everything else – slow, but it gave the company the confidence to ignore the ransom request with the FBI’s blessing. Soon afterwards, snapshots on disk were discovered that were immutable and safe from the virus. Those snapshots enabled recovery to be accomplished in a few days.
“We were able to restore everything and paid nothing,” said Mendoza. “Other than a few files, all data was recovered.”
Also read: Best Backup Solutions for Ransomware Protection
Lessons LearnedMondoza advises others that disk snapshots and offsite tape with an air gap are the best way to provide a sound recovery pathway after an attack. In his organization’s case, read-only immutable ZFS-based snapshots were stored on an HPE NAS system. Spectra Logic’s own systems were used for onsite and offsite tape storage.
Tape storage can take time to recover, but given Spectra Logic’s backup failures, tape media was a good additional backup measure to have.
“It took us almost a month to fully recover and get over the ransomware pain,” said Mendoza.
He agrees that threat protection is the first line of defense. But a breach is likely to happen eventually. Threat protection technology must be supported by immutability at the data level via snapshots as well as a tape air gap, he said.
That said, he stresses that security can’t take precedence over productivity. Both factors must be balanced. The company is also in favor of good security awareness education for users, as well as strong ransomware and cybersecurity insurance.
Soon after the attack, the company developed a thorough response and action plan and a cybersecurity plan.
“Our attack happened at the speed of disk and flash and was system-agnostic,” said Mendoza. “Air-gapped tape gave us the confidence to say no to paying a ransom and bought us time to find faster ways to recover.”
Ransomware attacks are happening with greater frequency than ever. A recent study published by Arcserve found that 50% of respondents were hit with ransomware attacks over the past year. More than a third (35%) said their organizations were asked to pay over $100,000 in ransom payments, and 20% were asked to pay between $1 million and $10 million.
Given the high cost of ransom payments and downtime, any company that relies on data would be wise to have a comprehensive plan and solutions in place.
Also read:
Tony Mendoza, Senior Director of Enterprise Business Solutions at Spectra Logic, laid out the details of the attack at the annual Fujifilm Recording Media USA (FRMA) Conference in San Diego late last month.
“We unplugged systems, as the virus was spreading faster than we could investigate,” Mendoza told conference attendees. “As we didn’t have a comprehensive cybersecurity plan in place, the attack brought the entire business to its knees.”
Cyber Insurer Provides HelpThe IT team spent the day assessing systems to see which ones were virus-free. Most had been infected. As Spectra Logic had the foresight to take out cyber insurance, Chubb representatives were professional and helpful, according to Mendoza. Chubb set the company up with Ankura, a cybersecurity recovery specialist that has also trained the FBI on cybersecurity.
Ankura immediately provided security operations center (SOC) services to stop the virus from spreading, protect against further damage, and to begin the process of removing it. Forensic analysis of the breach came to a quick conclusion – a phishing attempt had tricked a user with privileged access into clicking on a malicious link.
“The guys in the SOC discovered that the virus came in via a remote user, had spread over the VPN and then began to look for security flaws,” said Mendoza.
Fortunately, the email system escaped the virus. The team brought that system back online the next morning to enable the company to commence limited operations and to inform employees what had occurred. The IT team worked through the night for many days on intensive remediation efforts.
Also read:
Backups Wiped Out But Tape, Snapshots SurviveAs the backup account had been compromised and the backup server wiped out, online backups were useless. A detailed check revealed that no data had left the premises, although the criminals behind the hack had been stealing passwords. Instructions were issued to change passwords immediately.
Meanwhile, Spectra Logic got in touch with the FBI. The agency said the virus itself could not easily be undone in the systems that had been infected. At that point, it became a race against time to see if there was some way to recover systems before the ransom deadline arrived.
Although the backup server was useless, the company had retained a copy of all its data on tape. These tape cartridges were not impacted by the hack. Those stored on premises had been kept offline. Further tape copies were available at an offsite location.
“The air gap provided by offline tape cartridges gave us hope that we could actually get the company fully operational again in a reasonable time,” said Mendoza.
IT got to work restoring things from tape. Initial estimates put time to recovery from tape at about 30 days for Tier 1 production and up to six weeks for everything else – slow, but it gave the company the confidence to ignore the ransom request with the FBI’s blessing. Soon afterwards, snapshots on disk were discovered that were immutable and safe from the virus. Those snapshots enabled recovery to be accomplished in a few days.
“We were able to restore everything and paid nothing,” said Mendoza. “Other than a few files, all data was recovered.”
Also read: Best Backup Solutions for Ransomware Protection
Lessons LearnedMondoza advises others that disk snapshots and offsite tape with an air gap are the best way to provide a sound recovery pathway after an attack. In his organization’s case, read-only immutable ZFS-based snapshots were stored on an HPE NAS system. Spectra Logic’s own systems were used for onsite and offsite tape storage.
Tape storage can take time to recover, but given Spectra Logic’s backup failures, tape media was a good additional backup measure to have.
“It took us almost a month to fully recover and get over the ransomware pain,” said Mendoza.
He agrees that threat protection is the first line of defense. But a breach is likely to happen eventually. Threat protection technology must be supported by immutability at the data level via snapshots as well as a tape air gap, he said.
That said, he stresses that security can’t take precedence over productivity. Both factors must be balanced. The company is also in favor of good security awareness education for users, as well as strong ransomware and cybersecurity insurance.
Soon after the attack, the company developed a thorough response and action plan and a cybersecurity plan.
“Our attack happened at the speed of disk and flash and was system-agnostic,” said Mendoza. “Air-gapped tape gave us the confidence to say no to paying a ransom and bought us time to find faster ways to recover.”
Ransomware attacks are happening with greater frequency than ever. A recent study published by Arcserve found that 50% of respondents were hit with ransomware attacks over the past year. More than a third (35%) said their organizations were asked to pay over $100,000 in ransom payments, and 20% were asked to pay between $1 million and $10 million.
Given the high cost of ransom payments and downtime, any company that relies on data would be wise to have a comprehensive plan and solutions in place.
Also read:
RSS Feed
