It's like one of those game shows where you are just about to win and then the buzzer sounds.
All your hard work and your execution fade into the background. The spotlight shifts as the clock hits zero. You ran out of time and that is all that matters right now.
Fred Voccola, CEO of IT management company Kaseya, may be feeling that way today.
His company already knew about several Zero-Day vulnerabilities in its VSA application when the REvil ransomware gang exploited one of them.
The race to patch was over, time was up, and REvil won.
"This sucks," the CEO told his customers on video. "This is very disappointing to me personally. I feel like I let the community down, like I let my company down. And our company let you down. That is not going away. I'm not reading off of a script, I'm in my office right now and this is not BS. This is reality. And it sucks."
These are long days for Kaseya, its customers, and for up to 1,500 small and medium-sized businesses infected by ransomware because of the cyberattack.
As the recovery and response continue, it is time to discuss the elephant in the room: was Kaseya kicking security vulnerabilities down the road, or was it making a sincere effort to patch them? Here is what SecureWorld has learned.
[RELATED: Kaseya Ransomware Attack Through the Eyes of the Victim]
Kaseya on notice: you have multiple security vulnerabilities
REvil launched the ransomware attack against Kaseya on July 2, 2021.
But it was nearly three months earlier, on April 6th, when the Dutch Institute for Vulnerability Disclosure notified Kaseya that security researchers had found several problems in Kaseya's software—seven specific security vulnerabilities.
Security vulnerabilities come in lots of flavors; some are minor and an easy fix, while others are complicated to solve and pose great risk to an organization.
The Common Vulnerability Scoring System helps everyone sort this out.
The majority of the vulnerabilities discovered at Kaseya ranked 9.8 or higher out of 10. These were "critical" vulnerabilities that an attacker could take advantage of to do damage.
As the world knows now, that is what happened.
And perhaps this is one of the reasons Kaseya was so quick to identify the point of compromise in this attack. It shared this information just a few hours after detecting the attack:
"We believe that we have identified the source of the vulnerability and are preparing a patch to mitigate it for our on-premises customers that will be tested thoroughly. We will release that patch as quickly as possible to get our customers back up and running."
That was days ago, and its customers are still waiting on a restart, which is now promised by Sunday, July 11th.
But what about our elephant in the room? What did Kaseya do after being told about its "critical" security vulnerabilities?
Kaseya responds: it starts patching vulnerabilities
Let's go back to our timeline now.
The Dutch Institute for Vulnerability Disclosure (DIVD) told Kaseya about vulnerabilities at the company on April 6, 2021.
Just four days later, Kaseya issued a patch for CVE-2021-30118, a remote code execution (RCE) vulnerability. On May 8th, the company patched three more of these vulnerabilities: CVE-2021-30117, CVE-2021-30121, and CVE-2021-30201.
Other vulnerabilities remained unpatched at the time of the attack. And this includes one that could allow hackers to bypass two-factor authentication (2FA), a cross-site scripting bug, and a "credentials leak and business logic flaw" which corresponds to CVE-2021-30116.
Security researcher and author Kevin Beaumont claims this last exploit is the one that was used by REvil to launch its ransomware attack:
"Initial entry was using a zero day vulnerability in Kaseya VSA. This was CVE-2021–30116 (details have not been entered into CVE database, however it has been allocated for this)."
SecureWorld has since learned that two exploits were actually used, at least one of which was previously reported to the company.
It is common practice for an organization to be notified about a vulnerability and a CVE number assigned, with no technical details published until a patch is complete.
The DIVD says it has been caught in a storm of requests asking for technical details on the CVE. Why didn't it tell the world about this known risk?
"As the ransomware attack using Kaseya VSA software has shown, the effects of a malicious actor knowing the full details of a vulnerability can be devastating. This immediately poses a dilemma to anybody that discovers a critical vulnerability in a critical piece of software, do we disclose the details or not?
Say a security researcher discovers a vulnerability in a high-end car. When you kick the left rear bumper in just the right way, the car doors open, and the engine starts. What should the researcher do? Tell everybody, tell all the owners of this type of car, or inform the manufacturer so he can recall and fix the car?
If the full details are made public, it is evident that many cars will get stolen very soon. If you inform the owners, this will likely happen too. The chances of the details remaining secret are slim if you inform a broad audience.
Even if you limit the details to 'a security issue involving the bumper', you might tip off the wrong people. Telling the manufacturer there is a good chance that he comes up with a fix before large-scale car thefts are happening, and consider if you need to tell the owners to keep their car behind closed doors in the meantime."
One point not addressed by the analogy: what if the company you notify about its security vulnerability does not patch it?
Was Kaseya working on a patch to prevent the cyberattack?
What exactly do we know about Kaseya's intentions for the three remaining cybersecurity vulnerabilities reported by the DIVD, at least one of which REvil used?
CISO Frank Breedijk manages the DIVD, and in a recent post, he explained that Kaseya was incredibly responsive:
"Kaseya's response to our disclosure has been on point and timely; unlike other vendors, we have previously disclosed vulnerabilities to.
They listened to our findings, and addressed some of them by releasing a patch resolving a number of these vulnerabilities. Followed by a second patch resolving even more. We've been in contact with Kaseya ahead of the release of both these patches, allowing us to validate that these vulnerabilities had indeed been resolved by the patch in development.
Unfortunately, the worst-case scenario came true on Friday the 2nd of July. Kaseya VSA was used in an attack to spread ransomware, and Kaseya was compelled to use the nuclear option: shutting down their Kaseya Cloud and advising customers to turn off their on-premise Kaseya VSA servers. A message that unfortunately arrived too late for some of their customers."
Customers who are now waiting for a more secure restart of the VSA monitoring service.
But perhaps they will take some comfort in appearances: it seems that Kaseya was working on the vulnerability.
Kaseya CEO promises to make things right
Kaseya CEO Fred Voccola says the the company could have started the VSA service already, but he personally decided to delay things so that extra layers of security could be embedded into its service and software.
And he announced that the company will do a number of things to take care of managed service providers (MSPs) caught in the middle of this ransomware storm:
"We will be providing direct financial assistance to MSPs who have been crippled by these evil people, and the new adversaries that we face."
Adversaries that beat defenders in the race to patch.
[Update]
After the publication of this story, SecureWorld heard from one of our readers who relies on Kaseya. You can sense their frustration:
"Maybe you want to update that with some information like that till now - July 14th - there are still MSPs waiting for a working patch from Kaseya to get their VSAs back online.
The patch released Sunday night is not installable for some or maybe even all of us. This means the provided setup does not work.
I've no information about how many MSPs are affected, but I know there must be a lot as the Kaseya support is struggling to even just reply to emails within a reasonable time.
This is a big mess and I'm lucky not to have all our processes within the Kaseya system. We only use it for the technical ops. But I would assume that other IT companies having their billing and incident management (tickets) in the VSA are just f*cked for almost 2 weeks now."
And Kaseya also issued an update that running the patch may not mean you are patched, if you forget to check a box:
"When running the Kinstall patch on your VSA, if you chose to reinstall VSA and either unchecked the default option to install the latest patch, or reran the Reinstall VSA process a 2nd time without the “install patch” option selected – it’s possible your patch was not re-applied.
While these are rare edge cases, we recommend that you verify that the latest patch was installed properly. We have made a tool that enables you to ensure the patch is properly install."
See the ongoing list of Kaseya updates, here.
