Codecov Supply Chain Attack Explained

OK, we now have another supply chain attack that could become the next big hack. When April fools’ jokes were being published online, one company known as Codecov discovered something that was far from a joke. So, who is Codecov? Codecov is one of the many DevOps tools out there. It provides hosted testing reports and statistics. It is compatible with Github, Bitbucket, and Gitlab. If you didn’t know, these are the three largest web-based code hosting services used by developers and companies worldwide. The company was founded back in 2014 and I believe it is based out of New York. Unlike SolarWinds and Microsoft, Codecov is not a publicly-traded company, it does look like a startup but there are no reports of capital raise from an investor. It has less than 50 employees. The hackers were able to gain access to a Bash uploader script and backdoor it to hijack credentials. It allowed hackers to have access to private code repositories, which include source code and all other stuff that the application token was authorized for. The threat actor has been inside their network for over 2 months.

So, in this article, I wanna discuss the Codecov supply chain attack, the hack timeline, what happened, and the future ramifications as the investigation continue. The Codecov software development and distribution system hack is the latest supply chain attack to come to the surface. Last year in December, a similar compromise hit SolarWinds, which put about 18,000 of their customers at risk, including Fortune 500 companies and government agencies.

Why Supply Chain Attacks

The supply chain attacks make for great hacks for at least three reasons. Number One. The breadth and effectiveness. By compromising a single player in the software supply chain, hackers can potentially infect any organization who uses the hacked and backdoored product. Number Two. Digital signatures. The software is signed by the original developer as a whole so digital signatures indicate that it’s legit. Number Three. Virtually unlimited attack surface. There is no easy way to put a stop to these hacks. Why? Because nearly all software projects use third-party libraries, and it is nearly impossible to ensure that they are legit.

CodeCov Hack Details

Anyhow, here is what happened. It started with hackers gaining access to Codecov’s Google Cloud Storage keys. The hackers were able to spot a vulnerability in Codecov’s Docker image creation process and that allowed the actor to extract the credential required to modify the uploader script. The company didn’t know of the breach until a customer of theirs noticed that the digital fingerprint IE an SHA checksum for their bash uploader executable didn’t match with the fingerprint contained in the file downloaded from their website which means the file has been tampered with. So, there are two ways you can use the bash uploader and those are self-hosted and cloud-hosted. The vulnerability only lies in the cloud-hosted version.

Codecov and CI

As per Codecov, to be impacted, your CI pipeline would need to be fetching the bash uploader from codecov.io/bash instead of from your self-hosted Codecov installation. A customer can verify from where the bash uploader is being fetched, by looking at the CI pipeline configuration. Now, this enabled threat actors to potentially export information stored in users’ continuous integration or CI environments. If you didn’t know, Continuous Integration or CI is a DevOps practice where developers integrate code into a shared repository frequently, practically several times a day. Each integration can then be verified by an automated build and automated tests before it is merged into the master or main branch. There is evidence that the stolen credentials or data was exfiltrated to a command-and-control server. In terms of timeline, the unauthorized access took place on January 31 and was discovered on April 1.

Codecov Hack Ramifications

There is no news on exactly what has been stolen but any credentials, tokens, keys, code that was passing through their CI runner would be open to breach when the bash uploader script was executed. Beyond that, application credentials like any services or datastores are also fair game. It may sound weird, but the best case is that all code that existed in the private repos is now stolen, and the worst case is that everything I mentioned was stolen and that’s really bad news for the Codecov customers. There is no disclosure on how many customers were impacted but their website lists Atlassian, Mozilla, GoDaddy, and other high-profile customers that would make a great target. The remedy that Codecov has recommended to its customers is just to revoke and reset the credentials that the CI process had access to in customer environments.

Codecov and Supply Chain Attacks in the Future

This is an ongoing investigation so we will know more over time so here is my final thought. It is yet another scary reminder of how much of the third-party software integration ecosystem operates on a pure trust basis. Developers import libraries and even auto-update them who knows where all the time. Nothing stops a threat actor to even go on a library shopping spree so they can own a famous commonly used third-party library and then hijack anything and everything inside a target system. It would have been trivial for Codecov to detect the sha checksum change. So, the malicious version to exist in the wild for three months indicates that no one at Codecov bothered to perform the simple check. Anyhow, thank you for reading the article. Let’s continue the dialog in the comments.