On April 1st software testing firm, Codecov became aware of a security incident.

The company learned, that for over two months, Codecov’s Bash Uploader scripts used by hundreds or thousands of their customers had been altered with a malicious line of code that exfiltrated information in the environment variables present on the users’ CI/CD environments to an attacker’s IP address.

The flaw originated due to an error in the Docker image creation process, which, according to Codecov, “allowed the actor to extract the credential required to modify our Bash Uploader script.”

Codecov provides code coverage, testing, and stats to over 29,000 companies, and even has a handy GitHub app to integrate the tool right within your open-source software project.

As such, the security advisory released by Codecov strongly advised users to reset all of their credentials, tokens, or keys that were present in the environment variables in their CI processes that used Codecov uploaders.

Hundreds of networks reportedly breached

Although the initial compromise seemed limited to Codecov’s Bash Uploader, the scope of this breach was found to have expanded well beyond just that, when U.S. federal investigators hinted at hundreds of client networks having been breached by hackers as they managed to collect customer credentials using the tainted Bash Uploader tool.

And, as it turns out, these suspicions had been made by some industry experts as well:

Last week, a prominent Codecov client, HashiCorp disclosed that their GPG private key used to sign and validate software packages had been exposed as a result of this incident.

To illustrate all of this with a timeline:

But, the total list of affected projects and victims of the breach is expected to be much bigger and expansive, given the compromised projects themselves are used by very many customers.

Many more disclosures expected

Codecov breach has drawn comparison to the SolarWinds supply-chain incident because, despite some differences in the technical exploitation of the products in both cases, the after-effects of the incident cascaded many layers deep and have been felt by multiple organizations.

A simple search on GitHub for projects potentially affected by Codecov reveals a long list, with the users of those projects now beginning to raise questions:

Developers maintaining the open-source repositories of Netflix and vim also rotated some credentials as a safety measure, following the incident.

Although there remains more to be discovered as more security disclosures come out in the next few weeks incidents like these are testament to attackers targeting developer tools in supply-chain attacks due to the manifold returns they can expect from it.

Whether it be the tainted Codecov bash uploader, or open-source malware targeting developers, a successful attack means threat actors would now have access to not only a single compromised system, but to secrets that can then be misused to gain further access to a bigger number of proprietary assets.

As supply-chain attacks continue to evolve and become more sophistcated, novel software security approaches are needed to combat the adversaries.

Taking steps to add stringent security to your container images — such as checking for common configuration errors, suspicious traffic, and known vulnerabilities, is one way to spot and squash such incidents early on before they take off.

Further, adding automated checksum-based verification to critical assets and files is another way to spot anomalies before they escalate into successful attacks.

© 2021. Ax Sharma. All Rights Reserved.

About the author

Ax Sharma

Ax Sharma is an Indian-origin British security researcher, journalist and TV subject matter expert with a focus on malware analysis and cybercrime investigations. His areas of interest include open source software security, threat intel analysis, and reverse engineering. Frequently featured by leading media outlets like the BBC, Channel 5, Fortune, WIRED, The Register, among others, Ax is an active community member of the OWASP Foundation and the British Association of Journalists (BAJ).

See author's posts