Log4j from the eye of the storm

Unless you live in a cave, you couldn’t miss the last few days the flaws discovered on the Java log4j library.

I’m not going to make another post talk about this flaw, but rather to talk about my experience in the field on the impact of this flaw at the operational level.

The importance of having an up-to-date inventory

I talked about this on Twitter (in French), but for me, this flaw highlights the fact that many companies lack an up-to-date inventory of their resources.

When a breach like this occurs, it is important to be able to quickly identify which resources are at risk. But having an up-to-date inventory is not just about the machines, but also about their content.

This is even more important when you exploit the power of the cloud, having volatile machines make their management even more complex.

For that there are several solutions.

Doing only infra as code

This point will seem essential to many of you, but only doing infra as code simplifies things enormously in order to quickly visualize what is being deployed, on which project, which machine, which account, etc.

In a CI/CD pipeline, it is the role of a SAST (static application security testing) to identify common flaws when compiling new projects; however, there needs to be tooling for already compiled projects, for example, on GitHub, Dependabot showed me fairly quickly the projects in which it had identified the use of the library.

However, this point is clearly not enough. Just because I don’t explicitly define a library doesn’t mean it is not used.

Scanning already created artifacts

If your applications are already compiled, there are often solutions on your library managers to scan them.

For example, Docker Hub allows you to scan your images for the library.

However, this is still not enough. On your servers, you do not necessarily use only packages that you master and reference yourself.

For example, you can use an AWS image of a publisher in the marketplace, or have third-party applications deployed in your infrastructure.

Continuously scan your servers

This is where the third level of scanning comes into play: you can continuously scan your machines to quickly identify servers in danger.

To do this, there are tools like Nessus Tenable or Rapid7 InsightVM. The role of these tools will be to scan your servers and report to you the machines at risk.

A simple search with the identifier of a CVE allows you to quickly identify the impacted servers.

How I lived this stage

Like many in infosec these last days, this first step was complicated. Even with tools, there is always a risk of missing some resources.

I work in a small team and we reacted as best we could, but given the risk involved with this vulnerability, we are re-running several checks. This point is exhausting, because we have to do a lot of manual actions, not everything can be automated.

It is a laborious step, mentally exhausting, but essential and critical.

Once the servers are identified, what do I do?

Now that I know which of my resources are affected by the log4j vulnerability, what do I do?

The first thing we often forget: don’t panic!

This is a behavior I often see during security incidents, yet it is all the more important to keep a cool head as a mistake can be costly!

Patch, patch, patch…

The solution will often be the same here: patch or update the application. However, not all editors are as serious about this game.

At this point, I have seen several behaviors:

• Publishers who are very reactive and transparent about the risks
• Publishers who say they don’t know (which is not necessarily reassuring, by the way).
• Publishers who say they are impacted, but take several days to release a version with the necessary fix
• Those who say they are not impacted and then indicate the opposite
• Those who prefer to indicate that they are not impacted when they are far from java (for example HashiCorp and its tools written in GoLang)
  

Depend on other teams

At my position, I am not the one who patches, but in the team that will identify the risks, measure them and ask the necessary actions to other teams (Dev, Ops, DataOps, etc.)

It’s a position that can be frustrating, because you depend on people over whom you have no power of prioritization and not everyone necessarily perceives the risks of these flaws.

This is why it is important to have (once again) a pedagogical posture, and to explain the risks and why it is important to react quickly.

And it is also at this stage that it is important to trace all the requests and do the associated follow-up.

One of the temporary solutions can sometimes be to mitigate the risk to reduce as much as possible the impact of third party slowness, this can be done: by rules on WAF, package modifications directly on the machines (even if I’m not a fan of this solution), change the exposure of some machines (move machines behind a VPN for example) or interrupt some services temporarily.

The temporary solution depends, of course, on the assessment of the associated risk.

How I lived this stage

As I mentioned, I was a bit frustrated by the lack of response from some teams, or the impression that they didn’t care… For some teams, I’m just one incident among many already in progress, and I’m just adding a piece to the machine.

This step is not necessarily simple, but I find it simpler than having the inventory.

After log4j

For the moment the log4j storm is not over yet, with a flaw discovered the day before yesterday as I write these lines.

Invest in security

This storm, like others before it (Spectre, Meltdown, Heartbleed, etc.) is an opportunity for security teams to remind the importance of certain investments:

• Having a security team that keeps up to date
• Having the tools to quickly identify vulnerabilities
• Having an up-to-date inventory
• Educate teams on the importance of good security practices

I even reiterated this point (on Twitter once again, in French) recently:

Fighting the predatory behavior of big tech

This flaw reminds us of the harsh realities of open source projects: many projects are clearly exploited by many huge companies that rely on them without ever contributing either code or at least financially.

Log4j is a library exploited by millions of applications in the world, yet it is now by few people.

Many techs in the open source world have made the same observation (in French, once again):

This is not unlike recent battles:

• Elastic.co VS AWS
• MongoDB VS AWS

In the absence of a viable solution, the solution is often to go through the legal process by putting clauses preventing the use of these companies, because they exploit the wealth of these projects without ever giving anything in return.