We May Never Be Rid of Log4J. It Was, and Still Is, an Endemic Vulnerability. It’s the Long Tail That Is Concerning.

The below is a tweet from only a few days ago. We're now years into Log4J, and, like many things in technology, we tend to overestimate the impact of the short term (a new massive bug that we fix relatively quickly) and underestimate the impact over the long term (how long this bug will last).

To be fair, the above is a graph of attempts to exploit Log4J via RCE (remote code execution), not actual exploitation. But attackers (in theory) need to spend their resources wisely, and based on the above it would seem that it is still cost effective to go after the Log4J CVEs of yesteryear.

From the CISA July 11, 2022 report:

At this time, Log4j has become an “endemic vulnerability” that will be exploited for years to come. The impact to organizations over the long term will be difficult to assess without better tools for discerning real exploitation and centralized reporting of successful compromises. - https://www.cisa.gov/sites/default/files/publications/CSRB-Report-on-Log4-July-11-2022_508.pdf

As well, only a few months ago, we also had other reports on how Log4J was aging.

Veracode found that STILL more than one in three Java apps, 38%, were vulnerable to the Log4J issue. Glass half full: 2/3rds are not vulnerable. Glass half empty: 1/3rd are.

The bigger story at the two-year anniversary, however, is that there is still room for improvement when it comes to open-source software security. If Log4Shell was another example in a long series of wake-up calls to adopt more stringent open-source security practices, the fact that more than 1 in 3 applications currently run vulnerable versions of Log4j shows there is more work to do. The major takeaway here is that organizations may not be aware of how much open-source security risk they are exposed to and how to mitigate it. - https://www.veracode.com/blog/research/state-log4j-vulnerabilities-how-much-did-log4shell-change

Which takes us back to what some were saying in 2021...

“Log4Shell will be exploited for years because of its ubiquity. There are over three billion devices running Java, many of which are using Log4j. Getting every single one of these systems patched is no small feat, and requires a massive awareness campaign and guidance to developers on how to address this vulnerability,” she told Infosecurity. - https://www.infosecurity-magazine.com/news/experts-log4j-bug-could-be/

Java Will Be Around A Long Time

For some timeline comparisons, consider the age of some of the tools we use, for example the Spring framework reached 1.0 20 years ago. (Congratulations to Spring, by the way!)

Java was invented around 1995. Java 8 was released on 18 March 2014. Log4J November 2021. Java has a long, long, long life ahead of it, much longer than the Log4J problem, but this particular issue will be relevant for further into the future than most of us would have thought at the end of 2021.

Now, 20 years is not really that old in the grand scheme of things, and I fully expect Spring to be around for at least that long in the future, because it should be, but in our age of constant computing invention, with our fragmented history, we tend not to expect technologies to have that lengthy of a lifespan; that tools and apps and programming languages come and go like the seasons. In a similar vein, neither would we necessarily expect a massive vulnerability to have such a long term existence. But they do, and many of them will be around as long as we are.

The Long Tail of Vulnerabilities

There are a couple of "long tails" in vulnerabilities. (In the above image, the green section has the same area as the yellow.)

1) The first long tail is that we don't know all the vulnerabilities that exist, we'll fix a few that may be important, but many will still linger.

There's at least one academic paper that discusses this idea.

“Software metrics showed fat [long] tails in distributions of software defect data…Therefore, a correlation between the power law function and software metrics may exist.

However, to the best of our knowledge, few people have considered the power law function in predicting software defects.” - Paper: A Novel Approach for Software Defect prediction Based on the Power Law Function

2) The other long tail is that not every application using Log4J will be patched, as the previously mentioned articles discuss.

Conclusion

One thing I think about when I write this is I wonder what organizations, what applications, what products, haven't updated Log4J. I would imagine that most of the larger companies that have the resources to do so have managed to update their apps, or at least the Internet facing ones, and that where we are going to see problems is with companies that don't have the resources to fix these kinds of endemic software vulnerabilities, as well as any kind of hardware or device that has Java embedded in it (which of course will likely never be updated). As well, now that the hype around the vulnerability is over, people will just forget that it is an issue and use the older library without realizing.

Extra: History of Log4J

It's worth taking a quick look at the history of Log4J to remember how it happened, although the point of this post is that we're not done with the problem, and what we do from now on is what's important.

That said, what we're think of, or at least what I'm thinking of when I say "Log4J" (which isn't fair, because Log4J itself isn't the problem, just the fact that it had a major security problem, like many pieces of code), is the vulnerability that was discovered by Alibaba on 24 November 2021 and tweeted about in summary on 9 December 2021.

Again, from the CISA document:

On November 24, 2021, a security engineer from the Alibaba Cloud Security team, within the People’s Republic of China (PRC), reported a vulnerability in the JNDI feature to the Apache Software Foundation (ASF), a non-profit corporation that provides support for the Log4j project. While ASF was working to understand the issue and devise a fix, another party disclosed the vulnerability before ASF made an upgrade available to the general public. Such a disclosure of a significant vulnerability in any widely used piece of software immediately triggers a race between defense and offense: a race to apply upgrades before threat actors exploit vulnerable systems. The Log4j vulnerability was no exception.

Defenders faced a particularly challenging situation; the vulnerability impacted virtually every networked organization and the severity of the threat required fast action. The fact that there is no comprehensive “customer list” for Log4j, or even a list of where it is integrated as a sub-system, hindered defender progress. Enterprises and vendors scrambled to discover where they used Log4j. The pace, pressure, and publicity compounded the defensive challenges: security researchers quickly found additional vulnerabilities in Log4j, contributing to confusion and “patching fatigue”; defenders struggled to distinguish vulnerability scanning by bona fide researchers from threat actors; and responders found it difficult to find authoritative sources of information on how to address the issues. This culminated in one of the most intensive cybersecurity community responses in history. - https://www.cisa.gov/sites/default/files/publications/CSRB-Report-on-Log4-July-11-2022_508.pdf