UPDATE 11/11/22: Wired covers the latest in Log4Shell and the FTC's tone deaf threats of enforcement action against big and small businesses in the US if they do not take prompt corrective action. Interestingly, Starling Bank is highlighted as having taken rapid and appropriate action to detect Log4j in its own systems and in the 3rd party systems offered via its platform.
UPDATE 7/11/22: The Apache Software Foundation released its 5th patch for a Log4Shell vulnerability this week, with no signal that this is the last. Log4Shell attacks are spreading faster than Omicron variant. The vulnerability continues to be exploited widely, compromising many applications and data stores. While we tend to think of IT as back office, losses to ransomware and malware exploits can have very real consequences including loss and widespread disruption for targeted sectors such as maritime shipping and financial services. Insurance offers only partial cover. At this scale and scope, IT systems vulnerabilities could become a headwind to corporate profitability, economic stabilisation, and growth hopes in 2022.
Original post:
In September 2013 one of the volunteer coders who contributes to the Apache Software Foundation library of Java code objects injected a Log4j vulnerability into a logging library shared for reuse by other coders. The Apache Software Foundation is trusted by the open-source developer community as a reputable source, even if its tools and objects are written by unpaid volunteers. Log4j and other compromised logging library components have been used ever since to log diagnostic messages, such as 404 'page not found' errors. Hundreds of millions of devices are compromised globally, but the ubiquity of use means that the Log4j code may be hidden deep in layers of software, internet-of-things devices, or indirect in SaaS components of IT architecture.
The vulnerability is termed zero-day because it was either unknown to those who should have been interested in its mitigation, or if known by them, a patch was not developed until exposure.
Log4j is like a Java-based Lego brick of the global internet architecture that developers put into diverse applications to ‘log’ anything that needs logging for monitoring, alerts, diagnostics, or debugging enquiries. Log4j became very, very popular, reused millions of times by developers globally, and now part of applications we all depend on daily. The Log4j vulnerability permits remote code execution on the underlying servers that run vulnerable applications. Anyone aware of the vulnerability can abuse it by specifying custom code for formatting in a log message, for example submit software code to perform actions on a target computer or server. Exploits can steal sensitive information, take control of a target system, or slip malicious content to other users communicating with the server. There is a very low bar for exploiting Log4j so many can use it.
Logging is everywhere as it is fundamental to most software. Log4j is often bundled with other software so may be hard to find in a big application. Log4j is used in gaming, cloud architecture, a wide range of programming, and even security tools. While big companies may inventory and patch quickly, there are many smaller targets who won't acknowledge they have a problem.
How Log4j is used is relevant to patching it. Cisco has done a wholesale system update. Minecraft updated its software to a new version without the vulnerability. Deciding what to do requires analysis of how Log4j got there in the first place, so there is a delay between knowing a vulnerability is possible and deciding how it can be patched. Log4j is likely to prove a vulnerability for years to come. Users can do little except make sure software is up to date.
In 2016 presenters at the Black Hat Security Conference identified a method to exploit a broad class of software that included Log4j. The authors of Log4j and the Apache Software Foundation, if aware of the diagnosed vulnerability, elected to ignore it.
On 13 March 2021 a Github user from Beijing invited collaboration on a hack using Log4j, nice0e3/log4j_POC.
Fast forward to 2:51pm on 24 November 2021 when, according to Bloomberg, Chen Zhaojun, an employee of Alibaba Group Holding Ltd’s cloud-security team, alerted Apache with an email saying, “I want to report a security bug,” and “the vulnerability has a major impact.” The email described how a hacker could use Log4j to achieve ‘remote code execution’ that could remotely take over a server, computer or application hosting the Log4j object. Left unfixed, the software could give hackers – whether motivated by profit or state-controlled political objectives – unfettered access to millions of computer systems in governments, clouds, corporations and homes.
This was an ‘Oh crap’ incident at Apache, according to Gary Gregory who has worked on the volunteer team that maintained Log4j for nearly a decade. The volunteer programmers began working to fix the vulnerability before the rest of the world became aware of the problem.
On 8 December a further unwelcome email arrived from Alibaba’s Chen notifying Apache that details of the Log4j vulnerability had leaked to a Chinese blogging platform for all the world to see. “Some WeChat security chat groups are already discussing the details of the vulnerability, and some security researchers already have the vulnerability,” Chen wrote. “We [Alibaba] promise to keep it a secret until your official release version comes out. Please hurry up.”
Apache published its announcement and patch 20 hours later on 9 December. Since then hackers around the world have been testing systems for the vulnerability at an intensifying rate. The potential scope of impact is huge according to Christian Grobmeier of Apache, who had contributed to the code for a decade. “This is basically half the world, maybe even more. This is just crazy.”
Since publication both state and non-state actors have been ramping up Log4j exploit requests to test the vulnerability of state and non-state targets.
CloudFlare was already noticing by this time an upsurge in activity exploiting the flaw which NIST named as CVE-2021-44228 on 10 December. Cloudflare began collecting minute by minute data. John Graham-Cumming blogged the vulnerability on 10 December with recommendations for mitigation of risks. A separate blog began cataloguing attempted attacks using the Log4j vulnerability.
At this time there appears to be a lot of reconnaissance going on. Actors, good and bad, are scanning for vulnerable servers across the world. Eventually, some of that reconnaissance will turn into actual penetration of servers and companies. And, because logging is so deeply embedded in front end and back end systems, some of that won't become obvious for hours or days.
Like spores quietly growing underground some will break through the soil and into the light.
Log4j is critical for the health of global IT because its exploitation requires no authentication, bypassing all protections. It may have been exploited for years before detection and attempted mitigations. The ubiquity and design of Log4j mean that threat actors may already be inside networks, waiting to move from one protected system to another more vulnerable system. Even after patches and mitigations, everyone needs to assume breach and hunt actively for follow-on attacks taking advantage of access gained in the past through the Log4j exploit.
Cybersecurity is ever evolving, and as our dependence on technology grows and deepens, cybersecurity is a matter of national and economic security as well as a commercial and financial concern. We don't know if the Log4j vulnerability was deliberate in 2013 by a state or non-state actor to have wide access to applications and systems for espionage, compromise, exploitation, or sabotage. It may have been used quietly in the background since 2013, explaining why the 2016 warning went unheeded until the vulnerability was 'in the wild' with thousands of other state and non-state hackers in December 2021.
Even as IT teams around the world scramble to defend their IT infrastructure from penetration and exploit, the Log4j vulnerability is mutating and multiplying faster than Covid-19 and so are the hackers looking to profit from it.
We won’t ever really know the extent of the systems compromised by Log4j and its many variations, now collectively Log4Shell. Banks, corporations, governments, militaries, and other likely targets are unlikely to confirm if they have been compromised or even whether they are at risk. Only the US Department of Homeland Security has invited external hackers to help identify Log4j in its systems with a 'bug bounty' from its new program #HackDHS.
As of writing ransomware exploits by hackers may have already raked in millions. CISA and Five Eyes have issued guidance regarding Log4Shell attacks, and CISA has issued an open-source scanner to find vulnerable applications.
Ironically, the same day that Apache published its announcement and patch for the initial Log4j vulnerability Reuters reported on Collective Strength, an initiative for representatives of national treasuries and international institutions to collaborate over 10 days in Jerusalem in simulation of a major cyber attack on the global financial system. Officials of Israel, the United States, the UK, United Arab Emirates, Austria, Switzerland, Germany, Italy, the Netherlands and Thailand, as well as representatives from the International Monetary Fund, World Bank and Bank for International Settlements participated. Let’s hope they learned some useful lessons.
Wishing everyone happy holidays and a safe and prosperous 2022.