Using OpenShift Secrets With Spring Boot + Kafka

Going to skip right away to the juicy stuff as much as I can. I won’t discuss about what OpenShift is or what the Secrets feature does. This is only about my experience the first time I dealt with Secrets and how I used it for a Spring Boot Kafka consumer/client application.

The problem I needed to solve was to use Secrets to store a Java KeyStore (JKS). Then plug that into the container of the app. JKS in Java is the proprietary means to store SSL certificates for establishing secure connections – a requirement for the app to connect to a secure Kafka messaging server.

I was trying to figure out how to do all this, though I really did not go in with nothing. To start with, I had a handful of materials from another work that already had done what I was about to do. So it was not like uncharted territories. The only difference is that that Java application is J2EE only, while the one I worked on used Spring Boot.

One subtle difference is how that application was wired to start. Another is that a number of the Kafka properties were placed as environment variables, then were assigned to a Properties object within the Java code. While that can be done in the same way for a Spring Boot app, it’s just that using the more abstracted route seems to be the path of least resistance. Almost every, if not all, Kafka property that needs defining can be placed in the application.yaml file in Spring Boot’s case.

Going on, I figured later after much tinkering that some of the stuff done by the original implementation to load the JKS file into the containerized app, I could do without in Spring Boot. While another would cause it to crash the pods. For example, there were a chain of Linux shell commands that was defined in the Dockerfile, and the exact same bunch were also found in the DeploymentConfig.yaml file’s container command argument.

At first I tried to mimic all of these shell commands and placements to decode the Secret as they were, but it would not work. I had permission errors when creating a directory or writing to it. If not, then instead I’d find that during deployment the pod would just crash repeatedly.

I did many experimentations. Different combinations. 30 plus builds in total, or more. This was a crash course on OpenShift Secrets, all the while frantically reading and Google-ing documentations for OpenShift Secrets, even ConfigMaps and all other things related to my mission.

In the end, 2 approaches worked for me. Although do note there are now 3 solutions involved. Let’s name these:

  • J2EE-solution (did not work)
  • COPY-to-container-solution (works)
  • my-hybrid-solution (works)

The COPY-to-container-solution was derived later on from yet another team who COPY-ed the JKS file from the application resources directory into the container. Theirs is also a Spring Boot app. They did not store the JKS as a Secret. That part was skipped except for the JKS password. It was a clean and simple solution. No headaches. Not much more in terms of configurations than what came in from Initializr, and the password environment variable. The only caveat is that the JKS file was kind of exposed. Builds have to be run through the CI/CD pipeline. That means committing the file to the source code repository. It all boiled down to this:

In Dockerfile:

COPY src/main/resources/store/dev/keystore.jks /path/to/keystore/dir/dev/

In DeploymentConfig.yaml:

env:
  - name: MY_KEYSTORE_PWD
    valueFrom:
      secretKeyRef:
        name: my-keystore-pwd-secret
        key: my-keystore-pwd-secret-key

In application.yaml:

spring.kafka.ssl.key-store-location: "file:/path/to/keystore/dir/dev/keystore.jks"
spring.kafka.ssl.key-store-password: "${MY_KEYSTORE_PWD}"

J2EE-solution approach was to create a directory, pass the value of the Secret, decode it, then write it to the former. To visualize, this is how it would look like for example:

In Dockerfile:

CMD source dev_env.conf && mkdir -p "/path/to/keystore/dir" && echo -n "$MY_JKS_SECRET" | base64 --decode > "/path/to/keystore/dir/keystore.jks" && exec /bin/standalone.sh ${JAVA_OPTS}

In DeploymentConfig.yaml:

- command:
    - sh
    - -c
    - source dev_env.conf && mkdir -p "/path/to/keystore/dir" && echo -n "$MY_JKS_SECRET" | base64 --decode > "/path/to/keystore/dir/keystore.jks"  && exec /bin/standalone.sh ${JAVA_OPTS}
  env:
    - name: MY_JKS_SECRET
      valueFrom:
        secretKeyRef:
          name: my-keystore-secret
          key: my-keystore-secret-key
    - name: MY_KEYSTORE_PWD
      valueFrom:
        secretKeyRef:
          name: my-keystore-pwd-secret
          key: my-keystore-pwd-secret-key

Properties object defined within the code:

props.put("ssl.keystore.location","/path/to/keystore/dir/keystore.jks");
props.put("ssl.truststore.password","$MY_KEYSTORE_PWD");

My approach, or the my-hybrid-solution one, is more config-centered and a little bit of shell commands. No copy of files is done into the container. Both the JKS and password are still stored as Secrets. The password is mapped to an environment variable. While the JKS is mounted into a volume as a file.

Just to note, Secrets are usually saved as Base64-encoded string values. There are 2 ways, at the least, to consume the Secret in your application and that’s through, (1) environment variable mapping, and/or (2) as a mounted volume.

There is an extra mounted volume, which is initialized as an empty directory. It is used for storing the decoded Secret back to the original JKS file format.

Lastly, there is the Container LifeCycle Hook part. This is where the decoding and saving shell command voodoo happens.

The configuration file will be fatter. But not by much. Below is an illustration of how it looks like:

In DeploymentConfig.yaml:

(1)

env:
  - name: MY_KEYSTORE_PWD
    valueFrom:
      secretKeyRef:
        name: my-keystore-pwd-secret
        key: my-keystore-pwd-secret-key

(2)

# names must match in the volumes section
volumeMounts:
  - name: my-secret-volume
    mountPath: /mnt/secret
  - name: my-keystore-volume
    mountPath: /mnt/keystore

volumes:
  - name: my-secret-volume
    secret:
      secretName: my-keystore-secret
  - name: my-keystore-volume
    emptyDir:
      medium: memory

(3)

lifecycle:
  postStart:
    exec:
      command:
        - bash
        - -c
        - cat "/mnt/secret/my-keystore-pwd-secret-key" | base64 -d > "/mnt/keystore/keystore.jks"

Numbers 1 to 3 above are only for purposes of making it more obvious. All of these configurations are under the DeploymentConfig.yaml file.

In application.yaml:

(4)

spring.kafka.ssl.key-store-location: "file:/mnt/keystore/keystore.jks"
spring.kafka.ssl.key-store-password: "${MY_KEYSTORE_PWD}"

Before I end, let me say that I got very confused with the Secrets documentation. It doesn’t matter whether it’s OpenShift or Kubernetes. The way the examples, and from 3rd parties as well, were presented made it seem like the system will decode the Secret automatically upon volume mount. I might have misunderstood or there were caveats I do not know about and missed. All the while I thought the volume mounting was at fault.

Take this with a grain of salt, okay. Your mileage may vary.

In my case, it was crystal clear that something was wrong once the application started throwing exceptions that the JKS file is an Invalid Keystore Format. I had to check the Secret value in the mounted volume by printing it to standard out. Clear as day, everything in the mounted file is base64-encoded string.

Lastly, the original J2EE implementation works just fine. Not saying it didn’t or anything else. It did what it was supposed to do. Solid! Learned a lot from it. I just could not literally copy & paste it into the app I am working on. Hence, I did what I could do best to logically make my app do what the former is already doing. If it created a directory through shell commands, I’d use Docker features to do that to minimize the lengthy shell commands. And so on…

If anyone has suggestions of a better way to do this, or what I may be doing wrong, not including using third party Vaults or Service Accounts, feel free to message me. Would love to hear from you.

Similar Posts:

Notice: This article was published on October 2, 2020 and the content above may be out of date.
How To, linux, Programming, work work , , , , , , , ,