This is a blog regarding an error I encountered while trying connecting to Confluent Kafka Bootstrap server from self-manged KafkaConnect in my local minikube server in windows environment. If you are getting this error and are not using windows environment, this article might not be useful for you.

JAAS stands for Java Authentication and Authorization Service. And per Oracle JAAS reference guide it can be used for two purposes:

1. for authentication of users, to reliably and securely determine who is currently executing Java code, regardless of whether the code is running as an application, an applet, a bean, or a servlet; and
2. for authorization of users to ensure they have the access control rights (permissions) required to do the actions performed.

JAAS is the Java implementation of Pluggable Authentication Module (PAM) framework. And now where is JIM you may ask, which is the reason for this post, and I am getting to it, I promise

Now lets talk about the elephant in the room, 'JAAS config entry not terminated by semi-colon', we will get to JIM later I promise. Here is the link to the full source code of the JaasConfig class. For the lazy ones here is the snippet that we are concerned about:

if (tokenizer.ttype != ';')
  throw new IllegalArgumentException("JAAS config entry not terminated by semi-colon");

From the snippet we can figure out that it will throw an IllegalArgumentException if the config doesn't end with a semi-colon. My first thought after finding this out was okay I need to add a semi-colon then. But where exactly? And how do I add a semi-colon in a yaml file? Here is the yaml snippet, authentication part is what sets the JAAS config

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: kafka-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  tls:
    trustedCertificates: []
  authentication: 
    type: plain
    username: XXXXXXXXXXX
    passwordSecret:
      secretName: confluent-secret
      password: password
  ...

So how do you stick a semi-colon here? You don't. But that's what the error is saying: the error is wrong The error is wrong you say: yes

Some might say that the error is right, but I am not going to argue on the validity of it. My concern is to not get that error anymore, but the message is not helping. The issue was not that it was missing a semi-colon. If you look carefully at the yaml above, you can see the passwordSecret under spec->authentication Thats how you store secret in kubernetes world. Here password value is not the actual password but the label to the field in confluent-secret Secret that stores the password. Let's say you have the password in MY-PASSWORD.txt file What you do is first create a Secret in k8s

kubectl create secret generic confluent-secret --from-file=password=./MY-PASSWORD.txt -n Kafka (here Kafka is my k8s namespace)

Which creates this Secret object in k8s

apiVersion: v1
data:
  password: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
kind: Secret
metadata:
  name: confluent-secret
  namespace: kafka
  ...
type: Opaque
....

It is an object of kind Secret with name confluent-secret and it stores the password from MY-PASSWORD.txt to under data-> password. But you don't just store password in plain text in k8s world, technically it is still plain text but you encode it first. What kubectl does is base64 encodes the password before creating this Secret. And that my friend is the climax of the story

Now before revealing it, I have a promise to fulfill. There is no JIM in this story here only PAM, sorry folks I lied. But if there was ever any JIM, it would be any kafka service that tries to connect to a Kafka bootstrap server like a Kafka Connect instance or a MirroMaker instance. I actually figured out the solution to fix the 'not terminated by semi-colon' error by digging down a closed github bug in strimzi kafka operator which you can find here: not quite the semi-colon bug

Getting back to the climax, base64 encoding is everything that is wrong that causes this error. Now I can imagine Dwights of the world that are windows developer laughing at this saying I knew it all along. Wait dwight, and windows what's going on?

kubectl create secret doesn't work in windows environment

Let that sink in. There I said it, kubectl create secret is broken in windows. It is not encoding the password right. And poor JaasConfig trips up and returns the only error it can think of 'JAAS config entry not terminated by semi-colon'

The fix is simple, just don't use windows and use linux environment instead Or if you have to stick with windows here is the trick that I did to fix this issue

1. Open up a linux terminal, (you can use install WSL in windows
2. base64 encode your password

   echo XXXXXX | base64
   

3. Edit the Secret in windows:

   kubectl edit secret confluent-secret -n kafka
   

4. replace the password value with the encoded string from step 2

And live happily ever after