As Andrej Karpathy, former Director of AI at Tesla, observes: "The hottest new programming language is English" - highlighting how natural language is becoming the primary interface for software creation.

To create an application that stands out in this new era, developers must focus on a new design principle: the META framework - a unified experience that is both powerful and intuitive.

META stands for:

- Manager: AI as an intelligent orchestrator of tools

- Enhancer: AI as a creative collaborator

- Transparency: Inspectable AI actions and reasoning

- Augmentation: Amplifying human capability, not replacing it

Manager: Moving Beyond Static Tools

The next wave of AI applications will not require users to explicitly select every tool for every task. Instead, the application should serve as a high-level manager, capable of interpreting intent.

Karpathy further elaborates on this shift: "We're moving from a world where you tell the computer exactly what to do, to a world where you tell it what you want and it figures out how to do it." This is the essence of AI as Manager. For coding analogy we are moving away from imperative to declarative paradigm or moving from traditional LAMP stack management to Kubernetes orchestration.

Claude Code exemplifies this perfectly. When you ask it to "fix the authentication bug," it doesn't just write code - it autonomously searches the codebase with Grep, reads relevant files, analyzes the issue, writes the fix, runs tests, and even creates a git commit. The user never needs to specify which tools to use; Claude Code interprets intent and orchestrates the right sequence of actions.

Similarly, tools like Cursor and GitHub Copilot Workspace act as intelligent managers, integrating capabilities such as code search, editing, terminal access, and version control into a unified natural language interface. This removes the friction of navigating complex menus, allowing the user to focus on the outcome rather than the process.

Enhancer: Elevating the Creative Process

A key feature of these next-gen apps is their ability to assist with ideation. Rather than just executing commands, the AI should brainstorm, generate concepts, and refine ideas in a collaborative back-and-forth loop.

Greg Brockman, co-founder of OpenAI, describes this as "extending human creativity" - where AI becomes a thought partner that helps explore possibilities faster than ever before. Emad Mostaque, founder of Stability AI, adds: "AI tools should amplify human creativity, not constrain it to predefined templates."

v0 by Vercel demonstrates this enhancement beautifully. Designers can describe a UI component in natural language, and v0 generates multiple React component variations instantly. The designer iterates by refining prompts, transforming from a lone coder into a creative director with AI handling the implementation details.

ChatGPT Canvas and Claude Artifacts similarly enable this creative partnership, providing dedicated workspaces where users and AI collaboratively draft documents, code, and designs. The AI suggests improvements, restructures content, and visualizes ideas - making the creative process faster and more exploratory.

Transparency: Trust Through Inspectable Actions

For users to fully trust these advanced systems, the AI's actions must be inspectable. It is not enough for the system to work like magic; users need the ability to look "under the hood" to see how a result was generated.

Dario Amodei, CEO of Anthropic, emphasizes this principle: "The more powerful AI systems become, the more important interpretability and transparency become. Users should be able to understand and verify what AI is doing." Yann LeCun, Chief AI Scientist at Meta, similarly advocates for "observable reasoning chains" in AI systems.

Claude Code excels here by showing every tool it invokes - whether it's reading a file, running a bash command, or editing code. Users can see the exact grep searches performed, the specific lines of code modified, and the reasoning behind each action. This "human-in-the-loop" approach allows users to verify accuracy and provide necessary feedback.

Devin, the AI software engineer, similarly provides a real-time activity stream showing its planning, web searches, code changes, and terminal commands. Replit Agent displays its thought process and the sequence of files it creates or modifies. This transparency ensures that the human remains in control, able to intervene when needed.

Augmentation: Amplifying Human Capability

Ultimately, the purpose of these applications is not to replace human effort but to augment it. By automating tedious details and bridging the gap between intent and execution, AI empowers users to achieve more than they could alone.

Doug Engelbart, the pioneer of human-computer interaction, envisioned this decades ago with his concept of "augmenting human intellect."

Windsurf by Codeium demonstrates this by combining autonomous "Flows" with human-guided "Cascade" mode. Developers maintain creative control while the AI handles boilerplate code generation, test writing, and documentation. The result is a workflow that is not only more efficient but also deeply gratifying, as developers see their ideas come to life with unprecedented speed and fidelity.

Notion AI similarly augments writing and knowledge work by drafting content, summarizing documents, and extracting insights - letting users focus on strategic thinking rather than mechanical tasks.

The META framework represents a shift from static software to dynamic partnerships, where the "machine" is no longer just a calculator, but a collaborator capable of understanding and executing complex visions. The tools mentioned above - Claude Code, Cursor, v0, ChatGPT Canvas, Devin, Windsurf, and others - are pioneering this new paradigm, setting the standard for what next-generation AI applications should deliver.

References & Further Reading

Thought Leaders:

- [Andrej Karpathy](https://twitter.com/karpathy) - AI researcher, former Tesla AI Director

- [Greg Brockman](https://twitter.com/gdb) - Co-founder and President of OpenAI

- [Dario Amodei](https://www.anthropic.com/research) - CEO of Anthropic

- [Yann LeCun](https://twitter.com/ylecun) - Chief AI Scientist at Meta

- [Emad Mostaque](https://twitter.com/EMostaque) - Founder of Stability AI

Tools & Platforms:

- [Claude Code](https://docs.anthropic.com/en/docs/claude-code) - AI coding assistant

- [Cursor](https://cursor.sh) - AI-powered code editor

- [GitHub Copilot Workspace](https://githubnext.com/projects/copilot-workspace)

- [v0 by Vercel](https://v0.dev) - AI UI generation

- [ChatGPT Canvas](https://openai.com/index/introducing-canvas/)

- [Claude Artifacts](https://support.anthropic.com/en/articles/9487310-what-are-artifacts-and-how-do-i-use-them)

- [Devin](https://www.cognition-labs.com/introducing-devin) - AI software engineer

- [Replit Agent](https://replit.com/ai) - AI coding agent

- [Windsurf](https://codeium.com/windsurf) - AI development environment

- [Notion AI](https://www.notion.so/product/ai) - AI-powered workspace

In Kubernetes you can define CPU and memory (RAM) requirements for each container in a Pod. Resource Requests indicate the minimum amount of a resource that a container needs to run properly. The kube-scheduler uses this information to assign Pods to suitable nodes. Resource Limits, set by users, these limits prevent containers from consuming more resources than allocated. They are enforced to maintain fairness and resource efficiency within the cluster. Read more in kubernetes docs

If a node has surplus resources, a container might use more than its requested amount, subject to its defined limit. Consider a container with a 4GiB memory limit. If the container attempts to exceed this limit, the kubelet intervenes. This can result in process terminations within the container, often manifested as Out of Memory (OOM) errors.

However Kubernetes also includes eviction policies managed by the kubelet. These policies trigger Pod eviction if the node's available resources fall below specified thresholds. For instance, if the node's available memory dips below a defined threshold, the kubelet may evict Pods to reclaim resources, even though they are within their resource limit which brings us to the topic of this post.

Challenges of Memory Management Pre-AKS 1.29:

In previous versions of AKS, memory management posed challenges due to a higher eviction threshold and a complex reservation formula. These factors often led to inefficient resource allocation and pod instability.

• Higher Eviction Thresholds: Nodes would start evicting pods when the available memory dipped below 750 MiB, a mechanism to prevent system instability due to low memory.

• Complex Memory Reservation: A regressive formula determined the amount of memory reserved for system processes and Kubernetes components, which impacted the allocatable memory for pods.

Exploring the Changes in AKS 1.29:

AKS 1.29 introduced significant changes in memory management, optimizing how resources are allocated and used.

• Reduced Eviction Threshold: The new threshold is set at memory.available<100Mi, allowing for more efficient resource allocation and better pod stability.

• Simplified Memory Reservation: The new formula for kube-reserved memory is the lesser of 20MB * Max Pods + 50MB or 25% of the total system memory. This change aims to simplify and potentially increase the allocatable memory.

Example for an 8GB VM with 30 Max Pods:

Previous AKS versions

• Eviction Threshold: memory.available<750Mi

• The regressive formula typically reserved:

  • 25% of the first 4GB = 1GB (1024 MiB)

  • 20% of the next 4GB (4GB to 8GB) = 0.8GB (819.2 MiB)

• Total Reserved: 1024 MiB + 819.2 MiB = 1843.2 MiB

• Allocatable Memory: 8GB - 1.8432GB - 0.75GB ≈ 5.41GB

AKS 1.29

• Eviction Threshold: memory.available<100Mi

• Memory Reservation: 20MB * 30 + 50MB = 650MB (approximately 0.65GB)

• Allocatable Memory: 8GB - 0.65GB - 0.1GB = 7.25GB

Example for a 4GB VM with 70 Max Pods:

Previous AKS versions

• Eviction Threshold: memory.available<750Mi

• The regressive formula for a 4GB node typically reserved:

  • 25% of the first 4GB = 1GB (1024 MiB)

• Total Reserved: 1024 MiB

• Allocatable Memory: 4GB - 1GB - 0.75GB = 2.25GB

AKS 1.29

• Eviction Threshold: memory.available<100Mi

• Memory Reservation: Capped at 25% * 4GB = 1000MB (1GB), as it's lesser than 20MB * 70 + 50MB = 1450MB.

• Allocatable Memory: 4GB - 1GB - 0.1GB = 2.9GB

For a detailed understanding of resource reservations in AKS, refer to Microsoft's official documentation.

Conclusion

The transition from the regressive formula in previous versions to the new calculation method in AKS 1.29 can result in different amounts of memory being reserved for kubelet and system processes. In AKS 1.29, the changes generally lead to a more predictable and potentially higher percentage of allocatable memory, particularly for larger VM sizes. The reduced eviction threshold in AKS 1.29 also signifies a more proactive approach to managing memory availability, enhancing node stability.

To effectively manage memory in AKS 1.29, stay updated with new changes, apply best practices, and continually adjust to your application needs. Efficient memory management is an ongoing journey that enhances your AKS cluster's performance.

Introduction: The Significance of DNS in Kubernetes

In Kubernetes, DNS isn't just about resolving domain names. It's the heart of service discovery. When you create a service in Kubernetes, it's automatically given a DNS name, allowing other pods to discover and communicate with it without needing to know its IP address.

The Mechanics of Kubernetes DNS

• Service and Pod DNS Records: Every service and pod in Kubernetes is automatically assigned a DNS record. This ensures consistent communication paths. For example, a service named "my-service" in the "default" namespace would have a DNS name like my-service.default.svc.cluster-domain.example. Learn more from the official Kubernetes documentation.

• Namespaces and DNS: The namespace of a pod can influence DNS query results. A pod in the "test" namespace querying for "data" in the "prod" namespace would use data.prod.

• Headless Services: These are special services without a cluster IP. Their DNS name resolves to the IPs of all selected pods, rather than a single IP.

  Example: Imagine you have a headless service named "database-service" in the "backend" namespace, and it selects three pods with IPs 10.0.1.1, 10.0.1.2, and 10.0.1.3. When another pod tries to resolve the DNS name database-service.backend.svc.cluster-domain.example, it will get back a list of all three IPs (10.0.1.1, 10.0.1.2, 10.0.1.3) instead of a single service IP. This is particularly useful for applications that need to discover their peers like in distributed databases. Read more about headless services.

• SRV Records: SRV records are a type of DNS record used in Kubernetes and other systems to describe services offered by a domain. In Kubernetes, for services with named ports, SRV records are created to aid in service discovery across varying ports.

  • Structure: An SRV record typically has the format: _service._protocol.name. The _service and _protocol are prefixed with underscores, and the name is the domain name where the service is offered.

  • Example: If you have a service named "web-app" in the "default" namespace with a named port "http" at port number 8080, Kubernetes would create an SRV record like _http._tcp.web-app.default.svc.cluster-domain.example. This SRV record points to the port 8080 of the pods selected by the "web-app" service.

  • Usage: SRV records are particularly useful when you need to discover not just the IP address of a service but also the port number on which the service is running. This is common in systems where services might be running on non-standard ports.

  • Benefits in Kubernetes: In dynamic environments like Kubernetes, where pods can come and go and might be exposed on different ports, SRV records provide a way to dynamically discover services without hardcoding IP addresses and port numbers.

• Pod's DNS Policy: This determines how DNS queries from the pod are handled. The "ClusterFirst" policy, for instance, ensures non-cluster queries are forwarded to an upstream nameserver.

• Custom DNS Configurations: The dnsConfig field in a pod spec allows users to customize DNS settings, offering flexibility in service discovery.

• Windows Nodes and DNS: DNS resolution on Windows nodes has its quirks. For instance, all names with a dot (.) are treated as FQDNs. The Resolve-DNSName PowerShell cmdlet is recommended for DNS resolutions on Windows.

Advanced DNS Configurations in Kubernetes

• Pod's DNS Config: Provides granular control over DNS settings for a Pod. It allows specifying custom nameservers, search domains, and other DNS settings.

• DNS Resolution on Windows Nodes: Windows treats all names with a dot (.) as FQDN and skips FQDN resolution. It's recommended to use the Resolve-DNSName PowerShell cmdlet for DNS resolutions on Windows.

• DNS Search Domain List Limits: Kubernetes allows up to 32 search domains. The total length of all search domains should not exceed 2048 characters.

• Pod's setHostnameAsFQDN Field: When set to true, the kubelet writes the Pod's FQDN into the hostname for that Pod's namespace. This means both hostname and hostname --fqdn commands return the Pod's FQDN.

  Example: Suppose you have a Pod with the name "web-app" in the "production" namespace, and its FQDN is web-app.production.svc.cluster-domain.example. If setHostnameAsFQDN is set to true for this Pod, running the hostname command inside the Pod would return web-app.production.svc.cluster-domain.example instead of just web-app. Similarly, the hostname --fqdn command would also return web-app.production.svc.cluster-domain.example.

Why Does This Matter?

Understanding DNS in Kubernetes is crucial because it simplifies service discovery. Without DNS, pods would need to track the IP addresses of other services, a challenge given the ephemeral nature of pods. With DNS, services are easily discoverable, regardless of their current IP address.

Key Takeaways

• DNS in Kubernetes is central to service discovery.

• Kubernetes offers flexible DNS configurations to suit various needs.

• As applications grow and span multiple clusters, mastering Kubernetes DNS becomes essential.

Further Reading

For those keen on diving deeper, the official Kubernetes documentation provides comprehensive insights into DNS configurations and behaviors.

By understanding the nuances of DNS in Kubernetes, developers and administrators can build more resilient, scalable, and efficient applications. It's not just about name resolution; it's about seamless communication in a dynamic environment.

One popular set of connectors for Kafka are the Debezium connectors. These connectors are specifically designed for change data capture (CDC) scenarios, which involve capturing changes in a database and publishing them to Kafka topics. The Debezium connectors are available for various databases, including MySQL, PostgreSQL, Oracle, and SQL Server.

However, when setting up the Debezium connectors, there's an important consideration to keep in mind: the configuration of the Kafka topic used for storing the history of database changes. This topic is set using the database.history.kafka.topic parameter, and it's critical that this topic has infinite retention and a single partition. https://debezium.io/documentation/reference/1.9/install.html#_configuring_debezium_topics
Starting 2.0 debezium changed the paramater name to schema.history.internal.kafka.topic so if you are reading this wondering why you don't have that parameter you might be implementing the 2.x version of the connector

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: sql-connector-001
  labels:
    strimzi.io/cluster: debezium-connect-cluster
spec:
  class: io.debezium.connector.sqlserver.SqlServerConnector
  tasksMax: 1
  config:
    database.history.kafka.topic: schema-changes-topic
    ...

It's worth noting that Confluent Kafka, the most popular distribution of Kafka, sets its topics with only one week of retention by default. This means that if you're using Confluent Kafka, you'll need to override this default behavior for the history topic.

If you don't set the retention policy for the history topic correctly, you may run into issues with the Debezium connector not working properly. One common error message is "The db history topic or its content is fully or partially missing. Please check the database history topic configuration and re-execute the snapshot."

If you encounter this error, you'll need to take corrective action, which may involve nuking the topics, setting proper configurations, and starting from scratch. Alternatively, you could rename the connector to another name, which is a temporary fix that will only last another week.

In summary, when setting up Kafka Connect with Debezium Connectors, it's important to pay close attention to the configuration of the history topic. Make sure that it has infinite retention and a single partition to avoid issues with the Debezium connector not working properly. Open source no-code tools like Kafka Connect make it easier than ever to build a scalable, reliable, and distributed data pipeline. However, it's essential to be careful about configuring our connectors, which can often go wrong without us realizing it. By setting the right configuration parameters for the database history topic, we can ensure that our Kafka Connect pipeline runs smoothly and without any hiccups. I hope this blog post helps you avoid any potential issues and that you are reading this before you release your connectors to production.

Confluent Cloud offers a managed solution for CDC through its connectors, but the pricing model would not be favorable for our setup with hundreds of connectors. So, we had to come up with a solution that would work with our unique architecture and budget constraints. We ended up deploying self-managed Kafka Connect Instances in Azure Kubernetes Cluster

Here is the overall streaming architecture:

1. Some service updates the SQL database

2. The debezium connector, registered in Kafka Connect Instance deployed in Kubernetes environment, captures the changes

3. Debezium worker sends the events to confluent Kafka raw topic

4. The event processor processes the event and pushes that to another final topic for consumption

Why Kafka?

Kafka is a powerful distributed streaming platform that enables users to build scalable and reliable data pipelines and applications. It is based on the publish-subscribe messaging model and allows producers to publish messages to a Kafka topic, while consumers subscribe to the topic to receive messages. With its high throughput, low latency, and fault-tolerant architecture, Kafka is widely used for building real-time data streaming pipelines.

One popular use case for Kafka is Change Data Capture (CDC), which is the process of detecting and capturing changes to data sources and making those changes available in real-time to other systems or applications. This is where Debezium comes in. Debezium is an open source distributed platform that provides change data capture capabilities for Kafka. With Debezium connectors, Kafka can capture data changes from various data sources such as MySQL, PostgreSQL, MongoDB, and Oracle, among others (MSSQL in our case).

Deploying Debezium Connectors

Deploying Debezium connectors on a self-managed Kubernetes cluster was not an easy endeavor. Deploying a self-managed service does require more effort in terms of setting up and maintaining the cluster. Specially for someone who had not deployed any Kubernetes service before! But open source came to the rescue.

To deploy the Debezium connector on a self-managed AKS cluster we used the open-source Strimzi operator. Strimzi provides an easy and scalable way to manage Kafka and Kafka Connect clusters on Kubernetes. It allows for the deployment of Kafka brokers, ZooKeeper nodes, Kafka Connect workers, and Debezium connectors as Kubernetes resources. Strimzi also enables the configuration and scaling of Kafka and Kafka Connect resources using Kubernetes Custom Resource Definitions (CRDs). For our use case, we only needed to deploy Kafka Connect and Debezium Connectors

Kafka Connect Yaml

To deploy a Kubernetes service, you need to provide a YAML file. For production use cases I would not recommend to kubectl directly but use something like Helm charts (we use helm charts for all our kubernetes deployment). With that out of the way, lets dig into Kafka Connect

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: debezium-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  image: {{ custom-image/kafka-connect-with-x-connector }}
  replicas: 2
  bootstrapServers: {{ kafka-cluster-kafka-bootstrap:9092 }}
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
    #secret config
    config.providers: secrets
    config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider

The ones with {{ }} double braces need to be updated

First is the image, for KafkaConnect. By default base Kafka Connect image doesnot include any connectors. You need to add each one you need to /opt/kafka/plugins/ directory. Please follow deploying debezium with kafka connector for more details. It is quite old but is full of riches. I might write a blog on just the building image part later. The second variable is the bootstrapServers which is where the kafka cluster is. In our case it is the confluent but for local testing, it is {{ your kafka cluster name}}-bootstrap-server:9092. So if you kafka cluster is named "kafka-cluster" it would be kafka-cluster-kafka-bootstrap:9092

To scale all you have to do is increase the replicas value, in the example is 2. Since KafkaConnect is a custom type, provided by Strimzi CRDs, strimzi operator takes care of the rest of the deployment for you. In background, strimzi operator creates one Deployment with RollingUpdate Strategy with x replicas and Kafka Connect environment variables

Installing Kafka Connect

Assuming you have a Kubernetes environment and helm you can apply the yaml file to a specific namespace like 'kafka'. Prerequisite for setting your local environment:

1. Minikube : Minikube is local Kubernetes, focusing on making it easy to learn and develop for Kubernetes

2. Helm : Helm helps you manage Kubernetes applications

3. Hyper-V : Enable Hyper-V on your windows machine, if you are using windows

# create kafka namespace
kubectl create ns kafka
# add strimzi repo to helm chart 
helm repo add strimzi https://strimzi.io/charts/
# install strimzi operator
helm install strimzi strimzi/strimzi-kafka-operator --namespace kafka
# deploy kafka connect
kubectl apply -f kafka-connect.yaml --namespace kafka # yaml file shown above

Deploying Debezium Kafka Connector

Next is to deploy the debezium connector. And here is the sample yaml file

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: sql-connector-001
  labels:
    strimzi.io/cluster: debezium-connect-cluster
spec:
  class: io.debezium.connector.sqlserver.SqlServerConnector
  tasksMax: 1
  config:
    database.server.name: {{ server-name }}
    database.hostname: {{ host-name }}
    database.dbname: {{ $dbname | lower | quote }}
    database.user: "${secrets:kafka/dbsecret:username}"
    database.password: "${secrets:kafka/dbsecret:password}"
    #history events for database changes
    database.history.kafka.bootstrap.servers: {{ kafka-cluster-kafka-bootstrap:9092}}
    database.history.kafka.topic: schema-changes-topic
    #included tables
    table.include.list: "dbo.table_1"
    column.include.list: "dbo.table_1.id, dbo.table_1.name"

# secret file
apiVersion: v1
kind: Secret
metadata:
  name: dbsecret
  namespace: kafka
type: Opaque
data:
  username: {{ b64encoded username}}
  password: {{ b64encoded password}}

# applying secret
kubectl apply -f dbsecret.yaml 
# apply connector
kubectl apply -f sql-connector-001.yaml -n kafka

We are using KubernetesSecretConfigProvider in the KafkaConnect chart above. This allows to dynamically load db secrets in kafka connector above with properties:

database.user: "${secrets:kafka/dbsecret:username}"
database.password: "${secrets:kafka/dbsecret:password}"

This allows to add secrets or update them without restarting the Kafka Connect Instance as Debezium connectors are barely configurations attached to KafkaConnect

Conclusion

In conclusion, deploying Kafka Connect and Debezium Kafka Connector can be a great solution for capturing database changes and streaming them to Kafka. With the help of Kubernetes and Helm, deploying and scaling the solution is made much easier. While there are many ways to deploy Kafka Connect and Debezium Kafka Connector, using Strimzi can simplify the process and provide additional features like custom resource definitions for Kafka Connect and Kafka Connector. Additionally, using a fully managed Kafka service like Confluent Cloud can reduce the operational burden and allow you to focus on building your streaming application. With the right setup, deploying Kafka Connect and Debezium Kafka Connector can help you build real-time data pipelines and unlock valuable insights from your data.

I only scratched the surface of deploying Kafka connect and connectors. Please leave comments if you want to learn more about something I failed to mention

const axios = require('axios');
//DONOT use in production without error handling
const {data} = await axios.get('some-service');

Assuming you have some knowledge of Kubernetes(K8s), you know that you need a deployment and a service

To give basic info here is some description of deployment and service

Deployment

A Deployment provides declarative updates for Pods and ReplicaSets.

You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate. You can define Deployments to create new ReplicaSets or to remove existing Deployments and adopt all their resources with new Deployments.

You don't want to be creating pods manually or even replica sets but use deployment which handles rollout/rollback of your pods(not useful if you have only one replica as in the example below but good practice nonetheless)

Check out the K8s deployment official documentation for more info

And here is a sample YAML file for a deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: some-service-dpl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: some-service-dpl
  template:
    metadata:
      labels:
        app: some-service-dpl
    spec:
      containers:
      - name: some-service-dpl
        image: yourdockerregistry/some-service-dpl-image
        ports:
        - containerPort: 8080

Service

A Kubernetes Service is an abstraction that defines a logical set of Pods running somewhere in your cluster, that all provide the same functionality. When created, each Service is assigned a unique IP address (also called clusterIP). This address is tied to the lifespan of the Service, and will not change while the Service is alive.

Service by default is of type ClusterIP which exposes the service on cluster-internal IP which will not change while the service is alive. But it will if it is killed and a new one spins up so an IP address is not a full proof way to talk to a service

Check out the K8s service official documentation for more info

Here is YAML file for a service that is exposing the previous deployment

kind: Service
apiVersion: v1
metadata:
  name: some-service-svc
spec:
  selector:
    app: some-service-svc
  ports:
    - name: some-service-api
      targetPort: 8080
      port: 8080
      protocol: TCP

So how do we talk to this service backend? The whole point of creating the service was to get a stable IP for the pods managed by replica sets managed by some-service-dpl. So we don't want to talk to the pods directly

We also don't want to talk to the service directly using an IP address which can be found by this simple kubectl command. Especially not if we are deploying both services at the same time and there is no way to know the IP address beforehand

kubectl get svc some-service-svc
NAME               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)     AGE
some-service-svc   ClusterIP   10.0.15.149   <none>        8080/TCP    21s

What we do want to use is the name of the service itself and that's it!

This works because K8s uses DNS for services and pods and internally resolves http://some-service-svc:8080 to http://10.0.15.149:8080. Below you can see the fully qualified DNS name for a service

 {{service_name}}.{{namespace}}.svc.cluster.local

const axios = require('axios');
//DONOT use in production without error handling
const {data} = await axios.get('http://some-service-svc:8080');
//if the service was in some other namespace 
const {data} = await axios.get('http://some-service-svc.otherNamespace.svc.cluster.local:8080');

Just make sure that you keep track of the name of your service or add the service and port exposed by the service in your environment from config variables (from Helm values if you use Helm for example)

JAAS stands for Java Authentication and Authorization Service. And per Oracle JAAS reference guide it can be used for two purposes:

1. for authentication of users, to reliably and securely determine who is currently executing Java code, regardless of whether the code is running as an application, an applet, a bean, or a servlet; and
2. for authorization of users to ensure they have the access control rights (permissions) required to do the actions performed.

JAAS is the Java implementation of Pluggable Authentication Module (PAM) framework. And now where is JIM you may ask, which is the reason for this post, and I am getting to it, I promise

Now lets talk about the elephant in the room, 'JAAS config entry not terminated by semi-colon', we will get to JIM later I promise. Here is the link to the full source code of the JaasConfig class. For the lazy ones here is the snippet that we are concerned about:

if (tokenizer.ttype != ';')
  throw new IllegalArgumentException("JAAS config entry not terminated by semi-colon");

From the snippet we can figure out that it will throw an IllegalArgumentException if the config doesn't end with a semi-colon. My first thought after finding this out was okay I need to add a semi-colon then. But where exactly? And how do I add a semi-colon in a yaml file? Here is the yaml snippet, authentication part is what sets the JAAS config

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: kafka-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  tls:
    trustedCertificates: []
  authentication: 
    type: plain
    username: XXXXXXXXXXX
    passwordSecret:
      secretName: confluent-secret
      password: password
  ...

So how do you stick a semi-colon here? You don't. But that's what the error is saying: the error is wrong The error is wrong you say: yes

Some might say that the error is right, but I am not going to argue on the validity of it. My concern is to not get that error anymore, but the message is not helping. The issue was not that it was missing a semi-colon. If you look carefully at the yaml above, you can see the passwordSecret under spec->authentication Thats how you store secret in kubernetes world. Here password value is not the actual password but the label to the field in confluent-secret Secret that stores the password. Let's say you have the password in MY-PASSWORD.txt file What you do is first create a Secret in k8s

kubectl create secret generic confluent-secret --from-file=password=./MY-PASSWORD.txt -n Kafka (here Kafka is my k8s namespace)

Which creates this Secret object in k8s

apiVersion: v1
data:
  password: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
kind: Secret
metadata:
  name: confluent-secret
  namespace: kafka
  ...
type: Opaque
....

It is an object of kind Secret with name confluent-secret and it stores the password from MY-PASSWORD.txt to under data-> password. But you don't just store password in plain text in k8s world, technically it is still plain text but you encode it first. What kubectl does is base64 encodes the password before creating this Secret. And that my friend is the climax of the story

Now before revealing it, I have a promise to fulfill. There is no JIM in this story here only PAM, sorry folks I lied. But if there was ever any JIM, it would be any kafka service that tries to connect to a Kafka bootstrap server like a Kafka Connect instance or a MirroMaker instance. I actually figured out the solution to fix the 'not terminated by semi-colon' error by digging down a closed github bug in strimzi kafka operator which you can find here: not quite the semi-colon bug

Getting back to the climax, base64 encoding is everything that is wrong that causes this error. Now I can imagine Dwights of the world that are windows developer laughing at this saying I knew it all along. Wait dwight, and windows what's going on?

kubectl create secret doesn't work in windows environment

Let that sink in. There I said it, kubectl create secret is broken in windows. It is not encoding the password right. And poor JaasConfig trips up and returns the only error it can think of 'JAAS config entry not terminated by semi-colon'

The fix is simple, just don't use windows and use linux environment instead Or if you have to stick with windows here is the trick that I did to fix this issue

1. Open up a linux terminal, (you can use install WSL in windows
2. base64 encode your password

   echo XXXXXX | base64
   

3. Edit the Secret in windows:

   kubectl edit secret confluent-secret -n kafka
   

4. replace the password value with the encoded string from step 2

And live happily ever after

Recently I had to deal with reading/writing in-memory objects using blob storage. There was not much literature regarding doing this efficiently using the API. So I began the journey to find one, or rather stumbled into one

Here was the first pass at reading from blob storage:

using Newtonsoft.Json;

public async Task<T> ReadDataAsync (string blobId, CancellationToken c)
{
    var client = containerClient.GetBlobClient(blobId);
    using var stream = await client.OpenReadAsync(null, c);
    using var streamReader = new StreamReader(stream);
    using var json = new JsonTextReader(streamReader);
    return JsonSerializer.CreateDefault().Deserialize<T>(json);
}

This is okay performance wise, StreamReader and JsonTextReader should optimize memory usage but the code is too verbose. We can do better.

Here we go with the next pass on reading blob storage

using System.Text.Json;

public async Task<T> ReadDataAsync (string blobId, CancellationToken c)
{
    var client = containerClient.GetBlobClient(blobId);
    using var stream = await client.OpenReadAsync(null, c);
    return await JsonSerializer.DeserializeAsync<T>(stream, null, c);
}

This is much better, now we are using the new module System.Text.Json which helps tidy up the code a little better, reducing 3 lines from the first pass to just 1 line. Performance wise it is the same as before

Now lets see the first pass at writing to blob storage

public async Task WriteAsync (SomeData data, string blobId)
{
    var client = containerClient.GetBlobClient(blobId);
    await using var ms = new MemoryStream();
    var json = JsonConvert.SerializeObject(data);
    var writer = new StreamWriter(ms);
    await writer.WriteAsync(json);
    await writer.FlushAsync();
    ms.Position = 0;
    await client.UploadAsync(ms);
}

This is sub par to say the least as put in a PR comment. That's exactly what happened to me. I wrote this code and the PR comment recommended me to optimize it! Here is why this aprroach is not so good, first the data is serialized to json, next the serialized json is written into stream. And finally after resetting the stream position to 0 it is uploaded to blob storage. This is 3X the memory of the original data. We can definitely do better

Now the second pass at writing to blob storage

using System.Text.Json;

public async Task WriteAsync<T> (T data, string blobId)
{
    var client = containerClient.GetBlobClient(blobId);
    await using var ms = new MemoryStream();
    await JsonSerializer.SerializeAsync(ms, data);
    ms.Position = 0;
    await client.UploadAsync(ms);
}

I would say some improvement compared to the first pass, instead of 3x memory now it is down to 2x memory, but we can do better. With that PR review encouragement here is the final pass on writing to blob storage. Big thanks to this stackoverflow answer that mad use of Azure.Storage.Blobs.Specialized https://stackoverflow.com/questions/62279770/how-to-upload-stream-with-azure-storage-blobs

using System.Text.Json;
using Azure.Storage.Blobs.Specialized;

public async Task WriteAsync<T> (T data, string blobId)
{
    var client = containerClient.GetBlockBlobClient(blobId);
    var ms = await client.OpenWriteAsync(true);
    await JsonSerializer.SerializeAsync(ms, data);
}

The only difference between this and the previous code is that we are using a new api of GetBlockBlobClient instead of GetBlobClient from pass 2. This api gives us stream to write data directly into. And voila just 1x memory and it is uploaded to blob storage

With that we are finally at the most performant code that I can think of ! Let me know if there is something even better than this

Here is the final code for both read and write

using System.Text.Json;
using Azure.Storage.Blobs.Specialized;

public async Task<T> ReadDataAsync (string blobId, CancellationToken c)
{
    var client = containerClient.GetBlobClient(blobId);
    using var stream = await client.OpenReadAsync(null, c);
    return await JsonSerializer.DeserializeAsync<T>(stream, null, c);
}

public async Task WriteAsync<T> (T data, string blobId)
{
    var client = containerClient.GetBlockBlobClient(blobId);
    var ms = await client.OpenWriteAsync(true);
    await JsonSerializer.SerializeAsync(ms, data);
}