Data Streaming Repository

Kafka in action and further optional research links:

• Confluent kafka python library doc
• Kafka command line tools
• Kafka topics, producers and consumers
• Why does kafka need zookeeper?
• IBM Kafka architecture
• Setup Kafka on Kubernetes

The Number of Partitions

This Confluent article indicates how to plan the numnber of partitions.

• Number of partitions

Best pratices and anti patterns

Kafka can create topics automaticaly, but it is not recommended. The best practice is create them manually. It is possible create topic using script code, like sh or even terraform and ansible.

Terraform can control the topic creation and also the ACL management.

Installed Packages Locally

Distributed Systems Reading Group

• MIT group link

Compression

An article about the pros and cons, here is the link

Here Is a question: what is network overhead?

Kafka Local Cluster Config

Kind is used to create the local kafka cluster (using the kafka on kubernetes model).Kind command to create the cluster:

$ kind create cluster --config YAML_FILE --name CLUSTER_NAME

The local cluster is configured with 3 nodes (1 control plane and two worker nodes)