Technology Sharing

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

1. Controller Broker

There is a Controller Broker in the Kafka cluster, which is responsible for metadata management and coordination.

Kafka uses Zookeeper as a storage and management tool for cluster metadata. Zookeeper saves cluster status information, including all topics, partitions, leaders, and replicas.

When the cluster status changes, Controller Broker writes the change information to Zookeeper.

When the current Controller Broker fails, other Brokers in the Kafka cluster will detect this and conduct elections through Zookeeper.

After a Broker successfully runs for the new Controller Broker, it reads the latest cluster metadata from Zookeeper.

Guarantee mechanism

• Persistent metadata: Zookeeper persistently stores the metadata of the cluster to ensure that the latest status information can be obtained at any time.
• Heartbeat mechanism: Broker and Zookeeper maintain connection through the heartbeat mechanism to quickly detect Controller Broker failures.
• Election Mechanism: Through the Zookeeper election mechanism, a new Controller Broker can be quickly elected and metadata can be synchronized from Zookeeper.

2. Component Architecture

1. Log Manager

LogManagerMainly responsible for managing the storage and retrieval of Kafka logs.

For example: The producer sends the message to Leader Broker 1 of Partition 0.LogManagerWrite the message to the log file of Partition0 on Broker1.

2. Replication Manager

ReplicationManagerMainly responsible for managing the replication and synchronization of partition data.

The synchronization between the leader and followers of each partition is performed independently. In other words, each partition has its own synchronization process and does not depend on other partitions.

Although the synchronization process of each partition is independent, each Broker will start a corresponding replication thread for each partition it manages (whether Leader or Follower), which is responsible for handling specific synchronization tasks.

for example:ReplicationManagerOn Broker1, the newly written message is pushed to Follower Broker2 and Broker3 of Partition0.ReplicationManagerThe replication request received from Broker1 is processed on Broker2 and Broker3, writing the message to their respective log files.

3. SocketServer

SocketServerIt is a component in Kafka Broker that is responsible for handling network connections and I/O operations. It is responsible for accepting connection requests from clients and other Brokers and assigning a thread to each connection for processing.

4. NetworkServer

NetworkServerIt is a core part of Kafka's network communication framework, responsible for managing and scheduling network requests. It uses NIO (non-blocking I/O) to handle high-concurrency network connections.

5. ZKClient

Component that communicates with Zookeeper.