While the core of “Kafka” often refers to the low-latency storage system, the streaming platform includes other important components. First is Kafka Connect, an integration system that allows horizontally-scalable connectors to many important systems. This includes change data capture (CDC) connectors, cluster-to-cluster replication (MirrorMaker), and ability to write data to downstream systems such as lakehouses (Apache Iceberg), lakes (Avro or Parquet files on object storage), as well as databases such as BigQuery . Second, the Kafka projects ship with a set of powerful clients, including administrative command line clients for manipulating clusters and topics as well as high performance client libraries for reading and writing data.   

Historically, data processing was handled with periodic batch jobs, where raw data was first stored and later processed at arbitrary intervals. For example, a retail company might wait until the end of the day to analyze sales data. One of the limitations of batch processing is that it’s not real time. Increasingly, organizations and data scientists want to analyze data in real time as it is generated to make timely business decisions and power real-time AI models.This is where event streaming comes in. Event streaming is the process of continuously processing infinite streams of events, as they are created. This captures the time-value of data and enables push-based applications that take action whenever something interesting happens. For data scientists, this means the ability to perform real-time feature engineering and deliver low-latency predictions.