Kafka Connect #

Framework, um Kafka mit externen Systemen wie Datenbanken, Key-Value Stores, Suchindizes oder Dateisysteme zu verbinden.

Core concepts #

Connectors: The high level abstraction that coordinates data streaming by managing tasks
- Connector instance: Logical job that is responsible for managing the copying of data between Kafka and another system
  - Source Connector: A source connector ingests entire databases and streams table updates to Kafka topics. It can also collect metrics from all of your application servers into Kafka topics, making the data available for stream processing with low latency.
  - Sink Connector: A sink connector delivers data from Kafka topics into secondary indexes such as Elasticsearch or batch systems such as Hadoop for offline analysis.
- Connector plugin: Container for all of the classes that implement or are used by a connector
Tasks: the implementation of how data is copied to or from Kafka. Task state is stored in Kafka in special topics config.storage.topic and status.storage.topic and managed by the associated connector
Workers: the running processes that execute connectors and tasks
- Standalone worker
- Distributed worker: Share the same group.id. Under the covers, connect workers are using consumer groups to coordinate and rebalance
Converters: the code used to translate data between Connect and the system sending or receiving data
Transforms: simple logic to alter each message produced by or sent to a connector (vgl. Kafka Connect Transformations)