Kafka is a cornerstone of LinkedIn’s data infrastructure. It is the replication stream for Espresso; the message transport for Brooklin (our change capture system), Samza and Venice (our derived data serving store). We describe Kafka’s fundamental roles: data storage, movement, processing and analysis; and discuss the requirements to serve these data systems, issues that we hit and how we addressed them.
|Joel Koshy, Senior Staff Engineer, LinkedIn|
|Kartik Paramasivam, Director of Engineering, Streams Infrastructure, LinkedIn|