At Capital One we process billions of events everyday without any data loss and following exactly once processing semantics as well, some of these events are related to fraud detection, customer communications, payments processing...etc. Most of the times it is necessary for us to process these events in real time so we ReImagined auto scaling Kafka consumers to offer beyond predictable performance even during high volumes of inbound events as well as reduce resource utilization whenever possible. We are also making sure that there is no data loss and exactly once event processing when we are offering high tps. Unlike traditional applications, auto scaling an event driven real time stream consumer applications requires much more parameters and edge cases to consider. My talk will be covering all of these parameters, metrics, edge cases and steps to autoscale consumers safely..etc.
Here are highlights of my talk: Streaming architecture high level overview which is supporting processing billions of events Our journey to ReImagine Autoscaling of streams Autoscaling benefits and how to Autoscale consumers to offer beyond predictable performance Lessons learned along the way and best practices In addition to this, giving an overview on observability and monitoring streams.