Project Metamorphosis: Unveiling the next-gen event streaming platformLearn More

State Unemployment Systems Are Overwhelmed. Mainframe Offload with Apache Kafka Can Relieve the Pressure.

COVID-19 has created extraordinary challenges in virtually every industry. While much attention has been focused on the pandemic’s effect on travel, hospitality, airline, and healthcare industries, its impact on state and federal agencies has been just as profound. With millions of Americans suddenly out of work, state unemployment systems are struggling and many are buckling under the surge in demand leading to system failures and slowdowns.

In response to the crisis, the governor of New Jersey issued a plea for developers who know COBOL, the 60-year-old language used to develop the state’s unemployment system. While aging code bases are certainly part of the problem, a major contributing factor is the data and where it lives: locked away on mainframes. Efforts to scale overwhelmed unemployment systems are running headlong into the difficulties associated with getting data out of the mainframe. New Jersey and other states have found that as employees have relocated or retired over the decades, the tribal knowledge needed to maintain the systems—and access the data—has gone with them.

Any organization that has monolithic systems and brittle, aging code is susceptible to being hit with a new and unprecedented wave of demand. The pandemic is shining a bright light on the problem, as the failure of unemployment agencies to meet that demand is compounding the difficulties of those who have just lost their jobs.

Apache Kafka for mainframe offload

The good news is that this problem is not intractable. IT groups across many industries have begun using event streaming with Apache Kafka® to free data from its mainframe quarantine. As a result, these groups are able to make use of the data in modern, scalable applications that can readily be updated and adapted to meeting evolving demands and technology requirements.

With Kafka acting as a buffer between the mainframe and newer services, many of these mainframe offload initiatives can be completed relatively inexpensively without rewriting code or calling for a regiment of heroic COBOL developers to come out of retirement. On the scalability front, Kafka can persist data and play it back later, so when a spike in demand causes more data to flow in than a backend system can handle at once, Kafka can absorb that data for as long as needed and feed it steadily into the backend system without overloading it.

While Kafka by itself can address many of the mainframe-centric technical challenges facing unemployment agencies and other organizations struggling to keep up with the demands the pandemic has imposed upon them, clearing operational hurdles and meeting other environment-specific needs may require additional capabilities. Many agencies, for example, lack the personnel and resources to deploy and manage Kafka clusters on their own.

For these organizations, and others that have adopted a cloud-first mindset, having a fully managed cloud-native Kafka service such as Confluent Cloud can not only serve as a springboard for rapid deployment but also reduce maintenance overhead and manpower requirements in the long term. For environments running Kubernetes, Confluent Platform automates the deployment of Kafka on this runtime via the Confluent Operator, enabling a team to set up a production-ready event streaming platform in minutes, on premises, or in the cloud. In any environment, Control Center makes it easy to manage and monitor Kafka, enabling teams to track the health of their clusters and identify potential problem areas during peak loads.

Further, because the data stored on agency mainframes often includes sensitive personal information, it’s important to control access to this data as it is streamed—and Role-Based Access Control with Confluent Platform enables organizations to set up rules that do just that. Looking forward, past the immediate needs of the present situation, there are opportunities to use the newly available data in innovative ways, for example by using ksqlDB to enrich it and curate it midstream and in real time with simple SQL statements, in order to provide new and better services to the citizens being served.

Learn more about event streaming

If you want to learn more about making data from your mainframe systems available in a modern, event-driven architecture, this blog post that covers the entire journey is a good place to start. Plenty of real-world use cases are available as well. Alight Solutions, for example, lowered costs by offloading work and reducing demand on mainframe systems. RBC used Confluent Platform to rescue data off of its accumulated IT assets, including its mainframe, with a cloud-native, microservice-based approach. This Express Scripts online talk describes the company’s transformation from mainframe to a microservices-based ecosystem using Kafka and change data capture (CDC) technology. Finally, when you’re ready to get into the how-to details, check out the online talk on Mainframe Integration, Offloading and Replacement with Apache Kafka.

Bert Hayes is a solutions engineer for Confluent, supporting public sector customers with a focus on state and local governments and higher education. Bert has a background in information security and has been an incident handler for the University of Texas at Austin as well as a handful of Texas state agencies. Bert likes riding his bike.

Did you like this blog post? Share it now

Subscribe to the Confluent blog

More Articles Like This

Broadcom Modernizes Machine Learning and Anomaly Detection with ksqlDB

Mainframes are still ubiquitous, used for almost every financial transaction around the world—credit card transactions, billing, payroll, etc. You might think that working on mainframe software would be dull, requiring […]

La transition de CloudBank du système central au streaming avec Confluent Cloud

Le cloud est l'un des principaux moteurs de l'innovation. Les entreprises innovantes expérimentent avec des données pour arriver à quelque chose d'utile. Généralement, cela commence par l'ouverture d'une lance à incendie que [...]

Sign Up Now

Start your 3-month trial. Get up to $200 off on each of your first 3 Confluent Cloud monthly bills

Nouvelles inscriptions uniquement.

En cliquant sur le bouton « inscription » ci-dessus, vous acceptez que nous traitions vos informations personnelles conformément à notre Politique de confidentialité.

En cliquant sur « Inscription » ci-dessus, vous acceptez les termes du/de la Conditions d'utilisation et de recevoir occasionnellement des e-mails publicitaires de la part de Confluent. Vous comprenez également que nous traiterons vos informations personnelles conformément à notre Politique de confidentialité.

Gratuit à vie sur un seul broker Kafka
i

Le logiciel permettra une utilisation illimitée dans le temps de fonctionnalités commerciales sur un seul broker Kafka. Après l'ajout d'un second broker, un compteur de 30 jours démarrera automatiquement sur les fonctionnalités commerciales. Celui-ci ne pourra pas être réinitialisé en revenant à un seul broker.

Sélectionnez un type de déploiement
Déploiement manuel
  • tar
  • zip
  • deb
  • rpm
  • docker
ou
Déploiement automatique
  • kubernetes
  • ansible

En cliquant sur le bouton « télécharger gratuitement » ci-dessus, vous acceptez que nous traitions vos informations personnelles conformément à notre Politique de confidentialité.

En cliquant sur « Téléchargement gratuit » ci-dessus, vous acceptez la Contrat de licence Confluent et de recevoir occasionnellement des e-mails publicitaires de la part de Confluent. Vous acceptez également que vos renseignements personnels soient traitées conformément à notre Politique de confidentialité.

Ce site Web utilise des cookies afin d'améliorer l'expérience utilisateur et analyser les performances et le trafic sur notre site Web. Nous partageons également des informations concernant votre utilisation de notre site avec nos partenaires publicitaires, analytiques et de réseaux sociaux.