Celonis Kafka Connector: Real-time data streaming enables process optimization, business transformation

Exasperation. If you’re an air traveler who’s flight has been canceled or delayed, it’s an emotion you’ve likely felt. If you’re an airline, it’s an emotion you don’t want passengers to feel and one you’re constantly working to prevent. That means keeping your flights running smoothly and on time, even when bad weather strikes.

“A punctual flight is the basic customer satisfaction for everyone who travels with an airline,” said Philipp Grindemann, head of business development and project management at Lufthansa CityLine, in an interview with Celonis.

A key aspect of reducing or eliminating delays, weather-related or otherwise, is knowing which flights are at risk of being delayed in real-time, so that ground crews can take action to mitigate or eliminate the delay. And to know which flights are at risk, airlines need to collect, analyze and disseminate information in real-time, which is a tough technical challenge and one that the Celonis Kafka Connector was designed to solve.

“Celonis Kafka Connector enables Celonis EMS to create new business opportunities by reacting to emerging business conditions and modeling completely new business processes,” said Antonios Chalkiopoulos, VP of Streaming Data, Engineering, Action and Ecosystem at Celonis. “As part of the EMS, it has the potential to transform business operations across all industries.”

What is the Celonis Kafka Connector?

Developed by engineers from Lenses.io, which Celonis acquired in October 2021, Celonis Kafka Connector is an open-source component that can enhance any Apache Kafka® cluster (using the Kafka Connect framework) to add native capabilities to ship any stream of data from the customer's Apache Kafka cluster, into Celonis Execution Management System (EMS).

Bring streaming data from any Kafka into Celonis EMS

“Celonis Kafka Connector opens up new execution management possibilities and improves the data integration practices for data streaming into EMS,” said Chalkiopoulos.

Celonis Kafka Connector offers several benefits over other data ingestion methods: 

• All in real-time: Collect streaming data from transactional systems such as Oracle, SAP, Salesforce or Workday in real-time as a stream instead of batches, allowing you to take action before it’s too late.

• New data sources: Connect completely new forms of data from your custom microservices representing digital business, for example streaming data from IoT devices or 3rd party feeds like weather, traffic and market data.

• Better data integration architecture: Use Apache Kafka to avoid a multitude of point-to-point connections.

“The connector maximizes the value an organization generates from data by enabling entirely new applications and thereby unlocking hidden execution capacities,” said Jonas Weich, Associate Product Manager at Celonis.

Data ingested through Celonis Kafka Connector can support business execution in multiple ways:

• Real-time contextual data: Live traffic information from a 3rd party feed could enable you to take corrective action on an order that’s at risk for a late delivery. By knowing which supplies are impacted, you could split the order and ship the products that are in stock instead of allowing the whole order to be delayed.

• Real-time transactional data: You can ingest data via your streaming platform from business systems like Salesforce or SAP, but do so in real-time rather than in batches.

• Real-time data from event-driven applications: If your organization is already using Kafka to build event-driven applications or uses microservice architecture, you can integrate this data into Celonis EMS to understand new business processes that span your digital services. Continuing our airline example, the carrier could collect sensor data from catering, refueling, waste disposal and other services to optimize processes that lead to shorter turnaround times between flights.

“We’re so excited for the possibility of streaming weather data and other sources into the EMS with the help of Lenses and the new Kafka connector, and to address new real-time use cases that otherwise would not be possible,” said Maximilian Hoffmann, Process Digitization Manager, Lufthansa CityLine.

Download: Celonis Kafka Connector Brochure for detailed information on how it works

AI4Pro research project: Celonis Kafka Connector in action

Celonis Kafka Connector is currently being used in a research project to develop an AI- and CPS-based solution that helps small and medium-sized manufacturing companies optimize their production processes in real time.

Launched in 2020, the AI4Pro project is a partnership between the Fraunhofer Institute for Integrated Circuits (IIS) (the project lead), Celonis, AST-X GmbH, Maxsyma GmbH & Co. KG, PASS Stanztechnik AG, Rauschert Heinersdorf-Pressig GmbH. Sponsored by the Bavarian Ministry of Economic Affairs, Energy and Technology and funded with 2.3M €, the project is scheduled to run until August 2023.

Data is collected with IoT sensors and streamed into EMS through the Celonis Kafka Connector.

The solution being developed through the AI4Pro project will have two basic phases.

First, data is collected from storage, logistics, assembly and other production areas via IoT sensors located along the production process and using Celonis Kafka Connector. In order to derive process events, the raw data is merged and processed by a dedicated software engine backed by machine learning algorithms.

In the second phase, process analytics are created within Celonis EMS that enable actionable insights in real time. The solution is designed to give employees within the production process “fast, effective and adaptive recommendations for action” according to a Fraunhofer IIS AI4Pro press release.

Celonis Kafka Connector is now open source

The Celonis Open Source Kafka Connector is available as an Apache 2.0 licensed component. The Kafka Connector GitHub repository contains the latest code releases, documentation, examples, operational details and more. It is one of Celonis’ first open source projects and has been instrumental in the development of our open source strategy.

“Releasing the Open Source Kafka Connector is an important milestone to kick-start the open source initiative at Celonis and share our best practices and solutions with the community,” said Stefan Bocutiu, Director Engineering and Streaming Data at Celonis. “We have a few projects lined up, so stay tuned.”

Bill Detwiler

Editor, Technical Content & Ecosystem

Bill Detwiler is Editor for Technical Content and Ecosystem at Celonis. He is the former Editor in Chief of TechRepublic, where he hosted the Dynamic Developer podcast and Cracking Open, CNET’s popular online show. Bill is an award-winning journalist, who’s covered the tech industry for more than two decades. Prior his career in the software industry and tech media, he was an IT professional in the social research and energy industries.