Realtime Connectors — Bringing Extractions to the Next Level

At Celonis, we enable our users to make use of Realtime Connectors to get their source system data into the Celonis EMS -as the name suggests- in real-time. 

This is an avant-garde alternative to the traditional batch extractions. A modern technology in view that once the extraction is configured, our real-time connectors monitor every update in the source system and apply the changes on the Celonis side automatically, in a matter of seconds. 

We are going to walk through the benefits for real-time data ingestion, as well as the types of Realtime Connectors and how they function from a high-level manner. 

Why do we need Realtime Connectors?

In modern business, massive amounts of data are generated every second. Each update carries new information and users expect their data to be available the second it’s updated. The only way to do this is real-time connectivity. It is a necessity to keep up with modern business requirements. 

This is where Celonis comes in. By extracting process data from on-prem and cloud-based ERPs, CRMs, data warehouses, productivity tools and desktop activities (i.e task mining), you can turn raw data into meaningful insights. Celonis automates data extraction so you can execute on the most up-to-date information available. With a stable real-time pipeline, it is possible to quickly identify process inefficiencies and take further, even automated, action to fix them. In this sense, real-time connectivity is a fundamental step for unlocking execution capacity.

Our goal was to make this initial step as user friendly as possible. The subset of benefits of Realtime Connectivity stated below serve the objective of better user experience:

• Auto-connectivity — Data extractions are orchestrated automatically without user-defined schedules, thereby reducing the possibility of human error while keeping data more up to date.

• Auto self-recovery — In case of temporary failures, extractions will correct themselves and re-execute automatically which reduces the need for manual error handling.

• Reduced load on the source system — ​​Through intelligent frequent extractions, real-time connectivity decreases the load on the source system.

• Latest data — Since data connectivity is the first step within the Celonis EMS, all the decisions users make will be based on the latest data via real-time connectivity.

• Enabling real-time transformations and Action flows — All the automated tasks within the Celonis EMS will be performed based on the latest data.

Types of Realtime Connectors

Realtime Connectors are differentiated by the technology behind them. Let’s walk through two main Realtime Connector types and how they function in the Celonis EMS.

1. Realtime JDBC

Real-time connectivity to JDBC (Java Database Connectivity) based source tables can be achieved using the Replication Cockpit in Celonis. 

The Replication Cockpit executes a new extraction for the requested table every few minutes (the time interval is configurable) with a major twist of using change-log tables to detect changes in the source system. In technical terminology, the process of ensuring the consistency between two systems is called “replication”.

Let’s define 2 major components of a replication:

• Trigger: A simple SQL code snippet that needs to be run on the customer database to track changes of the corresponding tables

• Changelog Table: A table that would be populated when there is a change in the source table with the help of triggers

Real-time connectivity in JDBC can be achieved by creating changelog tables that would be populated when there is a change in the source table. The changes in the source table can be tracked with the help of triggers. 

In short, the replication process can be summarized in 3 steps:

1. Create changelog tables to store changes

2. Install triggers to monitor the changes

3. Cleanup of the processed changelog tables

The processing of the changelog tables is done by joining the changelog tables with the source table over the primary keys. We always expect the primary keys of the source table to be part of the changelog tables in order to join the tables correctly.

2. Streaming Extractions

For traditional source systems, the data is constantly present and can be requested at any time. For streaming based source systems this is not the case. The whole data is not available any time. The required data should be retrieved, processed and stored in realtime by communicating with the streaming service. In Celonis, for the supported streaming services Realtime Connectivity can be achieved using Streaming Cockpit. 

In order to accommodate these streaming source systems, we introduced a different type of extraction architecture. Each connection to the streaming source system is called "Subscription" in the Streaming Cockpit. These subscriptions can be turned on and off using the Streaming Cockpit UI.

Celonis supports different streaming services like Azure Service Bus, Azure Event Hub, Salesforce Streaming. Since the streaming services are not maintained by Celonis, we are playing the role of a “user”. To retrieve and process data, we have different strategies for different streaming services. Let’s check an example to concretize.

For example, Salesforce Streaming gives us a “replayID” which indicates the event number in order. Among all the present events Salesforce Streaming services provide, we know from where to process the data by the replayID. We can signal how many new messages we received and processed. In case of a temporary failure, extraction will correct itself and re-execute automatically by using the last successfully processed replayID.

Once we successfully process the data, we update our replayID, signal the changes and let the Salesforce Streaming decide whether to keep or delete the messages from their stream. We cannot enforce the streaming services to delete processed data because there might be other users watching the same stream.

Salesforce Streaming is just one example. Not all streaming services have the same structure, so we customize our solutions to the specific needs of the streaming services to have a better efficiency in every case. 

Takeaways

Realtime Connectivity solutions that Celonis provides enable users to have a better synchronization with the Celonis EMS. It reduces the required effort for the users to configure data extractions in the long run. We aim to minimize the user-errors, reduce the load on the source system, and introduce better auto self-recovery mechanisms.

We are aware that the wave of the future includes automation, intelligent solutions, and always being up-to-date. Introducing Realtime Connectors is one of the important steps towards this vision. Realtime Connectors help us to turn friction into flow.

Yaşar Fatih Enes Yalçın

Software Engineer

Software Engineer in Extractors Suite team, Enes works at the intersection of customer data and the Celonis EMS. He specializes in extraction of valuable data from different sources. He is a drone pilot. He can throw-and-catch a frisbee like a boomerang. Interested in fragrances.