Data Governance
Also available as:
PDF
loading table of contents...

Integrating Messaging with Atlas

So far we have covered integrating with Atlas using its REST API. In the Architecture section, we mentioned another method of integration: a Messaging interface.

The Atlas Messaging interface uses Apache Kafka. Apache Kafka is a scalable, reliable, high-performance messaging system. It provides a mechanism for integrating between Atlas and metadata sources that generate a high volume of metadata events.

Kafka also provides a durable message store. This way, metadata change events can be written to Kafka by sources even if Atlas is not available to process them at the moment. This allows for a very loosely coupled integration, and therefore generally more reliability in a distributed architecture. This is also true for consumers of metadata change events that Atlas communicates via Kafka.

While this durability offers safety guarantees, when Atlas is active (along with the other metadata sources and consumers) it can process metadata events in real time.

Kafka is used for two types of messages. Each message type is written to a specific topic in Kafka.

Note also that the Messaging integration is restricted to Entity notifications. Type-related changes are still handled via the API layer. This is acceptable because Types are created much less frequently than Entities.

The following sections provide the formats for messages that are written to ATLAS_HOOK and ATLAS_ENTITIES.