kafka Data Integration The data synchronization system replicates business data from databases such as Oracle and GBase 8s using tools such as Oracle Golden Gate (OGG) and GBase RTSync, and synchronizes it to GBase 8a MPP Cluster through Kafka. To cope with possible spikes in business systems, a Kafka message queue is added to the system as a buffer. The overall process is as follows: Figure 4-1. Process flow. The OGG sender (GoldenGate Extract) extracts transaction information from Oracle's online logs and archive logs and generates Trail files. The OGG receiver (GoldenGate Replicat) receives the Trail files, extracts the transaction information, converts it to the target format, and produces transaction messages to Kafka. The consumer consumes transaction messages from Kafka and updates the data to 8a MPP Cluster. The main function of the Kafka consumer is to synchronize Kafka data to 8a MPP Cluster: 1)Based on the configuration, the business to be synchronized can be specified; 2)During the synchronization process, the function of querying
GBase 8a MPP Cluster Technical White Paper General Data Technology Co., Ltd. - 23 - synchronization status is provided; 3)Implement high availability and transaction data consistency for data synchronization. Virtual Clusters and Mirror Clusters