averemee/oracdc

By averemee

Updated 20 days ago

Oracle Database CDC and Data Streaming using Kafka Connect

Image
Databases & Storage
Integration & Delivery
Message Queues

1.2K

NEW TAGS will be at https://hub.docker.com/r/a2solutions/oracdc

You can dynamically specify configuration values with environment variables. You can use the Docker -e/--env or --env-file flags to specify various settings. You need to convert connect-distributed.properties file variables as described below and pass them as environment variables:

  1. Convert to upper case
  2. Prefix with A2_ORACDC_
  3. Replace a period [.] with a single underscore (_]
  4. Replace a dash [-] with double underscores [__]
  5. Replace an underscore [_] with triple underscores [___]

Ports and volumes

Standard Kafka Connect port 8083 is exposed, for storing secrets and oracdc memory mapped files mountpoint /var/lib/oracdc is present

Mandatory settings

A2_ORACDC_BOOTSTRAP_SERVERS A list of Kafka brokers to connect to

A2_ORACDC_GROUP_ID A unique string that identifies the Connect cluster group this Worker belongs to

A2_ORACDC_OFFSET_STORAGE_TOPIC The name of the existing topic where connector and task configuration offsets are stored. This must be the same for all Workers with the same A2_ORACDC_GROUP_ID (group.id). Always create it as a compacted, highly replicated (3x or more) topic with a large number of partitions (e.g., 25 or 50, just like Kafka’s built-in __consumer_offsets topic) to support large Kafka Connect clusters.

A2_ORACDC_CONFIG_STORAGE_TOPIC The name of the existing topic where connector and task configuration data are stored. This must be the same for all Workers with the same A2_ORACDC_GROUP_ID (group.id). Always create this topic as a compacted topic with a single partition and a high replication factor (3x or more).

A2_ORACDC_STATUS_STORAGE_TOPIC The name of the existing topic where connector and task configuration status updates are stored. This must be the same for all Workers with the same A2_ORACDC_GROUP_ID (group.id). Always create this topic as a compacted, highly replicated (3x or more) topic with multiple partitions

oracdc configuration

For complete example please read How to Set Up Oracle Database Tables Replication in Minutes

OR

Perform RDBMS configuration steps according to the Required Oracle database setup for oracdc Source connector

Full list of oracdc configuration parameters is available here

Example configuration for connector itself

Create JSON config file (do not forget to replace connection details with your actual data) e.g.

{
   "name" : "oracdc-test",
   "config" : {
     "connector.class" : "solutions.a2.cdc.oracle.OraCdcLogMinerConnector",
     "tasks.max" : "1",
     "a2.resiliency.type" : "fault-tolerant,
     "a2.wallet.location" : "/var/lib/oracdc/admin/wallet",
     "a2.jdbc.url" : "jdbc:oracle:thin:@VISION?TNS_ADMIN=/var/lib/oracdc/admin/network",
     "a2.include" : "SCOTT.DEPT,SCOTT.EMP"
   }
 }

Create connector by invoking REST API at exposed port

curl -s -X POST -H 'Content-Type: application/json' --data @connector.json http://<CONTAINER_IP>:8083/connectors

Docker Pull Command

docker pull averemee/oracdc