Process PeopleSoft Sync Messages into Kafka Topics
Note: This project is under development. There is no usable service at this time.
ps-stream is a Python utility that collects and and parses PeopleSoft rowset-based messages generated by sync and fullsync services into Kafka messages and topics. PeopleSoft sync processes are normally used to sync data between PeopleSoft applications. However, they can also be used as a way to generate an externalized stream of PeopleSoft objects in streaming data pipelines.
There are two major commands in
collect accepts PeopleSoft rowset-based messages over http or https, and produces a Kafka message for each transaction in the PeopleSoft message, stored in one or more Kafka topics.
parse consumes transaction messages stored by
collect and produces Kafka messages in topics with
KTable semantics. Each record in the resulting stream is oriented to reflect records whose record key is the primary key or identifier of the transaction message.
Running a ps-stream container
Collect PeopleSoft sync messages.
$ docker run -p 8000:8000 -d ucalgary/ps-stream collect
This repository and image are currently maintained by the Research Management Systems project at the University of Calgary.