senzing/stream-producer repository overview

⁠stream-producer

⁠Synopsis

Populate a queue with records to be consumed by stream-loader⁠.

⁠Overview

The stream-produder.py⁠ python script reads files of different formats (JSON, CSV, Parquet, Avro) and publishes it to a queue (RabbitMQ, Kafka, AWS SQS). The senzing/stream-producer docker image is a wrapper for use in docker formations (e.g. docker-compose, kubernetes).

To see all of the subcommands, run:

$ ./stream-producer.py --help
usage: stream-producer.py [-h]
                          {avro-to-kafka,avro-to-rabbitmq,avro-to-sqs,avro-to-sqs-batch,avro-to-stdout,csv-to-kafka,csv-to-rabbitmq,csv-to-sqs,csv-to-sqs-batch,csv-to-stdout,gzipped-json-to-kafka,gzipped-json-to-rabbitmq,gzipped-json-to-sqs,gzipped-json-to-sqs-batch,gzipped-json-to-stdout,json-to-kafka,json-to-rabbitmq,json-to-sqs,json-to-sqs-batch,json-to-stdout,parquet-to-kafka,parquet-to-rabbitmq,parquet-to-sqs,parquet-to-sqs-batch,parquet-to-stdout,websocket-to-kafka,websocket-to-rabbitmq,websocket-to-sqs,websocket-to-sqs-batch,websocket-to-stdout,sleep,version,docker-acceptance-test}
                          ...

Queue messages. For more information, see https://github.com/Senzing/stream-
producer

positional arguments:
  {avro-to-kafka,avro-to-rabbitmq,avro-to-sqs,avro-to-sqs-batch,avro-to-stdout,csv-to-kafka,csv-to-rabbitmq,csv-to-sqs,csv-to-sqs-batch,csv-to-stdout,gzipped-json-to-kafka,gzipped-json-to-rabbitmq,gzipped-json-to-sqs,gzipped-json-to-sqs-batch,gzipped-json-to-stdout,json-to-kafka,json-to-rabbitmq,json-to-sqs,json-to-sqs-batch,json-to-stdout,parquet-to-kafka,parquet-to-rabbitmq,parquet-to-sqs,parquet-to-sqs-batch,parquet-to-stdout,websocket-to-kafka,websocket-to-rabbitmq,websocket-to-sqs,websocket-to-sqs-batch,websocket-to-stdout,sleep,version,docker-acceptance-test}
                              Subcommands (SENZING_SUBCOMMAND):
    avro-to-kafka             Read Avro file and send to Kafka.
    avro-to-rabbitmq          Read Avro file and send to RabbitMQ.
    avro-to-sqs               Read Avro file and print to AWS SQS.
    avro-to-stdout            Read Avro file and print to STDOUT.

    csv-to-kafka              Read CSV file and send to Kafka.
    csv-to-rabbitmq           Read CSV file and send to RabbitMQ.
    csv-to-sqs                Read CSV file and print to SQS.
    csv-to-stdout             Read CSV file and print to STDOUT.

    gzipped-json-to-kafka     Read gzipped JSON file and send to Kafka.
    gzipped-json-to-rabbitmq  Read gzipped JSON file and send to RabbitMQ.
    gzipped-json-to-sqs       Read gzipped JSON file and send to AWS SQS.
    gzipped-json-to-stdout    Read gzipped JSON file and print to STDOUT.

    json-to-kafka             Read JSON file and send to Kafka.
    json-to-rabbitmq          Read JSON file and send to RabbitMQ.
    json-to-sqs               Read JSON file and send to AWS SQS.
    json-to-stdout            Read JSON file and print to STDOUT.

    parquet-to-kafka          Read Parquet file and send to Kafka.
    parquet-to-rabbitmq       Read Parquet file and send to RabbitMQ.
    parquet-to-sqs            Read Parquet file and print to AWS SQS.
    parquet-to-stdout         Read Parquet file and print to STDOUT.

    sleep                     Do nothing but sleep. For Docker testing.
    version                   Print version of program.
    docker-acceptance-test    For Docker acceptance testing.

optional arguments:
  -h, --help                  show this help message and exit

⁠Contents

1. Demonstrate using Docker⁠
2. Demonstrate using docker-compose⁠
3. Demonstrate using Command Line Interface⁠
   1. Prerequisites for CLI⁠
   2. Download⁠
   3. Run command⁠
4. Configuration⁠
5. AWS configuration⁠
6. References⁠

⁠Preamble

At Senzing⁠, we strive to create GitHub documentation in a "don't make me think⁠" style. For the most part, instructions are copy and paste. Whenever thinking is needed, it's marked with a "thinking" icon :thinking:. Whenever customization is needed, it's marked with a "pencil" icon :pencil2:. If the instructions are not clear, please let us know by opening a new Documentation issue⁠ describing where we can improve. Now on with the show...

⁠Legend

1. :thinking: - A "thinker" icon means that a little extra thinking may be required. Perhaps there are some choices to be made. Perhaps it's an optional step.
2. :pencil2: - A "pencil" icon means that the instructions may need modification before performing.
3. :warning: - A "warning" icon means that something tricky is happening, so pay attention.

⁠Expectations

• Space: This repository and demonstration require 6 GB free disk space.
• Time: Budget 40 minutes to get the demonstration up-and-running, depending on CPU and network speeds.
• Background knowledge: This repository assumes a working knowledge of:
  • Docker⁠

⁠Demonstrate using Docker

1. Run Docker container. This command will show help. Example:

   docker run \
     --rm \
     senzing/stream-producer --help
   
   

   Copy

2. For more examples of use, see Examples of Docker⁠.

⁠Demonstrate using docker-compose

1. Deploy the Backing Services⁠ required by the Stream Loader.

2. Specify a directory to place artifacts in. Example:

   export SENZING_VOLUME=~/my-senzing
   mkdir -p ${SENZING_VOLUME}
   
   

   Copy

3. Download docker-compose.yaml file. Example:

   curl -X GET \
     --output ${SENZING_VOLUME}/docker-compose.yaml \
     https://raw.githubusercontent.com/Senzing/stream-producer/main/docker-compose.yaml
   
   

   Copy

4. Bring up docker-compose stack. Example:

   docker-compose -f ${SENZING_VOLUME}/docker-compose.yaml up
   
   

   Copy

⁠Demonstrate using Command Line Interface

⁠Prerequisites for CLI

:thinking: The following tasks need to be complete before proceeding. These are "one-time tasks" which may already have been completed.

1. Install Python prerequisites. Example:

   pip3 install -r https://raw.githubusercontent.com/Senzing/stream-producer/main/requirements.txt
   
   

   Copy
   1. See requirements.txt⁠ for list.
      1. Installation hints⁠

⁠Download

1. Get a local copy of stream-producer.py⁠. Example:

   1. :pencil2: Specify where to download file. Example:

      export SENZING_DOWNLOAD_FILE=~/stream-producer.py
      
      

      Copy

   2. Download file. Example:

      curl -X GET \
        --output ${SENZING_DOWNLOAD_FILE} \
        https://raw.githubusercontent.com/Senzing/stream-producer/main/stream-producer.py
      
      

      Copy

   3. Make file executable. Example:

      chmod +x ${SENZING_DOWNLOAD_FILE}
      
      

      Copy

2. :thinking: Alternative: The entire git repository can be downloaded by following instructions at Clone repository⁠

⁠Run command

1. Run the command. Example:

   ${SENZING_DOWNLOAD_FILE} --help
   
   

   Copy

2. For more examples of use, see Examples of CLI⁠.

⁠Configuration

Configuration values specified by environment variable or command line parameter.

• SENZING_NETWORK⁠

⁠AWS configuration

stream-producer.py⁠ uses AWS SDK for Python (Boto3)⁠ to access AWS services. This library may be configured via environment variables or ~/.aws/config file.

Example environment variables for configuration:

• AWS_ACCESS_KEY_ID⁠
• AWS_SECRET_ACCESS_KEY⁠
• AWS_DEFAULT_REGION⁠

⁠References

1. Boto3 Configuration⁠
2. Boto3 Credentials⁠
3. Development⁠
4. Errors⁠
5. Examples⁠
6. Related artifacts:
   1. DockerHub⁠