fnalabs/hive-stream-processor-js repository overview

This is the Hive^io⁠ framework Stream Processor microservice leveraging Node.js with Fastify in Docker. This is the base image⁠ to support most use cases.

⁠Supported tags and respective Dockerfile links

• 2.1.3, latest⁠ (Alpine Linux)
• 2.1.3-slim⁠ (Debian)
• 2.1.3-full⁠ (Debian)

⁠Getting Started

Stream Processors are multi-faceted in their responsibilities. By default, they handle the Command responsibilities in the CQRS pattern. Therefore, they are integrated with the domain layer to take commands and get existing aggregate data to pass to the domain layer for domain logic and validation. Once validated, it passes the returned event to the log and stores the updated snapshot of the aggregate to the caching layer. Depending on the needs of the domain model, the Stream Processor allows for transactional consistency if required. Essentially this makes it a Stream Producer as it is performing more than the Producer⁠, but for similar tasks.

The second role of the Stream Processor is to rebuild the caching layer from the transactional log. This is valuable when standing up new environments for various reasons like A/B testing, debugging, and deploying geolocated instances of the application stack. Essentially this makes it a Stream Consumer as it is performing the specific task of rebuilding the cache as opposed to the translations and queries of the Consumer⁠. Typically these would be a short-lived implementation and not used nearly as often as the default Stream Processor definition above.

The third role of the Stream Processor is the most complex. For more complex domain models, sometimes the need for a saga (or process manager) is required. A saga's job is to manage the complexities of inter-aggregate communication should the need arise. Since a Stream Processor is able to read events from the logs and also write to the logs (defined separately above), it is able to issue commands to the domain layer based on the events from one aggregate to another.

The Hive^io framework leverages Redis for a caching layer due to its high availability, distribution, and performance capabilities. Also, it employs the Redlock algorithm to provide transactional consistency and manage concurrency.

⁠Prerequisites

To use, you'll need:

• Required
  • Docker⁠
  • Kafka⁠
  • Redis⁠
• Recommended
  • OpenTelemetry Collector⁠
  • Zipkin⁠
  • Load Balancer or Service Mesh

⁠Installing

To start using in your own infrastructure, pull the base image:

docker pull fnalabs/hive-stream-processor-js:<[release]|latest>

⁠Examples

To use, write your own Dockerfile and add any additional dependencies, including the package with your domain Actors.

FROM fnalabs/hive-stream-processor-js:latest
RUN npm install hive-io-domain-example

⁠Environment variables

Below is a table describing the possible environment variables to run the Hive^io framework Stream Processor microservice. You can override these settings if/when required. This option works great if using the standard setup within a Docker container.

⁠Future

• feature requests via issues⁠