pathwaycom/pathway repository overview

⁠What is Pathway Live Data Framework?

Pathway Live Data Framework⁠ is an open framework for high-throughput and low-latency real-time data processing. It is used to create Python code which seamlessly combines batch processing, streaming, and real-time API's for LLM apps. Pathway Live Data Framework's distributed runtime (:crab:-:snake:) provides fresh results of your data pipelines whenever new inputs and requests are received.

Explore our Pathway Live Data Framework⁠ and dive into our GitHub repositories⁠:

• Core Framework⁠: The core of efficiency in real-time data processing.
• Examples⁠: Practical demonstrations of Pathway's adaptability.
• Benchmarks⁠: Performance metrics showcasing speed and reliability.

⁠How to use this image

⁠Image Variants

Currently, we have two types of images that can be used according to the user's needs:  - pathway:<version>-slim

This image contains only the Pathway Live Data Framework package and the dependencies required to use it.

 - pathway:<version>/pathway:latest

This image includes all the dependencies necessary to run Pathway, along with indexer examples.

⁠Create a Dockerfile in your Python app project

FROM pathwaycom/pathway:latest

WORKDIR /app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "python", "./your-script.py" ]

You can then build and run the Docker image:

docker build -t my-pathway-app .
docker run -it --rm --name my-pathway-app my-pathway-app

⁠Run a single Python script

When dealing with single-file projects, creating a full-fledged Dockerfile might seem unnecessary. In such scenarios, you can execute a Python script directly using the Python Docker image. For example:

docker run -it --rm --name my-pathway-app -v "$PWD":/app pathwaycom/pathway:latest python my-pathway-app.py

⁠License

Pathway Live Data Framework is distributed on a BSL 1.1 License which allows for unlimited non-commercial use, as well as use of the Pathway Live Data Framework package for most commercial purposes, free of charge. Code in this repository automatically converts to Open Source (Apache 2.0 License) after 4 years. Some public repos which are complementary to this one (examples, libraries, connectors, etc.) are licensed as Open Source, under the MIT license.