bushlab/privategpt repository overview

NOTE: Tag v1.2 is still being tested, do not use

⁠Introduction

This is a Docker container built to run privateGPT, using llama-cpp-python, compiled with CUDA 11.8 support, and an installation of NVIDIA SMI driver version 535.54.

⁠Contents

This container was built from the privateGPT Git repo using poetry.

The llama-cpp-python LLM was built with cuBLAS acceleration.

⁠Running the container

privateGPT and the LLM will be hosted on port 8001.

Run hostname -a to find your domain name.

Once the container has been pulled, run the following command to launch the container in interactive mode:

docker run -p 8001:8001 -it bushlab/privategpt:v1.1

Once inside the container, run:

poetry run python3.11 -m private_gpt

This will initiate the privateGPT module, which can be accessed through a Gradio UI at domain_name:8001.

Important: This container was built using mambaforge. All the necessary packages have been installed in a mamba env called gpt_env. If the container doesn't have gpt-env activated when entering it, run mamba activate gpt_env.

⁠References

PrivateGPT can be found here: (https://github.com/imartinez/privateGPT⁠).

llama-cpp-python can be found here. (https://github.com/ggerganov/llama.cpp⁠)