milkymap/gpt4pdf repository overview

⁠Docker image gpt4pdf

This Docker image allows you to create embeddings and build indexes for PDF documents using the GPT-3.5 model and the Qdrant indexing engine.

#Usage First, pull the Docker image using the following command:

docker pull milkymap/gpt4pdf

⁠Why do I need COHERE and OPENAI API keys?

To use this Docker image, you'll need API keys for both COHERE and OPENAI. Here's why: COHERE API key

COHERE is a language model that's used to calculate embeddings for the text chunks in your PDF documents. In order to use COHERE, you'll need an API key, which you can obtain by signing up for a COHERE account on their website. Once you have an API key, you'll need to pass it to the Docker container as an environment variable (using the -e option in the docker run command). OPENAI API key

GPT-3.5 is the language model used in this Docker image to tokenize your PDF documents. To use GPT-3.5, you'll need an API key from OPENAI, which you can obtain by signing up for an OPENAI account on their website. Once you have an API key, you'll also need to pass it to the Docker container as an environment variable (using the -e option in the docker run command).

By providing these API keys, you'll be able to leverage the power of these language models to create high-quality embeddings and indexes for your PDF documents.

⁠Creating an index

To create an index, run the following command:

docker run -it --rm --name gpt4pdf \
    -v /path/to/corpus:/home/solver/corpus \
    -v /path/to/qdrant_stoage:/home/solver/qdrant_storage \
    -e OPENAI_API_KEY=XXXX-XXXX \
    -e COHERE_API_KEY=XXXX-XXXX \
    -e NB_DIMS=768 \
        milkymap/gpt4pdf:latest build-index \
            --pdf_filename filename.pdf \
            --nb_threads 32 \
            --tokenizer gpt-3.5-turbo \
            --chunk_size 392 \
            --chunk_overlap 32 \
            --collection_name kb \
            --recreate_collection

This command will divide the PDF document into chunks, calculate embeddings for each chunk using the Cohere API, and build an index using the Qdrant engine.

⁠Exploring an index

To explore an index, run the following command:

docker run -it --rm --name gpt4pdf \
    -v /path/to/corpus:/home/solver/corpus \
    -v /path/to/qdrant_storage:/home/solver/qdrant_storage \
    -e OPENAI_API_KEY=XXXX-XXXX \
    -e COHERE_API_KEY=XXXX-XXXX \
    -e NB_DIMS=768 \
        milkymap/gpt4pdf:latest explore-index \
               --collection_name kb  \
               --top_k 7 \
               --code zh

this command will explore the index and return the top K similar chunks for a given query.