maadsdocker/tml-privategpt-no-gpu-amd64 repository overview

This private GPT container is 100% local, FREE, secure and production ready. This repo is cloned from https://github.com/imartinez/privateGPT⁠.

This a no GPU container. There is a container that will access NVidia GPU enabled server (https://hub.docker.com/repository/docker/maadsdocker/tml-privategpt-with-gpu-nvidia-amd64/general⁠).

This container can also be accessed via API:

Docker Run: docker run -d -p 8001:8001 --net=host --env PORT=8001 --env GPU=0 --env WEB_CONCURRENCY=1 --env COLLECTION=tml-cisco maadsdocker/tml- 
privategpt-no-gpu-amd64:latest

NOTE: You MUST have the Qdrant VectorDB container running as shown below:

docker run -d -p 6333:6333 -v $(pwd)/qdrant_storage:/qdrant/storage:z qdrant/qdrant

Now, enter in chrome or browser: http://localhost:8001⁠

Details of LLM:
llm_load_print_meta: format = GGUF V2
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32000
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: model type = 7B
llm_load_print_meta: model ftype = mostly Q4_K - Medium
llm_load_print_meta: model params = 7.24 B
llm_load_print_meta: model size = 4.07 GiB (4.83 BPW)
llm_load_print_meta: general.name = mistralai_mistral-7b-instruct-v0.2
llm_load_print_meta: BOS token = 1 ''
llm_load_print_meta: EOS token = 2 ''
llm_load_print_meta: UNK token = 0 ''
llm_load_print_meta: LF token = 13 '<0x0A>'
llm_load_tensors: ggml ctx size = 0.11 MB
llm_load_tensors: mem required = 4165.47 MB