Sign inSign up

opea/llm-eval

By opea

•Updated over 1 year ago

OPEA LLM microservice upon eval docker image for GenAI application

Image
0

1.3K

opea/llm-eval repository overview

⁠LM-Eval Microservice

This microservice, designed for lm-eval⁠, which can host a separate llm server to evaluate lm-eval tasks.

⁠CPU service

⁠build cpu docker
docker build -f Dockerfile.cpu -t opea/lm-eval:latest .
⁠start the server
  • set the environments MODEL, MODEL_ARGS, DEVICE and start the server
docker run -p 9006:9006 --ipc=host  -e MODEL="hf" -e MODEL_ARGS="pretrained=Intel/neural-chat-7b-v3-3" -e DEVICE="cpu" opea/lm-eval:latest
⁠evaluate the model
  • set base_url and tokenizer
git clone https://github.com/opea-project/GenAIEval
cd GenAIEval
pip install -e .

cd GenAIEval/evaluation/lm_evaluation_harness/examples

python main.py \
    --model genai-hf \
    --model_args "base_url=http://{your_ip}:9006,tokenizer=Intel/neural-chat-7b-v3-3" \
    --tasks  "lambada_openai" \
    --batch_size 2

Tag summary

Content type

Image

Digest

sha256:7f8575d21…

Size

3.3 GB

Last updated

over 1 year ago

docker pull opea/llm-eval