johnsnowlabs/jsl-llms repository overview

This Docker image provides a deployable inference server for medical large language models (LLMs) developed by JohnSnowLabs. Designed for healthcare and medical research applications, this service enables advanced medical AI capabilities with flexible deployment options.

⁠🔬 Supported Medical LLM Models

⁠🚀 Quick Start

⁠🔄 Pull the Docker Image

You can pull the image from 👉 Docker Hub⁠:

docker pull johnsnowlabs/jsl-llms:latest

⁠🐳 Run the Container

⁠Basic Docker Run Command

docker run -d \
    --gpus all \
    --env "SPARK_NLP_LICENSE=your_license_key" \
    -p 8080:8080 \
    --ipc=host \
    johnsnowlabs/jsl-llms \
    --model Medical-LLM-Small \
    --port 8080

⁠Using License File

Mount a license.json file for more secure license management:

docker run -d \
    --gpus all \
    -v "$(pwd)/license.json:/app/license.json" \
    -p 8080:8080 \
    --ipc=host \
    johnsnowlabs/jsl-llms \
    --model Medical-LLM-Small \
    --port 8080

⁠⚙️ Advanced Configuration

For detailed information about available parameters and configuration options, please refer to the official VLLM documentation: 📚 VLLM Server Parameters Documentation⁠

⁠Key Parameters to Customize

• --tensor-parallel-size: Set number of GPUs for tensor parallelism
• --max-model-len: Maximum sequence length
• --gpu-memory-utilization: Fraction of GPU memory to be used

⁠Memory Optimization Tips

• Use smaller sequence lengths to reduce KV-cache memory
• Leverage tensor parallelism for large models
• Select an appropriate model based on your GPU resources

⁠🌐 API Endpoints

⁠Health and Monitoring

• GET /health: Check service health status
• GET /ping: Quick server ping
• GET /version: Retrieve server version information

⁠Model Management

⁠List Available Models

• Endpoint: /v1/models
• Method: GET
• Description: Retrieve a list of available models on the server

⁠Model Interactions

⁠Chat Completions

• Endpoint: /v1/chat/completions
• Method: POST
• Example Request:

payload = {
    "model": "Medical-LLM-Small",
    "messages": [
        {"role": "system", "content": "You are a professional medical assistant"},
        {"role": "user", "content": "Explain symptoms of chronic fatigue syndrome"}
    ],
    "temperature": 0.7,
    "max_tokens": 1024
}

⁠Text Completions

• Endpoint: /v1/completions
• Method: POST
• Example Request:

payload = {
    "model": "Medical-LLM-Small",
    "prompt": "Provide a detailed explanation of rheumatoid arthritis treatment",
    "temperature": 0.7,
    "max_tokens": 4096
}

⁠🤝 Support

For technical support, inquiries, or enterprise solutions, contact:

• Email: [email protected]⁠
• Live Chat: Available on our website⁠ during business hours

⁠📝 Licensing

A valid Spark NLP license is required to use this inference server. Licenses can be obtained through JohnSnowLabs.