leybzon/llm-serving repository overview

This container serves the mosaicml/mpt-7b language model using FastAPI. It allows users to interact with the model through a web-based chat interface or REST API while supporting configurable parameters such as max_new_tokens, top_k, top_p, and temperature. Designed for text generation and other NLP tasks, this container provides a simple yet powerful tool for deploying LLMs.

⁠Features:

• Model: Uses the mosaicml/mpt-7b LLM, known for its versatility in text generation.
• API & Web Interface: Includes a chat interface and API endpoints for model interaction.
• Customizable Parameters: Adjust model behavior with various options like token limits, temperature, etc.

⁠Usage:

• Run the Container:

  docker run --gpus all -p 8000:8000 leybzon/llm-serving:v1
  

  Copy
• Access the Web Interface: Open http://localhost:8000 in a web browser for chat interaction.
• Use the API: Send POST requests to /generate to get model responses with customized parameters.
• Example: curl -X POST "http://localhost:8000/generate" -H "Content-Type: application/json" -d '{"prompt": "What is the capital of France?"}'

⁠Requirements:

• Hardware: NVIDIA GPU with CUDA support.
• Docker: Containerized application for easy deployment.

⁠Links:

• Model Information: mosaicml/mpt-7b on Hugging Face⁠
• Documentation: Detailed usage instructions and parameter settings are provided within the container.

This container is ideal for those looking to leverage the capabilities of the mosaicml/mpt-7b model in various NLP applications.