021bl00m/docker-model-runner repository overview

⁠Docker Model Runner with CI/CD

Docker-based ML model runner using Hugging Face's distilgpt2 for text generation, with automated GitHub Actions pipeline for building, testing, and deploying secure Docker images.

⁠What is a Model Runner?

A model runner is a containerized service that loads a machine learning model and exposes it via an API. This project demonstrates:

• 🤖 ML Model Serving: Hugging Face transformers (distilgpt2) for text generation
• 🐳 Docker Containerization: Multi-stage builds with security hardening
• 🔄 CI/CD Automation: GitHub Actions for automated builds and deployments
• 🔒 Security Scanning: Trivy and SonarCloud for vulnerability detection
• 📊 Code Quality: Automated code analysis with SonarCloud

⁠Architecture

The complete CI/CD pipeline from code push to deployment

⁠Project Structure

docker-model-runner/
├── README.md                    # This file
├── app.py                       # Flask API with model runner
├── requirements.txt             # Python dependencies
├── Dockerfile                   # Multi-stage Docker build
├── Makefile                     # Development commands
├── .gitignore                   # Git exclusions
├── .dockerignore               # Docker build exclusions
└── .github/
    └── workflows/
        └── docker-build.yml    # CI/CD pipeline

⁠Quick Start

⁠Prerequisites

• Docker Desktop installed and running
• GitHub account
• Docker Hub account
• SonarCloud account (optional but recommended)

⁠Step 1: Fork and Clone Repository

1. Fork this repository on GitHub (click "Fork" button at the top right)
2. Clone your fork locally:

git clone https://github.com/theCaptN21/docker-model-runner.git
cd docker-model-runner

Replace YOUR_USERNAME with your GitHub username.

⁠Step 2: Configure SonarCloud (Recommended)

Why SonarCloud? It provides automated code quality and security analysis on every push.

1. Sign up at https://sonarcloud.io⁠

   • Click "Log in" → "With GitHub"
   • Authorize SonarCloud

2. Import Your Repository

   • Click "+" → "Analyze new project"
   • Select your organization (GitHub username)
   • Choose docker-model-runner
   • Click "Set Up"

3. Get Your Token

   • Click profile icon → My Account → Security
   • Generate new token:
     • Name: github-actions
     • Type: User Token
     • Expiration: No expiration (for personal projects)
   • Copy the token immediately

4. Add Secrets to GitHub

   • Go to your repository on GitHub
   • Settings → Secrets and variables → Actions
   • Click "New repository secret" and add these three secrets:

   Secret 1: SONAR_TOKEN

   • Name: SONAR_TOKEN
   • Value: Your token from step 3

   Secret 2: SONAR_PROJECT_KEY

   • Name: SONAR_PROJECT_KEY
   • Value: Copy from SonarCloud Project Information (e.g., theCaptN21_docker-model-runner)

   Secret 3: SONAR_ORGANIZATION

   • Name: SONAR_ORGANIZATION
   • Value: Copy from SonarCloud Project Information (e.g., thecaptn21)

⁠Step 3: Configure Docker Hub

1. Create Docker Hub Access Token

   • Log in to hub.docker.com
   • Account Settings → Security → New Access Token
   • Description: github-actions
   • Access permissions: Read, Write, Delete
   • Copy the token immediately

2. Add Secrets to GitHub

   • Go to Settings → Secrets and variables → Actions
   • Add two secrets:
     • DOCKERHUB_USERNAME: Your Docker Hub username
     • DOCKERHUB_TOKEN: Your access token from above

⁠Step 4: Test Locally (Optional but Recommended)

Before pushing to GitHub, verify everything works:

# Build the Docker image
make build

# Run the container
make run

# In another terminal, test the API
curl http://localhost:5000/health

curl -X POST http://localhost:5000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "The future of AI is", "max_length": 50}'

# Stop the container
make clean

⁠Step 5: Push to GitHub

git add .
git commit -m "Initial commit: Docker Model Runner"
git push origin main

⁠Step 6: Verify Pipeline

1. Go to your repository → Actions tab
2. Watch the workflow run (takes ~5-10 minutes)
3. Verify all stages pass:
   • ✅ SonarCloud Analysis (code quality)
   • ✅ Security Scan (Trivy vulnerabilities)
   • ✅ Build and Test (Docker image)
   • ✅ Push to Registry (Docker Hub)
   • ✅ Post-Deployment (documentation)

⁠Local Testing & Verification

⁠Test the Model Runner

# Start the container
docker run -d -p 5000:5000 --name model-runner YOUR_DOCKERHUB_USERNAME/docker-model-runner:latest

# Wait for model to load (check logs)
docker logs -f model-runner

# Test health endpoint
curl http://localhost:5000/health
# Expected: {"status":"healthy"}

# Test text generation
curl -X POST http://localhost:5000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Once upon a time", "max_length": 100}'

# Stop container
docker stop model-runner && docker rm model-runner

⁠Prove It's a Model Runner

Test 1: Health Check (Server Running)

curl http://localhost:5000/health

Output: {"status":"healthy"} - proves Flask API is running

Test 2: Generate Text (Model Loaded)

curl -X POST http://localhost:5000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Hello world", "max_length": 30}'

Output shows generated text - proves distilgpt2 model is loaded and generating

Test 3: Different Prompts (Model Intelligence)

# Creative writing
curl -X POST http://localhost:5000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "In a galaxy far away", "max_length": 50}'

# Technical content
curl -X POST http://localhost:5000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Machine learning is", "max_length": 50}'

Different outputs for different prompts - proves model is doing inference

⁠Make a Code Change & See It Deploy

Step 1: Modify the model prompt processing

Edit app.py, find the generate function and add this line after getting the prompt:

# Add this line to make prompts more creative
prompt = f"[CREATIVE] {prompt}"

Step 2: Test locally

docker build -t model-runner:test .
docker run -d -p 5000:5000 --name test model-runner:test

curl -X POST http://localhost:5000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Hello", "max_length": 30}'

# Notice the prompt now has [CREATIVE] prefix in the response

Step 3: Push the change

git add app.py
git commit -m "Add creative prefix to prompts"
git push origin main

Step 4: Watch automated deployment

• GitHub Actions runs automatically
• New image built and pushed to Docker Hub
• Pull the new image and see your change in production

docker pull YOUR_DOCKERHUB_USERNAME/docker-model-runner:latest
docker run -d -p 5000:5000 YOUR_DOCKERHUB_USERNAME/docker-model-runner:latest
# Test and see the [CREATIVE] prefix in responses

⁠CI/CD Pipeline Stages

⁠1. SonarCloud Analysis

• Analyzes code quality and security
• Detects bugs, vulnerabilities, code smells
• Reports technical debt
• Runs first to catch issues early

⁠2. Security Scanning (Trivy)

• Scans dependencies for known CVEs
• Reports CRITICAL, HIGH, MEDIUM, LOW vulnerabilities
• Shows which versions fix issues
• Scans both filesystem and Docker image

⁠3. Build and Test

• Multi-stage Docker build (builder + runtime)
• Security hardening (non-root user, minimal base)
• Automated health checks
• API endpoint testing
• Container verification

⁠4. Push to Docker Hub

• Multiple tags: latest, main-<sha>, branch name
• OCI-compliant metadata
• Automated on successful builds
• Available immediately after push

⁠5. Post-Deployment

• Updates Docker Hub description
• Generates build summaries
• Provides deployment commands

⁠Docker Image Features

⁠Multi-Stage Build

# Stage 1: Builder (dependencies compilation)
FROM python:3.11-slim AS builder
# Install and compile dependencies

# Stage 2: Runtime (minimal production image)
FROM python:3.11-slim
# Copy only what's needed, run as non-root

Benefits:

• 300MB smaller final image
• No build tools in production
• Reduced attack surface
• Faster deployments

⁠Security Hardening

1. Non-root User

   • Runs as appuser (not root)
   • Limited permissions
   • Prevents privilege escalation

2. Minimal Base Image

   • python:3.11-slim (only essentials)
   • Fewer packages = fewer vulnerabilities
   • Regular security updates

3. Health Checks

   • Monitors container health
   • Auto-restart on failures
   • Integration with orchestrators

4. Dependency Pinning

   • Exact versions in requirements.txt
   • Reproducible builds
   • Controlled updates

⁠Available Commands (Makefile)

make help          # Show available commands
make build         # Build Docker image locally
make test          # Build and run full test suite
make run           # Run container locally
make clean         # Stop and remove containers/images

⁠Understanding the Model Runner API

⁠Endpoints

GET /health

curl http://localhost:5000/health

Response: {"status": "healthy"}

POST /generate

curl -X POST http://localhost:5000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Your text here",
    "max_length": 50
  }'

Response:

{
  "prompt": "Your text here",
  "generated_text": "Your text here and the model continues..."
}

⁠Model Information

• Model: distilgpt2 (Hugging Face)
• Task: Text generation
• Size: ~250MB
• Type: Transformer-based language model
• Use Cases: Text completion, creative writing, content generation

⁠Monitoring & Observability

⁠View SonarCloud Dashboard

1. Go to https://sonarcloud.io/projects⁠
2. Click docker-model-runner
3. View:
   • Overview: Quality gate status
   • Issues: Bugs, vulnerabilities, code smells
   • Security: Security hotspots
   • Measures: Metrics and coverage

⁠View Security Scan Results

1. Go to repository → Actions tab
2. Click latest workflow run
3. Expand "Security Scanning" job
4. View Trivy output with CVE details

⁠Check Docker Hub

1. Go to hub.docker.com
2. Navigate to your repository
3. View available tags
4. Check image layers and size

⁠Troubleshooting

⁠Pipeline Fails on First Run

SonarCloud Error:

• Ensure SONAR_TOKEN secret is added
• Verify project name is docker-model-runner in SonarCloud
• Check organization name matches your GitHub username

Docker Push Error:

• Verify DOCKERHUB_USERNAME and DOCKERHUB_TOKEN secrets
• Ensure Docker Hub token has write permissions
• Repository will be created automatically on first push

⁠Container Won't Start

# Check logs
docker logs model-runner

# Common issues:
# - Port 5000 already in use: use -p 5001:5000
# - Model download failed: check internet connection
# - Memory issues: Docker Desktop needs 4GB+ RAM

⁠Model Takes Long to Load

First run downloads the model (~250MB):

• Wait 30-60 seconds
• Check logs: docker logs -f model-runner
• Look for "Model loaded successfully"

⁠Best Practices

1. Always test locally before pushing
2. Monitor pipeline results
3. Keep dependencies updated
4. Use meaningful commit messages
5. Review code quality

⁠Security Considerations

Current dependencies (all vulnerabilities fixed):

flask==3.1.0          # Latest stable
transformers==4.48.0  # All CVEs fixed
torch==2.5.1          # Stable version
numpy==2.1.0          # Latest stable