pleosoft/categorize-transformer

•Updated over 1 year ago

Image

153

pleosoft/categorize-transformer repository overview

This example is not meant for production usage. It is showing how you could leverage different models to build an opinionated categorization api.

⁠Hierarchical Text Categorization System

A FastAPI-based system for categorizing text using TF-IDF and Naive Bayes with support for hierarchical category groups.

⁠Features

Hierarchical Category Support: Organize categories under different labels/groups
Adaptive Model Training: TF-IDF parameters adjust based on corpus size
Incremental Learning: Train and refine models over time with new data
Confidence Scoring: Get probability scores for each prediction
Top-N Filtering: Limit results to highest confidence predictions
Training Corpus Management: Save and load the training data for reuse

⁠Technologies:

FastAPI: Web framework for building the API
Transformers: Hugging Face's transformers library for NLP models
Docker: Containerization for easy deployment
PyTorch: Backend framework to run transformer models

⁠Running the API

The API will be available at http://localhost:8000⁠.

⁠Build the Docker image

docker build -t categorization-api .

⁠Run the container

docker run -p 8000:8000 categorization-api

⁠API Documentation

Once the server is running, visit http://localhost:8000/docs⁠ for the interactive Swagger UI documentation or http://localhost:8000/redoc⁠ for the ReDoc interface.

⁠Endpoints

⁠`/refine` (POST)

This endpoint refines the model with new data. If no model exists, it will create and train a new one; otherwise, it will update the existing model with new data. The model can be refined over time with multiple requests and new text-category pairs.

The refine method in the model class updates the model’s training data, enabling it to improve over time. For TFIDFModel, it retrains the model with the new data, while for BERTModel, this is a placeholder since fine-tuning BERT requires much more advanced handling.

{
  "texts": ["This is a sample text about technology", "Another text about healthcare"],
  "categories": [
    [{"label": "domain", "category": "tech"}, {"label": "topic", "category": "general"}],
    [{"label": "domain", "category": "healthcare"}]
  ]
}

⁠`/categorize` (POST)

Categorize texts using the trained model.

{
  "items": [
    {"key": "doc1", "text": "A document about artificial intelligence"},
    {"key": "doc2", "text": "Information about healthcare policies"}
  ],
  "labels_to_consider": ["domain", "topic"],
  "confidence_threshold": 0.4,
  "top_n": 3
}

⁠`/feedback` (POST)

This endpoint allows you to provide feedback on predictions made by the model. This feedback can later be used to adjust or retrain the model. This could eventually be used to trigger some form of online learning or to manually fine-tune the model later, mainly it submit corrections to improve the model.

{
  "texts": ["Document discussing machine learning algorithms"],
  "true_categories": [
    [{"label": "domain", "category": "tech"}, {"label": "topic", "category": "machine_learning"}]
  ]
}

⁠`/labels` (GET)

Get all known labels and their associated categories.

⁠`/status` (GET)

Check if the model is trained and get label information.

⁠`/corpus` (DELETE)

Clear the training corpus and reset the model.

⁠`/corpus/info` (GET)

Get information about the current training corpus.

⁠Usage Examples

⁠1. Training the Initial Model

import requests

# Train the model with initial data
response = requests.post(
    "http://localhost:8000/refine",
    json={
        "texts": [
            "Artificial intelligence is transforming industries",
            "Healthcare costs continue to rise in America",
            "New software development techniques improve productivity"
        ],
        "categories": [
            [
                {"label": "domain", "category": "tech"},
                {"label": "topic", "category": "ai"}
            ],
            [
                {"label": "domain", "category": "healthcare"},
                {"label": "topic", "category": "economics"}
            ],
            [
                {"label": "domain", "category": "tech"},
                {"label": "topic", "category": "software_dev"}
            ]
        ]
    }
)
print(response.json())

⁠2. Categorizing New Text

import requests

# Categorize new texts
response = requests.post(
    "http://localhost:8000/categorize",
    json={
        "items": [
            {"key": "doc1", "text": "Machine learning models require good data"},
            {"key": "doc2", "text": "Hospital patient care guidelines updated"}
        ],
        "confidence_threshold": 0.3,
        "top_n": 2
    }
)
print(response.json())

⁠Design Choices

⁠Text Preprocessing

The system uses NLTK for text preprocessing
Preprocessing steps include lowercasing, tokenization, lemmatization, and stopword removal

⁠Adaptive TF-IDF Parameters

The system adjusts TF-IDF parameters based on corpus size:

For 1-3 documents: More lenient parameters to handle limited data
For 4-10 documents: Moderate settings
For larger datasets: Standard settings

⁠Model Architecture

Each label has its own multi-output classifier
Uses Multinomial Naive Bayes for classification
MultiLabelBinarizer handles multiple categories per label

corpus refine:

{
    "texts": [
        "This is a document about technology and computers.",
        "This document discusses various financial investments.",
        "This text is about mobile phones and technology.",
        "This article covers stock markets and financial planning.",
        "Apple announces new iPhone with improved camera and battery life"
    ],
    "categories": [
        [
            {"label": "{http://www.alfresco.org/model/content/1.0}description", "category": "document"},
            {"label": "topic", "category": "computer"}
        ],
        [
            {"label": "{http://www.alfresco.org/model/content/1.0}description", "category": "finance"},
            {"label": "topic", "category": "banking"}
        ],
        [
            {"label": "{http://www.alfresco.org/model/content/1.0}description", "category": "technology"},
            {"label": "topic", "category": "mobile"}
        ],
        [
            {"label": "{http://www.alfresco.org/model/content/1.0}description", "category": "finance"},
            {"label": "topic", "category": "investment"}
        ],
        [
            {"label": "{http://www.alfresco.org/model/content/1.0}description", "category": "technology"},
            {"label": "topic", "category": "camera"}
        ]
    ]
}

categorize:

{
    "items": [
        { "key": "text1", "text": "Apple has new device with improved camera and battery life" },
        { "key": "441155", "text": "This document discusses various financial investments." }
    ],
    "labels_to_consider": ["{http://www.alfresco.org/model/content/1.0}description", "topic"], 
    "confidence_threshold": 0.4, 
    "top_n":1  
}

Tag summary

Recent tags

Content type

Image

Digest

sha256:f37ca8b58…

Size

105.2 MB

Last updated

over 1 year ago

docker pull pleosoft/categorize-transformer:1.0.0