Sign inSign up

dimanjet/piicloak

By dimanjet

Updated 4 months ago

Image
1

671

dimanjet/piicloak repository overview

PIICloak

Python 3.9+ License: MIT Code style: black PRs Welcome

Enterprise-grade PII detection and anonymization API

Fast · Accurate · GDPR/CCPA Ready · 31 Entity Types

Quick Start · Documentation · Use Cases · API Reference


🎯 What is PIICloak?

PIICloak is a production-ready REST API service for detecting and anonymizing Personally Identifiable Information (PII) in text and documents. Built on Microsoft's Presidio with custom recognizers optimized for:

  • 🏢 Salesforce data (Account/Contact/Case IDs)
  • ⚖️ Legal documents (Case numbers, contracts)
  • 💰 Financial data (Bank accounts, tax IDs)
  • 🏥 Healthcare (Medical records, HIPAA compliance)
  • 💻 Technical data (API keys, IP addresses)
Why PIICloak?
FeaturePIICloakAlternatives
Entity Types31 (including custom business entities)10-15 standard types
Organization Detection✅ NER-based (works with ANY company name)❌ Pattern-only
Salesforce Support✅ Native (Account/Contact/Case/Lead IDs)❌ Not included
Legal Document Support✅ Case numbers, contracts, dockets❌ Not included
API Keys Detection✅ OpenAI, AWS, GitHub, Stripe, generic⚠️ Limited
SDK✅ Python SDK included❌ API only
One-Line Installpip install piicloak⚠️ Complex setup
Docker Ready✅ Production-grade image⚠️ Basic
Metrics✅ Prometheus built-in❌ None
Auth✅ Optional API key❌ None

🚀 Quick Start

30-Second Setup
# Install
pip install piicloak

# Run
python -m piicloak

Server starts on http://localhost:8000 🎉

Instant Test
curl -X POST http://localhost:8000/anonymize \
  -H "Content-Type: application/json" \
  -d '{"text": "Email [email protected], SSN 123-45-6789"}'

Response:

{
  "anonymized": "Email <EMAIL_ADDRESS>, SSN <US_SSN>",
  "entities_found": [
    {"type": "EMAIL_ADDRESS", "text": "[email protected]", "score": 1.0},
    {"type": "US_SSN", "text": "123-45-6789", "score": 0.85}
  ]
}
Docker
docker run -p 8000:8000 dimanjet/piicloak
Python SDK
from piicloak import PIICloak

cloak = PIICloak()
result = cloak.anonymize("Contact John Smith at [email protected]")
print(result.anonymized)  # "Contact <PERSON> at <EMAIL_ADDRESS>"

✨ Features

Supported Entity Types (31)
Entity TypeDescriptionExample
👤 PERSONAL IDENTIFIABLE INFORMATION
PERSONNames of individuals (NER-based)"John Smith", "Jane Doe"
EMAIL_ADDRESSEmail addresses"[email protected]"
PHONE_NUMBERPhone numbers (multiple formats)"+1-555-123-4567", "(555) 123-4567"
US_SSNUS Social Security Numbers"123-45-6789"
US_PASSPORTUS Passport numbers"123456789"
US_DRIVER_LICENSEUS Driver's License numbers"D1234567"
ADDRESSPhysical addresses (NER + patterns)"123 Main St, New York, NY 10001"
💳 FINANCIAL INFORMATION
CREDIT_CARDCredit card numbers (all major brands)"4532-1234-5678-9010"
IBAN_CODEInternational Bank Account Numbers"GB82 WEST 1234 5698 7654 32"
US_BANK_NUMBERUS bank account numbers"123456789012"
BANK_ACCOUNTGeneric bank account patterns"ACC-123456789"
TAX_IDTax IDs (EIN/TIN)"12-3456789"
CRYPTOCryptocurrency addresses"1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa"
🏢 ORGANIZATIONAL DATA
ORGANIZATIONCompany names (NER-based)"Acme Corp", "Tech Industries Inc"
DOMAINInternet domains"example.com", "company.io"
SALESFORCE_IDSalesforce record IDs (Account/Contact/Case/Lead)"0015000000AbcDEF", "5005000000XyzABC"
ACCOUNT_IDGeneric account identifiers"ACC-123456", "A-987654"
⚖️ LEGAL DOCUMENTS
CASE_NUMBERCourt case numbers (Federal/State)"1:24-cv-12345", "CR-2024-001234"
CONTRACT_NUMBERContract and agreement numbers"CONT-2024-001", "AGR-123456"
💻 TECHNICAL & SECURITY
USERNAMEUsernames and login IDs"john_smith123", "@johndoe", "admin"
API_KEYAPI keys (OpenAI, AWS, GitHub, Stripe, generic)"sk-1234567890abcdef...", "ghp_abc..."
IP_ADDRESSIPv4 and IPv6 addresses"192.168.1.1", "2001:0db8::1"
URLWeb URLs"https://example.com/page"
🏥 HEALTHCARE & OTHER
MEDICAL_LICENSEMedical license numbers"MD-123456"
UK_NHSUK NHS numbers"123 456 7890"
NRPNúmero de Registro de Personas (Spanish ID)"12345678A"
LOCATIONGeographic locations (NER-based)"New York", "San Francisco"
DATE_TIMEDates and timestamps"2024-01-20", "January 20th, 2024"

Total: 31 entity types covering personal, financial, organizational, legal, technical, and healthcare data.

Anonymization Modes
# Replace with entity type (default)
{"mode": "replace"} → "Contact <PERSON> at <EMAIL_ADDRESS>"

# Mask with asterisks
{"mode": "mask"} → "Contact ******** at ****************"

# Redact (remove completely)
{"mode": "redact"} → "Contact  at "

# Hash (SHA256)
{"mode": "hash"} → "Contact a1b2c3d4... at e5f6g7h8..."

💼 Use Cases

Salesforce Data Protection
curl -X POST http://localhost:8000/anonymize \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Account: 0015000000AbcDEFG, Contact: Jane Doe ([email protected]), Case: 5005000000XyzABC"
  }'

Output:

Account: <SALESFORCE_ID>, Contact: <PERSON> (<EMAIL_ADDRESS>), Case: <SALESFORCE_ID>
curl -X POST http://localhost:8000/anonymize \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Case No. 1:24-cv-12345 - Plaintiff John Doe (SSN: 123-45-6789) vs. Acme Corp (EIN: 12-3456789)"
  }'

Output:

Case No. <CASE_NUMBER> - Plaintiff <PERSON> (SSN: <US_SSN>) vs. <ORGANIZATION> (EIN: <TAX_ID>)
API Keys & Secrets
curl -X POST http://localhost:8000/anonymize \
  -H "Content-Type: application/json" \
  -d '{
    "text": "OpenAI key: sk-1234567890abcdefghijklmnopqrstuv, GitHub: ghp_abcdefghijklmnopqrstuvwxyz1234567890"
  }'

Output:

OpenAI key: <API_KEY>, GitHub: <API_KEY>
.docx Files
curl -X POST http://localhost:8000/anonymize/docx \
  -F "[email protected]" \
  -F "mode=replace"

📖 Documentation

Installation
# Basic installation
pip install piicloak

# Download NLP model (required)
python -m spacy download en_core_web_lg

# Or install everything at once
pip install piicloak && python -m spacy download en_core_web_lg
Configuration

All settings use the PIICLOAK_ prefix and have sensible defaults:

Environment VariableDefaultDescription
PIICLOAK_HOST0.0.0.0Server host
PIICLOAK_PORT8000Server port (standard)
PIICLOAK_DEBUGfalseDebug mode
PIICLOAK_WORKERS4Gunicorn workers
PIICLOAK_LOG_LEVELINFOLogging level
PIICLOAK_SPACY_MODELen_core_web_lgspaCy model
PIICLOAK_SCORE_THRESHOLD0.4Min confidence score (0-1)
PIICLOAK_DEFAULT_MODEreplaceDefault anonymization mode
PIICLOAK_CORS_ORIGINS*CORS allowed origins
PIICLOAK_API_KEY""Optional API key (empty = no auth)
PIICLOAK_RATE_LIMIT100/minuteRate limiting
PIICLOAK_ENABLE_METRICStruePrometheus metrics

Example:

export PIICLOAK_PORT=9000
export PIICLOAK_API_KEY=your-secret-key
python -m piicloak

🔌 API Reference

Endpoints
POST /anonymize - Anonymize Text

Request:

{
  "text": "Contact John at [email protected]",
  "entities": ["PERSON", "EMAIL_ADDRESS"],  // optional
  "mode": "replace",                        // optional
  "language": "en",                         // optional
  "score_threshold": 0.4                    // optional
}

Response:

{
  "original": "Contact John at [email protected]",
  "anonymized": "Contact <PERSON> at <EMAIL_ADDRESS>",
  "entities_found": [...]
}
POST /analyze - Detect PII Only
curl -X POST http://localhost:8000/analyze \
  -H "Content-Type: application/json" \
  -d '{"text": "Contact [email protected]"}'
GET /entities - List Supported Entities
curl http://localhost:8000/entities
GET /metrics - Prometheus Metrics
curl http://localhost:8000/metrics
GET /health - Health Check
curl http://localhost:8000/health

🐳 Deployment

Docker
# Build
docker build -t piicloak .

# Run
docker run -p 8000:8000 piicloak

# With environment variables
docker run -p 8000:8000 \
  -e PIICLOAK_API_KEY=your-key \
  -e PIICLOAK_WORKERS=8 \
  piicloak
Docker Compose
docker-compose up -d
Production (Gunicorn)
pip install gunicorn
gunicorn -c gunicorn.conf.py "piicloak.app:create_application()"
Kubernetes

See docs/DEPLOYMENT.md for Kubernetes deployment guide.


🛠️ Development

Setup
# Clone repository
git clone https://github.com/dimanjet/piicloak.git
cd piicloak

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dev dependencies
pip install -e ".[dev]"

# Download spaCy model
python -m spacy download en_core_web_lg

# Run tests
pytest

# Run with coverage
pytest --cov=piicloak --cov-report=html

# Format code
black src/ tests/

# Lint
flake8 src/ tests/
Project Structure
piicloak/
├── src/piicloak/
│   ├── __init__.py          # PIICloak SDK class
│   ├── __main__.py          # CLI entry point
│   ├── app.py               # Application factory
│   ├── api.py               # REST API endpoints
│   ├── config.py            # Configuration
│   ├── engine.py            # Analyzer/Anonymizer setup
│   ├── recognizers.py       # Custom PII recognizers
│   ├── middleware.py        # Auth, CORS, logging
│   └── metrics.py           # Prometheus metrics
├── tests/                   # Comprehensive test suite
├── docs/                    # Documentation
├── Dockerfile               # Production Docker image
├── docker-compose.yml       # Docker Compose config
├── gunicorn.conf.py         # Gunicorn configuration
└── requirements.txt         # Dependencies

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Adding New Recognizers

To add a new PII recognizer:

  1. Add pattern(s) to src/piicloak/recognizers.py
  2. Create a factory function
  3. Add to SUPPORTED_ENTITIES
  4. Write tests in tests/test_recognizers.py
  5. Update README

Example:

def create_license_plate_recognizer() -> PatternRecognizer:
    patterns = [
        Pattern("US_PLATE", r"\b[A-Z]{2,3}[-\s]?\d{3,4}\b", 0.7),
    ]
    return PatternRecognizer(
        supported_entity="LICENSE_PLATE",
        patterns=patterns
    )

📊 Performance

  • Throughput: ~100 requests/second (single worker)
  • Latency: <100ms per request (average)
  • Memory: ~500MB (with spaCy model loaded)
  • Scalability: Stateless design, horizontally scalable

🔒 Security

  • Optional API key authentication
  • CORS configuration
  • Rate limiting support
  • Security headers included
  • No data retention
  • Stateless operation

Report security vulnerabilities to: [email protected]


📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

PIICloak is built on top of these excellent open-source projects:


🌟 Star History

If you find PIICloak useful, please consider giving it a star ⭐

Star History Chart


📫 Contact & Support


Made with ❤️ for the privacy-conscious developer community

⬆ Back to Top

Tag summary

Content type

Image

Digest

sha256:7a854448d

Size

566.6 MB

Last updated

4 months ago

docker pull dimanjet/piicloak