watrall/charm-market-intelligence-engine repository overview

⁠CHARM

Source repo: https://github.com/watrall/charm-market-intelligence-engine⁠

Containerized market-intelligence pipeline + Streamlit dashboard for cultural heritage and archaeology workforce-demand analysis (job postings + PDF reports → enrichment, skills extraction, geocoding, canonical artifacts, and briefings).

⁠Overview

CHARM is a container-ready market intelligence system that converts labor-market and sector-report signals into structured artifacts and a stakeholder-facing dashboard. It ingests targeted job postings and local PDF reports, performs normalization and enrichment (skills extraction, geocoding, optional sentiment), writes canonical outputs (CSV, SQLite, JSON, Markdown brief), and serves an interactive Streamlit UI over the processed dataset.

The Dockerized deployment separates concerns:

• Pipeline container: one-shot execution that builds/refreshes the dataset and artifacts.
• Dashboard container: persistent Streamlit service reading the latest processed outputs.

Cost-controlled operation is built in:

• Demo mode serves the UI from bundled synthetic artifacts and does not scrape sources or call external APIs.
• Optional integrations (LLM brief generation, Google Sheets export) are opt-in via environment flags.

⁠Outputs

• data/processed/jobs.csv — enriched job postings
• data/processed/analysis.json — summary statistics + schema version
• data/processed/insights.md — market brief (human-readable)
• data/charm.db — SQLite store (queryable + auditable)

⁠Quickstart

Start dashboard

docker compose up --build

Run pipeline

docker compose run --rm pipeline

Demo dashboard Set DEMO_MODE=1 and start the dashboard.

⁠Key configuration flags

• DEMO_MODE=1 — run the dashboard against bundled demo artifacts
• USE_LLM=0/1 — enable or disable LLM-assisted briefing
• USE_SHEETS=0/1 — enable or disable Google Sheets export

⁠Deployment notes

• Mount data/ as a volume for persistence across runs.
• Parsing/geocoding caches reduce repeat calls and improve determinism between runs.
• The dashboard is artifact-driven (reads data/processed/*), enabling scheduled pipeline refreshes without coupling UI availability to ingestion.