Sign inSign up

jncchds/analyze

By jncchds

•Updated 1 day ago

See what takes space on your drives, find duplicates and clean up safely: web UI, one container.

Image
0

2.1K

jncchds/analyze repository overview

ANalyze logo

⁠ANalyze

See what takes space on your drives, find duplicates, and clean up safely.
One Docker container (or a desktop app) with a web UI — nothing on disk changes until you approve a plan.

Docker pulls Image size Latest release Build License: GPL v3

Quick start⁠ · Desktop app⁠ · Configuration⁠ · Safety⁠ · Integrations⁠ · Development⁠

Source code, issues, desktop apps and release notes: github.com/jncchds/ANalyze⁠

⁠Features

FeatureWhat it does
📊OverviewTotals; size by category, type, extension, age and file size; largest files; "heaviest folders" (by files directly inside, so parents don't drown out the real culprits).
🗂️BrowseThe folder tree as a sized list or a treemap, with multi-select and integration labels (e.g. Plex titles). Compares each folder with the disk, so files that aren't scanned yet show up too.
🔎SearchBy name, path, category, type, extension, tag, size, age, folder name, duplicates, hard links, sparse files… Every result set can become a cleanup plan.
👯DuplicatesSame size → quick hash → full hash (xxHash128). Hard links are recognized and never counted. Similar folders finds folder pairs that share many identical files (copied backups, extracted archives).
💡InsightsDependency folders (node_modules, venvs…), caches & temp files, incomplete downloads, OS/NAS junk, files named like copies, huge/stale files, empty files and folders, broken symlinks, growth since the previous scan.
🏷️Categories & rules~35 built-in rules and 19 categories out of the box; edit them or add your own with a live preview of matching files; import/export as JSON.
✨AI (optional)Any OpenAI-compatible API (OpenAI, Ollama, LM Studio, vLLM, OpenRouter…) classifies files no rule recognizes and proposes new rules for you to accept.
👁️PreviewImages, video, audio, PDF, text/code (highlighted), Markdown (rendered), archive listings, embedded metadata (EXIF, tags), and a hex viewer for everything else.
🧹ActionsTrash, move (with path templates like Sorted/{category}/{mtime:yyyy}/{name}), delete, or replace duplicates with hard links. Every change is a plan you review first; trash and moves can be undone.
🔌IntegrationsPlex, Jellyfin, Emby, Sonarr, Radarr, Lidarr, qBittorrent, Transmission, SABnzbd, Immich, PhotoPrism, plus Docker (works out the path mappings between containers) and ANalyze parent (checks a backup kept on another ANalyze instance).
🛡️Integrity (optional)Hash every file and re-verify old hashes to catch bit rot: files whose content changed while size and date stayed the same are flagged as possibly corrupted.
📜Activity logWhat ANalyze is doing, step by step and live: scan phases with counts and times, skipped folders and why, unreadable files, integration syncs, plans and AI runs. Hover a step name in the sidebar for a short explanation.
🔗Links to every viewTabs, filters, pages, dialogs and the file panel are part of the URL: the browser's Back and Forward buttons work everywhere (Back closes a dialog first), and any view can be bookmarked or shared.
📝Release notesWhat changed in each version, right in the UI (click the version next to the logo).

⁠Quick start

services:
  analyze:
    image: jncchds/analyze:latest
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      ANalyze__Roots__0: /library
      # ANalyze__Password: "change-me"
    volumes:
      - analyze-data:/data
      - /srv/nas:/library          # read-write: actions enabled
      # - /srv/media:/media:ro     # read-only: analysis only (add ANalyze__Roots__1: /media)
volumes:
  analyze-data:

Open http://localhost:8080⁠ and press Scan now in the sidebar. With several roots, pick one in the scope selector to get Scan <root> next to Scan all; Settings also has a scan button per root. The first scan reads every folder; later scans only look at what changed (size, mtime, inode), so they are much faster. Schedule automatic scans in Settings.

Important The container runs as root so it can read — and, when allowed, move or delete — files of every user on the share. Folders it creates (move targets, trash) take the owner of their parent folder. To guarantee nothing is ever changed, mount roots with :ro or set ANalyze__ReadOnly=true.

⁠Scanning
  • Live results — results are stored while the scan runs (at least every 2 seconds), so Browse, Search and the integrations see folders as soon as they are walked.
  • Responsive while scanning — Overview, Insights and the sidebar counters show their last numbers at once and update in the background (at most once a minute), so they stay usable on slow or busy disks.
  • Walk first, the rest in the background — a scan only holds the index to itself while it walks the folders. File types, categories, archives, hashes, duplicates and re-verification then run in the background (Processing in the sidebar), next to integration syncs and the AI, and step aside whenever the next scan or a plan needs the index; they carry on afterwards where they were. Stop in the sidebar stops them; the next scan picks the work up again.
  • Other filesystems — disks, shares, ZFS datasets or btrfs subvolumes mounted below a root are entered unless Descend into other filesystems is turned off in Settings → Scanning. Then they are marked "not scanned" in Browse and listed in the activity log, and you choose the ones to enter under Other filesystems to enter (e.g. /mnt/main/media, which also covers the datasets below it) or with Enter this filesystem in Browse.
  • Ignoring — skip folders with Ignore in Settings (names like .snapshot, paths from the root like Media/Cache/**, or full paths like /mnt/main/backups) or with Ignore in Browse. What you ignore leaves the index at once, so it disappears from duplicates, search and insights without waiting for a scan (stop ignoring it and scan its folder to bring it back).
  • Priority — Settings → Scanning → Priority: Full speed, or Background: ANalyze runs at low CPU priority and gets disk time only when nothing else wants it, with one worker thread, so the server's other services stay responsive while it scans and hashes (it takes longer). In background mode the hashing and the other background work also check how busy the disks are: while something else keeps them more than half busy, they wait.
  • Not scanned yet — Browse compares each folder with the disk: what is there but not scanned yet, or ignored, is listed under the folder with buttons to scan the folder or (stop) ignoring an item. Folders in that list open, as deep as they go, read live from the disk.

⁠Desktop app

ANalyze also runs as a desktop app for Windows, macOS and Linux: the same server and UI in a window, for analyzing the disks of the computer it runs on. Download it from the GitHub releases⁠ — each version that reaches main gets a release v<VERSION> automatically.

OSFiles
🪟 Windows (x64 / arm64)ANalyze-<version>-win-x64.exe / -win-arm64.exe (installer), or .zip
🍎 macOS (Apple Silicon)ANalyze-<version>-mac-arm64.dmg, or .zip
🐧 Linux (x64 / arm64)ANalyze-<version>-linux-x86_64.AppImage / -arm64.AppImage, or .tar.gz

On first start, pick the folders or drives to analyze; add or remove them later in Settings → Roots or with File → Add folder…. The app's data (index, thumbnails, logs) is in its user-data folder (File → Open data folder). Video thumbnails need ffmpeg on the PATH (on macOS, Homebrew's is found too). Hard-link dedupe is only available on Linux.

Note The apps are not code-signed: Windows SmartScreen asks to confirm (More info → Run anyway); on macOS, if the app is reported as damaged, run xattr -cr /Applications/ANalyze.app.

The bundled server listens on 127.0.0.1 only and accepts only requests from the app's window (a random token per launch).

Build it yourself (needs .NET 10 and Node 22): cd desktop && npm ci && npm run dist → desktop/dist/. For development: npm run server once (publishes the server into desktop/server/<arch>), then npm start.

⁠Configuration

VariableDefaultMeaning
ANalyze__Roots__0, __1, …/library if it existsFolders to analyze (one per mount).
ANalyze__ReadOnlyfalseAnalysis only; no actions even on writable mounts.
ANalyze__Password—Protects the UI and API. Set it if the port is reachable by others.
ANalyze__ApiToken—Token for scripts: Authorization: Bearer <token>.
ANalyze__DataDir/dataDatabase (analyze.db, SQLite), thumbnails, logs, encryption keys. Keep it on a volume (also with PostgreSQL: the keys decrypt saved secrets).
ANalyze__Database__Providersqlitesqlite or postgres (see PostgreSQL⁠).
ANalyze__Database__ConnectionString—PostgreSQL: required, e.g. Host=db;Database=analyze;Username=analyze;Password=…. SQLite: optional, overrides the file in the data folder.
ANalyze__Database__ImportFrom—PostgreSQL: a SQLite analyze.db to copy over at startup while the PostgreSQL database has never been scanned.
ANalyze__Integrations__<id>__<field>—Seeds an integration's settings until they are saved in the UI (see Plex⁠).
TZUTCTime zone for schedules and dates.

Everything else (scan schedule, ignore patterns, duplicate search, hashing threads, hashing every file and re-verifying hashes, trash retention, AI) is set in Settings.

Build without ffmpeg (no video thumbnails, smaller image): docker build -f docker/Dockerfile --build-arg FFMPEG=false .

⁠PostgreSQL

The index lives in SQLite (/data/analyze.db) unless you choose PostgreSQL — worth it for very large libraries, when the data volume sits on a slow disk, or if you already run a PostgreSQL server. Everything works the same on both.

docker compose -f docker-compose.yml -f docker-compose.postgres.yml up -d   # ANalyze + a PostgreSQL 17 container

or point ANalyze at your own server (PostgreSQL 12 or newer, built with ICU like the official images; the user needs to be able to create tables in the database):

      ANalyze__Database__Provider: postgres
      ANalyze__Database__ConnectionString: "Host=db;Database=analyze;Username=analyze;Password=change-me"

The tables are created on first start. Keep the /data volume: thumbnails, logs and the keys that decrypt saved integration secrets stay there.

Moving an existing SQLite index to PostgreSQL (scans, hashes, rules, plans, settings — everything):

  • At startup: also set ANalyze__Database__ImportFrom: /data/analyze.db. While the PostgreSQL database has never been scanned, ANalyze copies the SQLite database into it before it starts (the log says how many rows); afterwards the setting is ignored, so it can stay. The SQLite file is left in place (brought up to date first) — delete it once you are happy.
  • Or as a one-off command, with ANalyze stopped: docker compose run --rm analyze --import-sqlite /data/analyze.db (outside Docker: ANalyze.Web --import-sqlite path/to/analyze.db). It refuses to overwrite a PostgreSQL database that holds scans unless you add --force, which replaces its contents.

To go back to SQLite, remove the two ANalyze__Database__… variables (the old analyze.db is used again as it was).

⁠How actions stay safe

  1. Selecting files and choosing Clean up… creates a draft plan — nothing changes yet. The dialog can also add the selection to an existing draft plan (the last one used is preselected); a file the plan already has gets the new action. From the file panel the panel stays open, so you can go through files one by one (‹ › or ← →) and plan each; the panel says which draft plan has the file.
  2. The review page lists every item with its target; untick anything to keep, change an item's action (or the action of every item shown), and Show opens the file in Browse. Conflicts (destination exists) are resolved by the chosen policy (rename/skip/overwrite); items on read-only roots are excluded.
  3. Apply re-checks each item right before it runs: still there, same size and mtime. Duplicate removals also re-hash both copies, so a file that changed since the scan is skipped.
  4. Trash moves items to <root>/.analyze-trash/<plan>/… (an instant rename on the same disk). It is emptied after the retention period (Settings) or from Actions → Trash. Undo restores trashed and moved items, and files replaced with the backup's copy.

⁠Duplicates

Files are grouped by size; a quick hash (first/middle/last 64 KB) filters candidates, and only colliding files get a full xxHash128. Hashes are cached and recomputed only for changed files. The dedupe wizard keeps the oldest, newest, shortest-path, deepest-path, or a copy inside preferred folders, and trashes, deletes or hard-links the others (hard links keep every path working while storing the data once).

⁠Extracted archives

Scans list the files inside zip, rar and 7z archives (from their headers, a small read) and look on every root for folders that hold them — same relative paths and sizes, also when the archive's top folder was not kept, and whatever else the folder contains. Duplicates → Extracted archives shows:

  • complete extractions (every file there) and partial ones, with the files that are missing or cut short (smaller than in the archive — an interrupted extraction);
  • Check against archive: reads the extracted files and compares each with the CRC32 the archive stores for it, so you know the copy is intact without reading the archive; files that differ are listed;
  • plans to remove the archive (complete extractions only) or the extracted copy (only the files that came from the archive).

tar, tar.gz, .gz, .xz, .zst… have no file index: turn on Also list tar and compressed files in Settings → Archives to decompress each one once (CRC32s are computed on the way; there is a size limit). Insights and the Search filter Archives list them too.

⁠Rules and categories

Rules run top to bottom on every file. Conditions combine with all/any and can test name, path, extension, MIME type, kind, size (100MB), days since modified/accessed, any parent folder name (node_modules) or depth, with operators equals / contains / starts / ends / glob / regex / one of / greater / less, optionally negated. The first rule that sets a category wins; tags from all matching rules add up. A category you set by hand (on a file or a whole folder) always wins; AI only fills gaps.

⁠AI

Settings → AI: base URL, key and model of any OpenAI-compatible server (e.g. http://ollama:11434/v1 with qwen2.5). By default only paths, names, sizes, types and dates are sent; optionally the first 2 KB of text files.

  • Classify uncategorized sends the largest unexplained files in batches. Answers above the confidence threshold are applied (and cached by content hash); the rest wait in AI → Review.
  • Suggest rules summarizes uncategorized files by extension and folder and asks for reusable rules; each suggestion shows how many files it would match before you add it.

⁠Integrations

Integrations are plugins with their own settings form, sync job, API and page. They label files and folders, and the labels appear in Browse and Search; click a label to open that integration's page at the title or folder it belongs to. Turn one on under Integrations: enabling it also adds its page to the sidebar, and disabling it hides it again.

IntegrationWhat it adds
Plex, Jellyfin, EmbyMovies, shows and music labelled in Browse; titles ranked by size and watch state (watched long ago, never watched); media in library folders the server does not know.
Sonarr, Radarr, LidarrManaged files labelled with quality; files the app does not track in its root folders (old upgrades, extras, failed imports); orphan folders; Unmonitor a title before cleaning it up so it is not downloaded again.
qBittorrent, Transmission, SABnzbdSeeding and downloading items are protected — cleanup plans skip them; finished downloads ranked by size and age; files in the download folders the client no longer knows; library files copied instead of hard-linked (fix them in Duplicates → Dedupe with hard link).
Immich, PhotoPrismLibrary photos and videos labelled; copies outside the library (exports, old backups, phone dumps); files in the library folders that were never indexed.
ANalyze parentWhich files and folders are synchronized to your backup server (when it runs ANalyze too), and which files in synchronized folders are corrupted (other size or hash there) or missing there — per folder in Browse and on the Backup page.
DockerFinds each app's container and turns the volume mounts into path mappings for the integrations above, so /movies in Radarr is matched to /library/movies here without typing anything. Labels folders that other containers mount.
⁠Path mappings

Apps in other containers see the same files under other paths. Each integration tries, in order:

  1. mappings you entered,
  2. mappings from Docker,
  3. mappings detected from sample files (name + size),
  4. file name + size per file.

A mapping you entered covers everything below its prefix (Docker never overrides part of it); for the same prefix the last one in the list wins. When ANalyze sees one host folder under two paths (/mnt/main mounted at /mnt/main, and its dataset /mnt/main/media again at /mnt/media), a mapping that leads to the path that is not scanned is followed through ANalyze's own mounts to the one that is — so /downloads → /mnt/main/media works with only /mnt/media as a root.

To use Docker, mount the socket read-only and enable Integrations → Docker:

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

Tip ANalyze only reads the container list (/containers/json). For tighter control, point it at a docker-socket-proxy⁠ with CONTAINERS=1 (tcp://docker-proxy:2375).

The app's container is found from the integration's URL (container name, compose service or IP), a published port, or its image name; name it explicitly under Container per app (radarr=radarr-4k) when several match.

The integration pages show each file three ways: the app's path, the path ANalyze looked at (after the mappings) and the file it found. Items not found are summed up in the activity log with the reason (e.g. inside a folder on another filesystem). Settings → Host paths (Docker only) tells ANalyze where its folders are on the host (/library → /mnt/media), so paths are shown with their host path too; with the Docker integration on, ANalyze's own mounts fill this in automatically.

⁠Backups between two ANalyze instances

When your NAS is backed up to another NAS and both run ANalyze, the instance being backed up (the child) can check its backup on the other one (the parent, the backup server):

  1. On the backup server, set ANalyze__ApiToken (and ANalyze__Password if the port is reachable by others).
  2. On both, turn on Settings → Integrity & backups → Hash every file (and, to catch bit rot, Re-verify hashes after N days). Without hashes, corrupted files are found by size only.
  3. On the child, open Integrations → ANalyze parent: the backup server's URL (http://backup-nas:8080), its token, and path mappings when the backup lives under another path there (/backup/nas1 → /library; leave empty to detect it from sample files, or press Suggest). Test, enable, Sync now.

How it decides:

  • a file is synchronized when the backup has a file at the same path;
  • a folder is synchronized when at least 80% of what is directly inside it (files and sub-folders, by count) is synchronized, or when it is identical to the backup's folder (so an empty folder that exists empty on the backup counts too); an empty folder that is not synchronized (only empty folders below it, missing or different on the backup) is left out of its parent's share, so it does not make the parent look less synchronized;
  • in a synchronized folder, a file is corrupted when the backup's copy has another size or content hash, and missing when the backup does not have it. Files outside synchronized folders are just not synchronized.

The Backup page shows how much is on the backup, corrupted and missing files, and a folder view with each folder's share; Browse labels every folder (Synced, Not synced · 40%) and each corrupted or missing file. The comparison runs after each scan of the child; the backup server only answers read-only requests, also while it scans. Downloading the backup server's file list doesn't wait for a scan on the child, and doesn't hold one up either.

Hashing a large library takes a long time, so the comparison doesn't wait for it: it compares sizes where a hash is still missing and uses each hash as soon as it exists. Files hashed on the child are checked against the backup batch by batch, and every 15 minutes the child fetches the hashes the backup server computed since then (backup server 0.1.10 or newer).

What happened? Next to each corrupted file, this button reads both copies completely, the child's here and the backup server's there, at the same time. No file content crosses the network, only a hash per block. It then tells you how they differ and which copy is likely the bad one:

VerdictMeaning
Identical nowSame content: an index was out of date (rescan)
Cut off / Grew since the backupOne copy is exactly the start of the other: an interrupted copy, or data appended later (the newer one is longer)
Zeros instead of dataWhere they differ, one copy holds only zeros: a copy that stopped while writing, or lost data
Silent corruptionA few blocks differ although size and date are the same: bit rot, a failing disk or RAM. If either instance had noticed the change while re-verifying, that copy is named
Header / tags changed, End changed, Middle changedThe rest is identical, also when the size changed (e.g. a tagger rewrote audio or video metadata)
Edited / Different fileScattered changes, or nothing in common (replaced, re-encoded, downloaded again); the newer copy is named

The rest of this page is on GitHub⁠.

Tag summary

Content type

Image

Digest

sha256:0a88eaddb…

Size

244.9 MB

Last updated

1 day ago

docker pull jncchds/analyze