biohazardious/marquee repository overview

⁠Marquee — the MAME library manager

A MAME library manager in the shape of Sonarr/Radarr: you declare what you want, it acquires only that, keeps it categorised, and moves it forward a version at a time without re-downloading the world.

Pleasuredome publishes MAME as two enormous torrents — 163 GB of ROMs and 1.13 TB of CHDs. Most of it is machines that do not work, are prototypes, are mechanical, or are in genres you would never play. Marquee works out which files you actually want, tells the download client to fetch only those, and builds a categorised library out of the result.

Upgrading is cheaper still. Moving a 0.282 library to 0.289 — seven releases — changes 244 machines and adds 877, which is 4.46 GB rather than 163 GB.

Those are measured figures, not estimates. See docs/ARCHITECTURE.md⁠ for how they are arrived at and why the approach works.

It is built for EmulationStation systems like Batocera, RecalBox and RetroPie, but works anywhere you need a categorised MAME set.

⁠At a glance

• Choose by genre, category or single game. One tree, shaped like the folders on your console, with the game count and the gigabytes each choice costs.
• Fetch only what you chose. Marquee narrows the Pleasuredome torrents in qBittorrent to the files you picked, prices it before anything starts, and never deselects a file you are already seeding.
• Upgrade a release at a time. It compares two MAME catalogues and fetches only the machines whose bytes changed.
• Prove what is on the console. Every zip is read against the release's CRCs, and every disk can be hashed against the torrent's own pieces — a copy that stopped half-way is found, not assumed.
• Write straight to the console. A local folder, SMB, FTP or SFTP; a gamelist.xml that keeps EmulationStation's favourites and play counts; title screens from libretro-thumbnails, matched by name, no scraper.
• Run it anywhere. One Docker image for amd64 and arm64, no database, a web UI that works from a phone. See Running it⁠.

⁠Screenshots

Overview — the four stages from source folder to console, each with the number that matters, and one recommendation for what to do next.	Every game — title screen, year, manufacturer, controls, screen, the folder it goes to, and whether its files match the release.
Selection — tick a genre, a category or one game; the counts and sizes are what that choice actually costs.	Transfer — what the next run will do, before it does it: 10,017 games left alone, 5 files renamed rather than copied again.
Wanted — what the selection still lacks, priced from the release torrent, one button away from the download client.	On a phone — the same pages, laid out for a small screen.

⁠Contents

At a glance⁠ · Screenshots⁠ · Running it⁠ · The pages⁠ · Starting from nothing⁠ · Keeping a library current⁠ · What it does⁠ · How it works⁠ · Command line⁠ · Development⁠ · Credits⁠ · Licence⁠

⁠Running it

The image is on Docker Hub as biohazardious/marquee and on the GitHub Container Registry as ghcr.io/biohazardious/marquee (amd64 and arm64, the same build and tags on both), built by the CI workflow⁠ from every push to master and every v* tag:

cp .env.example .env        # point CONFIG, DOWNLOADS and LIBRARY at your folders
docker compose pull         # or `docker compose build` to build from this checkout
docker compose up -d
docker compose logs         # the first line has the URL, with the API key in it

Without compose:

docker run -d --name marquee -p 8585:8585 \
  -v /path/to/config:/config -v /path/to/downloads:/downloads \
  -v /path/to/library:/library -e PUID=$(id -u) -e PGID=$(id -g) \
  biohazardious/marquee:latest

Tags: latest follows master; a release v0.4.0 is also 0.4.0 and 0.4; every build carries its short commit as sha-abc1234.

⁠On a NAS

TrueNAS SCALE (24.10 or later): Apps → Discover Apps → ⋮ → Install via YAML, and paste deploy/truenas/custom-app.yaml⁠ with its three host paths changed to your datasets. It runs as TrueNAS's own apps user (568), the one its qBittorrent app uses, so both see the torrent folder the same way. A catalog app for Discover Apps is in deploy/truenas/catalog⁠, tested with the catalog's own CI.

Unraid: save deploy/unraid/marquee.xml⁠ to /boot/config/plugins/dockerMan/templates-user/my-marquee.xml, then Docker → Add Container → Template: Marquee. The defaults follow Unraid's conventions (appdata, PUID 99, PGID 100).

Anywhere else that runs containers -- Synology, QNAP, Portainer, a Raspberry Pi -- the compose file above is all it needs: one container, three folders, one port.

Open the URL it prints. The key is stored in the browser after the first visit, so plain http://localhost:8585/ works from then on; it also lives in config/api_key if you need it again. A server bound to localhost only asks for no key at all.

Where the key comes from, in order: the MARQUEE_API_KEY environment variable if it is set (the thing to use on a NAS, where the container's log is the awkward place to fish it out of); otherwise api_key in the config volume, generated on the first start and kept across restarts. Change the variable and the old key stops working; clear it and the file's key is back in force.

The running version and build (v0.5.0@7c17217, or local for an image built by hand) are at the bottom of the sidebar and on the System page, which also says whether a newer release has been tagged -- checked against GitHub twice a day, with nothing downloaded.

Or without Docker:

pip install -e .
marquee --web                                # settings.ini beside the package
marquee --web --config ~/.config/marquee/    # or a config directory

--config takes either the settings file or the directory holding it, which is what the container mounts at /config. Outside Docker the UI is on port 8777.

Requires Python 3.9 or newer. The only hard dependency is pysmb; paramiko is needed for SFTP and is an optional extra.

⁠The pages

A dark web UI with nine pages, in the order the work happens:

• Overview — where the library stands and what to do next. The four stages (source folder → selection → fetch → library) as cards with the number that matters on each, one recommendation at the top worked out from the same figures ("2.9 GB ready to go into the library", "your settings moved on — rebuild the plan"), the health of the setup, the free space the next transfer needs, and the last runs. It is the page the app opens on. The topbar pill says the same thing from every other page, and the status bar under it says what is happening right now -- "Transferring 1.1 GB of 2.9 GB · 118 MB/s · 15s left · mslug.zip", "Checking the library 4,210 of 10,223", "Stopped: no space left on device" -- with the download client beside it: unreachable and why, connected, or "Downloading 1 release · 11.9 MB/s · 38 GB left · 55m left".
• Library — every machine the selection covers, downloaded or not, as a poster grid or a dense table. On a fresh install that is the whole catalogue: nothing is on disk, and the point of the page is choosing what should be. Title screens come from libretro-thumbnails, matched by MAME's own description, so there is no scraper and no lookup table; Artwork downloads them all so the page loads from disk and works offline, and the grid redraws itself when it finishes. Filter by genre, by sync state, or by whether a title is adult. Click a game for year, manufacturer, players, controls, screen type and resolution, where it will be filed, its files and its other revisions. Each game has its own link (#library/mslug). Filter by condition, too: everything listed passes MAME's working filter, but 3,229 of the 11,993 run with something imperfect about them and each one says what. Download asks the client for exactly what the filters are showing — it prices it first, and nothing is fetched until you say so.
• Selection — one tree shaped like the library on disk: genre → category → game, and then a branch named after your adult folder holding the same genres and categories underneath it. catlist marks adult categories in the category name, so the two halves are genuinely different categories filed in different folders, and each side ticks without disturbing the other. Every row is a tick box with its game count and size. Untick a genre, a category, or a single game; tick it again to put it back, then Save selection. The tick box decides; clicking the game's name opens it. The counts are what the selection asks for, not what happens to be downloaded, so the tree means something on a fresh install. It takes the library's filters too — genre, condition, adult, downloaded or not — and when one is set the tree becomes a flat list of what matched, with Leave all out / Put all back acting on the whole match rather than the page of it. Dropping every imperfectly emulated game is 11,993 → 8,764 games and 494.9 → 322.6 GB. Only the largest 500 games of a category are drawn, but every tick box covers all of them. Unticking something that is already on the console is a deletion waiting to happen, so the heading says so at once -- "· 312 in the library · 4.1 GB to delete" -- rather than only under Delete on the Transfer page after a rebuild. Nothing goes until that page's "Also delete" box is ticked.
• Wanted — the games the selection asks for that are not on disk yet, grouped by genre and priced from the release torrent, with a button that asks the download client for exactly those. Also where you move the library to a newer MAME release: it compares the two catalogues and fetches only the machines whose bytes actually changed. A partly-downloaded set is the normal state of things, so this gets a page rather than a wall of names in a log.
• Left out — everything that is not in the library, in the same tree as the library. Six reasons: five are the working filters (a driver that does not run, a BIOS set, a prototype) and are there so the question has an answer; the sixth is the games you excluded by name, and those have a Put back on them. Excluding a game used to be a one-way door — it was dropped before anything else ran, so it appeared in no list, no search and no tree.
• Transfer — what the next run will do, before it does it: what is copied, what is replaced, what is merely renamed, what would be deleted and what is left alone. Every row opens into the games and files it means. See What a transfer will do⁠.
• Activity — the running job, the download queue, and the log.
• Settings — media management, download client (with a Test button), indexer. Only the ROM folder and the library are required: the CHD folder defaults to the ROM folder (the disks are found inside it), and the MAME release to whatever the source folder's name says. The Library field has a Test button: for a share on the console (smb://192.168.1.20/Batocera3/roms/mame) it connects and reads the top level -- "Reached 192.168.1.20, share Batocera3: 48 entries (Music, Multiplay, …)" -- since the folder picker can only walk this machine's own disks.
• System — which XML and catlist are in use, what the library on disk records, Check the library, and the breakdown of every machine the filters dropped. On 0.289 that is 4,323 of 16,350 before your own exclusions — 2,861 with no screen at all, 949 whose driver does not work, 306 prototypes and betas, 206 BIOS and device sets.

⁠Starting from nothing

Nothing downloaded, an empty library folder, a qBittorrent that has never seen a MAME torrent. That is the case the whole thing is shaped around.

1. Settings → point Library at where the games should end up, fill in the Download client, press Test. Leave the ROM and CHD folders at the folder your client downloads into; Marquee finds the set inside it.
2. Build plan. No files are needed for this — it reads MAME's own XML and catlist.ini for the release you chose and works out the catalogue. About 12,000 games, filed by genre and category.
3. Library → filter. By genre, by search, by adult, by year — whatever narrows it to what you actually want.
4. ⬇ Download. It reads the release's file table — which means adding the torrent to your client stopped, so its own metadata arrives and none of its content — and tells you what that selection costs. 16 Metal Slug games: 936 MB of a 152 GB set. Confirm, and the client fetches those files and skips the other 44,150.
5. When it finishes, Build plan again, look at Transfer, and run it. The games land under Genre/Category/machine.zip, with a gamelist.xml and artwork for the console.

Where downloads go is qBittorrent's business, not Marquee's — it is configured there, per category. What Marquee needs is to be able to read them: if the client reports /data/torrents/... and this app sees the same files somewhere else, set a remote path mapping in Settings. The download drawer says where the client is putting things and warns when that path is not visible from here.

⁠Keeping a library current

Three questions, three answers. This is the part a plain copy tool cannot do.

⁠Is what is there still the right file?

A name match says nothing. MAME rebuilds ROM sets between releases — a bad dump replaced, a chip renamed, a mask ROM redumped — so strider2.zip from 0.252 and strider2.zip from 0.289 are the same name and different bytes. A library that only checks for existence carries those differences for ever, and that is the difference between a copy of a romset and one that is kept up to date.

Check the library, on the System page or beside the Transfer summary, works on a local folder and on an SMB or SFTP share alike (each zip's central directory is read in place; plain FTP cannot seek, so it is the one destination that cannot be checked). It reads every zip's central directory — entry names and CRC-32s, no unpacking — and compares them with the release's own ROM list, one ROM at a time. About 9 ms a game on a local disk (a 10,000-game library in under two minutes), and about five minutes for the same library on a share over the LAN.

Each game comes back as current, out of date (a ROM is there with the wrong contents, or one it owns outright is missing), missing an inherited ROM (what a merged or split set looks like from here, and not evidence of the wrong version), damaged, or absent. Anything out of date joins the download list, with the reason on the row:

NBA Showtime NBA on NBC      2.7.u27: 4242bf14 not 44a086a1
NFL Blitz 2000 Gold Edition  494_blitz_2000.u96: missing
Total Vice (ver EBA)         93c46.7k: 25aa0bd1 not 9c34554a

A real 0.252-era library of 9,904 games measured against 0.289: 9,350 current, 386 out of date, 168 missing an inherited ROM — 5.8 GB to bring up to date, against 250 GB to fetch the set again. What the check finds outranks what the file sizes say, so a redump that happens to weigh the same as the dump it replaces is still replaced.

⁠What a transfer will do

The Transfer page is the diff between the library and the selection, in five rows, each of which opens into the games and files it means:

Each kind can be read as one list or by genre -- the same genre → category → game tree the Selection page uses, so a run reads as what it does to the library ("Platform: 65 games, 38 GB, most of it Run Jump") and not as 1,144 lines.

Nothing is deleted unless the box is ticked, and every candidate says why it is one: you left it out, not in this release, a disk this release does not list for it. "Delete 917 files" is not something anyone can agree to; that list is.

The Move row is what makes an upgrade cheap. A machine's path changes between releases as well as its contents — catlist renamed Casino to Gambling in 0.289, and a clone with a disk of its own moves out of its parent's folder — and those are renames, not downloads. On a real 0.252-era library moving to 0.289 that was 954 files and 23.7 GB relocated rather than deleted and fetched again, and 939 games that stopped counting as missing.

⁠Updating a library you already have

Point Library at an existing romset and build a plan. What is already there and still wanted is counted as already there — not copied again, and not reported as something to download. A real 0.252 library came out as 7,256 of 8,764 games already present, 1,508 to fetch and the rest accounted for.

The source folder does not have to hold anything for this: the library is a statement about what you have, not only a place to put things.

⁠What it does

• Parse mameXXXX.xml, read the ROM data, and filter out non-working drivers, prototypes, betas, screenless devices, mechanical cabinets and BIOS sets
• Find the MAME XML and catlist.ini by itself, and refuse to run when the two are for different MAME versions
• Derive genres and file every machine under Genre/Category/
• Separate ROM and CHD source folders, and a separate destination folder for adult titles
• List the missing ROM and CHD files, so only what is needed is downloaded
• Download the matching catlist.ini by itself when it is not already on disk
• Exclude by ROM name, by genre or by category
• Dry run, a written report of what is missing, and a free-space check before copying
• Copy to a local folder, or to a console over SMB, FTP, FTPS or SFTP
• Sync rather than copy: work out what is new, changed, merely moved, or no longer wanted
• Relocate a recategorised machine instead of re-copying it
• Optionally remove destination files the plan no longer wants, for romset upgrades

⁠On the console

• Two layouts: every game under Genre/Category/, or every game in one folder — what MAME itself, RetroArch and most frontends read, since none of them look inside subfolders. The adult and console folders stay folders either way. Switching is a transfer of renames: on a test library, 85 games went to one folder and back with nothing copied, and the gamelist followed.

• Write a gamelist.xml at the root of the library, so EmulationStation shows Metal Slug - Super Vehicle-001, 1996, Nazca, 2 players instead of mslug — every field of it is already known, nothing is scraped

• Place the downloaded artwork beside the games

• One game, one ROM: keep each family's parent and leave its other revisions out. On MAME 0.289 that is 7,538 fewer machines and 80 GB less — Dragon's Lair alone ships four revisions at 11.5 GB each. A game whose parent did not survive the filters keeps one version, so nothing disappears.

• Preferred regions: pick World, USA, Europe (or any order you like) and a version from anywhere else is left out when the game has one from those; a game released only in Japan stays. MAME's own tags decide it — "(US, set 1)", "(German)", "(Export)" — and a country counts as its continent. On 0.289 that sets aside 1,582 versions and loses no game; 97% of the Japanese ones it finds were already on one user's hand-typed exclude list. With one game, one ROM, the first region that has a proper release is the version kept — never a bootleg over the real thing.

• What the cabinet can play: switch on the controls it has — joystick, twin sticks, wheel and pedals, spinner, trackball, light gun, mahjong and casino panel — the buttons per player and the screen's orientation. A game needing anything else goes to Left out, with the reason. "Gamepad" and "Arcade stick" presets included.

• A rating on the console: AntoPISA's bestgames.ini score goes into gamelist.xml as stars, only where the game has none — a rating you or a scraper gave is kept. The Library sorts by it too.

• The console's own MAME: Batocera ships its own, often a few releases behind the set, and older full sets are not published. Name it in Settings and every game is held to that release's XML: machines it does not have go under ZZ-Version-Mismatch/, games whose zip lacks a ROM it still asks for under ZZ-Missing-ROM/, genre and category kept inside. On a 0.289 library under Batocera's 0.285 that was 714 and 67 of 10,022 — the other 9,241 run as they are, since MAME finds a ROM by CRC. Nothing is copied or deleted: they move by rename, and move back when the console catches up.

• An ignore list for what is in the library but not Marquee's — a BIOS pack's neogeo.zip, pgm.zip and friends at the top level, for the older libretro cores. Never deleted, moved or counted. *.zip covers every zip at the top level and nothing below it; the Transfer page's delete list adds files with one click.

⁠Acquisition

• Read the Pleasuredome index and list every published set, its version