refcollect collects the literature on a topic from PubMed, applies your limits (publication date, minimum number of citations, reviews in/out/only, preprints such as bioRxiv in/out/only, maximum number of articles), optionally asks an LLM to screen each title/abstract for relevance, downloads the open-access PDFs, and lists every selected article for which no PDF could be retrieved.
It is a deterministic pipeline with only two LLM steps: writing the PubMed query and, optionally, screening abstracts. Every run records the exact query and the number of articles removed at each step, so the selection is reproducible.
The LLM steps require the use of:
/opt/.apikey: a plain-text file containing only your API key (DeepSeek or any other OpenAI-compatible provider):
config.json in the directory you mount as /data (your working directory) with the following format:
{
"provider_id": "deepseek",
"provider_name": "DeepSeek",
"npm": "@ai-sdk/openai-compatible",
"base_url": "https://api.deepseek.com/v1",
"model_id": "deepseek-chat",
"model_name": "DeepSeek Chat",
"email": "[email protected]"
}
If config.json is missing, DeepSeek (https://api.deepseek.com/v1, model deepseek-chat) is assumed.
Without an API key, refcollect still works if you give it the PubMed query yourself with --query or --query-file, but --screen is not available.
You should adapt and run the following command:
docker run --rm -it -v /your/data/dir:/data -v /opt:/opt pegi3s/refcollect "<your topic>" <options>
In this command, you should replace:
/your/data/dir with the directory that contains your config.json. Results are written there too."<your topic>" with your research topic in plain language, in quotes.-it lets refcollect show you the PubMed query it built and ask for confirmation before collecting ([Y]es / [n]o / [e]dit query). To run unattended, for example in a script, drop -it and add -y.
The 100 most recent research articles (no reviews) from the last 5 years on GULO evolution in animals, cited at least 10 times, screened for relevance by the LLM:
docker run --rm -it -v /your/data/dir:/data -v /opt:/opt pegi3s/refcollect "GULO gene evolution in animals" --years 5 --min-citations 10 --reviews exclude --max-articles 100 --screen
To check the query and the number of PubMed hits before collecting anything:
docker run --rm -it -v /your/data/dir:/data -v /opt:/opt pegi3s/refcollect "GULO gene evolution in animals" --years 5 --dry-run
Reviews only, from the last 10 years, without preprints:
docker run --rm -it -v /your/data/dir:/data -v /opt:/opt pegi3s/refcollect "GULO gene evolution in animals" --years 10 --reviews only --preprints exclude
To use your own PubMed query instead of the LLM's (no API key needed, and no topic needed):
docker run --rm -v /your/data/dir:/data pegi3s/refcollect -y --query '(GULO[tiab] OR "gulonolactone oxidase"[tiab]) AND (evolution[tiab] OR pseudogene*[tiab])'
To reuse a query that DeepSeek wrote in an earlier run, without sending a description to DeepSeek again (see Reusing a query):
docker run --rm -it -v /your/data/dir:/data -v /opt:/opt pegi3s/refcollect --query-file refcollect_gulo_gene_evolution_in_animals_20261103-1201/query.txt --years 3 --reviews only
To see the full help:
docker run --rm pegi3s/refcollect --help
| Option | Meaning |
|---|---|
--years N | Only articles published in the last N years |
--from-year YYYY / --to-year YYYY | Explicit publication-year range (instead of --years) |
--reviews include|exclude|only | Keep reviews (default), exclude them, or keep only reviews. Systematic reviews count as reviews. |
--preprints include|exclude|only | Keep preprints (default), exclude them, or keep only preprints. Covers every preprint server indexed by PubMed (bioRxiv, medRxiv, Research Square, arXiv, SSRN...). With exclude, a journal article's PDF is also never replaced by its preprint. |
--min-citations N | Keep only articles cited at least N times |
--citation-source icite|openalex | Where citation counts come from (default icite, see below) |
--screen | Ask the LLM to classify each title/abstract as include / exclude / uncertain, with a short reason |
--drop-uncertain | With --screen, also drop the "uncertain" ones (they are kept by default) |
--max-articles N | Keep only the N most recent articles that pass all the other criteria |
--query "..." | Use this PubMed query verbatim instead of asking the LLM. Don't add date, review or preprint limits to it; use the options above. The topic then becomes optional. |
--query-file FILE | Read the PubMed query from a file instead of asking the LLM: either the query.txt of a previous run, or a plain-text file with the query. The topic then becomes optional. |
--out DIR | Output folder (default refcollect_<topic>_<date>-<time>) |
--no-pdf | Collect metadata only |
--threads N | Parallel PDF downloads (default 4) |
--email ADDRESS | Your e-mail address |
--ncbi-api-key KEY | Optional NCBI API key for faster PubMed access (or env NCBI_API_KEY) |
--openalex-api-key KEY | Optional OpenAlex API key (or env OPENALEX_API_KEY) |
--config FILE / --api-key-file FILE | Other locations for config.json / the API key |
-y, --yes | Don't ask to confirm the query |
--dry-run | Show the query and hit count, then stop |
-d, --debug | Print the LLM responses and every PDF download attempt |
--max-articles worksPubMed results are processed from newest to oldest. Each article goes through the citation filter and then, if requested, the LLM screening, and collection stops as soon as N articles have passed. The N articles are therefore the N most recent ones that meet all criteria. Articles excluded along the way don't count toward N.
The N articles all appear in included.csv and the reference files, whether or not their PDF could be downloaded. The ones without a PDF are also listed in missing_pdfs.csv.
Every run saves the query it used in query.txt. Pass that file to --query-file to search with the same query again, for example with other limits or a few months later. In that case:
query.txt is reused locally (for the output folder name and screening); nothing is sent to DeepSeek to build a query.refcollect reads the Base query: line, which has no date, review or preprint limits. So a query first used with --reviews exclude can be rerun with --reviews only.# are ignored. Don't paste in the "PubMed's interpretation" text: it already contains the date and preprint/review limits of the run it came from, which would then stay fixed whatever --years, --reviews or --preprints you choose (refcollect warns you if a query contains such limits). Copy the Base query: line instead.--query-file FILE, not --query FILE: --query takes the query text itself. If you give it a file name, refcollect stops with an error instead of searching PubMed for the file name.--screen. With no topic at all, the LLM screens each abstract against the query itself.Google Scholar has no API and blocks automated access, so refcollect doesn't use it. Citation counts come from NIH iCite (default, built for PubMed articles) or OpenAlex. Both are usually lower than Google Scholar's counts, because Scholar also counts theses, preprints and grey literature, but they don't change between runs. If you are used to thinking in Google Scholar numbers, use a lower threshold. Very recent articles may not yet be in iCite and count as 0 citations.
NCBI asks that tools identify themselves, and Unpaywall, one of the PDF sources, refuses requests without an e-mail address. Give it with --email, the REFCOLLECT_EMAIL environment variable, or "email" in config.json. Without it, Unpaywall is skipped and fewer PDFs are found.
Everything goes to the output folder under /your/data/dir:
| File | Content |
|---|---|
query.txt | Date, topic, the base query (reusable with --query-file), the exact PubMed query with all limits (and PubMed's interpretation of it), the filters used, and how many articles each step removed |
included.csv | The selected articles: PMID, date, authors, title, journal, DOI, PMCID, publication types, citations, LLM decision and reason, PDF file, its source, whether it is a preprint copy (pdf_is_preprint), or reason for no PDF |
excluded.csv | Articles examined but rejected, with the reason (too few citations, or the LLM's reason) |
missing_pdfs.csv | Selected articles for which no PDF could be downloaded, with the reason |
pdfs/ | The PDFs, named <year>_<FirstAuthor>_<PMID>.pdf |
references.ris | For Zotero, EndNote and Mendeley (each record links to its PDF) |
references.bib | For BibTeX/LaTeX |
Only legal open-access copies are downloaded, tried in this order:
Every file is checked to be a real PDF before it is saved. Paywalled articles, and open-access articles whose publisher blocks automated downloads, end up in missing_pdfs.csv with the DOI and PubMed link, so you can download them by hand through your institution's access.
For a paper published in a journal, the only free copy is often the preprint it started as (e.g. on bioRxiv), and Unpaywall/OpenAlex sometimes label such copies as accepted manuscripts. refcollect recognises them by their address (bioRxiv, medRxiv, Research Square, arXiv, SSRN, Preprints.org, ChemRxiv... and their DOIs) and by the preprint stamp on the first page ("bioRxiv preprint", "not certified by peer review"...). What happens then depends on --preprints:
--preprints | Journal article whose only free copy is a preprint |
|---|---|
include (default) | The journal version is tried first; the preprint copy is used only as a last resort, and marked yes in the pdf_is_preprint column of included.csv |
exclude | The preprint copy is not downloaded. The article goes to missing_pdfs.csv, with the link to the preprint in the reason |
only | the selected articles are themselves preprints |
Re-running with --preprints exclude into an existing output folder also removes preprint copies saved there by an earlier run.
Re-running with the same --out folder skips PDFs that are already there.
Content type
Image
Digest
sha256:4ca7befa7…
Size
47.8 MB
Last updated
13 days ago
docker pull pegi3s/refcollector