Sign inSign up
refcollector logo

pegi3s/refcollector

By pegi3s

•Updated 13 days ago

Image
0

56

pegi3s/refcollector repository overview

⁠This image belongs to a larger project called Bioinformatics Docker Images Project (http://pegi3s.github.io/dockerfiles⁠)

⁠(Please note that the original software licenses still apply)

⁠refcollect

refcollect collects the literature on a topic from PubMed, applies your limits (publication date, minimum number of citations, reviews in/out/only, preprints such as bioRxiv in/out/only, maximum number of articles), optionally asks an LLM to screen each title/abstract for relevance, downloads the open-access PDFs, and lists every selected article for which no PDF could be retrieved.

It is a deterministic pipeline with only two LLM steps: writing the PubMed query and, optionally, screening abstracts. Every run records the exact query and the number of articles removed at each step, so the selection is reproducible.

⁠Requirements

The LLM steps require the use of:

  1. /opt/.apikey: a plain-text file containing only your API key (DeepSeek or any other OpenAI-compatible provider):

  2. config.json in the directory you mount as /data (your working directory) with the following format:

    {
      "provider_id":   "deepseek",
      "provider_name": "DeepSeek",
      "npm":           "@ai-sdk/openai-compatible",
      "base_url":      "https://api.deepseek.com/v1",
      "model_id":      "deepseek-chat",
      "model_name":    "DeepSeek Chat",
      "email":         "[email protected]"
    }
    

    If config.json is missing, DeepSeek (https://api.deepseek.com/v1, model deepseek-chat) is assumed.

Without an API key, refcollect still works if you give it the PubMed query yourself with --query or --query-file, but --screen is not available.

⁠Using the refcollect image in Linux

You should adapt and run the following command:

docker run --rm -it -v /your/data/dir:/data -v /opt:/opt pegi3s/refcollect "<your topic>" <options>

In this command, you should replace:

  • /your/data/dir with the directory that contains your config.json. Results are written there too.
  • "<your topic>" with your research topic in plain language, in quotes.

-it lets refcollect show you the PubMed query it built and ask for confirmation before collecting ([Y]es / [n]o / [e]dit query). To run unattended, for example in a script, drop -it and add -y.

⁠Examples

The 100 most recent research articles (no reviews) from the last 5 years on GULO evolution in animals, cited at least 10 times, screened for relevance by the LLM:

docker run --rm -it -v /your/data/dir:/data -v /opt:/opt pegi3s/refcollect "GULO gene evolution in animals" --years 5 --min-citations 10 --reviews exclude --max-articles 100 --screen

To check the query and the number of PubMed hits before collecting anything:

docker run --rm -it -v /your/data/dir:/data -v /opt:/opt pegi3s/refcollect "GULO gene evolution in animals" --years 5 --dry-run

Reviews only, from the last 10 years, without preprints:

docker run --rm -it -v /your/data/dir:/data -v /opt:/opt pegi3s/refcollect "GULO gene evolution in animals" --years 10 --reviews only --preprints exclude

To use your own PubMed query instead of the LLM's (no API key needed, and no topic needed):

docker run --rm -v /your/data/dir:/data pegi3s/refcollect -y --query '(GULO[tiab] OR "gulonolactone oxidase"[tiab]) AND (evolution[tiab] OR pseudogene*[tiab])'

To reuse a query that DeepSeek wrote in an earlier run, without sending a description to DeepSeek again (see Reusing a query⁠):

docker run --rm -it -v /your/data/dir:/data -v /opt:/opt pegi3s/refcollect --query-file refcollect_gulo_gene_evolution_in_animals_20261103-1201/query.txt --years 3 --reviews only

To see the full help:

docker run --rm pegi3s/refcollect --help

⁠Options

OptionMeaning
--years NOnly articles published in the last N years
--from-year YYYY / --to-year YYYYExplicit publication-year range (instead of --years)
--reviews include|exclude|onlyKeep reviews (default), exclude them, or keep only reviews. Systematic reviews count as reviews.
--preprints include|exclude|onlyKeep preprints (default), exclude them, or keep only preprints. Covers every preprint server indexed by PubMed (bioRxiv, medRxiv, Research Square, arXiv, SSRN...). With exclude, a journal article's PDF is also never replaced by its preprint.
--min-citations NKeep only articles cited at least N times
--citation-source icite|openalexWhere citation counts come from (default icite, see below)
--screenAsk the LLM to classify each title/abstract as include / exclude / uncertain, with a short reason
--drop-uncertainWith --screen, also drop the "uncertain" ones (they are kept by default)
--max-articles NKeep only the N most recent articles that pass all the other criteria
--query "..."Use this PubMed query verbatim instead of asking the LLM. Don't add date, review or preprint limits to it; use the options above. The topic then becomes optional.
--query-file FILERead the PubMed query from a file instead of asking the LLM: either the query.txt of a previous run, or a plain-text file with the query. The topic then becomes optional.
--out DIROutput folder (default refcollect_<topic>_<date>-<time>)
--no-pdfCollect metadata only
--threads NParallel PDF downloads (default 4)
--email ADDRESSYour e-mail address
--ncbi-api-key KEYOptional NCBI API key⁠ for faster PubMed access (or env NCBI_API_KEY)
--openalex-api-key KEYOptional OpenAlex API key (or env OPENALEX_API_KEY)
--config FILE / --api-key-file FILEOther locations for config.json / the API key
-y, --yesDon't ask to confirm the query
--dry-runShow the query and hit count, then stop
-d, --debugPrint the LLM responses and every PDF download attempt
⁠How --max-articles works

PubMed results are processed from newest to oldest. Each article goes through the citation filter and then, if requested, the LLM screening, and collection stops as soon as N articles have passed. The N articles are therefore the N most recent ones that meet all criteria. Articles excluded along the way don't count toward N.

The N articles all appear in included.csv and the reference files, whether or not their PDF could be downloaded. The ones without a PDF are also listed in missing_pdfs.csv.

⁠Reusing a query

Every run saves the query it used in query.txt. Pass that file to --query-file to search with the same query again, for example with other limits or a few months later. In that case:

  • No topic is needed. If you don't give one, the topic recorded in query.txt is reused locally (for the output folder name and screening); nothing is sent to DeepSeek to build a query.
  • The limits come from the new run, not the old one. refcollect reads the Base query: line, which has no date, review or preprint limits. So a query first used with --reviews exclude can be rerun with --reviews only.
  • Plain-text files work too: a file containing just a PubMed query. It can span several lines (they are joined), and lines starting with # are ignored. Don't paste in the "PubMed's interpretation" text: it already contains the date and preprint/review limits of the run it came from, which would then stay fixed whatever --years, --reviews or --preprints you choose (refcollect warns you if a query contains such limits). Copy the Base query: line instead.
  • Use --query-file FILE, not --query FILE: --query takes the query text itself. If you give it a file name, refcollect stops with an error instead of searching PubMed for the file name.
  • DeepSeek is contacted only if you add --screen. With no topic at all, the LLM screens each abstract against the query itself.
⁠Citation counts

Google Scholar has no API and blocks automated access, so refcollect doesn't use it. Citation counts come from NIH iCite⁠ (default, built for PubMed articles) or OpenAlex⁠. Both are usually lower than Google Scholar's counts, because Scholar also counts theses, preprints and grey literature, but they don't change between runs. If you are used to thinking in Google Scholar numbers, use a lower threshold. Very recent articles may not yet be in iCite and count as 0 citations.

⁠Why an e-mail address?

NCBI asks that tools identify themselves, and Unpaywall, one of the PDF sources, refuses requests without an e-mail address. Give it with --email, the REFCOLLECT_EMAIL environment variable, or "email" in config.json. Without it, Unpaywall is skipped and fewer PDFs are found.

⁠Output

Everything goes to the output folder under /your/data/dir:

FileContent
query.txtDate, topic, the base query (reusable with --query-file), the exact PubMed query with all limits (and PubMed's interpretation of it), the filters used, and how many articles each step removed
included.csvThe selected articles: PMID, date, authors, title, journal, DOI, PMCID, publication types, citations, LLM decision and reason, PDF file, its source, whether it is a preprint copy (pdf_is_preprint), or reason for no PDF
excluded.csvArticles examined but rejected, with the reason (too few citations, or the LLM's reason)
missing_pdfs.csvSelected articles for which no PDF could be downloaded, with the reason
pdfs/The PDFs, named <year>_<FirstAuthor>_<PMID>.pdf
references.risFor Zotero, EndNote and Mendeley (each record links to its PDF)
references.bibFor BibTeX/LaTeX
⁠Where the PDFs come from

Only legal open-access copies are downloaded, tried in this order:

  1. The PMC Open Access dataset⁠ (NCBI's official copy on AWS)
  2. Europe PMC
  3. Unpaywall⁠ open-access locations
  4. OpenAlex⁠ open-access locations

Every file is checked to be a real PDF before it is saved. Paywalled articles, and open-access articles whose publisher blocks automated downloads, end up in missing_pdfs.csv with the DOI and PubMed link, so you can download them by hand through your institution's access.

⁠Preprint copies of journal articles

For a paper published in a journal, the only free copy is often the preprint it started as (e.g. on bioRxiv), and Unpaywall/OpenAlex sometimes label such copies as accepted manuscripts. refcollect recognises them by their address (bioRxiv, medRxiv, Research Square, arXiv, SSRN, Preprints.org, ChemRxiv... and their DOIs) and by the preprint stamp on the first page ("bioRxiv preprint", "not certified by peer review"...). What happens then depends on --preprints:

--preprintsJournal article whose only free copy is a preprint
include (default)The journal version is tried first; the preprint copy is used only as a last resort, and marked yes in the pdf_is_preprint column of included.csv
excludeThe preprint copy is not downloaded. The article goes to missing_pdfs.csv, with the link to the preprint in the reason
onlythe selected articles are themselves preprints

Re-running with --preprints exclude into an existing output folder also removes preprint copies saved there by an earlier run.

Re-running with the same --out folder skips PDFs that are already there.

Tag summary

Content type

Image

Digest

sha256:4ca7befa7…

Size

47.8 MB

Last updated

13 days ago

docker pull pegi3s/refcollector