osirrc2019/anserini repository overview

⁠OSIRRC Docker Image for Anserini

Ryan Clancy⁠ and Jimmy Lin⁠

This is the docker image for the Anserini⁠ toolkit (v0.5.1) conforming to the OSIRRC jig⁠ for the Open-Source IR Replicability Challenge (OSIRRC) at SIGIR 2019⁠. This image is available on Docker Hub⁠. The OSIRRC 2019 image library⁠ contains a log of successful executions of this image.

• Supported test collections: robust04, core17, core18 (newswire); gov2, cw09b, cw12b (web)
• Supported hooks: init, index, search

⁠Quick Start

The following jig command can be used to index TREC disks 4/5 for robust04:

python run.py prepare \
  --repo osirrc2019/anserini \
  --tag v0.1.1 \
  --collections robust04=/path/to/disk45=trectext

The following jig command can be used to perform a retrieval run on the collection with the robust04 test collection.

python run.py search \
  --repo osirrc2019/anserini \
  --tag v0.1.1 \
  --output out/anserini \
  --qrels qrels/qrels.robust04.txt \
  --topic topics/topics.robust04.txt \
  --collection robust04 \
  --opts search_args="-bm25" out_file_name="run.bm25.robust04"

The arguments search_args and out_file_name can be customized (i.e., search_args="-ql -rm3" and out_file_name="run.ql.rm3.robust04")

⁠Retrieval Methods

The Anserini image supports the following retrieval methods:

• BM25: k1=0.9, b=0.4 (Robertson et al., 1995)
• QL (query likelihood with Dirichlet smooth): mu=1000 (Zhai and Lafferty, 2001)
• +RM3 (RM3 variant of relevance models, applied on top of initial BM25 or QL results): exact parameter settings⁠ (Abdul-Jaleel et al., 2004)
• +Ax (Semantic term matching in the axiomatic framework, applied on top of initial BM25 or QL results): exact parameter settings⁠ (Fang and Zhai, 2006)

⁠Expected Results

The following numbers should be able to be re-produced using the scripts provided in the bin⁠ directory.

⁠robust04

⁠core17

⁠core18

⁠gov2

⁠cw09b

⁠cw12b

⁠Implementation

The following is a quick breakdown of what happens in each of the scripts in this repo.

⁠Dockerfile

The Dockerfile installs dependencies (python3, etc.), copies scripts to the root dir, and sets the working dir to /work.

⁠init

The init script⁠ is straightforward - it's simply a shell script (via the #!/usr/bin/env sh she-bang) that invokes wget to download an anserini JAR from Maven Central.

⁠index

The index Python script⁠ (via the #!/usr/bin/python3 she-bang) reads a JSON string (see here⁠) containing at least one collection to index (including the name, path, and format). The collection is indexed and placed in a directory, with the same name as the collection, in the working dir (i.e., /work/robust04). At this point, jig takes a snapshot and the indexed collections are persisted for the search hook.

⁠search

The search script⁠ reads a JSON string (see here⁠) containing the collection name (to map back to the index directory from the index hook) and topic path, among other options. The retrieval run is performed (using additional --opts params, see above) and output is placed in /output for the jig to evaluate using trec_eval.

⁠References

• Stephen E. Robertson, Steve Walker, Micheline Hancock-Beaulieu, Mike Gatford, and A. Payne. (1995) Okapi at TREC-4. TREC
• ChengXiang Zhai and John Lafferty. (2001) A Study of Smoothing Methods for Language Models Applied to Ad Hoc Information Retrieval. SIGIR.
• Nasreen Abdul-Jaleel, James Allan, W. Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Donald Metzler, Mark D. Smucker, Trevor Strohman, Howard Turtle, and Courtney Wade. (2004) UMass at TREC 2004: Novelty and HARD. TREC.
• Hui Fang and ChengXiang Zhai. (2006) Semantic Term Matching in Axiomatic Approaches to Information Retrieval. SIGIR.

⁠Reviews

• Documentation reviewed at commit 68f8291⁠ (06/06/2019) by Chris Kamphuis⁠, Claudia Hauff⁠, Craig Macdonald⁠ and Antonio Mallia⁠