osirrc2019/olddog repository overview

⁠OSIRRC Docker Image for OldDog

Chris Kamphuis⁠ and Arjen de Vries⁠

This is the docker image for the OldDog⁠ project (based on work by Mühleisen et al.) conforming to the OSIRRC jig⁠ for the Open-Source IR Replicability Challenge (OSIRRC) at SIGIR 2019⁠. This image is available on Docker Hub⁠ has been tested with the jig at commit ee62fa8⁠ (06/17/2019).

• Supported test collections: robust04
• Supported hooks: init, index, search

⁠Quick Start

The following jig command can be used to index TREC disks 4/5 for robust04:

python run.py prepare \                                                         
  --repo osirrc2019/olddog \             
  --tag v0.2.0 \
  --collections robust04=/path/to/disk45=trectext

The following jig command can be used to perform a retrieval run on the collection with the robust04 test collection.

python run.py search \
  --repo osirrc2019/olddog \
  --tag v0.2.0 \
  --output $(pwd)/out \
  --qrels qrels/qrels.robust04.txt \
  --topic topics/topics.robust04.txt \
  --collection robust04 \
  --opts out_file_name="run.bm25.robust04"

⁠Retrieval Methods

The OldDog image supports the following retrieval models:

• BM25 (conjunctive variant): k1=1.25, b=0.75 (Robertson et al., 1995)

⁠Expected Results

The following results should be able to be re-produced using the jig search command.

⁠robust04

Note that the scores are lower than regular bm25 because of conjunctive query processing; all query terms need to appear in a document in order for the document to be considered relevant. Without this restriction you would find the results listed below. These can however as of now, not be produced with a jig command.

⁠robust04

⁠Implementation

The following is a quick breakdown of what happens in each of the scripts in the repo.

⁠Dockerfile

The Dockerfile installs dependencies (python3, monetdb, etc.), copies scripts to the root dir, and sets the working dir to /work

⁠init

The init script⁠ is a bash script (via the #!/bin/bash she-bang) that invokes wget to download an anserini JAR from Maven Central. Then it clones the OldDog project from github, which then is build using maven.

⁠index

The index Python script⁠ (via the #!/usr/bin/python3 she-bang) reads a JSON string (see here⁠) containing at least one collection to index (including the name, path, and format). The collection is indexed using Anserini (Yang et al., 2017) and placed in a directory, with the same name as the collection, in the working dir (i.e., /work/robust04). After the Lucene index has been created, the OldDog software uses this index to creates csv files from it that can be loaded in the monetdb (Boncz, 2002) column store. A monetDB databse is created and the csv-files are loaded into the database. This is followed by removing the Lucene index so commiting the image takes less time. At this point, jig takes a snapshot and the indexed collections are persisted for the search hook.

⁠search

The search script⁠ reads a JSON string (see here⁠) containing the collection name (to map back to the index directory from the index hook) and topic path, among other options. The retrieval run is performed and output is placed in /output for the jig to evaluate using trec_eval.

⁠Notes

• re:v0.1.0 We can not guarantee that version v0.1.0 still works. This version cloned the OldDog github repository. New versions download a released version and should keep working.

⁠References

• Hannes Mühleisen, Thaer Samar, Jimmy Lin, Arjen de Vries (2014) Old Dogs Are Great at New Tricks: Column Stores for IR Prototyping. SIGIR
• Stephen E. Robertson, Steve Walker, Micheline Hancock-Beaulieu, Mike Gatford, and A. Payne. (1995) Okapi at TREC-4. TREC
• Peilin Yang, Hui Fang, and Jimmy Lin (2017) Anserini: Enabling the Use of Lucene for Information Retrieval Research. SIGIR
• Boncz Peter (2002) Monet: A Next-Generation DBMS Kernel For Query-Intensive Applications. PhD Thesis

⁠Reviews

• Documentation reviewed at commit d3a9750⁠ (2019-06-13) by Jimmy Lin⁠
• Documentation reviewed at commit dd53191⁠ (2019-06-17) by Ryan Clancy⁠.