osirrc2019/terrier repository overview
OSIRRC Docker Image for Terrier
Arthur Câmara and Craig Macdonald
This is the docker image for the Terrier toolkit (v5.2) conforming to the OSIRRC jig for the Open-Source IR Replicability Challenge (OSIRRC) at SIGIR 2019. This image is available on Docker Hub
- Supported test collections:
robust04,gov2,cw09b,cw12b(web),core18(newswire) - Supported hooks:
init,index,train,search
Quick Start
The following jig command can be used to index TREC disks 4/5 for robust04:
python run.py prepare --repo terrier --collections robust04=/tmp/disk45/=trectext
The following jig command can be used to perform a retrieval run on the collection with the robust04 test collection, using BM25 as ranker:
python run.py search \
--repo osirrc2019/terrier\
--collection robust04 \
--topic topics/topics.robust04.txt \
--qrels qrels/qrels.robust04.txt\
--output /tmp/runs
Retrieval Methods:
(BM25)
python run.py search --repo terrier --collection robust04 --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt --output /tmp/runs
(BM25 + query expansion)
python run.py search --repo terrier --collection robust04 --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt --output /tmp/runs --opts config=bm25_qe
(PL2)
python run.py search --repo terrier --collection robust04 --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt --output /tmp/runs --opts config=pl2
(PL2 + query expansion)
python run.py search --repo terrier --collection robust04 --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt --output /tmp/runs --opts config=pl2_qe
(DFRD)
python run.py search --repo terrier --collection robust04 --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt --output /tmp/runs --opts config=DFRD
NOTE: for running DFRD, the index must be build using the --opts=block.index=true param
Learning to Rank Runs
Learning-to-rank will typically require that the index has more information, e.g. fields or blocks.
Indexing:
python run.py prepare --repo terrier --collections robust04=/tmp/disk45/=trectext --opts "FieldTags.process=HEADLINE"
Training:
You need to specify the features to be used by Terrier - see http://terrier.org/docs/v5.1/learning.html for more information about Terrier feature definitions.
python run.py train --repo terrier --collection robust04 --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt --test_split $PWD/sample_training_validation_query_ids/robust04_test.txt --validation_split $PWD/sample_training_validation_query_ids/robust04_validation.txt --model_folder /tmp/runs --opts features="SAMPLE;WMODEL:SingleFieldModel(BM25,0);QI:SingleFieldModel(Dl,0)"
Retrieval:
You will need to specify the bm25_ltr_jforest configuration.
python run.py search --repo terrier --collection robust04 --topic topics/topics.robust04.txt --qrels qrels/qrels.robust04.txt --output /tmp/runs --opts config=bm25_ltr_jforest
Expected Results
robust04
| MAP | BM25 | +QE | +Prox | +Prox + QE | DPH | + QE | +Prox | +Prox +QE | PL2 | +QE |
|---|---|---|---|---|---|---|---|---|---|---|
| TREC 2004 Robust Track Topics | 0.2363 | 0.2762 | 0.2404 | 0.2781 | 0.2479 | 0.2821 | 0.2501 | 0.2869 | 0.2241 | 0.2538 |
core18
| MAP | BM25 | +QE | +Prox | +Prox + QE | DPH | + QE | +Prox | +Prox +QE | PL2 | +QE |
|---|---|---|---|---|---|---|---|---|---|---|
| TREC 2018 Common Core Track Topics | 0.2326 | 0.2975 | 0.2369 | 0.2960 | 0.2427 | 0.3055 | 0.2428 | 0.3035 | 0.2225 | 0.2728 |
Tag summary
Content type
Image
Digest
Size
996.2 MB
Last updated
over 7 years ago
docker pull osirrc2019/terrier