frankierr/finn-wsd-eval repository overview

⁠Evaluation framework

⁠Setup

The evaluation framework is distributed together with all systems and requirements as a Docker image, which can be pulled like so:

$ docker pull registry.gitlab.com/frankier/stiff:latest

And run like so:

$ docker -v /path/to/working/dir/:/work/ run python eval.py /work/results.json /work/eurosense.eval/

For CUDA accelerated experiments, you can use nvidia-docker⁠. For running in shared computing environments, in which you don't have root access, I recommend udocker⁠.

You can also set up requirements manually. See the Dockerfile for the list of commands to run.

⁠Evaluation corpus

The one requirement which is not included in the Docker image is the evaluation corpus. Please follow the instructions in the STIFF README⁠.

⁠Filtering experiments with eval.py

You can run a subset of experiments by passing filters to eval.py, e.g.

$ python eval.py /work/results.json /work/eurosense.eval/ Knowledge 'Cross-lingual Lesk' mean=pre_sif_mean

⁠Making tables with table.py

You can make LaTeX tables with table.py, e.g.

$ python table.py results.json --filter='Knowledge;Cross-lingual Lesk' --table='use_freq;vec:fasttext,numberbatch,double;mean expand;wn_filter'

⁠Licenses

This project is licensed under the Apache v2 license. The code in ukb-eval is vendorized from UKB, and therefore licensed under the GPL. The scorer in support/scorer is under an unknown license, possibly public domain.

⁠See also

• STIFF⁠: Automatically created sense tagged corpus of Finnish and corpus wrangling tools.
• STIFF-explore⁠: Some exploratory coding related to STIFF.
• finn-man-ann⁠: Small, Finnish language, manually annotated word sense corpus.
• FinnTK⁠: Simple, high-level toolkit for Finnish NLP, mainly providing convenience methods for, and gluing together, other tools.
• extjwnl_fiwn⁠: Java code to make extjwnl interoperate with FinnWordNet.
• FinnLink⁠: Link between FinnWordNet and Finnish Propbank created by joining with PredicateMatrix.
• finn-sense-clust⁠: Sense clusterings of FinnWordNet.

⁠Forks/fixes

• ItMakeseSense⁠: ItMakesSense fork to support FiWN for use by finn-wsd-eval
• AutoExtend⁠: AutoExtend fork to support FiWN and ConceptNet Numberbatch
• babelnet-lookup⁠: babelnet-lookup fork to obtain BABEL2WN_MAP.
• FinnWordNet⁠: Temporary fixes to FinnWordNet 2.0.
• Eurosense⁠: Attempted fixes to Eurosense.