epwalsh/allennlp-manager repository overview

⁠[WORK IN PROGRESS] allennlp-manager

Your manager for AllenNLP⁠ experiments.

⁠Table of contents

• Road map⁠ :car:
• Dependencies⁠
• Installation⁠
• Quick start⁠ :rocket:
• Contributing⁠
• Advanced configuration⁠
• Updates⁠ :tada: :tada: :tada:

⁠Road map

The goal of this project is to build a CLI and dashboard for running, queueing, tracking, and comparing experiments.

This was inspired by other open source projects such as the resource manager slurm⁠ and visualization toolkit TensorBoard⁠, as well as commercial software such as Weights & Biases⁠ and Foundations Atlas⁠.

slurm and TensorBoard are both excellent tools, but they fall short for NLP researchers in a number of ways. For example, slurm is difficult to set up and use - especially on your own desktop or server - unless you're an experienced sys admin, and TensorBoard has limited functionality for searching, organizing, tagging, and comparing models. And while the commercial options are fairly easy to use and come with a solid set of features, they were built as generic tools and therefore don't "understand" all of AllenNLP's features.

allennlp-manager aims to leverage all of the convenient pieces of AllenNLP to provide you with a dashboard that let's you

• quickly search through all of your experiments based on properties like model type, training / validation set, or arbitrary tags,
• visualize the metrics from training runs of an experiment, and
• compare experiments in a number of ways, such as looking at a git diff of configuration files.

In addition to the dashboard, there will be a multi-purpose CLI with commands for serving the dashboard, updating to the latest version, and programmatically submitting training runs.

For the first release I intend to have all of the features implemented except for, possibly, the slurm-like resource manager and job queueing system, as that may become quite complex.

⁠Dependencies

AllenNLP, Python 3.6 (or higher), and Docker are required.

⁠Installation

pip install 'git+git://github.com/epwalsh/allennlp-manager.git#egg=mallennlp'

⁠Quick start

Create a new project named my-project:

mallennlp new my-project && cd my-project

Then edit the Project.toml file to your liking and start the server:

mallennlp serve

⁠For potential contributors

I chose to implement this project entirely in Python to make it as easy possible for anyone to contribute, since if you are using AllenNLP you must already be familiar with Python. The dashboard is built with plotly Dash⁠, which is kind of like Python's version of Shiny⁠ if you're familiar with R.

The continuous integration for allennlp-manager is a lot like that of AllenNLP. Unit tests are run with pytest⁠, code is type-checked with mypy⁠, linted with flake8⁠, and formatted with black⁠. You can run all of the CI-steps locally with make test.

If this is your first time contributing to a project on GitHub, please see this Gist⁠ for an example workflow.

⁠Advanced configuration

⁠Command completion

Since the CLI is implemented using Click, setting up completion for Bash or ZSH⁠ is easy. For example, you can just add

eval "$(_MALLENNLP_COMPLETE=source mallennlp)"

to your .bashrc. Note however that it is better to use the activation script approach⁠ instead, otherwise your shell may take a couple seconds to start.

⁠Updates

11/6

Added sqlite database and implemented login / authentication system.

11/5

Dashboard skeleton implemented with Page abstraction.

11/4

CLI implemented (using click). Project can be created with mallennlp new [PROJECT NAME] and dashboard can be served with mallennlp serve from within the project directory (dashboard just returns 'Hello from AllenNLP manager' at the moment).