Public | Automated Build

Last pushed: 2 months ago
Short Description
Full Description


.. image::

.. image::

.. image::

.. contents ::

Project Name

The Project Name is GEMET - GEneral Multilingual Environmental Thesaurus

Prerequisites - System packages

These packages should be installed as superuser (root).

Debian based systems

Install these before setting up an environment::

    apt-get install python-setuptools python-dev libmysqlclient-dev \
    libldap2-dev python-virtualenv mysql-server git

RHEL based systems

Install Python2.7 with PUIAS:

Run these commands::

curl | python2.7 -
pip2.7 install virtualenv
yum install mysql-server mysql git mysql-devel

Product directory

Create the product directory::

mkdir -p /var/local/gemet
mkdir /var/local/gemet/logs

Create a new user::

adduser edw

Change the product directory's owner::

chown edw:edw /var/local/gemet -R

Install dependencies

We should use Virtualenv for isolated environments. The following commands will
be run as an unprivileged user in the product directory::

  1. Clone the repository::

    git clone -o origin gemet
    cd gemet

2.1. Create & activate a virtual environment::

virtualenv --no-site-packages sandbox
echo '*' > sandbox/.gitignore
source sandbox/bin/activate

2.2 Make sure setuptools >= 0.8 is installed::

pip install -U setuptools
  1. Install dependencies::

    pip install -r requirements-dep.txt

  2. Create a local configuration file::

    cd gemet

    Follow instructions in to adapt it to your needs.

  3. Set up the MySQL database::

    Replace [user] and [password] with your MySQL credentials and [db_name]

    with the name of the database:

    mysql -u[user] -p[password] -e 'create database [db_name] CHARACTER SET utf8 COLLATE utf8_general_ci;'

    The database charset MUST be utf8.

  4. Update local configuration file with database credentials and database name

    • default section in DATABASES dict.
  5. Create initial database structure::

    ./ migrate

  6. Load fixtures data into the database::

    ./ loaddata gemet/thesaurus/fixtures/data.json

  7. Generate EIONET static templates::

    ./ fetchtemplates

  8. Import data, see Data Import_ below.

.. _Data Import:

Build production

Setup production environment using an unprivileged user::

cd /var/local/gemet
source sandbox/bin/activate

Change the file by setting debug mode off::

DEBUG = False
ALLOWED_HOSTS = ['localhost']  # Add allowed hosts to the list as needed

Configure supervisord and set the WSGI server port::

cp gemet/supervisord.conf.example supervisord.conf
supervisorctl reload 1>/dev/null || ./bin/supervisord

Build staging

Setup staging environment using an unprivileged user::

cd /var/local/gemet
source sandbox/bin/activate

Change the file by setting debug mode off::

DEBUG = False
ALLOWED_HOSTS = ['localhost']  # Add allowed hosts to the list as needed

Configure supervisord and set the WSGI server port (a different one from the
production, for example 8010)::

cp gemet/supervisord.conf.example supervisord.conf
supervisorctl reload 1>/dev/null || ./bin/supervisord


Details about configurable settings can be found in

Data Import

  1. Considering you have a dump of the old database (gemet.sql), import it in a
    separate database::

    mysql -u[user] -p[password] -e 'create database [db_name] CHARACTER SET utf8 COLLATE utf8_general_ci;'
    mysql -u[user] -p[password] [db_name] < gemet.sql

  2. Update the import section from DATABASES dict in the local
    configuration file with the name of the database used for import
    (gemet_old from the previous example).

  3. Run the management command for data import::

    ./ import

  4. Fix romanian characters::

    ./ fix_romanian

  5. Insert data that enables search to work properly::

    ./ insertdata

  6. Create reversed relations for all concepts::

    ./ fixrelations

  7. Import new terms from the spreadsheet::

    ./ importspreadsheet [spread_sheet_name]

Other commands

  1. Some romanian terms, definitions etc. are written with the wrong diacritical marks (cedillas instead of commas).
    The following custom management command fixes those characters and prints the number of objects changed::

    ./ fix_romanian

  1. Check the consistency of an excel file (.xlsx extension) containing new terms.

The custom command assures:

  • Old terms used in the file are defined in the database.
  • New terms used in broader, narrow relations etc. of other terms are also defined in the file.
  • An error containing the cell of the term is printed if it does not respect those rules.

Run the command providing a valid excel file::

 ./ check_spreadsheet file_name.xlsx


The documentation has been created using Sphinx. The source directories for the three sections of documentation can be found in the docs directory.

.. _Sphinx:
.. _docs:

In order to get the HTML output, you should run the following command inside one of the documentation directories (api, new_api or overview)::

make html

These static HTML files can be served via a web server (Apache, Nginx, etc).

Docs contents

  • api - old version of the API user guide, kept for reference;
  • new_api - current documentation for the GEMET API; duplicated in this file_ and published on Web services page;
  • overview - quick overview of the technical solution;

.. _this file:

Development hints


These packages should be installed as superuser(root)::

apt-get install libxml2-dev libxslt1-dev

Use requirements-dev.txt instead of requirements-dep.txt::

pip install -r requirements-dev.txt

Configure deploy

  • copy fabfile/env.ini.example to fabfile/env.ini
  • configure staging and production settings
  • run fab staging deploy or fab production deploy

Running unit tests

  1. Before running the tests make sure you have configured the test database

    cd gemet/

    Parameters values should match the ones used for the 'default' database

    entry in

  2. For the GEMET web application::

    ./ test

  3. For the API::

    python apitests/

Two optional parameters exist:

  • --public, which runs the tests against the production website;
  • --get, which calls the API methods through GET requests.
  1. Running tests with coverage measurement

Add to your TEST_RUNNER and NOSE_ARGS from
local_settings.example and run::

./ test

Sentry settings

Sentry is used to track errors in real-time.

Create an account and a project on Sentry_ .

Install the proper version of raven used by sentry::

pip install -r requirements-dep.txt

Configure local settings with your project's dsn.

.. _Sentry:


The project owner is Søren Roug (soren.roug at

Other people involved in this project are:

  • Iulia Chiriac (iulia.chiriac at
  • Andrei Melis (andrei.melis at
  • Diana Boiangiu (diana.boiangiu at
  • Cornel Nițu (cornel.nitu at
  • Alex Eftimie (alex.eftimie at
  • Mihai Tabără (mihai.tabara at
  • Mihai Zamfir (mihai.zamfir at



Minimum requirements:

  • 2048MB RAM
  • 2 CPU 1.8GHz or faster
  • 4GB hard disk space


  • 4096MB RAM
  • 4 CPU 2.4GHz or faster
  • 8GB hard disk space


Any recent Linux version, apache2, MySQL server, Python 2.7

Copyright and license

This project is free software; you can redistribute it and/or modify it under
the terms of the EUPL v1.1.

More details under LICENSE.txt_.

.. _LICENSE.txt:

Docker Pull Command
Source Repository