skilja/benchmarkserver repository overview

⁠Quick reference

• Maintained by: Skilja GmbH⁠

• Where to get help: The Partner Portal⁠ or via email [email protected]⁠

⁠Supported tags and Versioning

Image tags adhere to <major>.<minor>.<patch>.<revision> format.

<major>.<minor>.<patch>.<revision> points to a specific version. <Major>.<Minor> always points to the latest version. This version is compatible with all previous images of the same <Major>.<Minor> version. latest always points to the latest version, but such a version might require service and database schema updates.

For loading the latest but compatible version, we recommend to pull a <Major>.<Minor> with

docker pull skilja/benchmarkserver:<Major>.<Minor>

The latest image tag is:

• 1.0.1.37484⁠

⁠What is the Benchmark Server?

The Benchmark Server is a web service for offline evaluation of document extraction quality, measuring precision and recall per document type and field on uploaded work items. It uses Vinna as its data source, since Vinna closely mirrors production data, importing the internal document representation (STGDoc) with field values, confidences, and locations. The tool helps compare extraction methods to find the best fit for your goal.

⁠How to use this image

The following sample shows how to run a PostgreSQL database container and the Benchmark Server container together.

⁠1. Optional: start a PostgreSQL database

If you want to use a PostgreSQL database server, install and start a PostgreSQL container:

docker pull postgres:15.1
docker run --rm -v <localpath>:/var/lib/postgresql/data \
  -e POSTGRES_PASSWORD=MyPassword \
  -e POSTGRES_USER=laera \
  -p 5432:5432 \
  -d postgres:15.1

⁠2. Create the environment file

Create a file named envfile.txt and configure it to use the database user and password created above:

BENCHMARKSERVER_SERVICEENDPOINT: http://+:8048
BENCHMARKSERVER_SELFHOSTWEBSITE: "True"
BENCHMARKSERVER_DBSERVERTYPE: 2     (2 - PostgreSQL)
BENCHMARKSERVER_DBSERVER: <database-server-name>   (the host computer name in case of local docker installation)
BENCHMARKSERVER_DBNAME: BenchmarkServer             (if not there, it is created)
BENCHMARKSERVER_DBINTEGRATEDSECURITY: "False"
BENCHMARKSERVER_DBUSER: "laera"
BENCHMARKSERVER_DBPASSWORD: "MyPassword"

⁠3. Install and start the Benchmark Server

Open a cmd / terminal, navigate to your working directory and execute:

docker pull skilja/benchmarkserver:1.0
docker run -p 8048:8048 -d --rm --env-file envfile.txt skilja/benchmarkserver:1.0

Wait for it to initialize completely, then visit http://localhost:8048/ or http://host-ip:8048.

⁠... via docker-compose

Example docker-compose.yml running PostgreSQL and skilja/benchmarkserver together:

services:
  postgres:
    image: postgres:15.1
    environment:
      POSTGRES_USER: laera
      POSTGRES_PASSWORD: MyPassword
    volumes:
      - ./pgdata:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    restart: unless-stopped

  BenchmarkServer:
    image: skilja/benchmarkserver:1.0
    ports:
      - "8048:8048"
    environment:
      BENCHMARKSERVER_SERVICEENDPOINT: "http://+:8048"
      BENCHMARKSERVER_SELFHOSTWEBSITE: "True"
      BENCHMARKSERVER_DBSERVERTYPE: 2
      BENCHMARKSERVER_DBSERVER: postgres
      BENCHMARKSERVER_DBNAME: BenchmarkServer
      BENCHMARKSERVER_DBINTEGRATEDSECURITY: "False"
      BENCHMARKSERVER_DBUSER: "laera"
      BENCHMARKSERVER_DBPASSWORD: "MyPassword"
    depends_on:
      - postgres
    restart: unless-stopped

Run docker compose up, wait for it to initialize completely, and visit http://localhost:8048/ or http://host-ip:8048.

⁠HTTPS

You probably noticed that the described sample is not using an SSL certificate. The currently recommended way is to use a reverse proxy like NGINX or Traefik in front of the container.

⁠Service User

The service user for the Benchmark Server is consistent across all containers. For .NET-based containers, such as this one, the UID/GID 1654 is used. This is a non-root user called app. This becomes important when working with volume mounts.

⁠Environment Variables

The Benchmark Server image uses several environment variables, of which some are required others are optional.

⁠BENCHMARKSERVER_SERVICEENDPOINT (ServiceEndpoint)

URL of the Benchmark Server.

⁠BENCHMARKSERVER_PATHBASE (PathBase)

Sets the path base of the Benchmark Server. Empty by default. This option may be necessary when running behind a reverse proxy or in containerized environments.

⁠BENCHMARKSERVER_USEFORWARDEDHEADERS (UseForwardedHeaders)

true or false - If you are running behind an SSL offloading service such as traefik, you need to configure Benchmark Server to use the forwarded headers provided by that service. Set to false by default.

⁠BENCHMARKSERVER_HASAUTHENABLED (HasAuthEnabled)

If this property is true, authentication is enabled for the Benchmark Server. Users need to log in via the configured authorization server and need to have access permissions with corresponding roles. Custom applications need to provide an API key with each API call. If this property is false, no authentication and no API keys are required.

⁠BENCHMARKSERVER_AUTHSERVERURL (AuthServerUrl)

The URL of the authorization server if authentication is enabled.

⁠BENCHMARKSERVER_AUTHTRUSTCERTIFICATE (AuthTrustCertificate)

If the URL of the Authorization Server is using SSL (https) with an self-signed certificate that is unknown to the operating system, then this option allows to trust all SSL certificates for the Authorization Server. Otherwise any backend call to the Authorization Server fails because of the unknown and untrusted SSL certificates.

It's not recommended to have this option enabled for production installations because it bypasses the normal SSL validation. If this option is enabled, a warning message is written into the service log file on each start up.

⁠BENCHMARKSERVER_SERVERCLIENTID (ServerClientId)

The client ID of the server side application that is registered as a confidential client within the authorization server.

⁠BENCHMARKSERVER_SERVERCLIENTSECRET (ServerClientSecret)

The client secret corresponding to the client ID of the server side application.

⁠BENCHMARKSERVER_DBSERVERTYPE (Database.ServerType)

The database server type to connect to as integer for the Benchmark Server. Supported are SQL Server and PostgreSQL.

- 0 - SQL Server  
- 2 - PostgreSQL  
(Oracle not supported)

⁠BENCHMARKSERVER_DBSERVER (Database.Server)

Database server name hosting the Benchmark Server database.

⁠BENCHMARKSERVER_DBNAME (Database.Name)

The database name for the Benchmark Server database.

⁠BENCHMARKSERVER_DBUSESSL (Database.UseSSL)

Use SSL encryption on the Benchmark Server database connection. This is currently supported for MS SQL server and PostgreSQL server.

⁠BENCHMARKSERVER_DBTRUSTSERVERCERTIFICATE (Database.TrustCertificate)

If SSL is enabled, by default SSL certificate must be officially signed. Set this parameter to true for self-signed SSL certificates that are not officially trusted.

⁠BENCHMARKSERVER_DBINTEGRATEDSECURITY (Database.UseIntegratedSecurity)

When true, use integrated security for the database access to the Benchmark Server database when false, use SQL user and password.

⁠BENCHMARKSERVER_DBUSER (Database.User)

SQL user name if the integrated security option is false.

⁠BENCHMARKSERVER_DBPASSWORD (Database.Password)

SQL password if integrated security option is false.

⁠Trusting and using self-signed certificates

In general, it's recommended to use publicly trusted SSL certificates whenever possible for communication between docker containers. For test installations, or if the chosen URLs cannot be publicly signed, self-signed SSL certificates can be used within a Docker environment. In that case it's required to trust each certificate itself or the CA (certificate authority) that signed the certificates.

To add such certificates to the list of trusted certificates, perform these steps.

⁠Mounting certificate files into the docker image

You need the public key of the certificates that you want to trust as PEM-encoded files. These certificates are usually stored as .crt files. A single crt-file can contain one or multiple public keys.

Example:

-----BEGIN CERTIFICATE-----
MIIF...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIG...
-----END CERTIFICATE-----

To mount a single crt-file into the container:

docker run ... --mount type=bind,source=/HostDir/certs/my_cert.crt,target=/certs/my_cert.crt,readonly

To mount a complete directory with multiple crt-files:

docker run ... --mount type=bind,source=/HostDir/certs,target=/certs,readonly

Mounting files inside other docker runtime environments, such as Kubernetes, requires a different syntax but the functionality is basically the same.

⁠Adding additional certificates to the trust store

To add additional certificates to the trust store inside a docker container, define an environment variable. To add only a single crt-file, use the SSL_CERT_FILE environment variable.

Example:

SSL_CERT_FILE=/certs/my_cert.crt

To add all crt-files stored in a directory, use SSL_CERT_DIR instead.

Example:

SSL_CERT_DIR=/certs/

The newly trusted certificates are added to the existing list of root CAs of the Docker base image. This means the Docker image works with a combination of publicly trusted SSL certificates and self-signed certificates.

⁠Other Platforms

The service is also available for Windows installation. Please visit The Partner Portal⁠ or write us an email [email protected]⁠ for more information.