ulixee/ulixee-cloud repository overview

⁠Ulixee: The Open Data Platform

Ulixee is a sql database with built-in scraping engine⁠ functionality to enrich tables.

This repository is the development home to several of the tools that make it easy to build and manage these scripts, including Ulixee Desktop⁠, [Cloud][cloud] and Datastores⁠].

⁠Documentation

Ulixee documentation can be found at https://ulixee.org⁠.

⁠Usage

To use the image, we maintain a run.sh⁠ script that will run with a non-root user on your choice of port. All environmental configurations are listed here⁠.

⁠Ulixee Cloud

The Ulixee Cloud docker comes pre-installed with 2 versions of Chrome and several default fonts. It comes with a default unprivileged user as well, called "ulixee". To use it, you need to "run" your docker with the appropriate user --user ulixee. This feature helps use the chrome sandbox security model on linux.

⁠Install Docker on Ubuntu

If you're using something like Amazon ECS, you can extrapolate the parts that apply to you.

This script will install docker and register a service to run on bootup.

sudo apt update
sudo apt install docker.io
# start a daemon to auto-run service
sudo systemctl start docker.service
sudo systemctl enable docker.service
# allow non-root access
sudo usermod -aG docker $USER
reboot

⁠Pull Docker image from Github Container Registry

docker pull ghcr.io/ulixee/ulixee-cloud:latest
docker tag ghcr.io/ulixee/ulixee-cloud:latest ulixee-cloud

⁠Pull Docker image from Dockerhub

docker pull ulixee/ulixee-cloud:latest
docker tag ulixee/ulixee-cloud ulixee-cloud

⁠Copy/modify run.sh On Your Server

The run.sh⁠ file can be a starting point to run the docker on your server. It configures docker as follows:

• --init Register a process cleanup service.
• -p 8080:8080 By default, register 8080 as the internal and published port.
• --ipc=host Chrome can run out of memory if you don't use the system host for it's IPC communications.
• --user ulixee We build the docker to run as a non-root user.
• --restart unless-stopped Reboot the process on crash unless you explicitly call docker stop.
• --log-opt max-size=50m --log-opt max-file=3 --log-driver local Add rotating log files.
• --security-opt seccomp="$SECURE_PROFILE" Clones the default Docker seccomp profile to run reduced linux permissions. It adds user namespace permissions to enable Chrome Sandboxing without root user. NOTE: You MUST use the seccomp settings or turn off Chrome sandboxes.
• -v $DATASTORES_MOUNT:/home/ulixee/.cache/ulixee/datastores Mount and chmod 777 a directory on the host OS where Datastores can be packaged for deployment.
• -v $DATADIR_MOUNT:/tmp/.ulixee Mount a tmp directory and chmod 777 so that the ulixee user can store Session databases in this directory.
• -e DISPLAY=:99, xvfb-run Allows headed Chrome with a virtual display.
• -e DEBUG=ulx* Configures Ulixee to emit very verbose logs to the console. Removing * will remove devtools and internal network logging.