structure101/soda-server repository overview

⁠Quick Reference

• Maintained by: Structure101⁠

• Where to get help: [email protected]⁠

⁠What Is Structure101?

Structure101 is a software architecture platform that bridges the architect/programmer divide to deliver the benefits of well-structured codebases with defined and communicated architecture.

⁠The instructions below require the following 3 Docker images

• structure101/soda-server⁠
• structure101/soda-client⁠
• postgres-15-alpine⁠

⁠Restricted access to Docker Hub

In the event your network restricts access to Docker Hub you must first download the following required images before following our Docker instructions below:

• soda-server: https://s101-pickup.s3.amazonaws.com/generic/masterv7/soda-server.tar⁠

• soda-client: https://s101-pickup.s3.amazonaws.com/generic/masterv7/soda-client.tar⁠

• postgres-15-alpine: https://s101-pickup.s3.amazonaws.com/generic/masterv7/postgres-15-alpine.tar⁠

Once downloaded, use the docker load command to bring up the images in Docker Desktop.

docker load --input soda-server.tar soda-client.tar postgres-15-alpine.tar

⁠Prerequisites:

For the ease of usage its recommended to install Docker Desktop for your respective OS: https://www.docker.com/products/docker-desktop/⁠

Also make sure that the docker-compose command is working as expected (if you are going to use docker compose for orchestration).

https://docs.docker.com/engine/reference/commandline/compose/⁠

Create a directory named soda and under it create the following files:

• .env
• docker-compose.yml

The contents of the above required files are outlined below.

⁠Steps:

1] Inside the .env file, add the following content with the appropriate configuration. The explanation of each of the variables is given at the end of this Readme:

#
# db
#
PORT=8001
DB_PORT=5432
DB_NAME='sodadb'
DB_USER='soda'
DB_PASS='pass'
# (For Linux-based systems, use: DB_IP='172.17.0.1')
DB_IP='host.docker.internal' 
DB_SCHEMA='s101_20230112'
#
# general
#
DB_DIALECT='postgres'
DB_VERSION=21194
#
# Cache
#
CACHE_ENABLED='False'
CACHE_SIZE=48
CORS_WHITELIST=[]
DB_LIMIT=200000
RS_LIMIT=10000

The configuration of the .env depends on the use case. For example, you may already a database that you want to use, in which case, configure the DB_* parameters accordingly.

Note: The .env file format is dependent on your operating system. On Windows you must use single quotes around each string variable, e.g. DB_NAME='sodadb' & DB_IP='172.17.0.1' etc. While on Linux, single quotes are not required, only the values are needed, e.g. DB_NAME=sodadb & DB_IP=172.17.0.1

2] Inside the docker-compose file, add the following:

version: '3'
services:
  sodaserver:
    container_name: soda-server
    image: structure101/soda-server
    restart: always
    env_file:
      - .env
    depends_on:
      - db
  sodaclient:
    container_name: soda-client
    image: structure101/soda-client
    restart: always
    depends_on:
      - sodaserver
    ports:
      - 80:80
  db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_HOST=${HOST}
      - POSTGRES_PORT=${PORT}
      - POSTGRES_USER=${USER}
      - POSTGRES_PASSWORD=${PASSWORD}
      - POSTGRES_DB=${DB}
    volumes:
      - ./data:/var/lib/postgresql/data
    ports:
      - ${PORT}:5432

3] Now run one of the following commands to start the Structure101 Soda Dependency Database Docker containers:

• Windows (Powershell):
  • $env:PORT='5432'; $env:HOST='host.docker.internal'; $env:USER='soda'; $env:PASSWORD='pass'; $env:DB='sodadb'; docker-compose up
• Linux:
  • HOST=172.17.0.1 PORT=5432 USER=soda PASSWORD=pass DB=sodadb docker-compose up

4] You should now have a running Docker containers with 3 services:

• soda-client: Soda UI, this is the image where we have the Vue (Javascript) frontend hosted, that consumes the soda-server behind the scenes.
• soda-server: Soda APIs, all the API requests are handled by this image, basically the backend for your setup.
• soda-db: this image is going to handle the database, it is going to mount to the specified location in the docker-compose file to store the data persistently and serve the soda-server image.

⁠If you want to orchestrate your own infrastructure:

• Use docker pull to save the docker hub images on your machine as follows
  • docker pull structure101/soda-client:latest
  • docker pull structure101/soda-server:latest
  • docker pull structure101/soda-client:15-alpine

⁠Publishing to your Structure101 Dependency Database:

⁠Publishing using a Structure101 project

1] Create a soda-db.properties file with the following properties set appropriately:

dburl=jdbc:postgresql://localhost:5432/sodadb?currentSchema=s101_20230112
dbuser=soda
dbpwd=pass

2] Then create a structure101-build-conf.xml with the following arguments set appropriately:

<?xml version="1.0" encoding="UTF-8"?>
   <headlessversion="1.0">
      <arguments>
         <argumentname="repository" value="const(THIS_FILE)/soda-db.properties"/>
          <argumentname="project" value="my-project"/>
          <argumentname="label" value="1.0"/>
          <argumentname="local-project" value="const(THIS_FILE)/my-project.cpa.hsp">
      </argument>
   </arguments>
   <!-- OPERATIONS TO EXCEUTE -->
   <operations>
   <!-- PUBLISH OPERATION -->
   <operationtype="publish"></operation>
   </operations>
</headless>

Note: Structure101 has 4 primary parsers: C/C++ (cpa), C# (dotnet), Java (java), and a Generic (gen) language parser with support for Javascript/Typescript and Python. The build configuration file above and the command used below to run the publish command use C/C++ as an example. If you are not working with C/C++ (cpa) please replace the occurrences of cpa in the configuration file above, and command below, with the appropriate parser string.

3] Finally run the following publish command:

java -jar structure101-cpa-build.jar structure101-build-conf.xml

4] For more on Structure101 Build publishing please see the Structure101 Help here⁠. If using Structure101 for Clang C/C++ remember to also publish your function hashcodes and C++ attributes using the publish-cpa-attributes⁠ operation.

⁠Importing from an existing Structure101 XML repository

1] Simply run the following command (using the Structure101 Build jar for your preferred programming language):

java -cp structure101-java-build.jar com.structure101.headless.S101HeadlessCmdRunner db-import-from-repository -repository=/my-respositories/my-repo -dbid=0000

⁠Notes:

• You can access the soda-server directly (using the APIs without the Soda front-end UI), for example http://localhost/api/projects⁠.
• The complete Structure101 Soda Dependency Database API documentation is available from http://localhost/api-docs/⁠
• If you are using the Postgres docker image and the data is published locally, and you want to replicate the same in some other machine with the same data. Just copy the ./data directory that is created by the postgres-15-alpine image, and then boot up the images on another machine the same way you did on the first machine. This will make sure that the published data is brought up when we bring up the setup on another machine.
• You can also mount a shared file system as a data partition. More on that: https://hub.docker.com/_/postgres⁠
• When you run the docker-compose command with the parameters the first time, it sets the passed credentials as the default credentials for the Structure101 Soda Dependency Database. So you have to make sure that whatever credentials are passed the first time are the same ones added in the .env file, as they are then used by the soda-server to connect to the db service.
• The volumes that are mentioned under the soda-db service are required to have a persistent database so that the published data is not lost when Docker containers are shut down.
• If you want to access the database directly using third-party tools such as DBeaver, you can do so by having the localhost as the host, port, and credentials as the ones you have set while doing the docker-compose command for the first time.
• .env file variables:
  • PORT: This is the port that the soda-server will be running on internally, and that is the one that will be consumed by the soda-client via an nginx configuration.
  • DB_PORT: Database port that the soda-server image will use to connect to the database.
  • DB_NAME: Database name that the soda-server image will use to connect to the database.
  • DB_USER: Database user that was created when you first ran the docker-compose up command.
  • DB_PASS: Database password that was set.
  • DB_IP='host.docker.internal' (For Windows/OSX) For Linux-based systems, use: DB_IP=172.17.0.1.
  • DB_SCHEMA='s101_20230112': This will be static and based on the current build that you're using.
  • CACHE_ENABLED='False': In case you want to enable internal SQL results caching.
  • CACHE_SIZE=48: Size of the cache storage.
  • CORS_WHITELIST=[]: If you want to whitelist specific domains so that only they can access the soda-server APIs, you will have to add the server IP with port where you are running soda, as the migrations module 'POST' request needs cors to be enabled.
  • DB_LIMIT=200000: Record limit.
  • RS_LIMIT=10000: Record limit.
• The port mapping in the docker-compose file works as follows:
  • ${PORT}:5432 where the '${PORT}' is your external port/machine port and the '5432' is the internal docker container port. The docker container will not change as long as you are not setting the docker container in that way. The ${PORT} variable can be anything, based on the availibility on the machine you are using, just make sure if you are changing the host port, do change the database connection port in .env file (for soda-server connection) as well as the soda.properties (while publishing)
  • An extended example for the above, just in case your 5432 machine port is already engaged, you can pass the port to docker-compose up command as 5433 HOST=172.17.0.1 PORT=5433 USER=soda PASSWORD=pass DB=sodadb docker-compose up and then you will change the same in .env file for DB_PORT, and in soda.properties file the string will be changed to something like dburl=jdbc:postgresql://localhost:5433/sodadb?currentSchema=s101_20230112
• Just in case you want to have another database instead of using the postgres-15-alpine docker image, just pass the respective parameters to the .env file to connect to the database and remove the DB service from the docker-compose file.
• For postgres installation please use the appropriate guide for your platform:

  • Windows: https://www.postgresql.org/download/windows/⁠

  • Linux: https://www.devart.com/dbforge/postgresql/how-to-install-postgresql-on-linux/⁠

  • Unix: https://www.postgresqltutorial.com/postgresql-getting-started/install-postgresql-macos/⁠

⁠Mounting your own nginx.conf / enabling SSL

Generate respective certificates. How To Create a Self-Signed SSL Certificate for Nginx Modify your nginx configuration as follows:

user  nginx;
worker_processes  auto;
 
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
 
events {
  worker_connections  1024;
}
 
http {
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;
 
  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';
 
  access_log  /var/log/nginx/access.log  main;
 
  sendfile        on;
  #tcp_nopush     on;
 
  keepalive_timeout  65;
 
  #gzip  on;
 
  server {
    listen 443 ssl;
    server_name localhost;
 
    ssl_certificate /etc/nginx/certs/nginx.crt;
    ssl_certificate_key /etc/nginx/certs/nginx.key;
 
    location / {
      root   /usr/share/nginx/html;
      index  index.html index.htm;
      try_files $uri $uri/ /index.html;
    }
 
    location ~ ^/(api|api-docs)/ {
        proxy_pass http://172.17.0.1:8001;
        proxy_read_timeout 1800;
        proxy_connect_timeout 1800;
        proxy_send_timeout 1800;
        send_timeout 1800;
    }
 
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
      root   /usr/share/nginx/html;
    }
  }
}

Modify the sodaclient section in docker-compose as follows:

  sodaclient:
    container_name: soda-client
    image: structure101/soda-client
    restart: always
    depends_on:
      - sodaserver
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./cert:/etc/nginx/certs/
      - ./nginx.conf:/etc/nginx/nginx.conf

Above we are mounting the certificates and updating nginx.conf files as per our needs.