quicklizard/cli-agents repository overview

⁠Quicklizard CLI Agents

This repository contains a collection of command-line agents that are designed to be run locally, in your organization network and make it easier for your IT team to integrate with QL's REST API.

⁠API Proxy

The API Proxy agent is an HTTP service that serves as bridge between your network and QL's REST API, and signs API requests on your behalf.

The purpose of this agent is to reduce integration complexity, which usually arises from having to write custom code to sign API requests to QL. Since this agent signs API requests on your behalf, you can run it in your local organization network and call it instead of QL's REST API.

⁠Usage

The API proxy agent accepts 2 command-line arguments:

• c - The path to the agent's config file (defaults to config.yaml)
• k - The config file's segment (key) to use (defaults to default)

See the Config Section for more details on how to configure the agent.

Once the agent is running, you can start making API calls to its local address, instead of QL's REST API. For example, assuming your agent is running on localhost:8080 and you want to call the API endpoint /api/v2/recommendations/accepted, instead of calling https://rest.quicklizard.com/api/v2/recommendations/accepted you should call http://localhost:8080/api/v2/recommendations/accepted.

Since this agent is merely a bridge to QL's REST API, it supports any API endpoints available on https://rest.quicklizard.com. However, you should note that the by default, agent runs on HTTP rather than HTTPS. This means that by default, traffic from the agent to QL is secure (the agent calls QL's REST API HTTPS endpoint), but traffic from your HTTP client to the agent is not.

To secure traffic from your HTTP client to the API Proxy agent, you should provide the agent with SSL certificates, either self-generated or provided by a certificate authority. See the following section for example on how to generate self-signed SSL certificates.

⁠Self-signed SSL certificates

To create self-signed SSL certificates for the API Proxy agent, please run the following commands in your Bash terminal (Linux / Unix only).

$ openssl genrsa -out server.key 2048
$ openssl ecparam -genkey -name secp384r1 -out server.key
$ openssl req -new -x509 -sha256 -key server.key -out server.crt -days 365

The above commands should generate two files - server.key and server.crt. You may now pass the paths of these files to the agent's configuration, which will cause the agent to run in HTTPS rather than in HTTP mode. Assuming your agent is running on localhost:8080, you should call it on https://localhost:8080.

⁠Running the agent

docker run -p 8080:8080 -v /path/to/config.yaml:/tmp/config.yaml quicklizard/cli-agents:tagname /cli/api-proxy -c /tmp/config.yaml -k default

• change port binding to match your config if needed
• change /path/to/config.yaml to actual config path on your system
• change -k default to actual config key you want to use
• change :tagname to latest available docker image tag or use stable for latest stable versions
• optionally mount ssl certificates (add relevant config entries) - -v /path/to/ssl:/tmp/ssl

⁠Recommendations Syncer

The recommendations syncer is an agent that is designed to sync price recommendations from QL's REST API to a local database, along with related product and competitor data.

Using the recommendations syncer lets you collect and store a complete history of price recommendations provided by QL in a given timeframe. Data is synced from QL's REST API to a local database, which you can then query to sync information to your operational database / ERP, and use to generate reports.

⁠Usage

The recommendations syncer agent accepts 2 command-line arguments:

• c - The path to the agent's config file (defaults to config.yaml)
• k - The config file's segment (key) to use (defaults to default)

See the Config Section for more details on how to configure the agent.

Once the agent is running, it will periodically run the following procedures:

• recommendations - sync all price recommendations from QL to local DB
• DB cleanup - cleanup products & recommendations created before a given time frame (optional)

The DB cleanup procedure is optional, and is designed to help you keep your database at a reasonable size. It's important to note that the data accumulated by the recommendation syncer is not meant to span over long periods. Ideally, you should keep data around for up to 90-120 days.

If you need to keep a historical record of products, prices and recommendations for longer than 90-120 days, you should periodically export the data from the agent's DB to an archive.

The maintenance of the agent's DB is entirely up to you, and you should employ industry best-practices to maintain and secure it.

⁠Database structure

The recommendations syncer supports two database engines - SQLite and PostgreSQL. Choose SQLite when your IT team is unable to provide a separate database server or if you want to test the agent locally without having to install additional software. Otherwise, we recommend that you choose PostgreSQL as your database.

If you intend to sync more than one client key, then you should either use PostgreSQL or use separate SQLite DB files for each of your client keys.

You should use SQLite v3 and above, and PostgreSQL v9.6 and above.

The recommendations database includes four tables:

• products - contains information about your products, as created and updated on QL
• attr - contains product attributes, where each row references a product on the pid column
• price_recommendations - a historical record of price recommendations generated by QL, uniquely identified by product ID (pid) and recommendation date (recommended_at)
• competitor_prices - a historical record of competitor prices available for each price recommendation

⁠Database schema

⁠Database schema notes

The puid column in every table contains the QL product internal UID of the product related to that row. You can use this column to map your internal product ID (from the products.product_id column) to QL's product UID. Please note that this is not a JOIN column.

Row primary keys are saved as UUID v4. You can safely join tables on these columns.

Product attributes are uniquely identified using a name->value->pid combination. This means that an attribute is the unique combination of a product ID, name and value. Each product has many attributes. Each attribute belongs to a single product.

Price recommendations are uniquely identified using a pid->recommended_at combination. This means that what makes a recommendation unique is the combination of its product and recommendation date. One product has many recommendations. Each recommendation belongs to a single product.

Competitor prices are recommendation-specific. For each recommendation saved in the DB, only the competitor prices that are currently available for it will be saved in the DB. Each recommendation has many competitor prices. Each competitor price belongs to a single recommendation.

⁠Beta / Future features

The channel column in the price_recommendations table is currently a beta feature. Unless your account has support for multiple price channels, you can safely ignore its content.

The disabled column in the products table is currently a place-holder. Syncing enabled / disabled products is not supported at the moment.

⁠Running the agent

docker run -v /path/to/config.yaml:/tmp/config.yaml quicklizard/cli-agents:tagname /cli/syncer -c /tmp/config.yaml -k default

• change port binding to match your config if needed
• change /path/to/config.yaml to actual config path on your system
• change -k default to actual config key you want to use
• change :tagname to latest available docker image tag or use stable for latest stable versions

⁠Config

The CLI agents accept a configuration file in YAML format. The configuration file can contain multiple segments, one for each client key you wish to work with.

Each segment includes a common section, and one section per agent.

⁠Configuration sections

⁠Configuration notes

• syncer.dbUrl - Supports DB connection URLs as follows:
  • SQLite - sqlite:///path/to/db.sqlite
  • PostgreSQL - postgres://user:pass@host:port/database?sslmode=disable
• syncer.retentionPeriod - should be formatted in hours or days. Leave empty to disable data cleanup
• syncer.intervals - a map of sync intervals, where each key corresponds to a specific data synchronizer, and each value designates the data synchronization interval
  • syncer.intervals.recommendations - should be formatted in seconds or minutes. Cannot be over 30 minutes
• durations should be formatted as a numeric interval + period string. For example 30m for 30 minutes and 10h for 10 hours. Supported periods are:
  • s - seconds
  • m - minutes
  • h - hours
  • d - days (only available for syncer.retentionPeriod)

⁠Configuration example

client1:
  apiKey: 2e99758548972a8e8822ad47fa1017ff72f06f3ff6a016851f45c398732bc50c
  apiSecret: e1cb527cb19eff463fa117435d7241a865266b5d87033331f5b6a95dbd723e89  
  logLevel: "debug"
  apiProxy:
    port: 8080
    ssl: on
    tlsCert: "/tmp/ssl/server.crt"
    tlsKey: "/tmp/ssl/server.key"
  syncer:
    clientKey: client1
    dbUrl: sqlite:///tmp/db/recommendations_client1.sqlite
    retentionPeriod: 90d
    intervals:
      recommendations: 10m

client2:
  apiKey: 2e99758548972a8e8822ad47fa1017ff72f06f3ff6a016851f45c398732bc50c
  apiSecret: e1cb527cb19eff463fa117435d7241a865266b5d87033331f5b6a95dbd723e89  
  logLevel: "info"
  apiProxy:
    port: 8080
    ssl: on
    tlsCert: "/tmp/ssl/server.crt"
    tlsKey: "/tmp/ssl/server.key"
  syncer:
    clientKey: client2
    dbUrl: sqlite:///tmp/db/recommendations_client2.sqlite
    retentionPeriod: 90d
    intervals:
      recommendations: 10m