altosys/bliss repository overview

⁠Bliss

Bliss is a book price monitoring application written in Go to collect, store, and process price listings from online retailers. The collected price data is stored in a local database (InfluxDB⁠) and can be viewed on the supplied dashboard (Grafana⁠). State is saved in in-memory storage (Redis⁠). Bliss also supports sending push notifications using the Pushover⁠ mobile app so you can get notified on your phone when prices drop, books become available for order and more.

Bliss on Gitlab⁠
Bliss on Docker Hub⁠

⁠Screenshots

Pushover notifications on mobile

Grafana dashboard in browser

⁠Usage

The following topics explain how to use Bliss, what features are available, and the overall structure of the system. Bliss is packaged to run in docker, and set up to start "out of the box" using the default docker-compose.yml file in the repository with no changes needed to try it out.

⁠Running Bliss

Run docker-compose up --detach to start Bliss. Logs can be read with docker logs --follow bliss, log verbosity is set with BLISS_LOG_LEVEL.

Edit the lines of the BLISS_BOOKS variable to add or remove books and restart Bliss by running docker-compose up --detach again. The line format for BLISS_BOOKS is <ISBN> <Format> <Title>, where <ISBN> is an ISBN-13 number and <Format> is optional and one of SC (softcover) or HC (hardcover). Optionally it's possible to use BLISS_BOOKS_FILE to load the books list from a yaml file instead, see docker-compose.yml and books.example.yml for an example. If both BLISS_BOOKS and BLISS_BOOKS_FILE are defined BLISS_BOOKS will take precendence.

Note: indentation is important in yaml. See the default docker-compose.yml for examples.

To stop Bliss run docker-compose down and clean up docker images and volumes with docker system prune --all --volumes if uninstalling and removing all data.

Why should I specify the book format when Bliss uses ISBN anyway?
Some sites, like Amazon, list multiple formats of a book as purchase options even when searching for a specific ISBN. Specifying which format the book is in allows Bliss to fetch the correct price even if other formats are returned in the search results. If the book format is not specified in the configuration Bliss will fetch it automatically, in case Bliss fetches the wrong format for a book it can be overridden by adding the format to the configuration and restarting Bliss.

⁠Dashboard

Grafana is the supplied dashboard for visualizing the data collected by Bliss. The dashboard is reached at localhost:3000 by default.

⁠Metrics

Bliss exposes Prometheus metrics at localhost:8085/metrics by default (the http server listens to BLISS_HTTP_PORT). Metrics are only exposed for books which Bliss has been able to find listings for, i.e. if a metric is not exposed for a book it either hasn't been scraped yet, or the book is not available at any retailers. Consequently previously exposed lowest price metrics will be removed if books go out of stock.

⁠Configuration

The table below lists all configuration parameters that can be set for Bliss and what values are expected.
For example configuration, see the docker-compose.yml file.

⁠System Overview

Bliss as a system is composed out of 3 main components:

• Bliss - web scraper and data processor
• InfluxDB⁠ - a time series database which stores the collected data.
• Grafana⁠ - a web dashboard which displays the collected data as graphs and tables.
• Redis⁠ - an in-memory data storage solution for keeping state across restarts.

Optionally (but highly recommended) a user can set up the Pushover⁠ mobile app to receive event notifications from Bliss (see Notifications⁠). Pushover is the only component of Bliss which is not free, the cost of the app however is a small one-time charge.

flowchart LR
A(Bliss) -->|Listing| B{Bliss}
B{Bliss} -->|Request| A(Retailer)
B <-->|State| G(Redis)
B -->|Notifications| D(Pushover)
B -->|Metrics| E(HTTP /metrics)
B -->|Listings| F(InfluxDB)
F -->|Display| H(Grafana)
H o--o I(User)
D o--o I(User)

⁠Scraping

Bliss scrapes book prices from enabled sites by continuously iterating over the books defined in the BLISS_BOOKS variable or in a yaml file at the path defined by BLISS_BOOKS_FILE.

⁠Priority Order

The order of scraping depends on the existence of previous listings as detailed below. The prioritization of listings to be scraped optimizes the time to establish the baseline (see Listings Baseline⁠).

• If there are no previous listings the order of scraping is the numerical order of the ISBN numbers.
• If previous listings are loaded from the database, any books that are missing listings will be prioritized to be scraped first, after which scraping will continue from the next item after the last one scraped.

⁠Request Timing

Books are scraped with a time interval between requests, set as minutes with BLISS_INTERVAL. The number of books in the books list multiplied by the number of minutes in BLISS_INTERVAL make up a scrape cycle. The scrape cycle is the longest amount of time it can take to scrape all the books listed before starting over on a new cycle. The calculated cycle time is logged when Bliss starts up.

A random time jitter is used to shorten each scrape interval in order to add some irregularity to request times.

Example: A list of 10 books with a scrape interval of 6 minutes will take at most 10*6=60 minutes to scrape before starting over on a new cycle. The random time jitter always shortens the interval, meaning that the cycle will never be longer than 60 minutes in this case, but likely a bit shorter.

NOTE: BLISS_INTERVAL has a lowest accepted value of 1 minute to prevent excessive requests to sites.

⁠Notifications

Bliss can send notifications using the Pushover API⁠. Enable notifications by setting the BLISS_PUSHOVER_USER_KEY and BLISS_PUSHOVER_API_TOKEN variables, or omit them to leave notifications disabled.

⁠Listings Baseline

Event notifications will not be sent until baseline listings have been established. All enabled scrapers need to scrape all the books listed in BLISS_BOOKS at least once (or have a previous listing loaded from the database at startup) to complete the baseline. Once the baseline is established a notification will be sent to inform that event notifications have been enabled.

⁠Notification Types

These are the types of notifications sent by Bliss:

⁠Price Drop Notifications

Bliss can be configured to only send price drop notifications at and above a certain threshold. This can be useful to prevent excessive notifications being sent due to small price adjustments at retailers. Setting BLISS_NOTIFY_DROP_THRESHOLD to 10 will only send notifications when a price drops 10% or more.

⁠License

Bliss is licensed under the terms of GNU GPLv3⁠

⁠Did you find bliss?

Did you find Bliss useful, and perhaps managed to snag a book at a great price?
Consider showing your appreciation by donating!

Copyright © 2020 Tommy Alatalo