qmcgaw/htmlspitter repository overview

⁠HTMLSpitter

Lightweight Docker image with NodeJS server to spit out HTML from loaded JS using Puppeteer and Chrome

Medium story: HTML from the Javascript world⁠

The program is written in NodeJS with Typescript, in the src⁠ directory.

⁠Description

Runs a NodeJS server accepting HTTP requests with two URL parameters:

• url which is the URL to prerender into HTML
• wait which is the optional load event to wait for before stopping the prerendering. It can be:
  • load (wait for the load event)
  • domcontentloaded (wait for the DOMContentLoaded event)
  • networkidle0 (default, wait until there is no network connections for at least 500 ms)
  • networkidle2 (wait until there are less than 3 network connections for at least 500 ms)

For example:

http://localhost:8000/?url=https://github.com/qdm12/htmlspitter

• The server scales up Chromium instances if needed
• It limits the number of opened pages per instance to prevent one page crashing all the other pages
• It has a 1 hour cache for loaded HTML
• It has a queue system for requests once the maximum number of pages/chromium instances is reached
• Not compatible with other architectures than amd64 as Chrome-Beta is only built for amd64 for now and is required.

⁠Usage

1. Run the container

   docker run -it --rm --init -p 8000:8000 qmcgaw/htmlspitter
   

   Copy

   You can also use docker-compose.yml⁠.

⁠Environment variables

⁠Troubleshooting

⁠Chrome fails to launch

If you obtain the error:

{"error":"Error: Failed to launch chrome!\nFailed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted\n\n\nTROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md\n"}

Then you might need to use seccomp with the chrome.json⁠ file of this repository:

wget https://raw.githubusercontent.com/qdm12/htmlspitter/master/chrome.json
docker run -it --rm --init --security-opt seccomp=$(pwd)/chrome.json -p 8000:8000 qmcgaw/htmlspitter

⁠Details

⁠Program

• A built-in local memory cache holds HTML content obtained the last hour and is limited in the size of characters it contains.
• A built-in pool of Chromium instances creates and removes Chromium instances according to the server load.
• Each Chromium instance has a limited number of pages so that if one page crashes Chromium, not all page loads are lost.
• As Chromium caches content, each instance is destroyed and re-created once it reaches a certain number of page loads.

⁠Docker

• chrome.json⁠ may be required depending on your host OS.
• The --init flag is added to prevent eventual zombie Chromium processes to exist when the container stops the main NodeJS program.
• A built in healthcheck is implemented by running node build/healthcheck.js against a running instance.

⁠Performance considerations

• Chromium is written in C++ and multi threaded so it scales well with more CPU cores
• The NodeJS program should not be the bottleneck because all the work is done by Chromium
• The bottleneck will be CPU and especially RAM used by Chromium instance(s)
• You can scale up by having multiple machines running the program, behind a load balancer

⁠Development

• Either use the Docker container development image with Visual Studio Code and the remote development extension
• Or install Node and NPM on your machine

# Install all dependencies
npm i
# Transcompile the Typescript code to Javascript and run build/main.js with
npm run start

Test it with, for example:

wget -qO- http://localhost:8000/?url=https://github.com/qdm12/htmlspitter

You can also:

• Run tests

  npm t
  

  Copy

• Run the sever with hot reload (performs npm run start on each .ts change)

  npx nodemon
  

  Copy

• Build Docker

  docker build -t qmcgaw/htmlspitter .
  

  Copy

  You can also specify the branch of Google Chrome from beta (default), stable and unstable

  docker build -t qmcgaw/htmlspitter --build-arg GOOGLE_CHROME_BRANCH=unstable
  

  Copy

• There are two environment variables you might find useful:

  • PORT to set the HTTP server listening port
  • CHROME_BIN which is the path to the Chrome binary or Puppeteer-bundled

⁠TODOs

• Show Chrome version at start
• Fake user agents
• Prevent recursive calls to localhost
• Format JSON or raw HTML
• Limit Chromium instances in terms of RAM
• Compression Gzip
• Sync same URL with Redis (not getting twice the same URL)
• Sync Cache with Postgresql or Redis depending on size
• Limit data size in Postgresql according to time created
• Unit testing
• ReactJS GUI
• Static binary in Scratch Docker image

⁠Credits

• Credits to jessfraz⁠ for chrome.json⁠
• The Google Chrome team
• The Puppeteer developers

⁠License

This repository is under an MIT license⁠