gusdecool/web-crawl repository overview

⁠Web Crawler

A web crawler software to crawl your web pages and report a result like:

1. How long does it take for the page to load
2. Page screenshots
3. HTTP Response code
4. Javascript and style execution time
5. Console log

⁠How to use

1. Create a list of URL in each new line and save the file inside /page folder where do you want to run the container
2. Create empty folder called /output where the report will be put

⁠Container command explanation

yarn start test.csv --emulate="iPhone 6" --no-screenshots

1. yarn start is the base command
2. test.csv required. The name of the csv file in /page folder
3. iPhone 6 optional. The device name emulator, see this page⁠ for the list of emulator option.

⁠Docker single container

docker run --rm -it --privileged \
    -v ${PWD}/output:/app/output \
    -v ${PWD}/page:/app/page \
    gusdecool/web-crawl yarn start test.csv --no-screenshots

where test.csv is the name of csv file in /page folder

⁠Docker Compose

Please create docker-compose.yml file with this content

version: '3'
services:
    app:
        image: 'gusdecool/web-crawl'
        volumes:
            - ./output:/app/output
            - ./page:/app/page
        command: bash
        privileged: true

and run command

docker-compose run app yarn start test.csv

where test.csv is the name of csv file in /page folder