jiangweiyao/aws_html_scraper repository overview

⁠HTML link scraper and unzipper

This Jupyter notebook contains code showing how to scrape links from a HTML webpage using BS4, download the links, and unzip them. All the actions are also written into a single function for convenient use.

All the dependencies can installed via a Conda environment. Additionally, a Docker image was created from the included environment.yml and Dockerfile at https://hub.docker.com/r/jiangweiyao/aws_html_scraper⁠

⁠Download the repository

You can download the repository using a standard git command

git clone https://github.com/jiangweiyao/AWS_html_scraper.git

⁠Build the Conda Environment

A Conda environment can be replicated from included environment.yml file to include all the dependencies. Run the following command to create the environment:

conda env create -f environment.yml

Activate the environment and the containing jupyter-lab program using the following command:

conda activate htmlscraper
jupyter-lab

Use the link generated by jupyter to access the server via a browser. You can now open the included jupyter notebook, and run the code in the original environment it was written in.

⁠Run Container with Singularity

Activate the jupyter-lab from the container image using Singularity:

singularity exec docker://jiangweiyao/aws_html_scraper jupyter-lab

Use the link generated by jupyter to access the server via a browser. You can now open the included jupyter notebook, and run the code in the original environment it was written in.

The image can also be run in docker, but is more complicated and system dependent, and therefore out of the scope here.

⁠Author

• Jiangwei Yao

⁠License

Apache License 2.0