chusiang/crawler-book-info repository overview

⁠Crawler Book Info

A sample crawler for quick parser some books information.

⁠Initialization

1. Install the virtualenv.

   [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ sudo pip3 install virtualenv
   

   Copy

2. create virtualenv.

   [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ virtualenv -p python3 .venv
   

   Copy

3. Enter the virtualenv.

   [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ . .venv/bin/activate
   

   Copy

4. Install packages with pip.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ pip3 install -r requirements.txt
   

   Copy

⁠Usage

⁠tenlong.com.tw

1. Run crawler with ISBN-13.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ python3 tenlong.py 9781491915325
   

   Copy

2. (option) Run crawler via make.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ make telong 9781491915325
   

   Copy

⁠books.com.tw

1. Run crawler with url.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ python3 books.py https://www.books.com.tw/products/0010810939
   

   Copy

2. Run crawler with product number.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ python3 books.py 0010810939
   

   Copy

⁠View Result

1. Open html via Firefox on GNU/Linux.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ firefox index.html
   

   Copy

   

2. We can see the https://www.tenlong.com.tw/products/9781491915325⁠, it is clean, now.

⁠Run local Nginx for Evernote Web Clipper

The Evernote Web Clipper is not support local files, so we can clip it with Nginx.

1. Run Nginx container.

   $ docker run --name nginx -v "$(pwd)":/usr/share/nginx/html/ -p 80:80 -d nginx
   

   Copy

2. Open html via Firefox on GNU/Linux.

   (.venv) [ jonny@xenial ~/vcs/crawler-book-outline ]
   $ firefox http://localhost
   

   Copy

3. (option) Run Nginx container via make.

   $ make run_containers
   

   Copy

4. (option) Open web via make.

   $ make review_serve
   

   Copy

5. Finally, we can clip the information to Evernote with Evernote Web Clipper⁠.

⁠License

Copyright (c) chusiang from 2017-2019 under the MIT license.