phatpham9/scraper
An html scraper microservice based on x-ray & micro
10K+
phatpham9/scraper repository overview
scraper
An html scraper microservice based on x-ray & micro
Features
Usage
Request
Send a GET request to /scrape endpoint with query string if:
- Scraping a text
| Params | Required | Description |
|---|---|---|
| s-url | yes | destination website url to be scraped |
| s-selector | yes | css selector of data to be extracted |
- Scraping multiple of data objects
| Params | Required | Description |
|---|---|---|
| s-url | yes | destination website url to be scraped |
| s-scope | yes | css selector of data's scope |
| s-limit | no | limit number of objects returned |
| [selector] | yes | css selector of each data to be extracted |
Response
A text or an array of objects in json whose keys are specified selectors in the request's query string.
Examples
Scraping Bitcoin price in USD from CoinMarketCap
- Request (uri encoded):
https://scraper.fun/scrape?s-url=https://coinmarketcap.com&s-selector=%23id-bitcoin%20.price - Response: as shown below
Scraping top 3 coins' price
- Request (uri encoded):
https://scraper.fun/scrape?s-url=https://coinmarketcap.com&s-scope=table%23currencies%20tbody%20tr&name=.currency-name%20.currency-name-container&price=.price&s-limit=3 - Response: as shown below
Development & deployment guide
Getting started
Make sure NodeJS (9.0.0 or newer), Yarn or NPM installed on your local machine. Then install project dependencies by running:
yarn
Start developing
yarn start
The service will be up at 127.0.0.1:9000 by default
Testing
We use ESLint to lint source code. Simply run:
yarn test
Running in production mode
By the command:
PORT=80 yarn serve
The app will be up at 127.0.0.1
Deploy using Docker
You can use the existing docker image from https://hub.docker.com/r/phatpham9/scraper by running:
docker pull phatpham9/scraper
docker run -d -p 80:80 phatpham9/scraper
The app will be up at 127.0.0.1
Deploy to CaptainDuckDuck
CaptainDuckDuck is a nice heroku-liked tool to deploy your apps easily. You need to install CaptainDuckDuck client on your local, follow the instruction here to do it then run on your local:
captainduckduck deploy
That's it!
Deploy to Heroku
Click the below button to deploy to Heroku dyno
Contributing
- Fork this repository to your own GitHub account and then clone it to your local device
- Follow the Development guide or just simply run:
yarn start - Lint code by running: yarn test
- Create a pull request for us
Contributing
- Phat Pham (@phatpham9)
Tag summary
Content type
Image
Digest
Size
45.6 MB
Last updated
over 8 years ago
docker pull phatpham9/scraper