sanketdhoble/medium-scraper repository overview

1. Keep a pendingUrlsTobeScraped array and keep adding urls fetched from scraped pages into it.   
2. Used Redis cache/db to store already scrapedUrls and kept adding scrappedUrl into it.   
3. These scrapedUrls are used to filter out already scraped urls from pendingUrlsTobeScraped array.   
4. We need throttling of 5 req/sec so we make sure each request takes 200ms. i.e if req takes 50ms we add a delay of 150ms before making next request.    
5. Store Url data in db in following way:      
    Assume scraper parses 4 URLs:     
    - https://medium.com/some/thing.     
    - https://medium.com/some/thing?param1=abc    
    - https://medium/com/some/thing?param2=xyz    
    - https://medium/com/some/thing?param1=def&param3=xxx    
    Database will contain the URL https://medium.com/some/thing with the a    
    reference count of 4. A unique list of parameters containing param1,param2,param3.    

{
    "_id" : ObjectId("5dbd8d452e34896cde22fab0"),
    "parameters" : [ 
        "redirect", 
        "source", 
        "operation"
    ],
    "url" : "https://medium.com/m/signin",
    "refCount" : 1566
}