imshashank/sepana repository overview

⁠Sepana Search Module

⁠Build & Run

docker build -t imshashank/sepana .

docker run -it -p 80:80 imshashank/sepana

Then you can send search queries to http://127.0.0.1:80/search

⁠Infrastructure

Search Engine: ElasticSearch Trial (hosted on elastic.co) RateLimit: https://pypi.org/project/fastapi-limiter/⁠ + Redis Cloud

Starting server locally(for testing):

 uvicorn src.sepana_search.main:app --reload

⁠Example Search query:

URL: http://127.0.0.1:80/search⁠

Type: POST

{
    "post": "bitcoin",
    "user_bio": "crypto",
    "offset": 0,
    "limit": 20
}

⁠Indexing data:

Run file: ./src/sepana_search/import_data/import_data.py

The data is converted to a pandas dataframe and the entire dataframe is added to ES. This was the fastest approach for indexing around 64k records.

Other approaches used but discarded as they took too long:

• Read CSV using python and add individual record.

• Read CSV as Pandas df and send individual record.

• Read CSV as Pandas df and send batched record.

• P.S. The old code is in the file but has been commented out.

⁠Search

The search backend is ElasticSearch. I used the free elastic.co Trial.

The search implemented was based on the requirements in the doc and it can be easily modified to support other use-cases.

⁠Rate limit

Rate Limiting is implemented using the library: https://pypi.org/project/fastapi-limiter/⁠ with a redis database.

These are the rules for /seach API:

 # 10k API requests per 24 hours
 Depends(RateLimiter(times=10000, hours=24)),
 # 100 API calls/minute
 Depends(RateLimiter(times=100, minutes=1))

The rate limit is based on the user's IP address. A user-id based rate-limiter can also be implemented using the same library

⁠TODO/Recommendations

• Save all passwords in .env file
• Create classes to re-use objects. Most methods are static for now.
• Add tests
• Optimize Search query
• Add userID based rate limiting
• Implement Caching for queries