zenlaw/model repository overview

⁠MLBUDDY

⁠Run api server locally on port 8001 (HTTP + GRPC)

cd go
make local-api

⁠Generate protobuf and gorm files

make generate

⁠Python dependencies

Linux

sudo apt install cmake python3.9-dev libpoppler-cpp-dev python-numpy build-essential

macos:

brew install mupdf swig freetyp poppler pdftotext ocrmypdf pandoc antiword

for m1 mac users, tokenizer depency might be broken, just add tokenizers into python/spacy folder:

poetry add tokenizers 

⁠Deploy docker image with "testing" tag

cd go
make docker.build && docker tag zenlaw/mlbuddy:latest zenlaw/mlbuddy:testing && docker push zenlaw/mlbuddy:testing

⁠Backup SQL crontab

PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:
WEEKLY BACKUPS
00 04 * * 00  /path/to/mlbuddy/deployments/platform-dev/db_backup.sh weekly
MONTHLY BACKUPS
00 04 01 * * /path/to/mlbuddy/deployments/platform-dev/db_backup.sh monthly

don't forget to chmod u+x db_backup.sh for avoid permissions problems and make it executable make sure to create the folder db_weekly_backups, db_monthly_backups inside mlbuddy-api

⁠Load DB into local env

⁠retrieve the sql container

docker ps | grep maria

⁠copy source file into root directory of the container (909 is the last 3 number of the ID, if not working take the whole ID)

docker cp ./mlbuddy_db_bkp1608.sql 909:/

⁠get into the container and load the file into the db

docker exec -it 909 bash

mysql -u mlbuddy -puns3cur3 mlbuddy < mlbuddy_db_bkp1608.sql

⁠Mltools update

To update mltools package to the latest version:

If mltool package and its dependencies are updated, python/spacy package needs also to update its dependencies:

cd python/spacy
poetry lock
poetry install

⁠Scrapping

To be able to scrap document you need to have a running chrome process in headless mode, for this you have two options:

1. Run a chromedp docker container: docker run -d -p 9222:9222 --rm --name headless-shell --shm-size 5G chromedp/headless-shell
2. Run directly a chrome process with following parameters: --headless --remote-debugging-port=9222 --disable-gpu --no-sandbox https://chromium.org Windows

   C:\Program Files\Google\Chrome\Application>chrome.exe --headless --remote-debugging-port=9222 --disable-gpu --no-sandbox https://chromium.org
   

   Copy
   MacOS

   /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --headless --remote-debugging-port=9222 --disable-gpu --no-sandbox https://chromium.org
   

   Copy

If the scrapper doesn't want to stop after doing a ctrl-c, you can type: Pkill mlbuddy

⁠API Documentation

The api documentation is automatically generated in swagger 2.0 format through protoc-gen-swagger from mbapi_external proto file. We are using the platform readme.io as a frontend and host provider for our api docs. Here is the command to update the doc with a new swagger file: rdme openapi ../swagger/mbapi_external.swagger.json To install rdme cli: npm install -g rdme