blockaero/pdf2csv repository overview

⁠Overview

Block Aero Backend API uses a micro services based architecture. This API supports the following tasks and they are executed in the order shown below.

1. Upload⁠ a PDF file (Microservice: upload-endpoint Execution Time:0.03)
   • Split PDF if it is a multi page pdf

⁠Supported Upload Types

-  [x] multi page PDF
-  [x] single page PDF
-  [x] png image
-  [x] text file

2. OCR Pre-Processing⁠ (Microservice: ocr-tesseract Execution Time:2.5) - [x] See if the given page has text content or image content - [x] If the page has text content, forcefully convert it into image. The reason being, even though there is text content in the PDF, there could be images. - [x] Create highdpi grayscale image (=>300 DPI) for ocr after repair, clean, orient, deskew - [x] Create a lowdpi grayscale image (72) with 120X120 for thumbnail

1. OCR⁠ (Microservice: ocr-tesseract Execution Time:6.4)
   • Create an HOCR (coordinates of words identified during ocr) and text
2. Classify⁠ (Microservice: classify-fasttext Execution Time:0.9)
3. Extract⁠ Fields (Microservice: extract-fields Execution Time:1.5)

⁠Development Process - One microservice at a time.

⁠Development Environment.

You must use a proper IDE (not Notepad etc) with linting enabled for whatever programming language you use for this repo. For each microservice, you may choose whatever programming language of your choice as it fits the contract to the other microservices.

Development environment: Linux

⁠Python

IDE: Atom Python Version needed on your machine: 3.6.5 Additional requirement in your machine: pip install autopep8

Atom plugins to install:(use Atom settings view) ide-python, linter-python-pep8, language-python Python code does not autofix, the editor will show you the static lint errors only. So you run ./lint.sh path/to/python/file to do linting.

Please make sure you have no lint errors before you commit code. Make sure you make small functions which are testable and a function should do ideally only one 'function'. Make sure you don't have a high cyclomatic dependency lint error. If you have, your function design is not well planned.

⁠Bash

IDE: Atom Requirement in your machine: apt-get install shellcheck . Refer to shellcheck⁠ Atom plugins to install: language-shellscript, linter-shellcheck

⁠Docker & jq

Make sure you have the latest docker running in your machine. Make sure jq is installed in your machine.

Each microservice has one specific objective - nothing more. It expects a specific payload and outputs a specific response.

Make sure the following commands work properly

  //this should work without errors
  docker ps
  //you should see something equal or above 18
  docker version
  //this should work
  jq

⁠Issues

Any work you do, ideally make sure you have an issue associated with that, if not, create one with clear definition of work. When you commit, commit only the file related to that issue one at a time.

For example, you changed a file process.py and abc.py where process.py is changed related to issue 7 and abc.py is related to issue 8

git add processs.py
git commit -m 'fix #7'
git add abc.py
git commit -m 'fix #8'
git push origin master (if it is in master)

⁠Devops

We have three machines with the same configuration. They will be in AWS with names, api, stage.api and dev.api.

Except for production machine, other machines may or may not be available. When there is no active development, dev.api is to be deleted. Also when in fully production mode and when there is no need for staging testing, stage.api should also be deleted.

⁠How to create the machines.

This expects that you have aws previleges and has the aws cli configured with the profile name as 'blockaero'.

You need an IAM role with 1) AmazonEC2FullAccess and 2) user_creating_ec2_machine_with_role or an admin role. user_creating_ec2_machine_with_role is a custom role created with the following JSON payload.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:PassRole",
                "iam:ListInstanceProfiles",
                "ec2:*"
            ],
            "Resource": "*"
        }
    ]
}

Clone the repo, blockaero-devops and run the following command.

  //dev machine
  resources/scripts/linux/create_aws_machine_without_iac.sh dev.api
  //staging machine
  resources/scripts/linux/create_aws_machine_without_iac.sh stage.api
  //production machine
  resources/scripts/linux/create_aws_machine_without_iac.sh api

The additional step which you need to do is to update the route53 entries for the domain in aws console once the IP address is available. For this you need the IAM policy AWSRoute53FullAccess policy attached to your IAM role or request somebody who has that permission to create the entries.

For example, for dev you need dev.api.block.aero and minio.dev.api.block.aero mapped to the IP address of the dev machine. These entries will already be available, just change the IP address.

To add the domains in route53, navigate to list of hosted zones and click api.block.aero.Click 'Create Record Set'.In the name field, just enter the name of the domain, for example dev. Enter Value field with IP address to create an A record.

First time when you are setting up the machine, you need to update the .bashrc file in the $HOME folder where you are working. For example,

  export AIMODELS="/home/lucid/ai-models"
  export ROOTDOMAIN="api.block.aero"

With the above configuration, you can deploy the pipeline for example for dev as follows.

./deploy.sh dev

If you just want to make sure all dependencies are available and want to proceed with something else, for example, do s3 sync manually, do the following.

./deploy.sh dev install

⁠Syncing AIMODELS

You need to make sure you have synced the latest AIMODELS. Following is an example of usage based on the above shown directory structure.

aws s3 sync s3://blockaero-ai-models /home/lucid/ai-models &

⁠How to remove the machines

Same command as above, except that you add the keyword remove. For example, to remove dev.api, run the following. It will remove all resources connected to that machine.

  resources/scripts/linux/create_aws_machine_without_iac.sh dev.api remove

⁠Git Process

We are using a simplified gitflow in our approach.

1. Whenever you are creating a new feature, create a branch from master with an appropriate name. Use the DEV deployment method and test in your local environment.
2. Once your work is complete, create a branch with name dev and use the dev branch to deploy it to dev endpoint for testing. dev.api.block.aero
3. Once the business is happy with the new endpoint, merge with master and deploy to the staging endpoint. stage.api.block.aero
4. If there is no staging deployment needed, create a release and deploy to production endpoint, api.block.aero. Make sure you have updated the CHANGELOG⁠ before doing this.
5. Whenever you do any fixes on a release, create a minor version from the release branch and follow the same steps as outlined below.

⁠How to do dev release for testing.

1. Create a dev branch off your feature branch. Make sure you are on the dev branch.

  $ git branch
  dev

1. Fetch all remote updates

  $ git remote update
  Fetching origin
  $ git push origin dev

You should be then pulling dev in your dev endpoint machine and deploy based on the subdomainname.

⁠How to do a production release

1. Make sure you are in the dev branch

  $ git branch
  dev
  git checkout -b release/0.1.0 origin/dev

1. Push to a release repo

  git push origin release/0.1.0

1. Considerations If you go to GitHub, unlike feature branches, there won't be a notification bar waiting for you to open a new "pull request" for the release branch you've just pushed so you need to do it manually instead. To open a "pull request" for the release branch, hit "New pull request" button, compare master (base dropdown) branch to release/0.1.0 (compare dropdown) branch, write a subject and a description for it. This will open a "pull request" so that it can be reviewed by other team members. If what commits you're preparing to release are fine for the rest of the team members then there is nothing to worry about. This process just shows everyone what will be released. Do not use "Merge pull request" button.

2. Checkout into master branch.

  $ git checkout master
  Switched to branch 'master'
  Your branch is up-to-date with 'origin/master'.

1. Update local master branch

  $ git pull origin master

1. Merge release branch into master branch.

Do not use --no-ff flag otherwise merge will use "recursive" strategy instead of "fast-forward" and this will lead GitHub to create a "pull request" notification after pushing master branch to remote which is an unwanted behavior.

  $ git merge release/0.1.0

1. Tag the release point by creating a new tag.

  $ git tag -a 0.1.0 -m 'Create release tag 0.1.0'
  Verify the tag is created
  $ git tag
  0.1.0

1. Push master branch to remote repository.

In GitHub, this will also merge and close the "pull request" of remote release branch into master branch however it won't delete it. You'll see that in GitHub.

  $ git push origin --tags
  Counting objects: 1, done.

  Note: Clean up the dev after usage of the specific work if needed (only after testing is over and you have done the corresponding release.)
  $ git branch -D dev
  git push orign --delete dev

⁠Developing in development mode in a local laptop

//make sure you have made an entry 127.0.0.1       minio.localhost in your /etc/hosts file
./deploy.sh DEV

// you should be able to see the dev webhook by running the following
docker logs webhook --follow

⁠Deploying in production mode

In this mode, you cannot have any uncommitted code, the script will not deploy for changes you try to deploy directly. Make sure you have tested in a DEV environment first, before you attempt this.

// make sure you have an environment variable for ROOTDOMAIN
// for example, you may set ROOTDOMAIN=api.block.aero in your .bashrc file.
./deploy.sh subdomainname

//you should then be able to access endpoints in https://subdomainname.ROOTDOMAIN
//all minio will be accessible from https://minio.subdomainname.ROOTDOMAIN
//you should have created the necessary entries in aws route53

//./deploy.sh subdomainname will build whatever is in the repo and push to dockerhub before deploying.
//however, you may just want to give the customer the deployment script without the code. or you just want to standup something which is already there.
./deploy.sh subdomainname nobuild

⁠Scaling Services.

By default when you deploy, only one task will be done at a time. However, you may want to have parallel processing in your production deployment. However, how many you want or can have will depend on the host capacity or other joined machines to the host. We haven't automated that scenario for now. As this service is intended to be deployed by a customer and within his premises and it will greatly depend on the capacity there.

The service to scale for multi tasking is func_gateway. For example, if you wish 5 tasks to run parallelly, you may do as follows.

docker service scale func_gateway=5

As of now, most time consuming process is ocr-tesseract, you may scale that up. We know each will take a maximum of 4 CPUs, so for a 16 core machine, we could do as follows.

docker service scale ocr-tesseract=3

An example combination of scaling may look like the following.

  docker service scale func_gateway=30 && docker service scale ocr-tesseract=4 && docker service scale func_queue-worker=10 && docker service scale classify-fasttext=3

⁠Useful Browser Endpoints - Currently open(will be closed, restricted after MVP)

object storage⁠ function dashboard⁠

⁠Common Issues

If you get an error like "dial tcp 127.0.0.1:8080: getsockopt: connection refused, ", there could be various network reasons. Easy fix is to do the following.

// this is already done automatically in DEV. There is no need to do the following in DEV. Only in PRODUCTION if needed.
docker swarm leave --force
//then deploy again

// In dev environment. If calling 127.0.0.1:8080 to invoke function does not work, your network interface has some issue, use the exact IP which you can get by following.
NODE_ADDRESS="$(docker node inspect "$(docker node ls --format '{{.Hostname}}' --filter 'role=manager')" | jq '.[0].Status.Addr' | tr -d '"')"

⁠Useful Docker commands.

⁠Debug service.

docker service logs servicename Ex: docker service logs upload-endpoint docker service ls - lists the services running

⁠Vewing logs

For applicable services, papertrail centralized log management is enabled. Please login to papertrail⁠

As the application is a swarm cluster, the name of the machine will be the name underwhich systems will get auto registered.

⁠Data directory and persistence for functions

We are using Object Storage using minion which supports various providers like AWS, Azure etc. During development version (when working locally), same api is used where an object storage is created locally where there is no network traffic.

Blockaero multi-tenant scenario in the future may use various Azure/Object stores. (during mvp release, there is only one client, ie one Object Store in Azure). For each tenant, we use a strategy to run a separate minio service. Minio uses http protocol encapsulated and normal copy operation and sync operation will work.

Although we evaluated the possibility of NFS mount (Network File System) using EMC rexray, currently there is no standard way to have the plugin support multiple Object Stores of the same type (S3FS).⁠

Threfore the solution is to have the minio as a service in swarm, then all file operation to local and mirror using mc (minion client).

⁠Testing against production/staging API end point

Basically you need a webhook, thats it. Easiest way is to run the development environment as normal. Make a note of the ngrock webhook url. Then use that URL as the callback url when you call the api end point.