overview/pdfocr-native repository overview

⁠pdf-processor

Processes PDFs for Overview Server.

Requirements:

• PDF processing happens in a separate process, so its parent can handle out-of-memory errors.
• It's quick.

Right now, Overview's worker runs pdf-processor as two programs: split-pdf-and-extract-text and make-pdf-searchable. See their documentation for descriptions of those scripts.

In the future, we may make pdf-processor daemons that grab work off a queue. That would help us scale up when demand spikes. In the meantime, "quick" means: processes a simple PDF in 0.1s, including startup and shutdown overhead (but excluding OCR).

⁠Developing

1. Install Docker-CE⁠ and if you're on Linux, install docker-compose⁠.
2. Run ./make && ./python3 test/*_test.py

⁠Using in your service

Use a multi-stage build in your Dockerfile. Like this:

# Download binaries as a build stage
FROM overview/pdfocr-native:latest AS pdfocr

# Okay, back to your regularly-scheduled Dockerfile....:
FROM alpine:3.7

#...

# Now you can COPY in the pdfocr binaries:
COPY --from=pdfocr /split-pdf-and-extract-text /bin/

#...

From there, you have binaries in your service, and you can invoke them as /bin/split-pdf-and-extract-text.

⁠Legacy Overview Server deployment

To deploy to non-Dockerized Overview, run these steps after you push a new version of pdfocr-native:

1. ./make clean all to generate binaries on your local machine.
2. aws s3 cp split-pdf-and-extract-text s3://overview-builds.overviewdocs.com/pdfocr-native/latest/split-pdf-and-extract-text to copy your local binary to a special S3 bucket that Overview's deployment scripts use.
3. On the server, sudo -u overview aws s3 cp s3://overview-builds.overviewdocs.com/pdfocr-native/latest/split-pdf-and-extract-text /opt/overview/ && sudo chmod +x /opt/overview/split-pdf-and-extract-text to install the new binary. (Beware the race: if Overview tries running the program during the 1s-long copy, it'll fail on that document.)

⁠split-pdf-and-extract-text

Usage: split-pdf-and-extract-text --only-extract=true IN-FILE.pdf

Opens IN-FILE.pdf as a PDF. Streams it to a proprietary, streaming-friendly binary output format on stdout:

HEADER
PAGE
PAGE
...
FOOTER

Where HEADER is:

0x1       [ 1 byte ]
nPages    [ 4-byte integer, big-endian ]

PAGE is:

0x2       [ 1 byte ]
isOcr     [ boolean, 1 byte, either 0x0 or 0x1 ]
thumbLen  [ 4-byte integer, big-endian, may be 0x00000000 ]
thumbnail [ png-encoded thumbnail, may be empty ]
pdfLen    [ 4-byte integer, big-endian, may be 0x00000000 ]
pdf       [ pdf-encoded single page, may be empty ]
textLen   [ 4-byte integer, big-endian, may be 0x00000000 ]
text      [ utf8-encoded text from the page ]

And FOOTER is:

0x3       [ 1 byte ]
len       [ 4-byte integer, big-endian, may be 0x00000000 ]
error     [ utf8-encoded text describing error; empty on success ]

Only FOOTER is actually required. In a valid stream:

• There is at most one HEADER
• If there is a HEADER, it appears before any PAGE
• If there is no HEADER, there are no PAGE entries
• There are at most nPages PAGE entries
• There is only one FOOTER

The --only-extract option will produce empty PNGs on all pages except the first, and it will produce empty PDFs.

⁠dump-split-pdf-and-extract-text-output

Usage: split-pdf-and-extract-text IN-FILE.pdf | dump-split-pdf-and-extract-text-output OUTDIR

Dumps the streamed data to a directory. Files will be:

• OUTDIR/n-pages: number of pages
• OUTDIR/p1.is-ocr: "true" or "false"
• OUTDIR/p1.png: Thumbnail (only if non-empty)
• OUTDIR/p1.pdf: Single page (only if non-empty)
• OUTDIR/p1.txt: Text (even if empty)
• OUTDIR/error.txt: Error (only if non-empty)