HUB
Sign inSign up

blys/unoconv-s3

By blys

•Updated over 1 year ago

scalable document files format converter service

Image
0

1.8K

blys/unoconv-s3 repository overview

⁠Unoconv-S3 ver. 1.0.1

https://github.com/pulciux/unoconv-S3⁠

Is a scalable document files format converter service based on nodejs, AWS S3 object sotrage (and compatible), libreoffice and unoconv, made to run in a dockerized environment.

⁠The problem

Let's say that in your application you have to manage the conversion of document files such as MS-Word files, Open Document files, Rich text files, spreadsheets, presentations and so on, to a different format.
You want to be able to easily scale up this feature as the load of your system grows.

Unoconv⁠ can do conversions for you, exploiting the ability of LibreOffice⁠ and OpenOffice⁠ to work as a server and without GUI.

Unfortunately, it has to rely on the local filesystem, that it makes it a bit complicated to scale up in a distributed environment.

⁠A solution

You can store the files you need to convert, in an Object Store Service such as AWS S3 or Google Cloud Storage, that is an easy and efficient way to makes your files available by every node in your distributed system.

Probably you already have your files in an Object Storage.

So, why don't make Unoconv able to get the files from your Object Storage, convert them and then put them back from where they come?

This is the idea.

Add a simple RESTfull interface to control the process, pack everything in a Docker container, and you have Unoconv-S3.

NodeJS makes the trick!

⁠How to use it

You can run it in your laptop as well in a single node multi-container application using docker-compose or in an orchestrated multi-node distributed environment as Docker Swarm or Kubernetes.

In this latter case, the only thing you need to scale up is to say scale n pods or services. No file system mount to manage.

⁠Run the the docker image:
docker run --rm -d -p 80:9080 -e AWS_ACCESS_KEY=<your key> -e AWS_SECRET=<your secret> blys/unoconv-s3

Where AWS_ACCESS_KEY and AWS_SECRET are environment variables to allow the service to connect to your Object Storage.

See also: aws security credentials⁠

⁠Ask for converting a document file:

From a browser or by CLI with curl.

http://localhost/my-bucket-name/my-object-key?format=pdf

Where my-bucket-name is the bucket where the document to convert is stored, and my-object-key is the key of the object containing the data of the source file.

Parameter format is the target format you wish to obtain.

⁠The result:

If nothing different is specified, the converted file will be stored in the same bucket of the original one, with equal object key but different extension.

As result of the HTTP call, you'll get a JSON document describing what is happened.

e.g., Let's say you want to convert an MS-Word file, located in user-docs bucket and identified by my fantastic life.docx, into a PDF file.

Then call:

curl "http://localhost/user-docs/my fantastic life.doc?format=pdf"

And you'll get something similar to:

{
  "code": 0,
  "text": "conversion complete",
  "beginTime": 1517206908.701,
  "endTime": 1517206910.364,
  "execTime": 1.6630001068115234,
  "result": {
    "key": "my fantastic life.pdf",
    "bucket": "user-docs",
    "data": {
      "ETag": "\"843f90ab9ec5ea23ef23fb18a3cc1551\""
    },
    "metadata": {
      "userdeclaredsize": "149504",
      "userdeclaredtype": "application/msword",
      "masterDocMD5": "ad18fc9a2e1fd4761671e137d3a30fc6"
    }
  }
}

code: 0 says the conversion has succeeded, it took 1.66 seconds, and the converted document is now available in bucket user-docs with key my fantastic life.pdf .

Additional info are returned such as begin and end time, metadata which has been copied from the source object with in addition a new entry, namely masterDocMD5 whitch is the source object ETag without double quotes.

⁠Different bucket and/or object key target:

If you don't want the converted file to be stored in the same bucket of the original one or you want to specify a customized key, you can use two more optional parameters: dbucket and dkey.

curl "http://localhost/user-docs/my fantastic life.doc?format=pdf&dkey=my amazing life.pdf&dbucket=converted-user-docs"
⁠List of available parameters:
parammeaningmandatory
formatidentify the target format of the converted file.true
dbucketname of the bucket where the converted object will be stored. If not specified it will be the same as the source object.false
dkeykey of the converted object. If not specified it will be the same as the source object with the extension changed or added.false
⁠Alternatives to ASW S3 and not deafult region:

If you don't use the actual ASW S3 service, you can use a compatible one.

Some example: Google Cloud Storage⁠, Minio⁠, ...

To make Unoconv-S3 able to point to a different service, you can specify some additional environment variables.

variablemeaningmandatory
AWS_ACCESS_KEYAWS, or compatible, unique identifiertrue
AWS_SECRETAWS, or compatible, secret keytrue
AWS_ENDPOINTURL to the alternative servicefalse
AWS_REGIONthe name of the region to usefalse
AWS_SSLENABLEDif false it will allow to connect without SSL encryption. True is defaultfalse
AWS_S3FORCEPATHSTYLEif true it will force the path style. False is defaultfalse
⁠Usage of environment variables:

You can specify them using the -e option in docker run command.

You can also use them in a docker-compose file, of course.

⁠Example:
version: '3'
services:
    doc-converter:
        image: blys/unoconv-s3
        environment:
            - AWS_ACCESS_KEY=my-unique-key
            - AWS_SECRET=my-secret-key
            - AWS_ENDPOINT=s3.compatible.service.example.org
            - AWS_REGION=us-east-1
            - AWS_SSLENABLED=true
            - AWS_S3FORCEPATHSTYLE=true        
        networks:
            - objStorage
    doc-uploader:
        image: nginx
        volumes:
            - projFile:/var/www/html
        networks:
            - frontend
            - objStorage
⁠Return codes:

If something goes wrong the service will respond with an appropriate HTTP status code and additional info will be available by code and text parameters of the returned JSON document.

  • 0: conversion complete
  • 1: missing source object path
  • 2: missing destination format
  • 3: source object not found
  • 4: source object unavailable
  • 5: temporary data can't be instantiated
  • 6: source datablock can't be retrieved
  • 7: converted datablock can't be read
  • 8: conversion process can't be started
  • 9: conversion process didn't create a converted document
  • 10: converted datablock can't be stored

Tag summary

Content type

Image

Digest

Size

450.2 MB

Last updated

over 7 years ago

docker pull blys/unoconv-s3