alino/mongo2es repository overview

⁠Mongo2ES:

⁠v0.6

• Mongo2ES syncs data from MongoDB to ElasticSearch.
• Also automatically removes documents from ElasticSearch if they are removed in MongoDB.
• Mongo2ES is built with MeteorJs.

Why does this exist?

• ElasticSearch rivers are deprecated
• Transporter by compose⁠ is kind of stuck and unable to remove ES documents when they are removed in MongoDB.

⁠Installation

⁠install without docker

Mongo2ES runs as a Meteor application, so you need to have Meteor installed, first.

curl https://install.meteor.com/ | sh

then clone, set env variables and run it

git clone https://github.com/Alino/Mongo2ES.git
cd Mongo2ES
export elasticsearchHost="http://127.0.0.1:9200";
export MONGO_URL="mongodb://127.0.0.1:27017/dbname?replicaSet=rs&readPreference=primaryPreferred&w=majority&connectTimeoutMS=60000&socketTimeoutMS=60000"
meteor --port 3001

⁠install as docker container:

git clone https://github.com/Alino/Mongo2ES.git && cd Mongo2ES
docker build -t kuknito/mongo2es .
docker run --name Mongo2ES -d \
  -e ROOT_URL=http://localhost:3001 \
  -e MONGO_URL="mongodb://127.0.0.1:27017/dbname?replicaSet=rs&readPreference=primaryPreferred&w=majority&connectTimeoutMS=60000&socketTimeoutMS=60000" \
  -e MONGO_OPLOG_URL=mongodb://127.0.0.1:27017/local \
  -e elasticsearchHost="127.0.0.1:9200" \
  -p 3001:80 \
  kuknito/mongo2es

note: the docker image is also available at docker hub as an automated build. https://hub.docker.com/r/alino/mongo2es/⁠

⁠install as a Meteor package

meteor add alino:mongo2es

https://github.com/Alino/alino-mongo2es⁠

⁠environment variables

⁠Syncing data from MongoDB to ElasticSearch

If you want to sync MongoDB to ElasticSearch, you must define which collections you want to watch. For that, you have to write your watchers.

⁠creating a watcher

If you are ready to write your own watchers, go and create new file watchers.js in the project root. Then you create a watcher by creating new object from Mongo2ES class:

if(Meteor.isServer) {
  Meteor.startup(function() {
    new Mongo2ES(options);
  });
}

⁠options:

options = {
  collectionName: // String - name of the mongoDB collection you want to watch
  ES: {
    host: // String - url of your ElasticSearch host where you want to copy the data to.
    index: // String - ElasticSearch index
    type: // String - ElasticSearch type
  },
  transform: // Function - modify the document before copying it to ElasticSearch. Takes 1 argument - the document and should return the modified document.
             // NOTE: if you 'return false' within this function and then the transportation of this document would be totally skipped!
  copyAlreadyExistingData: // Boolean - true if  it should copy all existing data in this collection ( default: false )
}

⁠examples of watchers

You can get inspired from this example file watchersExample.md⁠

⁠stopping a watcher

watcher = new Mongo2ES(options); // create a watcher and put it into a variable
watcher.stopWatch() // stops the watcher

⁠logging

there are currently 2 options for logging in Mongo2ES.

1. Default behavior - simply shows up the logs, like console.log() does
2. logging to ElasticSearch with logstash - to enable this feature, you must set logitHost and logitPort environment variables.

Both logging options are using Meteor package alino:logit⁠

⁠Verbose mode

• to enable verbose output to container log, either set the environment variable MONGO2ES_VERBOSE=true when creating the container
• or set option verbose: true in options when instantiating Mongo2ES.

⁠limitations:

• only one mongo database can be synced to ES, because we are tailing single MONGO_OPLOG_URL

⁠contact

I can give you real time help for this project or meteor