obigbando/vcf2es repository overview

⁠overview

This container includes scripts to import genetic variants in VCF file to Elasticsearch.
sample docker run command

    docker run --network=${net} --rm --name vcf2es obigbando/vcf2es /bin/bash
    

⁠detailed description of inside container commands

VCF to elastcisearch import flow includes the following steps:

schema setup for elasticsearch /root/demo_table.json is a table schema for current demo elasticsearch table - /demo_table/variant. Use the following command to setup the table schema for /demo_table/variant:

    curl -XPUT --header "content-type: application/JSON" ${es_container_ip}:9200/demo_table -d @"${schema.json}"
    

use the following command to check whether to schema has been properly setup:

    curl ${es_container_ip}:9200/_cat/indices/demo_table?v

vcf parsing: vcf to json conversion You can also find a demo.vcf.gz at path /root/demo.vcf.gz, which is the VCF file for this workshop. You can use python script /root/vcf2json.py to do vcf to json conversion, with sample command:

    python3 /root/vcf2json.py /root/cat.sort.vcf.gz ${json_output_dir}
    

You can find converted json files at ${json_output_dir}.

bulk import : json documents to elasticsearch Then you can use elasticsearch _bulk api to do json file bulk import, e.g.

    curl -s -H "Content-Type: application/x-ndjson" -XPOST ${es_host_ip}:9200/_bulk --data-binary "@${chunk.json}"

After bulk import of all chunk json files, you can use the following command to check whether all documents have been imported to elasticsearch /demo_table/variant

    curl ${es_host_ip}:9200/_cat/indices/demo_table?v

There should be 90361 entries in the table.