Sign inSign up

toto1310/fscrawler

By toto1310

•Updated over 6 years ago

Docker Image for FSCrawler

Image
1

10K+

toto1310/fscrawler repository overview

⁠Docker for FSCrawler

⁠Quick Reference

⁠How to Run

Refer to Getting Started⁠, you can run FSCrawler that read its configuration files from /root/.fscrawler(i.e. --config_dir) and its target files from /tmp/es(i.e. fs.url).

$ docker run -it --rm -v ${PWD}/config:/root/.fscrawler -v ${PWD}/data:/tmp/es:ro toto1310/fscrawler fscrawler job_name

⁠Using with Elasticsearch installed by Docker

If you installing Elasticsearch with Docker⁠, you need to communicate between each container.

In the following, the following directory arrangement is assumed.

  .
  ├── config
  │   └── job_name
  │       └── _settings.yaml
  ├── data
  │   └── <your files>
  └── docker-compose.yml

For example, to connect to a docker container named elasticsearch, modify your settings.yaml or settings.json.

name: "test"
elasticsearch:
  nodes:
  - url: "http://elasticsearch:9200"

After that, you have two ideas to do so.

⁠As legacy way, use --link flag:

WARNING: That is easy, but DEPRECATED feature(Docker v19.03). For more details, please read Legacy container links | Docker Documentation⁠.

For example, you run Elasticsearch in a container named "elasticsearch" as follows:

$ docker run --rm --name elasticsearch -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.3.2

Then, you can run FSCrawler as follows:

$ docker run -it --rm --link elasticsearch -v ${PWD}/config:/root/.fscrawler -v ${PWD}/data:/tmp/es:ro toto1310/fscrawler fscrawler job_name
⁠Instead of above, use docker network:

For example, prepare the following docker-compose.yml.

version: '2.2'
services:
  # FSCrawler 
  fscrawler:
    image: toto1310/fscrawler
    container_name: fscrawler
    volumes:
      - ${PWD}/config:/root/.fscrawler
      - ${PWD}/data:/tmp/es
    networks: 
      - esnet
    command: fscrawler job_name
    
  # Elasticsearch Cluster
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
    container_name: elasticsearch
    environment:
      - node.name=elasticsearch
      - discovery.seed_hosts=elasticsearch2
      - cluster.initial_master_nodes=elasticsearch,elasticsearch2
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
    container_name: elasticsearch2
    environment:
      - node.name=elasticsearch2
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=elasticsearch,elasticsearch2
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata02:/usr/share/elasticsearch/data
    networks:
      - esnet

volumes:
  esdata01:
    driver: local
  esdata02:
    driver: local

networks:
  esnet:

Then, you can run Elasticsearch.

$ docker-compose up -d elasticsearch elasticsearch2

After starting Elasticsearch, you can run FSCrawler.

$ docker-compose up fscrawler

⁠Using environment variables in FSCrawler configuration

Out-of-the-box, FSCrawler doesn't support environment variables inside FSCrawler configuration such as _settings.yaml or _settings.json. But envsubst may be used as a workaround if you need to generate your FSCrawler configuration dynamically before FSCrawler starts.

Here is an example using docker-compose.yml:

version: '3'
services:
  # FSCrawler 
  fscrawler:
    image: toto1310/fscrawler
    container_name: fscrawler
    volumes:
      - ${PWD}/config:/root/.fscrawler
      - ${PWD}/data:/tmp/es
    environment:
      - FS_URL=/tmp/es
      - ELASTICSEARCH_URL=http://elasticsearch:9200
      - ELASTICSEARCH_INDEX=job_index
    command: |
      /bin/bash -c "
        envsubst < /root/.fscrawler/_template.yaml > /root/.fscrawler/_settings.yaml \
        && fscrawler job_name"

The _template.yaml file may then contain variable references like this:

fs:
  url: "${FS_URL}"

⁠Running as non-root

By default, FSCrawler needs the user home(exactly ~/.fscrawler ) as a configuration directory but not exist, so you need to change it using the --config-dir option in addition to docker's -u option.

$ docker run -it --rm -u 1000 -v ${PWD}/config:/tmp/config -v ${PWD}/data:/tmp/es:ro toto1310/fscrawler fscrawler --config_dir /tmp/config job_name

⁠Tags and Respective Dockerfile links

Tag summary

Content type

Image

Digest

Size

339.6 MB

Last updated

over 6 years ago

docker pull toto1310/fscrawler