Public | Automated Build

Last pushed: 2 years ago
Short Description
iRODS statistical information collector for iMon (iRODS monitoring framework)
Full Description

iMon (iRODS Monitoring framework)

iRODS monitor framework running based on EFK (ElasticSearch, Flume, and Kibana).

Overview

  • Flume agent on each iRODS server sends rodsLog to port 10000 on Flume container.

  • istats collects statistical information from each iRODS server by using python-irodsclient and sends it to port 9000 on Flume container every 10 minutes.

  • You can check these information on Kibana dashboards:

    • rodsLog dashboard on the Discovery tab.

    • Statistical usage dashbard on the Dashboard tab:

      • Number of Objects: per resource, per user

      • Object size: per resource, per user

      • Number of Accesses: per user, per access-from

      • Number of Errors: per error status

      • Number of Users

Demo

  • Execute following commands on your Docker host and you will have iMon environment:
export DOCKER_SERVER="xxx.xxx.xxx.xxx"
./demo.sh
  • Above shell script boots all necessary services including iRODS as Docker containers.

  • And then iRODS users and objects are created automatically every miniute for the demo.

  • You can browse Kibana monitoring page on https://DOCKER_SERVER .

    • Go to Discover page and open rodslog search.

    • Go to Dashboard page and open tempZone dashboard.

Monitoring against existing iRODS

  • Boot containers on your Docker host:
export DOCKER_SERVER="xxx.xxx.xxx.xxx"
docker run -d --name elasticsearch -v "$PWD"/es_data:/usr/share/elasticsearch/data wtakase/elasticsearch-imon:1.7
docker run -d --name flume -p 9000:9000 -p 10000:10000 --link elasticsearch:elasticsearch wtakase/flume-imon:1.6
docker run -d --name kibana -e ELASTICSEARCH_URL=https://${DOCKER_SERVER}/es wtakase/kibana-imon:4.1
docker run -d --name httpd -p 443:443 -e DOCKER_HOST=${DOCKER_SERVER} --link elasticsearch:elasticsearch --link kibana:kibana wtakase/httpd-imon:2.4
  • Exec following shell script and set cron job for iRODS log rotation
/path/to/imon/docker/icat/rotate_rodslog.sh
echo "*/10 * * * * root /path/to/imon/docker/icat/rotate_rodslog.sh" >> /etc/crontab
  • Install Flume (flume-ng and flume-ng-agent) to your iRODS server.

  • Put /path/to/imon/docker/icat/flume.conf to /path/to/flume/conf on your iRODS server and replace FLUME_HOST with your Docker hostname.

    • Any requests to port 10000 on your Docker host will be redirected to Flume container's port.
  • Start Flume service with the flume.conf.

  • Boot istats container with your iRODS server IP (replace xxx.xxx.xxx.xxx below) and password (replace xxxxx below):

docker run -d --name istats --link flume:flume -e IRODS_PORT_1247_TCP_ADDR=xxx.xxx.xxx.xxx -e IRODS_PASSWORD=xxxxx wtakase/istats

Dockerfile

  • All Docker images are built by using docker/xxx/Dockerfile.
Docker Pull Command
Owner
wtakase
Source Repository

Comments (0)