conscia/grafana repository overview

⁠Docker Compose files

Collection of Dockerfile for different environments suppported by Conscia

⁠Caviets

 # set on host machine
 sysctl -w vm.max_map_count=262144
 sysctl -w vm.swappiness=0

Update /etc/security/limits.conf

*         -     nofile    100000
*         -     memlock   unlimited
*         -     nproc     1000000

Update /etc/sysctl.conf

# raising the global open file limit
fs.file-max = 6815744
vm.swappiness=0
vm.max_map_count=262144

⁠Setting fs.file-max⁠

⁠Disable all swap files

Usually Elasticsearch is the only service running on a box, and its memory usage is controlled by the JVM options. There should be no need to have swap enabled.

On Linux systems, you can disable swap temporarily by running:

To disable it permanently, you will need to edit the /etc/fstab file and comment out any lines that contain the word swap.

⁠Disable swap permanently

# Following change will keep swap off after reboot too
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

⁠Elasticsearch Configuration

More information on configuring instances for optimal Elasticsearch performance can be found here : https://www.elastic.co/guide/en/elasticsearch/reference/6.x/system-config.html⁠

⁠Running Docker with UFW

If you insist on using the -p flag, tell docker not to touch your iptables by disabling them in /etc/docker/daemon.json and restarting:

{ "iptables" : false }

sudo service docker stop then sudo service docker start FINALLY docker is simply following the appropriate rules in UFW.

/etc/defaults/ufw: DEFAULT_FORWARD_POLICY="ACCEPT"
/etc/defaults/docker: DOCKER_OPTS="--iptables=false"

ufw enable
ufw allow 22/tcp
ufw deny 80/tcp
ufw reload

host# docker run -it --rm -p 80:8000 ubuntu bash
container# apt-get update
container# python3 -m http.server

⁠No space left on device (ERROR)

This issue caused by inodes not being freed after repeated builds. To fix this run :

docker rmi $(docker images -q --filter "dangling=true")

⁠Removing all services

docker service rm $(docker service ls -q)

⁠Removing all containers

docker rm -f $(docker ps -a -q)

⁠Removing All Unused Objects⁠

The docker system prune command will remove all stopped containers, all dangling images, and all unused networks:

docker system prune

You’ll be prompted to continue, use the -f or --force flag to bypass the prompt.

WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all dangling images
        - all build cache
Are you sure you want to continue? [y/N]

If you also want to remove all unused volumes, pass the --volumes flag:

docker system prune --volumes

WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all volumes not used by at least one container
        - all dangling images
        - all build cache
Are you sure you want to continue? [y/N] y

⁠Removing Docker Containers⁠

Docker containers are not automatically removed when you stop them unless you start the container using the --rm flag.

⁠Remove one or more containers⁠

To remove one or more Docker images use the docker container rm command followed by the ID of the containers you want to remove.

You can get a list of all active and inactive containers by passing the -a flag to the docker container ls command:

docker container ls -a

The output should look something like this:

CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS                      PORTS               NAMES
cc3f2ff51cab        centos                  "/bin/bash"              2 months ago        Created                                         competent_nightingale
cd20b396a061        solita/ubuntu-systemd   "/bin/bash -c 'exec …"   2 months ago        Exited (137) 2 months ago                       systemd
fb62432cf3c1        ubuntu                  "/bin/bash"              3 months ago        Exited (130) 3 months ago                       jolly_mirzakhani

Once you know the CONTAINER ID of the containers you want to delete, pass it to the docker container rm command. For example to remove the first two containers listed in the output above run:

docker container rm cc3f2ff51cab cd20b396a061

If you get an error similar to the following, it means that the container is running. You’ll need to stop the container before removing it.

Error response from daemon: You cannot remove a running container fc983ebf4771d42a8bd0029df061cb74dc12cb174530b2036987575b83442b47. Stop the container before attempting removal or force remove.

⁠Remove all stopped containers⁠

Before performing the removal command, you can get a list of all non-running (stopped) containers that will be removed using the following command:

docker container ls -a --filter status=exited --filter status=created

To remove all stopped containers use the docker container prune command:

docker container prune

You’ll be prompted to continue, use the -f or --force flag to bypass the prompt.

WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y

⁠Remove containers using filters⁠

The docker container prune command allows you to remove containers based on condition using the filtering flag --filter.

At the time of the writing of this article, the currently supported filters⁠ are until and label. You can use more than one filter by passing multiple --filter flags.

For example to remove all images that are created more than 12 hours ago, run:

docker container prune --filter "until=12h"

⁠Stop and remove all containers⁠

You can get a list of all Docker containers on your system using the docker container ls -aq command.

To stop all running containers use the docker container stop command followed by a list of all containers IDs.

docker container stop $(docker container ls -aq)

Once all containers are stopped you can remove them using the docker container stop command followed by the containers ID list.

docker container rm $(docker container ls -aq)

⁠Removing Docker Images⁠

⁠Remove one or more images⁠

To remove one or more Docker images use the docker images ls command to find the ID of the images you want to remove.

docker image ls

The output should look something like this:

REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
centos                  latest              75835a67d134        7 days ago          200MB
ubuntu                  latest              2a4cca5ac898        2 months ago        111MB
linuxize/fedora         latest              a45d6dca3361        3 months ago        311MB
java                    8-jre               e44d62cf8862        3 months ago        311MB

Once you’ve located the images you want to remove, pass their IMAGE ID to the docker image rm command. For example to remove the first two images listed in the output above run:

docker image rm 75835a67d134 2a4cca5ac898

If you get an error similar to the following, it means that the image is used by an existing container. To remove the image you will have to remove the container first.

Error response from daemon: conflict: unable to remove repository reference "centos" (must force) - container cd20b396a061 is using its referenced image 75835a67d134

⁠Remove dangling images⁠

Docker provides a docker image prune command that can be used to remove dangled and unused images.

A dangling image is an image that is not tagged and is not used by any container. To remove dangling images type:

docker image prune

You’ll be prompted to continue, use the -f or --force flag to bypass the prompt.

WARNING! This will remove all dangling images.
Are you sure you want to continue? [y/N] y

When removing dangling images, if the images build by you are not tagged, they will be removed too.

⁠Remove all unused images⁠

To remove all images which are not referenced by any existing container, not just the dangling ones, use the prune command with the -a flag:

docker image prune -a

WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y

⁠Remove images using filters⁠

With the docker image prune command, you can also remove images based on a certain condition using the filtering flag --filter.

At the time of the writing of this article, the currently supported filters⁠ are until and label. You can use more than one filter by passing multiple --filter flags.

For example to remove all images that are created more than 12 hours ago, run:

docker image prune -a --filter "until=12h"

⁠Removing Docker Volumes⁠

⁠Remove one or more volumes⁠

To remove one or more Docker volumes use the docker volume ls command to find the ID of the volumes you want to remove.

docker volume ls

The output should look something like this:

DRIVER              VOLUME NAME
local               4e12af8913af888ba67243dec78419bf18adddc3c7a4b2345754b6db64293163
local               terano

Once you’ve found the VOLUME NAME of the volumes you want to remove, pass them to the docker volume rm command. For example to remove the first volume listed in the output above run:

docker volume rm 4e12af8913af888ba67243dec78419bf18adddc3c7a4b2345754b6db64293163

If you get an error similar to the following, it means that the volume is used by an existing container. To remove the volume you will have to remove the container first.

Error response from daemon: remove 4e12af8913af888ba67243dec78419bf18adddc3c7a4b2345754b6db64293163: volume is in use - [c7188935a38a6c3f9f11297f8c98ce9996ef5ddad6e6187be62bad3001a66c8e]

⁠Remove all unused volumes⁠

To remove all unused volumes use the docker image prune command:

docker volume prune

You’ll be prompted to continue, use the -f or --force flag to bypass the prompt.

WARNING! This will remove all local volumes not used by at least one container.
Are you sure you want to continue? [y/N]

⁠Removing Docker Networks⁠

⁠Remove one or more networks⁠

To remove one or more Docker networks use the docker network ls command to find the ID of the networks you want to remove.

docker network ls

The output should look something like this:

NETWORK ID          NAME                DRIVER              SCOPE
107b8ac977e3        bridge              bridge              local
ab998267377d        host                host                local
c520032c3d31        my-bridge-network   bridge              local
9bc81b63f740        none                null                local

Once you’ve located the networks you want to remove, pass their NETWORK ID to the docker network rm command. For example to remove the network with the name my-bridge-network run:

docker network rm c520032c3d31

If you get an error similar to the following, it means that the network is used by an existing container. To remove the network you will have to remove the container first.

Error response from daemon: network my-bridge-network id 6f5293268bb91ad2498b38b0bca970083af87237784017be24ea208d2233c5aa has active endpoints

⁠Remove all unused network⁠

Use the docker network prune command to remove all unused networks.

docker network prune

You’ll be prompted to continue, use the -f or --force flag to bypass the prompt.

WARNING! This will remove all networks not used by at least one container.
Are you sure you want to continue? [y/N] 

⁠Remove networks using filters⁠

With the docker network prune command you can remove networks based on condition using the filtering flag --filter.

At the time of the writing of this article, the currently supported filters⁠ are until and label. You can use more than one filter by passing multiple --filter flags.

For example to remove all networks that are created more than 12 hours ago, run:

docker network prune -a --filter "until=12h"

⁠Truncating Logs

https://stackoverflow.com/questions/42510002/how-to-clear-the-logs-properly-for-a-docker-container⁠

truncate -s 0 /var/lib/docker/containers/*/*-json.log

Proper solution would be to restrict log size in /etc/docker/daemon.json or via the stack file. See for more info⁠

⁠Tasks fail with non-zero exit (137)

This error occurs when a container is out of memory

⁠Adding current $USER to docker group

1. Create the docker group.

$ sudo groupadd docker

1. Add your user to the docker group.

$ sudo usermod -aG docker $USER

⁠Configure Docker to start on boot

Most current Linux distributions (RHEL, CentOS, Fedora, Ubuntu 16.04 and higher) use systemd to manage which services start when the system boots. Ubuntu 14.10 and below use upstart.

systemd

$ sudo systemctl enable docker

To disable this behavior, use disable instead.

$ sudo systemctl disable docker

upstart

Docker is automatically configured to start on boot using upstart. To disable this behavior, use the following command:

$ echo manual | sudo tee /etc/init/docker.override

chkconfig

$ sudo chkconfig docker on

⁠Specify DNS servers for Docker

The default location of the configuration file is /etc/docker/daemon.json. You can change the location of the configuration file using the --config-file daemon flag. The documentation below assumes the configuration file is located at /etc/docker/daemon.json.

Create or edit the Docker daemon configuration file, which defaults to /etc/docker/daemon.json file, which controls the Docker daemon configuration.

 $ sudo vim /etc/docker/daemon.json

Add a dns key with one or more IP addresses as values. If the file has existing contents, you only need to add or edit the dns line.

  {
    "dns": ["8.8.8.8", "8.8.4.4"]
  }

If your internal DNS server cannot resolve public IP addresses, include at least one DNS server which can, so that you can connect to Docker Hub and so that your containers can resolve internet domain names.

Save and close the file.

Restart the Docker daemon.

$ sudo service docker restart

For more information on Docker post-installation setup : https://docs.docker.com/install/linux/linux-postinstall/#specify-dns-servers-for-docker⁠

⁠Truncating Redis appendonly.aof

Since Redis 2.4 the AOF rewrite is automatically triggered by Redis, however the BGREWRITEAOF command can be used to trigger a rewrite at any time.

Sometimes the OS will lock the file and prevent a rewrite. Manually doing this can sometimes free-up needed space

# Connect to the redis container
$ docker exec <redis-container-id> /bin/sh

# Run info command
# aof_rewrite_in_progress tells you if the rewrite is running
# aof_last_bgrewrite_status tells you if it finished successfully
# aof_base_size is the initial size of the AOF file
# aof_current_size is the current size of the AOF file
# More info on this command : https://redis.io/commands/info
$ redis-cli info | grep aof

# https://redis.io/commands/bgrewriteaof
# Instruct Redis to start an Append Only File rewrite process. The rewrite 
# will create a small optimized version of the current Append Only File.
$ redis-cli BGREWRITEAOF

# check the file size
$ ll -lh

For more info on BGREWRITEAOF and taming redis AOF file, check out https://redislabs.com/ebook/part-2-core-concepts/chapter-4-keeping-data-safe-and-ensuring-performance/4-1-persistence-options/4-1-3-rewritingcompacting-append-only-files/⁠

⁠Truncating Redis appendonly.aof (SERVER CANNOT START)

docker run -v ${PWD}:/data redis:5-alpine redis-check-aof --fix /data/.run.sh

# the following script answer yes to all prompts
# bash script is placed in the same directory as the data
yes | docker run -v ${PWD}:/data redis:5-alpine redis-check-aof --fix /data/appendonly.aof

⁠Elasticsearch 7 - Discovery configuration check

This bootstrap check ensures that discovery is not running with the default configuration. It can be satisfied by setting at least one of the following properties:

discovery.seed_hosts
discovery.seed_providers
cluster.initial_master_nodes 

discovery.seed_hosts Provides a list of master-eligible nodes in the cluster. Each value has the format host:port or host, where port defaults to the setting transport.profiles.default.port. Note that IPv6 hosts must be bracketed. The default value is ["127.0.0.1", "[::1]"]. See discovery.seed_hosts edit . This setting was previously known as discovery.zen.ping.unicast.hosts. Its old name is deprecated but continues to work in order to preserve backwards compatibility. Support for the old name will be removed in a future version.

discovery.seed_providers Specifies which types of seed hosts provider to use to obtain the addresses of the seed nodes used to start the discovery process. By default, it is the settings-based seed hosts provider. This setting was previously known as discovery.zen.hosts_provider. Its old name is deprecated but continues to work in order to preserve backwards compatibility. Support for the old name will be removed in a future version.

cluster.initial_master_nodes Sets a list of the node names or transport addresses of the initial set of master-eligible nodes in a brand-new cluster. By default this list is empty, meaning that this node expects to join a cluster that has already been bootstrapped. See cluster.initial_master_nodes edit .

https://www.elastic.co/guide/en/elasticsearch/reference/7.x/modules-discovery-settings.html⁠

https://www.elastic.co/guide/en/elasticsearch/reference/7.0/docker.html⁠

⁠Elasticsearch - Increasing max_shards_per_node

Maximum shards limit defaults to 1,000 shards per data node, and can be dynamically adjusted using the following property:

cluster.max_shards_per_node Controls the number of shards allowed in the cluster per data node.

curl -X PUT \
  {{esURL}}/_cluster/settings \
  -H 'Content-Type: application/json' \
  -H 'cache-control: no-cache' \
  -d '{
  "persistent": {
    "cluster.max_shards_per_node": {{shardCount}}
  }
}
'

⁠Elasticsearch - Increasing search.max_open_scroll_context

To prevent against issues caused by having too many scrolls open, the user is not allowed to open scrolls past a certain limit. By default, the maximum number of open scrolls is 500. This limit can be updated with the search.max_open_scroll_context cluster setting.

You can check how many search contexts are open with the nodes stats API⁠:

curl -X GET "localhost:9200/_nodes/stats/indices/search?pretty"

⁠Updating :

curl -X PUT \
  {{esURL}}/_cluster/settings \
  -H 'Content-Type: application/json' \
  -H 'cache-control: no-cache' \
  -d '{
    "persistent" : {
        "search.max_open_scroll_context": 1024
    },
    "transient": {
        "search.max_open_scroll_context": 1024
    }
}
'

### Resize Terminal in Docker Container

`` docker exec -ti --env COLUMNS=`tput cols` --env LINES=`tput lines` <container_id> /bin/bash ``

### Configure logging drivers

If the logging driver has configurable options, you can set them in the `daemon.json` file as a JSON array with the key `log-opts`. The following example sets two configurable options on the `json-file` logging driver:

​```json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "labels": "production_status",
    "env": "os,customer"
  }
}

If you do not specify a logging driver, the default is json-file. Thus, the default output for commands such as docker inspect <CONTAINER> is JSON.

To find the current default logging driver for the Docker daemon, run docker info and search for Logging Driver. You can use the following command on Linux, macOS, or PowerShell on Windows:

$ docker info --format '{{.LoggingDriver}}'

json-file

⁠Create services using templates⁠

You can use templates for some flags of service create, using the syntax provided by the Go’s text/template⁠ package.

The following flags are supported:

• --hostname
• --mount
• --env

Valid placeholders for the Go template are:

| Placeholder