Skip to main content

Freeing Up Monitor Disk Space

journalctl​

  1. Run the following command to check disk usage:

    $ journalctl --disk-usage

  2. Reduce journald's disk usage by running the following command:

    $ journalctl --vacuum-size=<integer>

docker unused images​

  1. Check for multiple images by running the following command:

    $ docker image ls -a --digests

    If there are multiple images, check for the existence of unused images by comparing the outputs of the following two commands:

    $ docker ps -a --no-trunc --format 'table {{.Image}}\t{{.ID}}\t{{.Status}}\t{{.Names}}'

    and

    $ docker image ls -a --digests

  2. The digest of a docker image is shown after the @ in the IMAGE of the output of the docker ps command.

  3. Determine if any of the images that are not being used by running containers are still needed. Remember that there may exist transient containers that are not currently running but are still in use.

  4. If you determine that there exist images not being used by running containers and that those images are no longer needed, run the following command to check for dangling images:

    $ ls -lah /var/lib/docker/overlay2/

    Any currently-running image in overlay2 has two directory entries that have nearly-identical names. The second name has - init appended. Any directory that is listed once and does not end in - init is unused.

  5. Alternatively, you can check each running container's name, image, and overlay2 paths by running the following command:

    $ docker inspect <container id from docker ps> | jq -r '.[] | .Name,.Image,.GraphDriver'

    Ensure that any image not in current use is indeed not transient and is in fact no longer in use.

  6. After you have confirmed that there exist overlay2 directories that are no longer needed, remove them and reclaim their disk space by using docker image prune. Consult the 'docker image prune' documentation before running docker image prune -a.

Docker Logs​

  1. By default, docker logs do not rotate. They grow throughout the uptime of the container. Check /etc/docker to confirm the presence of the daemon.conf file. If it is present, cat it to determine whether it contains "log-opts". "log-opts" control docker-container log rotation.

  2. Check the size of the container logs by running the following command:

    $ ls -lah /var/lib/docker/containers/*/*.log

    Remember that logs are lost on container exit.

  3. If large logs exist, check to make sure that it is safe to restart the container with the large log file or large log files. Note that the CONTAINER ID returned by docker ps is embedded in the path to the log file, for example /var/lib/docker/containers/<CONTAINER ID>/<CONTAINER ID>-json.log.

  4. If it is safe to restart the container, determine whether you should preserve the log file. Preserve the file if you need to preserve it, and then run the following commands to restart the docker container:

    In one terminal, monitor the following command:

    $ watch ceph -S

    In another terminal, restart the container by running a command of the following form:

    $ ceph orch daemon restart <ceph orch ps name>

General Docker Tips​

To map running containers to the overlay directory that they use, run the following script:

if [ -f rpt ]; then rm rpt; fi; echo "CONTAINER_ID NAME
OVERLAY_DIR" >> rpt; for d in $(docker ps | grep -v
CONTAINER | awk '{print $1}'); do name=$(docker inspect $d |
jq -r '.[] | .Name'); overlay=$(docker inspect $d | jq -r '.
[] | .GraphDriver.Data.MergedDir'); echo $d $name $overlay
>> rpt; done; cat rpt | sed 's|/ceph|ceph|' | sed
's|/merged||' | column -t

To get a brief overview of docker space, run the following command:

# docker system df