# Freeing Up Monitor Disk Space

The procedure below was developed by Joel Davidow, <jdavidow@nso.edu>. We would like to acknowledge and thank him for his valued partnership and technical contributions. It is based on the original source material, available [here](/assets/kb/cephadm/ceph-free-up-disk-space.pdf).

### journalctl[​](#journalctl "Direct link to journalctl")

1. Run the following command to check disk usage:

   ```bash
   journalctl --disk-usage

   ```

2. Reduce journald's disk usage by running the following command:

   ```bash
   journalctl --vacuum-size=<integer><K|M|G|T>

   ```

### docker unused images[​](#docker-unused-images "Direct link to docker unused images")

1. Check for multiple images by running the following command:

   ```bash
   docker image ls -a --digests

   ```

   If there are multiple images, check for the existence of unused images by comparing the outputs of the following two commands:

   ```bash
   docker ps -a --no-trunc --format 'table {{.Image}}\t{{.ID}}\t{{.Status}}\t{{.Names}}'

   ```

   and

   ```bash
   docker image ls -a --digests

   ```

2. The digest of a docker image is shown after the `@` in the `IMAGE` of the output of the `docker ps` command.

3. Determine if any of the images that are not being used by running containers are still needed. Remember that there may exist transient containers that are not currently running but are still in use.

4. If you determine that there exist images not being used by running containers and that those images are no longer needed, run the following command to check for dangling images:

   ```bash
   ls -lah /var/lib/docker/overlay2/

   ```

   Any currently-running image in overlay2 has two directory entries that have nearly-identical names. The second name has `- init` appended. Any directory that is listed once and does not end in `- init` is unused.

5. Alternatively, you can check each running container's name, image, and overlay2 paths by running the following command:

   ```bash
   docker inspect <container id from docker ps> | jq -r '.[] | .Name,.Image,.GraphDriver'

   ```

   Ensure that any image not in current use is indeed not transient and is in fact no longer in use.

6. After you have confirmed that there exist overlay2 directories that are no longer needed, remove them and reclaim their disk space by using `docker image prune`. Consult the ['docker image prune' documentation](https://docs.docker.com/reference/cli/docker/image/prune/) before running `docker image prune -a`.

### Docker Logs[​](#docker-logs "Direct link to Docker Logs")

1. By default, docker logs do not rotate. They grow throughout the uptime of the container. Check `/etc/docker` to confirm the presence of the `daemon.conf` file. If it is present, cat it to determine whether it contains "log-opts". "log-opts" control docker-container log rotation.

2. Check the size of the container logs by running the following command:

   ```bash
   ls -lah /var/lib/docker/containers/*/*.log

   ```

   Remember that logs are lost on container exit.

3. If large logs exist, `CONTAINER ID` returned by `docker ps` is embedded in the path to the log file, for example `/var/lib/docker/containers/<CONTAINER ID>/<CONTAINER ID>-json.log`.

4. The container can either be restarted or run a specific truncate command

   Truncate:

   ```text
   truncate -S 0 /var/lib/docker/containers/<CONTAINER ID>/<CONTAINER ID>-json.log

   ```

   Otherwise a restart would be trigger the truncate:

   In another terminal, restart the container by running a command of the following form:

   ```bash
   ceph orch daemon restart <ceph orch ps name>

   ```

## General Docker Tips[​](#general-docker-tips "Direct link to General Docker Tips")

To map running containers to the overlay directory that they use, run the following script:

```bash
if [ -f rpt ]; then rm rpt; fi; echo "CONTAINER_ID NAME OVERLAY_DIR" >> rpt; for d in $(docker ps | grep -v CONTAINER | awk '{print $1}'); do name=$(docker inspect $d | jq -r '.[] | .Name'); overlay=$(docker inspect $d | jq -r '.[] | .GraphDriver.Data.MergedDir'); echo $d $name $overlay >> rpt; done; cat rpt | sed 's|/ceph|ceph|' | sed 's|/merged||' | column -t

```

To get a brief overview of docker space, run the following command:

```bash
docker system df

```