Container update fails due to "no space left on device"

Problem

When updating a docker container, e.g. nginx-proxy-manager, you get a "no space left on device".

root@container-nginx:~#
root@container-nginx:~# docker pull jc21/nginx-proxy-manager:latest
latest: Pulling from jc21/nginx-proxy-manager
5b4d6ff92fc4: Extracting [==================================================>]  29.78MB/29.78MB
91199fe81166: Download complete
174e7ed16bc5: Download complete
[...]
35e0d2d62c51: Download complete
206a6810902a: Download complete
write /var/lib/docker/tmp/GetImageBlob3584628884: no space left on device
root@container-nginx:~#
root@container-nginx:~#
root@container-nginx:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/pve-vm--100--disk--1  9.8G  9.1G  141M  99% /
none                              492K  4.0K  488K   1% /dev
udev                               15G     0   15G   0% /dev/tty
tmpfs                              16G     0   16G   0% /dev/shm
tmpfs                             6.3G  252K  6.3G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                           9.8G  9.1G  141M  99% /var/lib/docker/overlay2/52ab08fe774e475ecebdbe1b580aee9936b6379c6ab614e61eae6fa729d5c450/merged
overlay                           9.8G  9.1G  141M  99% /var/lib/docker/overlay2/a1921a337d3130fb4aebf386a0ff6ae2f66b269167e5246b0b0cbe583914aff9/merged
root@container-nginx:~#
root@container-nginx:~#

(Example above shows a nginx-proxy-manager docker container, running in a lxc container, running on a proxmox system)

Solution

1. Check what is using space:
   
   docker system df
   
   
2. Remove dangling images, stopped containers, unused network, (not volumes)
   
   docker system prune 

root@container-nginx:~#
root@container-nginx:~#
root@container-nginx:~# docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          6         2         5.542GB   4.18GB (75%)
Containers      2         2         20.93MB   0B (0%)
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B
root@container-nginx:~#
root@container-nginx:~# docker system prune
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all dangling images
  - all dangling build cache

Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: jc21/nginx-proxy-manager@sha256:669be1a294d4491b842f542d238ea1d4bca8f76fbedd7d99b45b24cd5a8cff99
deleted: sha256:80a22773d011ec7aeaa4242af78e34640b8a020990194e44a5af2de4a5462a81
deleted: sha256:f528f208f3537a7a6e02847e6998a3ca723ae19f4db02f03b01f3650b9864f78
deleted: sha256:9fc2de54173749e63c639fd90eb69ca0edc9d67abbfe7f05f8eac9fa2a1b178e
deleted: sha256:6085eadda2cc2c82596fb251a8ccbb02d1959ffd765a6b0461b19a30511ea7ae
[...]
deleted: sha256:bc9a532f7d9e6eede7bf1211bc78e3ad9f26651a919a469267de6f43f1de9fb2
deleted: sha256:976266c7ef753f49025ef2a06f04057731f75237dd25c434a7cb4ca7c48154c7
deleted: sha256:94f50c2019ef7faaaa113d0f1630ca17c277f376b7d9d2d427ded0da1b5a112c
deleted: sha256:e2550d89489943916f5364c67bbab7a387e233eeb306e421819fc337abba1a90
deleted: sha256:ce5fed2b03d10fd7c44135669ea7a92259effb697be0a01221ef78477c4bee16
deleted: sha256:67ceb0469f70afb39b8eac819d1f2ddd4a170b9ebb031ecb22c8f56b81961451
deleted: sha256:678b5df8088d2c2e48fae0bffe380122f935a5b25e96d3e1b0d7b9341de11979
deleted: sha256:5f1ee22ffb5e68686db3dcb6584eb1c73b5570615b0f14fabb070b96117e351d

Total reclaimed space: 4.18GB
root@container-nginx:~#
root@container-nginx:~#
root@container-nginx:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/pve-vm--100--disk--1  9.8G  2.8G  6.5G  30% /
none                              492K  4.0K  488K   1% /dev
udev                               15G     0   15G   0% /dev/tty
tmpfs                              16G     0   16G   0% /dev/shm
tmpfs                             6.3G  252K  6.3G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                           9.8G  2.8G  6.5G  30% /var/lib/docker/overlay2/52ab08fe774e475ecebdbe1b580aee9936b6379c6ab614e61eae6fa729d5c450/merged
overlay                           9.8G  2.8G  6.5G  30% /var/lib/docker/overlay2/a1921a337d3130fb4aebf386a0ff6ae2f66b269167e5246b0b0cbe583914aff9/merged
root@container-nginx:~#
 

https://docs.docker.com/reference/cli/docker/system/prune/ --> docker system prune --> "Remove all unused containers, networks, images (both dangling and unused), and optionally, volumes."