English
A Linux Filesystem Is Full. Where Did the Space Go?
Free gigabytes, free inodes and visible files tell different stories. Find the exhausted resource before removing anything else.
IN THIS ARTICLE+
A service stops writing logs, but removing an old log file does not free the expected space. Before deleting anything else, identify the filesystem that actually rejected the write. The root filesystem, a mounted data volume and a container's writable layer can have very different answers.
Check both kinds of capacity
Run df -h /path/to/data to inspect block usage for the affected path. Then run df -i /path/to/data on a filesystem that reports inode counts. A workload that creates millions of tiny cache files can run out of inodes while many gigabytes remain free. Increasing disk capacity without understanding the file creation pattern may only postpone the next incident.
Use du -x -h --max-depth=1 /path/to/data on GNU systems to compare visible directory usage without crossing filesystem boundaries. Choose a suitable mount root and account for permissions. A scan of only one application directory cannot explain every byte on a shared filesystem.
Explain a large discrepancy
If filesystem usage is high but visible files account for much less, check for deleted files that processes still hold open. On Linux, lsof +L1 can help identify them when run with suitable permissions. Removing a directory entry does not release the blocks until the last reference closes. A supported log reopen or a controlled service restart may release them. Verify the process and the file before acting.
Other explanations include reserved blocks, snapshots and files hidden beneath a mount point. Each requires a different remedy. Restarting an unrelated service will not remove snapshot retention.
Confirm the recovery
Measure the same mount again, perform a small application-level write and confirm normal log rotation. Add monitoring for the resource that actually ran out. A disk-percentage alert alone misses inode exhaustion, and a deletion policy without a working reopen procedure can recreate the original failure.
Further reading
Related guide: Measure Storage Latency Before Buying Faster Disks.
No comments yet. Share the first perspective.