Your terminal throws "No space left on device," but du says your folders are empty? Welcome to the world of Linux storage ghosts. In this episode of Into the Terminal, Nate and Scott break down why your disk space vanishes, how to uncover invisible open files and exhausted inodes, and how to safely expand your LVM volume and filesystem live on Red Hat Enterprise Linux. Watch the full episode here or test your skills in the interactive labs.
The Essentials
When diagnosing storage problems, it's easy to just run df -h, but that doesn't always tell the whole story. Here are the essential tools to track down vanished storage:
| Command |
What it does |
df -hP |
Displays available disk space in human-readable format. The P ensures the output uses the POSIX format so long mount names don't break lines. |
df -i |
Shows inode usage instead of block usage. Crucial for when the disk is "full" but data blocks are still free! |
du --inodes |
Tracks down which directories are consuming the most inodes. |
| `lsof \ |
grep deleted` |
Advanced Features
Storage Ghost #1: Deleted but Not Gone
If a massive log file fills up your disk, your first instinct is to run rm /var/log/huge.log. However, if a service is actively writing to that file descriptor, the OS removes the file name from the directory tree but cannot free the data blocks until the process releases the file descriptor!
- How to find them: Use lsof | grep deleted to see which processes are hoarding space from deleted files. You either have to restart the service to release the file or manually kill the process.
- The better fix: Don't rm an active log file. Instead, truncate it in place: cat /dev/null > /var/log/huge.log. This zeroes out the data instantly without breaking the active file descriptor.
Storage Ghost #2: Inode Exhaustion
Every file, directory, and symbolic link on a Linux filesystem requires exactly one inode—an index that holds the metadata for the file (permissions, ownership, timestamps, and pointers to the data blocks).
- The ext4 Problem: On ext4 filesystems, the total number of available inodes is set permanently at format time. If you have an application creating millions of tiny files, you can consume 100% of your inodes while your disk is still 90% empty! Once inodes are gone, you cannot write new files, and you'll get a "No space left on device" error.
- The XFS Advantage: Modern XFS filesystems handle this gracefully. They dynamically allocate inodes as needed by converting free data blocks into inode blocks, making it incredibly difficult to run into inode exhaustion compared to ext4.
Safely Expanding a Disk (LVM & XFS)
If your disk truly is full and you've added a new physical disk (e.g., via your hypervisor or storage array), here is the standard critical path to growing your live filesystem using Logical Volume Manager (LVM):
- Partition the new disk:
fdisk /dev/sdb (or sda1 if extending a partition).
- Create the Physical Volume:
pvcreate /dev/sdb1
- Extend the Volume Group: Add the new PV to your existing Volume Group with
vgextend rhel_vbox /dev/sdb1
- Extend the Logical Volume: Allocate the new space to your Logical Volume with
lvextend -l +100%FREE /dev/rhel_vbox/root (or specify a size like -L +1G).
- Grow the Filesystem: Finally, expand the live
xfs filesystem to consume the new space inside the LV with xfs_growfs /dev/rhel_vbox/root. (If using ext4, use resize2fs instead).
Quick Reference Card
```bash
Safely empty an active log file without breaking the service
cat /dev/null > /var/log/messages
Find "deleted" files still consuming disk space
lsof | grep deleted
Check for Inode exhaustion
df -i
Track down which directory has all the inodes
du --inodes -S | sort -nr | head -n 20
Standard LVM Expand Workflow
pvcreate /dev/sdb1
vgextend rhel_vg /dev/sdb1
lvextend -L +5G /dev/rhel_vg/root
xfs_growfs /dev/rhel_vg/root
```
Links & Resources
Into the Terminal is a show dedicated to helping you grow your knowledge of critical administration skills for Red Hat Enterprise Linux. Whether you are new to Linux or new to RHEL, join us for a hands-on look at commands, processes, and tools.