While I was maintaining a monitoring service for an AWS-based infrastructure the service daemon stopped working. After some digging on the host where the service was hosted, I find out that the Linux system was full of inodes which caused the service to be down. Inodes are blocks on the disk which are storing integer numbers for the files on the system. So, potentially if your system is generating a lot of log files or if you have enabled log rotation without getting rid of the old ones you might face the full inodes on your Linux system.
In this tutorial, I will show you how you can detect and free inode usage.
Prerequisites
- Linux bash environment
- sudo privileges
Detect the Inode usage
Step 1. First, make sure that you have 100% inode partition usage.
sudo df -ih
Output:
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 119K 324 119K 1% /dev
tmpfs 121K 485 121K 1% /run
/dev/nvme0n1p1 2.5M 2.5M 1 100% /
tmpfs 121K 1 121K 1% /dev/shm
tmpfs 121K 5 121K 1% /run/lock
tmpfs 121K 16 121K 1% /sys/fs/cgroup
You can see that the /dev/nvme0n1p1 partition has 100% IUse%.
Step 2. Next, you need to go through your system and detect the folders with full inodes. My suggestion is to check the /var/log directories first as the most suspicious path.
for i in /var/logs/*; do echo $i; find $i |wc -l; done
Output:
/var/logs/icinga2/perfdata
1983440
/var/logs/icinga2/tmp
1
Free the Inode usage
Step 1. From the output above, we can clearly see that /var/logs/icinga2/perfdata has a lot of data. After clearing some data the usage of inodes has reduced to 70%.
sudo df -ih
Output:
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 119K 324 119K 1% /dev
tmpfs 121K 483 121K 1% /run
/dev/nvme0n1p1 2.5M 1.7M 773K 70% /
tmpfs 121K 1 121K 1% /dev/shm
tmpfs 121K 5 121K 1% /run/lock
tmpfs 121K 16 121K 1% /sys/fs/cgroup
But from my perspective, it’s really important to find out what is generating so many files to overload the inodes. In my situation, influxdb service was causing the issues with the inodes and generating those files. It might be a different service in your situation, cause it depends on what are you using. Nonetheless, if you don’t find out the root problem, clearing the data will be a temporary solution.
Conclusion
Detection and fixing the issues with inodes is crucial for the system to operate smoothly. So, if you need some further help do not hesitate to leave a comment below and if you find this tutorial useful, follow our official channel on Telegram.
