linux,

How to detect and free Inode Usage on Linux

Oct 09, 2021 · 2 mins read · Post a comment
How to detect and free Inode Usage on Linux

While I was maintaining a monitoring service for an AWS-based infrastructure the service daemon stopped working. After some digging on the host where the service was hosted, I find out that the Linux system was full of inodes which caused the service to be down. Inodes are blocks on the disk which are storing integer numbers for the files on the system. So, potentially if your system is generating a lot of log files or if you have enabled log rotation without getting rid of the old ones you might face the full inodes on your Linux system. In this tutorial, I will show you how you can detect and free inode usage.

Prerequisites

  • Linux based system

Detect the Inode usage

Step 1. First, make sure that you have 100% inode partition usage with:

sudo df -ih

Output:

Filesystem                                           Inodes IUsed IFree IUse% Mounted on
udev                                                   119K   324  119K    1% /dev
tmpfs                                                  121K   485  121K    1% /run
/dev/nvme0n1p1                                         2.5M  2.5M     1  100% /
tmpfs                                                  121K     1  121K    1% /dev/shm
tmpfs                                                  121K     5  121K    1% /run/lock
tmpfs                                                  121K    16  121K    1% /sys/fs/cgroup

You can see that the /dev/nvme0n1p1 partition has 100% IUse%.

Step 2. Next, you need to go through your system and detect the folders with full inodes. My suggestion is to check the /var/log directories first as the most suspicious path:

for i in /var/logs/*; do echo $i; find $i |wc -l; done

Output:

/var/logs/icinga2/perfdata
1983440
/var/logs/icinga2/tmp
1

Free the Inode usage

Step 1. From the output above, we can clearly see that /var/logs/icinga2/perfdata has a lot of data. After clearing some of the data the usage of inodes has reduced to 70%:

sudo df -ih

Output:

Filesystem                                           Inodes IUsed IFree IUse% Mounted on
udev                                                   119K   324  119K    1% /dev
tmpfs                                                  121K   483  121K    1% /run
/dev/nvme0n1p1                                         2.5M  1.7M  773K   70% /
tmpfs                                                  121K     1  121K    1% /dev/shm
tmpfs                                                  121K     5  121K    1% /run/lock
tmpfs                                                  121K    16  121K    1% /sys/fs/cgroup

But from my perspective, it’s really important to find out what is generating so many files to overload the inodes. In my situation, influxdb service was causing the issues with the inodes and generating those files. It might be a different service in your situation, cause it depends on what are you using. Nonetheless, if you don’t find out the root problem, clearing the data will be a temporary solution.

Conclusion

Detection and fixing the issues with inodes is crucial for the system to operate smoothly. So if you need some further help do not hesitate to leave a comment below and if you find this tutorial useful, follow our official channel on telegram.