File aperti e cancellati
Why is space not being freed from disk after deleting a file in Red Hat Enterprise Linux?
Solution Verified - Updated February 13 2022 at 3:06 AM - English
Environment
Red Hat Enterprise Linux (RHEL) Issue
Why is space not being freed from disk after deleting a file in Red Hat Enterprise Linux? When deleting a large file or files, the file is deleted successfully but the size of the filesystem does not reflect the change. I've deleted some files but the amount of free space on the filesystem has not changed.
The OS was holding several very large log files open with some as large as ~30G. The file was previously deleted, but only stopping and restarting the jvm/java process released the disk space. The lsof command shows the following output before restarting the java process Raw
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME : java 49097 awdmw 77w REG 253,6 33955068440 1283397 /opt/jboss/jboss-eap-5/jboss-as/server/all/log/server.log (deleted)
When you perform a df, the storage shows 90+% utilized, however, there is not really that much written to that space.
Resolution
Graceful shutdown of relevant process
First, obtain a list of deleted files which are still held open by applications: Raw
$ lsof | egrep "deleted|COMMAND" COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME ora 25575 8194 oracle oracle 33 REG 65,65 4294983680 31014933 /oradata/DATAPRE/file.dbf (deleted)
Note: check either the filesystem path within NAME field or the device number under DEVICE to match the filesystem of interest.
The lsof output shows the process with pid 25575 has kept file /oradata/DATAPRE/file.dbf open with file descriptor (fd) number 33.
After a file has been identified, free the file used space by shutting down the affected process. If a graceful shutdown does not work, then issue the kill command to forcefully stop it by referencing the PID.
Truncate File Size
Alternatively, it is possible to force the system to de-allocate the space consumed by an in-use file by forcing the system to truncate the file via the proc file system. This is an advanced technique and should only be carried out when the administrator is certain that this will cause no adverse effects to running processes. Applications may not be designed to deal elegantly with this situation and may produce inconsistent or undefined behavior when files that are in use are abruptly truncated in this manner. Raw
$ echo > /proc/pid/fd/fd_number
For example, from the lsof output above: Raw
$ file /proc/25575/fd/33 /proc/25575/fd/33: broken symbolic link to `/oradata/DATAPRE/file.dbf (deleted)' $ echo > /proc/25575/fd/33
The same reason will cause different disk usage from du command and df command, please refer to Why does df show bigger disk usage than du?
To identify the used file size (in blocks), use the command below: Raw
- lsof -Fn -Fs |grep -B1 -i deleted | grep ^s | cut -c 2- | awk '{s+=$1} END {print s}'
Root Cause
On Linux or Unix systems, deleting a file via rm or through a file manager application will unlink the file from the file system's directory structure; however, if the file is still open (in use by a running process) it will still be accessible to this process and will continue to occupy space on disk. Therefore such processes may need to be restarted before that file's space will be cleared up on the filesystem. Diagnostic Steps
Log Reaper will allow you to visualize and quickly narrow down the lsof data to exactly the subset you want to see
Product(s) Red Hat Enterprise Linux
Component lsof
Category Learn more
Tags file_systems kernel rhel
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form. 7 Comments