Investigating a Full Disk on Linux

A Linux server reports that its disk is almost full, but the application folder looks small. Which directories are growing, who can inspect them, and could a running program still hold space occupied? A few commands can answer these questions, provided you understand what each measures. This guide follows a small application called psc-demo through an investigation you can reproduce without changing system files.

The core idea: small tools, connected

The shell is the program that reads your commands and starts other programs. We will use Bash, a common Linux shell. Think of the command line as a workshop: each tool does a particular job, and you can pass one tool’s output to another.

Your filesystem organizes files into a tree of directories, also called folders. A path describes a location in that tree. Commands can inspect locations, read file contents, or examine a process, a running instance of a program.

The useful habit is to connect evidence. Finding a large log explains where space went; finding the process writing it explains why it keeps growing. Neither observation alone tells you whether deleting it is safe.

Follow the evidence step by step

For psc-demo, use this sequence:

  1. Establish your location so commands inspect the intended application.
  2. Check the filesystem’s capacity, then measure the application’s directories.
  3. Inspect permissions if you cannot read something.
  4. Find large files and search logs for relevant messages.
  5. Identify active processes and inspect recent activity.
  6. Choose a remedy that preserves needed data and operational history.

Our example has a configuration file holding application settings, a cache file, and a log. A cache stores data that an application may reuse; whether it can be regenerated depends on the application. A log records events. In production, fixing excessive logging or applying a retention policy—how long records remain—may solve the problem more reliably than repeatedly removing files.

Hands-on: investigate psc-demo

These examples assume Bash, Python 3, and GNU tools, a collection of utilities common on Linux distributions, packaged Linux operating systems. Some options differ in other implementations. Run the commands in the same shell so its variables remain available.

Create a disposable example

mktemp -d creates a uniquely named temporary directory. mkdir -p creates the requested directories, including missing parents. cd changes your current directory.

demo_dir=$(mktemp -d)
mkdir -p "$demo_dir/psc-demo/logs" "$demo_dir/psc-demo/data"
cd "$demo_dir/psc-demo"

printf 'LOG_LEVEL=debug\n' > app.conf
printf 'INFO started\nERROR disk write failed\nINFO retry scheduled\n' > logs/service.log

python3 - <<'PY'
from pathlib import Path
Path("data/cache.bin").write_bytes(b"x" * 4 * 1024 * 1024)
PY

chmod 640 app.conf logs/service.log

The shell’s $(...) runs a command and captures its output in demo_dir. Double quotes preserve paths containing spaces. printf writes text, and > directs it into a file, replacing any existing contents. The Python block writes an ordinary 4 MiB file: a mebibyte, or MiB, is 1,048,576 bytes. chmod sets access permissions: here, owner read/write, group read, and no access for others.

Successful setup produces no output. The log’s error is manufactured sample text, not evidence that your computer failed.

“Where am I, and what is here?”

pwd prints the current directory. ls lists its entries.

pwd
ls
cd logs
ls
cd ..

Illustrative output; your temporary path and listing layout can differ:

/tmp/tmp.<random>/psc-demo
app.conf  data  logs
service.log

An absolute path starts at /, the root of the directory tree. A relative path, such as logs/service.log, starts from your current directory. . means the current directory; .. means its parent. The final command returns to the application’s directory.

Check pwd before commands that change files. A correct command in the wrong directory can still cause damage.

“What is using my disk?”

Start with two different measurements:

df -h .
du -h --max-depth=1 . | sort -h

df reports capacity and usage for the filesystem containing .. du walks directories and totals the space allocated to their files. -h requests readable size units. GNU --max-depth=1 limits displayed directory totals to one level below the starting point; it still measures deeper contents. GNU sort -h orders lines by their human-readable sizes. The pipe symbol, |, passes du’s output to sort.

Representative du output follows; allocation varies by filesystem:

8.0K    ./logs
4.1M    ./data
4.1M    .

The last row includes the others; do not add all three. Directory storage and allocation rounding can make totals exceed file content sizes.

If df’s Used value is much larger than du’s directory total, check whether you measured the whole filesystem, missed unreadable directories, or encountered filesystem features such as retained snapshots, saved views of earlier filesystem state. A deleted file can also remain allocated while a process has it open.

On a server, du -x stays within the starting filesystem instead of crossing into other mounted filesystems, storage attached elsewhere in the directory tree. Large scans still consume disk activity, so scope them carefully.

“Why does this say permission denied?”

ls -l displays details, including permissions, owner, and group:

ls -l app.conf

Expected structure, with account names and timestamp varying:

-rw-r----- 1 <owner> <group> 16 <timestamp> app.conf

The first character - identifies a regular file. The next three groups describe access for its owner, its group, and everyone else. A group is a collection of user accounts used to assign shared access.

Permission On a file On a directory r: read Read contents List entry names w: write Modify contents Create or remove entries, normally with x x: execute Run it as a program Traverse it and access entries, subject to their permissions

chmod changes permissions. In 640, each digit combines read (4), write (2), and execute (1): the owner gets read/write, the group gets read, and everyone else gets nothing.

Inspect a directory itself with ls -ld logs. Reading a file also requires traversal access through its parent directories. Deleting a file primarily depends on the parent directory’s permissions, with additional restrictions possible.

Avoid chmod 777: it grants everyone every permission. sudo runs a command under another account, commonly root, the administrative account. Use it only when elevated access is appropriate; changing ownership or permissions blindly can break an application or expose data.

“Which files are large, and what do their logs say?”

find searches directory entries by attributes such as name, type, and size.

find . -type f -name '*.log'
find . -type f -size +3M

Expected output:

./logs/service.log
./data/cache.bin

-type f selects regular files. The quoted '*.log' pattern matches names ending in .log; quoting prevents Bash from expanding it before find receives it. GNU find uses MiB for M and rounds file sizes up to the requested unit; +3M selects sizes above three of those units.

This measures file length, not allocated disk space. A sparse file contains unwritten regions that read as zeros and may occupy much less storage than its length suggests.

grep searches file contents:

grep -Fn 'ERROR' logs/service.log

Expected output:

2:ERROR disk write failed

-n includes line numbers. -F treats the search text literally. Without -F, grep normally uses a regular expression, a pattern language in which characters such as . have special meanings.

To search every matching log while handling filenames safely:

find logs -type f -name '*.log' -exec grep -HnF -- 'ERROR' {} +

Expected output:

logs/service.log:2:ERROR disk write failed

-exec runs the following command with found paths; {} represents those paths, and + batches them. -H prints filenames. -- ends option parsing, so the search text cannot be mistaken for an option.

“Can I combine these results?”

A pipe connects one command’s standard output, its normal result stream, to another’s standard input, the stream it reads.

grep -F 'ERROR' logs/service.log | wc -l

Expected output:

1

wc -l counts lines. This counts matching log lines, not individual occurrences or unique incidents. grep -Fc 'ERROR' logs/service.log provides the same count directly; the pipeline demonstrates how tools connect.

Errors normally travel through standard error, a separate stream, so a pipe does not automatically capture permission failures.

Commands also return an exit status, a numeric result: zero usually means success. Bash normally uses the final command’s status for a pipeline. In scripts, set -o pipefail makes an earlier failure affect that status. Handle expected cases deliberately: grep returns 1 when it finds no matches and 2 for errors.

“Which process is running?”

Start a harmless demonstration process:

sleep 300 &
demo_pid=$!
ps -p "$demo_pid" -o pid,comm

sleep waits for the given number of seconds. & runs it in the background, leaving the shell available. $! holds the most recent background process’s PID, its process identifier. ps shows a snapshot; here, -p selects that PID and -o selects output columns.

Representative output from ps:

PID COMMAND
  24680 sleep

Bash may also print a background-job notification. Your PID will differ.

Stop only this captured demonstration process:

kill -TERM -- "$demo_pid"

kill sends a signal, a notification to a process. TERM requests termination and allows a program to perform cleanup. Successful delivery normally prints nothing; it does not guarantee that every program exits immediately.

For production applications managed by systemd, software that starts and supervises services, use its systemctl service-management command. Killing a supervised process directly may trigger an automatic restart.

“What happened recently, and where did the space go?”

Read the last three log lines:

tail -n 3 logs/service.log

Expected output:

INFO started
ERROR disk write failed
INFO retry scheduled

tail reads the end of a file. To watch additions, run tail -F logs/service.log and stop viewing with Ctrl+C. GNU -F follows the filename and retries when it disappears, which helps with log rotation, replacing or moving logs to control their size.

On a production system using systemd’s journal, its collected event store, this command reads up to 50 recent entries for a named service:

journalctl -u psc-demo.service --since "1 hour ago" -n 50 --no-pager

-u selects the service’s unit, its systemd management definition. --no-pager prints directly instead of opening an interactive viewer. The demo does not install this service; substitute a real unit name. Results and access permissions depend on the machine.

If a deleted log still occupies space, an optional tool can identify open files with no remaining directory links—names referring to a file:

sudo lsof +L1

lsof lists open files; +L1 selects those whose link count is below one. It may require installation and elevated access. Inspect the process and file before acting. Space becomes reclaimable when the last open reference closes; deletion alone does not force that.

Prefer application-supported rotation and a planned restart or reload, asking the application to reread its settings, when appropriate. Logs may contain credentials or personal information, so redact sensitive values before sharing them.

Tradeoffs: choose the tool for the question

Structured logs store named fields, such as an event’s importance level and time. Queries can use those fields instead of depending solely on message text.

Question Use When to choose an alternative Is this filesystem full? df Use du to locate allocated space within directories Which directory is large? du Use ncdu, an optional interactive browser, for exploration Which paths match? find Use a desktop file manager for small, visual searches Which lines match? grep Use structured-log queries when meaning depends on fields What happened across many servers? Local logs initially Use centralized logging to correlate machines and preserve history

Local tools suit focused diagnosis and quick checks. Avoid repeated full-tree scans on busy disks. Text searches also cannot reliably interpret every log format; an error word may appear in a successful recovery message. Permissions, clocks, and retention gaps can limit what the evidence proves.

Common mistakes and their fixes

· Running commands from the wrong directory. Check pwd and use explicit paths for consequential changes. · Using chmod 777 to fix access. Inspect the owner, group, and parent directories; grant only the needed access. · Leaving search patterns unquoted. Write -name '*.log' so find receives the pattern. · Redirecting output into its own input file. > truncates the destination before reading begins; write to a different file and verify it. · Hiding errors. Keep permission failures visible so incomplete totals are not mistaken for complete measurements. · Forcing shutdown immediately. Request graceful termination before considering KILL, a signal that prevents cleanup.

Quick recap

· Establish your location before acting. · Distinguish filesystem usage, allocated space, and file length. · Read permissions for both files and their parent directories. · Use find for paths, grep for contents, and pipes for composition. · Connect logs to processes before choosing a remedy.

Test yourself

  1. Why can df show more usage than a directory’s du total? Answer: The directory may cover only part of the filesystem, unreadable files may be missed, or deleted files may remain open. Filesystem features can also affect accounting.
  2. What access does chmod 640 app.conf grant? Answer: Owner read/write, group read, and no permissions for others. Reading still requires access through parent directories.
  3. Does grep -F 'ERROR' file.log | wc -l count every error occurrence? Answer: No. It counts matching lines. Several occurrences on one line count once, and a matching line does not necessarily represent a distinct incident.

Where to go next

· Bash scripting: learn variables, quoting, exit statuses, and error handling to make investigations repeatable. · Service operation: study systemd units, graceful shutdown, and log rotation. · Linux storage: explore mounts, sparse files, and filesystem accounting to explain discrepancies confidently.