Debugging Linux Program with strace

When a program on Linux is misbehaving, such as crashing, hanging, or failing silently, you may reach for gdb, logs, or guesswork.  Alternatively, there's another powerful, arguably underused tool: strace.

strace intercepts and logs system calls made by a process, letting you peek into how it interacts with the OS. It’s simple, fast, and requires no code changes.

Outline

• What Is strace?
• Why does it matter?
• Installation
• Basic Usage
• Common Debugging Patterns
• Complementary Debugging Tools
• Summary
• References

What Is strace?

Specifically, strace shows you what a process is asking the Linux kernel to do; for example, this includes file operations (such as open, read, and write), network I/O (including connect, sendto, and recvfrom), process management (like fork, execve, and waitpid), and so on.

Why does it matter?

strace is worth exploring because it has proven invaluable for debugging programs. Being able to see how a program interacts with the kernel can give you a basic understanding of what is going on. Below are some of the representative use cases of strace:

• Find the config file of the program
• Find files that program depends on, such as dynamically linked libraries, root certificates, data source etc
• Determine what happens during a program hang
• Coarse profiling
• Hidden permission error
• Command arguments passed to other programs

I hope this has sparked your interest in exploring this tool further. Up next, we'll dive into installation, basic usage and common patterns.

Installation

sudo apt install strace     # Debian/Ubuntu
sudo yum install strace     # CentOS/RHEL

Basic Usage

# Trace a new program
strace ./your_program

# Attach to a running process
strace -p <pid>

# Redirect output to a file
strace -o trace.log ./your_program

Common Debugging Patterns

1. Find the config file a program tries to load

strace -e trace=openat ./your_program

Purpose: Traces file opening syscalls using openat(), helping identify missing or mislocated configuration files.

What to look for: Look for ENOENT errors indicating missing files.

Example output:

openat(AT_FDCWD, "/etc/your_program/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)

Explanation: The program attempted to open a config file that does not exist.

2. Find dynamically loaded libraries, root certificates, or data files

strace -e trace=file ./your_program

Purpose: Captures file-related syscalls (open, stat, etc.), revealing all resources accessed.

What to look for: Locate paths to important libraries, certs, or data files.

Example output:

openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libssl.so.1.1", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = 4

Explanation: These lines show successful loading of required shared libraries and certificate files.

3. Diagnose why a program hangs (e.g., blocking on I/O)

strace -tt -T -p <PID>

Purpose: Attaches to a running process and shows syscall durations and timestamps.

What to look for: Find syscalls with unusually long durations.

Example output:

12:00:01.123456 read(4, ..., 4096) <120.000123>

Explanation: A read() call took 120 seconds—likely waiting for input or a blocked pipe.

4. Detect hidden permission errors

strace -e trace=openat ./your_program

Purpose: Reveals hidden EACCES errors that applications may suppress.

What to look for: Failed attempts with EACCES indicating permission denied.

Example output:

openat(AT_FDCWD, "/var/log/secure.log", O_RDONLY) = -1 EACCES (Permission denied)

Explanation: The program tried to read a log file it doesn't have permission to access.

5. Capturing inter-process communication or exec chains

strace -f -e execve ./script.sh

Purpose: Traces all execve calls and follows child processes using -f.

What to look for: Command-line arguments, incorrect binary paths.

Example output:

execve("/usr/bin/python3", ["python3", "-c", "print('Hello')"], ...) = 0

Explanation: A Python subprocess was launched with an inline script.

6. Profiling high-level syscall activity

strace -c ./your_program

Purpose: Displays syscall usage summary with counts and total time.

What to look for: Time-consuming or frequent syscalls that affect performance.

Example output:

% time     seconds  usecs/call     calls    syscall
------     -------  -----------    -----    --------
 40.12    0.120000         120       100    read
 30.25    0.090000          90        50    write

Explanation: Most time was spent on read() and write() syscalls.

7. Uncovering undocumented file access

strace -e trace=file ./your_program

Purpose: Detects unexpected or undocumented files accessed during execution.

What to look for: Configs, caches, or plugins accessed without explicit documentation.

Example output:

openat(AT_FDCWD, "/usr/share/your_program/theme.conf", O_RDONLY) = 3

Explanation: The program accessed a theming configuration file not mentioned in docs.

8. Investigating network behavior

strace -e trace=network ./your_program

Purpose: Monitors networking syscalls like connect(), sendto(), and recvfrom().

What to look for: Refused connections, DNS resolution failures, unreachable hosts.

Example output:

connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("93.184.216.34")}, 16) = -1 ECONNREFUSED (Connection refused)

Explanation: The program attempted to connect to a server but the connection was refused.

9. Monitoring signals or crashes

strace -e trace=signal ./your_program

Purpose: Shows received signals, useful for detecting terminations and faults.

What to look for: Signals like SIGSEGV, SIGBUS, SIGKILL indicating a crash or forced exit.

Example output:

--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x0} ---

Explanation: A segmentation fault occurred due to a null pointer access.

10. Audit child process creation (fork, clone, exec)

strace -f -e trace=clone,execve ./parent_program

Purpose: Traces child creation and execution using clone() and execve().

What to look for: Verify subprocess execution flow and command arguments.

Example output:

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, ...) = 1234
execve("/bin/ls", ["ls", "-la"], ...) = 0

Explanation: The parent process spawned a child which executed the ls -la command.

Tip: Clean Output

strace can be noisy. Use filters:

strace -e trace=read,write,open,close ./prog

Or output to file and search:

strace -o log.txt ./prog
grep ENOENT log.txt

Sample Output: Running strace on curl

To illustrate how strace works in practice, here’s a truncated real-world output of running strace on the curl command:

strace curl https://example.com

Essential system calls output:

execve("/usr/bin/curl", ["curl", "https://example.com"], ...) = 0
...
openat(AT_FDCWD, "/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = 3
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("93.184.216.34")}, 16) = 0
write(4, "GET / HTTP/1.1\r\nHost: example.com\r\n...", 96) = 96
read(4, "HTTP/1.1 200 OK\r\n...", 4096) = 320
...
close(4) = 0

What this tells us:

• execve: The initial execution of the curl binary.
• openat: Loads root certificates to verify the HTTPS connection.
• socket and connect: Establishes a TCP connection to the server.
• write/read: Sends HTTP request and reads response.
• close: Cleans up the socket connection.

This example demonstrates how you can use strace to observe low-level behavior of everyday tools and debug or analyze them more deeply.

Recap

Complementary Debugging Tools

While strace gives insight into system calls, combining it with other tools offers a fuller picture:

lsof: List Open Files

lsof -p <pid>

See which files, sockets, and devices a process is using. Great for checking if the process is stuck on a file or network call.

ps + top + htop: Process Status

• ps aux shows current process states.
• top and htop give a live view into CPU, memory, and I/O usage.
• htop allows filtering and killing processes interactively.

gdb: Interactive Debugging

Use gdb if you need to inspect memory, variables, and stack traces.

gdb ./app
(gdb) run
(gdb) bt  # backtrace on crash

perf: Performance Analysis

Find hotspots and performance bottlenecks.

perf top
perf record ./your_program
perf report

dmesg and journalctl: Kernel Logs

Check for kernel-level errors:

dmesg | tail
journalctl -xe

These can reveal permission denials, segmentation faults, or system-wide resource issues.

Summary

strace is an indispensable tool for Linux developers. It’s not a full debugger, but often it’s the only tool you need. Combine it with tools like lsof, gdb, htop, and perf for deeper diagnosis.

Next time your program fails silently, try this:

strace ./your_program

References

• strace man page - Accessed 2025-06-01
• strace source code - Accessed 2025-06-01
• Julia Evans – What problems do people solve with strace? (Twitter Post) - Accessed 2025-06-01