Debugging Linux Program with strace
When a program on Linux is misbehaving, such as crashing, hanging, or failing silently, you may reach for gdb, logs, or guesswork. Alternatively, there's another powerful, arguably underused tool: strace.
strace intercepts and logs system calls made by a process, letting you peek into how it interacts with the OS. It’s simple, fast, and requires no code changes.
Outline
What Is strace?
strace is a program on Linux that lets you determine what a program is doing without a debugger or source code.
Specifically, strace shows you what a process is asking the Linux kernel to do; for example, this includes file operations (such as open, read, and write), network I/O (including connect, sendto, and recvfrom), process management (like fork, execve, and waitpid), and so on.
Why does it matter?
strace is worth exploring because it has proven invaluable for debugging programs. Being able to see how a program interacts with the kernel can give you a basic understanding of what is going on. Below are some of the representative use cases of strace:
- Find the config file of the program
- Find files that program depends on, such as dynamically linked libraries, root certificates, data source etc
- Determine what happens during a program hang
- Coarse profiling
- Hidden permission error
- Command arguments passed to other programs
I hope this has sparked your interest in exploring this tool further. Up next, we'll dive into installation, basic usage and common patterns.
Installation
sudo apt install strace # Debian/Ubuntu
sudo yum install strace # CentOS/RHEL
Basic Usage
# Trace a new program
strace ./your_program
# Attach to a running process
strace -p <pid>
# Redirect output to a file
strace -o trace.log ./your_program
Common Debugging Patterns
1. Find the config file a program tries to load
strace -e trace=openat ./your_program
Purpose: Traces file opening syscalls using openat(), helping identify missing or mislocated configuration files.
What to look for: Look for ENOENT errors indicating missing files.
Example output:
openat(AT_FDCWD, "/etc/your_program/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)
Explanation: The program attempted to open a config file that does not exist.
2. Find dynamically loaded libraries, root certificates, or data files
strace -e trace=file ./your_program
Purpose: Captures file-related syscalls (open, stat, etc.), revealing all resources accessed.
What to look for: Locate paths to important libraries, certs, or data files.
Example output:
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libssl.so.1.1", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = 4
Explanation: These lines show successful loading of required shared libraries and certificate files.
3. Diagnose why a program hangs (e.g., blocking on I/O)
Purpose: Attaches to a running process and shows syscall durations and timestamps.
What to look for: Find syscalls with unusually long durations.
Example output:
12:00:01.123456 read(4, ..., 4096) <120.000123>
Explanation: A read() call took 120 seconds—likely waiting for input or a blocked pipe.
4. Detect hidden permission errors
strace -e trace=openat ./your_program
Purpose: Reveals hidden EACCES errors that applications may suppress.
What to look for: Failed attempts with EACCES indicating permission denied.
Example output:
openat(AT_FDCWD, "/var/log/secure.log", O_RDONLY) = -1 EACCES (Permission denied)
Explanation: The program tried to read a log file it doesn't have permission to access.
5. Capturing inter-process communication or exec chains
strace -f -e execve ./script.sh
Purpose: Traces all execve calls and follows child processes using -f.
What to look for: Command-line arguments, incorrect binary paths.
Example output:
execve("/usr/bin/python3", ["python3", "-c", "print('Hello')"], ...) = 0
Explanation: A Python subprocess was launched with an inline script.
6. Profiling high-level syscall activity
Purpose: Displays syscall usage summary with counts and total time.
What to look for: Time-consuming or frequent syscalls that affect performance.
Example output:
% time seconds usecs/call calls syscall
------ ------- ----------- ----- --------
40.12 0.120000 120 100 read
30.25 0.090000 90 50 write
Explanation: Most time was spent on read() and write() syscalls.
7. Uncovering undocumented file access
strace -e trace=file ./your_program
Purpose: Detects unexpected or undocumented files accessed during execution.
What to look for: Configs, caches, or plugins accessed without explicit documentation.
Example output:
openat(AT_FDCWD, "/usr/share/your_program/theme.conf", O_RDONLY) = 3
Explanation: The program accessed a theming configuration file not mentioned in docs.
8. Investigating network behavior
strace -e trace=network ./your_program
Purpose: Monitors networking syscalls like connect(), sendto(), and recvfrom().
What to look for: Refused connections, DNS resolution failures, unreachable hosts.
Example output:
connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("93.184.216.34")}, 16) = -1 ECONNREFUSED (Connection refused)
Explanation: The program attempted to connect to a server but the connection was refused.
9. Monitoring signals or crashes
strace -e trace=signal ./your_program
Purpose: Shows received signals, useful for detecting terminations and faults.
What to look for: Signals like SIGSEGV, SIGBUS, SIGKILL indicating a crash or forced exit.
Example output:
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x0} ---
Explanation: A segmentation fault occurred due to a null pointer access.
10. Audit child process creation (fork, clone, exec)
strace -f -e trace=clone,execve ./parent_program
Purpose: Traces child creation and execution using clone() and execve().
What to look for: Verify subprocess execution flow and command arguments.
Example output:
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, ...) = 1234
execve("/bin/ls", ["ls", "-la"], ...) = 0
Explanation: The parent process spawned a child which executed the ls -la command.
Tip: Clean Output
strace can be noisy. Use filters:
strace -e trace=read,write,open,close ./prog
Or output to file and search:
strace -o log.txt ./prog
grep ENOENT log.txt
Sample Output: Running strace on curl
To illustrate how strace works in practice, here’s a truncated real-world output of running strace on the curl command:
strace curl https://example.com
Essential system calls output:
execve("/usr/bin/curl", ["curl", "https://example.com"], ...) = 0
...
openat(AT_FDCWD, "/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = 3
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("93.184.216.34")}, 16) = 0
write(4, "GET / HTTP/1.1\r\nHost: example.com\r\n...", 96) = 96
read(4, "HTTP/1.1 200 OK\r\n...", 4096) = 320
...
close(4) = 0
What this tells us:
execve: The initial execution of the curl binary.openat: Loads root certificates to verify the HTTPS connection.socket and connect: Establishes a TCP connection to the server.write/read: Sends HTTP request and reads response.close: Cleans up the socket connection.
This example demonstrates how you can use strace to observe low-level behavior of everyday tools and debug or analyze them more deeply.
Recap
| Use Case |
Key Flag |
What to Look For |
Common Errors |
| Missing config |
-e openat |
ENOENT on config paths |
ENOENT |
| Missing libraries |
-e file |
failed open for .so, certs |
ENOENT, EACCES |
| Hang detection |
-tt -T |
long syscall durations |
– |
| Silent failure |
-e openat |
EACCES, hidden failures |
EACCES |
| Debug scripts |
-f -e execve |
wrong args, missing paths |
– |
| Profile behavior |
-c |
dominant syscall timings |
– |
| Trace file access |
-e file |
undocumented config/data |
– |
| Network issues |
-e network |
failed connect, DNS |
ECONNREFUSED |
| Crashes/signals |
-e signal |
SIGSEGV, SIGBUS |
– |
While strace gives insight into system calls, combining it with other tools offers a fuller picture:
lsof: List Open Files
See which files, sockets, and devices a process is using. Great for checking if the process is stuck on a file or network call.
ps + top + htop: Process Status
ps aux shows current process states.top and htop give a live view into CPU, memory, and I/O usage.htop allows filtering and killing processes interactively.
gdb: Interactive Debugging
Use gdb if you need to inspect memory, variables, and stack traces.
gdb ./app
(gdb) run
(gdb) bt # backtrace on crash
Find hotspots and performance bottlenecks.
perf top
perf record ./your_program
perf report
dmesg and journalctl: Kernel Logs
Check for kernel-level errors:
dmesg | tail
journalctl -xe
These can reveal permission denials, segmentation faults, or system-wide resource issues.
Summary
| Problem |
What to Look For With strace |
| Crash/Error |
Last syscalls, missing files |
| Hangs/Timeout |
Long gaps between calls |
| Wrong paths |
open, access, stat on wrong files |
| Permission issue |
EACCES in open or access calls |
strace is an indispensable tool for Linux developers. It’s not a full debugger, but often it’s the only tool you need. Combine it with tools like lsof, gdb, htop, and perf for deeper diagnosis.
Next time your program fails silently, try this:
References
