$manlookup

strace

(1)

trace system calls and signals

User commands157 optionsstrace 6.13+ds-1
strace [-ACdffhikqqrtttTvVwxxyyYzZ] [-a column] [-b execve] [-e expr]... [-I n] [-o file] [-O overhead] [-p pid]... [-P path]... [-s strsize] [-S sortby] [-U columns] [-X format] [--seccomp-bpf] [--stack-trace-frame-limit=limit] [--syscall-limit=limit] [--tips[=format]] { -p pid | [-DDD] [-E var[=val]]... [-u username] command [args] } strace -c [-dfwzZ] [-b execve] [-e expr]... [-I n] [-O overhead] [-p pid]... [-P path]... [-S sortby] [-U columns] [--seccomp-bpf] [--syscall-limit=limit] [--tips[=format]] { -p pid | [-DDD] [-E var[=val]]... [-u username] command [args] } strace --tips[=format]

Options

157
-eexpr

A qualifying expression which modifies which events to trace or how to trace them. The format of the expression is:

-Evar

--envvar

Run command with var=val in its list of environment variables.

-Evar

--envvar

Remove var from the inherited list of environment variables before passing it on to the command.

-ppid

--attachpid

Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to many processes in addition to command (which is optional if at least one -p option is given). Multiple process IDs, separated by either comma (“,”), space (“ ”), tab, or newline character, can be provided as an argument to a single -p option, so, for example, -p "$(pidof PROG)" and -p "$(pgrep PROG)"

-uusername

--userusername

Run command with the user ID, group ID, and supplementary groups of username. This option is only useful when running as root and enables the correct execution of setuid and/or setgid binaries. Unless this option is used setuid and setgid programs are executed without effective privileges.

-uUID:GID

--userUID:GID

Alternative syntax where the program is started with exactly the given user and group IDs, and an empty list of supplementary groups. In this case, user and group name lookups are not performed.

--argv0name

Set argv[0] of the command being executed to name. Useful for tracing multi-call executables which interpret argv[0], such as busybox or kmod.

-bsyscall

--detach-onsyscall

If specified syscall is reached, detach from traced process. Currently, only execve(2) syscall is supported. This option is useful if you want to trace multi-threaded process and therefore require -f, but don't want to trace its (potentially very complex) children.

-D

--daemonize

--daemonize

Run tracer process as a grandchild, not as the parent of the tracee. This reduces the visible effect of strace by keeping the tracee a direct child of the calling process.

-DD

--daemonize

--daemonize

Run tracer process as tracee's grandchild in a separate process group. In addition to reduction of the visible effect of strace, it also avoids killing of strace with kill(2) issued to the whole process group.

-DDD

--daemonize

Run tracer process as tracee's grandchild in a separate session ("true daemonisation"). In addition to reduction of the visible effect of strace, it also avoids killing of strace upon session termination.

-f

--follow-forks

Trace child processes as they are created by currently traced processes as a result of the fork(2), vfork(2) and clone(2) system calls. Note that -p PID -f will attach all threads of process PID if it is multi-threaded, not only thread with thread_id = PID.

--output-separately

If the --output=filename option is in effect, each processes trace is written to filename.pid where pid is the numeric process id of each process.

-ff

--follow-forks--output-separately

Combine the effects of --follow-forks and --output-separately options. This is incompatible with -c, since no per-process counts are kept.

-Iinterruptible

--interruptibleinterruptible

When strace can be interrupted by signals (such as pressing CTRL-C).

--syscall-limitlimit

Detach all tracees when limit number of syscalls have been captured. Syscalls filtered out via --trace, --trace-path or --status options are not considered when keeping track of the number of syscalls that are captured.

--kill-on-exit

Apply PTRACE_O_EXITKILL ptrace option to all tracee processes (which sends a SIGKILL signal to the tracee if the tracer exits) and do not detach them on cleanup so they will not be left running after the tracer exit. --kill-on-exit is not compatible with -p/--attach options.

-esyscall_set

-esyscall_set

--tracesyscall_set

Trace only the specified set of system calls. syscall_set is defined as [!]value[,value], and value can be one of the following:

-eset

-eset

-eset

-eset

--trace-fdsset

Trace only the syscalls that operate on the specified subset of (non-negative) file descriptors. Note that usage of this option also filters out all the syscalls that do not operate on file descriptors at all. Applies in (inclusive) disjunction with the --trace-path option.

-eset

-eset

-eset

--signalset

Trace only the specified subset of signals. The default is signal=all. For example, signal=!SIGIO (or signal=!io) causes SIGIO signals not to be traced.

-eset

--statusset

Print only system calls with the specified return status. The default is status=all. When using the status qualifier, because strace waits for system calls to return before deciding whether they should be printed or not, the traditional order of events may not be preserved anymore. If two system calls are executed by concurrent threads, strace will first print both the entry and exit of the first system call to exit, regardless of their respective entry time. The entry and exit of the second system call to exit will be printed afterwards. Here is an example when select(2) is called, but a diff

-Ppath

--trace-pathpath

Trace only system calls accessing path. Multiple -P options can be used to specify several paths. Applies in (inclusive) disjunction with the --trace-fds option.

-z

--successful-only

Print only syscalls that returned without an error code.

-Z

--failed-only

Print only syscalls that returned with an error code.

-acolumn

--columnscolumn

Align return values in a specific column (default column 40).

-esyscall_set

-esyscall_set

--abbrevsyscall_set

Abbreviate the output from printing each member of large structures. The syntax of the syscall_set specification is the same as in the -e trace option. The default is abbrev=all. The -v option has the effect of abbrev=none.

-esyscall_set

-esyscall_set

--verbosesyscall_set

Dereference structures for the specified set of system calls. The syntax of the syscall_set specification is the same as in the -e trace option. The default is verbose=all.

-esyscall_set

-esyscall_set

--rawsyscall_set

Print raw, undecoded arguments for the specified set of system calls. The syntax of the syscall_set specification is the same as in the -e trace option. This option has the effect of causing all arguments to be printed in hexadecimal. This is mostly useful if you don't trust the decoding or you need to know the actual numeric value of an argument. See also -X raw option.

-eset

-eset

-eset

--readset

Perform a full hexadecimal and ASCII dump of all the data read from file descriptors listed in the specified set. For example, to see all input activity on file descriptors 3 and 5 use -e read=3,5. Note that this is independent from the normal tracing of the read(2) system call which is controlled by the option -e trace=read.

-eset

-eset

-eset

--writeset

Perform a full hexadecimal and ASCII dump of all the data written to file descriptors listed in the specified set. For example, to see all output activity on file descriptors 3 and 5 use -e write=3,5. Note that this is independent from the normal tracing of the write(2) system call which is controlled by the option -e trace=write.

-eset

-eset

-eset

-eset

--quietset

--silentset

--silenceset

Suppress various information messages. The default is quiet=none. set can include the following elements:

-eset

--decode-fdsset

Decode various information associated with file descriptors. The default is decode-fds=none. set can include the following elements:

-eset

--decode-pidsset

Decode various information associated with process IDs (and also thread IDs, process group IDs, and session IDs). The default is decode-pids=none. set can include the following elements:

-e

--kvm

Print the exit reason of kvm vcpu. Requires Linux kernel version 4.16.0 or higher.

-i

--instruction-pointer

Print the instruction pointer at the time of the system call.

-n

--syscall-number

Print the syscall number.

-k

--stack-trace

Print the execution stack trace of the traced processes after each system call.

--stack-trace-frame-limitlimit

Print no more than this amount of stack trace frames when backtracing a system call (the default is 256). Use this option with the --stack-trace (or -k) option.

-ofilename

--outputfilename

Write the trace output to the file filename rather than to stderr. filename.pid form is used if -ff option is supplied. If the argument begins with '|' or '!', the rest of the argument is treated as a command and all output is piped to it. This is convenient for piping the debugging output to a program without affecting the redirections of executed programs. The latter is not compatible with -ff option currently.

-A

--output-append-mode

Open the file provided in the -o option in append mode.

-q

--quiet

--quiet

Suppress messages about attaching, detaching, and personality changes. This happens automatically when output is redirected to a file and the command is run directly instead of attaching.

-qq

--quiet

Suppress messages attaching, detaching, personality changes, and about process exit status.

-qqq

--quiet

Suppress all suppressible messages (please refer to the -e quiet option description for the full list of suppressible messages).

-r

--relative-timestampsprecision

Print a relative timestamp upon entry to each system call. This records the time difference between the beginning of successive system calls. precision can be one of s (for seconds), ms (milliseconds), us (microseconds), or ns (nanoseconds), and allows setting the precision of time value being printed. Default is us (microseconds). Note that since -r option uses the monotonic clock time for measuring time difference and not the wall clock time, its measurements can differ from the difference in time reported by the -t option.

-sstrsize

--string-limitstrsize

Specify the maximum string size to print (the default is 32). Note that filenames are not considered strings and are always printed in full.

--absolute-timestampsformat

--timestampsformat

Prefix each line of the trace with the wall clock time in the specified format with the specified precision. format can be one of the following:

-t

--absolute-timestamps

Prefix each line of the trace with the wall clock time.

-tt

--absolute-timestamps

If given twice, the time printed will include the microseconds.

-ttt

--absolute-timestamps

If given thrice, the time printed will include the microseconds and the leading portion will be printed as the number of seconds since the epoch.

-T

--syscall-timesprecision

Show the time spent in system calls. This records the time difference between the beginning and the end of each system call. precision can be one of s (for seconds), ms (milliseconds), us (microseconds), or ns (nanoseconds), and allows setting the precision of time value being printed. Default is us (microseconds).

-v

--no-abbrev

Print unabbreviated versions of environment, stat, termios, etc. calls. These structures are very common in calls and so the default behavior displays a reasonable subset of structure members. Use this option to get all of the gory details.

--strings-in-hexoption

Control usage of escape sequences with hexadecimal numbers in the printed strings. Normally (when no --strings-in-hex or -x option is supplied), escape sequences are used to print non-printable and non-ASCII characters (that is, characters with a character code less than 32 or greater than 127), or to disambiguate the output (so, for quotes and other characters that encase the printed string, for example, angle brackets, in case of file descriptor path output); for the former use case, unless it is a white space character that has a symbolic escape sequence defined in the C standard (that is,

-x

--strings-in-hex

Print all non-ASCII strings in hexadecimal string format.

-xx

--strings-in-hex

Print all strings in hexadecimal string format.

-Xformat

--const-print-styleformat

Set the format for printing of named constants and flags. Supported format values are:

-y

--decode-fds

--decode-fds

Print paths associated with file descriptor arguments and with the AT_FDCWD constant.

-yy

--decode-fds

Print all available information associated with file descriptors: protocol-specific information associated with socket file descriptors, block/character device number associated with device file descriptors, and PIDs associated with pidfd file descriptors.

--pidns-translation

--decode-pids

If strace and tracee are in different PID namespaces, print PIDs in strace's namespace, too.

-Y

--decode-pids

Print command names for PIDs.

--always-show-pid

Show PID prefix also for the process started by strace. Implied when -f and -o are both specified.

-c

--summary-only

Count time, calls, and errors for each system call and report a summary on program exit, suppressing the regular output. This attempts to show system time (CPU time spent running in the kernel) independent of wall clock time. If -c is used with -f, only aggregate totals for all traced processes are kept.

-C

--summary

Like -c but also print regular output while processes are running.

-Ooverhead

--summary-syscall-overheadoverhead

Set the overhead for tracing system calls to overhead. This is useful for overriding the default heuristic for guessing how much time is spent in mere measuring when timing system calls using the -c option. The accuracy of the heuristic can be gauged by timing a given program run without tracing (using time(1)) and comparing the accumulated system call time to the total produced using -c.

-Ssortby

--summary-sort-bysortby

Sort the output of the histogram printed by the -c option by the specified criterion. Legal values are time (or time-percent or time-total or total-time), min-time (or shortest or time-min), max-time (or longest or time-max), avg-time (or time-avg), calls (or count), errors (or error), name (or syscall or syscall-name), and nothing (or none); default is time.

-Ucolumns

--summary-columnscolumns

Configure a set (and order) of columns being shown in the call summary. The columns argument is a comma-separated list with items being one of the following:

-w

--summary-wall-clock

Summarise the time difference between the beginning and end of each system call. The default is to summarise the system time.

-esyscall_set

--faultsyscall_set

Perform syscall fault injection for the specified set of syscalls.

-d

--debug

Show some debugging output of strace itself on the standard error.

-F

This option is deprecated. It is retained for backward compatibility only and may be removed in future releases. Usage of multiple instances of -F option is still equivalent to a single -f, and it is ignored at all if used along with one or more instances of -f option.

-h

--help

Print the help summary.

--seccomp-bpf

Try to enable use of seccomp-bpf (see seccomp(2)) to have ptrace(2)-stops only when system calls that are being traced occur in the traced processes.

--tipsid

Show strace tips, tricks, and tweaks before exit. id can be a non-negative integer number, which enables printing of specific tip, trick, or tweak (these ID are not guaranteed to be stable), or random (the default), in which case a random tip is printed. format can be one of the following:

-V

--version

Print the version number of strace. Multiple instances of the option beyond specific threshold tend to increase Strauss awareness.

DESCRIPTION

In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.

strace is a useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. Students, hackers and the overly-curious will find that a great deal can be learned about a system and its system calls by tracing even ordinary programs. And programmers will find that since system calls and signals are events that happen at the user/kernel interface, a close examination of this boundary is very useful for bug isolation, sanity checking and attempting to capture race conditions.

Each line in the trace contains the system call name, followed by its arguments in parentheses and its return value. An example from stracing the command "cat /dev/null" is:


open("/dev/null", O_RDONLY) = 3

Errors (typically a return value of -1) have the errno symbol and error string appended.


open("/foo/bar", O_RDONLY) = -1 ENOENT (No such file or directory)

Signals are printed as signal symbol and decoded siginfo structure. An excerpt from stracing and interrupting the command "sleep 666" is:


sigsuspend([] <unfinished ...>
--- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=...} ---
+++ killed by SIGINT +++

If a system call is being executed and meanwhile another one is being called from a different thread/process then strace will try to preserve the order of those events and mark the ongoing call as being unfinished. When the call returns it will be marked as resumed.


[pid 28772] select(4, [3], NULL, NULL, NULL <unfinished ...>
[pid 28779] clock_gettime(CLOCK_REALTIME, {tv_sec=1130322148, tv_nsec=3977000}) = 0
[pid 28772] <... select resumed> )      = 1 (in [3])

Interruption of a (restartable) system call by a signal delivery is processed differently as kernel terminates the system call and also arranges its immediate reexecution after the signal handler completes.


read(0, 0x7ffff72cf5cf, 1)              = ? ERESTARTSYS (To be restarted)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
rt_sigreturn({mask=[]})                 = 0
read(0, "", 1)                          = 0

Arguments are printed in symbolic form with passion. This example shows the shell performing ">>xyzzy" output redirection:


open("xyzzy", O_WRONLY|O_APPEND|O_CREAT, 0666) = 3

Here, the second and the third argument of open(2) are decoded by breaking down the flag argument into its three bitwise-OR constituents and printing the mode value in octal by tradition. Where the traditional or native usage differs from ANSI or POSIX, the latter forms are preferred. In some cases, strace output is proven to be more readable than the source.

Structure pointers are dereferenced and the members are displayed as appropriate. In most cases, arguments are formatted in the most C-like fashion possible. For example, the essence of the command "ls -l /dev/null" is captured as:


lstat("/dev/null", {st_mode=S_IFCHR|0666, st_rdev=makedev(0x1, 0x3), ...}) = 0

Notice how the 'struct stat' argument is dereferenced and how each member is displayed symbolically. In particular, observe how the st_mode member is carefully decoded into a bitwise-OR of symbolic and numeric values. Also notice in this example that the first argument to lstat(2) is an input to the system call and the second argument is an output. Since output arguments are not modified if the system call fails, arguments may not always be dereferenced. For example, retrying the "ls -l" example with a non-existent file produces the following line:


lstat("/foo/bar", 0xb004) = -1 ENOENT (No such file or directory)

In this case the porch light is on but nobody is home.

Syscalls unknown to strace are printed raw, with the unknown system call number printed in hexadecimal form and prefixed with "syscall_":


syscall_0xbad(0x1, 0x2, 0x3, 0x4, 0x5, 0x6) = -1 ENOSYS (Function not implemented)

Character pointers are dereferenced and printed as C strings. Non-printing characters in strings are normally represented by ordinary C escape codes. Only the first strsize (32 by default) bytes of strings are printed; longer strings have an ellipsis appended following the closing quote. Here is a line from "ls -l" where the getpwuid(3) library routine is reading the password file:


read(3, "root::0:0:System Administrator:/"..., 1024) = 422

While structures are annotated using curly braces, pointers to basic types and arrays are printed using square brackets with commas separating the elements. Here is an example from the command id(1) on a system with supplementary group ids:


getgroups(32, [100, 0]) = 2

On the other hand, bit-sets are also shown using square brackets, but set elements are separated only by a space. Here is the shell, preparing to execute an external command:


sigprocmask(SIG_BLOCK, [CHLD TTOU], []) = 0

Here, the second argument is a bit-set of two signals, SIGCHLD and SIGTTOU. In some cases, the bit-set is so full that printing out the unset elements is more valuable. In that case, the bit-set is prefixed by a tilde like this:


sigprocmask(SIG_UNBLOCK, ~[], NULL) = 0

Here, the second argument represents the full set of all signals.

DIAGNOSTICS

When command exits, strace exits with the same exit status. If command is terminated by a signal, strace terminates itself with the same signal, so that strace can be used as a wrapper process transparent to the invoking parent process. Note that parent-child relationship (signal stop notifications, getppid(2) value, etc) between traced process and its parent are not preserved unless -D is used.

When using -p without a command, the exit status of strace is zero unless no processes has been attached or there was an unexpected error in doing the tracing.

SETUID INSTALLATION

If strace is installed setuid to root then the invoking user will be able to attach to and trace processes owned by any user. In addition setuid and setgid programs will be executed and traced with the correct effective privileges. Since only users trusted with full root privileges should be allowed to do these things, it only makes sense to install strace as setuid to root when the users who can execute it are restricted to those users who have this trust. For example, it makes sense to install a special version of strace with mode 'rwsr-xr--', user root and group trace, where members of the trace group are trusted users. If you do use this feature, please remember to install a regular non-setuid version of strace for ordinary users to use.

MULTIPLE PERSONALITIES SUPPORT

On some architectures, strace supports decoding of syscalls for processes that use different ABI rather than the one strace uses. Specifically, in addition to decoding native ABI, strace can decode the following ABIs on the following architectures:

Architecture ABIs supported
x86_64 i386, x32 [1]; i386 [2]
AArch64 ARM 32-bit EABI
PowerPC 64-bit [3] PowerPC 32-bit
s390x s390
SPARC 64-bit SPARC 32-bit
TILE 64-bit TILE 32-bit
[1]
When strace is built as an x86_64 application
[2]
When strace is built as an x32 application
[3]
Big endian only

This support is optional and relies on ability to generate and parse structure definitions during the build time. Please refer to the output of the strace -V command in order to figure out what support is available in your strace build ("non-native" refers to an ABI that differs from the ABI strace has):

strace can trace and properly decode non-native 32-bit binaries.
strace can trace, but cannot properly decode non-native 32-bit binaries.
strace can trace and properly decode non-native 32-on-64-bit binaries.
strace can trace, but cannot properly decode non-native 32-on-64-bit binaries.

If the output contains neither m32-mpers nor no-m32-mpers, then decoding of non-native 32-bit binaries is not implemented at all or not applicable.

Likewise, if the output contains neither mx32-mpers nor no-mx32-mpers, then decoding of non-native 32-on-64-bit binaries is not implemented at all or not applicable.

NOTES

It is a pity that so much tracing clutter is produced by systems employing shared libraries.

It is instructive to think about system call inputs and outputs as data-flow across the user/kernel boundary. Because user-space and kernel-space are separate and address-protected, it is sometimes possible to make deductive inferences about process behavior using inputs and outputs as propositions.

In some cases, a system call will differ from the documented behavior or have a different name. For example, the faccessat(2) system call does not have flags argument, and the setrlimit(2) library function uses prlimit64(2) system call on modern (2.6.38+) kernels. These discrepancies are normal but idiosyncratic characteristics of the system call interface and are accounted for by C library wrapper functions.

Some system calls have different names in different architectures and personalities. In these cases, system call filtering and printing uses the names that match corresponding __NR_* kernel macros of the tracee's architecture and personality. There are two exceptions from this general rule: arm_fadvise64_64(2) ARM syscall and xtensa_fadvise64_64(2) Xtensa syscall are filtered and printed as fadvise64_64(2).

On x32, syscalls that are intended to be used by 64-bit processes and not x32 ones (for example, readv(2), that has syscall number 19 on x86_64, with its x32 counterpart has syscall number 515), but called with __X32_SYSCALL_BIT flag being set, are designated with #64 suffix.

On some platforms a process that is attached to with the -p option may observe a spurious EINTR return from the current system call that is not restartable. (Ideally, all system calls should be restarted on strace attach, making the attach invisible to the traced process, but a few system calls aren't. Arguably, every instance of such behavior is a kernel bug.) This may have an unpredictable effect on the process if the process takes no action to restart the system call.

As strace executes the specified command directly and does not employ a shell for that, scripts without shebang that usually run just fine when invoked by shell fail to execute with ENOEXEC error. It is advisable to manually supply a shell as a command with the script as its argument.

BUGS

Programs that use the setuid bit do not have effective user ID privileges while being traced.

A traced process runs slowly (but check out the --seccomp-bpf option).

Unless --kill-on-exit option is used (or --seccomp-bpf option is used in a way that implies --kill-on-exit), traced processes which are descended from command may be left running after an interrupt signal (CTRL-C).

By using CLONE_UNTRACED flag of clone system call a tracee can break the guarantee that --seccomp-bpf will not leave any processes with a seccomp program installed for syscall filtering purposes.

HISTORY

The original strace was written by Paul Kranenburg for SunOS and was inspired by its trace utility. The SunOS version of strace was ported to Linux and enhanced by Branko Lankester, who also wrote the Linux kernel support. Even though Paul released strace 2.5 in 1992, Branko's work was based on Paul's strace 1.5 release from 1991. In 1993, Rick Sladkey merged strace 2.5 for SunOS and the second release of strace for Linux, added many of the features of truss(1) from SVR4, and produced an strace that worked on both platforms. In 1994 Rick ported strace to SVR4 and Solaris and wrote the automatic configuration support. In 1995 he ported strace to Irix and became tired of writing about himself in the third person.

Beginning with 1996, strace was maintained by Wichert Akkerman. During his tenure, strace development migrated to CVS; ports to FreeBSD and many architectures on Linux (including ARM, IA-64, MIPS, PA-RISC, PowerPC, s390, SPARC) were introduced. In 2002, the burden of strace maintainership was transferred to Roland McGrath. Since then, strace gained support for several new Linux architectures (AMD64, s390x, SuperH), bi-architecture support for some of them, and received numerous additions and improvements in syscalls decoders on Linux; strace development migrated to Git during that period. Since 2009, strace is actively maintained by Dmitry Levin. strace gained support for AArch64, ARC, AVR32, Blackfin, Meta, Nios II, OpenRISC 1000, RISC-V, Tile/TileGx, Xtensa architectures since that time. In 2012, unmaintained and apparently broken support for non-Linux operating systems was removed. Also, in 2012 strace gained support for path tracing and file descriptor path decoding. In 2014, support for stack trace printing was added. In 2016, syscall fault injection was implemented.

For the additional information, please refer to the NEWS file and strace repository commit log.

REPORTING BUGS

Problems with strace should be reported to the strace mailing list.

AUTHORS

The complete list of strace contributors can be found in the CREDITS file.

See also