Conclusion

Troubleshooting Linux is an art, but these tools can help you master it. You can read more usage details about these tools on their respective man pages. Remember that knowing how to use a tool is not the same as knowing when to use it. As you encounter different problems and tackle them, you'll eventually learn the art of diagnosing trouble and fixing problems on your Linux system.

ldd<nobr> <wbr></nobr>/path/to/executable:

Show dynamic linker library resolution for executable. Incredibly useful when handling weird issues due to library path problems etc.

gdb:

duh. The GNU Debugger. Even on non-debug binaries

it's incredibly useful in diagnosing crashes

(except on Debian, who strip their binaries

of all symbol information). Distros with

-debuginfo packages make it even more handy.<nobr> <wbr></nobr>/proc/$PID

The<nobr> <wbr></nobr>/proc file system has a lot of very useful info. You can `kill -STOP' a process (or not) and poke around in it's<nobr> <wbr></nobr>/proc/pid directory for things like open file handles, memory maps, etc.

ps

Almost too obvious to mention, but yet often overlooked in detail. Learn the details of ps, it can do some great stuff. I find the `wchan' field particularly useful as it shows WHERE in the kernel a process is blocked. Top can also be told to show this with an rc file edit.

`ip'

Your networking swiss army knife. `ip route show' is obviously useful, but ip does a lot more, and most of it is worth learning when facing network issues.

`netstat'

Display network status information. Impossibly useful, especially with the `-p' argument (requires root) to show which process is associated with a socket. That said, lsof provides essentially all netstat's functionality and much more besides. To the person asking about dtrace: No, nothing like that in standard kernels. There are some 3rd party tracing patches of various quality and utility. Personally I'm interested in finding a tool to list mandatory and advisory locks on a file/dir, so if anyone knows of one I'm all ears. I'm sure it's a standard tool, I just haven't found it.

---