This little project is aimed at symbolic debugging of machines which do not have a serial ports. Using Firewire (called i.Link by Sony), (when completed) it could be used like kgdboe (kgdb over ethernet using netpoll, no interrupts), except for one point: Target memory can be read (by gdb) and dumped even when the system is hung or crashed (DRAM refresh and DMA need to work though) without any debugger, memory dumper or kdump in operation in the crashed kernel or system. This works now. The origin of the project is Benjamin Herrenschmidt's tool firescope, which is a FireWire frontend for the xmon kernel monitoring/debugging tool for ppc: firescope controls a remote xmon over a FireWire cable, using FireWire's direct memory access (reading/writing to memory areas) for communication. The FreeBSD guys followed benh's example with gdb and system console over firewire in 2003. Physical DMA is part of the IEEE1394 specification and all OHCI-1394 compatible controllers implement it. I do not have a FireWire card which is not OHCI-1394-compatible. I could not test the Texas Instruments PCILynx chip (have no such controller available), however. The BIOS of one of my Laptops even supports booting from Firewire and it allows to read at least 4k of memory even before Linux is booted. You might say that allowing physical reads/writes to arbitray addresses is a big security hole, but when looking closer, it always depends on physical access to the machine, similar to using USB packet sniffing to capture your keystrokes. Of course you might want to disable physical DMA access by reloading ohci1394 with the module option phys_dma=0. Some links for further details: http://en.wikipedia.org/wiki/FireWire#Security_issues http://lists.freebsd.org/pipermail/freebsd-security/2004-November/002475.html http://www.derkeiler.com/Mailing-Lists/FreeBSD-Security/2004-11/0020.html On Mac OS X laptops, apparently the easyest solution was to put epoxy into the ports of the laptop (until it was possible to disable DMA): http://rentzsch.com/macosx/securingFirewire The default however has stayed as DMA=on, at least until this blog entry. Andi Kleen ported the xmon-independent parts of firescope to x86_64/i386 and implemented dmesg buffer display without target cooperation. Using the System.map of the remote kernel, it shows you the dmesg buffer of the remote system by telling the remote FireWire controller to read the memory directly using DMA and send it to firescope over firwire. I have taken the firewire code from Andi's firescope port and added a GDB remote protocol backend to it (similar to the gdbstub in the kernel), and this is fireproxy: It proxies the gdb remote protocol to firewire (so far it allows reading and writing remote memory by gdb). Traditional remote debugging: debugger Program or system being debugged gdb(+frontend) gdbserver (Program debugging) gdbstub/kgdb (Kernel debugging) +-----------+ +-----------+ | | | | | Machine A |<- GDB Remote Protocol ->| Machine B | | gdb | (serial, Ethernet) | gdbstub | +-----------+ +-----------+ Fireproxy (when complete) provides the same functionality, but the machine being debugged is not machine B, but machine C: gdb(+frontend fireproxy Final target +-----------+ +-----------+ +-----------+ | | | | | | | Machine A |<- GDB Remote Protocol ->| Machine B |<---->| Machine C | | Connects | using TCP/IP network | TCP Port | IEEE | (gdbstub) | +-----------+ or tunnel to +-----------+ 1394 +-----------+ Naturally, one can achieve similar results by using remote login from Machine A to Machine B and running the debugger (and its frontend) directly on Machine B. This only works when the ohci1394 driver has enabled DMA, so an early ohci1394 init would be needed for early boot debugging. (or the hanging kernel is started using kexec) Quickstart: 1) Copy vmlinux.debug from Machine C's kernel to Machine A and 2) Machine C: modprobe ohci1394 (if not loaded already) 3) On Machine B, after unpacking the tarball, run: # make && ./fireproxy System.map-from-MachineC (if you have) Sample output: Loaded system.map <../System.map> <879897> bytes 2 nodes available, local node is: 1 0: ffc0, uuid: 00080100 fa360220 1: ffc1, uuid: 00080100 cc8b0120 [LOCAL] pick a target node: not a ppc utsname addr: ffffffff80323ca0 Attached to node 'f229' System : x86_64 Version: 2.6.16-rc5-git2-3-default (#1 Tue Feb 28 09:16:17 UTC 2006) Target : ffc0 Gen : 3 Ready to accept on port 4 * On Machine A: gdb /path/to/vmlinux/with/debuginfo ... (gdb) target remote Machine_B.somewhere.net:4 getting this message is normal: 0x0000000000000000 in ?? () Dumping the printk buffer to a file is possible (if it has not wrapped yet) with: (gdb) dump binary memory dmesg.out log_buf (log_buf+log_end) You could get also get a full memory dump this way and analyze it using lcrash (lkcdutils) or RH's crash tool. (Note: There is a memory leak somewhere, so this may have to be fixed first..) You can use usual commands like: (gdb) p system_utsname.release The included .gdbinit expects the vmlinux file at ../vmlinux and has a few useful macros which do symbolic stack backtraces of the tasks on the system. Here is an example: (kgdb) btpid 5507 looking for task_struct of pid 5507... This may take a few seconds or up to a few minutes (with many tasks) pid 5507 - nautilus: ------------------- 803203a0 init_task in section .data 8012fba5 __mod_timer + 169 in section .text 802bc8e2 schedule_timeout + 150 in section .text 803df758 per_cpu__tvec_bases + 280 in section .bss 803df958 per_cpu__tvec_bases + 792 in section .bss 8012f6c5 process_timeout in section .text 803df640 per_cpu__tvec_bases in section .bss 8017ad10 do_sys_poll + 629 in section .text 8017a257 __pollwait in section .text 8017ae0c sys_poll + 58 in section .text 8010a5e8 tracesys + 209 in section .text looking for task_struct of pid 1... This may take a few seconds or up to a few minutes (with many tasks) (kgdb) It could work better if I had implemented thread support in fireproxy (or be able to use the thread support in the kernel gdbstub) already, which would show Linux processes like threads in GDB but bear with me a little, I have started coding only a few days ago. Of course, the goal is to control a remote gdbstub in the same way as if you'd have a serial connection, just at 400,000,000 bps :) The next goal is to make fireproxy talk with a kernel-gdbstub, which would mean that you could get CPU backtraces, breakpoints, watchpoints and single-stepping. I guess that such communication stub may be even possible on top of kdb, even mostly as a loadable module so for quick development of the stub. The next, easyer goal would be to use the same module interface which kgdboe is using with kgdb, and implement communication with fireproxy using memory areas. To implement the "Ctrl-C" function to enter the remote kernel debugger by a remote signal, the ohci1394 driver could subsequently be modified to pass control to the kernel debugging stub when it is receives an interrupt caused by a unique packet. A detailed README which talks about all current issues and caveats too (read it if you want to want to try it now) is at: http://www.suse.de/~bk/firewire/README.txt You can download the latest tarball using this directory listing: http://www.suse.de/~bk/firewire/ There is quite quite some room for improvement (possible improvements documented in the README), but since it can be already used to debug some real problems, I wanted to make the tool known. If you have ideas or patches for improvement, they will of course be appreciated. If other developers are interrested in joining development, a project on some open source software development management system could be opened to have a common repository. Please send me a mail if you like to get involved. Bernhard PS: FireWire is a trademark of Apple, Sony uses the name i.Link for the same bus and the generic name is IEEE 1394 and just 1394 (thirteen-ninetyfour) when talking about it. FireWire is just the most popular name. Much of techical information uses IEEE 1394 to refer to the standard. PPS: Good information is in: http://en.wikipedia.org/wiki/FireWire