Changing a process’ file descriptor on the fly

source:http://ingvar.blog.redpill-linpro.com/2010/07/10/changing-a-process-file-descriptor-on-the-fly/

Changing a process’ file descriptor on the fly

I come back to a specific problem every once in a while: Changing a program’s file descriptor while the same program is running.

From time to time, we do stupid things, like running a very important shell command, and adding debug output to see that it works, then realizing that it will take hours to finish, and spitting gigabytes of debug to an xterm through ssh does not help.

An other typical example could be a finding an ill managed system with some daemon without proper logfile handling. Restarting that process right now is just out of the question, copy-truncating that 16GB logfile will take too much time, and by the way, the disk is almost full.

After a hint from the ever helping kjetilho, and a Google search, I ended up with what seems to be a well-known trick, but I did not know about it till now. Welcome to the dark side of gdb.

With the power of gdb at your hand, you can hook into the inner parts of any running program, and change, well, virtually anything. Sounds insanely dangerous for systems in production, right? Yes I agree. Still, this hack is not that ugly. It does just what the doctor ordered: It changes a process’ fds for you while it’s running.

Google found this hack by Robert McKay, and I added a couple of lines for robustness. Here it is: fdswap.sh.

For a simple test, (yes, do test this before smashing your production environment to pieces) make small shell script like this

#!/bin/bashn=0while true; do ((n++)); echo $n; sleep 1; done

Run that script in a separate window. Pick up the pid, and find out what vty is connected to that process’ standard output

ls -l /proc/$pid/fd/0

Then run the script, be really afraid of all that scary gdb output, but watch that number pumping magically stop.

fdswap.sh /dev/pts/$vty /dev/null $pid

Have a look at the fds again

ls -l /proc/$pid/fd

You can use strace to check that the process is still happily pumping numbers, but now to /dev/null

strace -p $pid -e write

fdswap.sh:

#!/bin/bash # # Swap/Roll a logfile # # Usage:              <old logfile>  <new logfile> [ optional pids ] #         ./swap.sh /var/log/logfile /tmp/logfile  [pids] # # Author: Robert McKay <[e-mail]> # Date:   Tue Aug 14 13:36:35 BST 2007 ## Update: Added usage message when needed, a fuser format fix, #         some whitespace cleanup, and a localization fix.#         Ingvar Hagelund <[e-mail]># Date:   Sat Jul 10 02:11:49 CEST 2010if [ "$2" = "" ]; thenecho "Usage: $0 /path/to/oldfile /path/to/newfile [pids]Example: $0 /var/log/somedaemon.log /var/log/newvolume/somedaemon.log 1234Example: $0 /dev/pts/53 /dev/null 1234"exit 0fiif gdb --version > /dev/null 2>&1; thentrueelseecho "Unable to find gdb."exit 1fisrc=$1 dst=$2 shift; shift pids=$* for pid in ${pids:=$( /sbin/fuser $src | cut -d ':' -f 2 )}; do echo "src=$src, dst=$dst"echo "$src has $pid using it" (     echo "attach $pid" echo 'call open("'$dst'", 66, 0666)' for ufd in $(LANG=C ls -l /proc/$pid/fd | grep $src\$ | awk ' { print $9; } ');     do echo 'call dup2($1,'"$ufd"')'done echo 'call close($1)'echo 'detach' echo 'quit' sleep 5) | gdb -q -x -done