Showing scp's Progress Using zenity (pseudo terminal example)

I recently discovered zenity for making quick, straightforward, graphical interfaces for shell scripts. One such script I wrote uses scp for backups. Since scp prints its percentage progress to stdout I figured it would be pretty easy to simply use a regex to parse the output and pipe it to zenity. I guessed this venture would take about ten minutes (I am not very experienced with regex), since zenity --progress just expects a number printed to its stdin in order to update the progress bar. I couldn't have been more mistaken; after four hours of hacking and wtfs I gave up. Finally, after a discussion with my old operating systems professor and a little bit of coding I finally have a solution.

After piping scp's output to various c and perl programs I determined that the problem was probably a terminal mode problem. A standard (canonical) terminal sends input to the program as lines, thus if there are never any new lines (such as in scp's output) the program would never receive the data. After figuring out how to put c (and perl) in to raw mode, as well as playing with stty, I managed to get a couple programs working that could read input before a newline was sent. I tested these with stdin and pipes and they seemed to work fine, but they still wouldn't work with scp. After a bit of debugging I figured out that when the data was piped to my c program I couldn't put the terminal in to raw mode. Go figure, when you pipe data to a program it doesn't use a terminal... it uses a pipe (duh!). This is the point where I gave up, as I didn't really feel like digging through the scp source code, and I certainly didn't want to run a modified copy of scp (that would kind of defeat the whole purpose).

After discussing the problem with an old professor he hypothesized that scp was checking if it was being run on a terminal, and if it wasn't then it was disabling output. Sure enough, on line 396 of scp.c:

if (!isatty(STDOUT_FILENO))
showprogress = 0;


As a possible solution he mentioned pseudo terminals. Apparently back in the old days psuedo terminals were quite a pain since you had to find a free one manually before you could use it. Luckily linux provides a convenient function, forkpty(), that finds a free pseudo terminal, forks a new process, and attaches the new process to the terminal. Sweet. Now with some simple fileio it works fine:


#include <unistd.h>
#include <sys/types.h>
#include <stdio.h>

main(int argc, char **argv) {
int fd;
pid_t pid;
char c,c1,c2;

if (argc != 3) {
printf("usage: [[user@]host1:]file1 [[user@]host2:]file2\n\nThis is a program that wraps scp and prints out the numeric progress on separate lines.\n");
fflush(stdout);
_exit(1);
}

pid = forkpty (&fd, NULL, NULL, NULL);
if (pid == 0) {
execlp ("scp", "scp", argv[1], argv[2], (char *) NULL);
_exit (1);
} else if (pid == -1) {
return 1;
} else {
FILE *F;

F = fdopen (fd, "r");

//reading character by character isn't horribly efficient...
while((c = fgetc(F)) != -1) {
if (c == '%') {
if (c1 == ' ')
printf("%c\n", c2); //one digit progess
else if (c1 == '0' && c2 == '0')
printf("100\n"); //done
else
printf("%c%c\n", c1,c2); //two digit progress
}
fflush(stdout);
c1 = c2;
c2 = c;
}

fflush (F);
wait (0);
}
return 0;
}


Notably, I went ahead and parsed the scp output in c. It seemed to be a cleaner solution than piping this output to a perl script and then to zenity. I initially missed the fflush() in the while loop, which took me a while to figure out :/.

To compile it you need to use the -lutil switch. So first copy and paste this c program in to scpwrap.c. Then run "gcc -lutil scpwrap.c -oscpwrap". Before you can use it you have to get rid of scp's authentication prompt by setting up some authentication keys:


ssh-keygen -t rsa
cat .ssh/id_rsa.pub | ssh user@example.com 'cat >> .ssh/authorized_keys'


Now to use the wrapper you can do something like:


./scpwrap /home/ubuntu/somefile user@example.com:~ | zenity --progress


Of course you can add whatever zenity switches you want. You cannot, however, add any scp switches without modifying the c program... Sorry.

---------------------Notes and Sources---------------------------

Terminal raw mode "stty raw".

Perl expression to parse the scp output:
perl -015 -l -ne 'print /(\d+)%/'
Notice the "-015" that tells it to use carriage returns as the line delimiter. Pretty cool, it probably would have worked if scp wasn't checking if it was running on a terminal.

Perl code to put terminal in to raw mode:

#!/usr/bin/perl

use Term::ReadKey;

ReadMode 5; # Turn off controls keys
while (($key=getc) ) {
print "$key\n";
last if $key eq "q";
};
print "Get key $key\n";
ReadMode 0; # Reset tty mode before exiting


C code to put terminal in to raw mode.

Zenity Progress Example 1:

(for a in `seq 1 100` ;


do
echo $a;
sleep 0.03;


done) | zenity --auto-close --progress \


--text="Slow counting from 1 to 100" \
--title="Example Progress"


Zenity Progress Example 2:

#!/bin/sh
(
echo "10" ; sleep 1
echo "# Updating mail logs" ; sleep 1
echo "20" ; sleep 1
echo "# Resetting cron jobs" ; sleep 1
echo "50" ; sleep 1
echo "This line will just be ignored" ; sleep 1
echo "75" ; sleep 1
echo "# Rebooting system" ; sleep 1
echo "100" ; sleep 1
) |
zenity --progress \
--title="Update System Logs" \
--text="Scanning mail logs..." \
--percentage=0

if [ "$?" = -1 ] ; then
zenity --error \
--text="Update canceled."
fi


Update: I modified the scp source (openssh5.2p1) to include a '-n' switch to print out the progress meter on new lines, regardless of whether stdout is a tty. I proposed this new feature to the openssh-unix-dev listserv, but I doubt they will adopt it. The email thread and patch can be found in their archives.

Update 2: Hrm, the pseudo terminal hack doesn't work if you run it from a launcher (kind of the whole point).