I needed to cycle through a collection of external files, but I wanted to do it in a random order, and I also wanted to go through all of the files before I went through any file twice.

There are all sorts of ways to implement this, so that's not really a big deal. Somewhere I had an error that was giving me repeated files before I had finished a complete cycle, so I need to track that down, and I did, and it was fixed.

But was it really fixed? I wondered how I could test this. I could simply count every time I saw the file and compare all the counts at various times. That works.

I wanted to watch it though. I didn't want to speed read as a bunch of text lines scrolled past on the terminal. I wanted so compact output that would fit on the screen and updated as things happened, so I wrote a little Curses visualizer.

The particulars of the program aren't important. It output a string with two numbers which identified the resource. The first number was between 1 and 9 and the second was between 1 and 80. On the screen, I framed a matrix for these. At the right spot, I'd put something on the screen to show something just happened for the file with those two numbers.

I started with the count of the times I had seen that file. When I saw the file once, I'd put a "1" in the matrix, when I saw it twice, a "2", and so on. That kinda worked, but it was hard to read. Then I used other characters in place of number and alternated between characters with lots of empty space and those without a lot, such as ( . @ ^ # , $ | & + ). That was better, but still not good enough. I was running this things for a couple implementations and I wanted to see an anomaly instantly.

I changed to simple, colored blocks instead of characters. I just made a bunch of Curses pairs that had the same foreground and background and assigned each color to a number of times I saw the file. I went in the order of the rainbow: blue for 1 time, green for 2 times, and so on.

Too bad I can't show some images inline, so you'll have to be satisfied with some external images: first pass through the corpus (black is 0 times and blue is 1 time ), second pass (green is 2 times).

I could have also done this with sound by playing a specific pitch for each count and I wouldn't even have to watch it. I could almost tune out because the odd pitch would get my attention. 

#!/usr/bin/perl

use Curses;

init();
make_frame();

while(1)
    {    
    my( $M, $m ) = 
        `command` 
            =~ m/M: (\d+), m: (\d+)$/g;
        
    put( $M, $m, ++$count{$M}{$m} );
    }
    


# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
sub init
    {
    $SIG{__DIE__} = $SIG{TERM} = sub { endwin; exit };
    instiller;
    curs_set(0);
    start_color();

    my @colors = ( COLOR_BLACK, COLOR_BLUE, 
        COLOR_GREEN, COLOR_YELLOW, COLOR_RED );
    
    foreach my $color ( 1 .. @colors )
        {
        init_pair( $color, ( $colors[ $color ] ) x 2 )
        }

    }
    
sub make_frame
    {
    foreach my $y ( 1 .. 80 )
        {
        my $Y = $y;
        if( $y % 10 == 0 ) { $y /= 10 };
        
        $y %= 10;
        
        addch(  0, $Y+1, "$y" );
        addch( 10, $Y+1, "$y" );
        }
        
    foreach my $x ( 1 .. 9 )
        {
        addch( $x,  1, "$x" );
        addch( $x, 82, "$x" );
        }
    
    refresh;
    }
    
sub put
    {
    my( $major, $minor, $n ) = @_;
    
    attron( COLOR_PAIR($n) );
    addch( $major, $minor+2, " " );
    attroff( COLOR_PAIR($n) );
    
    refresh;
    }

[download]
--
brian d foy <bdfoy@cpan.org>

Replies are listed 'Best First'.
Re: Visualizing bugs
by graff (Chancellor) on Dec 19, 2004 at 22:26 UTC
    Using text-based graphics to watch a non-deterministic process like this is a great idea -- both really rich and really simple.

    But for the sake of debugging a process with respect to a spec of the sort:

    ... cycle through a collection of external files, ... do it in a random order, and ... go through all of the files before [doing] any file twice.

    it seems to me there are easier methods that would involve listing problem cases to an error-log file, which can be read at whatever pace with full attention to relevant details (as opposed to watching a live animation with a constant data rate, where momentary distractions can make you miss some events, and the level of detail is limited).

    For that matter, there are well-established cookbook techniques for efficiently randomizing any list of items, with no repeats and no drop-outs. I'm a little surprised to see this matter come up as something that requires special measures for debugging. (Maybe your process is more complicated, and you just decided to spare us the details?) Still, thanks and ++ for the display technique.

    (BTW, I noticed from the screen shots that you're using a mac. I just tried installing Curses.pm on my mac so I could play with your code, and the CPAN build failed because of what appear to be some conflicts in header files -- e.g. I seem to have a "/usr/include/curses.h" (from Sep 2003) that contains some conflicts with a few different *.h files in "Perl/5.8.1/darwin-thread-multi-2level/CORE/", so Curses.c gets into trouble trying to use that curses.h along with those other *.h files. Did you have this trouble, and if so, how did you get around it?)

[reply]

      The actual randomizing wasn't the problem. I had something else going on that messed things up. The trick is the boundary of two cycles. An object can be the last thing of the first cycle and the first thing of the next cycle, so effectively gets chosen twice in a row, which is bad. I wanted to spread those apart. It's not big deal: just the stupid sort of stuff that happens at 3 in the morning. I had wanted this technique for something else, so I tried it out. It doesn't really take that much attention: as long as I look at it and there are only two colors, everything is fine.

      I recall some odd problem with headers and that I had to remove something. I think I found the answer by plugging "curses perl mac os x" into Google. There is a fink port, and there is probably a Darwin Ports port too.

      If you are going to use this on a Mac, the vt100 terminal type might not work for you. I switched to dtterm (there is also Apple_Terminal), which is really just changing the TERM environment variable however your avorite shell does that.

      --
      brian d foy <bdfoy@cpan.org>
[reply]
[reply]
        That worked -- thanks!
[reply]
Re: Visualizing bugs
by CountZero (Bishop) on Dec 19, 2004 at 17:43 UTC
    Nice, I always like computer displays which have blinking lights.

    CountZero

    "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

[reply]
Re: Visualizing bugs
by The Mad Hatter (Priest) on Dec 19, 2004 at 18:14 UTC
    The sound idea is a nice one... I might have to use that sometime.
[reply]

      Many years ago I wrote some software for use on a laptop by biologists studying migrating birds of prey. The laptop collected several key data items for each group of birds that was observed. The input was specificly designed to be easy to enter with one hand and without looking at the keyboard. This is because the person using the software is likely holding a pair of binoculars and staring at the birds trying to determine what values to record before they fly too far away (or one gets lured in to be measured and tagged).

      The problem with this is that if they make a mistake, they likely won't notice until it is too late.

      So the software played middle C when the first value was entered correctly. Then it played D when the second value was entered correctly. This chromatic scale continued until the final value was entered correctly when it played a major third progression (C E G).

      They were happy with the software so I got to spend several nights on a desert mountain top watching birds and even helping to capture, measure, and tag a few (and fixing any problems with the software). (Capturing migrating birds of prey is astonishingly like fishing, only up-side-down and less boring because you can clearly see who you are trying to fool and how they are reacting to the bait.)

      The other civilians (a.k.a. non-biologists) at camp were fond of telling me how lucky I was because they had each paid quite a bit of money in order to have the privilege of volunteering to help.

      I didn't sleep much, despite the kind donation of a couple self-inflating foam sleeping pads (I think each was only about 1/2" thick). But it was a great trip, mostly because all of the biologists and civilians were such good company. I think having had to sacrifice in order to have the privilege of being there combined with knowing that if anyone got sick or hurt helicopters would have to get involved made for unusually good interpersonal relationships.

      - tye        

[reply]
        Cool story. I wonder about the possibilities if you had pianists for bird-watchers. You'd give them a MIDI keyboard and let, oh, a minor fourth stand for one bird, and a diminished major seventh another. Or perhaps when a flock passed over your heads you'd get to listen to a spontaneous fugue.

        /me remembers helping zoologists in a bat cave once

[reply]

      I've always wanted a sound-based firewall that plays a tone each time a packet is rejected, with a different tone for each port. I think with a bit of fine-tuning this could give you a kind of wide-spectrum Geiger counter that would make it very easy to recognise a change in the normal patterns that might represent a problem.

      Hugo

[reply]

        In my first ever multi-threaded program, written to run under OS/2 pre-1.0, the only reentrant IO function available was Beep( freq, duration ). So

        1. thread_1 did "beeeb....beeep...beeep";
        2. thread_2 did "beep beep....beep beep";
        3. thread_3 did "bip bip bip....bip bip bip" etc.

        You could suspend/resume each of the threads by typing it's thread number on the keyboard.

        It wasn't much of a program, but you'd be amazed how many grown men (and women) spent an inordinate amount of time being amused by it:)

        Examine what is said, not who speaks.        The end of an era!
        "But you should never overestimate the ingenuity of the sceptics to come up with a counter-argument." -Myles Allen
        "Think for yourself!" - Abigail        "Time is a poor substitute for thought"--theorbtwo         "Efficiency is intelligent laziness." -David Dunham
        "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
[reply]
[d/l]
      Years ago, I was working in the system administration department of some company where another system administrator had gone a bit overboard with the various monitor scripts he had installed. Not only would the sysadmin be paged and emailled if there was a problem, but also his workstation (in case of a server problem), or the workstation of a developer (in case of a problem with his/her computer) would speak. I wasn't so pleased as his desk was right behind mine. (I made it a habit to remotely turn down his volume if his computer would speak while he was away of his desk). And the developers often got a scare if their swap-space/tmp dir filled up (this was Solaris), because then their workstation would vocally complain. Once I ran an at job - sceduled to run while I was out of the office - that remotely filled up everybody's /tmp directory to slightly over treshold that triggered the spoken warning. Steathely (open file, unlink file, write file, wait for warning, closely). Too bad I wasn't there to see the faces of the developers amid 100 talking workstations. (We had cubicles - with low walls).
[reply]
Re: Visualizing bugs
by dragonchild (Archbishop) on Dec 20, 2004 at 13:58 UTC
    I was reminded of this when reading your OP.

    Being right, does not endow the right to be rude; politeness costs nothing.
    Being unknowing, is not the same as being stupid.
    Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
    Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

[reply]

Back to Meditations