As for your questions:

1. It sounds like your main problem in #1 is synchronizing your intercommunicating processes. Each process only wants to take "the next step" when it has results from all of its neighbors for the previous step.

Both of your solutions are essentially "polling": running a relatively tight loop, checking for a condition until something interesting happens. This can suck up CPU time like crazy. It's much more efficient to tell the OS "I'm not doing anything right now; wake me when something interesting happens" so the kernel can give another process use of the CPU. You can do this in a few ways. The easiest way is to continue polling, but put a "sleep" in your loop. This will greatly slow down your process, since the shortest sleep time is 1 second. But it'll give other processes more CPU time. Easiest, not best.

Depending on your design, it may be a Lot better to use select or IO::Select to wait for input from all of your peers. This is more difficult, but also much more efficient. I'd check the IO::Select perldoc and post a followup if you need more help with it.

2. Does tcpdump let you inspect the loopback? If so, there are other tools such as trafshow that may give you similar but more readable information. I'm not really sure on this one.

3. I'm also not sure what to do for debugging. However, I'd suggest creating a test environment and simplifying things while testing. Instead of running 10 interconnected processes, run one process which is connected to a parent test process, just to make sure it works with one peer. Then try having the parent test process open multiple connections to the child, and test that. Once you're sure the child works very well in a sandbox, spawn a bunch of them in an interconnected way and it should be much easier to ferret out the few remaining bugs.

As for particular debugging tools: people seem to like  ddd a lot, and it looks cool from what I've seen, but I haven't spent the time to become very familiar with it. This is probably also very difficult to use with multiple processes.

In any case, if you have all your processes log to the same file, and use file locking on that file so each child's output is sent all at once instead of being interrupted by other children, debugging would be a lot easier than trying to read a streaming STDERR.

I hope this helps :)

Alan

tcpdump -i lo -w /tmp/loopback.log
[download]

This will log all packets on lo to the file /tmp/loopback.log. You can then read/analyze them with:

tcpdump -r /tmp/loopback.log 'filter stuff here'
[download]

Update:

I thought about this a little more this morning, and if you have the extra resources, and know a little about writing rules, you should be able to use an IDS (intrusion detection system) -- like Snort -- to monitor for/alert on specific events. Now that I think about it, an IDS could be a really good debugging tool for a project like you describe. (And, if you get to the point where you are testing on multiple system on a LAN, your IDS will still be able to help as long as it is located on the same segment.)

And in the spirit of using Perl for reinventing wheels, a TCP packet dumper using Net::PcapUtils ... :-)

use Net::PcapUtils;
use NetPacket::IP;
use NetPacket::TCP;
use NetPacket::Ethernet qw/:types/;
use strict;

Net::PcapUtils::loop(
    sub {
        my ($arg, $header, $packet) = @_;
        my $ethernet = NetPacket::Ethernet->decode($packet);
        if ($ethernet->{'type'} == ETH_TYPE_IP) {
            my $ip = NetPacket::IP->decode($ethernet->{'data'}, $ether
+net);
            my $tcp = NetPacket::TCP->decode($ip->{'data'});

            print
                $ip->{'src_ip'}, ":", $tcp->{'src_port'}, " -> ",
                $ip->{'dest_ip'}, ":", $tcp->{'dest_port'}, "\n\n";

            my @data = split //, $tcp->{'data'};
            while (@data) {
                print "\t";
                for (0..7) {
                    print sprintf("%02x", shift(@data)), " ";
                }
                print "\n";
            }
            print "\n";
        }
    },
    'DEV'       =>  'lo'
);
[download]

 

Ethereal's another good freely avialable packet sniffer. It seems to be able to watch the lo interface fine on my RH7.3 box.

Status at the moment is: I have a simple game of life working (where each peer only handles one GOL square - obviously this makes network traffic massive!) but need to make the Organizer (which tells peers where they are and who their neighbours are) more reliable. The Organizer is clientserver based at the moment - it is easier to have one peer marshalling all the others into position before they all start talking to each other - but it's nicely subclassed so a truly self-organizing space will be quite easy to write. (Essentially, when a newbie peer arrives,each peer "points" it to the next peer, until the newbie gets to the frontier and finds a free position; all peers know their universe size and don't let anyone get beyond it.)

dave hj~