Intro

I hereby present to a X11::GUITest script that automates the Facebook App "Graffiti". There are quite a few of those tools available on the internet, i just thought of doing an open source Perl version of it.

Also, it's quite easy to adapt to other programs and apps. See after the code for details on usage to give you an idea on how to do that.

Code

#!/usr/bin/perl -w
use strict;
use warnings;
use X11::GUITest qw(GetMousePos MoveMouseAbs ClickMouseButton SendKeys
+ PressReleaseKey :CONST);
use Image::BMP;
use Time::HiRes qw[sleep];

# ~~~~~~ Change timing here (for slow computers increase sleep times) 
+~~~~~~~~
my $sleeptime = 0.5;
my $shortsleeptime = 0.1;


my $stepsize = 1; # for debugging (skip pixels)

my $img = Image::BMP->new();
# ~~~~~~~ Change image file here ~~~~~~~~
$img->open_file('dune_sunset_big.bmp');
my ($width, $height) = ($img->{Width}, $img->{Height});

print "Optimizing rendering sequence...\n";
my $colnochange = 0;
my $colchange = 0;
my %pixels;
for(my $y = 0; $y < $height; $y+=$stepsize) {
    for(my $x = 0; $x < $width; $x+=$stepsize) {
        my @colors = $img->xy_rgb($x, $y);
        my $colortext = '';
        foreach my $grey (@colors) {
            my $greytext = lc(sprintf("%x", $grey));
            if(length($greytext) == 1) {
                $greytext = "0$greytext";
            }
            $colortext .= $greytext;
        }
        
        #print "$x $y $colortext\n";
        if(!defined($pixels{$colortext})) {
            my @tmp = ($x, $y);
            $pixels{$colortext} = \@tmp;
            $colchange++;
        } else {
            $colnochange++;
            push @{$pixels{$colortext}}, $x, $y;
        }
    }
}
print "Can use $colnochange cached colors clicks.\n";
print "Have to change color $colchange times, though!\n";

my @colmenubutton = getPos("Mouse to color select button");
my @colmenutext = getPos("Mouse to color select textfield");
my @topleft = getPos("Mouse to top left of image");
print "------ READY TO PAINT ----\n";
getPos("GET READY!");

my $didcolchange = 0;
foreach my $destcolor (reverse sort keys %pixels) {   
    setColor($destcolor);
    $didcolchange++;
    my @tmp = @{$pixels{$destcolor}};
    while(@tmp) {
        doPaint(shift @tmp, shift @tmp);
    }
    print "Finished cycle $didcolchange of $colchange\n";
}

print "Done painting!\n";

sub getPos {
    my $text = shift;
    my $x = 10;
    while($x > 0) {
        print "$text ... $x\n";
        sleep(1);
        $x--;
    }
    return GetMousePos();
}

sub setColor {
    my $greytext = shift;
    MoveMouseAbs(@colmenubutton[0, 1]);
    ClickMouseButton(M_LEFT);
    sleep($sleeptime);
    MoveMouseAbs(@colmenutext[0, 1]);
    ClickMouseButton(M_LEFT);
    sleep($sleeptime);
    #SendKeys("{BS}{BS}{BS}{BS}{BS}{BS}$greytext\n");
    my @keyarray = ('{BS}' x 6, (split//, $greytext), "\n");
    foreach my $pkey (@keyarray) {
        SendKeys($pkey);
        sleep(0.1);
    }
    sleep($sleeptime);
}

sub doPaint {
    my ($x, $y) = @_;
    
    # We're scaling the image by factor two, since we
    # have trouble using the smallest brush
    my $paintx = $x * 2 + $topleft[0];
    my $painty = $y * 2 + $topleft[1];
    
    MoveMouseAbs($paintx, $painty);
    ClickMouseButton(M_LEFT);
    sleep($shortsleeptime);

}
[download]

Usage

First, change the name of the image file at the indicated place in the code (currently, only BMP supported for simplicity). The most time generally used up on changing colors, so prefer a 256 color or greyscale indexed BMP. Also important to know: There is a slight problem with opacity when using the smallest 1-pixel brush, so the program is designed to use the 2-pixel brush and scale the image accordingly.

Open the Facebook Graffiti app in a browser side-by-side to your commandline or debugger window - you need to see the programs output. Also, set the Graffiti to a blank canvas (or whatever rocks your boat), change the opacity to nearly full and select the second-smallest brush.

Then start the script. At startup, it optimizes the rendering sequence in a way to use the least amount of color changes. Basically, it scans the image and puts all the same color pixel coordinates into the same hash-of-arrays bin. I could have paint-time scans of the whole image, but i figured this way is easier to change the algorithm later or change the order in which colors are applied to the canvas (Graffiti has an option to play back the painting generation).

Now, the script asks you to place the mouse at certain points on the screen, which is effectively a calibration for screen coordinates for when the script takes over control of your mouse and keyboard: First, on the color select button, then (click on the button to open the color menu) after the last digit of the HEX display for the color code and then (press enter to close color dialog) at the start pixel where you want to place your painting. Everytime you'll see a ten second countdown.

At this point you'll see the "GET READY!" 10 second countdown. This is your last change to either stop the script or make sure that for the duration of automated paint operation the windows wont move and your (hardware) mouse stays untouched. Unplugging/switching of the mouse during the countdown is a good way of assuring that.

Ok, you calibrated the mouse correctly, let the final countdown run and you have made sure the (hardware) mouse wont be moved. You'll also took out that good book you always wanted to read (since this will take a while).

The mouse pointer starts to move on its own painting pixels and sometimes opening the color dialog. When that happens, the mouse pointer clicks at the end of the HEX number for the colors, the script backspaces and puts in automatically the new color value. Then it presses ENTER to accept and close the dialog, continuing to paint more pixels. This process repeats until minutes (or hours) later the final picture appears and the script finishes.

Other uses

The principle is simple enough. Adapting it to other Web- or offline graphic tools is more or less a matter of writing down how to do some operations manually and then changing the setColor() and doPaint() operations accordingly. It's also quite likely that quite a few graphic tools could be automated with no programatic changes whatsoever - just with doing the calibration required anyway.

BREW /very/strong/coffee HTTP/1.1
Host: goodmorning.example.com

418 I'm a teapot