Does it help at all if I re-phrase and re-format it like this (pretend that the predicates are prefixed with 'on_' even though they aren't):

lambda sub {
    context $socket;
    on_writable sub {
        print $socket "GET $url HTTP/1.0\r\n\r\n";
        my $buf = '';
        on_readable sub {
            my $n = sysread( $socket, $buf, 1024, length($buf));
            return "read error:$!" unless defined $n;
            return $buf unless $n;
            again;
        }
    }
}
[download]

When the lambda closure is executed, the 'on_writable' sets a callback to be executed when the $socket is writable. When the closure finishes, IO::Lambda sees that the socket is writable and executes the callback. That callback executes and it sets another callback for when the socket is readable. When the writable callback finishes, IO::Lambda sees that the socket is readable and executes the readable callback. That callback returns a value when all input is read, or else re-queues itself for the next time the socket is readable using again.

When all that is done, the value returned from running (er, 'wait'-ing for the lambda) is the value returned from the last callback to run -- in this case, the "return $buf".

The 'readable' part has to be set after the 'writable' part runs, otherwise, IO::Lambda could call them in any order, trying to read from the socket before the request is sent.

At least, that's how I think it works.

I agree that the nesting syntax is confusing and the way values are returned is likewise confusing.

I thought it should be read differently, then I read it more carefully and I agree with your reading. But since seeing the same thing said multiple ways often helps, let me present it my way. First merlyn's example, formatted differently.

lambda {
    context $socket;
    writable {
        print $socket "GET $url HTTP/1.0\r\n\r\n";
        my $buf = '';
        readable {
            my $n = sysread( $socket, $buf, 1024, length($buf));
            return "read error:$!" unless defined $n;
            return $buf unless $n;
            again;
        };
    };
};
[download]

Here it is again with comments stating what each piece does according to my understanding.

# This sets up one of many parallel closures that process
# in parallel.  It will be called at the start.
lambda {
    # This sets the context of what connection this happens
    # on.  This association is remembered within the engine.
    context $socket;

    # writeable sets up a possible event to monitor, when
    # $socket is writeable, execute the closure.
    writable {
        # The engine discovered we can write, so do so.
        print $socket "GET $url HTTP/1.0\r\n\r\n";

        # This variable needs to stay shared across
        # multiple invocations of our readable closure, so
        # it needs to be outside that closure.
        my $buf = '';

        # readable registers another event to monitor -
        # that $socket is readable.  Note that we do not
        # need to set the context again because when we get
        # here, the engine knows what context this command
        # took place in, and assumes the same context.
        readable {
            # This closure is executed when we can read.
            my $n = sysread( $socket, $buf, 1024, length($buf));

            # If we return without registering a follow-up
            # handler, this return will be processed as the
            # end of this sequence of events for whoever is
            # waiting on us.
            return "read error:$!" unless defined $n;
            return $buf unless $n;

            # We're not done so we need to do this again.
            # Note that the engine knows that it just
            # called this closure because $socket was
            # readable, so it can infer that it is supposed
            # to set up a callback that will call this
            # closure when $socket is next readable.
            again;
        };
    };
};
[download]

And here we see the reason for the nesting. You nest whenever one action is contingent on another having already happened. Given that lambda just registers a callback, and you always want to do something, somewhere, you always nest at least once. But you can nest more times.

One thing that is still not entirely clear here is how the $socket parameter gets passed aroud - or rather: why it needs to be passed in two ways - as a shared variable in the closure and via context? Actually - maybe I know the answer - this is because the "conditions" read from the context.

Whoa, that is a surprise, thank you! May I ask if I can copy your code comments into the documentation? If you should like to do the same to other comments or code, you are very welcome, I'd be happy with such help.

The minor thing I decided not to correct, but just changed that decision: it's not quite

This sets up one of many parallel closures that process in parallel.

but rather

This sets up a new lambda object with attached one of many closures that process sequentially.

Per one lambda, only one writable will ever be executed; after it in turn registers readable, socket won't be listened for on_write events. Same is valid for readable.

Now, back to your question. "{{" notion is not tied programmatically to the way results are returned, it's just a set of nested closures. However, it is tied conceptually, indeed it means more to a programmer, because it introduces two important parallels between plain old declarative style and all this callback and closure mess.

The first parallel is that the sequence matters. As in normal, blocking code, you would expect programming a HTTP request as

print $socket ...
readline $socket;
[download]

writable {
   syswrite ...
readable {
   sysread ...
}}
[download]

This fact, in my opinion, is also a step forward when comparing with programming using the traditional event-driven frameworks, like POE or IO::whatever:

on_write => sub { ... },
on_read  => sub { ... },
[download]

The second parallel is that return quits the scope. In the declarative style, no matter how deep the execution is down in inner cycles, return $x quits the scope of the current subroutine and sends $x to the caller. In IO::Lambda, no matter how many "}}" execution is inside, readable inside writable inside whatever, return $x does basically the same (given of course some restrictions, that no others callbacks are waiting, and no again is called). Whoever awaits for the lambda, be that asynchronous execution by tail/tails/etc, or synchronous by wait, it can count on $x being returned no matter how many stages the lambda internally went through. Again, to compare with the event-driven style, where it's surely possible, but is not that elegant and is not the part of the syntax.

From this parallel one gets an important feature, easy wrapping of lambdas, one inside another. For example, we have a lambda that returns HTTP response, as we just've discussed, created by function http. Let's write a wrapper that accepts not URL and returns plaintext, but HTTP::Request and HTTP::Response:

sub http2 {
    my $req = shift;
    lambda {
        my $uri   = $req-> uri-> as_string;
        my $host  = $req-> uri-> host;
        my $port  = $req-> uri-> port;
        my $addr  = sockaddr_in( $port, inet_aton( $host));
        context http( $addr, $uri);
    tail {
        my $res = shift;
        return ( $res =~ /^(\w+ error:)/) ?
             $res :
             HTTP::Response-> parse( $res)
    }}
}
[download]

sub http3 {
    my $req = shift;
    lambda {
        context http2($req);
    tail {
        my $res = shift;
        return $res unless ref($res);
        return $res if $res-> code !~ /^3/;

        $req-> uri( $res-> header('Location'));
        context http2( $req);
        again;
    }}
}
[download]

I hope that helps, please ask further if it doesn't.

Now, you may think you just communicated something to me in:

First parallel is that the sequence matters. As in normal, blocking code, you would expect programming a HTTP request as

print $socket ...
readline $socket;
[download]

Here, the code is different:

writable {
   syswrite ...
readable {
   sysread ...
}}
[download]

but the sequence is the same! To emphasize this fact, there's no additional indentation for inner closures, to highlight that the execution is top-down, one way, linear.

But nothing clicked for me. Why are they nested? When does the closure passed to readable {} get executed? When does the code within the closure passed to writable get executed {}? And why?

This is the magic that is then fundamental to the rest of your description. Please explain it so we mere mortals can follow along. I can't understand anything further, because it seems to be based on something you think you are showing with this code.

In other words, you have writable, which takes a coderef. And readable, which takes a coderef. Somehow, the coderef passed to writable is built up by calling some other code and then the result of calling readable. Ooof. Too many layers. Head is hurting. What's the flow, and why?

And why are they nested!?!

-- Randal L. Schwartz, Perl hacker

I'm gonna take a swing at it, because I think I get it now, and someone will correct me if I'm wrong :-)

writable {...} creates an event for the IO::Lambda event loop to wait for the socket to be writable before it executes the code ref. But the actual call to writable doesn't block, if you put code after writable's closing brace, it would get executed almost immediately. One of the examples in the docs confused me at first, where there was an again;, with some code right after it. At first I was thinking that "again" was like a goto (of the next/retry sort), and the code after would never be executed, but then I realized that it was just setting up another event for IO::Lambda to wait for, and the actual call to again() would return immediately.

So when the socket is writable, you write to it in the code ref. But if you want IO::Lambda to wait for something else after you've written to the socket, it needs to be inside writable's code ref, after you've written. So that's when we call readable() to make IO::Lambda wait for the socket to be readable.

When does the closure passed to readable {} get executed? When does the code within the closure passed to writable get executed {}?

As soon as the enclosing lambda {...} thing is started, writable is executed, and the event (of waiting for $socket to be writable) is added to the event loop. The code ref is called when $socket is writable. When the code ref is called, readable is called, and the event loop will now wait for the socket to be readable (along with any other events that it's still waiting for), and when the $socket is readable, that code ref will be called.

So if writeable {..} followed readable {...} without the nesting, you'd be creating two distinct events at the same time, and it would call one code ref if the socket was readable, and the other if it was writable in whichever order the events happened.

All right, the key word "flow" has been uttered. Here's the flow:

1. lambda {X} has just been called. It returns an empty object, blessed hash of IO::Lambda with a single coderef X, which is not executed yet. This object has a method wait(), which is then called.

2. wait() figures out that lambda (object, that is) hasn't been started yet. It switches the lambda into "started" state, and invokes callback X. Then it enters event loop sleep/select cycle, until the lambda state gets "finished":

sub wait # simplified
{
    my $lambda = shift;
    $lambda-> start if $lambda-> is_passive;
    event_loop while not $lambda-> is_finished;
    return $lambda-> peek;
}
[download]

3. Coderef X creates a socket, calls a non-blocking TCP connect on it using IO::Socket::new, and passes that socket, along with a coderef Y to standard condition writable:

context $socket;
writable { Y }
[download]

and then exits. The lambda state is unchanged by calling writable, however it adds the socket into the listening queue of the underlying event loop (select, AnyEvent, POE, whatever), and asks the loop to do something when that socket gets writable. Meanwhile, wait still processes the event loop.

4. TCP syn-synack-ack succeeded, event loop says the socket is writable, and calls the lambda that waits for the socket. Lambda figures out that the socket waiting conditions should result in calling the coderef Y, which it immediately does.

5. Coderef Y does two things: it prints to the socket, and registers another coderef Z, that will be called when the socket gets readable:

syswrite $socket ....;
readable { Z }
[download]

Note, that these callbacks are one-shot only. The subscription to the writable events is not renewed, so the socket is not listened for write events anymore.

6. Socket gets readable, Z gets executed. It reads some bytes, but wants more, so it re-subscribes on the same socket, with the same coderef Z, to be woken up when there are more data. This is done with calling again. The latter knows the last coderef used, and spares the programmer from calling readable with the same coderef.

7. Socket gets readable, Z gets executed again. It reads zero bytes, figures it's an EOF, it says "return $buf". Before that, result of the last executed coderef was accumulated inside the lambda object. Now, $buf overwrites that and is stores its own $buf, which is then accessible through method peek.

8. Lambda, having called coderef Z, finds out that it has no more subscriptions, so it switches itself to "finished" state.

9. wait sees that the lambda is finished, it stops the cycle and and returns whatever was left on lambda, in our case $buf.

I almost feel how it gets better :) Please tell me what do you think.