First up, it is not a good idea to mix blocking code such as LWP::Simple with POE.
And therein lies the rub with POE.
Once you invite it into your code, forget everything else you know about writing Perl. Forget all those familiar modules that you've been using forever. Forget reading a file, or querying a directory, or asking the user a quick question, because unless there is a module in the POE::* namespace that does exactly what you want to do, and that module has been thoroughly tested and kept up to date with all the other POE::* modules you're gonna need, you're stuffed.
Because once you go the 'build our own cooperative scheduler' route, everything--and that means everything--in your program has to be be broken up into small bite-sized chunks, and retain state across those chunks, so that the non-preemptive scheduler doesn't get locked out.
It like using Lego. So long as everything you want to build has straight sides and is either a multiple of some fixed sized unit, or there is a (usually expensive) off-the-shelf part to suit, it's great. But if you want curved sides; or cirular holes; or 120° angles; or anything else that is vaguely custom--you're stuffed.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |
Once you invite it into your code, forget everything else you know about writing Perl. ....
I call bogus.
File and directory access is generally insignificant. If you're writing high-performance programs that simply can't wait for the filesystem, POE and IO-AIO get along nicely.
As we've seen elsewhere in the thread, POE-Component-Generic can wrap an asynchronous interface around all those blocking modules you know and love, assuming there isn't a POE::Component to your liking and you're not sufficiently motivated to write one.
Oh, right, and some POE components aren't maintained as well as they might be. Welcome to CPAN.
Because once you go the 'build our own cooperative scheduler' route, everything--and that means everything--in your program has to be be broken up into small bite-sized chunks, and retain state across those chunks, so that the non-preemptive scheduler doesn't get locked out.
Semi-accurate but exaggerated.
It really depends on the program in question. It's generally accepted that well-written code is already decomposed into bite-sized pieces: small functions and methods that do one thing well, and larger ones that are composed of glue logic between calls to the smaller ones. In this case, you may find that a lot of your code will work as-is.
POE provides ways to maintain state between cooperative callbacks, but the POE-agnostic parts of a program don't even need that.
A program that requires major restructuring to work in a cooperative environment may already have bigger problems.
It like using Lego. So long as everything you want to build has straight sides and is either a multiple of some fixed sized unit, or there is a (usually expensive) off-the-shelf part to suit, it's great. But if you want curved sides; or cirular holes; or 120° angles; or anything else that is vaguely custom--you're stuffed.
Completely bogus.
Lego are proprietary hardware. Most users can't fabricate their own bricks. POE is a suite of free, open-source software libraries, with internal and external extension APIs. If you can't find just the right component, you're encouraged to write it and share it with the rest of the world.
| [reply] |
File and directory access is generally insignificant.
Sorry, but searching or parsing a large file, or scanning a directory tree are certainly *not* insignificant.
And your suggestion that I might need to dump binmode, close, closedir, dbmclose, dbmopen, die, eof, fileno, flock, format, getc, print, printf, read, readdir, rewinddir, seek, seekdir, select, syscall, sysread, sysseek, syswrite, tell, telldir, truncate, warn, write, -X, chdir, chmod, chown, chroot, fcntl, glob, ioctl, link, lstat, mkdir, open, opendir, readlink, rename, rmdir, stat, symlink, sysopen, umask, unlink, utime (and more) et al. and instead learn to use the aio_* varients from IO::AIO, along with all the entirely new and very different coding methods it requires:
use IO::AIO;
aio_open "/etc/passwd", O_RDONLY, 0, sub {
my $fh = shift
or die "/etc/passwd: $!";
...
};
aio_unlink "/tmp/file", sub { };
aio_read $fh, 30000, 1024, $buffer, 0, sub {
$_[0] > 0 or die "read error: $!";
};
# version 2+ has request and group objects
use IO::AIO 2;
aioreq_pri 4; # give next request a very high priority
my $req = aio_unlink "/tmp/file", sub { };
$req->cancel; # cancel request if still in queue
my $grp = aio_group sub { print "all stats done\n" };
add $grp aio_stat "..." for ...;
no matter how good that module may be, confirms my
"forget everything else you know about writing Perl" statement exactly.
So *not* bogus.
POE-Component-Generic can wrap an asynchronous interface around all those blocking modules you know and love,
Can I use Devel::SmallProf on POE code and get sensible, usable numbers? How about Devel::Trace? Or Devel::Size? Or Algorithm::FastPermute on large set with its callback? Or Data::Rmap with it's callback? Or Inline::C? Or Win32::API::Prototype? Or the analysis functions in Math::Pari or the datatset manipulations or graphics functions in PDL? Many, many others.
All of these modules have calls that can run for a substantial amount of time (minutes or hours), but need to run within the same process as the data they are operating on. Shipping large volumes of data to another process, and then reading the results back again is not efficient for those few for which this could be done. It just doesn't work at all for most of them.
So, I say again. This "forget everything else you know about writing Perl" is not bogus. You have to learn an entirely different way of working. I'm not saying that POE isn't brilliant for working that way. I'm am saying that it requires a programmer to learn an entirely different way of working that is much harder to learn, much harder to code and much harder to debug than the standard linear flow--do this, then do that, then do something else--that a single tasking program uses. And that *every programmer* learns to program.
Instead, it substitutes a--do a bit of this (and remember where we got to); and do a bit of that (and remember where we got to); and do a bit more of this (and remember...); and oh, do a bit of something else (and remember...); and do a bit more of that (and remember...); and ...--paradigm.
Semi-accurate but exaggerated.
It really depends on the program in question. It's generally accepted that well-written code is already decomposed into bite-sized pieces: small functions and methods that do one thing well, and larger ones that are composed of glue logic between calls to the smaller ones. In this case, you may find that a lot of your code will work as-is.
Sorry again, but you cannot even run a simple sort on a moderately large dataset within a cooperative environment, because it cannot be interrupted. How about running a moderately complex regex on a large string? You cannot interupt that either. The larger the data, the bigger the problem; and the larger the data, the greater the inefficiency of your proposed solution--transfering the large dataset to a separate process and then shipping the results back again.
POE provides ways to maintain state between cooperative callbacks, but the POE-agnostic parts of a program don't even need that.
You mean I can't just use lexical variables anymore? Isn't that a retrograde step?
A program that requires major restructuring to work in a cooperative environment may already have bigger problems.
That sounds a lot like 'if your double buggy doesn't fit through my turnstill; it's your double buggy that's a fault'!
Lego are proprietary hardware. Most users can't fabricate their own bricks.
Most (Perl) users can't (or don't want to and shouldn't have to; and for most programming problems, don't have to), fabricate their own (POE) bricks. Much less have to re-fabricate other peoples existing, working, tested, freely availble open-source software bricks. Again, *not* bogus.
And with threads, they don't have to.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply]
[d/l] |
I feel the same way. I'm always amazed when people casually suggest using POE, or a similar non-blocking I/O scheme. It makes trivial things difficult. That's why the forking/threading approach is so popular: you can ignore these issues most of the time.
| [reply] |
See I find this to be less and less true as POE develops. For example last year I wanted to hook a POE based IRC bot up to a web service. I didn't want to figure out how to make PoCo::Client::HTTP work when I knew I could get things off the ground quickly with LWP. So I wrote it with LWP, in a straight OO fashion and then used POE::Component::Generic to handle making it non-blocking for POE.
Perhaps I have an odd case where my issue is just simple enough to work, but increasingly POE is becoming a decent solution for non-blocking code. The results are in the Net::AIML dist.
In my experience threading/forking doesn't change the issues most people are going to have, it just means you have to re-write a bunch of wheels dealing with concurrency and blocking (literally POE::Wheel's in the case of forking) instead of learning the POE API for doing the same tasks. But then I'm obviously biased toward POE which had a stable API when I started writing concurrent code, where as threading didn't.
| [reply] |
