Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

Re: Multithreading Parsers

by djantzen (Priest)
on Mar 22, 2005 at 01:06 UTC ( #441354=note: print w/replies, xml ) Need Help??

in reply to Multithreading Parsers

A streaming parser doesn't seem like a very threadable kind of thing insofar as you're talking about walking a document tree in parallel. I think you'd end up doing a great deal of work trying to rig something where, for example, you'd have dedicated threads for processing particular node types.

I think I'd be more inclined to try to break the raw input into separate chunks that could run in parallel processes, something your 28 processor machine could do very well. So, maybe rather than reengineering your existing code to use threads, you could write a frontend script to divide whole units of work and then fork off processes.

Hard to say much more without concrete examples.

"The dead do not recognize context" -- Kai, Lexx

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://441354]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (8)
As of 2021-01-26 16:36 GMT
Find Nodes?
    Voting Booth?