Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re^3: How to download html with threads?

by Trizor (Pilgrim)
on Jul 31, 2007 at 06:26 UTC ( [id://629746]=note: print w/replies, xml ) Need Help??


in reply to Re^2: How to download html with threads?
in thread How to download html with threads?

There aren't multiple threads on a single file in my example code,only the capability becuase WriteOut was wrapped in a sub to be made a thread. Only one writer thread is created, to atomically dequeue processed data and write it out.

As for the overhead issue, while in its current state the overhead doesn't merit separate threads, if this grows and starts using some form of HTML Parser in the parse stage then the split begins to make more sense as HTML parsers can be slower than downloading the document to feed them, separating the processes allows the download to finish faster and make room for the parsing.

  • Comment on Re^3: How to download html with threads?

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://629746]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others browsing the Monastery: (4)
As of 2024-04-16 22:17 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found