Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re: I keep getting "b/bc/Wiki.png" instead of the actual thing I want

by tangent (Vicar)
on Apr 06, 2016 at 22:29 UTC ( #1159770=note: print w/replies, xml ) Need Help??


in reply to I keep getting "b/bc/Wiki.png" instead of the actual thing I want

What you get back from that URL is a full HTML page so I think you will need to bring in a HTML parser. I guess that you are looking to get at the actual image file and this is how you could get the URL for that using HTML::TreeBuilder::XPath
use HTML::TreeBuilder::XPath; # ... for my $count ( 0 .. 417 ) { my $url = "http://crappyfartsgohome.wikia.com/wiki/File:".$myFileN +ame[$count]; my $request = HTTP::Request->new(GET => $url); my $response = $browser->request($request); if ($response->is_error()) {print $response->status_line, "\n";} my $contents = $response->content(); my $tree = HTML::TreeBuilder::XPath->new; $tree->parse($contents); $tree->eof; my @links = $tree->findnodes('//div[@class="fullImageLink"]/a'); my $image_link = $links[0]; my $image_url = $image_link->attr('href'); print "$myFileName[$count]\n$image_url\n\n"; next if $image_url =~ m/$reject1/; next if $image_url =~ m/$reject2/; # do stuff with image URL }
OUTPUT:
Kniro-Lippies V.6 Concept.JPG http://vignette2.wikia.nocookie.net/crappyfartsgohome/images/b/b3/Knir +o-Lippies_V.6_Concept.JPG/revision/latest?cb=20140714143948 Kniro concept thing.png http://vignette3.wikia.nocookie.net/crappyfartsgohome/images/4/43/Knir +o_concept_thing.png/revision/latest?cb=20140714143947 Kniro og.png http://vignette3.wikia.nocookie.net/crappyfartsgohome/images/3/3e/Knir +o_og.png/revision/latest?cb=20140714143946
Note that I changed $reject1 to 'OKAY_I_SERIOUSLY_CANNOT.png'

Replies are listed 'Best First'.
Re^2: I keep getting "b/bc/Wiki.png" instead of the actual thing I want
by Nyro46 (Initiate) on Apr 07, 2016 at 01:01 UTC

    Okay ... I installed the HTML treebuilder thing through cpan or whatever.

    Am I supposed to take everything from mine and replace the "# ..." part? I still get some errors (especially around the rejects)

    Are you able to paste your entire input including the other part from mine so I can check to make sure I have it typed right? Thanks :)

      You could paste yours and we can tell if you typed it right ;)

        Okay

        use HTML::TreeBuilder::XPath; use strict; use warnings; use LWP::Simple; use LWP::UserAgent; use HTTP::Request; use HTTP::Response; my @myFileName=(''); $myFileName[0]="Kniro-Lippies V.6 Concept.JPG"; $myFileName[1]="Kniro concept thing.png"; $myFileName[2]="Kniro og.png"; ... (redacted because there are a lot) $myFileName[700]="Theta's redesign.jpg"; $myFileName[701]="THETA..jpg"; $myFileName[702]="Lippies Book 8 Page 10.jpg"; my $agentName="User:Nyro_the_Leopard (http://lippies.shoutwiki.com/wik +i/User:Nyro_the_Leopard) grabbing some data using ExtractImages.pl"; my $browser = LWP::UserAgent->new(); $browser->timeout(500); my $string='crappyfartsgohome/images/'; my $endString='"'; my $position=0; my $endPosition=0; #my $prefix='http://vignette.wikia.nocookie.net/crappyfartsgohome/imag +es/; my $prefix=''; my $delimiter="\n"; my $reject1='OKAY_I_SERIOUSLY_CANNOT.png);'; my $reject2='Yum_yum.jpg'; my $newArrayIndex=0; for (my $count=0; $count<=417; $count++){ my $url="http://crappyfartsgohome.wikia.com/wiki/File:".$myFileNam +e[$count]; my $request = HTTP::Request->new(GET => $url); my $response = $browser->request($request); if ($response->is_error()) {printf "%s\n", $response->status_line; +} my $contents = $response->content(); $position=index($contents,$string,0)+length($string); $endPosition=index($contents,$endString,$position); my $fileName=substr($contents,$position,$endPosition-$position); if ($position!=-1 && $fileName ne $reject1 && $fileName ne $reject +2){ #print $prefix.$fileName.$delimiter; print '$myFileName['.$newArrayIndex.']="'.$fileName.'";'.$deli +miter; $newArrayIndex++; } } for my $count ( 0 .. 417 ) { my $url = "http://crappyfartsgohome.wikia.com/wiki/File:".$myFileN +ame[$count]; my $request = HTTP::Request->new(GET => $url); my $response = $browser->request($request); if ($response->is_error()) {print $response->status_line, "\n";} my $contents = $response->content(); my $tree = HTML::TreeBuilder::XPath->new; $tree->parse($contents); $tree->eof; my @links = $tree->findnodes('//div[@class="fullImageLink"]/a'); my $image_link = $links[0]; my $image_url = $image_link->attr('href'); print "$myFileName[$count]\n$image_url\n\n"; next if $image_url =~ m/$reject1/; next if $image_url =~ m/$reject2/; # do stuff with image URL }

        I basically just took my code and replaced the "# ..." from tangent's code with it because I was guessing that's what I was supposed to do

        Also if he meant the "next if $image_url =~ m/$reject1/;" was supposed to have the "OKAY_I_SERIOUSLY_CANNOT.png" thing replacing $reject1, I tried that (and replaced reject2 with the yum yum one) but it didn't work.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1159770]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (3)
As of 2022-05-20 14:09 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you prefer to work remotely?



    Results (73 votes). Check out past polls.

    Notices?