comment on

With some prodding from shmem, I was able to make progress, culminating in the code post above for "Update 2".

However, I still can't match the text I want, becuase the page fetched with WWW::Mech differs from the text I get when I open the url in firefox and do view source.

This is either no longer a javascript issue, or if it is a javascript issue I don't understand how to troubleshoot it.

At any rate, to simply reiterate the test case for my problem (basically the same as Update 2 above, but shorter and simpler):

use strict;
use warnings;

use WWW::Mechanize;
use Data::Dumper;

my $price = '249,9';
print "price: $price\n";

my $url = 'http://www.baur.de/is-bin/INTERSHOP.enfinity/WFS/Baur-BaurD
+e-Site/de_DE/-/EUR/BV_ExternalCall-Start?ArticleNo=515358&NUMSArt=444
+3504&NUMSArtPc=4488615&AffiliateID=pangora-bd&Name=pangora-produktdat
+en-baur&ActionID=preis-produkt-suche-baur&WKZ=79&IWL=101';

my $mech = WWW::Mechanize->new();
$mech->agent('Firefox/1.0 (Windows; U; Win98; en-US; Localization; rv:
+1.4) Gecko/20030624 Netscape/7.1 (ax)');
$mech->get( $url );
my $html = $mech->content;
print "html from $url doesn't match $price\n" unless $html =~ /$price/
+ ;
print "but paste into browser and view source, and it does\n";
print "final url after firefox redirect (but not www::mech redirect) i
+s something like " . 'http://www.baur.de/is-bin/INTERSHOP.enfinity/WF
+S/Baur-BaurDe-Site/de_DE/-/EUR/BV_DisplayProductInformation-ArticleNo
+;sid=7oVhaTsE5oZsaX6rnON4q25Uv6S6Ixu_PzIwW50ajEGxS04TwoV1a_bGFYiItw==
+?ArticleNo=515358&ls=0&firstPage=true&showGewinnspiel=true&showW3B=fa
+lse' . "\n";
# uncomment this to print html, which is totally different from what y
+ou get from firefox, show source.
# print "html: $html";
[download]

Can anyone prod me further in the right direction?

PS: I'm still wondering if I would get better results with Win32::IE::Mechanize or WWW::Selenium or Mozilla::Mechanize.

UPDATE: better user agent string after perrin comment.

In reply to WWW::Mechanize fetch differs from firefox browser result (was Re: WWW::Mechanize or WWW::Selenium with javascript redirect) by tphyahoo
in thread WWW::Mechanize or WWW::Selenium with javascript redirect by tphyahoo

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Perl Monk, Perl Meditation
	PerlMonks