Re^3: HTTP::Recorder Problem

Hmmm... I can't access it now, either. When I talked to leira, she did say she was in the middle of a cross-country move; that may have something to do with it.

I would normally never post another's work, but given the fact the module is released under the GNU Public License and that it was publically downloadable at least as of September 27, 2005 when I retrieved it, it is probably ok to post a diff of the current CPAN version and the version I downloaded. (Plus I ran the idea by the folks in the CB first. :-) )

Note: HTTP::Recorder is the work of leira, a.k.a. Linda Julien, and is Copyrighted by same.

$ diff -u Recorder.old Recorder.pm > Recorder.diff
[download]

Recorder.diff:

--- Recorder.old    2005-11-01 13:56:10.155897600 -0600
+++ Recorder.pm    2005-09-27 11:48:12.000000000 -0500
@@ -87,11 +87,12 @@
 use strict;
 use warnings;
 use LWP::UserAgent;
+
 use HTML::TokeParser;
 use HTTP::Recorder::Logger;
 use URI::Escape qw(uri_escape uri_unescape);
 use URI::QueryParam;
-use HTTP::Request::Params;
+use HTTP::Recorder::Request;
 
 our @ISA = qw( LWP::UserAgent );
 
@@ -182,12 +183,15 @@
 
     my $response;
 
+    # make a HTTP::Recorder::Request object
+    my $newrequest = HTTP::Recorder::Request->new(request => $request
+,
+                          prefix => $self->{prefix});
+
     # special handling if the URL is the control URL
-    if ($request->uri->host eq $self->{control}) {
+    if ($newrequest->uri->host eq $self->{control}) {
 
     # get the arguments passed from the form
-    my $arghash;
-    $arghash = extract_values($request);
+    my $arghash = $newrequest->get_params;
 
     # there may be an action we need to perform
     if (exists $arghash->{updatescript}) {
@@ -226,11 +230,39 @@
                     $content,
                     );
     } else {
-    $request = $self->modify_request ($request)
-            unless $self->{ignore_favicon}
-                && $request->uri->path =~ /favicon\.ico$/i;
+    my $values = $newrequest->get_params;
+    my $action = $values->{"$self->{prefix}-action"};
+    if ( $self->{ignore_favicon} && 
+         $newrequest->uri->path =~ /favicon\.ico$/i) {
+        # don't do anything
+    } elsif (!$action) {
+        $newrequest->unmodify_query();
+
+        if (!$newrequest->headers->referer) {
+
+        # log a blank line to give the code a little breathing room
+        $self->{logger}->LogLine();
+        $self->{logger}->GotoPage(url => $newrequest->uri);
+        }
+    } else {
+        $self->log_actions($values, $action);
+
+        # undo what we've done
+        $newrequest->unmodify_query();
+    }
+
+    my $https = $values->{"$self->{prefix}-https"};
+    if ( $https && $https == 1) {
+        my $uri = $newrequest->uri;
+        $uri->scheme('https') if $uri->scheme eq 'http';
+
+        $newrequest = new HTTP::Request($newrequest->method, 
+                        $uri, 
+                        $newrequest->headers, 
+                        $newrequest->content);
+    }
 
-    $response = $self->SUPER::send_request( $request );
+    $response = $self->SUPER::send_request( $newrequest );
 
     my $content_type = $response->headers->header('Content-type') || 
+"";
 
@@ -243,25 +275,15 @@
     return $response;
 }
 
-sub modify_request {
+sub log_actions {
     my $self = shift;
-    my $request = shift;
+    my $values = shift;
+    my $action = shift;
 
-    my $values = extract_values($request);
+    return unless $action;
 
     # log the actions
-    my $action = $values->{"$self->{prefix}-action"};
-
-    my $referer = $request->headers->referer;
-    if (!$action) {
-    if (!$referer) {
-        my $uri = $self->unmodify($request->uri);;
-
-        # log a blank line to give the code a little breathing room
-        $self->{logger}->LogLine();
-        $self->{logger}->GotoPage(url => $uri);
-    }
-    } elsif ($action eq "follow") {
+    if ($action eq "follow") {
     $self->{logger}->FollowLink(text => $values->{"$self->{prefix}-te
+xt"} || "",
                 index => $values->{"$self->{prefix}-index"} || "",
                 url => $values->{"$self->{prefix}-url"});
@@ -309,62 +331,6 @@
     $self->{logger}->LogLine();
     }
 
-    # undo what we've done
-    $request->uri($self->unmodify($request->uri));
-    $request->content($self->unmodify($request->content));
-
-    # reset the Content-Length (if needed) to prevent warnings from
-    # HTTP::Protocol
-    if ($action && ($action eq "submitform")) {
-    $request->headers->header('Content-Length' => length($request->co
+ntent()) );
-    
-    }
-
-    my $https = $values->{"$self->{prefix}-https"};
-    if ( $https && $https == 1) {
-    my $uri = $request->uri;
-    $uri->scheme('https') if $uri->scheme eq 'http';
-
-    $request = new HTTP::Request($request->method, 
-                     $uri, 
-                     $request->headers, 
-                     $request->content);
-    
-    }        
-
-    return $request;
-}
-
-sub unmodify {
-    my $self = shift;
-    my $content = shift;
-
-    return $content unless $content;
-
-    # get rid of the arguments we added
-    my $prefix = $self->{prefix};
-
-    for my $key ($content->query_param) {
-    if ($key =~ /^$prefix-/) {
-        $content->query_param_delete($key);
-    }
-    }
-    return $content;
-}
-
-sub extract_values {
-    my $request = shift;
-
-    my $parser = HTTP::Request::Params->new({
-    req => $request,
-    });
-
-    # un-escape all params
-    for my $key (keys %{$parser->params}) {
-    $parser->params->{$key} = uri_unescape($parser->params->{$key});
-    }
-
-    return $parser->params;
 }
 
 sub modify_response {
@@ -565,7 +531,7 @@
     $fields .= ("<input type=hidden name=\"$self->{prefix}-formname\"
+ value=\"$name\">\n");
     $fields .= ("<input type=hidden name=\"$self->{prefix}-formnumber
+\" value=\"$number\">\n");
     if ($https) {
-    $fields .= ("<input type=hidden name=\"$self->{prefix}-https\" va
+lue=\"$https\">\n");
+    $fields .= ("<input type=hidden name=\"$self->{prefix}-https\" va
+lue=\"$https\">\n");
     }
 
     return $fields;
@@ -586,7 +552,7 @@
 </head>
 <body>
 <h1>Start Recording</h1>
-<p>Type a url into the browser's adddress field to begin recording.
+<p>Type a url into the browser adddress field to begin recording.
 </html>
 EOF
 
@@ -682,12 +648,12 @@
     my $js = <<EOF;
 // find the top-level opener window
 var opwindow = window.opener;
-while (opwindow.opener) {
+while (opwindow && opwindow.opener) {
   opwindow = opwindow.opener;
 }
 // update it with HTTP::Recorder's control panel
 if (opwindow) {
- opwindow.location = "http://http-recorder/";
+ opwindow.location = "http://$self->{control}";
 }
 EOF
 
@@ -725,7 +691,7 @@
 Please submit any feature requests, suggestions, bugs, or patches at
 http://rt.cpan.org/, or email to bug-HTTP-Recorder@rt.cpan.org.
 
-If you're submitting a bug of the type "X doesn't record correctly,"
+If your bug is of the type "X doesn't record correctly,"
 be sure to include a (preferably short and simple) HTML page that
 demonstrates the problem, and a clear explanation of a) what it does
 that it shouldn't, and b) what it should do instead.
@@ -735,8 +701,8 @@
 You can read more about L<HTTP::Recorder>, including browsing the
 current source tree, at http://www.bitmistress.org/.
 
-There's a mailing list for users and developers of HTTP::Recorder.
-You can subscribe at
+You can subscribe to the mailing list for users and developers of
+HTTP::Recorder at
 http://lists.fsck.com/mailman/listinfo/http-recorder, or by sending
 email to http-recorder-request@lists.fsck.com with the subject
 "subscribe".
@@ -749,7 +715,7 @@
 Copyright 2003-2005 by Linda Julien <leira@cpan.org>
 
 Released under the GNU Public License.
-
+'
 =cut
 
 1;
[download]

Update: added <readmore> tags

HTH,

planetscape

Comment on Re^3: HTTP::Recorder Problem Select or Download Code

Replies are listed 'Best First'.
Re^4: HTTP::Recorder Problem by sivathar (Initiate) on Nov 04, 2005 at 16:06 UTC
Hi planetscape, I made the changes as your diff file and now I need Request.pm file. siva.	[reply]
Re^5: HTTP::Recorder Problem by planetscape (Chancellor) on Nov 05, 2005 at 02:36 UTC
Looks like you also have to have HTTP::Request installed, which is part of libwww. HTH, planetscape	[reply]
Re^6: HTTP::Recorder Problem by Anonymous Monk on Nov 07, 2005 at 16:42 UTC
When I tried to install HTTP::Request through following method. 1. perl -MCPAN -e 'shell' 2. o conf makepl_arg INSTALLDIRS=perl 3. Install HTTP::Request ftp://ftp.nrc.ca/pub/CPAN/authors/01mailrc.txt.gz it timeout while trying to connect to that site. Any idea. Siva	[reply]
Re^7: HTTP::Recorder Problem by planetscape (Chancellor) on Nov 07, 2005 at 22:34 UTC
Re^4: HTTP::Recorder Problem by roytech88 (Initiate) on Mar 31, 2006 at 02:06 UTC
how come that some other website's log-in information would no t be recorded in the HTTP::Recorder Control Panel? like mail.yahoo.com..and some other website's links would not be recorded like..https:/www.math.gatech.edu/ do i need to modify the recorder.pl? thanks	[reply]
Re^5: HTTP::Recorder Problem by planetscape (Chancellor) on Mar 31, 2006 at 07:09 UTC
Well, I don't know about https:/www.math.gatech.edu/, but I do have a Yahoo mail account I tested it on. While there are instructions here on "Recording SSL sessions" using HTTP::Recorder, I was unable to successfully record anything useful when accessing http://mail.yahoo.com. My proxy script: #!/usr/bin/perl # myHTTPProxy.pl # Run script under Cygwin; use Netscape for browser # Goto Tools \| Options... \| General \| Connection Settings... # Check "Manual proxy configuration" # HTTP Proxy = localhost # Port = 8080 # Click "OK" # # Also, see Perl Testing: A Developer's Notebook, pp. 135-138 # use HTTP::Proxy; use HTTP::Recorder; my $proxy = HTTP::Proxy->new(); # create a new HTTP::Recorder object my $agent = new HTTP::Recorder( showwindow => 1 ); # set the log file (optional) $agent->file("/tmp/myfile"); # set HTTP::Recorder as the agent for the proxy $proxy->agent( $agent ); # start the proxy $proxy->start(); 1; [download] complains of the following: `Can't locate object method "host" via package "URI::_generic" at /usr/ +lib/perl5/site_perl/5.8/HTTP/Recorder.pm line 191.` [download] which has not been remedied by upgrading URI::_generic. So, I'd have to say your best bet is either to contact HTTP::Recorder's author, leira, or hope some other Monk comes along who knows the answer. Sorry I can't be of further help, although if I do find something in future, I'll return and update this node. HTH, planetscape	[reply] [d/l] [select]


more useful options
	PerlMonks