LWP::UserAgent vs. NTLM Challenge/Response

jpk236 has asked for the wisdom of the Perl Monks concerning the following question:

Monks:

I'm working on a project for work where I need to screen scrape an internal website. All of the internal sites use Single Signon (SSO).

This is the current code I've been working with:

#!/usr/bin/perl -w

use strict;
use LWP::UserAgent;

my ($ua, $req, $res, $output);

$ua = new LWP::UserAgent(keep_alive=>1);
$ua->agent('Internet Explorer/6.0');
$ua->credentials('domain.com:80', '', "domain\\user", 'pass');

$req = HTTP::Request->new(POST => 'http://domain.com/cgi-bin/test.cgi'
+, HTTP::Headers->new('WWW-Authenticate' => 'NTLM'));
$req->content_type('application/x-www-form-urlencoded');
$req->content('testvar=testvalue');

$res = $ua->request($req);

$output = $res->content;

print "output:\n".$output;
[download]

This is the output I get when running this script:

output:
<HTML>
<HEAD>
<TITLE></TITLE>
</HEAD>
<BODY onLoad="document.AUTOSUBMIT.submit();">
This page is used to hold your data while you are being authorized for
+ your request.<BR><BR>You will be forwarded to continue the authoriza
+tion process. If this does not happen automatically, please click the
+ Continue button below.

<FORM NAME="AUTOSUBMIT" METHOD="POST" ACTION="http://logon.domain.com/
+ntlm/creds.ntc?CHALLENGE=&SMAGENTNAME=$SM$7itOP8YnQhR1qQcOO6pp75JDJJF
+C96mLihRDilLmOSr8oAm7zddwfw%3d%3d&TARGET=$SM$http%3a%2f%2domain%2ecom
+%2fcgi-bin%2ftest%2ecgi">

<INPUT TYPE="HIDDEN" NAME="SMPostPreserve" VALUE="L2JDNDdmSXpOWVVOT2wy
+YUt4VjZpb1JkbkkxRVlZbUd4ZVdadkFoaGpQZWJVZHE4eW9IcGNJNEx3dUdmYi81Qw">
<INPUT TYPE="SUBMIT" VALUE="Continue">

</FORM>

</BODY>
</HTML>
[download]

I've tried a number of methods to handle the challenge/response, but to no avail. I've even tried using a stand-alone curl command, which has a --ntlm option.

Any input from the ranks is appreciated.

Thanks!

- Justin

Comment on LWP::UserAgent vs. NTLM Challenge/Response Select or Download Code

Replies are listed 'Best First'.
Re: LWP::UserAgent vs. NTLM Challenge/Response by jettero (Monsignor) on Oct 04, 2007 at 14:15 UTC
I don't know what ntlm is, but I'd check WWW::Mechanize before I gave up... If it requires javascript, you may not ever get it to work, but if it's just those query params, mech can probably handle it with little or no effort on your part. -Paul	[reply]
Re^2: LWP::UserAgent vs. NTLM Challenge/Response by bigmacbear (Monk) on Oct 04, 2007 at 18:01 UTC
There is JavaScript in that page, but all it does is auto-submit the embedded form containing the challenge and response. I've never had to do this so I don't know for certain it will work, but if you can use the "submit" method in WWW::Mechanize to submit the form presented right back at the server, that should get you logged in.	[reply]
Re: LWP::UserAgent vs. NTLM Challenge/Response by atemon (Chaplain) on Oct 04, 2007 at 14:36 UTC
In many cases, the site with SSO sets session cookies to keep track of authenticated sessios. Please try enabling cookie jars for LWP::Useragent, to store cookies either by adding `$ua->cookie_jar({ file => "$ENV{HOME}/.cookies.txt" });` [download] or `require HTTP::Cookies; $ua->cookie_jar(HTTP::Cookies->new(file => "$ENV{HOME}/.cookies.txt" +));` [download] or even `$ua->cookie_jar({}); #this will store cookie in memory. Not in file` [download] to your code. Enable cookie jar before you send the request. Hope this helps. --VC	[reply] [d/l] [select]
Re^2: LWP::UserAgent vs. NTLM Challenge/Response by jpk236 (Monk) on Oct 04, 2007 at 15:18 UTC
VC: Thanks for the reply. I tried all three of your examples and none of them seemed to help. This is all that was written to cookies.txt after I ran the script a few times with the first two examples: `#LWP-Cookies-1.0` [download] - Justin	[reply] [d/l]
Re: LWP::UserAgent vs. NTLM Challenge/Response by blahblahblah (Priest) on Oct 05, 2007 at 01:09 UTC
I believe the stuff that LWP does deals with HTTP headers, and in your case it looks like the info is in form fields. You're not likely to find a pre-made solution that fits this method. You could possibly roll your own with Authen::NTLM. The first step would probably be to log into the site using your browser and watch how it works using something like Paros or Charles. Speaking of those tools, an alternative might be to watch if they set a cookie, like vcTheGuru suggested. If so, then you might be able to hardcode a spoofed cookie to trick the site into thinking you have one long never-ending session. Even if neither of my ideas work out, I'd suggest trying one of the tools I linked to above. They'll help take the guesswork out of what you need to do. Joe	[reply]
Re: LWP::UserAgent vs. NTLM Challenge/Response by phio (Acolyte) on Jul 27, 2009 at 11:23 UTC
You can use `LWP::UserAgent` along with `LWP::Authen::Ntlm`. Here's a example modified from the POD document of `LWP::Authen::Ntlm` use LWP::UserAgent; use HTTP::Request::Common; $url = 'http://foo.bar.com/some_path'; # Set up the ntlm client # and then the base64 encoded ntlm handshake message $ua = new LWP::UserAgent(keep_alive=>1); $netloc = 'foo.bar.com:80'; $domain_user_name = 'Domain\User'; $password = 'blahblah'; $ua->credentials($netloc, '', "$domain_user_name", "$password"); $request = GET $url; print "--Performing request now...-----------\n"; $response = $ua->request($request); print "--Done with request-------------------\n"; if ($response->is_success) { print "It worked!->" . $response->code . "\n" } else { print "It didn't work!->" . $response->code . "\n" } print $response->content; [download]	[reply] [d/l] [select]
Re: LWP::UserAgent vs. NTLM Challenge/Response by Giuoco (Initiate) on May 05, 2009 at 19:22 UTC
jpk236, Any resolution to this? I have the same issue with SSO and LWP::UserAgent. Thanks, G.	[reply]


Come for the quick hacks, stay for the epiphanies.
	PerlMonks