Odd LWP Problem

mdkoehn has asked for the wisdom of the Perl Monks concerning the following question:

The following script works, reading from the DATA section:

#! /usr/bin/perl -C255

use strict;
use warnings;
use LWP::Simple;
use Encode 'decode';
use URL::Encode ':all';
use feature 'unicode_strings';
use utf8;
use open ':encoding(UTF-8)';

open INFIL, "<:encoding(UTF-8)", "temp.txt" or die; #This line generat
+es a warning.

#while (<INFIL>) {
while (<DATA>) {
    chomp;
    my $title = $_;
    $title =~ s/\/.*//;
    $title =~ s/\[by.*//;
    print "Title Searched: $title\n";
    $title = url_encode_utf8($title);
    my $html = getprint("https://clio.columbia.edu/catalog?q=$title&se
+arch_field=title_starts_with&f%5B-format%5D%5B%5D=Online&commit=Searc
+h") or die;
}

exit;

__DATA__
How to conduct your own survey / Priscilla Salant and Don A. Dillman.
[download]

But, if I change it to read from a file, like so:

#! /usr/bin/perl -C255

use strict;
use warnings;
use LWP::Simple;
use Encode 'decode';
use URL::Encode ':all';
use feature 'unicode_strings';
use utf8;
use open ':encoding(UTF-8)';

open INFIL, "<:encoding(UTF-8)", "temp.txt" or die;

#while (<DATA>) { ###CHANGE HERE###
while (<INFIL>) {
    chomp;
    my $title = $_;
    $title =~ s/\/.*//;
    $title =~ s/\[by.*//;
    print "Title Searched: $title\n";
    $title = url_encode_utf8($title);
    my $html = getprint("https://clio.columbia.edu/catalog?q=$title&se
+arch_field=title_starts_with&f%5B-format%5D%5B%5D=Online&commit=Searc
+h") or die;
}

exit;

__DATA__
How to conduct your own survey / Priscilla Salant and Don A. Dillman.
[download]

I get the error: 501 Protocol scheme 'https' is not supported (LWP::Protocol::https not installed) <URL:https://clio.columbia.edu/catalog?q=How+to+conduct+your+own+survey+&search_field=title_starts_with&f%5B-format%5D%5B%5D=Online&commit=Search>

The input file contains only a single line, the same as the DATA section.

I've tried this on both Strawberry Perl 5.24.1 and Cygwin Perl 5.22.3

All feedback appreciated.

Thanks, Michael

Comment on Odd LWP Problem Select or Download Code

Replies are listed 'Best First'.
Re: Odd LWP Problem by choroba (Cardinal) on May 03, 2017 at 19:13 UTC
That's really weird. Both versions work the same for me (5.18.2 on Linux). ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l]
Re: Odd LWP Problem by kennethk (Abbot) on May 03, 2017 at 19:56 UTC
I concur with choroba that it's odd. I see no difference under Perl 5.22 ActiveState (or 5.20): `This is perl 5, version 22, subversion 1 (v5.22.1) built for MSWin32-x +86-multi-t hread-64int (with 1 registered patch, see perl -V for more detail) Copyright 1987-2015, Larry Wall Binary build 2201 [299574] provided by ActiveState http://www.ActiveSt +ate.com Built Dec 24 2015 12:36:28` [download] But get an unhandled exception under 64-bit ActiveState 5.22 and 5.24: This is perl 5, version 22, subversion 1 (v5.22.1) built for MSWin32-x +64-multi-t hread (with 1 registered patch, see perl -V for more detail) Copyright 1987-2015, Larry Wall Binary build 2201 [299574] provided by ActiveState http://www.ActiveSt +ate.com Built Jan 4 2016 12:12:58 ... This is perl 5, version 24, subversion 0 (v5.24.0) built for MSWin32-x +64-multi-t hread (with 1 registered patch, see perl -V for more detail) Copyright 1987-2016, Larry Wall Binary build 2400 [300558] provided by ActiveState http://www.ActiveSt +ate.com Built Jun 9 2016 21:44:59 [download] and it happens for this reduced example: `#! /usr/bin/perl use strict; use warnings; use LWP::Simple; open my $inf, "<", "temp.txt" or die; #This line no longer generates a + warning. while (<$inf>) { #while (<DATA>) { my $html = getprint("https://clio.columbia.edu/catalog?q=How%20to% +20conduct%20your%20own%20survey%20&search_field=title_starts_with&f%5 +B-format%5D%5B%5D=Online&commit=Search") or die; } exit; __DATA__ How to conduct your own survey / Priscilla Salant and Don A. Dillman.` [download] Given that I'm seeing something slightly different, can you verify you still see the issue for this posted code? Otherwise, can you edit your code down at all and still see bad behavior? #11929 First ask yourself `How would I do this without a computer?' Then have the computer do it the same way.	[reply] [d/l] [select]
Re^2: Odd LWP Problem by mdkoehn (Novice) on May 04, 2017 at 17:12 UTC
OK, I've cut out all of the code that does anything. The following, despite doing nothing with the value read from the file, still fails `#! /usr/bin/perl use strict; use warnings; use LWP::Simple; open INFIL, "temp.txt" or die; while (<INFIL>) { #while (<DATA>) { my $html = getprint("https://clio.columbia.edu/) or die; } exit; __DATA__ How to conduct your own survey / Priscilla Salant and Don A. Dillman.` [download] I have a workaround for now -- I dumped all 40,000+ lines to be processed in the DATA section. Still, it would be nice to figure out if this is user error (doubtful at this point), or, if it's a perl-related bug, get it fixed. By the way, here's the perl -V for the cygwin perl: $ perl -V Summary of my perl5 (revision 5 version 22 subversion 3) configuration +: Platform: osname=cygwin, osvers=2.6.1(0.30553), archname=cygwin-thread-multi uname='cygwin_nt-6.3 cygwin 2.6.1(0.30553) 2016-12-16 11:55 x86_64 + cygwin ' config_args='-des -Dprefix=/usr -Dmksymlinks -Darchname=x86_64-cyg +win-threads -Dlibperl=cygperl5_22.dll -Dcc=gcc -Dld=g++ -Accflags=-gg +db -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/mnt/ +share/maint/perl.x86_64/build=/usr/src/debug/perl-5.22.3-1 -fdebug-pr +efix-map=/mnt/share/maint/perl.x86_64/src/perl-5.22.3=/usr/src/debug/ +perl-5.22.3-1 -fwrapv' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='gcc', ccflags ='-DPERL_USE_SAFE_PUTENV -D_GNU_SOURCE -U__STRIC +T_ANSI__ -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-pref +ix-map=/mnt/share/maint/perl.x86_64/build=/usr/src/debug/perl-5.22.3- +1 -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/src/perl-5.22.3=/us +r/src/debug/perl-5.22.3-1 -fwrapv -fno-strict-aliasing -fstack-protec +tor-strong -D_FORTIFY_SOURCE=2', optimize='-O3', cppflags='-DPERL_USE_SAFE_PUTENV -D_GNU_SOURCE -U__STRICT_ANSI__ - +ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/mn +t/share/maint/perl.x86_64/build=/usr/src/debug/perl-5.22.3-1 -fdebug- +prefix-map=/mnt/share/maint/perl.x86_64/src/perl-5.22.3=/usr/src/debu +g/perl-5.22.3-1 -fwrapv -fno-strict-aliasing -fstack-protector-strong +' ccversion='', gccversion='5.4.0', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 +, doublekind=3 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1 +6, longdblkind=3 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', + lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='g++', ldflags =' -Wl,--enable-auto-import -Wl,--export-all-sym +bols -Wl,--enable-auto-image-base -fstack-protector-strong' libpth=/usr/lib libs=-lpthread -lgdbm -ldb -ldl -lcrypt -lgdbm_compat perllibs=-lpthread -ldl -lcrypt libc=/usr/lib/libcygwin.a, so=dll, useshrplib=true, libperl=cygper +l5_22.dll gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags=' --shared -Wl,--enable-auto-import -Wl +,--export-all-symbols -Wl,--enable-auto-image-base -fstack-protector- +strong' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS PERL_DONT_CREATE_GVSV PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_IMPLICIT_CONTEXT PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV PERL_USE_SAFE_PUTENV USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE +_TIME USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API Built under cygwin Compiled at Jan 15 2017 13:11:06 %ENV: PERL5LIB="/home/mdk2126/perl5/lib/perl5" PERL_LOCAL_LIB_ROOT="/home/mdk2126/perl5" PERL_MB_OPT="--install_base "/home/mdk2126/perl5"" PERL_MM_OPT="INSTALL_BASE=/home/mdk2126/perl5" @INC: /home/mdk2126/perl5/lib/perl5/cygwin-thread-multi /home/mdk2126/perl5/lib/perl5 /usr/lib/perl5/site_perl/5.22/x86_64-cygwin-threads /usr/lib/perl5/site_perl/5.22 /usr/lib/perl5/vendor_perl/5.22/x86_64-cygwin-threads /usr/lib/perl5/vendor_perl/5.22 /usr/lib/perl5/5.22/x86_64-cygwin-threads /usr/lib/perl5/5.22 . [download] And here's the same for Strawberry: O:\My Documents\ReCAP Transfer\perl>perl -V Summary of my perl5 (revision 5 version 24 subversion 1) configuration +: Platform: osname=MSWin32, osvers=6.3, archname=MSWin32-x86-multi-thread-64in +t uname='Win32 strawberry-perl 5.24.1.1 #1 Sun Jan 15 20:28:53 2017 +i386' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef useithreads=define, usemultiplicity=define use64bitint=define, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='gcc', ccflags =' -s -O2 -DWIN32 -DPERL_TEXTMODE_SCRIPTS -DPER +L_IMPLICIT _CONTEXT -DPERL_IMPLICIT_SYS -fwrapv -fno-strict-aliasing -mms-bitfiel +ds', optimize='-s -O2', cppflags='-DWIN32' ccversion='', gccversion='4.9.2', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678 +, doubleki nd=3 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1 +2, longdbl kind=3 ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='lo +ng long', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='g++', ldflags ='-s -L"C:\perl\perl\lib\CORE" -L"C:\perl\c\lib" +' libpth=C:\perl\c\lib C:\perl\c\i686-w64-mingw32\lib C:\perl\c\lib\ +gcc\i686-w 64-mingw32\4.9.2 libs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 +-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm + -lversion -lodbc32 -lodbccp32 -lcomctl32 perllibs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdl +g32 -ladva pi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lw +inmm -lver sion -lodbc32 -lodbccp32 -lcomctl32 libc=, so=dll, useshrplib=true, libperl=libperl524.a gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs, dlext=xs.dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-mdll -s -L"C:\perl\perl\lib\CORE" -L"C +:\perl\c\l ib"' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES HAVE_INTERP_INTERN MULTIPLICITY PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS PERL_MALLOC_WRAP PERL_PRESERVE_IVUV USE_64_BIT +_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF Built under MSWin32 Compiled at Jan 15 2017 20:39:08 @INC: C:/perl/perl/site/lib/MSWin32-x86-multi-thread-64int C:/perl/perl/site/lib C:/perl/perl/vendor/lib C:/perl/perl/lib . [download]	[reply] [d/l] [select]
Re^3: Odd LWP Problem by marinersk (Priest) on May 04, 2017 at 19:21 UTC
Just picking nits, because, you know, Perl does: The updated code you posted does fail for me also, but works when I add the closing quote in the `getprint()`call: `my $html = getprint("https://clio.columbia.edu/") or die; ^` [download] Any chance the "still doesn't work" could be a different error this time, and maybe, expecting failure, you didn't check the error message as closely? I know, it's a long shot, but, you know, pick all the nits, right?	[reply] [d/l] [select]
Re^3: Odd LWP Problem by kennethk (Abbot) on May 04, 2017 at 21:53 UTC
It looks like there may be an open ticket on this issue: https://github.com/libwww-perl/libwww-perl/issues/244 Reading what's posted, my gut says they are unrelated, but who knows. Regardless, I would say you probably have enough to give the dev a good bug report. With regard to getting your work done, I agree that the dumb `DATA` block solution is the right one. #11929 First ask yourself `How would I do this without a computer?' Then have the computer do it the same way.	[reply] [d/l]
Re: Odd LWP Problem by marinersk (Priest) on May 04, 2017 at 07:26 UTC
A thought from way out in left field: See if the behavior changes if you switch `-C255`to `-C191`. I'm wondering if you have a locale issue different from choroba and kennethk.	[reply] [d/l] [select]
Re^2: Odd LWP Problem by mdkoehn (Novice) on May 04, 2017 at 17:02 UTC
-C191 didn't make a difference. Since I'm testcasing with non-Unicode at the moment, I removed -C entirely -- still no difference.	[reply]


No such thing as a small change
	PerlMonks