Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Script stopped working...

by gentoobob (Novice)
on Mar 18, 2018 at 02:18 UTC ( #1211171=perlquestion: print w/replies, xml ) Need Help??

gentoobob has asked for the wisdom of the Perl Monks concerning the following question:

First off, I've never scripted with Perl. I'm a Network Engineer. Script below was not created by me, but by an ex employee years ago. This script will only work on a 2011'ish Debian box. We wanted to migrate the script to a new Oracle linux (redhat) VM server. The Debian box was using Perl 5.10. and the Oracle box is using 5.16. We downloaded the modules that was being used in the script but the script will no longer work on the new linux VM. The script accesses Cisco's Call Manager (phone system), it pulls data off the pages. It then parses it (XML) into two columns, users and extensions, and dumps it into a .CSV file.

When this script is ran on the new linux VM, it gives an error trying to access the URL. I found that it was getting denied for two reasons 1) authentication 2) SSL Cert. So after troubleshooting, I learned that I needed to use LWP::UserAgent, not LWP::Simple (or at least I think thats the case). Once I got the UserAgent set up correctly to log in and ignore the SSL cert, I was able to access the site. Then I learned that something with the XML::Simple wasn't working right on the new linux VM. I keep getting a "Not a GLOB reference" error. I've gone as far as trying to get Data::Dumper to work. I'm at a complete lost at this point. To be honest, I can't seem to grasp this GLOB reference or where in the script is the true problem.

Glob error I get..."Not a GLOB reference at /usr/share/perl5/vendor_perl/XML/SAX/PurePerl/Reader/UnicodeExt.pm line 10."

I do not know if something has changed in the versions of code or modules from the old Debian linux box and the newer Oracle linux VM. Any help would be greatly appreciated.

I will list the original script, then list what I have modified.

---Original (works ONLY on the Debian linux Perl 5.10 box)...

#!/usr/bin/perl use LWP::Simple; use XML::Simple; my $xml = new XML::Simple; my @userdata; $page = 1; #Get Unity subscriber data from cucxnpub one page at a time, 2000 reco +rds to a page while(1) { #Uses ADQuery as the Unity service account my $url = "https://User:password\@cucxnpub/vmrest/users?rowsPerPage= +2000\&pageNumber=$page"; my $content = get($url); die "Error getting $url" unless defined $content; my $data = $xml->XMLin($content); $size = @{$data->{User}}; #If we don't get at least one user end the loop if(@{$data->{User}} < 1) { last; } #Build the userdata array, each entry contains "username,extension" $start = (($page-1) * 2000); for($i=$start;$i<=$start + @{$data->{User}} - 1;$i++) { push(@userdata,"$data->{User}->[$i-$start]->{Alias},$data->{User}- +>[$i-$start]->{DtmfAccessId}"); } $page++; } #Create a timestamp containing year, month, and day ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time +); $year+=1900; $mon++; $timestamp = sprintf("%4d%02d%02d",$year,$mon,$mday); #Dump the results of the unity query to a file open(UNITY,">/usr/scripts/unityLDAP/$timestamp-unity.csv"); for(@userdata) { print UNITY "$_\n"; } close(UNITY);

--Modified by me, I get at it to login to the site but get the GLOB error mentioned above...

use LWP::UserAgent; use LWP::Protocol::https; use XML::Simple; $xml = new XML::Simple; @userdata; $userName = ‘user’; $passwd = ‘password’; $page = 1; #Get Unity subscriber data from cucxnpub one page at a time, 2000 reco +rds to a page while(1) { my $url="https://cucxnpub/vmrest/users?rowsPerPage=2000\&pageNumber= +$page"; $ua = LWP::UserAgent->new(ssl_opts => { SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE, verify_hostname => 0, }); $header = HTTP::Headers->new; $req = HTTP::Request->new(GET => $url); $req->authorization_basic($userName,$passwd); $req->content_type('application/xml'); $content = $ua->request($req); my $data = $xml->XMLin($content); $size = @{$data->{User}}; #If we don't get at least one user end the loop if(@{$data->{User}} < 1) { last; } #Build the userdata array, each entry contains "username,extension" $start = (($page-1) * 2000); for($i=$start;$i<=$start + @{$data->{User}} - 1;$i++) { push(@userdata,"$data->{User}->[$i-$start]->{Alias},$data->{User}- +>[$i-$start]->{DtmfAccessId}"); } $page++; } #Create a timestamp containing year, month, and day ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time +); $year+=1900; $mon++; $timestamp = sprintf("%4d%02d%02d",$year,$mon,$mday); #Dump the results of the unity query to a file open(UNITY,">/usr/scripts/unityLDAP/$timestamp-unity.csv"); for(@userdata) { print UNITY "$_\n"; } close(UNITY);

Replies are listed 'Best First'.
Re: Script stopped working...
by hippo (Chancellor) on Mar 18, 2018 at 08:20 UTC
    $content = $ua->request($req); my $data = $xml->XMLin($content);

    This is a bug. $content in this context is an HTTP::Response object, not something which can be blindly passed to XMLin() successfully. You first need to extract the XML from it. eg. by using decoded_content. I'd therefore refrain from calling the response $content in the first place.

    Try replacing those two lines with:

    my $response = $ua->request ($req); my $content = $response->decoded_content; my $data = $xml->XMLin ($content);

    That ought to get you a little bit farther along at least.

      Yep! Wow. I figured I was missing some kind of decoding which is why I was trying to mess around with Data::Dumper but I couldn't get anywhere with it. This works! Thank you so much. Ive been trying to fix this script for 2 weeks now and putting serious time into. Thank you again.

Re: Script stopped working...
by Marshall (Canon) on Mar 18, 2018 at 06:45 UTC
    Evidently HTTPS stuff changed with LWP version 6. I read Crypt-SSLeay.

    I agree with NetWallah that you should be using both strict and warnings. This will spot compile and run time errors. With strictures, each variable has to be declared at first use with "my". I did some of that below to get you started. The use of a while(1) as a loop structure and burying the exit condition in the body is in my opinion a bad idea. More common would be to have the looping exit condition appear right as the loop starts. However, I didn't change that. I did clean up a bit of this $size stuff although I guess that should have been called $n_users? This C-style for loop is a bit weird, but I didn't change that either.

    The first step is to clean up any "my" declarations that I might have missed and get this thing to compile cleanly and start running. We can help with any obscure error messages. Then, the next step is to make sure that you are actually getting the first page (i.e. that you ua modifications are working). I added a "die" message if $content is not defined. Stuffing an undefined value into the XML gizmo could cause the GLOB error that you are seeing - maybe. See below and un-comment the print to see the $content and give us enough of it for us to get the idea of what that is (probably don't need the whole thing). Hope this helps.

    Update: See the suggestion by hippo at Re: Script stopped working.... Hippo is correct. The response is an object, not a simple scalar. I modified my efforts below with his suggestion and added in the right kind of die message showing the response->status if the req did not succeed. The code doesn't do re-tries but I don't think you need to do that. What I remember from my last LWP effort in 2016, I saw a transient error maybe every 3,000 requests and I implemented re-tries because I had to do a lot of requests. I think it is worthwhile to check if the request succeeded, but I don't recommend mucking with this code any more than necessary. Note: the right place to end the loop may not be if($size<1){} depends upon what the server does when you "run out of valid pages".

    use strict; use warnings; use LWP::UserAgent; use LWP::Protocol::https; use XML::Simple; my $xml = new XML::Simple; my @userdata; my $userName = ‘user’; my $passwd = ‘password’; my $page = 1; #Get Unity subscriber data from cucxnpub one page at a time, 2000 reco +rds to a page while(1) { my $url="https://cucxnpub/vmrest/users?rowsPerPage=2000\&pageNumber= +$page"; my $ua = LWP::UserAgent->new(ssl_opts => { SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE, verify_hostname => 0,}); my $header = HTTP::Headers->new; my $req = HTTP::Request->new(GET => $url); $req->authorization_basic($userName,$passwd); $req->content_type('application/xml'); # changed this line in update re: [hippo]'s post # this ain't right #my $content = $ua->request($req) or die "unable to get content for +$url $!"; ### need defined content my $response = $ua->request ($req); #response is an object if ( !($response->is_success)) { die $response->status_line; } my $content = $response->decoded_content; my $data = $xml->XMLin ($content); #print "$content\n"; ### un-comment for to see print of xml documen +t #exit; my $size = @{$data->{User}}; #If we don't get at least one user end the loop if($size < 1) { last; ### <- THIS IS HOW LOOP ENDS } #Build the userdata array, each entry contains "username,extension" my $start = (($page-1) * 2000); my $i; for($i=$start;$i<=$start + $size - 1;$i++) { push(@userdata,"$data->{User}->[$i-$start]->{Alias},$data->{User}- +>[$i-$start]->{DtmfAccessId}"); } $page++; } #Create a timestamp containing year, month, and day my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(t +ime); $year+=1900; $mon++; my $timestamp = sprintf("%4d%02d%02d",$year,$mon,$mday); #Dump the results of the unity query to a file open(UNITY,">/usr/scripts/unityLDAP/$timestamp-unity.csv"); for(@userdata) { print UNITY "$_\n"; } close(UNITY);

      Yes. Like I was telling NetWalla I do not want to run bad code, especially when I'm purely amateur at Perl scripting. I will definitely look over everything in more detail Monday morning. I really appreciate your help and will be looking into more resources and learning tools. I kind of wondered if something had changed in one of the modules. Makes sense. Thank you so much again. I've been beating my head in for about 2 weeks and a lot of hours. haha

        As compared with the average complete amatuer question, you get an A+++.
        Your question was clear and it was immediately obvious to me and others that you have some brain cells that are working hard. PerlMonks is a great place to learn, but this only works if you also work. You are working hard and I figure that the result will be great!

        I think you are very close to "working code".
        There are some fine points to be dealt with and some obvious things like "don't push when you can print!". The code creates an array of strings and then later the code loops over that array to print the strings. No need to do that.. just barf it out right away (e.g. print)

Re: Script stopped working...
by NetWallah (Canon) on Mar 18, 2018 at 04:01 UTC
    I'm surprised you got a response at all.

    Your declaration of "my $url" is out-of-scope by the time you attempt to get $content:

    while(1) { my $url="https://cucxnpub/vmrest/users?rowsPerPage=2000\&pageNumber= +$page"; $ua = LWP::UserAgent->new(ssl_opts => { SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE, verify_hostname => 0, }); #<<<< $url is OUT of scope $header = HTTP::Headers->new; $req = HTTP::Request->new(GET => $url); #IF you had "use strict", th +is would show an error.
    Please do your self a favour and add this to the top of you script:
    use strict; use warnings;
    I also dont know why the "while (1)" is being used.
    You should be setting the user agent up only one time, not in an indifinete loop.

    I don't know how you are able to exit that loop.

    If you are sure you have got past the URL issue, and are able to fetch $content, we can work on the remainder of the script, if you pshow us a sample of the $content.

    Depending on what you are pulling data from, Cisco::IPPhone may be relevant.

    Update:Thanks , gentoobob,syphillis and Marshall for pointing out that I mistook a mis-indentation for $url being out of scope. It is not.

                    Python is a racist language what with it's dependence on white space!

      Thats interesting! I will definitely do that first thing Monday morning. I don't want to run bad code on a live environment. We are just trying to move this script over off a server going bad and the guy who made it, isn't with us anymore. Guess I might need to buy a Perl book too. haha. Thanks again!

Re: Script stopped working...
by poj (Abbot) on Mar 18, 2018 at 15:18 UTC

    It looks very likely hippo's solution will solve your GLOB error but there are 2 further issues to be aware of. When you add use strict you may encounter the error

    Can't use an undefined value as an ARRAY reference at ...

    on this line.

    my $size = @{$data->{User}};

    This might occur on the last request when no more records are in the response. A fix would be

    my $size = @{$data->{User} || []};

    The other error with XML::Simple is more subtle and will only appear it there is one record in the response. In this case the error would be

    Not an ARRAY reference at ...

    The fix is to use ForceArray on the User element

    my $xml = new XML::Simple(ForceArray => ['User']);

    Example with strict

    poj

      Ok great! I will def look into that. Thank you for your response. I greatly appreciate it.

        From various replies:

        Ive been trying to fix this script for 2 weeks now and putting serious time into.
        I greatly appreciate it.
        Hmm... If only there were some way to show your appreciation...   (hint, hint...)


        Give a man a fish:  <%-{-{-{-<

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1211171]
Approved by davido
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (1)
As of 2020-12-05 06:07 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How often do you use taint mode?





    Results (63 votes). Check out past polls.

    Notices?