Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

pack() returns an unusable string

by syphilis (Bishop)
on May 26, 2021 at 13:39 UTC ( #11133064=perlquestion: print w/replies, xml ) Need Help??

syphilis has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

This post arises from an XY problem (for which I already have a very serviceable, albeit inferior, PQ solution).
The following demo script works fine when I set $template to 'd' or 'd<' or 'd>':
# test.pl use warnings; $template = 'd<'; $nv = 2.4; $p = pack $template, $nv; $s = "'$p'"; system $^X, '-wle', "print unpack('H*', $s);";
You can run that on any recent perl, and, for that given template, it should correctly output 3333333333330340.

The problem arises when I change the template to 'D' or 'D<' or 'D>'.
Of course, with the capitalized "D" templates, we're dealing with long doubles. And if you're running perl-5.32.x or earlier, these "D" templates are disallowed unless perl's nvtype is "long double".
However, if you're running perl-5.34.0, then the "D" templates are allowed irrespective of nvtype, thanks to the wonders of modern insanity.
(It's quite sane that the "D" templates now work on -Dusequadmath builds ... but I'm still aghast that they now also work on perls where nvtype is double.)

So ... if you have one of those permitted perl configurations, and you change the template to one of the 3 permitted "D" forms, you'll see the script fail, with the diagnostic warning:
Can't find string terminator "'" anywhere before EOF at -e line 1.
I wonder why that happens ? Is it because of the embedded NULLs in $p ?
More importantly, how do I get to pass that string $p to the system command so that when the template is set to 'D<' the script successfully outputs the expected (and correct) 9a999999999999990040000000000000 (or 00999999999999990040000000000000 if perl is 5.34.0 && nvtype is double).

I'm hitting the same issue on both Windows 7 and Ubuntu-20.04.

Cheers,
Rob

Replies are listed 'Best First'.
Re: pack() returns an unusable string
by choroba (Archbishop) on May 27, 2021 at 14:22 UTC
    Your problem is unrelated to pack and can be easily demonstrated without it:
    my $p = "X\x00Y"; system $^X, '-wE', "say '$p'";

    The problem is the zero byte in an argument (at least on Linux).

    map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
Re: pack() returns an unusable string
by kcott (Bishop) on May 27, 2021 at 08:45 UTC

    G'day Rob,

    I'm not sure if this helps, but it seems to reproduce what you're describing on a different platform: Cygwin 3.2.0 on Win10. I kept most of your posted code as is; I added some info up front; and, I looped through all of the templates on both 5.32 and 5.34.

    $ cat pm_11133064_pack_test.pl use warnings; print "OS: $^O; Perl: $^V;\n", `perl -V:usequadmath`, `perl -V:nvtype`, "\n"; my @templates = qw{d d< d> D D< D>}; for $template (@templates) { print "TEMPLATE: $template\n"; $nv = 2.4; $p = pack $template, $nv; $s = "'$p'"; system $^X, '-wle', "print unpack('H*', $s);"; }
    $ perl pm_11133064_pack_test.pl OS: cygwin; Perl: v5.32.0; usequadmath='undef'; nvtype='double'; TEMPLATE: d 3333333333330340 TEMPLATE: d< 3333333333330340 TEMPLATE: d> 4003333333333333 TEMPLATE: D Invalid type 'D' in pack at pm_11133064_pack_test.pl line 11.
    $ perl pm_11133064_pack_test.pl OS: cygwin; Perl: v5.34.0; usequadmath='undef'; nvtype='double'; TEMPLATE: d 3333333333330340 TEMPLATE: d< 3333333333330340 TEMPLATE: d> 4003333333333333 TEMPLATE: D Can't find string terminator "'" anywhere before EOF at -e line 1. TEMPLATE: D< Can't find string terminator "'" anywhere before EOF at -e line 1. TEMPLATE: D> Can't find string terminator "'" anywhere before EOF at -e line 1.

    — Ken

      I'm not sure if this helps ...

      Heh ... it helps ... but only in that it reaffirms what I already knew ;-)

      I'll elaborate a little on how this all came about.
      As I mentioned in the opening post, with perl-5.34.0, pack's 'D' templates are allowed on all builds irrespective of whether $Config{nvtype} is 'double' or 'long double' or '__float128'.
      In earlier perl versions the 'D' templates were supported only if $Config{nvtype} was 'long double'.

      With perl-5.34.0 and nvtype of 'long double' we get:
      C:\>perl -wle "print unpack 'H*', pack 'D', 2.4;" 9a999999999999990040000000000000
      whereas with perl-5.34.0 and nvtype of double, the same command yields a slightly different result:
      C:\>perl -wle "print unpack 'H*', pack 'D', 2.4;" 00989999999999990040000000000000
      That's pretty much as I expected because the string being unpacked in the first one-liner is different to the string being unpacked in the second one-liner.
      But then I wanted to see what the result would be if the string created on the long double build (in the first one-liner) was fed to the 'double' build (in the second one-liner).
      Would the result be 9a999999999999990040000000000000 or 00989999999999990040000000000000 ?
      (I was quite sure it would be the former, but I knew that if I didn't check then I'd get it wrong.)

      So I thought I'd do something cute:
      I decided that, on my 'long double' build, I would run the following script:
      use warnings; $template = 'D<'; $nv = 2.4; $p = pack $template, $nv; $s = "'$p'"; # Now pass $p to the 'double' build of perl. $double_perl = "C:/perl-5.34.0/bin/MSWin32-x64-multi-thread/perl.exe"; system $double_perl, '-wle', "print unpack('H*', $s);";
      But, as already demonstrated, that doesn't DWIM.

      In the end, I took the clunky approach of running 2 scripts.
      Firstly, on the 'long double' build of perl-5.34.0 I ran the following script that would print the string returned by pack() to a file:
      use warnings; $p = pack("D", 2.4); open WR, '>', 'packstr.txt' or die "Opening: $!"; binmode(WR); print WR $p; close WR or die "Closing: $!";
      Then, on the 'double' build of perl-5.34.0, I ran a script that passed the string that was saved in packstr.txt to unpack():
      use warnings; open RD, '<', 'packstr.txt' or die "Opening: $!"; binmode(RD); $p = <RD>; close RD or die "Closing: $!"; print unpack("H*", $p);
      That script output 9a999999999999990040000000000000.
      (Hmph ... as if it was ever going to do anything else ...)

      It just would have been so much cleaner and simpler if I could have got the first approach to work.

      Cheers,
      Rob
Re: pack() returns an unusable string
by tybalt89 (Prior) on May 26, 2021 at 18:37 UTC

    From your warning message, I would guess that your packed string '$p' has either a single quote "'" in it or an escape or $ or @ at the end that eats the closing single quote. Try looking at a dumped wersion of $p.

      I would guess that your packed string '$p' has either a single quote "'" in it or an escape or $ or @ at the end that eats the closing single quote

      I should add that there's nothing special about the given NV value of 2.4.
      The behaviour I'm seeing is happening for all NV values that I've tested.

      To get a dump of $s:
      # test.pl use warnings; use Devel::Peek; $template = 'D<'; $nv = 2.4; $p = pack $template, $nv; $s = "'$p'"; Dump $s; system $^X, '-wle', "print unpack('H*', $s);";
      That outputs the following (which does, at least, contain an "@"):
      C:\_32\pscrpt>perl test.pl SV = PV(0x4bcf98) at 0x571248 REFCNT = 1 FLAGS = (POK,pPOK) PV = 0x4b7e88 "'\232\231\231\231\231\231\231\231\0@\0\0\0\0\0\0'"\0 CUR = 18 LEN = 20 Can't find string terminator "'" anywhere before EOF at -e line 1.
      But if I change the value of $nv to 2.4e100, I get:
      C:\_32\pscrpt>perl test.pl SV = PV(0x3ecf98) at 0x383348 REFCNT = 1 FLAGS = (POK,pPOK) PV = 0x3e7e88 "'\vL\177v\257zb\206\264>\0\0\0\0\0\0'"\0 CUR = 18 LEN = 20 Can't find string terminator "'" anywhere before EOF at -e line 1.
      Interestingly, changing the value to 3.1e-100 results in a different warning:
      C:\_32\pscrpt>perl test.pl SV = PV(0x26cf98) at 0x5311f8 REFCNT = 1 FLAGS = (POK,pPOK) PV = 0x267e88 "'9\302\271\243\"\211\224\255\264>\0\0\0\0\0\0'"\0 CUR = 18 LEN = 20 Use of uninitialized value $_ in print at -e line 1.
      Perhaps something to do with the internal double quote.

      For a value of 13.16e-100, I get:
      C:\_32\pscrpt>perl test.pl SV = PV(0x38cf98) at 0x4c51b8 REFCNT = 1 FLAGS = (POK,pPOK) PV = 0x387e88 "'\245^,\356\266\0208\270\266>\0\0\0\0\0\0'"\0 CUR = 18 LEN = 20 Can't find string terminator "'" anywhere before EOF at -e line 1.
      Is there some way to pass the string returned by pack() to that system command ?
      I assume that it ought to be possible, and that I'm just doing something dumb.

      Cheers,
      Rob

        And here's a version with Data::Dumper and some config vars that also works.

        #!/usr/bin/perl -l use strict; # https://perlmonks.org/?node_id=11133064 use warnings; my $template = 'd<'; my $nv = 2.4; $nv = 4135; # NOTE errors for this number my $p = pack $template, $nv; use Data::Dumper; $Data::Dumper::Terse = $Data::Dumper::Useqq = 1; my $s = Dumper $p; print ">$s<\n"; system $^X, '-wle', "use strict; print unpack('H*', $s);";

        Outputs:

        >"\0\0\0\0\0'\260\@" < 000000000027b040

        note that it adds a newline, but perl doesn't care.

        This seems to work :)

        #!/usr/bin/perl -l use strict; # https://perlmonks.org/?node_id=11133064 use warnings; my $template = 'd<'; my $nv = 2.4; $nv = 4135; # NOTE errors for this number my $p = pack $template, $nv; use Data::Dump qw(pp); my $s = pp $p; print ">$s<\n"; system $^X, '-wle', "use strict; print unpack('H*', $s);";

        Outputs:

        >"\0\0\0\0\0'\xB0\@"< 000000000027b040

      Guessing is sometimes fine, but the OP gave an entire and complete test script which you can examine and even run. If I make a small modification to the test script given we can test the theory:

      # test.pl use warnings; $template = 'd<'; $nv = 2.4; $p = pack $template, $nv; $s = "'$p'"; print ">$s<\n"; system $^X, '-wle', "print unpack('H*', $s);";

      prints:

      >'333333@'< 3333333333330340

      running a 64 bit build of 5.32. Note that there is an ETX character between the final 3 and the @ character, which, when you think about it, is the string that the unpack is showing us so for the success case there is a trailing @ and it isn't swallowing the trailing '.

      Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond

        OP said error only happens on D<, not d<, which doesn't run on my machine.

        However, here is a number for which the error occurs on 5.32 with d<

        #!/usr/bin/perl -l use strict; use warnings; my $template = 'd<'; my $nv = 2.4; $nv = 4135; # NOTE errors for this number my $p = pack $template, $nv; my $s = "'$p'"; print ">$s<\n"; system $^X, '-wle', "print unpack('H*', $s);";

        Outputs:

        >''@'< Can't find string terminator "'" anywhere before EOF at -e line 1.

        which shows that my guess that the pack produces a single quote character is plausible.

Re: pack() returns an unusable string
by ikegami (Pope) on May 29, 2021 at 03:08 UTC

    Parameters can't contain the NUL character. Your so-called working code breaks if you change 2.4 to 2.5 (which generates at least one a NUL byte).

    I recommend that you pass the hex of the packed string. You could also use base64, JSON, etc, etc, etc.


    Furthermore, you have a code-injection bug. "'$p'" is not an appropriate way to generate a Perl string literal. $p could easily contain byte 0x27 (single quote). A trailing byte 0x5C (backslash) would also be problematic.

    I recommend that you pass the value as an argument.


    use strict; use warnings; my $template = 'd<'; my $nv = 2.5; my $p = pack $template, $nv; system $^X, '-wle', 'print $ARGV[0]', unpack('H*', $p);

    Seeking work! You can reach me at ikegami@adaelis.com

      I recommend that you pass the value as an argument

      What I was wanting to do was to pass the actual string as the argument.
      It didn't really occur to me that doing that would be such a problem. I thought that people would be able to show me that this could be achieved with little fuss. (I was wrong ;-)
      What I was interested in doing is easily demonstrated in the following script, where I've used the code that you supplied, but have replaced the last line with 2 lines:
      use strict; use warnings; my $template = 'D<'; my $nv = 2.4; my $p = pack $template, $nv; # I'm running this script on perl-5.34.0 # with nvtype of 'long double'. # I want to see the result of perl-5.34.0 # with nvtype of 'double' unpacking $p : my $perl = "C:/perl-5.34.0/bin/MSWin32-x64-multi-thread/perl.exe"; system $perl, '-wle', 'print unpack("H*", $p)';
      If that DWIMmed, it would output :
      9a999999999999990040000000000000
      but, of course, all that it emits is the warning:
      Use of uninitialized value $_ in print at -e line 1.
      Sure, I could take a leaf out of your book and alter my last line to:
      system $perl, '-wle', 'print $ARGV[0]', unpack('H*', $p);
      But that unpack('H*', $p) is being evaluated by the 'long double' build of perl.
      I wanted to see what happened when it was evaluated by my 'double' build of perl-5.34.0.
      I've since accomplished this by other means.

      Cheers,
      Rob

        unpack "H*" hasn't changed, so what you're saying makes no sense. Encode the string in the parent (say using unpack "H*"), then recreate the original in the child if so you desire (using pack "H*").

        Seeking work! You can reach me at ikegami@adaelis.com

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11133064]
Approved by philipbailey
Front-paged by stevieb
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (5)
As of 2021-12-02 10:58 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    R or B?



    Results (18 votes). Check out past polls.

    Notices?