Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

PerlIO::Layers 0.011 fails to build in macOS 10.13.3

by perlboy_emeritus (Scribe)
on Mar 25, 2018 at 22:05 UTC ( [id://1211706]=perlquestion: print w/replies, xml ) Need Help??

perlboy_emeritus has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks, I'm trying to build PDL from CPAN on:
This is perl 5, version 18, subversion 2 (v5.18.2) built for darwin-th +read-multi-2level (with 2 registered patches, see perl -V for more detail)
in macOS 10.13.3. The PDL build fails because PerlIO::Layers 0.011 build fails. When I attempt to compile Layers.c stand-alone, to see what the compiler is seeing and doing I get:
$ cc -I/System/Library/Perl/5.18/darwin-thread-multi-2level/CORE -DXS_ +VERSION="0.011" -DVERSION="0.011" -c -arch x86_64 -arch i386 -g -pipe + -fno-common -DPERL_DARWIN -fno-strict-aliasing -fstack-protector -Os + -o lib/PerlIO/Layers.o lib/PerlIO/Layers.c lib/PerlIO/Layers.c:249:5: error: expected ')' XS_VERSION_BOOTCHECK; ^ /System/Library/Perl/5.18/darwin-thread-multi-2level/CORE/XSUB.h:328:6 +1: note: expanded from macro 'XS_VERSION_BOOTCHECK' Perl_xs_version_bootcheck(aTHX_ items, ax, STR_WITH_LEN(XS_VERSION +)) ^ <command line>:1:20: note: expanded from here #define XS_VERSION 0.011 ^ lib/PerlIO/Layers.c:249:5: note: to match this '(' /System/Library/Perl/5.18/darwin-thread-multi-2level/CORE/XSUB.h:328:4 +8: note: expanded from macro 'XS_VERSION_BOOTCHECK' Perl_xs_version_bootcheck(aTHX_ items, ax, STR_WITH_LEN(XS_VERSION +)) ^ /System/Library/Perl/5.18/darwin-thread-multi-2level/CORE/handy.h:348: +26: note: expanded from macro 'STR_WITH_LEN' #define STR_WITH_LEN(s) ("" s ""), (sizeof(s)-1) ^ 1 error generated.
The compiler is:
$ file `which cc` /usr/bin/cc: Mach-O 64-bit executable x86_64 $ cc -v Apple LLVM version 9.0.0 (clang-900.0.39.2) Target: x86_64-apple-darwin17.4.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin
The Build file seems to be expanding the XS_VERSION_BOOTCHECK macro correctly as -DXS_VERSION="0.011" is being passed as a quoted string. Any ideas, as this is a deal-killer for building PDL from sources? As of now the latest binary build for macOS of PDL will not execute on 10.13.3, probably because it needs to be recompiled with the 10.13 compiler and libraries.

Thanks for your help.

Replies are listed 'Best First'.
Re: PerlIO::Layers 0.011 fails to build in macOS 10.13.3
by syphilis (Archbishop) on Mar 26, 2018 at 00:39 UTC
    It's rather odd.
    AFAICT, the XS_VERSION_BOOTCHECK macro (in XSUB.h) is the same for all systems, as is also the case for the STR_WITH_LEN macro (in handy.h). And both macros are correctly stated in the output you provided.

    If it were me, I'd try removing the -DXS_VERSION="0.011" argument from the command that's producing the error.
    I don't know if that will allow the build to succeed, but I do know that it will avoid the error you're getting (because XS_VERSION_BOOTCHECK is a noop if XS_VERSION is not defined).

    AIUI, defining XS_VERSION to VERSION allows a mismatch of XS component and Perl component to be detected at boot. Otherwise, such a mismatch will lead to confusing runtime errors or crashes.
    So, if my suggested approach does succeed, you lose the re-assurance of that check at boot.

    <afterthought>
    Given the nature of this error, I'm thinking it should occur every time you build a module that defines XS_VERSION prior to the inclusion of XSUB.h (which is what is happening here).
    I don't think there are a lot of modules that do that, though I haven't checked on that. (Correction ... seems to be defined for modules containing XS files.)
    I notice that XS_VERSION_BOOTCHECK has changed since perl-5.18 and no longer directly calls STR_WITH_LEN.
    Perhaps this means that updating to perl 5.26 would avoid the issue you've struck.
    </afterthought>

    Cheers,
    Rob
      Thanks Rob, for reply and suggestion. I think Perl v5.18.2 in my macOS is hopelessly broken, at least for what I want to do. There have been other reports of issues with Perls built by Apple, as in:

      https://groups.google.com/forum/#!msg/perl.perl5.porters/mKhKYbPqrmQ/2Z7rAsyJOsMJ

      Notice Hafen McCormick's reference to a 'diagnostic wrapper', presumably a Perl shared library built from an .xs file that would appear to have shipped with the writer's Perl distro from Apple. That wrapper may be mangling the output from the STR_WITH_LEN macro. I am giving up on Perl from Apple, especially in High Sierra, given the huge number of bugs reported by HS users, particularly the almost totally broken backward support for USB 3.0. For example, I have an On-The-Go external enclosure I use to clone/backup my mac with Carbon Copy Cloner. I can clone to a drive in that enclosure but I can't boot from it. If my internal SSD were to fail, I could move the drive from the enclosure to inside the mac and boot from it while waiting for a replacement SSD. Then, clone to the SSD in the enclosure and reverse the process. Apple has yet to respond to my bug report or to the numerous others from USB 3.0 users whose backups are hosed. No point in shopping for/buying a new enclosure since all with firmware prior to High Sierra are probably broken and their seller's don't know it and/or can't or won't certify it as bootable. But I digress...

      I looked at several Perls I have access to and as you reported, all have identical XSUB.h and handy.h headers, at least as they relate to macro XS_VERSION_BOOTCHECK, which means the problem must be something either compiled into my Perl or a shared library.

      I installed Homebrew and downloaded Perl 5.26.1 and am in the process of configuring my environment to use it instead of Apple's Perl. But, if it were possible to learn that 1) 'yes', such a 'diagnostic wrapper' does exist, and 2) can it simply be removed to make Apple's Perl behave as expected, I would try that. I hate to give up on a problem without knowing why it's a problem. How can I learn whether my Perl is so configured?

      Damn, something just occurred to me. Apple does not want macOS users to install additional bits in Apple protected directories, where Perl is installed, so, deliberately hosing the XSUB build process is one way to do that, and then ignore all bug reports related to it. Even Apple recommends installing Perl with brew. I am not a conspiracy theorist but dealing with Apple has made me very paranoid :-(

Re: PerlIO::Layers 0.011 fails to build in macOS 10.13.3
by bliako (Monsignor) on Mar 26, 2018 at 13:39 UTC
    The problem is when macro:
    /* concatenating with "" ensures that only literal strings are accepte +d as * argument */ #define STR_WITH_LEN(s) ("" s ""), (sizeof(s)-1)

    is passed a non-literal string (i.e. something which is not quoted, for example the XS_VERSION => 0.011 (via XS_VERSION_BOOTCHECK) which is just a naked 0.011 . Actually this is mentioned in the comments before the macro def (see above code).

    By double-quoting contents of -DVERSION and -DXS_VERSION, the command line you pasted succeeds for me:

    -DVERSION="\"0.011\"" -DXS_VERSION="\"0.011\""
    The complete command line:
    cc -I/opt/perlbrew/perls/perl-5.18.0/lib/5.18.0/darwin-2level/CORE -DV +ERSION="\"0.011\"" -DXS_VERSION="\"0.011\"" -c -fno-common -DPERL_DAR +WIN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include + -I/opt/local/include -arch x86_64 -framework System -Wa,--noexecstac +k -I/opt/local/include -I/opt/X11/include -O3 -o lib/PerlIO/Layers.o + lib/PerlIO/Layers.c

    Of course I am not sure if "causes no error" or "succeeds" is the same as "works as intended". But an unquoted 0.011 can be found in the resultant object file:

    $ strings lib/PerlIO/Layers.o 0.011 v5.18.0 ...
    bliako
      The problem is when macro ... is passed a non-literal string

      Good catch.
      I had seen the comments in handy.h, and noted the absence of quotes in perlboy_emeritus' build - but still failed to put it all together properly.
      (I had myself thinking that the strange ("" s "") construct must have been magically enforcing the requisite stringification but, of course, it merely throws the error if s is not a string.)

      So the bug is in the version of Module::Build that perlboy_emeritus has installed - and any perl extension that is built with ExtUtils::MakeMaker (instead of Module::Build) will be free from this issue.
      The version of Module::Build that I was using (on Windows) is 0.4214, which was fine - so I guess another alternative might have been to simply update Module::Build to a later version.

      Here's a little C demo that I ran just to satisfy my limited understanding:
      C:\_32\C>type try.c #include <stdio.h> int main (void) { #define FOO(s) printf("%s\n", "" s ""); FOO(VER); return 0; } C:\_32\C>gcc -o try.exe -DVER="\"0.1\"" try.c C:\_32\C>try 0.1 C:\_32\C>gcc -o try.exe -DVER=0.1 try.c try.c: In function 'main': try.c:5:2: error: expected ')' before numeric constant
      Cheers,
      Rob

        with your little c-program:

        gcc -DVER="0.1" -E try.c

        does the preprocessor pass and expands FOO to:

        printf("%s\n", "" 0.1 "")

        and

        gcc -DVER="\"0.1\"" -E try.c

        expands it to:

        printf("%s\n", "" "0.1" "")

        so, whereas

        printf("%s\n", 0.1)

        would have caused just a warning at compile time and a crash later,

        printf("%s\n", "" 0.1 "")

        causes an error right away

        and
        printf("%s\n", "" "0.1" "")

        is legitimate.

        bliako
      Bravo bliako :-) However, the behavior of that module's build process is still bizarro. I went into ./_build and grepped for '0.011' and found file build_params to contain:
      'dist_abstract' => 'Querying your filehandle\'s capabilities', 'dist_version' => '0.011', 'build_bat' => 0, 'dynamic_config' => 1, 'c_source' => undef, 'extra_compiler_flags' => [], 'verbose' => undef, 'build_class' => 'Module::Build', 'quiet' => undef, 'install_sets' => {}, 'use_rcfile' => 1, '_added_to_INC' => [ '/Users/carly/perl5/lib/perl5/5.18.2/darwin-thread-multi-2level' +, '/Users/carly/perl5/lib/perl5/5.18.2', '/Users/carly/perl5/lib/perl5/darwin-thread-multi-2level', '/Users/carly/perl5/lib/perl5', '/Users/carly/src/perl5/lib' ],
      among other expressions. This is obviously not a hand-generated file. So, I double quoted that version number => "0.011" and then did 'perl ./Build.PL' and then Build and 'Build test'. Like magic, all was well, all tests were successful but... get this, the module Module::Build as in:
      # This should have just enough arguments to be able to bootstrap the r +est. my $build = Module::Build->resume ( properties => { config_dir => '_build', orig_dir => $orig_dir, }, ); $build->dispatch;
      reset that double-quoted string to '0.011'. Man, talk about perverse. I poked Leon Timmerman's, the author/maintainer of PerlIO::Layers with this issue but have yet to hear back. How do we alert all of the Perl-heads who might be inadvertently creating this problem?

        As <<syphilis>> said:

        I notice that XS_VERSION_BOOTCHECK has changed since perl-5.18 and no longer directly calls STR_WITH_LEN.

        If I were you I would upgrade my perl and never think about it again.

        In OSX something that really works for me well is:

        1. perlbrew (note Homebrew and perlbrew are not the same thing, perlbrew allows you to select which version of perl to use and also downloads, installs and uses any version of perl you instruct it to),
        2. cpanm for installing perl modules,
        3. important: if using perlbrew, shebang should now be: #!/usr/bin/env perl

        #1 makes a lot of difference and definetely pokes a wooden pole in the chest of apple's perl.

        glad you found a solution.

Re: PerlIO::Layers 0.011 fails to build in macOS 10.13.3
by Haarg (Priest) on May 08, 2022 at 10:45 UTC

    This is an old post, but the real problem with the standalone compile you tried to do is that you used the wrong command, because you were mislead by Module::Build. When running ./Build, it prints out a command line it is using to compile the c files, but it prints it out incorrectly. When it runs the command, it bypassed the shell, which means that the quotes it shows are actually passed along to cc. As a shell command, it would be running the equivalent of:

    cc -I/System/Library/Perl/5.18/darwin-thread-multi-2level/CORE "-DXS_V +ERSION=\"0.011\"" "-DVERSION=\"0.011\"" -c -arch x86_64 -arch i386 -g + -pipe -fno-common -DPERL_DARWIN -fno-strict-aliasing -fstack-protect +or -Os -o lib/PerlIO/Layers.o lib/PerlIO/Layers.c

    This means that the errors you saw in your standalone compile don't reflect whatever errors there were in a normal compile of the module. Any further diagnosis would require the output of a normal ./Build.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1211706]
Approved by kcott
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others goofing around in the Monastery: (None)
    As of 2024-04-25 01:45 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      No recent polls found