Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

The Perl Compiler (turning perl scripts into binary executables)

by crazyinsomniac (Prior)
on Jul 10, 2001 at 10:32 UTC ( [id://95246]=perlmeditation: print w/replies, xml ) Need Help??

Hello monks.

If you ever wanted to compile a perl script into binary executable, you've probably been told you need to get ActiveState's perl2exe (which ain't free or opensource).

I want to change that.

I've created a sourceforge project i've dubbed the "PerlCompiler" (through helpful suggestions from Jouke and Petruchio -- /tell me if i forgot somebody).

The "description" is:

Do you have a perl script you'd like to distribute in binary executable form?
Then the Perl Compiler is for you, whatever your OS may be, as long as there is a perl port for it (you can run perl on it).

The website resides at http://perlcompiler.sourceforge.net/, where you can find more information about the project (currently not containing much more than this post, but will be the place to go for the Perl Compiler). I am new to sourceforge (and running an open source project), so be patient.

This project is GPL'ed (mening it's open source, and if you make any changes to it, you gotta open source it as well).

It is currently in its planning stage, which is the reason for this meditaton.

Help is needed in every possible way.

In particular, me and Jouke (the two members so far), have no idea what is needed programmatically, besides an interface ;).

Originally, we'll develop it as a console application, adding on a gui once we have a console version that works.

Now I've heard of B::Bytecode, and it is experimental, but it's a starting point. I've also run accross perlcc, but we don't wanna produce c-source from the perl sourcecode, we just wanna produce an executable.

So I figure we gotta gain access to a compiled version of a perl program, preferably directly from perln, whether it be in that machine's native machine code or pure perl bytecode and produce a valid binary executable for that machine.

Keep in mind that the language we're leaning for doing this in is preferably perl, but c++ or java will do fine. I'm a little skeemish about pure c, but hey, if you know how to achieve this, it's all good.

This is an open call for any and all ideas on how to achieve this (any relevant experience always preffered ;).

ps. - please don't just reply saying you think this is a cool idea, I know it is, I just don't know where to start to implement it ;)

update: This will be the only update to this node, short of tyepo's. See what Jouke said, and note his enthusiasm. Let's do it!

 
___crazyinsomniac_______________________________________
Disclaimer: Don't blame. It came from inside the void

perl -e "$q=$_;map({chr unpack qq;H*;,$_}split(q;;,q*H*));print;$q/$q;"

  • Comment on The Perl Compiler (turning perl scripts into binary executables)

Replies are listed 'Best First'.
Re: The Perl Compiler (turning perl scripts into binary executables)
by Jouke (Curate) on Jul 10, 2001 at 10:41 UTC
    OK crazyinsomniac! Great that you've set this up so far. A few remarks:
    • ActiveState has PerlApp, which only runs on WinNT and only creates executables for Win32 platforms
    • IndigoSTAR has perl2exe which runs on multiple platforms and creates executables for multiple platforms
    • perlcc has been written by Malcolm Beattie and should compile into C OR executable, but needs a C-compiler to actually create the binary executable. And it's not stable. Far from that. I've tried it on a Linux box with Perl 5.00503 and it failed to compile a simple script that used a few modules.
    Let's build a stable multiplatform compiler!

    Update:toma wondered if the goal would be to hide the source of the perlscript by compiling it. This would be interesting for commercial use, but it's not the goal as far as I'm concerned.


Jouke Visser, Perl 'Adept'
Using Perl to help the disabled: pVoice and pStory
      ActiveState PDK v3.0 comes with versions of PerlApp for both Windows and Linux / Solaris.

      Of course, it's not free or open source. Good luck!

      Error: Keyboard not attached. Press F1 to continue.
Re (tilly) 1: The Perl Compiler (turning perl scripts into binary executables)
by tilly (Archbishop) on Jul 10, 2001 at 15:50 UTC
Re: The Perl Compiler (turning perl scripts into binary executables)
by BrentDax (Hermit) on Jul 10, 2001 at 10:48 UTC
    Hmm...well, there's the obvious techniques--store the text of the program as a string, extract the bytecode and turn it back into the parse tree on execution, extract the parse tree and recreate it on execution--but those are all possible with current stuff. (Well, the last two, and especially the last one, would be hard, but whatever...) There's also a few questions of the heaviness of the code--do you want to bring the entire interpreter? Maybe you want to have parts of it, e.g. the regular expression engine, excludable with the right switches?

    I think you have to start by saying, "Do I care more about time to develop the project, time to run compiled code, or space the compiled code takes up" and go from there. It'd be relatively easy to write something that just stuck your script in a big ol string in the C source and compiled it, but I wouldn't really recommend that, as all you'd get for the work was something you could grab and drop on a different box with the same architecture.

    As for coders...I'm willing to help. It'll be my first collaborative project and my first work with Perl internals, but if that doesn't scare you off, count me in. :^)

    =cut
    --Brent Dax

    @HPAJ=split("", "rekcaH lreP rentonA tsuJ"); print reverse @HPAJ; #sucky but who cares?
Re: The Perl Compiler (turning perl scripts into binary executables)
by joefission (Monk) on Jul 10, 2001 at 22:00 UTC
    PerlApp runs on Win32, Linux, and Solaris.

    perlapp (and perl2exe) seems to:
    1. Scan the script.
    2. Grab the scanned use and require modules/files found in  @inc.
    3. use some sort of self-extracting zip/tar and tarball perl, the script, associated modules, files, etc.
    4. Run this executable on the host. The self-extractor (after untarring/unzipping into /temp) executes perl script.

    Of course the devil is in the details :-) The version of perl that is tarred has @inc of /temp and the local directory.
    The self-extractor/perl-boot-loader might be the hardest part (then again, I haven't looked for it).

    Are we trying to build a multi-platform compiler, cross-platform compiler, or a free-standing executable of perl scripts?
    The difference (to me) being:
    multi-platform means win32 compiler for win32 executables, linux for linux (I think this is what is being said)
    cross-platform means I can build a linux executable on a win32 machine (this would be crazy, huh?)
    free-standing would be the sleazy method described above. I guess that could be considered multi-platform, but I don't know if it's actually a compiler.

      Multi-platform compiler. The goal is to be able to write a Perl script on a system that has perl, run it through the compiler, and have an executable you can drop into another system running the same OS and processor but not necessarily containing perl. (At least I think so. crazyinsomniac, correct me if I'm wrong...)

      =cut
      --Brent Dax

      @HPAJ=split("", "rekcaH lreP rentonA tsuJ"); print reverse @HPAJ; #sucky but who cares?
      I think you're wrong about the way Perl2Exe and PerlApp work. I think it's for a reason that PerlApp only works on WinNT and not on Win98. Like Activestate states, it's because the Win98 API is lacking something that is needed for the compilation. That means more is involved than just creating a self-extracting thing.
      Besides, while running it does not extract anything anywhere.

      We are trying to create a multiplatform, maybe eventually a crossplatform compiler that creates free-standing executables.
      This means you would not need to have Perl installed on your machine to run the script.

      Jouke Visser, Perl 'Adept'
      Using Perl to help the disabled: pVoice and pStory
        Besides, while running it does not extract anything anywhere.

        I respectfully disagree. The free-standing executable's @inc paths are . and %temp%\scriptname\uniqueid.
        The subdirectory contains perl56.dll and perlapp.dll which looks like the interpreter (based on file size) + some other goodies.

        Update: I didn't mean to imply that all of the script-dependent modules are in the subdirectory. They aren't. Now I wonder what's going on in Linux and Solaris.

        I've seen WinNT errors thrown when a stand-alone script is run without proper permissions(actually, wrong user). The error stated that Perl.exe was causing it. I hope I saved a screen shot of it somewhere...

        You're right Jouke, I don't know exactly how it works. I'm just gues^H^H^H^H reasoning it out. The Win32API that needs NT to build, but not to run has me stumped. Hopefully Win9x/ME will be dead before anyone would want to use the perl monk compiler on it.

        The reason PerlApp only works on NT (not 9x) is the lack of functions for manipulating EXE files. Obviously having OS support for that makes writing any kind of compiler easier!

        Specifically, it works by taking the stub EXE and sticking all the files to bundle as "resources" to a copy of that EXE. I'm pretty sure it's uncompressed, too!

        Instead of using the resource fork as an archive, you could implement it as a tarball concatenated after the stub, as with a typical self-extracting archive. Then you would not need those exotic functions, and the source for the stub would be portable (as opposed to using OS-specific features to read the resources).

        —John

Re: The Perl Compiler (turning perl scripts into binary executables)
by srawls (Friar) on Jul 10, 2001 at 21:09 UTC
    I'm with jouke, I'm very excited about this! I don't know much about how to turn something into a binary executable file, but I'm all up for learning. As a quick question, though, what do you think we should do with evals? That is, would we need the Perl Interpreter in the executable to handle an eval statement?

    Update: Just incase it wasn't clear before: Count me in! I definitely would like to help code this. As for ideas, there are two main routes I can think of. One is to convert the perl code into relavent C code, this is probably a slow solution. The other is to convert it straight to a machine representation, this is probably not a portable solution. I just joined sourceforge, and can't wait to start. (P.S. I'll be gone tommorow for a robotics demonstartion, so if you try to contact me, I'll have to respond on Thursday)

    The 15 year old, freshman programmer,
    Stephen Rawls

      I think we would have to rely heavily on Perl itself. Not as a programminglanguage to write the compiler itself in, but the C-source of Perl to use, or the B::Bytecode or something to get to the backend of the Perl compilation process itself.

      I think the last thing we should do is to rewrite the Perl engine itself!! How could we ever keep up with new versions if we were writing every part ourselves?

      No, re-use what has been done, and write only the last part of the process, namely the creation of the binary executable.

      Jouke Visser, Perl 'Adept'
      Using Perl to help the disabled: pVoice and pStory
      Observation 1: You can let the compiler out if the source doesn't contain the letter 'e'! Observation 2: Normal programs can be written without the letter 'e'. 'E' doesn't count.
Re: The Perl Compiler (turning perl scripts into binary executables)
by Nitsuj (Hermit) on Jul 11, 2001 at 02:25 UTC
    Speaking as someone who has written a compiler before. Let me point you in the direction that you need to be going.

    1. You are not going to do this in PERL, you will rot your brain and go through WAYYY more frustration than is necessary trying to do so. There is already a REALLY nice suite of tools for writing compilers in C. PERL isn't written in PERL ;-)
    2. The first tool you will want to learn about is LEX. Lex generates lexical analyzers.
    3. The second tool that you will want to learn to use is YACC (or Bison). Bison will allow you to form the proper grammars for your language, to write the semantic analyzer, and to allow you to generate intermediate code. Intermediate code will be one of the key factors in making your compiler cross platform. You might even consider bootstrapping whatever you write to GCC, so as to to make this system cross platform that way, it will save you a lot of headaches, as you will not have to re-release for every platform.
    4. These 2 tools are generally used with C. They generate sources that can be linked to each other and to your own source.
    5. If you don't know anything about grammars and automata, you can forget about writing a compiler. Read up on them.
    6. A textbook that you might use to get yourself started is Compilers: Principals, Techniques, and Tools by Aho, Sethi, and Ullman.


    Just Another Perl Backpacker
      I'm not quite sure if you get what we're trying to do. We're not writing a compiler IN Perl, we're writing one FOR Perl. That basically requires us to figure out a way to take Perl's parse tree and somehow encode it into an executable such that we can regenerate it. (crazyinsomniac--I'll be sending you an e-mail soon explaining my idea for this.) The perl source will do all of the parsing for us--we're just parsing its parse trees, so to speak.

      Besides, even if we *were* writing a compiler in Perl, we'd probably use Parse::RecDescent. It's easily powerful enough to parse C, and supposedly the next version will be able to parse Perl.

      =cut
      --Brent Dax

      @HPAJ=split("", "rekcaH lreP rentonA tsuJ"); print reverse @HPAJ; #sucky but who cares?
Re: The Perl Compiler (turning perl scripts into binary executables)
by Cybercosis (Monk) on Jul 11, 2001 at 04:42 UTC
    To solve the eval problem, might I suggest you take a look at `C for ways to implement dynamic compilation.

    ~Cybercosis

    nemo accipere quod non merere

Re: The Perl Compiler (turning perl scripts into binary executables)
by strfry() (Monk) on Oct 17, 2001 at 20:36 UTC

    a few things to note...

    first, perlcc can compile perl source code into a binary:  perlcc perl_source.pl -o perl_bin_executable - and there's no offending C source involved. unfortunately, it's also less than stable (i get terrible errors with it on 5.6.1, when i use map{}, and have had no luck making it play nicely with Socket.pm)

    secondly, if you know a teensy bit of C (or are good at faking it <g>) you can always shove your very own perl interpreter into a .c file, and include the code in a char'd string (see perlman:perlembed for oodles of information on that ((: )

    now, i agree with the many others who have said that compiling perl scripts won't protect the source.. but as far as i could tell, that wasn't what all this was about, was it?

    Being able to write a decent snippet, one-liner, or full- fledged program in perl, and then distribute it to those who -don't- have some/any/all of the following:

    • some perl module or another
    • a perl interpreter! (:
    • any programming knowledge whatsoever
    installed would be, at times, infinitely helpful. i myself have lost track of ho w many times i've wished i had something as simple as a 'comment cutter' (eg.  map{print if not m/^(#|$)/;} <fh>;) on a dos/win32 platform.

    lastly, if you need help on this project, i'd be more than happy to help (assuming the need isn't already filled by the mega-experienced perl adepts and gods who haunt the monastery) (:


    Good Luck,
    strfry()

Re: The Perl Compiler (turning perl scripts into binary executables)
by mattr (Curate) on Jul 12, 2001 at 14:18 UTC
    Is it really going to save time to start from scratch instead of say putting energy into B::CC/perlcc ? I have not used it myself, but it looks like something that could benefit from the community, if Malcolm Beattie is still working on it. Otherwise quite excited, we've needed this for a looong time.
      No-one said we would start from scratch....we may be using something that Malcolm wrote, but his basic idea is to use a c-compiler to create the executables and that's not likely to become our way to do it...

      But who knows....

      Jouke Visser, Perl 'Adept'
      Using Perl to help the disabled: pVoice and pStory
Re: The Perl Compiler (turning perl scripts into binary executables)
by Anonymous Monk on Jul 11, 2001 at 00:52 UTC
    First, let me ask you a question. Why do you need to compile perl scripts into binary format? Second, there is no way to hide your code and eventually smart people whill decipher your code. Just my 2cents on the issue. h.
      As someone who uses perl2exe let me throw in my two cents..
      I create perl programs that work as simple helper apps to some of our internal Windows software. These scripts I compile into an exe format to keep from having to install Perl on every workstation who may need to use it, then having to keep everyone up to date with the perl versions and modules. It is not a matter of 'hiding' the source code as much as it making the program easier to access.
      SO for me it's a matter of keeping my administration overhead low

      On your second note about not being able to hide the internals I agree fully and I don't use the complier to 'secure' anything

      On the idea of this project I would love to see it happen. I can't do much to help other than test (I'm not that great of programmer). From a user perspective I have found the EXEs created by perl2exe tend to be quite large in file size and it would be nice to see smaller exes
      Like I stated in my first reply to this thread, we are trying to compile the source...not to hide it. A compiler is not the same as an obfuscator.

      The main idea (for me) for a Perl compiler is that it enables you to write Perl programs and enable anyone to run it, regardless if they installed a Perl interpreter or not (like I did with pVoice).

      Thanks to OeufMayo and neophyte my program is now compiled for the Win32 platform, which simplifies installation enormously for people who want to use the application instead of co-develop it.

      Jouke Visser, Perl 'Adept'
      Using Perl to help the disabled: pVoice and pStory

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlmeditation [id://95246]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others perusing the Monastery: (3)
As of 2024-04-19 16:00 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found