http://qs321.pair.com?node_id=388048

sutch has asked for the wisdom of the Perl Monks concerning the following question:

O'Reilly has a new offering, Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Apps. The book looks interesting to me, since it details how to automate most aspects of project management that I now perform by hand. I would not purchase this book myself, since it is Java specific (not that I have anything against Java, just that I don't use it).

My question is: what do others use for handling project automation tasks?

Most of the applications which I develop or maintain are web based and use some sort of SQL storage. My office has discussed building a utility which takes an application's source code (possibly from CVS), builds it on a test machine, configures it with some saved values, configures Apache, creates or upgrades a database, populates a database, and runs some tests. We now do most of this by hand. It would be highly useful to automate this process for testing purposes as well as for pushing new or updated applications into production.

Replies are listed 'Best First'.
Re: Pragmatic project automation
by tachyon (Chancellor) on Sep 02, 2004 at 19:05 UTC

    I would think that scripted code checkout from CVS followed by compile and running the test suite was pretty usual. Given that this is only a few lines of shell or Perl why would you NOT automate it every night?

    I hate doing the same stuff over and will typically spend 2-3 x as long to script it so I never have to do it over ever again. Mostly you win, sometimes you lose.....

    cheers

    tachyon

      In our case it took a long time to create the automation but we recovered the cost (and time) very quickly, within days actually. My advice on this is to track how long is currently takes you now to do everything, that way you can show the benefit in $$$ to management. Good for those reviews and CYA. :-)
        Did you build your own tools? What tools are readily available?
      I would think that scripted code checkout from CVS followed by compile and running the test suite was pretty usual

      Oh how I wish that were true.... for a start you're assuming that everybody actually has some kind of source control (gosh - I love consulting :-)

Re: Pragmatic project automation
by johnnywang (Priest) on Sep 02, 2004 at 21:39 UTC
Re: Pragmatic project automation
by terra incognita (Pilgrim) on Sep 02, 2004 at 19:55 UTC
    Where I currently work I do the following automatically for both windows and UNIX. Install, configure, start and stop services, run tests, log results to a DB, and email out status messages. Building is out of my hands but anything else I can automate. Most of this is done via perl with some shell scripts and java that other people wrote. This has saved huge amounts of time and effort. PLus now we get to do cool stuff rather than all that boring stuff. :-)
Re: Pragmatic project automation
by EdwardG (Vicar) on Sep 03, 2004 at 11:01 UTC

    I've built one of these, and it was a lot of fun.

    The interesting thing was, to me, how this automation script blossomed over time into what now stands as a utility very much like ant and nant. To give you an idea of how far I went, here is the list of "modules" that I could batch together and configure for the many different flavours of "build" I had on my plate at the time:

    • Check_For_Missing_Interfaces.pl
    • Check_Valid_Build_Number.pl
    • CompareBuilds.pl
    • CopyFile.pl
    • CopyFiles.pl
    • Create_Database.pl
    • Create_Dir.pl
    • Create_Readme.pl
    • Create_Zip.pl
    • DelDir.pl
    • DelFile.pl
    • Detach_Database.pl
    • Detach_db_and_zip.pl
    • Encrypt_Prod_Files.pl
    • Expand_Top_Level_DB_Script.pl
    • GatherModules.pl
    • GetArchiveRevisionNumbers.pl
    • Log_Message.pl
    • MSSQL_Attach_db.pl
    • MSSQL_Autofix_Users.pl
    • PVCS_Get.pl
    • PVCS_Get_Lock.pl
    • PVCS_Unlock.pl
    • Restore_Test_DB.pl
    • Run_SQL_Command.pl
    • Run_SQL_Script.pl
    • ShowErrorsAndFail.pl
    • Unzip.pl
    • Unzip_Attach_DB.pl
    • Update_DB_Script_Files.pl
    • Update_Version_and_Model_Number.pl
    • VSS_Get_Project_Files.pl
    • VSS_Get_DB_Scripts.pl
    • VSS_Label_Tip.pl
    • VSS_Remove_Project_Label.pl
    • VSS_Set_Project_Label.pl
    • WriteToControlFile.pl
    • WriteToLog.pl

    I controlled builds by composing INI files (guess which platform I was on ;-), and I had a simple GUI I used to check/uncheck different steps in each. I was about to launch into writing a miniature scripting language, or perhaps just a mechanism for passing state information between steps, when I realised that I was reinventing the wheel in a major way and it was time to stop. Soon after, I changed jobs and the need for such a tool diminished.

     

Re: Pragmatic project automation
by eyepopslikeamosquito (Archbishop) on Sep 03, 2004 at 09:05 UTC

    We use multiple perl scripts to do the job: one to build the C++ code, one to run the C++ white box regression tests, one (Test::Harness prove) to run the Perl white box regression tests, another to run the black box regression tests, and so on. Writing each script as a stand-alone tool has proved useful because developers can use them standalone, plus the same tool can be called from the high level nightly smoker script.

    We don't do continuous builds. We do complete builds and regression tests nightly. This is mainly because we support a dozen or so different platforms from a single source code base. We use rsh to control running the nightly smoker across the many build and test machines.