Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

How has Perl saved you time at your job this week|month|year|decade?

by missingthepoint (Friar)
on Apr 06, 2009 at 09:14 UTC ( [id://755666]=perlmeditation: print w/replies, xml ) Need Help??

I occasionally am required to do some exceptionally boring task at $work. Such tasks drive me nuts (click, drag, good monkey!). The upside is, most of them can be automated to some extent, which tasks provide an opportunity to improve my Perl-fu and save time in the process - I find that performing the whole task manually is usually slower than writing a Perl script to do the job.

The most recent annoyance involved a static-HTML web page that we had neither time nor budget to rewrite using a database. I needed certain page elements each given a unique ID tag (to facilitate a drop-down, flashy image gallery thing). And the elements needed to be sorted by title in alphabetical order. Worse yet was that I didn't have all the elements to be added up front; a designer was sending them to me rather serially. I wasn't about to renumber 100/2 (on average) IDs each time an element was added, so out came Perl:

#!perl -pi.bak BEGIN { $i = 0; $next_regex = qr/toggle[(]/i; } $i++ if /$next_regex/; $num = sprintf "%02d", $i; s/toggle[(]'\d+'[)]/toggle('$num')/; s/icon_details_\d+/icon_details_$num/; s/details_\d+/details_$num/; s/gallery_\d+/gallery_$num/;

How easy was that! Run each time the page got a new element, it saved me half an hour and took maybe 5 minutes to write. Thanks, Perl! :)

So, how has Perl saved you time at your job recently? Specific examples would be great; code would be even better.


undef $lunches{free}

Replies are listed 'Best First'.
Re: How has Perl saved you time at your job this week|month|year|decade?
by moritz (Cardinal) on Apr 06, 2009 at 09:36 UTC
    I spent some time porting a Fortran application to C++. That was quite error-prone, so I had to compare intermediate values in the process, which were all large matrices of complex numbers. Comparing them is boring, takes long and is error-prone in itself, so I wrote a script which compares those matrices for me.

    It's really a hack, but it works fine for me. Now I also use it to test my C++ application when I change internals that shouldn't affect the output.

    Actually I wrote quite many small scripts for similar tasks: converting matrices from one format to another one so it can be used as input by a different library, sanity check (check symmetries in my output matrices) etc, this is just the most recent example.

    All of them saved much time, either my own or that of co-workers.

Re: How has Perl saved you time at your job this week|month|year|decade?
by Limbic~Region (Chancellor) on Apr 06, 2009 at 14:44 UTC
    missingthepoint,
    I don't program professionally as my primary vocation and yet perl saves me time at work all the time. More often, it saves my cow orkers time. I will use this past Friday as an example.

    There are scanned documents in image format sitting in a work queue table in the database. Each image needs to be opened by a human to read so a search can be performed to determine which record the document belongs to. This is a time consuming process.

    I wrote code that uses OCR technology to convert the scanned document to text and then wrote an algorithm to match the document against records waiting for an attached document. Since OCR itself is not perfect, this process does not completely eliminate the need for a human. On the other hand, in roughly 2 hours I was able to reduce the human work by 60 to 70%. The reason why it only took a couple of hours is because 90% of the code was already written (CPAN). I just needed to tie it together.

    Cheers - L~R

      "cow orkers"!

      Now *that's* funny!

      Sorry, just couldn't resist. It made me almost shoot diet cola out of my nose like in pre-school. I really loved the reply and all of the other ones.

      ack Albuquerque, NM
Re: How has Perl saved you time at your job this week|month|year|decade?
by blokhead (Monsignor) on Apr 06, 2009 at 15:33 UTC
    I don't have much opportunity to program much in my own work. But I saved my wife a lot of time in her work this year. She is doing research that involves a (non-computational) linguistic analysis on a large corpus of court transcripts. I wrote just a few little simple scripts to help her workflow. The scripts combined took me at most an hour to put together, and were entirely trivial from my point of view, but she thinks they saved her several days of work. It's nice to feel wanted, and empowering to be able to automate your environment.

    Some of the little scripts I recall writing were as follows:

    The transcripts were digitized using OCR scans, resulting in pretty strangely formatted text files. Perhaps the simplest thing I wrote was just a utility to strip the tons and tons of extra whitespace, making everything consistent and readable. To be user-friendly, the script opened up a window with Xdialog where it would accept pasted text. Then it would open another dialog with the cleaned-up text, ready for copying. I think it also gave some statistics like word/line counts. This script was not rocket science, but when I saw that my wife was going to mindlessly clean up several hundred pages of text by hand, I had to intervene!

    Another script I wrote was to help her classify different phrases according to the categories she was interested in. She had already split up the text into clauses (hard to do automatically). I wrote a script that would take as input (through our little cut & paste interface) a long list of clauses. Then she could enter a list of the categories she was interested in. Then it would present each clause, one at a time, and let her choose the category. At the end, it would output the final categorizations and some statistics. The alternative was that she would have copied and pasted from one word processor window (the list of unclassified clauses) into another, scrolling around to find the right category to paste it into.

    I know most of us here probably take for granted being able to do these kinds of things in our sleep (I mean, how hard is it to strip whitespace or count lines?). But even the Perl hacker's simplest technology is indistinguishable from magic to an outsider ;)

    blokhead

      But even the Perl hacker's simplest technology is indistinguishable from magic to an outsider

      Indeed - I've hacked together various other time-saving things at work which are simple to me, but unimaginable to my boss. That's no negative reflection on him; he's not really a hacker in any language and doesn't know what's possible (or easy).

      ++ for the Arthur C. Clarke reference! (even if unintentional) :)


      undef $lunches{free}
Re: How has Perl(Monks) saved you time at your job this week|month|year|decade?
by toolic (Bishop) on Apr 06, 2009 at 23:38 UTC
    Probably the most recent (of many) examples:

    We use a queueing system to submit thousands of jobs in a Linux environment (LSF job arrays). It is necessary to periodically check the status of the jobs. However, I have been unable to coax the status command to provide meaningful names for the jobs, rather than just their job ID number (for example, if I submit 5000 jobs, I want to know if job "foo" and job "bar" are still running, instead of ID647 and ID4902).

    To do this, I would have to sift through the vebose and poorly-formatted output of the status command to find the directories holding the output status files. Then I'd have to grep throught the output files to correlate ID's with job names. Bor-ing.

    Within a few hours, I cranked out some decent Perl code to do the job. But, I must admit that hanging out at PerlMonks for the last couple years has given me the confidence and skills to rapidly converge on and implement a solution. A year and a half ago, I probably would have thrashed around for a few days or a week on that task, or perhaps I would have found the task too daunting.

    Had I never joined, this...

    • would have taken days to write, instead of hours
    • would have been 5 times as many lines of code
    • would have been more poorly laid out
    • would certainly have more bugs

    Instead, what I have...

    • formats a multi-line command output to simplify parsing
    • uses a compact regex to extract just the info I need
    • performs input checking
    • cleanly opens and parses the needed files
    • stores the relevant info in a data structure which facilitates displaying
    • uses POD for standard-looking documentation
    • uses Pod::Usage for standardized help

    I guess this is as much a PerlMonks testimonial as it is a Perl testimonial. Without Perl, I'd be in a world of hurt. Without the Monks, I'd have a really embarrassing heap of unmaintainable, unreadable, undocumented, buggy code!

Re: How has Perl saved you time at your job this week|month|year|decade?
by xyzzy (Pilgrim) on Apr 06, 2009 at 15:09 UTC

    I'm an IT monkey for the project management team (the PHBs of the PHBs) in a corporate IT department. A while ago I decided to start learning Perl so I wrote a 150 line script (most of it is just SQL) to automate the process of running several ODBC queries in Access and dump that information into a formatted Excel spreadsheet. Doing it Perl has the benefits of not having to use Access, which is demon-spawn. If there ever arises a need for something new (most of my work is reporting) Perl will undoubtably be my first choice. Working with VBA last year atrophied my brain.



    Everything is true." "Even false things?" "Even false things are true" "How can that be?" "I dunno man, I didn't do it."
Re: How has Perl saved you time at your job this week|month|year|decade?
by targetsmart (Curate) on Apr 07, 2009 at 09:48 UTC
    Recently I had a requirement to write program(in 1 day)
    1) which sends ISO8583(for financial transactions) message to a server receives response and creates some temporary files, from where other program can take understandable information.
    2) Logs all the request and response(output, debug message, error logs)
    3) It should be easily maintainable,configurable,scalable,reliable
    4) Target environment is only linux(x86).

    it is not purely ISO8583 it is BASE24 which is based on ISO8583, I searched in CPAN and in GOOGLE about any perl modules for this purpose, but no help from there, I decided to write it of my own using Perl

    I used Log::Agent for logging, I used IO::Socket:INET and IO::Select for socket connections, then my core module BASE24 for sending and receiving APIs, and a global configuration file(in perl, just hash and array references for STATIC definition of request and response fields, its length, its type, etc).

    I was able to do this in one day time with some unit testing at my side. Second day it went live and started working. Thanks to Perl and its community.

    I thoroughly agree with toolic
    Within a few hours, I cranked out some decent Perl code to do the job. But, I must admit that hanging out at PerlMonks for the last couple years has given me the confidence and skills to rapidly converge on and implement a solution. A year and a half ago, I probably would have thrashed around for a few days or a week on that task, or perhaps I would have found the task too daunting.

    Had I never joined, this...

    * would have taken days to write, instead of hours
    * would have been 5 times as many lines of code
    * would have been more poorly laid out
    * would certainly have more bugs

    Instead, what I have...

    * entire API structure in STATIC hash and array references, which is easily configurable(by developers atleast).
    * performs error checking & reporting at every subroutine(at every stage) of the program.
    * maintains the relevant info in a data structure which facilitates mapping of request and response properly
    * socket connections, sending and receiving API was done with proper modules in an abstract way.
    * with proper comments.


    Vivek
    -- In accordance with the prarabdha of each, the One whose function it is to ordain makes each to act. What will not happen will never happen, whatever effort one may put forth. And what will happen will not fail to happen, however much one may seek to prevent it. This is certain. The part of wisdom therefore is to stay quiet.
Re: How has Perl saved you time at your job this week|month|year|decade?
by JavaFan (Canon) on Apr 06, 2009 at 15:23 UTC
    So, how has Perl saved you time at your job recently?

    This is a question that pops up over and over again. There are usually lots of people giving answers, but almost always the answer means "programming saved me some time". It just happens that in a Perl forum, most people use Perl as their main tool. But over at forum X-monks, people solve almost identical problem using language X.

    Over the years, I got paid to solve computer related problems. Perl has been my favourite tool. And sure, the tools I wrote in Perl saved time over doing it by hand - but I got paid to do it in a faster way than doing it by hand. And if I hadn't learn Perl, I might have learned Python or Ruby or REXX or be better at using the sh/awk/C trident, and would have solved to problems by using whatever language(s) I had learned instead.

      I have to disagree. It's true that Perl is not the only language that saves people time, and it's true that many of the things people mention in threads like this could have been done equally well in Python, say. But it's not true that anything Turing-complete is equally good at time-saving.

      In my case, when I say "Perl saved me time", I typically mean "CPAN saved me time". The extraordinary depth and variety of libraries available at the touch of a button does make a real difference in my work, and does save me a significant amount of effort relative to working in some other languages, such as C++.

      Even for languages with rich libraries, the learning curve for those libraries may differ wildly. Just recently I needed to produce a simple GUI application in a hurry. I wasn't familiar with any of the options. I actually gave some serious thought to using Java, since that would have had certain advantages in this instance. But I took a long hard look at Swing, and then I took a short hard look at Perl/Tk,* and it was quite simply a good order of magnitude simpler. It would have taken me several days to get anything working in Java, while with Perl/Tk I had a functional prototype by the end of the afternoon.

      (Could Tcl have saved me just as much time? I doubt it; I was using a number of other CPAN libraries in the same project. Did Perl itself have anything to do with the simplicity? Yes; it has proper closures with a lightweight syntax, which makes defining callbacks much simpler than Java's cumbersome anonymous-inner-class approach. Am I claiming it's better than Java for all purposes? Not at all. I would hesitate to deploy a Perl/Tk app enterprise-wide; Java's lack of expressiveness comes hand-in-hand with a much greater availability of maintenance programmers, and I have no end of respect for their remarkable ability to tolerate endless VerboseBoilerplateFactoryVisitors.)

      * And some sidelong glances at Gtk2, etc, but Perl/Tk was already installed on all the servers, and that's a BIG bonus.
Re: How has Perl saved you time at your job this week|month|year|decade?
by girarde (Hermit) on Apr 06, 2009 at 21:01 UTC
    snmpconf from net-snmp-utils.

    If you haven't checked for a perl script that solves your problem for you, you haven't checked all the easy fixes yet.

Re: How has Perl saved you time at your job this week|month|year|decade?
by ack (Deacon) on Apr 08, 2009 at 22:15 UTC

    On each of our systems projects, building and deliverying remotely deployable systems that accept commands and delivery telemetry data, we use Perl to script all of our Baseline Functional Tests. We do system integration and testing over about 12 to 18 months; during which we test, extensively all commands and all of the telemetry points that the system can accept/provide.

    However, we have rarely gone back and VALIDATED that we have done such an extensive composite of testing.

    About a month ago, our management began to say "prove it to us." Prove to us that every single command and every single Telemetry point was tested.

    We typically have about 800 to 1,000 commands and about 1,000 to 2,000 different telemetry points per system and all of our testing dutifully generates log files of everything that was done.

    So I proposed that I could process the roughly 10,000 lines across about 800 log files to find the answer.

    So I sat down and over about a 3 day period (most of which time was reviewing and being sure I knew the structure, syntax and oddities of the log filling processes) I developed and ran a fairly simple and uncomplicated script that generated exactly the results that were looked for.

    Turned out that 1: The bosses were delighted and impressed and 2: we found that there were 3 commands that had not actually ever been tested...great BONUS!

    I've tried similar tasks in the past with various languages (FORTRAN, VISUAL BASIC, C) and have always hated the process and difficulty of the task. Perl made it a true "no brainer"!

    ack Albuquerque, NM
Re: How has Perl saved you time at your job this week|month|year|decade?
by raisputin (Scribe) on Apr 17, 2009 at 23:14 UTC
    Perl saves me countless hours at work. Here are some short examples (no code, sorry, because I am still learning and it would probably make you shoot yourself in some cases :) )

    1) I wrote a small program to provision cable modems. What used to take a full day to do by hand, can now be accomplished in just under 10 minutes :)

    2) We use RT for our ticketing system. Unfortunately it doesn't provide enough information to our after-hours support team in any meaningful way. I used perl to modify the way information is displayed in a given ticket. It queries our CMTS and displays IP information, CM Information such as upstream/downstream power, et al. and allows our unprivileged technicians the ability to access user email via the webmail interface without knowing the users password, though our privileged users can see the password as well.

    3) Used Perl and Mason to create an installation system for technicians in the field. They take a cable modem to a customers house fill out some basic information and it provisions the modem for the customer. I am currently working on version 2.0 of this that will make it even less work for the technicians.

    4) Created some simple scripts that will query each of our mail servers for a given date and piece of information we are looking for and then return it all to the current shell for further investigation.

    5) I created a system that monitors our Cable modem system showing uptime, downtime, signal to noise ratio and % of modems online on a given node and stores it all in a mysql database in 5 minute intervals.

    There are dozens of other little tools that I have created that save us time as well, but those are some of the cooler ones :)

    I estimate I have saved thousands of man-hours over the course of my employment so far :) let's hope I get a nice raise soon ^^

Re: How has Perl saved you time at your job this week|month|year|decade?
by grizzley (Chaplain) on May 14, 2009 at 06:34 UTC
    My colleague wanted to extract all struct names of C++ code. He wanted a regexp to insert it into find/replace box. I really enjoyed watching him running perl -lne "print if /^\s*struct\b" file.cpp and starring at immediate result. It has saved him 3 hours work (he says so).
Re: How has Perl saved you time at your job this week|month|year|decade?
by Your Mother (Archbishop) on Apr 17, 2009 at 23:50 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlmeditation [id://755666]
Approved by citromatik
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others surveying the Monastery: (4)
As of 2024-03-29 04:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found