Your Mother has asked for the wisdom of the Perl Monks concerning the following question:
- What is the largest system|code|site you have (>=90% of the work was your own) supported|created|inherited which was primarily in Perl?
- What are the main strategies|techniques|modules|brands-of-coffee that allowed you to pull it off (how about a bulleted list)?
- What was the single biggest mistake (most costly to your own time) you made while doing it?
Addition: Excellent stuff so far! Keep it coming. I'll see about doing a précis of the replies when the thread has died down.
|
---|
Replies are listed 'Best First'. | |
---|---|
Re: What's the biggest piece of work you've done alone with Perl?
by dragonchild (Archbishop) on Apr 21, 2003 at 22:26 UTC | |
Strategies? Your basic refactoring strategies, I guess. Mistakes? The biggest mistake anyone can make is to not have a design document, preferably coupled with a requirements document. This isn't an ivory-tower nicety. This is a real-world, in-the-trenches, life-and-death necessity. Spend the 100 hours to make one and you will recoup that 100 hours in the next four weeks. (In case you think I'm exagerrating, ask Elian what he thinks of design documents.) ------ Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement. Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified. | [reply] |
Re: What's the biggest piece of work you've done alone with Perl?
by Abigail-II (Bishop) on Apr 21, 2003 at 23:49 UTC | |
The previous administrator had made a mess of things - he would just sit in his cubicle, and only move his ass when people started to complain they could no longer do their job. Nothing was being monitored. Backups were automated, but since everything was hardcoded, and not touched for a long time, most databases, including the more important ones were not updated - and noone was inspecting the output for errors. When I took over, I was working 10+ hours/day just to keep things running. At my first day as a dba, I hacked my first monitoring program together, and at least I would get paged if a database became unavailable, instead having to wait till user started to complain. Within months, everything was database and every server was being monitored for their vital statistics, all databases would be backed up daily (no recoding needed if databases were added or deleted), databases would be dbcc-ed regulary, and many problems would be solved automatically within a minute of them occurring, and I was down of spending less than 4 hours a day on my dba tasks, while getting an uptime of over 99.95% outside of the maintainance windows. With the vital statistics of the most important statistics being graphed - to please manglement. This turned out to be my biggest mistake as well. When the company had to lay off people, they decided a dba was no longer needed. :( As for strategies/techniques/modules, the code itself wasn't too much. All monitoring less than 1000 lines of code. The Sybase reference and sysadmin modules of the documentation where vital though. I used Sybase::DBlib to make the database connections, gnuplot for the graphing, and /bin/mailx for the paging. Abigail | [reply] [d/l] |
Re: What's the biggest piece of work you've done alone with Perl?
by dws (Chancellor) on Apr 22, 2003 at 00:31 UTC | |
If you're exploring Perl's power, I suggest you rethink your first question, or consider adding a fourth. Perl is suitable for more than just solo programming. Teams can use it, too. I once managed a team that had written an 80KLOC application server in Perl. (The core of the team that wrote it was ~6 people, more were added later to extend it). Personally, the biggest single thing I have in Perl is about 1500 lines, plus another 1000 lines of unit tests, but not including CPAN stuff (HTML::Template, DBI, plus a couple of others). A strategy that helped me pull it off was to use Test-Driven Development. Figuring out how to write tests first helped me think through APIs with an actual client (the test) in mind. That helped avoid a lot of nonsense. Having a large body of unit tests gave me confidence that I wasn't breaking things as I went. The biggest mistake I made was not getting the code body under source-code control soon enough. (I've since set up a home CVS pserver.)
| [reply] |
by perrin (Chancellor) on Apr 22, 2003 at 04:16 UTC | |
| [reply] |
by Your Mother (Archbishop) on Apr 22, 2003 at 08:16 UTC | |
The power of the communit(y|ies) is evident just by visiting this amazing site. I'm curious about the extreme edge of the leverage that Perl gives the individual. There are many, many users here who own scads of modules or big parts of Perl. What better group to put this question to? The second question does include "what modules?" I'm not seeking the Alceste formula... unless of course you have it. :) | [reply] |
by perrin (Chancellor) on Apr 22, 2003 at 22:26 UTC | |
Re: What's the biggest piece of work you've done alone with Perl?
by Juerd (Abbot) on Apr 22, 2003 at 00:24 UTC | |
What is the largest system|code|site you have (>=90% of the work was your own) supported|created|inherited which was primarily in Perl? I don't know since most of my projects are under 2000 lines of code (Perl's too damn powerful). What are the main strategies that allowed you to pull it off?
K.I.S.S. Other than that, document every choice and every API for you WILL forget the specifics. What are the main techniques|modules that allowed you to pull it off?
Stuff things in modules if it makes sense. Use OO where it makes sense. Release code you use in almost every project to CPAN (let others discover bugs for you so you can fix them before things go terribly wrong *grin*). That's why PLP, DBIx::Simple, again and Exporter::Tidy are on CPAN. They wouldn't be if I didn't use them myself. how about a bulleted list? Bulleted lists form a great strategy, a reasonable technique and perhaps a nice module. But they're a terrible brand of coffee! What was the single biggest mistake (most costly to your own time) you made while doing it? Not documenting was my biggest mistake. Lack of documentation saves time at first, but takes a LOT of time when you need to change the code a few years later. Another VERY big mistake was starting to code in Perl without actually knowing the language well enough. I do know Perl quite well now, but my old code is disastrous. Even though I did make enough money off my first projects to start my own business, I do regret that I did not wait another year or maybe two. And I wish I had a good book back then: I learnt most of what I know about Perl from perldocs and IRC. One very good thing you can do is letting people here and on IRC review your code. Disclosing source usually also means tidier code :) Another piece of advice: choose a readable and CONSISTENT programming style. Whitespace DOES matter!
| [reply] |
Re: What's the biggest piece of work you've done alone with Perl?
by JPaul (Hermit) on Apr 22, 2003 at 15:10 UTC | |
And in this case 'Platform' is the operative word; From the get-go the specs changed from week to week depending on what idiot the manager had talked to and told him what his application had to have and how it had to function in just specifically a certain way. As the bank we had partnered with became more and more difficult to deal with the manager and accountant would think up more and more ridiculously obscure and abstract ways of accomplishing tasks that had become over-regulated. The system ended up being big. Like, way big. Like, you don't really want to know how big. The only part of the software not written in perl was the code used to talk to the bank via a frame relay VPN for credit card transactions, and that was only because it was based on existing code (That didn't work).
JP, | [reply] |
Re: What's the biggest piece of work you've done alone with Perl?
by Anonymous Monk on Apr 22, 2003 at 03:51 UTC | |
>=90% of the work was your own Has such a project ever been written, by anyone? Have you looked at the modules you're using? My Perl projects have all had more reused code written by others than original code written by me. Is this a bad thing? No, because that's not what I use Perl for. If I want to write an entire application from scratch, I'd choose a different language. Perl's power comes directly from third-party modules. | [reply] |
(z) Re: What's the biggest piece of work you've done alone with Perl?
by zigdon (Deacon) on Apr 22, 2003 at 17:16 UTC | |
-- zigdon | [reply] |
Re: What's the biggest piece of work you've done alone with Perl?
by The Mad Hatter (Priest) on Apr 22, 2003 at 17:28 UTC | |
CGI, DBI, and HTML::Template (plus a few of my own modules) made the job a lot easier. My biggest mistake was not modularizing the code from the beginning. I copy and pasted instead of creating my own modules for general functions. This led to problems when I updated code in one area but not another and general bloat. It also made it harder to switch backends. I've since shoved most of the general stuff in modules, but there is still some things that I haven't gotten around to yet. | [reply] |
Re: What's the biggest piece of work you've done alone with Perl?
by cyberconte (Scribe) on Apr 22, 2003 at 18:11 UTC | |
Probably the biggest mistake i made was trying to do everything in one shot. Working on a project alone, every time i would complete 1 task, i would htink of 3 others that would be great to have. The end result was a .0 release that, in all honesty, should've been a .9 release. As of right now, i'm working on a .1 release which is what i imagined the .0 release should've been. If i had a technical document, i would've been much better off and had much clearer goals. I would reccomend one. | [reply] |
Re: What's the biggest piece of work you've done alone with Perl?
by thom (Beadle) on Apr 22, 2003 at 18:36 UTC | |
What is the largest system|code|site you have (>=90% of the work was your own) supported|created|inherited which was primarily in Perl? I wrote a 10'000 lines network test utility (database with hosts, pinging, host/IP resolving, tracerouting and proxy check) with a HTML interface. At the moment I'm writing a library of useful classes for system administration scripts which has about 5'000 lines. What are the main strategies|techniques|modules|brands-of-coffee that allowed you to pull it off (how about a bulleted list)? Planning: Flow charts for scripts and UML for bigger libraries. Bulleted lists and TODO/FEATURES lists also help a lot. Also look at CPAN if someone has already done something similar. Modules: CGI, HTML::Template, CGI::Application, DBI What was the single biggest mistake (most costly to your own time) you made while doing it? Improper design at the beginning -> rewrinting some parts from scratch and having done very little documentation (that was before I discovered POD :)). Thomas | [reply] |
by thom (Beadle) on Apr 23, 2003 at 07:50 UTC | |
YABM (Yet Another Biggest Mistake): Roling my own webapplication framework with HTML::Template which did approximately the same as CGI::Application but with a procedural interface. I wrote only one application with it and abandoned it. | [reply] |
Re: What's the biggest piece of work you've done alone with Perl?
by gawatkins (Monsignor) on Apr 22, 2003 at 22:29 UTC | |
The largest single piece of code that I have written takes the back end database files from our Avaya CMS server, renames them, inserts them to .xls files, and writes them to a database. The first part involves the renaming the files from their default name of chrXXX.XXX where the X's are numbers. One of our remote reporting teams likes to receive them as csv files, so I simply open the directory and copy the files to a directory with the extension csv.The second part was a bit more difficult, I have been using the Spreadsheet::WriteExcel::Big CPAN module to write the files to an excel spreadsheet (for one manager that only uses MS Excel) containing all of the call records each day. I had some difficulties, but solved them by chopping the files at about 14000 lines. The last piece involved writing the daily call records to a MS SQL database, then a MYSQL database for our local reporting team to generate reports from. My main strategies involved trying to go out of my way to get everyone and what format they wanted and to help out local reporting guy who was spending all his time doing this task manually. As for the modules, I found Spreadsheet::WriteExcel::Big to be very good. My biggest mistake was using MS SQl as a db the first time. The tables csv files have 55 columns, somewhere between 20,000 and 30,000 records each day. I had a terrible time with the DBI, each record was taking around 4 second to write. Finally I wised up and was able to write the all the records in just a couple of hours each day to the MYSQL db with the exact same piece of code, go figure. Thanks Greg W | [reply] |
Re: What's the biggest piece of work you've done alone with Perl?
by hsmyers (Canon) on Apr 22, 2003 at 14:47 UTC | |
--hsm "Never try to teach a pig to sing...it wastes your time and it annoys the pig." | [reply] |
Re: What's the biggest piece of work you've done alone with Perl?
by sfink (Deacon) on Apr 28, 2003 at 03:59 UTC | |
Things that enabled me to pull it off: Biggest mistake: probably allowing myself to flail when I get stuck on things. Instead of stepping back and figuring out a simpler way of approaching the whole area I was having problems with, I would tend to accrete special cases, debugging code, large commented state machines, painstakingly detailed descriptions of what some insanely complicated function should be doing, etc. There's probably a better way to summarize that -- perhaps it would be "my biggest mistake is trying to solve problems by making them more complex instead of simpler"? Something like that. | [reply] |