Strategies? Your basic refactoring strategies, I guess.

• Get everything to run under strict.
• Identify common code that seems to be repeated and get it to live in one place.
• Identify bits of business logic that should live together, then make them live together, preferably in some object.
• Identify bits of application logic that yadda, yadda, yadda.
• If there's a way to make Perl do your work for you, then be intelligently-lazy. For example, all the messaging was done by having each message be its own class. But, I didn't want to maintain 1000+ messages in three different strains by hand. So, I developed a common data structure that would be inflated by Perl into the classes. (That was over 75% of the code and about 45% of the success.)
• Build a logging methodology and put it everywhere. Unless you know exactly what went wrong, you'll never make it right.
• Get the buy-in from your group leader, but ignore anyone higher. All they want to know is that it works, not the cool things you did to get it there.
• Write migration tools. Part of this effort involved changing the configuration files that the userbase used. Absolutely necessary. However, if I wanted acceptance, I had to make it as painless as possible to migrate.

Mistakes? The biggest mistake anyone can make is to not have a design document, preferably coupled with a requirements document. This isn't an ivory-tower nicety. This is a real-world, in-the-trenches, life-and-death necessity. Spend the 100 hours to make one and you will recoup that 100 hours in the next four weeks. (In case you think I'm exagerrating, ask Elian what he thinks of design documents.)

------
We are the carpenters and bricklayers of the Information Age.

Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

The previous administrator had made a mess of things - he would just sit in his cubicle, and only move his ass when people started to complain they could no longer do their job. Nothing was being monitored. Backups were automated, but since everything was hardcoded, and not touched for a long time, most databases, including the more important ones were not updated - and noone was inspecting the output for errors. When I took over, I was working 10+ hours/day just to keep things running. At my first day as a dba, I hacked my first monitoring program together, and at least I would get paged if a database became unavailable, instead having to wait till user started to complain. Within months, everything was database and every server was being monitored for their vital statistics, all databases would be backed up daily (no recoding needed if databases were added or deleted), databases would be dbcc-ed regulary, and many problems would be solved automatically within a minute of them occurring, and I was down of spending less than 4 hours a day on my dba tasks, while getting an uptime of over 99.95% outside of the maintainance windows. With the vital statistics of the most important statistics being graphed - to please manglement.

This turned out to be my biggest mistake as well. When the company had to lay off people, they decided a dba was no longer needed. :(

As for strategies/techniques/modules, the code itself wasn't too much. All monitoring less than 1000 lines of code. The Sybase reference and sysadmin modules of the documentation where vital though. I used Sybase::DBlib to make the database connections, gnuplot for the graphing, and /bin/mailx for the paging.

Abigail

If you're exploring Perl's power, I suggest you rethink your first question, or consider adding a fourth. Perl is suitable for more than just solo programming. Teams can use it, too. I once managed a team that had written an 80KLOC application server in Perl. (The core of the team that wrote it was ~6 people, more were added later to extend it).

Personally, the biggest single thing I have in Perl is about 1500 lines, plus another 1000 lines of unit tests, but not including CPAN stuff (HTML::Template, DBI, plus a couple of others).

A strategy that helped me pull it off was to use Test-Driven Development. Figuring out how to write tests first helped me think through APIs with an actual client (the test) in mind. That helped avoid a lot of nonsense. Having a large body of unit tests gave me confidence that I wasn't breaking things as I went.

The biggest mistake I made was not getting the code body under source-code control soon enough. (I've since set up a home CVS pserver.)

I agree about the team vs. solo. If someone told me he had written a huge application in Perl all by himself and 90% of the code was his, I would not see that as a good thing. Much better to hear something like "I wrote a successful application with a zillion features that was able to turn on a dime when the business asked for changes, and took only a few hundred lines of code, and I did it by collaborating well and reusing tons of open source code from CPAN."

Team or individual, it's prudent to see the quality of the code before judging, wouldn't you agree? 90% The Conway™ just might stand a chance of being better than 90% temp-SDEs. There have been fantastic large things done mostly by an individual with Perl (like Mason, right? or like the amazing sounding wireless stuff dragonchild wrote about, or CGI.pm for that matter—almost 1/2 of its 6,400 lines is code and I know that many have worked on it today but at one point it was the work of one and that single module drove a staggering portion of web development).

The power of the communit(y|ies) is evident just by visiting this amazing site. I'm curious about the extreme edge of the leverage that Perl gives the individual. There are many, many users here who own scads of modules or big parts of Perl. What better group to put this question to?

The second question does include "what modules?" I'm not seeking the Alceste formula... unless of course you have it. :)

What is the largest system|code|site you have (>=90% of the work was your own) supported|created|inherited which was primarily in Perl?

I don't know since most of my projects are under 2000 lines of code (Perl's too damn powerful).

What are the main strategies that allowed you to pull it off?

K.I.S.S.
No feature overkill, no complex designs, no bureaucratic nonsense.

Other than that, document every choice and every API for you WILL forget the specifics.

What are the main techniques|modules that allowed you to pull it off?

• Perl for everything;
• Imager for almost every graphics thing;
• PLP for web programming;
• DBIx::Simple for database stuff, usually with DBD::SQLite or DBD::Pg;

Stuff things in modules if it makes sense. Use OO where it makes sense. Release code you use in almost every project to CPAN (let others discover bugs for you so you can fix them before things go terribly wrong *grin*). That's why PLP, DBIx::Simple, again and Exporter::Tidy are on CPAN. They wouldn't be if I didn't use them myself.

how about a bulleted list?

Bulleted lists form a great strategy, a reasonable technique and perhaps a nice module. But they're a terrible brand of coffee!

What was the single biggest mistake (most costly to your own time) you made while doing it?

Not documenting was my biggest mistake. Lack of documentation saves time at first, but takes a LOT of time when you need to change the code a few years later.

Another VERY big mistake was starting to code in Perl without actually knowing the language well enough. I do know Perl quite well now, but my old code is disastrous. Even though I did make enough money off my first projects to start my own business, I do regret that I did not wait another year or maybe two. And I wish I had a good book back then: I learnt most of what I know about Perl from perldocs and IRC.

One very good thing you can do is letting people here and on IRC review your code. Disclosing source usually also means tidier code :)

Another piece of advice: choose a readable and CONSISTENT programming style. Whitespace DOES matter!

Juerd
- http://juerd.nl/
- spamcollector_perlmonks@juerd.nl (do not use).

• All shop owners would host their data on our site, but could link our site into theirs
• We had an internal mail system between shop owners and customers, including an address book and all
• Based on figures from various international courier/freighters the shop owner picked a freight company, and we worked out all the charges for them and contacted the freight company for item pick-up ourselves
• The entire system was insured, and as such had an Insurance module allowing representatives of the insurance company to review any insurances claims (Which had to be made internally) on bad sellers
• All sales history was naturally recorded, and there was a mountain of ways of displaying and graphing this data, thanks to the accounts two floors down
• Our partner bank decided not to allow us to pay money into a credit card, as such we would collect all of our shop owners profits during the day and then wire amounts to each country appropriately. Then we would manually dial in via modem to whatever chosen issuing bank we had talked to, and spread the money out to bank accounts from there, thusly avoiding a telegraphic transfer charge for every bank account we sent money to.

JP,
-- Alexander Widdlemouse undid his bellybutton and his bum dropped off --

>=90% of the work was your own

Has such a project ever been written, by anyone? Have you looked at the modules you're using? My Perl projects have all had more reused code written by others than original code written by me. Is this a bad thing? No, because that's not what I use Perl for. If I want to write an entire application from scratch, I'd choose a different language. Perl's power comes directly from third-party modules.

• Peeron LEGO Inventories.
• Modules:
  • CGI
  • GDBM_File at first then DBI
  • HTML::Template
  • strict
• Biggest mistake - not using an SQL backed from the start. I still haven't recovered from that completely. Should have written OO code, again, transitioning to it is HARD.

-- zigdon

CGI, DBI, and HTML::Template (plus a few of my own modules) made the job a lot easier.

My biggest mistake was not modularizing the code from the beginning. I copy and pasted instead of creating my own modules for general functions. This led to problems when I updated code in one area but not another and general bloat. It also made it harder to switch backends. I've since shoved most of the general stuff in modules, but there is still some things that I haven't gotten around to yet.

• Vanilla Coke. By the bucket. Diet counts too.
• Modularize where you can. (Althoug this was mostly appropriate due to the project.)
• DBI is your friend.
• Keep things neat and organized! I can't tell you how much of a difference that made (both good and bad when done/not done, respectivly). subs and "comment blocked" code, so that its easy to come back to a section after working on an entirly different part of the code for long periods of time. As your project gets large, this will occur more frequently and thus becomes more important.

What is the largest system|code|site you have (>=90% of the work was your own) supported|created|inherited which was primarily in Perl?

I wrote a 10'000 lines network test utility (database with hosts, pinging, host/IP resolving, tracerouting and proxy check) with a HTML interface. At the moment I'm writing a library of useful classes for system administration scripts which has about 5'000 lines.

What are the main strategies|techniques|modules|brands-of-coffee that allowed you to pull it off (how about a bulleted list)?

Planning: Flow charts for scripts and UML for bigger libraries. Bulleted lists and TODO/FEATURES lists also help a lot. Also look at CPAN if someone has already done something similar.

Modules: CGI, HTML::Template, CGI::Application, DBI

What was the single biggest mistake (most costly to your own time) you made while doing it?

Improper design at the beginning -> rewrinting some parts from scratch and having done very little documentation (that was before I discovered POD :)).

Thomas

YABM (Yet Another Biggest Mistake): Roling my own webapplication framework with HTML::Template which did approximately the same as CGI::Application but with a procedural interface. I wrote only one application with it and abandoned it.

The largest single piece of code that I have written takes the back end database files from our Avaya CMS server, renames them, inserts them to .xls files, and writes them to a database.

The second part was a bit more difficult, I have been using the Spreadsheet::WriteExcel::Big CPAN module to write the files to an excel spreadsheet (for one manager that only uses MS Excel) containing all of the call records each day. I had some difficulties, but solved them by chopping the files at about 14000 lines.

The last piece involved writing the daily call records to a MS SQL database, then a MYSQL database for our local reporting team to generate reports from.

My main strategies involved trying to go out of my way to get everyone and what format they wanted and to help out local reporting guy who was spending all his time doing this task manually. As for the modules, I found Spreadsheet::WriteExcel::Big to be very good.

My biggest mistake was using MS SQl as a db the first time. The tables csv files have 55 columns, somewhere between 20,000 and 30,000 records each day. I had a terrible time with the DBI, each record was taking around 4 second to write. Finally I wised up and was able to write the all the records in just a couple of hours each day to the MYSQL db with the exact same piece of code, go figure.

Thanks

Greg W

• Change Nothing
• If you have to change it, make the smallest change possible
• Make all changes in parallel---implement the new code and compare it's results with the previous
• Change Nothing without a test to back it up with...

--hsm

Things that enabled me to pull it off:

• Primarily, focusing on having something releasable without being perfect. If I had "finished" any single part of it to my satisfaction, I never would have released anything.
• Heavy use of CPAN modules, of course!
• Don't think too hard about making everything nicely modular initially. It's much easier to write a huge wad of stream-of-consciousness code and then go back and figure out what can be refactored into reusable pieces. (This might be a personal bias, because I suck so badly at up-front design.)
• Come up with a small number of nontrivial and representative test cases early on. I personally don't care whether it's before you start implementing or during, and I tend to leave the exhaustive test cases until afterwards. During the main implementation push, I prefer to work from a small number of tests that I get to know intimately well.
• Try very hard to plow through the things that are more likely to be showstoppers early on, when you're still enthusiastic and flying along.

Biggest mistake: probably allowing myself to flail when I get stuck on things. Instead of stepping back and figuring out a simpler way of approaching the whole area I was having problems with, I would tend to accrete special cases, debugging code, large commented state machines, painstakingly detailed descriptions of what some insanely complicated function should be doing, etc. There's probably a better way to summarize that -- perhaps it would be "my biggest mistake is trying to solve problems by making them more complex instead of simpler"? Something like that.