Two things come to mind.

First, write unit tests. jlp just wrote a very small and clever module for Jellybean that makes running tests very easy. If you have more complex needs, clemburg works on the Test-Unit bundle on the CPAN. Having a comprehensive test suite will prove baseline functionality, freeing you to clean things up inside without having to wonder if your changes broke something. (It's not 100% guaranteed, but if you move things around and a test suddenly fails, you know exactly why.)

The second thing is to keep your design as simple as it needs to be. That doesn't mean you design everything at the start and stick with that until the bitter end. It doesn't mean that you find yourself in a corner and rewrite everything. It means that you add features only when absolutely necessary. After you add a feature and its tests pass, go back over the code and see if you can simplify it further. The tests will tell you if you break something.

This all presupposes that you're a competent enough programmer to have good habits from the start -- you are. If you weren't, you wouldn't have recognized this and you wouldn't be asking for advice. Trust your instincts, but cultivate good habits to keep you from false Laziness.

Finally, when you're stuck, look at the project and ask yourself, "What needs to be done?" Organize that list according to importance. Then break the most important thing into individual units that can be added, tested, and debugged in a single sitting. Then do it.

If you can follow this half of the time, you'll be twice the programmer in short order. The tests are the key.

I've done the same with some of my projects. I write enough to see what my mistakes are with my code, then dump it and start again. It's by no means perfect, but that initial "oops" phase can strongly help any software project by ironing out the potental bumps in programming.

Now, if you backing against a deadline and in this situation, it's a bit hard to do that, but I would at least try to find the ones that need dire rewriting and do that, such that other programmers at least see where you are headed.

Even if you are working by yourself, you might try to get a few friends to help you with this exercise just for fun.

Apparently this gives a lot of insight into what the bottlenecks will be in real life and how it will work.

It also plays well into a tidbit I learned a long time ago about memory. People who study memory techniques claim that the bigger you make a visualization, and the more senses you get involved, the better your brain is at working with the image, and the better you will be able to remember it later. For instance imagining walking into a room and looking around is far better than imagining holding a model of the room and peering in.

Well the visualization of standing in a circle, throwing things, for the model of how a program works is a pretty big visualization that involves lots of your senses...

Example:

• What is the startup state of the script, environment variables, internal variables
• How does the script leave startup to a running state
• How does it handle signals and exceptions, not in the normal flow - what state does it go to and then leave on exception
• What inputs drive the script to what new states - like reading lines in a file, processing each line, looking for EOF, scanning for input errors
• Are errors handled gracefully
• How does the script terminate - what does the terminate state do

  I once had to write a text restructuring program that had three variables with two states each for a word processor ruler function. It wouldn't translate ceratin content right. Then I realized that the scanner had eight states (two to the third power) but I had only coded for six. Me dumb. Twenty minutes of new code and she shebanged. So.. are all of the states accounted for in YOUR code.

  Also on the advice from above, use the simplest means you can employ to get it done.

  Also, I usually prepare by writing psuedo code-like comments. The stunning wonder of Perl is the move from the psuedo code to the Real Thing is so short. Sometimes I draw state diagram pictures, especially if I have to explain it to others.

  -diskcrash

At the same time, in another file (like a table of contents or a master record), I put the name of the routine and a list of all the places I call it. I ignore certain super common pieces (like my own rolled cgi parser). And usually because I am working against databases, I work on the schemas for the data.

All along I am preparing a list of questions for the client.

Then I'll go through and make scripts, again empty, that are the framework of the real scripts with ridiculous comments. Usually while I am doing this I am adding and modifying the database schema and the subroutines.

Once I have all this schtuff sort of laid out, I go through and format indentation, making the scripts and bits legal (but not necessarily functional). Again, relooking at things means I am modifying again.

It's usually at this point I actually try to run things. Of course nothing actually works but a bunch of now completely inane prints....

So I print the code. And save off a copy. Send the questions to the client.

And go and do something completely different for at least a day.

I have found if I concentrate on something like this, I cannot comprehend the overall flow, but instead focus on the little details, so a day away gives my addled brain a chance to work it through. The printout is nice cause I can make notes on it, without actually sitting down and trying to work out code bits.

Then I go back through it, make whatever changes make sense and print it again.

At this point I start actually adding working code, parts at a time, testing along the way, right on top of all the faux schtuff I have already written out.

Usually I'll spend 10 to 30 percent of my time in the working out the flow phase and sometimes what I end up actually resembles what I started with.

As far as what I do when my code is ummm....disorganized...I deliver something mostly - partially - somewhat - working schtuff usually before schedule, ask for changes, and rewrite using the above method a second time. The second time through goes a lot faster, and I have lots of code to put into place, usually with mostly minor self induced changes.

Usually the before schedule version buys me enough time to make it nice if I have to go over schedule. If not, I apologize and bite the bullet. Fortunately, I get better all the time so these things happen less frequently.

EEjack

I usually try to do a process of constant refinement. I program the first working part of the program, then the next. Any time I feel the urge to cut and paste from the first program I pop the routine into a shared module. So my modules are built from routines that absolutely have to be shared, and nothing else.

The trouble with this approach is I am practically garanteed to paint myself into a corner at least once. So at some point I have to take some time off and redo functions in the modules (combine some, separate others, etc). I then have no choice but to take some time to reprogram stuff I've already done. To aid this process I usually document every variable and function using Lyx, and constantly update the docs as I move functions around. The upside of this is I can tidy up the notes and give them to the client as part of the documentation.

____________________
Jeremy
I didn't believe in evil until I dated it.

When you're at the point where you look at the project and see a mess, it's best to distance yourself from it and work on a new design. You'll be able to design it better given the experience you've had so far. There will be the temptation to keep building on top of the mess in the interest in time, but unless you absolutely cannot afford to spend the extra time it's often wiser to rebuild. In a lot of cases, you'll find that it won't take as much time to rewrite as you thought.

If it's really the mess you say, you'll find that the cost of maintenance escalates. It'll be harder to find bugs, and that won't make you or your client happy. And 6 months later, when you client asks for some additions, it could be a headache wading through that mess again--especially since it's not fresh on your mind anymore. I've had this experience before, and it's not fun at all.

The bad news is that architecting skills are acquired only with time and experience. You can read a score of books on the subject, but you really only know what you're doing after years of doing it.

To properly architect a project, begin with use cases. Buy several books on the subject. (Hint: O'Reilly's UML book is a great reference, but not good for learning.) They seem silly at first, but they are a great way to formally organize your thoughts and the thoughts of those who are providing your specifications. This will also be the time at which you will identify specifications that will cost the client more in development time and money than they will see in return; i.e., this is the time at which you should feel free to say, "No."

Next, diagram your objects using class, object, sequence, and activity diagrams. If you don't have a high-powered diagramming software (I recommend Dia if you don't have a big budget to work with, and have found Object Domain quite useful, significantly cheaper, and much friendlier to Linux and *nix users than Rational Rose's software), draw your ideas on a sheet of paper.

Next, create a prototype. Figure out what you did wrong. Repeat the above steps until you're comfortable, however you define that.

Putting together all of the above involves a learning curve if you haven't done it already, but you need to buckle down and learn it even if you're the only Mohican on staff. In the words of your mother, "It's for your own good." Trust me.

In lieu of having the time to learn and implement the above (e.g., if you work at a dot-com), do as much as you can, making sure that you take the time to learn, and look up "refactoring." This is the process of taking existing code and making it better. Knowing nothing more about your situation, this may very well be what you need to do for pragmatism's and expediancy's sake.

If you arrive at the conclusion that you need to rewrite most of what you've done, you will merely be in the same boat as most coders with years of experience have been at at some point in their careers. Don't beat yourself up about it. Simply go forward as best as you can, knowing that you did as good of a job as your experience and knowledge base would allow, and make sure you educate yourself in the process.

Now for the good news: While your employer doesn't care how much you learn as long as you get it done, if you take the time to educate yourself on the above concepts and simply force yourself to adhere to good coding practices, you will over time acquire an intuitive feel for how to address the questions that you ask.

You will rarely feel like you know exactly how to architect anything perfectly, but you will see yourself get better and better over time until you are clearly head and shoulders above less experienced coders on staff.

In the end if you are doing object oriented programming, you want to break it up into objects. The modern school looks for the most important nouns around, say passenger, ticket, and airplane, for one simple example. An older school looks for verbs or server processes, which is really a lot harder to get into your head. If you use nouns (I would) you can say, okay, I'm a passenger, what do I do next. Or I'm an airplane, what kind of information do I know? If you have a lot of information and don't know where to start, you can look for organizing principles so for example if you are doing a business logic project you might want to ask "Where does the money go?" and follow the flow of money, or maybe a purchase order, through an organization. These are the kinds of things that get into those CRC cards too, like What other classes (objects/nouns) do I need to know? and so on.

This is for bigger systems, maybe more than you want, but the general ideas still may work. There is also a group of programming concepts called design patterns which you may read about on the web which can help broaden your conceptual vocabulary if you think you need to get out of the box. Or if you're feeling like a commando armed with Perl and nothing can get in your way you could take a look at Tim Conway's Quantum Reality modules, if he can bend perl around that anything's possible. :)

Me, on a simple project I often just write pseudo code, whether in the browser (commented out, then fill in between the comments if it's really simple), or on paper. Maybe make some little mockups of the screens I want to see. Whatever it is, you probably want to think about what is going to have to get used or maybe extended the most, and see if that could become an object. Some people from the Xerox Parc (Palo Alto Research Center) school like to draw little sketches of people's heads and diagram how they interact with each other, to model a communications system. Something visual and dynamic that keeps your hand moving and head working is good. even stream of consciousness typing is fine if you clean it up later. Give yourself a day or so to leave and come back and look at it again.

The most important thing I think is to document it to yourself as you are going, saying what you are going to do. If once you dive in it starts to get hairy, then you have somewhere to retreat and take a breath, and update your worldview.

By the way, the XP stuff is neat too, I'm talking about the idea of making peace with yourself in the knowledge that this software may never be done. Just find a small part of the cosmic all in your head and make it work. You can create a series of goals and successes when you solve each one and then tear things up if you don't like them.

One nice thing about perl is that it can be lots of things to lots of people. You don't have to recreate structures as if it was Java or C++ or something else that has lots of rules. Though I didn't believe it, it seems to be helpful if you have that perl programmer trait of laziness (well, let's say you want to get something done quickly and easily) for you can then focus your energy on rigorous consideration of what you are trying to do.

Maybe other people will have Perl-specific suggestions. To my mind anyway, it will save you lots of time if you stick with pen and paper for a while. You also will be less disappointed if your thought models reflect the problem you are trying to solve, because a good model of the world is going to give you ahah! synergies, and continue to open doors to realizing things about your problem which you did not see at first.

Finally, if you are already into the project like you said, maybe the most important decision is to ask what simple tasks have I already solved and are definitely working okay. Don't change those. Then be willing to tear out chunks and leave the rest of the monolith standing there, just because if you are doing it on a deadline it has to work, not be pretty. If it is too scary, just tear it all down, it's not worth tweaking a monstrosity because it will just keep throwing you off balance (I know from recent experience). You can then try to step back, break the problem up into black box components which you just have to talk to, not know intimately, and then you can start thinking about how they interact together.

So first step, clear off the desk. Get a big liter bottle of ice coffee or something like it. get a pen and start attacking it anywhere, even on the edges, trying to figure out what you want to have happen and not how you get it done, and while you are at it keep alert for lines of cleavage where the problem seems to fall apart into one or more pieces. These may give you a clue about how to organize your efforts.