Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Virtues of Software Engineering

by vladb (Vicar)
on Dec 28, 2001 at 01:03 UTC ( [id://134719]=perlmeditation: print w/replies, xml ) Need Help??

As I read through a pretty decent book on software 'construction' ("Code Complete"), I have a few questions that the book leaves scarcely covered yet are somewhat concerning to me. For one, I'd categorize myself as a self-taught hacker. I didn't get to finish my degree in Computer Science (was hired in my second year, and offer wasn't bad so I thought to slack off ;-) and therefore may not be aware of certain things that go into constructing a sound software piece/program. This book, in particular, reinforces certain practices that if followed and executed properly will yield in a more coherent software product (it's architecture, code implementation etc). To prove it's point, it also mentions certain statistics covering and case studies that reveal that only proper execution of those software engineering principles will result in productive and cost effective projects.

The actual question that I have to fellow monks is what is their approach in building particular program? I'm specifically interested, since Perl is a language that is very tempting even to the point where one (such as myself) is more willing to jump into 'building code' sooner than later therefore foregoing certain design steps etc. Also, this urge to start coding earlier is also spurred by the fact that prototyping in Perl is somewhat easier than such languages as C, for example.

I'm sure there's a lot of people who at one point in their career found themselves in similar position that I am in now (quick to hack, slow to think first ;). I'd appreciate it very much if fellow monks could share into their experiences and why 'formal' approaches to 'software engineering' (e.g. requirements specification, functional specs, architectural design before actual 'coding' stage) are almost always guaranteed to yield better results. Also, if someone has a few examples where the reverse would work better. I hope this will be a fruitful discussion ;).



"There is no system but GNU, and Linux is one of its kernels." -- Confession of Faith

Replies are listed 'Best First'.
Re: Virtues of Software Engineering
by chromatic (Archbishop) on Dec 28, 2001 at 01:59 UTC
    My approach is pretty informal. I'm currently writing a Slash plugin for an article to plug an upcoming book. After coming up with the idea (an entirely new subject in itself), I did a little design.

    Most of that design is breaking the problem into tasks. Each task is supposed to be a standalone piece of functionality that can be accomplished in under two hours. This is a brainstorming session, where I figure out what's really necessary and often come up with useful enhancements. I rarely put the enhancements in, the first time. I end up with a small text file similar to:

    - display a page - current version - no arguments - op=display - ver=version number - list of previous versions - author - changes ? - links - restore previous version - only for administrators - delete everything later ? - edit a page - allow wiki markup - save rendered version - save previous versions - access control - restrict based on seclev - optional restriction based on user flag - index - list pages - search pages - recent changes - markup parser Schema ------ - wikipage - user (uid) - title (name of page) - date (change date) - version (autoinc) - description (char 255) - wid (wikitextid) - wikitext - wid (wikitextid) - text - rendered - version
    They're arranged in *rough* order of importance. Each indentation level allows further detail. I don't keep many design notes at this stage, but occasionally I'll think of something that may work.

    Then I go on to coding. I usually write tests before code. This time, I was revising an existing bit of code, and wrote the tests later. That made for more debugging time and brought shame to my ancestors.

    I have the display and list bits coded, and the parser is there as well. I've left the database for a bit later, but that's probably next on my list as I put in the edit/save files.

    Does this make any sense at all?

Data first. Code later. (was Re: Virtues of Software Engineering)
by chip (Curate) on Dec 28, 2001 at 04:57 UTC
    Fred Brooks, in The Mythical Man-Month, hit the key principle that always guides me:

    Show me your (code design) but hide your data structures,
      and I will remain as ignorant as before.
    But show me your data structures,
      and I won't need your (code design); it will be obvious.

    Or as Deep Throat might have said: "Follow the data."

        -- Chip Salzenberg, Free-Floating Agent of Chaos

      I prefer running that through Joseph McNamara: It's the data, stupid!
Re: Virtues of Software Engineering
by grinder (Bishop) on Dec 28, 2001 at 03:51 UTC
    I don't write anything big any more, that way I don't have to bother about design :)

    I should qualify that remark, things are never that simple. I used to work on huge monolithic systems, with data dictionaries, naming conventionsm APIs and library versions, the whole software engineering circus. Heaps of time was spent up front using reams of papers to produce reports that were then put on a shelf and never consulted again (I am barely exaggerating).

    These days, the "design" is much more diffuse. It's not only in the code. It's in the router configuration, the firewall rules, it's reflected in the database, in fact, all the systems the company uses have to be taken into account. The systems I have to deal with are small pieces loosely joined (a term stolen from David Weinberger). And Perl is a natural fit. Taking a new feed from the accounting system into the payroll. Dealing with bank transfers in a new protocol. If I can't write it in a day or two, it's not worth doing. There must be an easier way. A program that can be written in a day or two is easy enough to keep in your head. Once it's done I write a technical paper, drawing up the loose ends, what assumptions were used, what to look for if things go wrong.

    The one thing I do do, however, that greatly improves the creation process, is to make sure that the program always compiles. At the lowest level, this means that when I start typing in a control structure, I type the control skeleton first, and then go back and fill in the statements. Something like:

    if( defined( my $r = foo() )) { } else { }

    Once I enter that opening brace I keep on going until I type the closing brace following the else. The point being that were I to save the file at that instant, what I just typed would not cause any syntax errors.

    By extension, I do the same thing with resource acquisition, such as opening a file, and then closing it. So I'll type something like

    open IN, $filename or die "Cannot open $filename for input: $!\n"; close IN;

    I will then go back and change that to

    open IN, $filename or die "Cannot open $filename for input: $!\n"; while( <IN> ) { } close IN;

    ... and from there...

    open IN, $filename or die "Cannot open $filename for input: $!\n"; while( <IN> ) { next if //; } close IN;

    And then go back and fill out the regexp. And so on. I believe that writing code this way is a good habit to get into. I can often type in 50 or 60 lines of code and have it run on the first try. That, as much as anything (not having to go back and fix up silly errors like missing a closing parenthesis and thus losing track of your train of thought) is a big productivity boost. I think this is part of what Code Complete was on about (it's been years since I read it).

    Template-directed syntax editors, if you can find one you feel comfortable with, can be a great help. Personally, I hate them, but some people swear by them.

    --
    g r i n d e r
    just another bofh

    print@_{sort keys %_},$/if%_=split//,'= & *a?b:e\f/h^h!j+n,o@o;r$s-t%t#u';
Re: Virtues of Software Engineering
by FoxtrotUniform (Prior) on Dec 28, 2001 at 02:01 UTC

    The first thing that I do when I start working on a new project is to sketch out a design. Not a complete spec, but I try to at least write down what I'm supposed to be doing, what kinds of input I'm expecting, what kinds of output, and so on. (The next thing I do, once I have some idea of how I'm going to solve the problem, is hit CPAN.)

    Why does this work better than just sitting down and typing a shebang and going from there? It works better because I can more easily break the problem down into steps, and solve each smaller problem one at a time. I can think about which parts of the program need access to what data, and keep my code more loosely coupled as a result. (That way, when I have to change something, it's less work.) Because I put some thought into how I'm going to solve the problem, it's easier to figure out which modules on CPAN are going to be most helpful, and which ones are nifty but tangentially related to my particular problem. In general, this way works better because it gives you a better understanding of the problem.

    On larger projects, my design docs include prototypes and mockups. You can still start coding right away even if you don't start coding right away. :-)

    --
    :wq
Re: Virtues of Software Engineering
by talexb (Chancellor) on Dec 28, 2001 at 09:29 UTC
    Describe what you want to do, think about the inputs and the outputs. Do that as much as possible while staring out the window and drinking coffee|tea|cola.

    Don't start work on it yet. Do other stuff. Let your brain come up with clever ideas. Jot them down. My DayTimer is always to hand (like right now) just in case something pops up.

    Talk it over with someone. Don't write code yet or think about building tables. Bad idea. Ponder. Think of much better ways to do it, then play devil's advocate and knock those challengers down. Or not.

    Gently, start drawing boxes with arrows. Data diagrams, flow control, UI elements. Try a few test pieces of technology. Get comfortable with the whole idea.

    OK, start doing stuff. Build a table. Look at it a bit. Build another. Write some more code. Tinker. Finish building the tables. Try some of the code. Confirm you're on the right track with your diagrams, your early scribbles.

    Continue, all the while doing testing as if this were a finished product, remembering the pieces you still need to add.

    When finished (you think), leave it for a few days. Don't look at it. Come back with a fresh mind. Impress yourself with the good indentation (see earlier replies), the logical variable names, the clever use of .. something.

    If you've done mostly what you set out to do, make an archive copy of the code, then start using it (or release it). Start fiddling after that. It's so easy to get code working nicely, only to have it fall to bits when you start hacking again.

    Oh, I forgot documentation, didn't I. Umm .. you did comment your code, didn't you?

    --t. alex

    "Excellent. Release the hounds." -- Monty Burns.

Re: Virtues of Software Engineering
by hsmyers (Canon) on Dec 28, 2001 at 10:30 UTC

    “ I'd appreciate it very much if fellow monks could share into their experiences and why 'formal' approaches to 'software engineering' (e.g. requirements specification, functional specs, architectural design before actual 'coding' stage) are almost always guaranteed to yield better results.”
    </body>Well first off, think of it this way, imagine the same project without any of the above. Why would anyone expect this to work? Perhaps with a great deal of talent something could be salvaged, but it would be up-hill and against the wind all the way. Not that it doesn't happen this way in the Real World(™)—of course it does. But take the same crew and give them the advantage of actually designing the product, stand back and watch them fly!

    I suppose it depends on your background, i.e. how you came to the art. You talk of yourself as a hacker or at least self-taught. And you certainly show some of the symptoms (many around here and elsewhere should seriously start a 12-step program real soon now), particularly your preference for 'code before design'. I don't think that is necessarily a bad thing, the most likely reason is that you simply love to code. I would recommend only a slight change in course, keep the love for coding, but explore the hell out of design. Design is your friend, think of it as an early warning system that can alert you to the presence of Bad Things! As it turns out, there is not a lot of difference between designing and coding. Just different pigments and maybe paper versus canvas, but it still is yet another form of the creative process and can provide a 'fix' just as powerful as a good hack any day.

    I mentioned background as a strong influence on hackerhood. In my own case I came from an arena where 'design' was not optional, not required, just flat out mandatory. I'd spent a number of years as a silver-smith and not being independently wealthy, every commission required that the piece in question be spec'ed, designed and executed all on paper first. And at that, well enough to make the sale! Think of it this way, silver and gold cost way too much to whip up a piece on the off chance the client might like it. You might increase your chances of a sale with a good portfolio, but even that is just paper—photographic paper true, but still paper.

    So how does that condition my own programming process? Well, not too surprisingly, quite a lot! I don't do renderings of screen shots, but one way or another, I usually know what the product will look like before I cut my first line of code. Occasionally I cheat, for instance, I have no problems with using VB to work out the entire menu structure before hand. And similar techniques to give as realistic a picture of the product as I can without having to actually build the silly thing before its time. But once it's firmed up, approved by the client and by me, then on to coding. And the really remarkable thing is that it is a good deal easier to code when you know what you are building. Not to mention faster! And did I mention guilt-free? After all you're doing it by the book, even those software engineers at ACM and IEEE would approve…

    –hsm

    "Never try to teach a pig to sing…it wastes your time and it annoys the pig."
Re: Virtues of Software Engineering
by Fengor (Pilgrim) on Dec 28, 2001 at 13:39 UTC
    Many good things have already been stated in this Meditation so I'll stick with a little real-life tale of how i learned to appreciate Software Design:

    As some of you may know I'm one of those guys who often goes to LAN-Parties. I even helped organizing some and during that I learned to appreciate the Design-Prototype-Redesign cycle.

    The actual case was, that myself and a friend of mine have been in the organization team of DSK-Lan II, a little Lan-Party (approx. 100-130 people) here in Germany. Since we both had some Perl experience we got the ttask of building the inter/intranet site and the scripts for the organizing of tournaments etc.
    Of course we had a list of features in our head that we liked to have implemented and so we startet coding soon. During coding we got aware of some problems with our code and the database table (things that could be optimized or done a better way even some things that HAD to be changed in order to make it work). As a result our code was quite a mess but finally we got it to work (well sort of). But then another problem arose as we tried to merge the different parts of our code and got all the hell of problems with our different naming conventions etc.

    As you can clearly see, this was a very bad way of doing things. But I'm not feeling sorry that I did it that way cause it taught me a valuable lesson: Never underestimate Design issues!
    Meanwhile I have read the book The Pragmatic Programmer (which i can truly recommend to everyone) and learned a lot from it. And since I want to learn from my errors I declared the messed up script as a prototype and started to redesign. Which means that we are now creating a feature list (simply a brain storming of what features would be cool to have) which will be categorized and transformed in a roadmap (covering which feature will be implented in the differend phases of the project and till when this has to happen).

    Well I'm curious if it will work out this time ;-)
    fengor

    --
    "WHAT CAN THE HARVEST HOPE FOR IF NOT THE CARE OF THE REAPER MAN"
    -- Terry Pratchett, "Reaper Man"

Re: Virtues of Software Engineering
by lachoy (Parson) on Dec 28, 2001 at 20:01 UTC

    If I'm creating a new object (or subclassing an existing one) and it's relatively non-trivial, I find one of the best things to do is create code that uses the object. Act as if the necessary modules have already been written and don't worry about how they'll be implemented, just how you'll use them.

    I used this recently for adding import and export functionality to SPOPS, and it seems to have worked very well. Of course, from there it's only one step to coding your tests first. And from there a simple five steps to conquering the world. Who would have known?

    This has two attractive side-benefits. The first is that you tend toward simple interfaces. I don't like complicated interfaces as a user, so I won't write sample code that uses them. This is opposed to my developer-side, which in a vacuum creates kitchen-sink-functionality dealing with cases that will rarely (if that) occur and then pares it down afterward. I'm trying to unlearn this bad habit (practice, practice!), and this helps tremendously.

    The second benefit is that if you have any confusion about what you're doing with your module, creating implementation code will help clear it away. (Other people clear away confusion by being really smart, which unfortunately isn't available through an 800 number...)

    Chris
    M-x auto-bs-mode

Re: Virtues of Software Engineering
by newbie00 (Beadle) on Dec 29, 2001 at 12:14 UTC
    In brief, as engineering students several years ago, we learned 'structured' programming to establish our baseline technique. In essence, each function or subroutine had its own 'theme' or purpose and expected results. Each subroutine was likened to a paragraph having its own 'main idea' so to speak. This type of programming also seemed to 'reduce the clutter' of some programs that are written in a top-down manner (maybe 'clutter' is not the word I am looking for -- it may be that 'structured' programming makes it easier to see the 'overview' of the program and then be able to view the details of the subroutine when and if desired). Yes, you must jump to the subroutine to see what it does, but all of the code associated with that routine is in one place as a 'module'. At times, you may not need to see the detail of each subroutine in viewing the program's logic. Sometimes, just the 'overview' of its structure is needed. Also, subroutines can reduce 'code duplication' within a program.

    Each 'module' could be tested before adding another 'module', and so on. It just seemed easier to add 'modules' than to squeeze multi-lined code between other code.

    Adding variables instead of hard-coding PLUS adding a sufficient amount of comments which explain what the code is to do, etc., makes it easier to adapt certain 'modules' and snippets to other programs as well as to make it easy to go back to a program in the future and quickly determine 'what' is going on and 'why'.

    For me, it also seemed to make it easy for me to learn multiple programming languages, just by adapting the syntax to the structure of the code, and by looking at the 'overview' of other programs written structurally, as a guide. Of course, this can also be true of 'top-down' coding, but structured is clearer to me due to its 'overview' nature.

    What helps me to formulate the 'logic' of a project, depends on the project. Sometimes, creating an 'outline' as though I am writing a technical report is appropriate. At other times, creating the 'screens' geared toward what the end-user is going to see may be appropriate for another project. At other times, I may use a combination of the two. Next, I may create a flowchart whereby I also create 'what-if' scenarios. These 'what-ifs' may be for the purpose of minimizing security concerns or to minimize errors that the user may encounter if they enter data incorrectly (this helps with creating a more user-friendly environment). Flowcharts greatly enhances the building of the program's logic so that it is more efficient and self-explanatory to the user, others, and to me. Creating notes for future user and 'admin' documentation should also be done during the program's development. Trying to recap at the end leads to 'weaker' documentation.

    Sometimes, depending on the project, I may begin coding as my first step, because I could see the project clearly in my head, and then document and design as I go along, to carry me to the next step. It all depends. I guess this is the same part of my brain that allows me to hear a song and 'play it by ear' on the piano. That is why I enjoy designing (not just programs but anything mechanical, processes, etc.) because it combines the analytical with the creative juices to form a nice end-result. I guess it is the 'freedom' of thought that is my 'carrot'. Sometimes, rigidity and an overabundance of constraints stymies creativity. One of my previous boss' (an product design manager) philosophy was, for us to design WITHOUT the budget being a constraint. Most times, we would not know if there was a specified budget for a given project. He felt that a better product would be created without adding the monetary constraint which tends to restrict creativity. He preferred to streamline the product's definition AFTER the design in order to fit the budget, otherwise, maximum creativity may not be achieved.

    Well, this is it in a nutshell. Structure is great; outline the specifications up-front as required or practical or to use as a guideline; define the expected outcome; reducing the likelihood of operator error; increase security; and optimize the 'flow' and efficiency of the code; and increase the software's 'user-friendliness' as practical. Continually revisiting these items throughout the software's development is a must to any successful project. How you get there is most times an 'individual thing'. That is the best part. I don't have to think like someone else to design whatever. The project and I become 'one' (as much as practical) until I and/or others know it is 'good and very good'...

    Hope this helps.
    --newbie00

Re (tilly) 1: Virtues of Software Engineering
by tilly (Archbishop) on Jan 05, 2002 at 11:17 UTC
    I am all over the map, depending on the problem at hand. But for the most part I tend towards incremental ongoing design. A random link on that topic is Why Late-Binding Is Good.

    For a smorgasborg of design options you could do a lot worse than to pick up another book by Steve McConnell, Rapid Development.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlmeditation [id://134719]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others rifling through the Monastery: (4)
As of 2024-03-28 18:01 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found