Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

What's wrong with re-inventing wheels

by jimt (Chaplain)
on Jul 10, 2006 at 20:23 UTC ( #560216=perlmeditation: print w/replies, xml ) Need Help??

Seriously. What's the harm? I was reading RFC: Templating without a System and the usual debate ensued about re-inventing a templating language, what's the point, use solution Foo::Bar, blah blah blah. Why does everyone get their panties in a bunch?

For starters, the whole term is misleading. "Don't re-invent the wheel." The silly thing is that the wheel has been re-invented multiple times. Didn't we start off with solid wooden discs? Carved stone? Then we moved to carved wheels. Carved wheels with metal bands around the edge. Metal wheels. Metal wheels with rubber tires. Metal wheels with pneumatic tires. I sure don't go around on my bicycle with stone wheels, I use one of the re-invented ones.

The argument is even sillier when someone is told not to re-invent the wheel because there are existing solutions. For the above discussion, it was "Don't write your own templating system, use..." HTML::Template? Template::Toolkit? Mason? HTML::Seamstress? It's a fine case of there not being a single The Solution, there are multiple solutions.

And the solutions are slightly different. They were crafted with different goals and different points. Personally, I don't care for Template::Toolkit and its Yet-Another-Template-Syntax. But that's because I've always worked in shops where the perl guys (and only the perl guys) do the template code as well and I don't have a need for it beyond an artificial restriction of trying to use the separate language to enforce code separation. I can see why it'd be easier to teach to non-perl folks, though, so I get it. HTML::Template tackles the problem differently. Mason slightly differently (and with more involved, admittedly) and on and on and on and on.

So, the wheel is already being re-invented. A lot. All the time. Concurrently. Perhaps the discussion could morph into "which wheel should I choose?" but that's hiding the issue.

From my perspective, it seems to come down to some mystical class of programmer that's allowed to re-invent wheels and then people say, "Hey, that's neat!" and then their wheel is rattled off as one of the ones that you should use when the problem occurs. But how does one enter that mystical class of programmer?

It doesn't appear to simply be inventing a new wheel. That's usually responded to with scorn and admonishment and the tacit assumption that this new wheel can't possibly be as good because it's not widely used or well known, or simply because it's new. If you want to contribute, people are told, you should add onto an existing product.

And you know what? Maybe this new creation of yours isn't that great. Maybe it sucks. Maybe it's slow or not as general or does have bugs that the competition has long since quashed. You may be the most phenomenal programmer to walk the face of the earth, but nobody's perfect. Something went wrong, and it's silly to claim otherwise.

But maybe, just maybe, this new wheel is superior. Or will become superior, at least. Those pneumatic tires that we all use were horrible when they were first invented because they frequently popped and went flat and needed to be changed. Sure, when they worked, they were okay, but nobody ever got a flat tire on a wooden wheel with metal rims. But the people excited about it kept at it and now they're pretty reliable.

I firmly believe that the best and easiest software any of us will ever use has not been written yet, and there are good odds it won't be written by any of us. But somebody is going to come along and do it at some point, and it's probably going to be similar to some existing product, that was at least partially borrowed or re-created or alterred.

What's it going to take to get you (the generic, general you) to look at one of these new projects that merely re-invents a wheel to see if it has merit? What's the harm? Maybe a few hours? A few days? What happens next?

Maybe the software sucks. You tell the author and he gets pissed and doesn't talk to you. No harm, no foul - he would've been pissed if you'd just dismissed it out of hand. Maybe there's some merit to it, but it's rough around the edges, and you make some suggestions and he goes off and implements them and comes back. Then are you interested in it? Maybe there are more bugs that he should fix and he does. Then are you interested?

How many iterations would it take? Surely there must be some subset of developers out there that like creating their own things, if merely for the joy of creating them than nothing else, and surely there must be some good portion of ideas that they have in their new wheels. Is it really worthwhile to always dismiss them out of hand without review?

And if it's not, then what can these new wheel inventors do to get people to look at their thing? If it's shiny and new and they're impressed, then what's the harm? Heck, maybe they solved a problem that an existing wheel didn't and it could get integrated into an existing system and you'll miss it if you don't look.

Finally, it's also not a waste of time. Now, if you don't have the time to spare, that's one thing, I guess. But otherwise, you'll either make the person go away because their idea is half baked, you'll help them become a better programmer which helps everyone, or you'll see that their wheel is a superior solution and win out tremendously.

Replies are listed 'Best First'.
Re: What's wrong with re-inventing wheels
by philcrow (Priest) on Jul 10, 2006 at 20:50 UTC
    I like your point that new versions of the wheel are invented at intervals and that the genuine improvements are adopted. It's just that we've all seen home rolled templating systems, in part because we've written a few ourselves, and they were bad. Seeing so many of them has raised a big red template replacement button which has a hair trigger by this point.

    Expirementation gives us things like XML::Twig, but it also gives us those template systems that lie about the legacy areas of our shops, requiring occasional painful work. So, we advise against starting from scratch based on one or a few quibbles with existing approaches.

    Also remember that for each inventor who succeeeded, there were dozens of others in the same period of time, worried about the problems. Many of those, whose ideas died, thought themselves unfairly persecuted, some even imagined that their ideas were so powerful that entrenched interests arrayed against them. Most were delusional about those claims.

    Generally, we try to discourage people from pursuits that might lead them to the funny farm. Instead we try to engage them with the community. If, in the end, they don't find our advice helpful, off they go to do their own thing. It might be fabulous, but most likely it won't be.

    In summary, please bear with us when our crack pot path alerting sensors peg. But don't think we will always reject a better mouse trap, it just might take a bit of convincing.

    Phil

      It's reasonable to say that choosing templating systems as the rallying point was not the best move, but it's what got me rattled up, anyway. :-)

      And you are absolutely correct - the crackpot to genius ratio is quite high. Being skeptical is quite healthy. It's the cynical part I don't like.

      I guess the other point I want to raise and seek to discuss is - how are the geniuses expected to differentiate themselves from the crackpots? If the world at large isn't even willing to look at your better mousetrap, then what do you do next?

      Some people toil on in obscurity, content that it works well for them and the public be damned. Some get furious and abandon the project. Some get violent and argue against their attackers who don't see the light of the New True Way.

      And in the end? Meh. If you toil on in obscurity, you're somewhat satisfied at the end of the day, because you have the superior product (at least for you and your own uses). But the world sure wouldn't be beating a path to your door. The other responses up above there would benefit no one.

      Perhaps it's more of a marketing and product development question - if you've built your better mousetrap, how can you convince the public to at least take a look? If something existing is truly the superior product and you don't at least fulfill a niche, then you'll be ignored and relegated to niche status, being labeled eccentric (if you're lucky). And if it's good, then hoorah - off you go to power and glory. But no one will look, then you're just completely spinning your newly re-invented wheels.

        Being skeptical is quite healthy. It's the cynical part I don't like

        Well, let's put it this way. For every great new, improved idea, there are dozens, hundreds, thousands of wannabees. After hearing many dozens of times about how great something is, and getting disappointed (or burned, if you believed them without checking), a bit of cynicism is to be expected.

        Your question is much broader than programming, of course. How does anyone ever tell the difference between the genius and the crackpot? To reject the claim is a self-fulfilling prophecy. You deny that they have a valid idea. Nothing comes of it. Therefore, it wasn't valid. It's fallacious, but safe, because chances are good you won't be proved wrong.

        If you accept the idea, then you risk it being shown to be invalid. If you've committed lots of resources to it, you may be harmed - lose time, money, reputation, whatever.

        I once read a comment by a physicist (I don't remember the book or the person). He said something to the effect that he wasn't sure he'd have recognized Einstein's papers of 1905 for being works of genius. He might have dismissed Einstein as being a crackpot, with his crazy ideas about space and time.

        For every Einstein, there are endless Bozo's out there. If you have no better way of judging, then defaulting to skepticism will almost always make you right.

        At the end of the day, only time will tell whether a new idea is a better idea. People will have to try it out, test it, play around with it. There are probably many unsung geniuses out there whose ideas were ridiculed, ignored, maybe even forgotten. That's unfortunate, but it's bound to happen. We are likely to err, false positives and false negatives being unavoidable. But the price of a false positive is immediate and can be severe. Jumping onto a new, but wrong, idea can be very costly, and you'll usually find out fairly quickly that you've been burned. Ignoring a genius might be more costly to the world in the long run, but it's safer for the decision-makers, and the error won't usually show up while it matters to them. Or does that sound cynical? ;-)

        ... how are the geniuses expected to differentiate themselves from the crackpots? ...

        That's what open source and peer review are all about. The code speaks for the coder (sometimes the coder gets to speak for him/herself too, when asked), and the community sorts out the competent from the incompetent.

        Of course even in this arena, whether a good piece of open source code really gets the usage and notariety it deserves may still be a hit-or-miss, luck-of-the-draw kind of process. (And it's possible that some really poor code gets into the "standard tool set" for a while, but sooner or later, it will get phased out.)

Re: What's wrong with re-inventing wheels
by chromatic (Archbishop) on Jul 10, 2006 at 21:54 UTC
    What's the harm?

    Technical risk. Did you read my concerns over inventing Yet Another Templating System?

    I've done it myself. I did it because I didn't take the time to understand the existing systems. If I'd continued to use it, I would have had to add more and more features and make the code worse and worse to handle those features.

    I don't doubt that eventually I could refine my design into a decent templating system, but I'm not in the business of writing or maintaining templating systems. It's far from what I need to do and very far from what I want to do and it's a huge risk for me to get sucked into such a task.

    I don't object to people trying to improve the world, or at least a small piece of it. I do object to people doing their own thing because they don't understand and won't take the time to learn prior art. That's a silly waste of time.

Re: What's wrong with re-inventing wheels
by perrin (Chancellor) on Jul 10, 2006 at 22:10 UTC
    Usually, people say they are writing a new one because it's faster or easier or does something that the old one doesn't. Usually, they are also flat-out wrong about these assertions, and would have known it if they had spent a little more time looking at the existing solutions.

    It doesn't harm me if someone who I don't work with writes their own templating system. But what if they put it on CPAN and add to the pile of half-baked unmaintained solutions that swamp anyone searching for a templating system there? What if I have to work on some of their code someday, and it uses this monstrosity?

    And the more subtle issue is, how much better might TT be if people tried to improve it instead of writing their own half-assed version that gets tossed out in a year?

    There are real reasons why re-inventing the wheel is usually wrong. Ask anyone who had to support someone else's bespoke templating system at their company. It's not just a knee-jerk reaction.

      It would have been harder for me to make the necessary improvements to TT than it was to write my own. So I did. And my solution was better, in that it did everything I needed it to (just like TT would have) in a zillionth of the time and memory.

      These days I use TT because I have a faster server and so no longer have to care so much about speed and size. This is one reason why lots of people now use Java when they would have previously used C. It is trading elegance for convenience, and I don't like it.

Re: What's wrong with re-inventing wheels
by grinder (Bishop) on Jul 10, 2006 at 23:43 UTC

    I see the point you're trying to make, but I think choosing templating systems as the point of focus misses an obvious point: invent wheels by all means, but make sure that you're not trying to invent something in an overcrowded domain.

    Templating systems are not difficult to write, they don't have to do much, and there aren't many different ways of doing it. It's just a glorified macro processor. And Perl is naturally good at that sort of stuff. Writing the 85th templating system is not going to set the world on fire: you are never going to achieve significant market share. Inertia, and worse is better will kill you before you build any momentum up.

    Now a place where I'd be happy to see a wheel or two invented is in the realm of LDAP processing. Net::LDAP is great, but it's low level. It takes half a dozen statements just to get an encrypted session going. The error handling is bezerk. The LDAP query language has a funny lisp-like syntax. But you specify what you want to get back as a list. Some of the things you get back have one value. Some have one or many values. Some always have many values. Sometimes you want to move a bunch of leaves in an LDAP branch according to some criteria. Sometimes you want to do search and replace missions. Sometimes you want to add an attribute to a whole pile of objects.

    Doing these sorts of ad-hoc operations with a few lines of user code is my grail. It is really, really hard. With Net::LDAP, you wind up writing a lot of make-work code. I've been writing a lot of bespoke code to do this stuff for a few years now. Despite my best efforts and stepping back and looking at what I write, trying to synthesise a more generalised approach that doesn't sacrifice flexibility, the solution remains elusive.

    If you come up with a concise method for specifying general LDAP operations with short, idiomatic Perl code and the world will beat a path to your door. Write a new templating system, and no-one will ever hear you.

    • another intruder with the mooring in the heart of the Perl

Re: What's wrong with re-inventing wheels
by Tanktalus (Canon) on Jul 11, 2006 at 04:46 UTC

    What's wrong with reinventing wheels, he asks? Absolutely nothing. If.

    If you have properly and fully evaluated existing solutions. If you have discussed with others experienced in the field what needs you believe these solutions are not meeting. If you have confirmed that they do, indeed, fail to meet those needs. And if you can add new features without introducing new bugs to make your better wheel.

    With real wheels, each progression you stated was most likely made in a similar fashion: existing wheels were evaluated and found wanting. And then were tweaked into something new. Don't forget to do likewise here.

    Now, if you were trying to do this as a way to learn how such templating systems are developed, go ahead. That's done in many other areas as well, not just software. But, here as much as anywhere else, when your learning is done, you're best to work with off-the-shelf standardised tools when you want to get real work done instead of reinventing what is already out there.

Re: What's wrong with re-inventing wheels
by spiritway (Vicar) on Jul 11, 2006 at 05:20 UTC

    I think reinventing the wheel has its place. As you point out, that's where many improvements come from. OTOH, for every decent new wheel, there are many crummy ones that simply don't work, or that are worse than the ones they try to fix.

    Part of the problem, IMNSHO, is that we don't always see the difficulties when we're looking at a program from the outside. We think of an "obvious" solution, not realizing that there is some compelling - often overwhelming - reason why that solution doesn't work for this problem. Even as we delve into the problem, we might not recognize that our solution is faulty - or, we might not be willing to admit that we have to let it go. I can't speak for anyone else, but I get very attached to my ingenious solutions, and have a hard time recognizing when they're flat-out stupid.

    One reason why there are so many wheels in CPAN is that many people write less than optimum code and upload it, not understanding all the finer points of the problem. This creates problems for people who want to use a module for something - how can they tell which ones are decent, which are less than decent?

    This is where Perl Monks comes in... you can ask people here, and get a good idea of what really works. This is a real life-saver if you don't have the luxury of testing each module yourself. But the point is, having multiple wheels for the same thing often makes it difficult to find quality solutions. You can waste a lot of time trying to get some half-fast module to do what you need, only to find that it's simply the wrong tool for the job.

    I think it's a great idea to reinvent the wheel for your own edification - to learn how a problem might be solved, to get a feel for what's involved. That might even give you an idea of what to look for in an off-the-shelf solution. But I also think it's unwise to upload it without running it past a whole lot of people, and listening carefully to their criticisms (such as, "it doesn't work", or "it's no better than the others").

    I don't think most people deny the *possibility* that you might have built a better mousetrap. It's just that, like so many other things, the odds are against you. Maybe you could write a better sort program - but I wouldn't bet on it. Most of the solutions around have been reworked, optimized, and otherwise polished until there isn't much (apparent) room for improvement.

    I think that another consideration - a vital one - is why you're writing programs to begin with. Professional programmers often (probably usually) don't have time to play around with unproved solutions, just to see if they'll work. They also don't have time to write things themselves, that already exist. Sure, if the modules just don't do what they need, of course they'll write it, or rewrite an existing module, or something. But because they have deadlines and schedules (and get tired, and sometimes even have a life) they want to work efficiently and competently. It's safer and less time-consuming to use what's already out there.

    Amateurs have the time to play around. They can take as long as they want to get something right, or reinvent a wheel. They can look at new modules and hammer on them, see when they break, offer feedback to the programmer, all that good stuff. They can beta test stuff that professionals simply can't. Our goals are different - the pro needs to get work done for bosses or clients, and it better be right. Amateurs don't have these pressures. They don't lose money if they take forever to get something written; they don't get fired if their program decides to trash the hard drive. So yes, amateurs can putter around with things that the professionals can't.

    Unfortunately, the range of competence among amateurs is wider than among professionals. Most pro's can't remain incompetent for very long. They either learn, or lose their jobs (unless they work at certain larger software houses whose naMeS will not be mentioned). Amateurs have no such limitations. Many amateurs are highly competent - in fact, many are also pro's just having fun in their spare time. Some, though, are frighteningly incompetent, yet they manage to figure out how to upload to CPAN. This is where the harm is. And this is one very important reason why we need Perl Monks.

Re: What's wrong with re-inventing wheels
by eric256 (Parson) on Jul 11, 2006 at 06:07 UTC

    It is quite simple actualy. If your bike/car gets a flat tire do you: a.) repair it, b.) replace it, c.) invent an entirely new wheel

    Most people of coures will try A and B in that order. Some will skip straight to B. Others (our geniuses and crackpots) will consider a and b, then decided that C is going to be more fun! So if you skip to C we are going to consider you either a genius or a crackpot, that decision will be made on the reasoning you present.


    ___________
    Eric Hodges

      The expedient thing to do (in this case) is replace the wheel (or dig around in CPAN and evaluate a few, depending on your metaphor) and continue on.

      But at SOME point you have to decide if it is ok for you to simply trust in "The Magic of Wheels".

      You can dig around in someone's module all you like; run test cases and nod at little green lights (or green lava lamps if you're of that religion.)

      But how can you stand behind a (sub)?system you're creating if when asked how it works, you have "and there's this bit over here I got that I use for soandso and it seems to work well."?

      I've written a couple thoroughly incomplete hackish expedient templating systems for personal use in my time, being pretty damn self-assured that it would be easy. But unfailingly I've found that there are some interesting wrinkles to be ironed out.

      Now I know enough to really evaluate other templating systems. There's no way I could have before.

      Turns out mine is pretty much never the right solution. But I sure wouldn't have known it without writing it.

      To do it or not to do it depends too much on the context in which you're operating to resolve to Always or Never.

        "and there's this bit over here I got that I use for soandso and it seems to work well."

        That is an argument agianst all modules good and bad, big and small, core and non-core. Realy though it doesn't hold water. In any langauge you use peices written by other people, maybe its a sort algorythm, a linked list library, or some other such thing. In all cases you have to combine a bit of trust, a bit of research, and a lot of faith in the community that these thing work. This is part of the reason agianst lots of template systems. Right now it is easy to say (and remember) that TT is good for one thing and HTML::Template is good for others and maybe sometimes module Z. It is also easier to research them because there arn't tons. If however we let many many bad template modules get onto CPAN then we have a whole forest of bad answers with a few hidden flowers. So your issue with trusting code is part of the very heart of not wanting tons of Template wheels rolling around.


        ___________
        Eric Hodges
Re: What's wrong with re-inventing wheels
by tinita (Parson) on Jul 11, 2006 at 10:38 UTC
    just a recent bad example of reinventing the wheel: http://search.cpan.org/~abalama/Template-1.21/

    i know this is a bit offtopic, but to get a relation to the topic:
    there needs to be a review like perlmonks. otherwise code like the above does its way to CPAN.

    it's really important for a module author to hear "don't reivent the wheel" and "couldn't you just use module Y instead?" otherwise people will upload stuff like the above which is really bad because of several reasons (name clash, bad documentation (i can't see a simple working example, just fragments of code), no strict and warnings, no tests, no comparisons to other templating systems). actually, when i saw the above module, i wished there was a possibility to delete code from CPAN. the author itself doesn't care about the mentioned problems.

    if you think about the questions you get asked as a re-inventor you either stop reinventing the wheel or you figure out why you have to do it and document it.

Re: What's wrong with re-inventing wheels
by ruzam (Curate) on Jul 11, 2006 at 02:54 UTC
    Say it loud, say it proud! The rest of you guys can throw up your arms and cry it's been done before. I for one will re-invent until I'm old and grey.

    There's ALWAYS a better mousetrap out there somewhere. Maybe it won't come from me but you can bet it won't come from anyone who settles for the one that's been done before.
Re: What's wrong with re-inventing wheels
by qq (Hermit) on Jul 11, 2006 at 19:22 UTC

    It is easier to understand code that you have written yourself than code you've got from CPAN, and I think that is often a motivation for reinvention. Not because your own code is better, of course, but simply because you've thought into it, and because the code itself is an expression of your own idea. This is a trap for decent, but not great, programmers.

    I think you can legitimately reinvent large projects only if you have a really thorough understanding of the existing projects. Which is hard, and time consuming. But is also likely to increase your skills as much or more than implementing something half-baked.

Re: What's wrong with re-inventing wheels
by dorko (Prior) on Jul 10, 2006 at 23:47 UTC
    Quoth the poster:
    What's wrong with re-inventing wheels?

    I would contend it is exceedingly difficult to get right.

    Cheers,

    Brent

    -- Yeah, I'm a Delt.

      Or restating using the obligatory Simpsons quote:

      I guess one person can make a difference, but most of the time, they probably shouldn't.
      – Marge, Itchy and Scratchy and Marge
Re: What's wrong with re-inventing wheels
by davido (Cardinal) on Jul 11, 2006 at 07:30 UTC

    If you're interested in re-inventing wheels, don't stop at a new module that does something slightly different from an existing one. Noooo. Start with writing your own version of Perl. Too much work? Ok, so there's a limit then.

    But maybe that's not your limit. ...maybe you want to create a new language. But first you get sidetracked on finding a better way to build a microprocessor. But as you start working on that, you realize that a new transistor design would be fantastic, and before you can do that, vacuum tubes should have been designed slightly differently for better longevity.

    My point (that I'm making in a rediculous way) is that you can easily sidetrack a very good project by wasting vast quantities of time building specialized tools, when generalized tools would do the job just fine. NASA used to carry the "invent your own" philosophy to an extreme, and the result was $10,000 hammers, $30,000 toilet seats, etc. If you can ONLY accomplish your task with a $10,000 specialized type of hammer, you're going to have to innovate and make one. But if you can get the job done fairly well with a $15 hammer from the hardware store, you'll have some budget and time left to actually build the rest of the house.


    Dave

Re: What's wrong with re-inventing wheels
by explorer (Chaplain) on Jul 11, 2006 at 17:59 UTC

    Re-inventing wheels IS wrong...

    Wheels IS the CONCEPT.

    Re-inventing the PROCESS for to make BEST wheels is the correct.

    Example:

    Concept
    Templating System for dynamic web pages
    Traditional solution
    Perl CGI scripts from server side, with modules for template systems.
    Re-inventing the solution: Rasmus Lerdorf, 1994
    Small set of Perl scripts, followed by a rewritten set of CGI binaries written in C in 1994 to display his résumé and to collect certain data, such as how much traffic his page was receiving. "Personal Home Page Tools" was publicly released on 8 June 1995 after Lerdorf combined it with his own Form Interpreter to create PHP.

    We know the result...

Re: What's wrong with re-inventing wheels
by shmem (Chancellor) on Jul 17, 2006 at 09:26 UTC

    May I add my 0.02$ as the author of RFC: Templating without a System?

    IMHO there's also one compelling reason for re-inventing wheels - if you realize that existing wheels are not wheels, but either complete vehicles or wheels with brakes, or axles, or self-inflating tyres, whatever, i.e to get at the wheel. You need not delve into the making of the car to tell it from a wheel. Providing a basic wheel on which most agree that it's basic may result much more difficult than providing a complete group of axles with steering device and electronics, the more as the latter is handier to assemble a car with, and nobody seems to need a bare wheel in the first place.

    --shmem

    update: changed axes to axles (thanks wfsp), I didn't mean tools for ork slaying :-)

    _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                  /\_¯/(q    /
    ----------------------------  \__(m.====·.(_("always off the crowd"))."·
    ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlmeditation [id://560216]
Approved by Corion
Front-paged by Old_Gray_Bear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (2)
As of 2022-05-21 00:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you prefer to work remotely?



    Results (76 votes). Check out past polls.

    Notices?