http://qs321.pair.com?node_id=275994

In Re: Re: Re: Re: Re: Slowness when inserting into pre-extended array, tilly raises an interesting point.

How does anyone know how much they know?

How do I know (or perhaps find out) how much of what I think I know is accurate? How much of what I think I know was once accurate but is no longer? How much of what I think I know as a result of some specific experience or test or reference that I have read, is actual knowledge and how much is just conclusions or suppositions that fit the facts as I saw them at the time?.

I originally had that last paragraph worded with "you" instead of "I", but changed to avoid being seen as pointing the finger at anyone in particular., but the questions remain the same regardless of whether its "you" or "I" or "anyone".

I gained a fairly extensive formal education in Comp.Sci. & Math, but that was 20+ years ago. I also have a fairly extensive background in real-world, practical IT, though I been away from that for some months, not by choice and for longer than I would like, and given the pace at which this industry evolves, I can no longer claim to be current even in those fields I was once considered something of an expert. I'm a relative newbie to the world of perl, and even though I've probably racked up an average of over 60 hrs a week for the last year doing pretty much nothing else than perl, there is very limited depth behind whatever level of knowledge I have acquired.

That said, I've probably tackled a wider range of problems, experimented with more algorithms, and explored the minutiae of perl to a much greater degree in that year than I did in all the projects I was involved in, and in all the languages that I used over the past 20 years. Even so, there is no guarantees that what I think I know still holds a few weeks or months after I acquired the knowledge. As an example of this, in a recent thread, I thought that I was saying the "right thing" when I attempted to correct Zaxo in this Re: Re: suffix arrays in the light of information I had 'discovered' in this thread about 4 months earlier. However, as Zaxo pointed out here, my knowledge was about a week out of date. Initially I was chagrined by this, but when I thought about it further, unless the OP was using the very latest release, development build of perl, my information was still current as far as they were concerned, so they were likely to benefit anyway.

This is a constant problem, even for the acknowledged experts in our field. Even Donald Knuth continues to re-release and update and correct his 'definitive' work, meaning that at some point, some of what he previously said with great authority has been superceded.

So the question becomes: How does anyone know how much of what they 'know' is accurate?

The only way that I am aware of is to interact with a community of persons who are active in your field of interest, expound your wisdom as the occasion arises and accept correction when that happens. In many scientific fields this is done through writing papers and submitting them to peer review through industry journals (Nature, Scientific American etc.). In this industry, there are such vehicles as the Communications of ACM amongst others. However, the entry-level for acceptance of ones efforts in journals of this type are extremely high, restricting them in most cases to those who derive an income from the efforts they expend in deriving their knowledge and writing it up. There is also a problem that the baseline for knowledge of highly dynamic and rapidly evolving subject areas like perl are a constantly moving target, and the processes involved in the peer review and publishing of articles through journals is time consuming one.

I believe that the future of such processes is limited and will quite rapidly be replaced by mechanisms more akin to that exemplified by the one we see here at PM. I don't think that PM is perfect yet, but it is a pretty good base to work from. There are probably various ways that the mechanisms that drive PM could be modified and evolved to allow the veracity or otherwise of individual posts and posters to better reflect their true worth, but what we have seems to be working reasonably well. Perhaps the greatest improvement that could be made to the site is for the participants (myself included) to more readily accept correction. The current tendency for this to lead to loggerhead debates and polarisation of feelings is probably more detrimental than anything else.

Just as I believe that no one should be afraid to ask any question, no matter how simple, I also believe that no one should be afraid to offer their 'knowledge', regardless of whether that 'knowledge' is ultimately accurate.

Very few people will knowingly express their knowledge if they 'know' it is wrong, so it is a fair assumption that when you see someone say something, accurate or not, they believe it to be so. The alternative is for people to hold back on saying what they think they know in fear that they might be wrong. And for others to refrain from challenging the knowledge expounded by the few that have enough confidence in what they know to say it out loud, for fear of treading on toes, or being embarrassed or embarrassing others.

Perhaps the greatest trick to cordial relations in this place, is for people to have a sensitivity to the feelings of others. Often it is the manner (or tone) of a correction that most excites the person being corrected. Whilst it is notoriously difficult to determine the intended tone from a textual reply, we are all aware that some ways of writing things are more likely to offend than others. Most people are also quite aware when what they write is likely to be perceived as offensive, belittling or derogatory to the person with whom they are corresponding.

Just as the freedom of speech carries with it the flip side that others must be accorded that same freedom. So the freedom to express opinion, and impart ones knowledge carries a similar flip side. If one expounds knowledge, accept (and relish) that it may be challenged and corrected. If one expresses an opinion, accept that a counter-opinion maybe expressed. You cannot correct someone else's opinion, only counter it on the basis of logic or example. You can 'correct' your own opinions, but only in the light of new knowledge or clarification of existing knowledge, but only if you engage in discourse. There are very few that can arrive at new knowledge, conclusions or opinions on the basis of study or thought in isolation. Isolation engenders dogma and extremism.

Just as a squash or chess player or a racing driver needs to interact with people better than themselves to improve their own game/ skill, so a programmer needs to interact with programmers with wider, different or more in depth knowledge in order to grow. PM is the best example of this I have ever encountered. With less egos and more debate, I think it could be even better.

The only way I am ever going to know if what I think I know is correct is if people who know better correct me when I am wrong.

So please, correct me. I welcome your input. I may not directly accept it at face value, but if I don't, I'm not saying that you are wrong, I'm only saying that I am not sure that you are right and I would like further input to allow me to reach that state of grace.


Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller

Replies are listed 'Best First'.
Re: How do I know what I 'know' is right?
by tilly (Archbishop) on Jul 20, 2003 at 08:54 UTC
    In no particular order, here are some of my thoughts:
    1. I have a sister who is a nurse. In training she was told that every year about 30% of what she was taught would be revised and need to be relearned. She tells me that she didn't believe it then, but she does now since that is about what has happened since. So the problem of knowledge going stale does not just happen in computers.
    2. I have always had a pretty good ability to pick factoids up. I wasn't always able to judge their accuracy. I have a brother who took it on himself to fix that, any time I repeated something he would make me figure out where I heard that and say whether I think that is a reliable source. It wasn't any fun, but it became a habit that has served me well. So the next time I spout some tidbit, ask me where I learned it, and when I respond, imagine me being lectured by my brother, "Now, Benny, you can't just believe everything you hear..."
    3. The need to be able to accept ongoing feedback reminds me of what I was getting at in What you refuse to see, is your worst trap.
    4. I hadn't realized that I had raised the point which started your post, but if you wish to credit me with good thoughts you had while reading something I wrote... ;-)
Re: How do I know what I 'know' is right?
by cLive ;-) (Prior) on Jul 20, 2003 at 07:52 UTC

    I've evolved a few simple rules:

    • welcome mistakes and don't try to pretend they didn't happen - you learn more that way, and leave the bad thought process open to others to see the assumptions you made :)
    • always be paranoid that there's a better/more secure/more efficient way of doing something
    • accept that this is not the end of the world if you can still get the job done (you can always apply new stuff learned at a later date)

    My general philosophy of coding is that you're doing well as a programmer if you know what question to ask. By understanding the question, you can work out the answer. It's when you don't know the question that things go wrong.

    Basically, BrowserUK, I think you're suffering from programmers' paranoia. It's perfectly normal and I'm sure you'll feel better in the morning :)

    .02

    cLive ;-)

•Re: How do I know what I 'know' is right?
by merlyn (Sage) on Jul 20, 2003 at 08:01 UTC
    Every abstraction that is not a tautology contains some lie, and is therefore only applicable to some subset of all cases. The better abstractions cover more cases.

    So, everything I know is a lie. I should always be prepared to eject any knowledge I have in light of some abstraction that covers more of the territory. It's all models, but some models are better than others.

    -- Randal L. Schwartz, Perl hacker
    Be sure to read my standard disclaimer if this is a reply.

      Every abstraction that is not a tautology contains some lie, and is therefore only applicable to some subset of all cases.

      Such as..."all premises, major or minor, involved in the construction of a syllogism are lies".

      Hmmm. Granted, deductive reasoning is only a subset of logic.

      :)

      Matt

        And logic is only a subset of real life ;-)

Re: How do I know what I 'know' is right?
by Zaxo (Archbishop) on Jul 20, 2003 at 05:57 UTC

    As you figured out, your correction was entirely proper and helpful. I dwimmed and was lucky that my shiny new perl dwimmed right along. I was only dimly aware of the bug, and didn't think of it in that note.

    I'm educated as a scientist, and that helps me know that what I think must be tested. What I know must be questioned. To progress with all that doubt, I need to judge what principles are better than others. And I could be wrong. Anybody can tell me so, and if they respect the same values I have to listen.

    Give me that over immutable truth, any day!

    After Compline,
    Zaxo

Re: How do I know what I 'know' is right?
by liz (Monsignor) on Jul 20, 2003 at 07:44 UTC
    "...my knowledge was about a week out of date..."

    I find out-of-datedness less of a problem, as it will give you a sense of history in the end. Which will hopefully make you realize that all those nice things in Open Source didn't get there by itself, but were the result of hard work of countless, and often nameless, contributors. As long as it is clear which knowledge can be applied where (as in: in which version), I don't see a problem.

    It will also make you realize that there is no "definitive" truth in programming (as anywhere else in the world for that matter). And that you should always verify. "Trust, but verify". There are even <plug>conferences</plug> about this.

    Personally, I find that the combination of my 25 years experience in programming, combined with my programmers' laziness, tends to trip me up when I start making assumptions. Assumptions based on my experience, of course, but not necessarily having any foundation in "current" reality anymore. In that manner, I find the openness of Perl Monks, and the willingness to correct (in my case, mistakenly perceived) authority, refreshing.

    So I can only concur with BrowserUk: "The only way I am ever going to know if what I think I know is correct is if people who know better correct me when I am wrong.

    So please, correct me. I welcome your input. I may not directly accept it at face value, but if I don't, I'm not saying that you are wrong, I'm only saying that I am not sure that you are right and I would like further input to allow me to reach that state of grace.".

    Liz

Re: How do I know what I 'know' is right?
by allolex (Curate) on Jul 20, 2003 at 10:00 UTC

    ++ to you, BrowserUK and welcome to Epistemology Monks! ;)

    There are a few groundrules here:

    • You can Super Search, but keep in mind that knowledge is constantly growing and what may be true today is not necessarily true next week. Anyway we don't know if anything else really exists anyway, so what's the point, really?
    • It's not so much about what you know, but rather about your basic ability to acquire new knowledge and apply it. (I see some others have mentioned this.) Being intelligent helps, but also conditioning one's mind to think efficiently plays a huge role.
    • It follows that Perl Monks (our sister site) doesn't just expose people to things they don't know, but also teaches them to think in different ways in order to solve a problem. (map, baby! That's all I have to say.)
    • Hedge a lot. People who are used to participating in scientific/academic discourses tend to qualify the things they say a lot, using "academic caution"*. Academic caution is what results from knowing that you know something about something, but still knowing that your knowledge is potentially incomplete, so you go up one level of abstraction and say that the work reflects general principles, even though the experimental data used are quite case-specific.
    • In Perl Monks, the knowledge bit seems to flow freely. Most of the hedging done is social hedging---trying not to hurt someone's feelings. That is very important because people won't ask questions and learn anything if they feel they are being attacked. (Just adding my voice to your last couple of points.)

    OK, all jokes aside, I also feel these things are important, so thanks for the post.

    Cheers,

    --
    Allolex

    * Academics can be vicious.

Re: How do I know what I 'know' is right?
by mojotoad (Monsignor) on Jul 20, 2003 at 07:58 UTC
    You can 'correct' your own opinions, but only in the light of new knowledge or clarification of existing knowledge, but only if you engage in discourse. There are very few that can arrive at new knowledge, conclusions or opinions on the basis of study or thought in isolation. Isolation engenders dogma and extremism.

    Go away. I'm thinking.

    ;)

    I'm presuming that by 'discourse' you also mean reading and scholarly study (not that direct interaction with others is equivalent...but I'd hate to exclude one over the other).

    Perhaps a decent analogy is between knowledge and a field or crop -- the seeds must be sown, the soil must be fertilized, the plants must be tended...and eventually harvested. Repeat.

    On a more traditional note, one phrase I appreciate and embrace is "the more you learn, the more you realize how little you know".

    To coin my own paraphrase:

    You can't outrun the horizon.

    Something to ponder unless you're a Flat-Earther.

    Cheers,
    Matt

Re: How do I know what I 'know' is right?
by chunlou (Curate) on Jul 20, 2003 at 09:35 UTC

    The "know" has two levels (for which I lack good terminologies):

    • abstraction, theory ("black box"), model, attitude, perception, etc. and
    • data (input), information, fact (instance of abstraction), etc.

    People had had very good and pretty correct and accurate astronomical data for a long while even during the ancient times but their theories and perceptions about the universe were still "incorrect" (incorrect by today's standard).

    In programming if you get some factual, syntactic or logical things wrong, your program may not even run. But if you got your model or design wrong, it might take a while before you'd figure out. If the goal or business criteria for a program are fuzzy, you might hardly know whether the program itself is correct or not.

    Attitude and perception are hard to change. Maybe that's partly why major theories and models propagate slowly (a macro stability, so to speak), albeit data and facts kept pouring in (a micro race). Galois theory in math, quantum mechanics in physics, input-output analysis in economics, for instance, they all took decades to be generally accepted. Once accepted, the (micro) race to applying them began.

    Sometimes I feel a CS student learnt so much about algorithms and stuff, but not enough about "analysis" or problem modeling, which is partly art, partly science. A manifestation would be like, someone could write a fairly efficient query but fail to design an efficient and correct database structure that effectively solves the real life problem.

    It is like being right at the micro level but wrong at the macro level.

      ...People had had very good and pretty correct and accurate astronomical data for a long while even during the ancient times but their theories and perceptions about the universe were still "incorrect" (incorrect by today's standard)...

      To further humble ourselves: today's "standard" will be incorrect/corrected tomorrow. Check out: Higgs Field and realize that the concept of "ether" may not have been that wrong after all. ;-)

      Liz

Re: How do I know what I 'know' is right?
by artist (Parson) on Jul 20, 2003 at 06:14 UTC
    ++ BrowserUk For the good thought.

    This is certainly inline with "You don't know even what you don't know". ie.. You are simply not aware of the other facts that can have influence in this matter, forget about knowing its influence in the matter.

    artist.

      This was very thought provoking read, and thought provocation doesn't occur with me that often, so thanks and ++ to You Browser_uk sir.
Re: How do I know what I 'know' is right?
by husker (Chaplain) on Jul 21, 2003 at 20:00 UTC
    You know what you know by comparing what you know vs. what you observe. You can make observations in many ways: by seeing that your code fails to produce the results you predicted; by educating yourself with relevant material (which is typically just reading about other people's observations); and by interacting with other observing individuals who are also formulating what they "know". What you "know" should be subject to change every day. Tomorrow you may make some observation that contradicts something you know. How you reconcile what you know with what you observe is a key factor in learning. (Throwing out the offending observation in favor of your "knowledge" is called "denial"! Or, in some cases, "faith".)

    There are no scientific "laws". Scientific laws are just statements that science makes that seem to agree with our observations. Because we have accumulated a vast amount of observations about, say, how the pressure of a gas relates to it's volume, we get Boyle's Law. But it's not a law ... just a statement that fits our observations and has held true for a large set of observations. Tomorrow we may make some observation which conflicts with Boyle's Law, and so we will have to throw out Boyle's Law and come up with something that fits the observations we have.

    So it is in the Monastery.