that is what is so nice about Perl - it translates nearly exactly over to C when you are writing it

Perhaps that's part of your problem. I don't mean that in a nasty way - I know when I started writing Perl my code looked a lot like C too. The secret to writing Perl that runs fast is to use Perl's built-in functions to do the work for you, since they are written in optimised C (albeit optimised for a general case). The more lines of Perl code that you write, the more time Perl will need to spend branching and switching between ops. Whereas if you can pare your code down so that the work gets done by map, foreach, tr, regexes etc then the execution time should drop.

It really raises a red flag for me that your Perl implementation ended up longer than the equivalent thing written in C. These days when I have to write C I get frustrated at how many lines of code I have to execute to do anything useful.

That is a great point. I suppose I should clarify that statement by saying that my idea of "nearly exactly" in regards to translating to C was that it is much easier than say to forth - the math logic is the same and the functions are setup in a very similar manner.

In terms of length difference, I agree when dealing with strings, C ends up taking multiple lines to do what Perl can do in one line. Aside from that, the line count is IMO largely due to comments and I'm not really sure how many lines are just comments and haven't bothered to really determine that exactly. Were I to strip out all comment blocks from the C and Perl code - I would imagine the Perl to be shorter - I likely just estimated how many they have of each to be wrong.

Whereas if you can pare your code down so that the work gets done by map, foreach, tr, regexes etc then the execution time should drop.

I'm certainly interested in speeding it up just out of curiosity (since I already have a C version now that takes 1.4sec, which I don't foree the Perl version ever doing - but I am curious where I made decisions that slowed the Perl down).
I converted every for loop to a foreach loop instead - that slowed the code down by 6 seconds. I assume I didn't get the optimal use of it because I still need to be aware of where I am in the array and not just what $_ is, so I kept a counter along with it - which I would assume would be pretty much just like a for loop in that case.

Perhaps I'm just being dense, but the only place I see where map would really apply to my stuff is when I'm originally manipulating my arrays - and I've already benchmarked that to take less than a second, and it is only done once - so that isn't the speed concern there.

I'm only doing math inside the loops, so the tr is useless to me in that case - again, the only text manipulation I do is at the start and at the end of the program - outside of any of the loops. Other than that, it is just math.

Again, I don't see how a regex will help me when I just want to iterate N days of values and add them together, and then divide by some number.


Were I parsing text and seeing this huge a time difference, I would be very suspicious of it all - but this is nearly all math inside the loops (3000+ times it iterates through the outer loop, and inside of that is a large 1000+ loop iteration with a few smaller 5-20 iterations that depenind on what algorithm is used is done on each 1000 as well).

I understand that I'm being pretty useless in not putting up the actual code so that people can point out "you did this and THIS would be way faster" - but I think in the end I mainly wanted to show another example with some numbers of what speed differences for various applications exist. When I was orginally going to port it over to C, I looked around for numbers of what sort of difference I might see, and very few people talk about doing it.
So now there is at least one more node about it :)

Is there a resource out there that lets me know that "doing XYZ is more efficient in Perl than doing ABC"? I'm curious which things I'm doing that might be sped up, but the bulk of what is inside the loops is if/else statements, and just basic math ((a+b)/c)

You might be able to achieve a huge performance gain by taking your algorithms and memoizing them. I.e

my %multiply = ();

$str = "$num1 $num2";
if (!$multiply{$str}) {
    $result = $num1 * $num2;
    $multiply{$str) = "$result";
} else {
    $result = "$multiply{$str}";
}

# use result to do something else below
[download]

You could even turn this into a sub function, which takes a string as an argument, shifts it as it goes, determines what to do with the different expressions on the line, and then returns the final result. Thinking about that though.. You would need to add logic to deal with precedence, and parentheses. Also it would be very expensive to start, but once you are through a fair portion of your data, it will really help

This is probably a very poor example, and could be cleaned/optimized, but you see the point I hope. Floating point math is always leaps and bounds slower then interger math. On top of that you moved from a windows machine to a FreeBSD machine. The kernel architecture, memory management, threading mechanisms, as well as application overhead all play massive roles in how your code will perform, especially in regards to loops such as you are talking about.

Also as long as the data sets do not rely on each other, you could take all your data, parse it, store it in a hash or array, then take slices of those structures and fork off seperate processes which can do the computations in parallel, and then collect the results from files, or even share memory space, and have them all report back when they are done if you have ithreads enabled, and know how to use them.

I really am interested to see your code, much like other monks. I have started a few threads about speed, and squeezing the last bit I can out of perl. I've received great help and optimizations, which have also allowed me to peer a little deeper into how perl works and what it likes to see. It really does worry me, just like the other monks have stated, that your perl code looks like your C code. You could be missing out on some really great stuff there.

O well just my 2 cents..

/* And the Creator, against his better judgement, wrote man.c */

My best guess based on your breif description is that when you split your float data into the array, the C-version is performing the ascii-to-binary conversion once and the array iterated over is float *; or double * and all subsequent process of these numbers id performed on them in their binary form.

In the Perl version, the numbers are being maintained in the array in their ascii form, and each time any calculation with or comparison of them is done, perl has to perform the conversion to binary (though this step can be skipped if this scalar was use for numeric purposes previously I think as perl keeps a copy of the binary representation once a scalar has been used for math), do the math and then convert back to ascii representation and store.

The upshot is that every calculation between 2 floats, (ex. num1 *= num2; in the C version comes down to

load reg a, num1;  move 8 bytes from memory (probably cache)
; maybe 2 or 4 clock cycles barring stalls which given the 
;C array is probably contigious memory probably happens every 128K val
+ues 
;after the array is initially addressed depending on the size of the L
+1/L2 caches 
;and what else the surrounding code is doing

load reg b, num2; ditto

fmul reg a, regb; A Floating Point processor instruction
; Depending on the processor could be 1 to 10 or maybe 20 clock cycles

store reg a, num1; 8 bytes stored.
; Another 2/4 clock cycles.
[download]

maybe 30/40 clock cycles at most, and usually much less.

Whereas for Perl, the equivalent processor instructions involve

1. locating the base SV,
2. indexing that to find the XVPLV.
3. Locating, and possible performing ascii-binary conversion on the index var,
4. then using the value to calculate the offset into the storage pointed at by the XVPLV to
5. locate the SV pointing to the float element.
6. Loading the base of that SV,
7. checking to see if the NOK flag is set.
8. If it is load the previous binary value of this scalar.
9. If not, chase the XPVNV to the ascii representation.
10. Read the string byte by byte in a loop and perform the math to convert it to binary form.
11. Repeat from step 1 for the second variable.
12. Finally we have the two floats in binary form, but in (temporary?) storage not registers.
13. Perform the actual math using essentially the same steps as outlined above for the C version.
14. Now peform the reverse of the first 10 steps, twice, remembering we must always do the binary-to-ascii convertion on store as the next use of the variable may be as a string (eg. print), but that we also store both binary reps (which may save considerable time if the next use is numeric).
15. Do lots of flag checking/setting and other housekeeping.

I make no claims for the above being complete, correct or accurate in anyway whatsoever, but it gives a feel for whats involved

As you can see, the process of adding two floats held in an array in perl is considerably more involved than in C and takes in the order of 100's if not low 1000's of clock cycles, as opposed to 10's in C. That's the price you pay for Perl's infinitely simpler and more flexible type-polymorphism, DWIMery and transparent housekeeping.

It makes me wonder if Perl 6 couldn't reverse the magic used by Perl 5 as far as numbers are concerned. That is to say, once a value has been used as a number, if the ascii representation of that var couldn't be flagged as 'dirty' and not maintained until such times as an attempt is made to use it's contents in a non-numeric capacity. Ie. As well as the IOK and NOK flags, have a AOK flag, allowing the ascii representation not to be updated until necessary?

That said, in the above sequence, it's the pointer chasing and flag waving required by perls infinitly flexible array representation that makes the biggest difference beween it and the C-float array process, so it probably wouldn't make a whole heap of difference.

The upshot is, that if you want to do math intensive programming, use a language that lends itself to that, or find a module that allows you to hand over the storage and processesing of numeric data to a library written in such a language.

It would be nice to thing that our favorite language could be extended in its upcoming 'new' form to allow us to gives clues when we declare vars as to their most prevelent usage and that it could transparently (and without too much overhead) allocate, store and manipulate them ways more efficient to that usage, whilst retaining the DWIMery and flexibility of the standard Perl data representations. (You have no idea how hard it was to write that sentance without mentioning types :). I'm sort of thinking about the perl 6 attributes here.

Perl 6 will certainly support type declarations in spots that will help the optimizer. In particular, you'll be able to say something like

    my num @array;
[download]

to get a compact array of doubles. The very-soon-to-be-released A6 will have a section on the new type system. And then you won't have to be afraid to utter the word "type". :-)

Thanks for the info, it sounds really interesting. is there currently any shortlist of proposed types? Or do we must wait for the official skinny?

---

Examine what is said, not who speaks.
1) When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
2) The only way of discovering the limits of the possible is to venture a little way past them into the impossible
3) Any sufficiently advanced technology is indistinguishable from magic.
Arthur C. Clarke.

Are you sure about that last bit where once NOK is set it also reifies the string value as well? That sounds mighty suspicious to me (not that I actually /know/ one way or another). perl -MDevel::Peek -e '$k = "1234.1234"; $k++; Dump $k' shows that while there is a string still stored its not kept up to date with the numeric version unless I insert something that stringifies like print $k. Once NOK is set then it isn't going to be unset unless its string representation is altered so it keeps the packed representation around.

---

Seeking Green geeks in Minnesota

Um, no. I am not sure at all, hence the disclaimer.

I should have known that my 'reverse the magic', AOK optimisation idea was so obvious that it would already have have been implemented long ago.

I did attempt to verify that bit, but I won't explain (my stupidity) that made me think I had confirmed it.

Suffice to say, even without the need to re-ascii-ify the numbers between math ops, just the process of addressing the values held in Perl arrays is costly. This is the cost of the infinite flexibility with which they can be grown, shrunk, used to hold any type (small t) of data, sliced, spliced, diced; created and thrown away with relative impunity. For most applications, this flexibility far outweights the costs, but understanding the costs in the key to knowing when they are inappropriate for a given purpose. Math intensive manipulations of large numerical datasets is one such case.

As C arrays are simply contiguous lumps of memory, looping over numeric arrays in C, involves incrementing a pointer by a fixed integer value, and loading a one or two (64- or 32-bit) registers via the pointer, performing the fp-op and then writing the result back via the same pointer.

A very tight, register bound process. Even if two arrays are being processed in parralel, it's still quite likely that all the imtermediate terms and pointers can be kept in registers on most processors.

The equivalent process using Perl's arrays is considerably more involved, requiring lots of pointer chasing, flag waving and (potentially) exspensive format-conversion. This is neither a surprise, nor a burden in the majority of programs, but knowing that this is what is involved is key to understanding what things perl is really good at doing and what things it is less-than-optimal for.

---

Examine what is said, not who speaks.
1) When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
2) The only way of discovering the limits of the possible is to venture a little way past them into the impossible
3) Any sufficiently advanced technology is indistinguishable from magic.
Arthur C. Clarke.

No, when you use a Perl scalar that contains (just) a string in a numeric context, the string is converted to a number and the numeric value is cached in the scalar along with the string value. Future uses of that scalar in a numeric context simply fetch the cached numeric value.

                - tye

Yes. I understood, (and indicated this above?) that when a scalar (that has previously been used in a numeric context) is fetched, the binary version is accessed and used saving the need for a ascii-to-binary conversion.

The statement that diotalevi rightly took me to task for--though I have again attempted to verify this without success--is the one were I suggested that when a scalar that has been used in a numeric context and therefore has a binary version available, is modified numerically, the ascii version is also updated.

Part of what made me think this was the case is that I cannot see any mechanism in the data structures whereby perl would be able to know whether the ascii version needed updating from the binary version.

To clarify (my own thoughts mostly), in the following situation

my $num = '5.1'; ## scalar is a string, NOK is false
if ($num == 5.1) { print 'It is'; }
[download]

At this point, $num has been used in a numeric context, so a ascii-to-binary conversion has been done, NOK is true, and subsequent references to $num in numeric contexts can re-use the binary value directly avoiding an ftoi().

$num++; ## binary value fetched, incremented and stored
$num++; ## binary value fetched, incremented and stored
[download]

The question is, did the stringy version of the value get updated when those modification occurred, or does that get delayed until $num is used in a string context?

If the stringy version was not updated, then by what mechanism does perl know that it must do so when, sometime later, I do print $num;?

---

Examine what is said, not who speaks.
1) When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
2) The only way of discovering the limits of the possible is to venture a little way past them into the impossible
3) Any sufficiently advanced technology is indistinguishable from magic.
Arthur C. Clarke.