http://qs321.pair.com?node_id=11119793


in reply to Re^11: Amicable divorce
in thread Amicable divorce

A purist could pop in and say to you "no, I want Perl to just be P.E.R.L. and only that, no fancy additions."

I don't agree with that view but we should not forget what Perl was created for.

Asking for shared memory alone is vague. Asking for *efficient* shared memory is much more accurate. Only then we can see who is "swimming naked" ("You only find out who is swimming naked when the tide goes out" hehe, and let the source remain nameless) when it comes to comparing to other systems. Although I do not dispute that Perl's shared memory is inefficient.

For the realist, if a solution must involve Perl, perhaps it should not be as fine-grained as to require sharing the million variables of a parallel algorithm. A coarse-grain implementation could be more suitable for Perl (being further away from hardware, compared to C, and this is the reason we all enjoy other perks!). That is, limit the amount of what is shared or access it in larger chunks of raw data where the inefficient lock mechanism does ot matter much. On the other hand, MMAPing a raw-memory buffer and sharing it, is easy but each thread interpreting it as a Perl nested data structure or object will take some efficiency away. Interpreting it as 100 such data structures in 100 threads it will be much slower. And then locking each of these.

Additionally, there is the other route to harnessing state-of-the-art computer science developments: by using extenral libraries. For example the excellent implementation of AI::MXNet by Sergey Kolychev. Which is still a well-kept secret (I think? Anybody using it? Although regularly update. Thanks Sergey). Also see his relevant post http://blogs.perl.org/users/sergey_kolychev/2017/02/machine-learning-in-perl.html and my brief introduction to using it for very simple Dataflow programming Dataflow programming on CPU and GPU using AI::MXNet even on a GPU.

These are great doors opened from within Perl. Fair enough, not what exactly you are asking for: transparent shmem-smp. But just write your algorithm as a Graph and let MXnet worry about parallelising it or even having the GPU solve it.

Perhaps setting a concrete programming challenge can show the distance between practice and my theoretical blabber above?

bw, bliako