Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re^8: Optimizing with Caching vs. Parallelizing (MCE::Map) (PDL: faster)

by etj (Deacon)
on Apr 24, 2022 at 13:15 UTC ( [id://11143242]=note: print w/replies, xml ) Need Help??


in reply to Re^7: Optimizing with Caching vs. Parallelizing (MCE::Map) (PDL: faster)
in thread Optimizing with Caching vs. Parallelizing (MCE::Map)

A long, exhaustive search of the PDL source code (using https://github.com/PDLPorters/pdl/search?q=structure+recursion) shows where that message originates (which interestingly it looks like you retyped rather than copy-pasting - the actual message says "PDL:Internal").

The macro there (since at least v1.99987, from 1998) uses a process-global, function-static __nrec to attempt to track recursion depth. The problem on Windows will be because in Perl, its "fork" actually just makes a new thread. C global variables will still be process-global, so that variable will be getting incremented by lots of different threads, both POSIX threads and process-faking threads.

The solution to this might be attempted by using some sort of thread-local storage to limit the scope of that variable. A much better solution would be to change the relevant functions to just pass a depth-count as a stack parameter, which would obviate this whole problem.

Separately, turning off PDL autopthread behaviour seems to me the correct behaviour for MCE. Otherwise you're having two different types of parallelism, which seems likely to cause chaos.

  • Comment on Re^8: Optimizing with Caching vs. Parallelizing (MCE::Map) (PDL: faster)
  • Download Code

Replies are listed 'Best First'.
Re^9: Optimizing with Caching vs. Parallelizing (MCE::Map) (PDL: faster)
by etj (Deacon) on Apr 24, 2022 at 15:44 UTC
    The change to using a recurse_count argument in the relevant function (PDL.make_physical and friends) has now been implemented in git (doesn't break ABI as the API functions are now wrappers for the recurse_count versions).

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11143242]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others wandering the Monastery: (6)
As of 2024-04-23 10:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found