> the tricks I have done in XS sometimes... though, I would never say that disallowing those would be an unacceptable trade off!

well you know my opinion, that lexical macros would be the best way to go!

Hope to see you in one of the next conferences again. :)

Well, perhaps we could have a Platypus XS-emulator? It will not be able to modify runtime internals, unless through an API, but the rest of XS code can be run independently of current internals. Right now (with my minimal understanding, hence the hand-waving) these internals are accessed via macros and global variables. Introduce a layer to isolate the internals: macros become functions, global variables access replaced with accessor functions. Better still, a Perl interpreter behaves like an object. It is no longer the "centre of the universe" (as https://perldoc.perl.org/perlxs.html#CAVEATS says) but it is controlled by a meta-interpreter. This way, and with an API in place for controlling and accessing interpreters, a meta-interpreter can spawn as many Perls and many different Perl versions. As long as they adhere to the API. For older Perl's a wrapper will be needed to provide this API.

Also, again with my minimal understanding, perhaps it will no longer be needed to use XS for performance but, instead, write algorithm in any Platypus-supported language and use Platypus to access it from Perl.

Here is a funny idea: statically link XS code with Perl-X.Y.Z libraries and access it via Platypus from any other Perl version. Essentially one runs two different perl versions one inside the other. Though, again, one can't control the internals and do the tricks.

edit: also relevant to this thread Re^2: Modernizing the Postmodern Language?.

It is not just that internals are accessed through macros and global variables, it is that the current data structures are open.

For instance, the way perl scalars are represented internally (SV) is open. You can access the structure from XS, change it, get pointers to its slots, whatever. There is no reasonable way you could provide an API emulating that on top of a substantially different guts.

Also, any proposed solution needs to be simple, reliable and portable.

Regarding, efficiency, in many cases, efficiency is just being able to access the raw data without it passing through intermediate layers. Take for instance my module Sort::Key::Radix. Besides the algorithm, one of the reasons why it is so fast is because it manipulates directly the perl data structures. There isn't a conversion from Perl to some agnostic format when any of its subs is called or when returning from it.

I agree that data translations will be costly. But nothing stops using raw memory pointers typecasted (i.e. interpreted) to the appropriate data structure. In plain C that would be akeen to typecasting raw memory/void to a struct. In OOP, an SV could be any class implementing the SV interface with, say, slot(name, value). Since all these will be part of the same process, it wont be necessary to use shared memory etc.

It's true that any XS code using SV sv; sv.IV = 42; will break (is this the sort of tricks you mentioned doing?). But, perlguts's doc shows that the proper way is to use sv_setiv(&sv, 42) i.e. a setter macro/function which it is very easy for the suggested intermediate layer to hijack transparently. But perhaps it all leads to re-inventing a virtual machine, which you mentioned.

PS. I respond to this thread, and I am sorry if I waste your time with possible nonsense, as a means of throwing ideas/brain storming. I consider it fundamental that the "modernisation" process starts from the guts. Or equal attention be given to the guts too. And having a data-exchange framework as a when-all-else-fails option for the API when dealing with different Perl versions but also for when accessing external libraries with Platypus and also when accessing external environments (like R) and their native data structures (like R's data.frame) would be great and future-proof and make Perl a true polyglot.

refs: Parrot Overview and SO thread https://stackoverflow.com/questions/118141/what-exactly-is-parrot