|Welcome to the Monastery|
Dataflow programming on CPU and GPU using AI::MXNetby bliako (Parson)
|on Dec 18, 2019 at 12:04 UTC||Need Help??|
Computational pipelines, often called Dataflows, are just Graphs describing the interaction of Data with Operators and other Data to produce more Data.
a*b + c*d is a Dataflow.
And so is a (feed forward) Neural Network: read input, distribute input matrix to first layer, pass it through activations, sum outputs and distribute to second layer and so on until the output layer. Deep Neural Networks are so complex, deep and convoluted that they thought dataflow programming (which is not new as a field) will aid their training and use. And so TensorFlow and MXNet and others were created.
The really unfortunate thing is that most of these frameworks are investing heavily on Python interfaces even if internally they use C/C++ for obvious reasons :) . The fact that Python is a TIOOWTDI regime will lead this field to serious cargo-culting (e.g. "In a sparse network, itís more likely that neurons are actually processing meaningful aspects of the problem." which I have seen before when working with Neural Networks, mid-90's), script-kidding practices and eventual stagnation. Of course the field will recover. And faster and sooner, if and when Python is replaced or other high-level-er (script) languages are equally supported. Nothing stops the machine taking over ...
In Perl, there is an excellent set of modules written by Sergey Kolychev under AI::MXNet based on Apache's MXNet. Note that it is active and very recently updated: Feb 23, 2019 !!! That's a good sign.
My cpu and I have spent a lot of cycles trying to install the pre-requisite libraries of Apache's MXNet written in C/C++ and offering also CUDA capabilities. Two hints: do not use github repo version and use Makefile (instead of that cmake).
Now that I have it all installed (MXNet libraries, Perl module and also R package - unrelated to this post) I would like to share with you all just what size of doors this package opens by introducing a basic dataflow operating on scalars and matrices both on CPU and GPU! The very fact that this package offers, on the side, GPU capabilities within Perl makes it, in my opinion, very very promising. I can't see much offering GPU at CPAN at the moment and here we have one package which opens both GPU and Deep Learning worlds to Perl hackers. (I have no affiliation with S.Kolychev whatsoever)
So, here is some code to get you started on implementing a pipeline to calculate e=a*b+c*d. At first these AI::MXNet::Symbol (e.g. a,b,c,d,e) will be scalars represented as 1-dimensional AI::MXNet::NDArray (which is somewhat similar to PDL's arrays):
And this is the result:output: 
Now, let's replace 1-dimensional data with 2x2 arrays! Just create your data as thus:
But, wait because the * operator on matrices denotes element-to-element multiplication and not the actual matrix multiplication. For that we use:my $e = $a->dot($b) + $c->dot($d);
And the result is:
If not evaluating ginormous matrix expressions using a Dataflow framework (which will also parallelise when parallelisation is possible) was not enough we have here another huge door opening for you number crunchers and Perl hackers: the ability to do this using the GPU via CUDA! Just replace the context above with the GPU-related code. And there is light! In this way, CUDA can be used via Perl not only for doing neural network stuff but for any kind of computation which you can express in a Dataflow. For example, load two images as AI::MXNet::NDArray and multiply them on the GPU! Or do some signal processing by loading an mp3 voice file as NDArray! (this paragraph was a 5' later addition)
One thing that does not add up is that AI::MXNet::NDArray says: "However, NDArray is row-major, unlike the PDL that is column-major.". But PDL is row-major too! So have that in mind.
If that does not keep you excited over the Festivus, I don't know what will ...
Thank you Sergey Kolychev ++
ps. some last 5 min additions and cosmetic changes. Also note the comments in the code provided for more information.