Re^2: Dataflow programming on CPU and GPU using AI::MXNet

Do you think you will be able to make a series of posts about how to use AI::MXNet?

Yes, that's my intention. I will post here when I make more progress. The above was my first visit to MXNet-land. And I wanted to stress the fact that 1) you get GPU access on the side and 2) The underlying Dataflow "engine" by MXNet does parallelisation and lazy evaluation as it sees fit without the user being concerned, so this is quite higher up than more traditional ways to do such computations. And applications using these features do not have to be within neural networks...

If anyone has any real applications outside neural networks, involving tensor/matrix/vector operations then we can work out a demo of the above 2 features. This (i mean non-neural-network apps) will expand the user-base of AI::MXNet

Next stop neural networks. Possibly sooner than a month.

bw, bliako

Comment on Re^2: Dataflow programming on CPU and GPU using AI::MXNet

Replies are listed 'Best First'.
Re^3: Dataflow programming on CPU and GPU using AI::MXNet by etj (Priest) on Apr 25, 2022 at 10:43 UTC
PDL natively has forward dataflow (if you opt in by using `$pdl->doflow` - see the next PDL CPAN version for an updated PDL::Dataflow doc that lays all this out, or check out the GitHub version in `Basic/Pod`), and has for decades. Lazy evaluation currently only happens with slice-like operations, but there are plans to switch to lazy evaluation in order to have not only loop-fusion by creating custom operations that would e.g. do the `ab + cd` with only one round-trip from RAM through the CPU to RAM, but also GPU processing. See https://github.com/PDLPorters/pdl/issues/349 for discussion and/or to participate!	[reply] [d/l] [select]
Re^4: Dataflow programming on CPU and GPU using AI::MXNet by etj (Priest) on Aug 04, 2024 at 23:58 UTC
Update from the future: PDL now has better, working dataflow that you can use better, with the `inplace`-like operation `flowing`. It replaces `doflow`. See the latest PDL's PDL::Dataflow and also Example of PDL dataflow to implement 3D space calculations. EDIT: Having read a bit more about "dataflow programming", that's not entirely the same thing I was thinking of. PDL can (at this writing) have automatically-updated ndarrays that depend on other ndarrays. "Dataflow programming" appears to be more stream-orientated, where operations update when all their (smaller, more granular) inputs become valid. PDL can do that a little bit (see the molecule/graph-theory bit in `demo 3d`), but being more event-driven isn't fantastically well-supported yet.	[reply] [d/l] [select]