Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Insight needed: Perl, C, Gtk2, Audio, Threads, IPC & Performance - Oh My!

by Joost (Canon)
on Jun 01, 2006 at 14:30 UTC ( [id://553060]=perlquestion: print w/replies, xml ) Need Help??

Joost has asked for the wisdom of the Perl Monks concerning the following question:

This is a long rambling account of a project I'm working on and some of the problems I have. I'm writing this to get my head cleared up and hopefully recieve some suggestions.

Read on at your own peril ;-)

Introduction

Some time ago I was toying with writing my own "modular audio composer" / sequencer. Sort of like Buzz for Linux, only better ;-)

I started with an XS library that wraps the LADSPA API in a nice perl OO API, just so I could play with the many existing LADSPA plugins (these plugins are mostly written in C / C++ and can be loaded into the program at runtime).

Then I added a bunch of perl code to make it all easier to use and manage. I'm still finetuning it, but that part is reasonably stable and available on CPAN.

So far so good, but this project needed a GUI. After two initial attempts in Perl/Tk that got quite messy and didn't work as well as I'd hoped, I abandoned the whole thing for a two years.

Right now, I've got some spare time and I'm rewriting the interface in Gtk2. Once you get used to it, coding in Gtk is pretty easy, you can use nice OO techniques if you want, it's easy to refactor and it looks a whole lot prettier than Tk.

Though I'm still not finished, I'm mostly happy with how the code looks and the progress there.

Problems

This program should be able to generate audio streams that can be sequenced and changed on the fly via the GUI (and eventually also via MIDI controls). It works, with some latency, but:

Some GUI updates are a bit slow and cause breakups in the audio stream.

It's possible that I could buffer more output to offset the problem but if the delay between GUI actions and output bcomes too long, the whole thing will feel sluggish and it just won't work very well.

For larger networks the whole thing slows down.

A large network here is one with lots of connected plugins. I still need to do some work to figure out where the bottle neck is, exactly.

If needed I could rewrite some of the perl code in Audio::LADSPA::Network in C/XS, but to get real good speed I might have to break the possibility of writing plugins in Perl. That would be a shame, since I use that option right now for "special" plugins that glue the network to the GUI and add extra functionality. I could also rewrite those plugins in C/XS, which wouldn't be too bad, but I like having the option to quickly make & change prototypes in Perl first and optimize later.

I'm looking for a good audio IO library for perl

At the moment I'm using Audio::Play. Works well and it's very easy to use, but it only supports monophonic output and doesn't allow me to check the output buffer state.

Since I've got a 6 in / 6 out audio card that works very well with ALSA, I'd like to be able to use all of those inputs & outputs, either directly using the ALSA api or via Jack or both.

I might end up writing my own XS code to handle this anyway, but I would appreciate suggestions for existing solutions.

Constraints & design decisions

Currently, the highlevel design is simple: there's one perl process that initializes the Audio::LADSPA::Network code and sets up the GUI. GUI actions modify the network, and network changes are reflected in the GUI via Observer /Observable messages. (Note: due to a bug in Class::Observable, this functionality is not yet available in the CPAN release of Audio::LADSPA).

Every few milliseconds, a Gtk2 timeout callback will ask the network to generate a bit of audio, which may be send to the audio output if the network contains a Play plugin (which uses Audio::Play)

Simplified diagram of the current design

+--------------------------------------------------------+ | Main process | | +--------------+ +----------------------+ | | | GUI (Gtk2) | ---update--> | Network with plugins | | | | | <--update--- | | | | +--------------+ | +------------------+ | | | +--------------+ | | audio output | | | | | Timer (Gtk2) | ---- run --> | +------------------+ | | + +--------------+ +----------------------+ | +--------------------------------------------------------+

Updates are things like: adding & removing plugins, adjusting parameters etc. The run action is what actually generates the audio stream (every run action generates about 2.5 milliseconds worth of audio).

Now, because all this runs in a single thread, the timer isn't completely reliable (gui updates can delay timer calls). Also, as noted above, it seems that the network code could use a few improvements to make it faster.

What I really want to do is to seperate out the run actions from the rest of the process. I can think of three ways to do that: threading, multi-process or a combination of the two.

Threading

Now that perl has some measure of thread support (based on pthreads it seems), I might be able to use that. Basically, put the run() calls in a seperate thread.

This seems ideal, especially if I want to use the Jack audio API, which more or less requires a seperate thread for its callback routine (Jack callbacks look like they could be fairly easily mapped to network runs).

Since I haven't really played with perl's threading for ages, and I haven't done any pthread programming in C, I do have some questions:

  • Is it possible to share perl objects between threads?
  • If so, how efficient is that?
  • Does it also work with XS backed objects?
  • If I can't share perl objects, would it be possible to create a 100% C thread that directly calls into the C code/data in another thread?
  • Are there any pitfalls I should look out for?

Forking / multi-process

Since the Network/Plugin API is getting pretty stable, I might be able to run the whole network in a seperate process. This would mean I need some kind of IPC mechanism. I'd prefer not to write a whole IPC library from scratch if I can help it :-)

More questions:

  • What are the choices for good, lightweight, preferably OO IPC libraries like Java RMI, or Distributed Ruby?
  • Since I might end up rewriting the Network/Plugin code in C, what are my choices if I want to do IPC from Perl to C and vice-versa?
  • I still might need to run a seperate C thread to handle callbacks if I want to use Jack.
One alternative I've been looking at: Net::LibLO, which is more or less designed to do just this kind of thing.

Conclusion

This is more or less everything I can think up now. I hope it wasn't too incoherent.

I'm looking for suggestions, answers, links, anecdotes, questions, anything. In any case, if you've made it this far, thank you for reading it all. :-)

Cheers, Joost.

  • Comment on Insight needed: Perl, C, Gtk2, Audio, Threads, IPC & Performance - Oh My!
  • Download Code

Replies are listed 'Best First'.
Re: Insight needed: Perl, C, Gtk2, Audio, Threads, IPC & Performance - Oh My!
by jdhedden (Deacon) on Jun 01, 2006 at 15:56 UTC
    Let me address your threading questions because they are rather generic in nature.
    Now that perl has some measure of thread support (based on pthreads it seems), I might be able to use that.
    Perl's threading support has improved greatly since its introduction in 5.8.0, and continues to improve. As such, you should use the most recent version of Perl (currently 5.8.8) for your system, and be sure to install the lastest versions of the threads and threads::shared modules. This will give you the latest feature sets as well as the best reliability.
    * Is it possible to share perl objects between threads?
    Objects can be shared between threads. However, the class must be written to specifically support threads and sharing. You can do all this from scratch, of course, but I strongly recommend an object support module that handles all the necessary technicalities for you. Object::InsideOut is one such module that is very full-featured and fully supports threads and sharing.
    * If so, how efficient is that?
    Thread data sharing is implemented using a tie methodology. Sharing objects between threads is as efficient as sharing any other data.
    * Does it also work with XS backed objects?
    Yes, but you have to make sure it is done correctly. As an example, you might look at Math::Random::MT::Auto which has XS-backed objects, and uses Object::InsideOut for its object paradigm.
    * If I can't share perl objects, would it be possible to create a 100% C thread that directly calls into the C code/data in another thread?
    Since you can share objects between threads, this question is moot. However, regarding the technicalities of using C-based threads from Perl, it should be possible if you don't access any Perl data in those threads, but I would imagine that such an approach would be fraught with difficulities.
    * Are there any pitfalls I should look out for?
    If you want to write your classes to handle all the details of threading and sharing yourself, then you need to be cognizant of the CLONE method (see Making your module threadsafe), and how to create sharable objects (see annotations for threads::shared).

    However, if you use Object::InsideOut, all these details are taken care of for you.

    Regardless of your approach, you need to be conversant with the threads and threads::shared PODs.


    Remember: There's always one more bug.
Re: Insight needed: Perl, C, Gtk2, Audio, Threads, IPC & Performance - Oh My!
by zentara (Archbishop) on Jun 01, 2006 at 15:52 UTC
    This is only a comment, since it seems what you are doing is very complex. I didn't see you mention SDL::Mixer. You can have multiple channels of sound, see Tk Game Sound demo

    If you can get SDL::Mixer to work for you, you should be able to incorporate it with Gtk2. You may be able to use the "do 1 loop" trick of Gtk2

    Gtk2->main_iteration while Gtk2->events_pending;
    to avoid having to use the Gtk2 Mainloop. You could let the SDL event loop run things, just repeadetly call the Gtk2->main_iteration to keep the gui responsive. As an example with Tk
    #!/usr/bin/perl -w use strict; use Gtk2; use Tk; my $mw = MainWindow->new(-title=>'Tk Window'); Gtk2->init; my $window = Gtk2::Window->new('toplevel'); $window->set_title('Gtk2 Window'); my $glabel = Gtk2::Label->new("This is a Gtk2 Label"); $window->add($glabel); $window->show_all; my $tktimer = $mw->repeat(10, sub{ Gtk2->main_iteration while Gtk2->events_pending; }); $mw->Button(-text=>' Quit ', -command => sub{exit} )->pack(); MainLoop; __END__

    As far as threads go, the recent versions of Gtk2 have a way to share Gtk2 objects across threads, but it is tricky and not generally gauranteed to always work. It has an enter and leave method, to tell Gtk2 to enter and leave the thread.

    use Gtk2 qw/-init -threads-init/; die "Glib::Object thread safetly failed" unless Glib::Object->set_threadsafe (TRUE); ..... ..... sub thread_work{ Gtk2::Gdk::Threads->enter; ......... ......... Gtk2::Gdk::Threads->leave; }

    I'm not really a human, but I play one on earth. flash japh
      Actually, the mixing part of the program is more or less done already - it's all handled by compiled C plugins. I discarded SDL from consideration because it doesn't seem to give me enough low level access: I need to be able to generate audio streams quickly and dump them on the sound device (something like 40 samples at a time, that's once every ~0.001 second).

      SDL has a lot of nice routines for playing & mixing long samples & streams, but nothing that gives me that kind of response. Or so it seems to me reading the docs. If you or anyone else have tried something like this with SDL, I'd be happy to hear your stories :-)

        I use SDL_mixer, and I would echo your conclusion: it doesn't give enough low level access. It's fine for playing one background music track and doing fire & forget sound effects, but that's it. For my project, that's just barely kinda mostly sorta good enough for now, but the next time I revisit audio I'll be switching to either OpenAL or the raw ALSA API. I want to be able to change the volume of a triggered sound effect while it's playing. I want an immediate notification of when a sound has completed so I can sequence something else (non-audio) after it. I want to get feedback about how far through a sound I am. I want to do pitch bending. I wouldn't mind some 3D sound processing (although I think SDL_mixer can do some of that.)
Re: Insight needed: Perl, C, Gtk2, Audio, Threads, IPC & Performance - Oh My!
by kvale (Monsignor) on Jun 01, 2006 at 15:26 UTC
    Processing audio in straight perl can be pretty slow; traversing the parse tree takes a good deal of time, to the point that operations like regex matching have their own hand-optimized C engines.

    One alternative to generic perl that is great for processing vector data (like audio streams) is the Perl Data Language, or PDL. The vector and matrix operations are implemented in highly optimized C code and are programmed in a Matlab-like dialect that makes it easy to express DSP algorithms.

    -Mark

      I am aware of PDL :-) I don't actually do any audio processing in perl itself (though the API allows it) - the plugins written in C take care of all the audio data (except for a few conversions to Audio::Play, but even that is mostly C code).

      The perl based plugins only do "control channels", that is - they generate only 1 float per channel each "tick" (with ticks once every 40 .. 200 samples). PDL can't really help with the control data, I think, since you're not acting on arrays of numbers, only single values.

      I do see there's now a PDL::Audio module on CPAN. That wasn't there when I started this (2 years ago). I'll have to check it out.

Re: Insight needed: Perl, C, Gtk2, Audio, Threads, IPC & Performance - Oh My!
by toma (Vicar) on Jun 02, 2006 at 00:26 UTC
    I tried to use Perl for audio but did not get the performance I was looking for, so I switched to C++. I noticed that a lot of my C++ code was repetetive, so I used perl to generate the C++ code using templates.

    For my C++ generation code, the perl functions specify a signal-processing netlist. I used the graph modules to solve the dependencies in the netlist, so that the C++ code came out in the correct order. It works great, and the generated code is clean. It looks like it was typed in by a very patient programmer! A typical example is a 51 line perl program that turned into a 9102 line Jack application.

    It should work perfectly the first time! - toma
      That sounds very interesting. Do you have any code lying around that I can take a look at? :-)

      At the moment, the graph I'm dealing with is a net of connected plugins that are mostly written in C (LADSPA plugins). I just calculate the right order once it's needed (in perl, using the Graph modules) and cache that until the the network is changed. After that the speed is acceptable - not great, but good enough for small experiments at least - since all I do is call one method for each plugin to generate a partial stream of X samples. The rest is normally handled in C.

        I don't have anything that I can post right now.

        I used HTML::Template on the C++ code, and I was in the process of switching to Text::Fillin when I stopped working on it. My code is from the early days of Jack when it was a moving target. I hope to get back to it someday and publish it in an updated form.

        Our approaches are somewhat similar, and they are also similar to the PD (Pure Data) approach. The difference is that we like perl code, and PD uses a GUI instead.

        I don't know if you are using the low-latency linux patches, etc, but if you are not, you should visit Planet CCRMA at home. The PlanetCCRMA web site and mailing list is a great place to find help and advice on linux audio development!

        It should work perfectly the first time! - toma
Re: Insight needed: Perl, C, Gtk2, Audio, Threads, IPC & Performance - Oh My!
by Ace128 (Hermit) on Jun 15, 2006 at 15:39 UTC
    I was wondering when this is ready? :) How far have you come on this interesting project?
      I've got a working core - all the functionality to connect plugins and produce sound are there (that's Audio::LADSPA::Network plus a few additional plugins), and there's a functional GUI for it, including loading & saving networks.

      Currently I'm working on the sequencing part, which still needs a lot of GUI work. I don't know when that'll be done, since I'm also pretty busy doing paid work.

      By the way, I've just uploaded the latest version (0.018) of Audio::LADSPA to CPAN. This doesn't include the GUI but at least it allows you to play around with any ladspa plugins you might have installed.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://553060]
Approved by marto
Front-paged by Courage
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others wandering the Monastery: (4)
As of 2024-04-19 23:14 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found