iPhone Core Audio Part 2 – Setting up the AUGraph

Previous: iPhone Core Audio – Getting Started

In the previous section we went over the tedious details of getting a project set up. Now we are almost ready to start exploring some code. But first, some core audio concepts.

  • Audio Unit – A collection of audio processing code  that is bundled up for reuse in many applications. This is the same concept as a VST or AudioSuite plug-in. On OS X there is a good sized set of built in Audio Units that do common tasks like filtering or reverb and you can install 3rd party AU plug-ins. On iPhone OS the built-in set is more limited. There is not (yet) any way to load 3rd party AudioUnits.
  • AUGraph – A set of functions for loading, managing, and connecting AudioUnits into a signal processing system.
  • remoteIO – An audio unit that connects to the iPhone’s audio input and output hardware.
  • AudioComponentDescription – A struct used to identify an AudioUnit.
  • AudioStreamBasicDescription – A struct that holds information about constant bitrate audio that will be transferred; either to a file, through audio units, or out to audio hardware.
  • CAStreamBasicDescription –  A wrapper around the AudioStreamBasicDescription struct that provides some convenience methods for modifying values and printing.

Inside of the AudioController class we will set up an AUGraph containing a remoteIO audio unit and a multi-channel mixer audio unit. In your AudioController.h file you will need to:

  • Import the AudioToolbox
  • Import  CAStreamBasicDescription.h
  • Create instance variables to represent an AUGraph and a Mixer audio unit.
  • Add methods to initialize, start and stop the AUGraph.

//  AudioController.h
//  iPhoneAudio

#import <Foundation/Foundation.h>
#import <AudioToolbox/AudioToolbox.h>
#import "CAStreamBasicDescription.h"

@interface AudioController : NSObject {

    // Audio Graph Members
	AUGraph   mGraph;
	AudioUnit mMixer;

	// Audio Stream Descriptions
	CAStreamBasicDescription outputCASBD;	

	// Sine Wave Phase marker
	double sinPhase;
}

- (void)initializeAUGraph;
- (void)startAUGraph;
- (void)stopAUGraph;

@end

Now it’s time to hop over to AudioController.mm At the top we import and include some files and make a static declaration of the sample rate. The default rate on the iPhone is 44.1 KHz.

//  AudioController.mm
//  iPhoneAudio

#import "AudioController.h"

// Native iphone sample rate of 44.1kHz, same as a CD.
const Float64 kGraphSampleRate = 44100.0;

@implementation AudioController

It’s always good practice to take care of memory clean up right away, before you forget. So far we just need to get rid of the AUGraph.

// Clean up memory
- (void)dealloc {

     DisposeAUGraph(mGraph);
     [super dealloc];
}

Here are the methods that tell the AUGraph to start and stop audio processing.

// starts render
- (void)startAUGraph
{
	// Start the AUGraph
	OSStatus result = AUGraphStart(mGraph);
	// Print the result
	if (result) { printf("AUGraphStart result %d %08X %4.4s\n", (int)result, (int)result, (char*)&result); return; }
}

// stops render
- (void)stopAUGraph
{
    Boolean isRunning = false;

    // Check to see if the graph is running.
    OSStatus result = AUGraphIsRunning(mGraph, &isRunning);
    // If the graph is running, stop it.
    if (isRunning) {
        result = AUGraphStop(mGraph);
    }
}

Now for the big one, setting up the AUGraph, there’s a lot going on here. Remember, this is what is getting done.

  • Create a new AUGraph
  • Create two AUNodes that will hold our mixer and our output AudioUnits.
  • Filling out an AudioComponentDescription for each AudioUnit.
  • Adding the two nodes to the graph along with the description of the audio units we want made.
  • Connect the mixer node output to the output node input.
  • Call AUGraphOpen.
  • Setting up the mixer node inputs and their expected audio stream format.
  • Providing a callback function for the mixer input to call when it wants a chunk of audio data.
  • Setting up the output unit’s audio stream format.
  • Call AUGraphInitialize

Note how, mostly, you don’t have to handle AudioUnits directly. You use the AUGraph methods and pass information in and out by reference.

- (void)initializeAUGraph
{
	//************************************************************
	//*** Setup the AUGraph, add AUNodes, and make connections ***
	//************************************************************
	// Error checking result
	OSStatus result = noErr;

	// create a new AUGraph
	result = NewAUGraph(&mGraph);

        // AUNodes represent AudioUnits on the AUGraph and provide an
	// easy means for connecting audioUnits together.
        AUNode outputNode;
	AUNode mixerNode;

       // Create AudioComponentDescriptions for the AUs we want in the graph
       // mixer component
	AudioComponentDescription mixer_desc;
	mixer_desc.componentType = kAudioUnitType_Mixer;
	mixer_desc.componentSubType = kAudioUnitSubType_MultiChannelMixer;
	mixer_desc.componentFlags = 0;
	mixer_desc.componentFlagsMask = 0;
	mixer_desc.componentManufacturer = kAudioUnitManufacturer_Apple;

	//  output component
	AudioComponentDescription output_desc;
	output_desc.componentType = kAudioUnitType_Output;
	output_desc.componentSubType = kAudioUnitSubType_RemoteIO;
	output_desc.componentFlags = 0;
	output_desc.componentFlagsMask = 0;
	output_desc.componentManufacturer = kAudioUnitManufacturer_Apple;

        // Add nodes to the graph to hold our AudioUnits,
	// You pass in a reference to the  AudioComponentDescription
	// and get back an  AudioUnit
	result = AUGraphAddNode(mGraph, &output_desc, &outputNode);
	result = AUGraphAddNode(mGraph, &mixer_desc, &mixerNode );

	// Now we can manage connections using nodes in the graph.
        // Connect the mixer node's output to the output node's input
	result = AUGraphConnectNodeInput(mGraph, mixerNode, 0, outputNode, 0);

        // open the graph AudioUnits are open but not initialized (no resource allocation occurs here)
	result = AUGraphOpen(mGraph);

	// Get a link to the mixer AU so we can talk to it later
	result = AUGraphNodeInfo(mGraph, mixerNode, NULL, &mMixer);

	//************************************************************
	//*** Make connections to the mixer unit's inputs ***
	//************************************************************
        // Set the number of input busses on the Mixer Unit
	// Right now we are only doing a single bus.
	UInt32 numbuses = 1;
	UInt32 size = sizeof(numbuses);
       result = AudioUnitSetProperty(mMixer, kAudioUnitProperty_ElementCount, kAudioUnitScope_Input, 0, &numbuses, size);

	CAStreamBasicDescription desc;

	// Loop through and setup a callback for each source you want to send to the mixer.
	// Right now we are only doing a single bus so we could do without the loop.
	for (int i = 0; i < numbuses; ++i) {

		// Setup render callback struct
		// This struct describes the function that will be called
		// to provide a buffer of audio samples for the mixer unit.
		AURenderCallbackStruct renderCallbackStruct;
		renderCallbackStruct.inputProc = &renderInput;
		renderCallbackStruct.inputProcRefCon = self;

                // Set a callback for the specified node's specified input
                result = AUGraphSetNodeInputCallback(mGraph, mixerNode, i, &renderCallbackStruct);

		// Get a CAStreamBasicDescription from the mixer bus.
                size = sizeof(desc);
		result = AudioUnitGetProperty(  mMixer,
						kAudioUnitProperty_StreamFormat,
						kAudioUnitScope_Input,
						i,
						&desc,
						&size);
		// Initializes the structure to 0 to ensure there are no spurious values.
		memset (&desc, 0, sizeof (desc));        						

		// Make modifications to the CAStreamBasicDescription
		// We're going to use 16 bit Signed Ints because they're easier to deal with
		// The Mixer unit will accept either 16 bit signed integers or
		// 32 bit 8.24 fixed point integers.
		desc.mSampleRate = kGraphSampleRate; // set sample rate
		desc.mFormatID = kAudioFormatLinearPCM;
		desc.mFormatFlags      = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked;
		desc.mBitsPerChannel = sizeof(AudioSampleType) * 8; // AudioSampleType == 16 bit signed ints
		desc.mChannelsPerFrame = 1;
		desc.mFramesPerPacket = 1;
		desc.mBytesPerFrame = ( desc.mBitsPerChannel / 8 ) * desc.mChannelsPerFrame;
		desc.mBytesPerPacket = desc.mBytesPerFrame * desc.mFramesPerPacket;

		printf("Mixer file format: "); desc.Print();
		// Apply the modified CAStreamBasicDescription to the mixer input bus
		result = AudioUnitSetProperty(  mMixer,
						kAudioUnitProperty_StreamFormat,
						kAudioUnitScope_Input,
						i,
						&desc,
						sizeof(desc));
	}
	// Apply the CAStreamBasicDescription to the mixer output bus
	result = AudioUnitSetProperty(	 mMixer,
					 kAudioUnitProperty_StreamFormat,
					 kAudioUnitScope_Output,
					 0,
					 &desc,
					 sizeof(desc));

	//************************************************************
	//*** Setup the audio output stream ***
	//************************************************************

	// Get a CAStreamBasicDescription from the output Audio Unit
        result = AudioUnitGetProperty(  mMixer,
					kAudioUnitProperty_StreamFormat,
					kAudioUnitScope_Output,
					0,
					&desc,
					&size);

	// Initializes the structure to 0 to ensure there are no spurious values.
	memset (&desc, 0, sizeof (desc));

	// Make modifications to the CAStreamBasicDescription
	// AUCanonical on the iPhone is the 8.24 integer format that is native to the iPhone.
	// The Mixer unit does the format shifting for you.
	desc.SetAUCanonical(1, true);
	desc.mSampleRate = kGraphSampleRate;

       // Apply the modified CAStreamBasicDescription to the output Audio Unit
	result = AudioUnitSetProperty(  mMixer,
					kAudioUnitProperty_StreamFormat,
					kAudioUnitScope_Output,
					0,
					&desc,
					sizeof(desc));

        // Once everything is set up call initialize to validate connections
	result = AUGraphInitialize(mGraph);
}

Now we need to tell the Audio Controller to run those methods at the appropriate time. Hop on over to iPhoneAudioViewController.mm

  • Call initializeAUGraph to create the AUGraph.
  • Call startAUGraph to have the graph start processing audio.
- (void)viewDidLoad {
    [super viewDidLoad];

	[audioController initializeAUGraph];
	[audioController startAUGraph];
}
  • Call stopAUGraph to tell the graph to stop audio processing.
- (void)dealloc {
	[audioController stopAUGraph];
	[audioController release];

    [super dealloc];
}

That whole mess won’t work yet because it references a callback function which we still have to write.
So, on to the next post. iPhone Core Audio Part 3 – Callbacks

About these ads

25 thoughts on “iPhone Core Audio Part 2 – Setting up the AUGraph

  1. I’m having a little trouble understanding the mixer – if I want to trigger multiple sounds at the same time should I set up a new source and input bus for each sound? Will the mixer then handle the mixing for me or how does that work?

    I want to trigger sounds with the lowest possible latency so am I better off writing my own mixing or what do you recommend?

    • The mixer is just there as a convenience to help make connections easier.
      Where you want to do the mixing is pretty application dependent.
      You could do the mixing yourself in your callback if the application is not too complex.

      In the App I’m working on now I will have several instruments that can be added
      and removed as needed. Each instrument will be an object that supplies a callback to a mixer input bus.
      You can set the volume of each mixer input using AudioUnitSetParameter()

      I could be wrong, but I don’t believe the mixer adds any latency.

  2. Note that the CAStreamBasicDescription class is just a bunch of C++ methods on top of a AudioStreamBasicDescription struct, so it’s not too hard to get rid of the ugly C++ stuff. Print() and SetAUCanonical() are the only methods you’re using here, and they’re easy to replace with straight C code.

    • I had contemplated stripping the tutorial down to straight C, but thought that the print() statement was helpful for people to see what is going on while troubleshooting. Once you’re up and running though, you’re right, there’s not much reason to keep the C++ around.

  3. You have done a great service for those of us getting started on CoreAudio. I have been poking around in Apple’s documentation and nothing I have seen from them comes close to the simple, logical, comprehensive (well, relatively speaking, it’s a vast CoreAudio jungle out there) tutorial you have prepared. Many many thanks!

  4. Pingback: iPhone Audio Programming Tips « Zero Inverse

  5. Pingback: iOS Development Link Roundup: Part 1 | iOS/Web Developer's Life in Beta

  6. Thanks for this post, it has been very helpful.

    I’m am having trouble viewing all the source code because it is clipped by a frame(?).

    Are the sources for this posted somewhere or can you fix this clipping?

    Thanks,

    jjm

  7. Hi,
    Many thanks for the post.
    I have two questions!
    First: Is your book similar to this post? I mean I have to do audio processing on iPhone platform for my MASc thesis. Will your book be helpful for me?

    Second is about the code,
    / /Setup render callback struct
    // This struct describes the function that will be called
    // to provide a buffer of audio samples for the mixer unit.
    AURenderCallbackStruct renderCallbackStruct;
    renderCallbackStruct.inputProc = &renderInput;
    renderCallbackStruct.inputProcRefCon = self;
    I get this error “renderInput wasn’t declared in this scope” for the second line of the above code I pasted. What should I do?

  8. Great post. It got me over the hump. Just a question about the section of code you have titled “Setup the audio output stream”. If I understand correctly aren’t you copying the CASBD from the output AU and applying it to the mixer AU output. That means that the mixer makes the conversion between its inputs and outputs and then if you happen to change that a little the output AU has to convert also.

    The way I understand it, that step seems unnecessary and possibly counterproductive. Am I not understanding the process correctly?

    Thanks in advance

    • Not sure what you mean by “happen to change” the mixer output.
      The output unit doesn’t do any converting, it always expects 8.24 fixed point samples, so that is what the mixer unit has to provide.

  9. Hello Tim,

    Thanks for all this useful information. I set up everything and it works fine. Now, I want to use the input from the microphone as an input of the mixer. Without the mixer unit, it works fine, I can get the audio from the io data in my callback, but using AudioOutputUnitStart, not AUGraphStart. When I add the mixer unit to the AUGraph, the same way you do there, the mixer callback gets used, but never the io.

    I do connect the node’s input from io to int mixer’s node input. Using a CAShow(graph), I don’t see the io callback function.

    If you have a bit of time, could you please let me know how come io does not get called with AUGraphStart and how would you do such connection to receive mic input to the mixer?

    Thanks a lot in advance.
    Best

    • The result variable is a place to put error codes. If you were making a real app you would ideally check the result of
      each operation and handle the error if the result != noErr

      I left out a lot of error checking because it clutters up the functional bits and makes it hard to see what’s going on.

      On most CoreAudio functions, the return value is an error code and data is passed in and out as reference parameters.

  10. Loved your example, how would you independently control left and right channels and individually control each channel volume?

    • You could either do a separate mono callback for left and right into different inputs of the mixer, then pan the mixer, and adjust volume using AudioUnit parameters.

      Or you could set up the mixer to have a stereo input, this callback would have an ioData with 2 buffers to fill.

      • Trying to set up the mixer to have a stereo input as suggested but can’t get the stereo. Have setup another buffer for right channel as: AudioSampleType *outB = (AudioSampleType *)ioData->mBuffers[1].mData and fill it with same data.
        Have also tried changing desc.mChannelsPerFrame to 2 in the CAStreamBasicDescription setup but get distortion. If I change desc.mFramesPerPacket to 2 I do get stereo but my sound is also distorted. Individually disabling outA then outB switches the sound from left to right ear so I am on the right track.Suspect interleaving errors ? ? Need guidance.

      • Hi Earle,

        I use this for stereo:

        AudioStreamBasicDescription desc;
        desc.mSampleRate = kSampleRate; // set sample rate
        desc.mFormatID = kAudioFormatLinearPCM;
        desc.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked;
        desc.mBitsPerChannel = sizeof(AudioSampleType) * 8; // AudioSampleType == 16 bit signed ints
        desc.mChannelsPerFrame = 2; // Stereo
        desc.mFramesPerPacket = 1; // 1 frame = 1 packet when using PCM
        desc.mBytesPerFrame = desc.mBytesPerPacket = 2 * desc.mChannelsPerFrame;

        And then this in the callback:

        SInt32 *outA = (SInt32 *)ioData->mBuffers[0].mData;
        for (int i = 0; i < inNumberFrames; i++) {
        // Stereo white noise
        outA[i] = (arc4random() % 100)/100.0f * 32768;
        outA[i] += (SInt16)((arc4random() % 100)/100.0f * 32768)<<16;
        }

  11. Pingback: Remote I/O application with callback | Applerr.com All about Apple Products

  12. Many thanks again for sharing this. One potential bug-fix –

    In initializeAUGraph, in the third major section of code (“Setup the audio output stream”) – the comments claim you are going to set the Description of the Output AudioUnit, however the code actually just re-sets the Description of the Mixer AudioUnit, which was already accomplished in the second major section.

    If I was to guess at your intent in the third major section of code, I’d say it was to set the format of the output bus of the Mixer AudioUnit to be the 8.24-bit fixed-point linear PCM that the iPhone’s RemoteIO Audio Unit is expecting. If this is correct, the following two fixes would correct the code and comments and clear up the duplication:

    1) In the second major section of code (“Make connections to the mixer unit’s inputs”) – Remove the last line of code. Just to be clear, this is the line prefixed by the comment “Apply the CAStreamBasicDescription to the mixer output bus”. This duplicates partially the functionality of the third major section of code, and is unnecessary.

    2) In the third major section of code (“Setup the audio output stream”) – change the comment “AUCanonical on the iPhone is the 8.24 integer format” to the correct “AUCanonical on the iPhone is the 8.24 fixed-point format” (Change “integer” to “fixed-point”)

    One final thought – if you just delete the third major section of code entirely, everything still works fine. The default output format of the Multi-Channel Mixer is the 8.24 fixed-point linear PCM format that the iPhone RemoteIO Audio Unit is expecting. I tested this in the simulator and also on my iPhone 4, however I didn’t test it on an iPad. It may be that in the future these two formats (mixer output format and remote_io input format) don’t match up, and this code will then do something useful. So I’d leave it in.

    again, many thanks for your time and energy in sharing this.
    Ian Charnas

  13. Is there any reason for this code immediately after setting up the mixer inputs?

    // Apply the CAStreamBasicDescription to the mixer output bus
    result = AudioUnitSetProperty( mMixer,
    kAudioUnitProperty_StreamFormat,
    kAudioUnitScope_Output,
    0,
    &desc,
    sizeof(desc));

    You call the same function just a little ways down, and wouldn’t that call render this one meaningless, or am I not seeing what is going on there?

    Thanks
    Bob

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s