Bits to notes. To money?

Previously on Stream of unconsciousness: a first (simple) audio streaming implementation has been put in place using the Audio File Stream Example. The main parser mechanism has been analysed but a lot of mysteries about the callbacks have to be uncovered. If you are unsatisfied with this micro-summary, you can read the whole story on this post.

Time has come to introduce another audio-related library: the Audio Queue Services library. This is a C API under the under the Audio Toolbox framework (the same framework of the Audio File Stream library introduced in the previous post).

This library can be used to record and, as in our case, to play audio. Two fundamental documents to understand audio playing are: About Audio Queues that introduces the library main concepts and structure, and Playing Audio that, as its name implies deals with the details of audio playing.

As described in these documents, playing audio is just a matter of:

1. Define a custom structure to manage state, format, and path information.
2. Write an audio queue callback function to perform the actual playback.
3. Write code to determine a good size for the audio queue buffers.
4. Open an audio file for playback and determine its audio data format.
5. Create a playback audio queue and configure it for playback.
6. Allocate and enqueue audio queue buffers. Tell the audio queue to start playing. When done, the playback callback tells the audio queue to stop.
7. Dispose of the audio queue. Release resources.

Let’s see how these tasks are dealt with in the code example starting with the first callback we still have to describe: PropertyListenerProc.

The PropertyListenerProc callback definition is:

typedef void (*AudioFileStream_PropertyListenerProc) (
   void                         *inClientData,
   AudioFileStreamID            inAudioFileStream,
   AudioFileStreamPropertyID    inPropertyID,
   UInt32                       *ioFlags
);

This callback has to be implemented in your code and is invoked by the parser each time it parses (it is its job, after all) or sets an audio property. The meaning is (parser speaking now): “Let’s see these bytes, and these, and these..oh wait I’ve found/set an audio property!!”.

The example code does the following:

• retrieves the user data (as a MyData structure, more on this later)
• if the property that has been set is a Ready to Produce Packets property (kAudioFileStreamProperty_ReadyToProducePackets), meaning that the parser has covered all the useful properties and is going to parse the audio data:
• the audio queue and the audio queue buffers are initialized
• a listener (MyAudioQueueIsRunningCallback callback) is attached to the change of state of the audio queue ‘running’ property.

The first thing worth some explanation is the MyData structure and its use in the code example. Remember that the Audio File Stream library is function oriented; in particular note that the inPropertyListenerProc and inPacketsProc parameters of the AudioFileStreamOpen function are bound to specific function signature and cannot be ObjC instance methods. Each time a callback is called it has no context per-se (it is somehow stateless); the context of this callback (i.e. all the relevant information that describe the streaming session) has to be passed as a parameter, and that’s the meaning of the inClientData parameter passed to the AudioFileStreamOpen function. The client code uses this parameter to track a certain streaming session state and to retrieve its context in the different callbacks. In the case of this code example, a struct, called MyData, is used to keep track of everything the process is in need of (e.g. the audio file stream identifier, the audio queue and queue buffers, the mutexes to synchronize the bytes consumption with the music being played and so on).

The structure is declared as follows:

struct MyData
{
    AudioFileStreamID audioFileStream;  // the audio file stream parser

    AudioQueueRef audioQueue;                               // the audio queue
    AudioQueueBufferRef audioQueueBuffer[kNumAQBufs];       // audio queue buffers

    AudioStreamPacketDescription packetDescs[kAQMaxPacketDescs];    // packet descriptions for enqueuing audio

    unsigned int fillBufferIndex;   // the index of the audioQueueBuffer that is being filled
    size_t bytesFilled;             // how many bytes have been filled
    size_t packetsFilled;           // how many packets have been filled

    bool inuse[kNumAQBufs];         // flags to indicate that a buffer is still in use
    bool started;                   // flag to indicate that the queue has been started
    bool failed;                    // flag to indicate an error occurred

    pthread_mutex_t mutex;          // a mutex to protect the inuse flags
    pthread_cond_t cond;            // a condition varable for handling the inuse flags
    pthread_cond_t done;            // a condition varable for handling the inuse flags
};
typedef struct MyData MyData;

The definition of this structure clears the point 1 of the above list.

Two other important things that happen under the PropertyListenerProc callback are the audio queue and queue buffers initialization and the MyAudioQueueIsRunningCallback set up.

The audio queue is initialized (AudioQueueNewOutput function) with the format of the audio to play (&asbd), an audio queue callback (MyAudioQueueOutputCallback, see point 2 of the above list) and the timeless MyData structure. An output parameter is used to save a reference to the newly created playback audio queue; guess where??? In a field of the MyData structure, of course!! This covers the points 4 (more or less) and 5 of the list.

As a part of the audio queue initialization, the magic cookie is also initialized. Some file formats (e.g. MPEG 4 AAC) need a structure that contains some audio metadata: that’s the magic cookie. In the code example its value is read from the audio stream session (AudioFileStreamGetProperty function) and it is written in the audio queue (AudioQueueSetProperty function).

Three (as suggested by the documentation) audio queue buffers are also initialized (AudioQueueAllocateBuffer function) with a constant size (this should shortcut the point 3 of the list). Part of the point 6 of the list is also covered (later for the other part).

That’s the whole setup thing that comes with the PropertyListenerProc callback, called when the parser is ‘ready to play’.

When everything is in place, the real audio packets make their appearance. Each time some packets are parsed the PacketsProc callback is called.

It is declared as follows:

typedef void (*AudioFileStream_PacketsProc) (
   void                          *inClientData,
   UInt32                        inNumberBytes,
   UInt32                        inNumberPackets,
   const void                    *inInputData,
   AudioStreamPacketDescription  *inPacketDescriptions
);

This callback fills the queue buffer in use with the audio data (the bytes of the audio packets) and, each time a buffer is full, it enqueues it (MyEnqueueBuffer function that calls the AudioQueueEnqueueBuffer function and starts the queue if needed) and asks for the next buffer to be filled. The latter task is performed by the WaitForFreeBuffer function, this function asks for a ‘used’ buffer (a buffer with already played content) and waits for it stopping the thread with a mutex. Keep in mind that the bytes producer (the stream parser) is faster than the bytes consumer (the audio ‘player’) and some ‘retard’ has to be artificially put on the producer size (and that is exactly what this function does). That completes the point 6 of the above list. The point 7 (dispose and release) is managed at the end of the loop in the main function.

Each time an audio buffer has been ‘played’, the audio queue callback (MyAudioQueueOutputCallback) is called. It allows unlocking the mutex set by the WaitForFreeBuffer function. Remember the listener on the change of state of the audio queue ‘running’ property (MyAudioQueueIsRunningCallback)? This function is also used to unlock the mutex when the audio queue has done playing.

More or less that’s the whole story of the Audio File Stream Example. It seems a lot convoluted (OK, it IS actually convoluted), but the more you study it and the more you feel at ease with it.

The example is just this, an example to introduce two audio libraries; it is far from complete (for example it doesn’t handle CBR data) and has some limitations. For a detailed analysis of its problems and a brilliant example of how to overcome them have a look at Streaming and playing an MP3 stream and its revision: Revisiting an old post: Streaming and playing an MP3 stream. They offer a very interesting analysis of the  Audio File Stream Example and its limitations and provide an extension to the example to implement a sample application that streams and plays an audio file from a URL on the iPhone or Mac (source code downloadable by Github!!!). The whole blog (Cocoa with Love) by Matt Gallager is a GREAT place to find technical advice, hints, code examples and more; definitely a place to go if you are involved in OS X / iOS development (we’ve also added the link to our blogroll section on the right).

That’s it. See you next time with our inglorious first implementation of the audio streaming.

It just works!!!

With the Bluetooth connection in place (even in its simplest form), it was time for ourselves to penetrate the audio streaming deepest secrets. It was much a tougher challenge than we had anticipated, one full of trials and errors and wrong assumptions, one that deserves more than one single post.

After some browsing, the first relevant document we found was the Audio File Stream Services Reference. It details the use of the Audio File Stream library under the Audio Toolbox framework.

Trying to write some reasonable code, we stumbled upon a really useful code example. The AudioFileStreamExample, even if targeted at the Mac OS X, allowed us to quick start some audio streaming experiment. It contains just 2 .cpp files (aptly named afsclient and afsserver) and provides a straightforward example of the library usage.

Let’s delve into this example.

The server does really the barely minimum. It:

• listens for socket connection requests by clients
• accepts them
• for each connection sends an entire audio file to the corresponding client. The file is actually sent in chunks of 32768 (2 ^ 15) bytes.

The client manages the hard part and actually demonstrates the use of the Audio File Stream library.

At a very high level the client:

• Opens an audio file stream (AudioFileStreamOpen)
• Parses (in loop) each file chunk (bytes array) that receives from the server (AudioFileStreamParseByte)
• Closes the audio file stream (AudioFileStreamClose)

OSStatus  AudioFileStreamOpen (
   void                                  *inClientData,
   AudioFileStream_PropertyListenerProc  inPropertyListenerProc,
   AudioFileStream_PacketsProc           inPacketsProc,
   AudioFileTypeID                       inFileTypeHint,
   AudioFileStreamID                     *outAudioFileStream
);

inClientData is used to pass a context to the parsing session. The parsing process will involve some callback invocation (see below) and the inClientData object (or structure) will be passed along in the callbacks. Keep in mind that this library is not object-oriented but function-oriented. The state relative to a particular parsing session should be managed by your code and the use of this parameter can provide a great help.

inPropertyListenerProc is the callback that will be invoked by the parser each time it manages to parse an audio property (more on the callback structure later on).

• inPacketsProc is the callback that will be invoked by the parser each time it manages to parse some audio packets (more on the callback structure later on).
• inFileTypeHint is used to give the parser any hint about the audio file type (pass 0 if you are unsure of the actual type). It is very useful in case the file type is not univocally recognizable from its content.
• outAudioFileStream returns an identifier created by the library and associated to the newly opened parsing session. You have to use this identifier in all the other library function calls to specify the streaming session you are referring to.

A little clarification is needed: when I talk about a ‘parser’ (and I’ll also depict one in the coming diagrams) I don’t mean an actual parser object (since I’ve just said that the library is function-oriented). Think of it as an abstraction or as the library itself or as a service provided by the library, anything that helps visualizing the problem at hand. But remember that you won’t invoke any methods of any objects, just public functions provided by the library.

OSStatus AudioFileStreamParseBytes (
   AudioFileStreamID  inAudioFileStream,
   UInt32             inDataByteSize,
   const void         *inData,
   UInt32             inFlags
);

• inAudioFileStream is the identifier returned by the AudioFileStreamOpen function (as the outAudioFileStream parameter). It identifies the streaming session you are referring to.
• inDataByteSize is the number of bytes you are passing the function.
• inData is the actual bytes array.
• inFlags allows you to declare a discontinuity in the stream. Pass 0 by default.

The following diagram gives an overview of the process:

Parse this

1. The AudioFileStreamParseBytes is invoked repeatedly for each file chunk (bytes array)
2. When the parser recognizes (parses) a property it invokes the PropertyListenerProc callback
3. When the parser recognizes (parses) some audio packets it invokes the PropertyListenerProc callback

We will further analyse the callback in the next posts.

Finally, the AudioFileStreamClose function is used (any guess?) to close an audio file stream parsing session.

OSStatus AudioFileStreamClose (
   AudioFileStreamID inAudioFileStream
);

• inAudioFileStream is the identifier returned by the AudioFileStreamOpen function (as the outAudioFileStream parameter). It identifies the streaming session you are referring to.

That actually concludes the AudioFileStreamExample usage of the Audio File Stream library. Yes, because before having any meaningful sound produced by your device, at least another library (and its usage in the example) has to be introduced: the Audio Queue Services library.

But that’s it for now. It’s late in the morning. It’s Saturday. The sun is shining. The sea is beautiful. I’ve to take my wife at school (teacher, NOT student). A lot of reasons to stop posting.

Will be right back for Part 2.

Chitty chatty chitty chatty chittychittychitty chatty

Operational, at last. It’s time to begin sharing some code, some trials and errors and some discoveries we made while experimenting with iOS audio streaming. No secret recipe here or in the coming posts, just some little value uncovered during the process and to be returned to the community (haven’t you still read the Wikinomics book?? No?? Run and GET IT!!!).

Since our initial equipment comprised a MacBook Air and an iPad (version 1), our first concern was about running and testing the code on 2 different device (since, you know, audio self-streaming isn’t so exciting after all, as well as other self-something stuff). Hence Mission n. 1 was to verify the feasibility of a Bluetooth connection between the iPad and the iPhone/iPad simulator that comes with the iOS SDK.

The code for the Bluetooth connection was derived by the Beginning iOS 4 Application Development book and is kind of standard use of the GKPeerPickerController class of the GameKit Framework. We will use this code only in the first scouting phase because, although it is really simple, it alerts the user about the connection requests and asks him/her to accept it (see picture below, could be very annoying for multiple connections). An alert-less connection is already implemented in some (but not all!!) of the apps described in the previous posts. I guess this will involve a departure from the ‘picker’ approach and the direct use of the GKSession class, but I’ll leave it for later (any suggestion welcomed!!).

Do you accept connection? Do you? Do You?? DO YOU?????

Let’s look at some code:

   GKPeerPickerController picker = [[GKPeerPickerController alloc] init];

   picker.delegate = self;

   picker.connectionTypesMask = GKPeerPickerConnectionTypeNearby;

   [picker show];

It’s really as simple as this!! In the code sample above a picker variable (of type GKPeerPickerController):

• is allocated and initialized (it’ll be released soon..)
• its delegate is assigned to some class instance (self in my case, see below for the protocol involved)
• its connection type is set to ‘nearby’ (i.e. using Bluetooth; the other possibility is GKPeerPickerConnectionTypeOnline for an internet-based connection)
• finally the picker view itself is shown (see picture below)

Let me pick something...

The picker interface allows to discover nearby devices and to send (and accept) connection requests. All this is managed under the surface of your code that is in charge of just managing the delegate events (via the GKPeerPickerControllerDelegate protocol).

You may, for example, be interested in detecting when a peer connects to you or cancel your connection request; you will only need handling these two protocol methods:

• peerPickerController:didConnectPeer:toSession:
• peerPickerControllerDidCancel:

-(void)peerPickerController:(GKPeerPickerController *)pk didConnectPeer:(NSString *)peerID toSession:(GKSession *)session
{
   //Do something (for example alert the user that a peer has just connected
   //Or just do nothing, life is too short to be annoyed by alerts!!

   self.currentSession = session;

   session.delegate = self;

   [session setDataReceiveHandler:self withContext:nil];

   pk.delegate = nil;

   [pk dismiss];

   [pk release];
}

Please don’t question about my opening bracket that is sooo C, C++, C# and so feeew Java, ObjectiveC and so on; my need for vertical symmetry doesn’t allow me to experiment with fancy formatting for now.

In the example above we have:

• saved the connection session (a GKSession object) in a member variable
• set the delegate of the session to an object that implements the GKSessionDelegate protocol (it will be detailed soon)
• set the object that handles data received from other peers (in our case the self object, more on this later)
• set the picker delegate to null (or nil or nothing or ..)
• dismissed the picker view
• released the picker object

A similar, but simpler, code can be implemented for the peerPickerControllerDidCancel: method:

-(void)peerPickerControllerDidCancel:(GKPeerPickerController *)pk didConnectPeer:(NSString *)peerID toSession:(GKSession *)session
{
   ...

   [session setDataReceiveHandler:self withContext:nil];

   pk.delegate = nil;

   [pk dismiss];

   [pk release];
}

The peerPickerController:didConnectPeer:toSession: example (former above) contains two interesting lines of code: the first one is the one in which we assign the session delegate to an object (self in our case). This object must implement the GKSessionDelegate protocol that allows tracking things like peers state changes, connection requests and connection failures. This protocol has 4 required (mandatory) methods:

• session:peer:didChangeState:
• session:didReceiveConnectionRequestFromPeer:
• session:connectionWithPeerFailed:withError:
• session:didFailWithError:

The second, much more interesting, code line in the peerPickerController:didConnectPeer:toSession: example sets the object that handles data received from other peers; the handler object must implement a method with the following signature:

- (void) receiveData:(NSData *)data fromPeer:(NSString *)peer inSession: (GKSession *)session context:(void *)context;

This is the method used to handle data received from a peer; we will see an example of it in the following posts when we will try to do some real audio streaming.

So, what about Mission n. 1? Is it actually possible to connect a real device and a simulator instance via Bluetooth? The good news is YES. Even with some weird behaviour I’ve not spent time to deal with (the device picker doesn’t detect the simulator but the simulator picker detects the device, and it’s all we need), the connection works!!

Here we are, eventually. We have 2 devices (please, don’t tell the simulator he’s not a real device), that ignored each the other, now fully connected. The deaf can hear and the dumb can speak. OK, they’ve nothing to say each to the other, but how many miracles do you expect in a single blog post???