MIDI Driven Animation using CoreMIDI in Objective C

现在开始写、记博客

原文地址：http://www.deluge.co/?q=midi-driven-animation-core-audio-objective-c

MIDI Driven Animation using CoreMIDI in Objective C

Submitted by bensmiley on Wed, 03/21/2012 - 23:27

So in this post I'm going to explain how to produce MIDI driven animation on OSX or iOS using the CoreMIDI and CoreAudio frameworks. When I first started trying to do this I thought it would be easy - just register a callback in the MIDI player which is called every time a MIDI message is played. Unfortunately this is not possible and I ended up spending three long days figuring it out from the limited documentation available. Hopefully this post will save someone some time!

Project files

A fully working X-Code project can be downloaded here.

The Goal

In this guide I will explain how to do the following:

• Load and play a MIDI sequence from a file using a MusicPlayer
• Play the MIDI notes with an instrument effect (SoundFont) using an AUGraph
• Create a virtual endpoint to intercept and display the MIDI messages in realtime

 

Load and play a MIDI Sequence

The following tasks are needed to load and play a MIDI file:

• Create a MusicSequence to hold the MIDI information
• Get a NSURL to hold the path to the MIDI file
• Load the sequence file into the sequence using MusicSequenceFileLoad
• Create a new MusicPlayer, add the sequence and play the sequence

Now here's the code. You will need to include the following frameworks: CoreAudio, CoreMIDI and AudioToolbox as well as the import: AudioToolbox/MusicPlayer.h

    // Create a new music sequence
    MusicSequence s;
    // Initialise the music sequence
    NewMusicSequence(&s);
 
    // Get a string to the path of the MIDI file which
    // should be located in the Resources folder
    // I'm using a simple test midi file which is included in the download bundle at the end of this document
    NSString *midiFilePath = [[NSBundle mainBundle]
                               pathForResource:@"simpletest"
                               ofType:@"mid"];
 
    // Create a new URL which points to the MIDI file
    NSURL * midiFileURL = [NSURL fileURLWithPath:midiFilePath];
 
 
    MusicSequenceFileLoad(s, midiFileURL, 0, 0);
 
    // Create a new music player
    MusicPlayer  p;
    // Initialise the music player
    NewMusicPlayer(&p);
 
    // Load the sequence into the music player
    MusicPlayerSetSequence(p, s);
    // Called to do some MusicPlayer setup. This just 
    // reduces latency when MusicPlayerStart is called
    MusicPlayerPreroll(p);
    // Starts the music playing
    MusicPlayerStart(p);
 
    // Get length of track so that we know how long to kill time for
    MusicTrack t;
    MusicTimeStamp len;
    UInt32 sz = sizeof(MusicTimeStamp);
    MusicSequenceGetIndTrack(s, 1, &t);
    MusicTrackGetProperty(t, kSequenceTrackProperty_TrackLength, &len, &sz);
 
 
    while (1) { // kill time until the music is over
        usleep (3 * 1000 * 1000);
        MusicTimeStamp now = 0;
        MusicPlayerGetTime (p, &now);
        if (now >= len)
            break;
    }
 
    // Stop the player and dispose of the objects
    MusicPlayerStop(p);
    DisposeMusicSequence(s);
    DisposeMusicPlayer(p);

Hopefully you will have heard a rather mechanical scale followed by a chromatic scale. It's basic but at least it's a start. The next step is to create an AU graph so that we can play our MIDI file with an instrument effect.

Creating an AUGraph

When I first started reading about AU Graphs I thought it sounded horribly incomprehensible and opaque. In reality it's not too bad just a bit fiddly to set up.

An AUGraph is a container to hold a collection of AUNodes. AU Nodes are effects units which are supplied by Apple. Really it's just like music units in real life. Maybe you have a MIDI keyboard and you want to output the sound as a trumpet with an echo effect. You would need to plug your keyboard into a box which translates MIDI messages and turns them into trumpet sounds. This box would need to be plugged into an echo unit which is plugged into the speakers.

Choosing your AUNodes

In CoreAudio you choose the type of AUNode you need using three properties (defined by ENUMs):

• componentManufacturer: The author of the AUNode in this case we will be using audio units from Apple - kAudioUnitManufacturer_Apple
• componentType: The unit type
• componentSubType: The sub unit type

The unit type and sub-unit type can be found in the Apple documentation or in the header file AUComponent.h. Basically to find the audio unit you need it's easiest to use Google. But say I want a high pass filter, I look in the AUComponent.h header file and find kAudioUnitSubType_HighPassFilter - this is the sub type. I then count how many sub type definitions there were before this one - in this case 2. I then look at the top of the document and look at the third Audio unit type defined kAudioUnitType_MusicEffect. Now I have my manufacturer, type and sub type and I can use the Audio Unit.

For this example we will be using the following two Audio Units:

• Sampler: This is a unit converts MIDI to music sounds defined in a Sound Font or AUPreset and is available on iOS 5
• RemoteIO: This unit allows us to output sounds to iPhone speakers

  So here's the code - adapted from an example provided by Apple but with extra comments.

  - (BOOL) createAUGraph {
   
      // Each core audio call returns an OSStatus. This means that we
      // Can see if there have been any errors in the setup
  OSStatus result = noErr;
   
      // Create 2 audio units one sampler and one IO
  AUNode samplerNode, ioNode;
   
      // Specify the common portion of an audio unit's identify, used for both audio units
      // in the graph.
      // Setup the manufacturer - in this case Apple
  AudioComponentDescription cd = {};
  cd.componentManufacturer     = kAudioUnitManufacturer_Apple;
   
      // Instantiate an audio processing graph
  result = NewAUGraph (&_processingGraph);
      NSCAssert (result == noErr, @"Unable to create an AUGraph object. Error code: %d '%.4s'", (int) result, (const char *)&result);
   
  //Specify the Sampler unit, to be used as the first node of the graph
  cd.componentType = kAudioUnitType_MusicDevice; // type - music device
  cd.componentSubType = kAudioUnitSubType_Sampler; // sub type - sampler to convert our MIDI
   
      // Add the Sampler unit node to the graph
  result = AUGraphAddNode (self.processingGraph, &cd, &samplerNode);
      NSCAssert (result == noErr, @"Unable to add the Sampler unit to the audio processing graph. Error code: %d '%.4s'", (int) result, (const char *)&result);
   
  // Specify the Output unit, to be used as the second and final node of the graph
  cd.componentType = kAudioUnitType_Output;  // Output
  cd.componentSubType = kAudioUnitSubType_RemoteIO;  // Output to speakers
   
      // Add the Output unit node to the graph
  result = AUGraphAddNode (self.processingGraph, &cd, &ioNode);
      NSCAssert (result == noErr, @"Unable to add the Output unit to the audio processing graph. Error code: %d '%.4s'", (int) result, (const char *)&result);
   
      // Open the graph
  result = AUGraphOpen (self.processingGraph);
      NSCAssert (result == noErr, @"Unable to open the audio processing graph. Error code: %d '%.4s'", (int) result, (const char *)&result);
   
      // Connect the Sampler unit to the output unit
  result = AUGraphConnectNodeInput (self.processingGraph, samplerNode, 0, ioNode, 0);
      NSCAssert (result == noErr, @"Unable to interconnect the nodes in the audio processing graph. Error code: %d '%.4s'", (int) result, (const char *)&result);
   
  // Obtain a reference to the Sampler unit from its node
  result = AUGraphNodeInfo (self.processingGraph, samplerNode, 0, &_samplerUnit);
      NSCAssert (result == noErr, @"Unable to obtain a reference to the Sampler unit. Error code: %d '%.4s'", (int) result, (const char *)&result);
   
  // Obtain a reference to the I/O unit from its node
  result = AUGraphNodeInfo (self.processingGraph, ioNode, 0, &_ioUnit);
      NSCAssert (result == noErr, @"Unable to obtain a reference to the I/O unit. Error code: %d '%.4s'", (int) result, (const char *)&result);
   
      return YES;
  }

  Next we need to create a function to start the AUGraph running. This is equivalent to turning on the physical devices.

  // Starting with instantiated audio processing graph, configure its 
  // audio units, initialize it, and start it.
  - (void) configureAndStartAudioProcessingGraph: (AUGraph) graph {
   
      OSStatus result = noErr;
      if (graph) {
   
          // Initialize the audio processing graph.
          result = AUGraphInitialize (graph);
          NSAssert (result == noErr, @"Unable to initialze AUGraph object. Error code: %d '%.4s'", (int) result, (const char *)&result);
   
          // Start the graph
          result = AUGraphStart (graph);
          NSAssert (result == noErr, @"Unable to start audio processing graph. Error code: %d '%.4s'", (int) result, (const char *)&result);
   
          // Print out the graph to the console
          CAShow (graph); 
      }
  }

  So, now we've created a new audio graph with a sampler and an output unit. We've connected the sampler unit to the output unit and we've started the graph. Finally we need to set up the instrument effect, connect the music sequence and play.

  Set up the sound effect

  // Load a sound effect from a SoundFont file
  -(OSStatus) loadFromDLSOrSoundFont: (NSURL *)bankURL withPatch: (int)presetNumber {
   
      OSStatus result = noErr;
   
      // fill out a bank preset data structure
      AUSamplerBankPresetData bpdata;
      bpdata.bankURL  = (__bridge CFURLRef) bankURL;
      bpdata.bankMSB  = kAUSampler_DefaultMelodicBankMSB;
      bpdata.bankLSB  = kAUSampler_DefaultBankLSB;
      bpdata.presetID = (UInt8) presetNumber;
   
      // set the kAUSamplerProperty_LoadPresetFromBank property
      result = AudioUnitSetProperty(self.samplerUnit,
                                kAUSamplerProperty_LoadPresetFromBank,
                                kAudioUnitScope_Global,
                                0,
                                &bpdata,
                                sizeof(bpdata));
   
      // check for errors
      NSCAssert (result == noErr,
             @"Unable to set the preset property on the Sampler. Error code:%d '%.4s'",
             (int) result,
             (const char *)&result);
   
      return result;
  }

  This code takes a sound font NSURL and a preset number as input. The NSURL should point to the Sound Font file in your Resources directory. Sound Fonts can hold a number of instrument effects so the presetNumber defines which one should be used.

  Now we just repeat what we did before but with a few added lines (marked by stars).

  // Create a new music player
      MusicPlayer  p;
      // Initialise the music player
      NewMusicPlayer(&p);
   
   
     // ************* Tell the music sequence to output through our new AUGraph 
      MusicSequenceSetAUGraph(s, self.processingGraph);
   
   
      // ************* Load the sound font from file
      NSURL *presetURL = [[NSURL alloc] initFileURLWithPath:[[NSBundle mainBundle] pathForResource:@"Gorts_Filters" ofType:@"sf2"]];
   
      // ************* Initialise the sound font
      [self loadFromDLSOrSoundFont: (NSURL *)presetURL withPatch: (int)10];
   
      // Load the sequence into the music player
      MusicPlayerSetSequence(p, s);
      // Called to do some MusicPlayer setup. This just 
      // reduces latency when MusicPlayerStart is called
      MusicPlayerPreroll(p);
      // Starts the music playing
      MusicPlayerStart(p);
   
      // Get length of track so that we know how long to kill time for
      MusicTrack t;
      MusicTimeStamp len;
      UInt32 sz = sizeof(MusicTimeStamp);
      MusicSequenceGetIndTrack(s, 1, &t);
      MusicTrackGetProperty(t, kSequenceTrackProperty_TrackLength, &len, &sz);
   
   
      while (1) { // kill time until the music is over
          usleep (3 * 1000 * 1000);
          MusicTimeStamp now = 0;
          MusicPlayerGetTime (p, &now);
          if (now >= len)
              break;
      }
   
      // Stop the player and dispose of the objects
      MusicPlayerStop(p);
      DisposeMusicSequence(s);
      DisposeMusicPlayer(p);

  From the sample project you should understand how to play a MIDI file with a Sound Font effect. The final step is to get real time access to the messages being parsed by the MusicPlayer. To do this we need to add an extra step to our chain. Currently it looks like this:
  MIDI File -> Sequence -> Sampler -> IO Unit -> Speakers
  We want it to look like this:
  MIDI File -> Sequence -> callback function to read messages -> Sampler -> IO Unit -> Speakers
  With this system we will receive the messages in real-time before passing them on to the Sampler unit. This cal be achieved by creating a new MIDI end point. A MIDI endpoint is a destination where midi messages can be sent. This could be another MIDI app on your iPhone, an external MIDI instrument or, in this case, a callback function.

  
  
   

  Creating a new MIDI end point

  In order to capture the MIDI messages we need a destination that they can be sent to. This can be done by creating a MIDI end point:

      // Create a client 
      // This provides general information about the state of the midi engine to the callback MyMIDINotifyProc 
      MIDIClientRef virtualMidi;
      result = MIDIClientCreate(CFSTR("Virtual Client"),
                                MyMIDINotifyProc,
                                NULL,
                                &virtualMidi);
   
      NSAssert( result == noErr, @"MIDIClientCreate failed. Error code: %d '%.4s'", (int) result, (const char *)&result);
   
      // Create an endpoint
      // In this endpoint we define our client, a name: Virtual Destination
      // a callback function which will receive the MIDI packets: MyMIDIReadProc
      // a reference to the sampler unit for use within our callback
      // a point to our end point: virtualEndpoint 
      MIDIEndpointRef virtualEndpoint;
      result = MIDIDestinationCreate(virtualMidi, @"Virtual Destination", MyMIDIReadProc, self.samplerUnit, &virtualEndpoint);
   
      NSAssert( result == noErr, @"MIDIDestinationCreate failed. Error code: %d '%.4s'", (int) result, (const char *)&result);

  We also need to implement the callbacks in our code. This example will log each note as it's played:

  // Get general midi notifications
  void MyMIDINotifyProc (const MIDINotification  *message, void *refCon) {
      printf("MIDI Notify, messageId=%d,", message->messageID);
  }
  // Get the MIDI messages as they're sent
  static void MyMIDIReadProc(const MIDIPacketList *pktlist,
                             void *refCon,
                             void *connRefCon) {
   
      // Cast our Sampler unit back to an audio unit
      AudioUnit *player = (AudioUnit*) refCon;
   
      MIDIPacket *packet = (MIDIPacket *)pktlist->packet;
      for (int i=0; i < pktlist->numPackets; i++) {
          Byte midiStatus = packet->data[0];
          Byte midiCommand = midiStatus >> 4;
   
         // If the command is note-on
          if (midiCommand == 0x09) {
              Byte note = packet->data[1] & 0x7F;
              Byte velocity = packet->data[2] & 0x7F; 
   
             // Log the note letter in a readable format
              int noteNumber = ((int) note) % 12;
              NSString *noteType;
              switch (noteNumber) {
                  case 0:
                      noteType = @"C";
                      break;
                  case 1:
                      noteType = @"C#";
                      break;
                  case 2:
                      noteType = @"D";
                      break;
                  case 3:
                      noteType = @"D#";
                      break;
                  case 4:
                      noteType = @"E";
                      break;
                  case 5:
                      noteType = @"F";
                      break;
                  case 6:
                      noteType = @"F#";
                      break;
                  case 7:
                      noteType = @"G";
                      break;
                  case 8:
                      noteType = @"G#";
                      break;
                  case 9:
                      noteType = @"A";
                      break;
                  case 10:
                      noteType = @"Bb";
                      break;
                  case 11:
                      noteType = @"B";
                      break;
                  default:
                      break;
              }
              NSLog([noteType stringByAppendingFormat:[NSString stringWithFormat:@": %i", noteNumber]]);
   
              // Use MusicDeviceMIDIEvent to send our MIDI message to the sampler to be played
              OSStatus result = noErr;
              result = MusicDeviceMIDIEvent (player, midiStatus, note, velocity, 0);
   
          }
          packet = MIDIPacketNext(packet);
      }
  }

  The final step is to modify our main function to set the MusicSequence destination to our new endpoint:

  OSStatus result = noErr;
   
      self.graphSampleRate = 44100.0; 
   
   
      [self createAUGraph];
      [self configureAndStartAudioProcessingGraph: self.processingGraph];
   
      // Create a client
      MIDIClientRef virtualMidi;
      result = MIDIClientCreate(CFSTR("Virtual Client"),
                                MyMIDINotifyProc,
                                NULL,
                                &virtualMidi);
   
      NSAssert( result == noErr, @"MIDIClientCreate failed. Error code: %d '%.4s'", (int) result, (const char *)&result);
   
      // Create an endpoint
      MIDIEndpointRef virtualEndpoint;
      result = MIDIDestinationCreate(virtualMidi, @"Virtual Destination", MyMIDIReadProc, self.samplerUnit, &virtualEndpoint);
   
      NSAssert( result == noErr, @"MIDIDestinationCreate failed. Error code: %d '%.4s'", (int) result, (const char *)&result);
   
   
   
      // Create a new music sequence
      MusicSequence s;
      // Initialise the music sequence
      NewMusicSequence(&s);
   
      // Get a string to the path of the MIDI file which
      // should be located in the Resources folder
      NSString *midiFilePath = [[NSBundle mainBundle]
                                 pathForResource:@"simpletest"
                                 ofType:@"mid"];
   
      // Create a new URL which points to the MIDI file
      NSURL * midiFileURL = [NSURL fileURLWithPath:midiFilePath];
   
   
      MusicSequenceFileLoad(s, (__bridge CFURLRef) midiFileURL, 0, 0);
   
      // Create a new music player
      MusicPlayer  p;
      // Initialise the music player
      NewMusicPlayer(&p);
   
      // ************* Set the endpoint of the sequence to be our virtual endpoint
      MusicSequenceSetMIDIEndpoint(s, virtualEndpoint);
   
      // Load the ound font from file
      NSURL *presetURL = [[NSURL alloc] initFileURLWithPath:[[NSBundle mainBundle] pathForResource:@"Gorts_Filters" ofType:@"sf2"]];
   
      // Initialise the sound font
      [self loadFromDLSOrSoundFont: (NSURL *)presetURL withPatch: (int)10];
   
      // Load the sequence into the music player
      MusicPlayerSetSequence(p, s);
      // Called to do some MusicPlayer setup. This just 
      // reduces latency when MusicPlayerStart is called
      MusicPlayerPreroll(p);
      // Starts the music playing
      MusicPlayerStart(p);
   
      // Get length of track so that we know how long to kill time for
      MusicTrack t;
      MusicTimeStamp len;
      UInt32 sz = sizeof(MusicTimeStamp);
      MusicSequenceGetIndTrack(s, 1, &t);
      MusicTrackGetProperty(t, kSequenceTrackProperty_TrackLength, &len, &sz);
   
   
      while (1) { // kill time until the music is over
          usleep (3 * 1000 * 1000);
          MusicTimeStamp now = 0;
          MusicPlayerGetTime (p, &now);
          if (now >= len)
              break;
      }
   
      // Stop the player and dispose of the objects
      MusicPlayerStop(p);
      DisposeMusicSequence(s);
      DisposeMusicPlayer(p);

  So there you have it! Play your MIDI file through a nice reedy SoundFont while collecting the messages to drive your animation! I hope this saves you the 3 days it took me to figure it out! Here's the link again to the project files in case you missed it at the top of the guide. Project Files.

  
  
   

  Update:

  It's been pointed out to me that several resource files are missing from the project - a midi file called simpletest.mid and a sound font file called Gorts_Filters.SF2. These files can be downloadedhere. To add them to the project you need to right click on the resources folder in XCode and click "Add Files". As a side note, this code should work with any MIDI file and any Sound Font file. The only thing to watch with sound font files is that the preset/patch that you're requesting exists.

  If you want to ask a general question about CoreAudio or discuss your CoreAudio issue please ask your questions in the CoreAudio section of the forum.