Proof of concept for Silverlight and Speex microphone capture

There is a lot of buzz due to the new capabilities of Silverlight 4, one of them are webcam and microphone capturing.

Here are two great examples that you can use as a starting point: Accessing Web Camera and Microphone by Tim Heuer and Silverlight 4 Augmented Reality Proof Of Concept by René Schulte.

The problem is that in the current API for video and audio capturing, there is support only for RAW data, and that means 32bpp ARGB video and PCM audio samples. Also you have nothing in the server side to handle streaming from client to server, which prevents you from develop any real scenario of video/audio communication tool.

However, this is only the current Beta stage and I’m pretty sure Microsoft will push hard for complement this new features in the incoming versions.

Meanwhile, I handle to port (from JSpeex) the Speex audio codec completely to C# and modified the Tim Heuer’s sample to encode the audio. I uploaded the source code to CodePlex at CSpeex – http://cspeex.codeplex.com/. There is a new button that allows you to save the recorded audio. The audio is encoded in real time to a memory Ogg stream and when you press “Save Audio” button, this stream is decoded and saved as a WAV in your computer.

Here it is the code snippet for the encoding part.

protected override void OnSamples(long sampleTime, long sampleDuration, byte[] sampleData)
{
    for (int i = 0; i < sampleData.Length; )
    {
        int len = Math.Min(sampleData.Length - i, temp.Length - tempOffset);

        Buffer.BlockCopy(sampleData, i, temp, tempOffset, len);

        if (len < temp.Length - tempOffset)
        {
            tempOffset += len;
        }
        else
        {
            tempOffset = 0;

            speexEncoder.processData(temp, 0, temp.Length);

            int encsize = speexEncoder.getProcessedData(temp, 0);

            if (encsize > 0 && (memFile.InnerStream.Position + encsize < ((MemoryStream)memFile.InnerStream).Capacity))
            {
                 writer.writePacket(temp, 0, encsize);
             }
         }

         i += len;
    }
}

Remember, there is a long path to a real chat application, since you need to handle how to upload the encoded stream to the server and do the actual communication between two parties.