SFML community forums

Bindings - other languages => DotNet => Topic started by: s.baus on June 07, 2014, 12:28:53 pm

Title: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on June 07, 2014, 12:28:53 pm
Hello everybody,

I'm currently working on SFML.Net to expand with mp3 support. Therefore I wrote a Stream class which uses NLayer MpegFile to decode the mp3.

   
public class Mp3StreamSFML : SoundStream
    {
        private MpegFile mp3file;
        private int currentBufferSize;
        private short[] currentBuffer;

        public Mp3StreamSFML(String _filename)
        {
            mp3file = new MpegFile(_filename);
            Initialize((uint)mp3file.Channels, (uint)mp3file.SampleRate);
            currentBufferSize = 0;
            currentBuffer = new short[currentBufferSize];
        }

        #region implemented abstract members of SoundStream

        protected override bool OnGetData(out short[] samples)
        {
            if (currentBufferSize <= mp3file.Position)
            {
                byte[] buffer = new byte[512];
                if (mp3file.ReadSamples(buffer, 0, buffer.Length) > 0)
                {
                    Array.Resize(ref currentBuffer, currentBuffer.Length + (buffer.Length / 2));
                    Buffer.BlockCopy(buffer, 0, currentBuffer, currentBufferSize, buffer.Length);
                    currentBufferSize = currentBuffer.Length;
                }
                samples = currentBuffer;
                return true;
            }
            else
            {
                samples = currentBuffer;
                return false;
            }
        }

        protected override void OnSeek(TimeSpan timeOffset)
        {
            mp3file.Position = (long)timeOffset.TotalSeconds;
        }

        #endregion
    }

I use it this way:

                   
try
                    {
                        stream = new Mp3StreamSFML(this.objProgram.getObjCuesheet().getAudiofilePath(true));
                        stream.Play();
                        log.debug("samplerate = " + stream.SampleRate);
                    }
                    catch(Exception ex)
                    {
                        log.fatal(ex.ToString());
                    }

Unfortunately, there is not the correct sound played, its just "juttering" and sound really weird. What I'm doing wrong? Seems like a problem between the 2 Frameworks.

NLayer:https://nlayer.codeplex.com/ (https://nlayer.codeplex.com/)

Thanks for your help ;).
Sven
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on June 10, 2014, 08:46:02 am
No one any ideas? ;)
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: zsbzsb on June 10, 2014, 02:20:17 pm
Without ever using the library in question... You appear to be using it improperly. You assume as long as the MpegFile.ReadSamples returns > 0 that your entire buffer (512 bytes) will be filled. However from taking a glance at the source code (https://nlayer.codeplex.com/SourceControl/latest#NLayer/MpegFile.cs) the function actually returns the length of what was filled in your buffer.

Also, I do not believe simply converting the array as you are doing is the proper way to convert audio data from a byte array to a short array (PCM) that SFML requires.
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on June 11, 2014, 10:24:22 am
Thanks for your help, I changed the Method this way:

protected override bool OnGetData(out short[] samples)
        {
            if (currentBufferSize <= mp3file.Position)
            {
                byte[] buffer = new byte[512];
                int readSamples = mp3file.ReadSamples(buffer, 0, buffer.Length);
                if (readSamples > 0)
                {
                    Array.Resize(ref currentBuffer, currentBuffer.Length + (readSamples / 2));
                    Buffer.BlockCopy(buffer, 0, currentBuffer, currentBufferSize, readSamples);
                    currentBufferSize = currentBuffer.Length;
                }
                samples = currentBuffer;
                return true;
            }
            else
            {
                samples = currentBuffer;
                return false;
            }
        }

Also, I do not believe simply converting the array as you are doing is the proper way to convert audio data from a byte array to a short array (PCM) that SFML requires.

I have read in the internet, that this would be the safest way of converting byte[] to short[]. How should it be done?
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: Laurent on June 11, 2014, 10:36:55 am
This is not just a type conversion. You first have to find out what sample format NLayer is giving you. It's a byte array, but what does it really contain? Apparently the returned array contains 32-bits float samples. So the conversion would be a little more complicated.
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on June 11, 2014, 11:08:30 am
Ok, thanks again for your information. I'm not familiar with all this audio stuff, so I'm asking a bit dumb, you might think.
You said, this is a 32-bits-float sample, so I would say 4 elements of the read byte array represent 1 sample, correct? This 1 sample would take 2 elements in the short array passed to sfml,correct?
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on June 11, 2014, 02:07:57 pm
I tried now this (from http://www.codeproject.com/Articles/501521/How-to-convert-between-most-audio-formats-in-NET (http://www.codeproject.com/Articles/501521/How-to-convert-between-most-audio-formats-in-NET)), but this also doesn't work:

        protected override bool OnGetData(out short[] samples)
        {
            if (currentBufferSize <= mp3file.Position)
            {
                byte[] buffer = new byte[512];
                int readSamples = mp3file.ReadSamples(buffer, 0, buffer.Length);
                if (readSamples > 0)
                {
                    Array.Resize(ref currentBuffer, currentBuffer.Length + buffer.Length);
                    //Buffer.BlockCopy(buffer, 0, currentBuffer, currentBufferSize, readSamples);
                    int destOffset = currentBufferSize;
                    for (int sample = 0; sample < buffer.Length; sample++)
                    {
                        // adjust volume
                        float sample32 = buffer[sample];
                        // clip
                        if (sample32 > 1.0f)
                            sample32 = 1.0f;
                        if (sample32 < -1.0f)
                            sample32 = -1.0f;
                        currentBuffer[destOffset++] = (short)(sample32 * 32767);
                    }
                    currentBufferSize = currentBuffer.Length;
                }
                samples = currentBuffer;
                return true;
            }
            else
            {
                samples = currentBuffer;
                return false;
            }
        }
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: zsbzsb on June 11, 2014, 02:29:37 pm
If the returned byte array really is floats in the form of bytes you can use BitConverter.ToSingle(...) (http://msdn.microsoft.com/en-us/library/system.bitconverter.tosingle%28v=vs.110%29.aspx) to convert 4 bytes to a float.
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on June 11, 2014, 02:46:48 pm
Working with this code, I get only some "clicking" sound:

protected override bool OnGetData(out short[] samples)
        {
            if (currentBufferSize <= mp3file.Position)
            {
                byte[] buffer = new byte[512];
                int readSamples = mp3file.ReadSamples(buffer, 0, buffer.Length);
                if (readSamples > 0)
                {
                    Array.Resize(ref currentBuffer, currentBuffer.Length + buffer.Length);
                    //Buffer.BlockCopy(buffer, 0, currentBuffer, currentBufferSize, readSamples);
                    for (int i = 0; i < (buffer.Length / 4); i++)
                    {
                        currentBuffer[currentBufferSize + (i/4)] = (short)(BitConverter.ToSingle(buffer, i * 4) * 32767);
                    }
                }
                samples = currentBuffer;
                return true;
            }
            else
            {
                samples = currentBuffer;
                return false;
            }
        }

How can I check, what soundformat is read by nlayer?
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: Laurent on June 11, 2014, 04:49:03 pm
Quote
How can I check, what soundformat is read by nlayer?
Yeah, before trying to integrate to SFML, make sure you read the samples correctly with NLayer. Which means studying its documentation, and asking on their forum or whatever they have, if the documentation is not enough (I couldn't find one on their website).
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on July 03, 2014, 07:36:42 pm
Sorry for answering so late, but I had to get the information from NLayer developer. Here is the answer:

Quote
It returns 32-bit little-endian floats. You can copy it to a float[] buffer (use Buffer.BlockCopy), then do your processing from there.

The samples are started in channel-interleaved format (just like WAV files).

Do you have a hint for me, how to do the work from float buffer to short buffer? Thanks in advance ;).
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: zsbzsb on July 03, 2014, 07:45:07 pm
Ok, so it returns a float array in the form of a byte array (why it does I have no clue). Now you need to know what the range of the floats are. Is it the normal range of wav files [-32768 ... +32768] or normalized [-1.0 ... +1.0] or maybe something else?
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on July 05, 2014, 12:55:58 pm
Thank you again, here the answer :):

Quote
It's nominally [-1.0 ... +1.0], but you should expect some out of range values with "loud" sounds. Deal with them like you would any other out-of-range samples (usually clipping, but an audio compressor will also do the trick if set up correctly).

Do we need more information? :)
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: zsbzsb on July 05, 2014, 04:04:35 pm
Something like this should work.

var normalizedaudiodata = new float[] { 1.0f, -1.0f, 0.5f, -0.5f, 2.0f, -2.0f }; // normalized data from decoder
var pcmaudiodata = new short[normalizedaudiodata.Length]; // converted data
for (int i = 0; i < normalizedaudiodata.Length; i++)
{
    // clip the data
    if (normalizedaudiodata[i] > 1.0f) normalizedaudiodata[i] = 1.0f;
    else if (normalizedaudiodata[i] < -1.0f) normalizedaudiodata[i] = -1.0f;

    // convert to pcm data
    pcmaudiodata[i] = (short)(normalizedaudiodata[i] * short.MaxValue);
}
// do whatever you want with the converted data
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on July 07, 2014, 08:42:06 am
Thank you for your great help. I use this code, but now I hear nothing (turned the volume up to 100 ;)).:

protected override bool OnGetData(out short[] samples)
        {
            if (currentBufferSize <= mp3file.Position)
            {
                byte[] buffer = new byte[512];
                int readSamples = mp3file.ReadSamples(buffer, 0, buffer.Length);
                if (readSamples > 0)
                {
                    float[] normalizedaudiodata = new float[readSamples / 4];// { 1.0f, -1.0f, 0.5f, -0.5f, 2.0f, -2.0f }; // normalized data from decoder
                    Buffer.BlockCopy(buffer, 0, normalizedaudiodata, 0, buffer.Length);
                    short[] pcmaudiodata = new short[normalizedaudiodata.Length]; // converted data
                    for (int i = 0; i < normalizedaudiodata.Length; i++)
                    {
                        // clip the data
                        if (normalizedaudiodata [i] > 1.0f)
                        {
                            normalizedaudiodata [i] = 1.0f;
                        }
                        else
                        {
                            if (normalizedaudiodata [i] < -1.0f)
                            {
                                normalizedaudiodata [i] = -1.0f;
                            }
                        }
                        // convert to pcm data
                        pcmaudiodata[i] = (short)(normalizedaudiodata[i] * short.MaxValue);
                    }
                    // do whatever you want with the converted data
                    int startIndex = currentBuffer.Length;
                    Array.Resize(ref currentBuffer, currentBuffer.Length + pcmaudiodata.Length);
                    for (int i = startIndex; i < currentBuffer.Length; i++)
                    {
                        currentBuffer [i] = pcmaudiodata [i - startIndex];
                    }

                    //Array.Resize(ref currentBuffer, currentBuffer.Length + buffer.Length);
                    //Buffer.BlockCopy(buffer, 0, currentBuffer, currentBufferSize, readSamples);
//                    for (int i = 0; i < (buffer.Length / 4); i++)
//                    {
//                        currentBuffer[currentBufferSize + (i/4)] = (short)(BitConverter.ToSingle(buffer, i * 4) * 32767);
//                    }
                }
                samples = currentBuffer;
                return true;
            }
            else
            {
                samples = currentBuffer;
                return false;
            }
        }


What I am doing wrong? How can I find an error?
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: zsbzsb on July 07, 2014, 01:30:36 pm
Why are you reading a byte array when the library provides an overload that uses floats?
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: Laurent on July 07, 2014, 01:47:56 pm
Quote
if (currentBufferSize <= mp3file.Position)
What's the purpose of this test? I don't think you need it at all.

Quote
Array.Resize(ref currentBuffer, currentBuffer.Length + pcmaudiodata.Length);
OnGetData only expects a new chunk of fresh data. You mustn't return all the samples read in the previous calls, those have already been processed.
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on July 07, 2014, 04:02:07 pm
Hello, thanks for your help. I now optimized the code to the following:

protected override bool OnGetData(out short[] samples)
        {
            float[] normalizedaudiodata = new float[128];
            int readSamples = mp3file.ReadSamples(normalizedaudiodata, 0, normalizedaudiodata.Length);
            short[] pcmaudiodata;
            if (readSamples > 0)
            {
                pcmaudiodata = new short[normalizedaudiodata.Length]; // converted data
                for (int i = 0; i < normalizedaudiodata.Length; i++)
                {
                    // clip the data
                    if (normalizedaudiodata [i] > 1.0f)
                    {
                        normalizedaudiodata [i] = 1.0f;
                    }
                    else
                    {
                        if (normalizedaudiodata [i] < -1.0f)
                        {
                            normalizedaudiodata [i] = -1.0f;
                        }
                    }
                    // convert to pcm data
                    pcmaudiodata[i] = (short)(normalizedaudiodata[i] * short.MaxValue);
                }
                samples = pcmaudiodata;
                return true;
            }
            else
            {
                samples = null;
                return false;
            }
        }

But this also just doesn't work, I hear nothing :(. What can I do now?
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: zsbzsb on July 07, 2014, 04:19:10 pm
new short[normalizedaudiodata.Length]

You need to use the actual number of samples that were read, not just the maximum length of your buffer.
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on July 08, 2014, 08:11:38 am
new short[normalizedaudiodata.Length]

You need to use the actual number of samples that were read, not just the maximum length of your buffer.

Thanks for your help, I modified the code this way:

protected override bool OnGetData(out short[] samples)
        {
            float[] normalizedaudiodata = new float[128];
            int readSamples = mp3file.ReadSamples(normalizedaudiodata, 0, normalizedaudiodata.Length);
            short[] pcmaudiodata;
            if (readSamples > 0)
            {
                pcmaudiodata = new short[readSamples]; // converted data
                for (int i = 0; i < readSamples; i++)
                {
                    // clip the data
                    if (normalizedaudiodata[i] > 1.0f)
                    {
                        normalizedaudiodata[i] = 1.0f;
                    }
                    else
                    {
                        if (normalizedaudiodata[i] < -1.0f)
                        {
                            normalizedaudiodata[i] = -1.0f;
                        }
                    }
                    // convert to pcm data
                    pcmaudiodata[i] = (short)(normalizedaudiodata[i] * short.MaxValue);
                }
                samples = pcmaudiodata;
                return true;
            }
            else
            {
                samples = null;
                return false;
            }
        }

I'm hearing nothing, Windows Sound sees my application wants to start audio output (I'm using windows 7) but there is no peak and nothing to hear. What I'm doing wrong?
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: Laurent on July 08, 2014, 08:25:05 am
Can you show the other functions of your class?
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on July 08, 2014, 09:09:45 am
Can you show the other functions of your class?
No Problem, here is the whole class ;):

public class Mp3StreamSFML : SoundStream
    {
        private MpegFile mp3file;
        private int currentBufferSize;
        private short[] currentBuffer;

        public Mp3StreamSFML(String _filename)
        {
            mp3file = new MpegFile(_filename);
            Initialize((uint)mp3file.Channels, (uint)mp3file.SampleRate);
            currentBufferSize = 0;
            currentBuffer = new short[currentBufferSize];
        }

        #region implemented abstract members of SoundStream

        protected override bool OnGetData(out short[] samples)
        {
            float[] normalizedaudiodata = new float[128];
            int readSamples = mp3file.ReadSamples(normalizedaudiodata, 0, normalizedaudiodata.Length);
            short[] pcmaudiodata;
            if (readSamples > 0)
            {
                pcmaudiodata = new short[readSamples]; // converted data
                for (int i = 0; i < readSamples; i++)
                {
                    // clip the data
                    if (normalizedaudiodata[i] > 1.0f)
                    {
                        normalizedaudiodata[i] = 1.0f;
                    }
                    else
                    {
                        if (normalizedaudiodata[i] < -1.0f)
                        {
                            normalizedaudiodata[i] = -1.0f;
                        }
                    }
                    // convert to pcm data
                    pcmaudiodata[i] = (short)(normalizedaudiodata[i] * short.MaxValue);
                }
                samples = pcmaudiodata;
                return true;
            }
            else
            {
                samples = null;
                return false;
            }
        }

        protected override void OnSeek(TimeSpan timeOffset)
        {
            //TODO
            //mp3file.Position = 0;
            //mp3file.Position = (long)timeOffset.TotalSeconds;
        }

        #endregion
    }
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: Laurent on July 08, 2014, 09:31:55 am
Ok, you haven't forgotten to call Initialize ;)

You should check (with the debugger or the console) the important values involved, to make sure everything is ok:
- mp3file.Channels
- mp3file.SampleRate
- readSamples everytime OnGetData is called
- if possible, a complete chunk of samples (in the middle of the music, to avoid silences that may occur at the beginning and end) to make sure the values are not weird

By the way, what is the "0" argument of ReadSamples? And shouldn't the array be passed with the "out" or "ref" keyword, if it is modified inside the function?
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: zsbzsb on July 08, 2014, 01:17:55 pm
Try writing to the console the values of each sample before and after conversion and let us see the output.

Quote
what is the "0" argument of ReadSamples?

Probably the offset at which to write the samples into the buffer.
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on July 09, 2014, 07:43:12 am
I tried a bit around, and you know what works? Setting the buffer size to 2048 (just a bit trying):

protected override bool OnGetData(out short[] samples)
        {
            float[] normalizedaudiodata = new float[2048];
            int readSamples = mp3file.ReadSamples(normalizedaudiodata, 0, normalizedaudiodata.Length);
            short[] pcmaudiodata;
            if (readSamples > 0)
            {
                pcmaudiodata = new short[readSamples]; // converted data
                for (int i = 0; i < readSamples; i++)
                {
                    // clip the data
                    if (normalizedaudiodata[i] > 1.0f)
                    {
                        normalizedaudiodata[i] = 1.0f;
                    }
                    else
                    {
                        if (normalizedaudiodata[i] < -1.0f)
                        {
                            normalizedaudiodata[i] = -1.0f;
                        }
                    }
                    // convert to pcm data
                    pcmaudiodata[i] = (short)(normalizedaudiodata[i] * short.MaxValue);
                }
                samples = pcmaudiodata;
                return true;
            }
            else
            {
                samples = null;
                return false;
            }
        }

But I remember having read, that a too large buffer could decrease performance and therefore should be as little as possible. Do you have an idea, why it works with large buffer?
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: Laurent on July 09, 2014, 07:52:13 am
In fact a small buffer is unlikely to work. I should have noticed before, 128 is indeed too small. At 44100 Hz, it's only 3 milliseconds of sound, which may not be enough to cover the time needed to decode another chunk in parallel. In other words, the sound stream spends all its time decoding small chunks. You're supposed to feed it with large chunks so that the audio driver is busy enough while you decode the next one (the Music class uses 1 second buffers).
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: s.baus on July 09, 2014, 09:11:03 am
No Problem, you helped me really much ;). How can calculate the sound buffer size to have 1 second of data?
Title: Re: NLayer MpegFile to SFML.Net SoundStream
Post by: Nexus on July 09, 2014, 09:25:04 am
How can calculate the sound buffer size to have 1 second of data?
Wolfram Alpha (http://www.wolframalpha.com/input/?i=1s+*+44.1kHz)

But in this case, it's trivial: there are 44100 samples per second.