How to process audio recording and manage graphic in (almost) real time ?

[fluck]

Hello !

I am building a music visualizer. I already made the analyser, but I load the music using a buffer with the function LoadFromFile. Here is what it looks like :


Now I would like to use the internal input of my sound card for the sound input, to be able to display this animation live during parties and live dj set.

So I begun looking at the Audio Recording part of SFML, and made this little code to test it :

Code: [Select]

#include <SFML/Audio.hpp>
#include <SFML/Graphics.hpp>

using namespace  std;



void drawSamples(const sf::Int16* samples, sf::Uint64 samplecount, sf::RenderWindow &window){

    sf::VertexArray Lines;
    Lines.setPrimitiveType(sf::LineStrip);
    for(int i = 0;i < samplecount;i++){
        Lines.append(sf::Vertex(sf::Vector2f(i,samples[i]/100+window.getSize().y/2),sf::Color::White));
    }

    window.draw(Lines);

}

int main()
{

    sf::RenderWindow window(sf::VideoMode(1440,900,32),"TEST RECORDER");
    sf::SoundBufferRecorder recorder;
    sf::SoundBuffer buffer;
    sf::Event event;


    //Make sure there is something in the buffer
    recorder.start();
    sf::sleep(sf::milliseconds(1));
    recorder.stop();



    while(window.isOpen()){

        while(window.pollEvent(event)){
            switch (event.type)
            {
            case sf::Event::EventType::Closed:
                window.close();
                break;
            }

        }
        recorder.stop();
        buffer = recorder.getBuffer();
        window.clear(sf::Color::Black);
        const sf::Int16* samples = buffer.getSamples();
        sf::Uint64 samplecount = buffer.getSampleCount();
        drawSamples(samples, samplecount, window);
        window.display();
        recorder.start();
    }
    return 0;
}




When I run it, I find it really slow. I would like to hit almost 60fps to be able to run it smoothly on a video projector.
Where do you think this slowness comes from ? Is it because the recording is taking a lot of time ? Is it because of the the fact that I copy the buffer every time ?

There is another part in this problem. To achieve a very good looking results like the one on the video I use a bufferSize of  2^13 (8192) integer. I guess if I want to run my program with a live sound source I need to be able to get that data fast enough so that it won’t interfere with the displaying.
What should I use ? I feel like I should use a custom recorder that is fast enough, but I also feel like this should use an audio stream because I want to load some data that cannot be loaded directly in memory (because it doesn’t exist yet…). I would like to know if recording with a SoundBufferRecorder is a good solution to provide data for an audio stream.

I hope my problem appears clear to you, and I say thanks in advance to anyone willing to help me !

Thanks,

Louis

[eXpl0it3r]

If you can use a recording device as an input, then it's probably best to write your own recorder class, where you can access to the samples as they come in: https://www.sfml-dev.org/tutorials/2.5/audio-recording.php#custom-recording

You can use the setProcessingInterval (default 100ms) to get a lower latency, but you also need to keep in mind that the OS/OpenAL has limitations, so just setting it 0ms doesn't mean, that you're application will now have no latency whatsoever.

[fluck]

Ok I got it working !   
I was confused at first because I didn't understand how to retrieve the data from the sound recorder ! I thought i had to stop it every time I wanted to get access to the audio data, but in reality you can manage the data in realtime ! I didn't realise you could just create a buffer to capture the incoming audio data

I wrote my class like this :

Code: [Select]

#include <cstdlib>
#include <iostream>
#include <SFML/Audio.hpp>
#include <vector>

using namespace std;
using namespace sf;

class FastRecorder : public SoundRecorder
{
public:
    bool onStart() override;
    bool onProcessSamples(const Int16 *samples, std::size_t sampleCount) override;
    void onStop() override;
    void setBufferSize(int bufferSize);
    vector<Int16> getBuffer();
    int getBufferSize();

private:
    Int16 globalmax;
    const Int16*  msamples;
    Uint64 msamplecount;
    vector<Int16> mbuffer;
    int mark;
    int mbufferSize;
};


Code: [Select]

#include "fastrecorder.h"

bool FastRecorder::onStart()
{
    globalmax = 0;
    mark = 0;
    mbufferSize = 512;
    mbuffer.resize(mbufferSize);
    setProcessingInterval(milliseconds(20));

    return true;
}

bool FastRecorder::onProcessSamples(const Int16 *samples, std::size_t sampleCount)
{
        // Do something with the new chunk of samples (store them, send them, ...)
        msamples = samples;
        msamplecount = sampleCount;

        //If the whole samples array can fit in the vector just fill it
        //If not, then fill it as much as we can then go back to the beginning of the vector.
        for(int i = 0; i<msamplecount;i++){
            mbuffer[(i+mark)%mbufferSize] = samples[i];
        }

        mark = (msamplecount-1+mark) % mbufferSize;




        // Return true to continue playing
        return true;
}

void FastRecorder::onStop()
{}

void FastRecorder::setBufferSize(int bufferSize){

    mbufferSize = bufferSize;
    mbuffer.resize(mbufferSize);

}


vector<Int16> FastRecorder::getBuffer()
{
    return mbuffer;
}

int FastRecorder::getBufferSize()
{
    return mbufferSize;
}

This is just a quick test and not the final result, but it works enough for a first try. I used a vector<sf::Int16> to store all the samples because my old function needed a vector. I fill this vector (named buffer in the code) in a circular manner, and then I give it to my fft function. Works like a charm.

Thanks for your advice eXpl0it3r ! 

Louis