Welcome, Guest. Please login or register. Did you miss your activation email?

Author Topic: Help Beat Detection  (Read 8330 times)

0 Members and 1 Guest are viewing this topic.

Foaly

  • Sr. Member
  • ****
  • Posts: 453
    • View Profile
Help Beat Detection
« on: April 27, 2012, 09:57:12 am »
Hello everybody!
As I already said in this post http://en.sfml-dev.org/forums/index.php?topic=7642.0 I wanted to make a simple Beat Detection using SFML. I used the algorithm I found here: http://www.flipcode.com/misc/BeatDetectionAlgorithms.pdf (to be precise I implemented the third version of the first algorithm, because I want to use it for techno and electronic music later anyway).
In the thread above you can see that it didn't quiet work with sf::Music, because I couldn't acces the Samples in time, so I tried using sf::SoundRecorder, because I wanted to use a direct input later anyway. I could get it to work and using a custom SoundStream I could even get it play the samples afterwards. It seems to work fairly good if I put a mic in front of a speaker. It doesn't work too too well when i plug a cable from an ipod into the mic input directly.

So now here is the thing. I am new to the whole concept of threads and mutexs and such, but I would like to use this code in a later project. I know there are a lot of very talented C++ programmers here, so would anybody mind to look at my code and tell me what they think about it and what to improve. I'm not quiet sure if i use the mutexes correctly or if the way I pass the buffers around is the most efficient one or if there is some other way of optimizing my code... Please have a look and tell me what you think.
Also does anyone have an idea why plugging in a cable into the mic input directly makes the code constantly detect a beat?

Thanks in advance,
Foaly :)



Here is my code:
#include <SFML/System.hpp>
#include <SFML/Graphics.hpp>
#include <SFML/Audio.hpp>
#include <iostream>
#include <vector>

sf::Mutex GlobalMutex;

class CustomRecorder : public sf::SoundRecorder
{
    std::vector<sf::Int16> m_Samples;
    std::size_t m_ChunckSize;

    virtual bool onStart() // optional
    {
        // Initialize whatever has to be done before the capture starts
        m_ChunckSize = 1024;

        // Return true to start playing
        return true;
    }

    virtual bool onProcessSamples(const sf::Int16* samples, std::size_t sampleCount)
    {
//        std::cout << "Recording: " << sampleCount << "Samples" << std::endl;

        sf::Lock Lock(GlobalMutex);
        std::copy(samples, samples + sampleCount, std::back_inserter(m_Samples));

        // Return true to continue playing
        return true;
    }

    virtual void onStop() // optional
    {
        // Clean up whatever has to be done after the capture ends
    }

public:

    // Fill the given buffer with 1024 samples  (if possible)
    bool FillBuffer(std::vector<sf::Int16>& VectorToFill)
    {
        if(m_Samples.size() >= m_ChunckSize)
        {
            sf::Lock Lock(GlobalMutex);
            VectorToFill.insert(VectorToFill.begin(), m_Samples.begin(), m_Samples.begin() + m_ChunckSize + 1);
            m_Samples.erase(m_Samples.begin(), m_Samples.begin() + m_ChunckSize + 1);
            return true;
        }
        return false;
    }

};

class CustomPlayer : public sf::SoundStream
{
private:

    sf::Mutex m_mutex;
    std::vector<sf::Int16> m_Samples;
    bool b_FirstTime;

    virtual bool onGetData(sf::SoundStream::Chunk& Data)
    {
        sf::Lock Lock(m_mutex);

        // Check if there is enough data to stream
        if (m_Samples.size() < 1024)
        {
            // Returning false means that we want to stop playing the stream
            return false;
        }

        if(b_FirstTime)
            b_FirstTime = false;
        else
            m_Samples.erase(m_Samples.begin(), m_Samples.begin() + 1024 + 1);

        // Fill the stream chunk with a pointer to the audio data and the number of samples to stream
        Data.samples   = &m_Samples[0];
        Data.sampleCount = 1024;

        return true;
    }

    virtual void onSeek(sf::Time timeOffset)
     {
         // Change the current position in the stream source
     }


public:

    CustomPlayer()
    {
        b_FirstTime = true;
        initialize(1, 44100);
    }

    void FillBuffer(std::vector<sf::Int16>& SourceVector)
    {
        sf::Lock Lock(m_mutex);
        m_Samples.insert(m_Samples.end(), SourceVector.begin(), SourceVector.end() + 1);
    };

};


int main()
{
    // Check that the device can capture audio
    if (CustomRecorder::isAvailable() == false)
    {
        std::cout << "Sorry, audio capture is not supported by your system" << std::endl;
        return EXIT_SUCCESS;
    }

    srand(time(NULL));

    sf::RenderWindow window(sf::VideoMode(500, 500), "SFML works!");
    sf::RectangleShape Block;
    Block.setSize(sf::Vector2f(20, 200));
    Block.setOrigin(10, 100);
    Block.setPosition(250, 250);
    Block.setFillColor(sf::Color::Green);
    Block.setScale(1, 2);

    sf::Clock Clock;
    float FrameTime;

    CustomRecorder Recorder;
    CustomPlayer Player;

    std::vector<sf::Int16> Buffer;

    // Create the energy history buffer and set it to zero
    std::vector<float> EnergyHistoryBuffer;
    EnergyHistoryBuffer.resize(43, 0.f);

    Recorder.start();

    while (window.isOpen())
    {
        sf::Event event;
        while (window.pollEvent(event))
        {
            // Close Event
            if (event.type == sf::Event::Closed)
                window.close();

            // Escape key pressed
            if ((event.type == sf::Event::KeyPressed) && (event.key.code == sf::Keyboard::Escape))
                window.close();

        }

        FrameTime = Clock.getElapsedTime().asSeconds();
//        std::cout << "Framerate: " << 1.f / FrameTime << std::endl;
        Clock.restart();


        if(Recorder.FillBuffer(Buffer))
        {
            GlobalMutex.lock();

            // Compute instant energy
            float InstantEnergie = 0;
            for (std::vector<sf::Int16>::iterator itor = Buffer.begin(); itor < Buffer.end(); itor++)
            {
                InstantEnergie += ((*itor) / 32767.f) * ((*itor) / 32767.f);
            }

            // Compute average local energy
            float AverageLocalEnergySum = 0;
            for (std::vector<float>::iterator itor = EnergyHistoryBuffer.begin(); itor < EnergyHistoryBuffer.end(); itor++)
            {
                AverageLocalEnergySum += (*itor);
            }
            float AverageLocalEnergy = (1/43.f) * AverageLocalEnergySum;

            // Calculate the Variance of the last samples
            float VarianceSum = 0;
            for (std::vector<float>::iterator itor = EnergyHistoryBuffer.begin(); itor < EnergyHistoryBuffer.end(); itor++)
            {
                VarianceSum += ((*itor) - AverageLocalEnergy) * ((*itor) - AverageLocalEnergy);
            }
            float Variance = (1/43.f) * VarianceSum;

            // Calculate the Sesibility Constant
            float SensibilityConstant = (-0.0025714 * Variance) + 1.5142857;

            // Insert the Instant energy at the front of the history buffer and remove the last entry
            EnergyHistoryBuffer.insert(EnergyHistoryBuffer.begin(), InstantEnergie);
            EnergyHistoryBuffer.pop_back();

//            std::cout << "Instant Energy: " << InstantEnergie << " Local Energy: " << SensibilityConstant * AverageLocalEnergy << std::endl;
            // see if we have a beat
            if(InstantEnergie > SensibilityConstant * AverageLocalEnergy)
            {
//                std::cout << "We have a BEAT!"  << std::endl;
                Block.setScale(1, 2);//(rand()) / RAND_MAX * (0.4) + 1.8 );
            }

            Player.FillBuffer(Buffer);
            Player.play();
            Buffer.clear();

            GlobalMutex.unlock();
        }

//        Block.rotate(50 * FrameTime);
        if(Block.getScale().y > 0.8)
            Block.setScale(1, Block.getScale().y - 0.9 * FrameTime);

        window.clear();
        window.draw(Block);
        window.display();
    }

    Recorder.stop();

    return 0;
}

edit: maybe a little suggestion for the forum: what about a "spoiler" function or something like that? So that you only see the code when you click on it :)

eXpl0it3r

  • SFML Team
  • Hero Member
  • *****
  • Posts: 11045
    • View Profile
    • development blog
    • Email
Re: Help Beat Detection
« Reply #1 on: April 27, 2012, 01:20:59 pm »
Not sure if anyone really takes hi freetime to look though all this in detail...

My question is why you use mutexes but only 1 thread? ???

I didn't look at it all and it would take me even more time to really understand what you're trying to do but as I quickly looked over it I noticed GlobalMutex.lock() and GlobalMutex.unlock(). Which is not a good design, because it doesn't guarantee that unlock() will be always called, no matter what and you could end up with a deadlock. Since you write that class on your own, you can easily implement RAII.
Official FAQ: https://www.sfml-dev.org/faq.php
Official Discord Server: https://discord.gg/nr4X7Fh
——————————————————————
Dev Blog: https://duerrenberger.dev/blog/

Laurent

  • Administrator
  • Hero Member
  • *****
  • Posts: 32498
    • View Profile
    • SFML's website
    • Email
Re: Help Beat Detection
« Reply #2 on: April 27, 2012, 01:27:05 pm »
Quote
My question is why you use mutexes but only 1 thread?
Sound recorders and streams are threaded internally.
Laurent Gomila - SFML developer

eXpl0it3r

  • SFML Team
  • Hero Member
  • *****
  • Posts: 11045
    • View Profile
    • development blog
    • Email
Re: Help Beat Detection
« Reply #3 on: April 27, 2012, 11:54:50 pm »
Sound recorders and streams are threaded internally.

Ah okay! Good to know.
I've never done anything with sounds in SFML. I just quickly scanned over the source code and was kind of confused to see a mutex and no threads. :D
Official FAQ: https://www.sfml-dev.org/faq.php
Official Discord Server: https://discord.gg/nr4X7Fh
——————————————————————
Dev Blog: https://duerrenberger.dev/blog/

Foaly

  • Sr. Member
  • ****
  • Posts: 453
    • View Profile
Re: Help Beat Detection
« Reply #4 on: April 28, 2012, 09:38:46 am »
Yeah exactly. As Laurent said the soundstream and the soundrecorder are using threads internally, so I actually have 3 threads.
Thanks for pointing out the globalMutex.lock() I'll replace that with a sf::Lock. I don't remember why I didn't do that in the first place. Are all the other mutex ok though? I havent worked with mutexes before and I don't want my memory to be corrupted. :)

Nexus

  • SFML Team
  • Hero Member
  • *****
  • Posts: 6287
  • Thor Developer
    • View Profile
    • Bromeon
Re: Help Beat Detection
« Reply #5 on: May 04, 2012, 05:55:47 pm »
Some code parts that took my attention:
  • Instead of erase() on m_Samples, you could just use an index offset to know where the remaining sequence starts. Be aware that std::vector::erase() needs to copy all elements after the erased ones to fill the gap, which can be very inefficient.
  • Once you lock a mutex before calling m_Samples.size() and once after it. If m_Samples can be changed by another thread, make sure the mutex also protects it.
  • You should use a constant for 1024, like const std::size_t BlockSize = 1024;
  • The +1 offsets are incorrect. Note that STL iterator ranges are half-open intervals, that is begin refers to the first element and end to one-past the last element (it may even be not dereferencable).
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development:

Foaly

  • Sr. Member
  • ****
  • Posts: 453
    • View Profile
Re: Help Beat Detection
« Reply #6 on: May 07, 2012, 06:28:56 pm »
Thank you very much for your reply :D
I started implementing your suggestions, but that brought up a couple of questions. I'll just go point by point.
  • I never realized that erase() can be such a performance problem, makes totally sense though now that you say it. The problem is I kinda need the erase()... My Code works like this: FillBuffer() is called with a Buffervector as Argument. If there are more than 1024 Samples available, copy those 1024 samples into the given Buffervector and delete them from the Samplesvector. So an index offset doesn't really solve this problem, because then I end up with a bunch of memory that i don't need anymore, but which is not freed. Is there any other way to solve this problem? Like maybe split the Samplesvector into two vectors (at position 1024) or something like that?
  • As I said mutexes are new to me... Makes sense though. I changed that.
  • Changed that too.
  • Are you really sure the indexes are incorrect? In the doc for insert and erase it says, that (if you pass three iterators) the position specified by the last iterator is NOT included. So if I want to delete 1024 elements I would have to pass vector.begin() + 1024 + 1 , don't I?
Thanks again, Foaly :)

Silvah

  • Guest
Re: Help Beat Detection
« Reply #7 on: May 08, 2012, 07:28:35 pm »
Are you really sure the indexes are incorrect? In the doc for insert and erase it says, that (if you pass three iterators) the position specified by the last iterator is NOT included. So if I want to delete 1024 elements I would have to pass vector.begin() + 1024 + 1 , don't I?
If you want to delete 1024 elements, the first one is vector.begin() + 0; vector.begin() + 1024 is the 1025th one, which is the first one you don't want to delete. So no, don't add 1.

Foaly

  • Sr. Member
  • ****
  • Posts: 453
    • View Profile
Re: Help Beat Detection
« Reply #8 on: May 11, 2012, 06:11:40 pm »
Oh ok! Perfect thank you very much! I now see that it was a rather stupid mistake...
About the problem with the erasing. I don't really see a way to solve this when using vectors. So maybe would it be a good idea to use a std::list? Because from what I read in the docs the only drawback compared to vectors is that I can't access an element directly. Since I only iterate over the elements anyway (which should be just as fast) that shouldn't be a problem. The advantage is that I can erase elements from any position, without having to copy everything around. So wouldn't a std::list be better? Or is there anything I'm missing?

Nexus

  • SFML Team
  • Hero Member
  • *****
  • Posts: 6287
  • Thor Developer
    • View Profile
    • Bromeon
Re: Help Beat Detection
« Reply #9 on: May 12, 2012, 11:26:27 am »
Not std::list, it has a per-element overhead which is far too big for elements of 2 bytes size. Consider std::deque instead.
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development:

Foaly

  • Sr. Member
  • ****
  • Posts: 453
    • View Profile
Re: Help Beat Detection
« Reply #10 on: May 14, 2012, 04:58:59 pm »
So would you recommend a std::deque over a std::vector in this case?
I'm asking because I haven't worked with std::deque before and in the documentation it says, that they are not safe to use with pointer arithmetics and the Fillbuffer function uses a reference to the vector that is supposed to be filled. Would that still work?

Nexus

  • SFML Team
  • Hero Member
  • *****
  • Posts: 6287
  • Thor Developer
    • View Profile
    • Bromeon
Re: Help Beat Detection
« Reply #11 on: May 16, 2012, 06:19:46 pm »
In a std::vector, you have the equivalence
&container[index] == &container[0] + index

This doesn't apply for the other containers. I assume this is meant with "unsafe pointer arithmetics", although this is an unfortunate description.
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development:

Foaly

  • Sr. Member
  • ****
  • Posts: 453
    • View Profile
Re: Help Beat Detection
« Reply #12 on: May 20, 2012, 11:12:15 pm »
ah ok i see! But since I use the insert method and iterators, that shouldn't be a problem, right?

edit: I guess it would be a good idea to use a std::deque for the EnergyHistoryBuffer too, because I add an element to the front and remove one from the back on a regular basis, right? (right now I use a vector and that means that all the elements are copied on to the right when I add an element to the front, which is not really efficient, am I corrent?!)
« Last Edit: May 21, 2012, 12:02:51 am by Foaly »

Celtic Minstrel

  • Jr. Member
  • **
  • Posts: 80
    • View Profile
Re: Help Beat Detection
« Reply #13 on: June 26, 2012, 04:04:12 am »
std::vector is guaranteed to be stored in contiguous memory (ie, just like an array), so it's not great when you're doing a lot of insertion but is one of the best if you need random access on a group of objects that doesn't change a lot.

std::deque still uses array-style allocation, but it may split the array up into chunks in completely different areas of memory. If you need random access and fast insertion/deletion at the front and back, it's a good choice. If you need insertion in the middle, I think it's not as good, though I'm unsure on that one.

std::list puts each element in its own location in memory with a pointer to the next and previous elements, so they could be widely spaced out in memory. This means you can't have random access, but you have fast insertion and deletion anywhere in the list.

In other words, yes, deque is a good choice for what you're doing. A list would work too; I do see what Nexus means by the space overhead though, since it stores two pointers along with each element, which means the pointers take up well over half the space used. I doubt this would be a problem in most cases, but apart from that either deque or list would be equally suitable for what you're doing. So you might as well use the deque.

Bonus of using deque is that you can push_front to insert at the front, instead of using insert. ;)