Music getPlayingOffset() more accurate?

[Lauthai]

I am working on a rhythm style game and am using sf::Music to load and play my song files.  However, when I try to use the getPlayingOffset().asMilliseconds() function, it only is producing updates to the number every 20 milliseconds. For example:

// Create and open a song file
sf::Music song;
song.openFromFile(gameState.getSongPlaying().getAudioFilePath());

song.play();

// Test by outputting the offset.
while(1) {
    cout << song.getPlayingOffset().asMilliseconds() << endl;
}
 

I would expect that it would output something continouous like 1, 2, 3, 4, ... but instead get 0, 0, ..., 0, 20, 20 and repeat by 20s.

Is there a way to make it so that the offset is more accurately outputting to the millisecond instead of 20? Would that involve modifying a compiling a custom copy of the audio.dll?  Or would you have a better suggestion to get the current millisecond since the start of the song (i.e. would output 1000 if the song has been going for 1 second)?

EDIT: Would it be better to load the song into a sound buffer and then play using sf::Sound instead? Would the reduce the update delay?

[eXpl0it3r]

Probably better to keep control of a buffer, that way you know exactly where in the buffer you are.

[Stepland]

I was looking for something like this for my own project so I searched how other rhythm games were handling this, turns out it's a really common problem yet I did not find much regarding SFML on that matter.

My goals for a solution were :

• Having a linear output, avoiding both the step function shape and jitter
• Being at most a few ms off regardless of playback time
• Not hogging the CPU

Here's a lengthy guide through a few methods I came up with.

The first basic (and wrong) idea all the following methods are build upon is starting an sf::Clock when you call sf::Music::play() and just reading elasped time from said sf::Clock, this has two problems :

• sf::Music::play() is not blocking, since it just instructs another thread that playback has been requested, for this reason and maybe others having to do with the underlying sound lib, there will be an unpredictable delay between the time this function returns and the time playback actually starts
• I'm not sure why (audio hardware using its own clock ?) but sf::Clock will pretty quickly drift away from the time reported by sf::Music::getPlayingOffset() (as much as a few seconds per minute), and the drift factor itself is pretty unpredictable

Method #1 : The Watchdog Thread

• Have a "watchdog" thread monitor changes in the value returned by sf::Music::getPlayingOffset()
• Each time a new value is detected, reset an sf::Clock
• Report offset as : last detected value + time elapsed on the sf::Clock

This works better because even if we are still using an sf::Clock() to keep time elapsed since the last buffer update, sf::Clock does not drift too much from the audio clock on a timespan as short as a buffer.

Here's my code as a subclass of sf::Music :

(click to show/hide)

#include <thread>
#include <atomic>

#include <SFML/Audio.hpp>

struct PreciseMusic : sf::Music {
    PreciseMusic(std::string path);
    ~PreciseMusic();

    sf::Time getPrecisePlayingOffset() const;

    // Position update thread
    void position_updater_main();
    std::thread position_updater;
    std::atomic<bool> stop_position_updater = false;
    std::atomic<bool> should_correct = false;
    std::atomic<sf::Time> last_music_position;
    sf::Clock reference_clock;
};

(click to show/hide)

#include "PreciseMusic.hpp"

PreciseMusic::PreciseMusic(std::string path) {
    if (not this->openFromFile(path)) {
        throw std::invalid_argument("Could not open "+path);
    }
    position_updater = std::thread(&PreciseMusic::position_updater_main, this);
}

PreciseMusic::~PreciseMusic() {
    stop_position_updater.store(true);
    position_updater.join();  
}

sf::Time PreciseMusic::getPrecisePlayingOffset() const {
    if (should_correct.load()) {
        return last_music_position.load()+reference_clock.getElapsedTime();
    } else {
        return this->getPlayingOffset();
    }
}

void PreciseMusic::position_updater_main() {
    sf::Time next_music_position;
    while (not stop_position_updater.load()) {
        if (this->getStatus() == sf::Music::Playing) {
            next_music_position = this->getPlayingOffset();
            if (next_music_position != last_music_position.load()) {
                last_music_position.store(next_music_position);
                reference_clock.restart();
                should_correct.store(true);
            }
        }
        sf::sleep(sf::milliseconds(1));
    }
}

Here's my test code :

(click to show/hide)

#include <iostream>
#include <thread>
#include <chrono>

#include <SFML/Audio.hpp>

#include "PreciseMusic.hpp"

int main(int, char const **) {
    PreciseMusic m{"test.ogg"};
    m.play();
    std::cout << "raw,corrected" << std::endl;
    while (true) {
        auto raw = m.getPlayingOffset();
        auto corrected = m.getPrecisePlayingOffset();
        std::cout << raw.asMicroseconds() << "," << corrected.asMicroseconds() << std::endl;
        std::this_thread::sleep_for(std::chrono::milliseconds(1));
    }
    return 0;
}

And the resulting chart :

(click to show/hide)

You can clearly see how the code waits for the first step to correct the raw offset

The downside of this method is the watchdog thread, my implementation uses a 1ms loop, it's still not unreasonably fast but the impact is noticeable when you look at CPU usage. A minimal executable simply playing back an audio file using sf::Music takes about 1.5% of the CPU on my machine, bare playback with Method #1 takes 5%.

Method #2 : overriding onGetData()

Instead of restarting the clock using a busy loop in the watchdog thread, override the buffer callback, onGetData(). Since it should be called on every buffer request from the underlying sound lib, we should get similar results while reducing CPU load right ?

Here's my code :

(click to show/hide)

#include <thread>
#include <atomic>

#include <SFML/Audio.hpp>

struct PreciseMusic : sf::Music {
    PreciseMusic(std::string path);

    sf::Time getPrecisePlayingOffset() const;

    std::atomic<bool> should_correct = false;
    std::atomic<sf::Time> last_music_position;
    sf::Clock last_position_update;
protected:
    bool onGetData(sf::SoundStream::Chunk& data) override;
};

(click to show/hide)

#include "PreciseMusic.hpp"

PreciseMusic::PreciseMusic(std::string path) {
    if (not this->openFromFile(path)) {
        throw std::invalid_argument("Could not open "+path);
    }
}

sf::Time PreciseMusic::getPrecisePlayingOffset() const {
    if (should_correct.load()) {
        return last_music_position.load()+last_position_update.getElapsedTime();
    } else {
        return this->getPlayingOffset();
    }
}

bool PreciseMusic::onGetData(sf::SoundStream::Chunk& data) {
    bool continue_playback = sf::Music::onGetData(data);
    if (continue_playback) {
        last_music_position.store(getPlayingOffset());
        last_position_update.restart();
        should_correct.store(true);
    }
    return continue_playback;
}

And here are the results :

(click to show/hide)

You can clearly see it's way too early, my understanding is that audio data is queued/buffered way earlier than when it is actually played. We are wayyy off (still only about 10ms early) and we have weird glitches at the beggining, but at least CPU usage is back to normal.

Method #3 : Taking Lag Into Account

Another idea I had was to make a hybrid of the first two methods :

• Measure the lag by measuring the time between :
  
  • the first onGetData() call
  • the first step in the raw offset given by getPlayingOffset()
• Report offset as previously but substract lag

The first step in the raw offset is monitored by a fast busy loop in a watchdog thread as previously but since we only need to check for the first step to measure lag, the thread only runs for about one second at most.

Here's my code :

(click to show/hide)

#include <thread>
#include <atomic>

#include <SFML/Audio.hpp>

struct PreciseMusic : sf::Music {
    PreciseMusic(std::string path);
    ~PreciseMusic();

    std::thread position_update_watchdog;

    void position_update_watchdog_main();
    std::atomic<bool> should_stop_watchdog = false;

    sf::Time getPrecisePlayingOffset() const;

    bool first_onGetData_call = true;
    sf::Clock lag_measurement;
    std::atomic<sf::Time> lag;
    std::atomic<bool> should_correct = false;
    std::atomic<sf::Time> last_music_position;
    sf::Clock last_position_update;
protected:
    bool onGetData(sf::SoundStream::Chunk& data) override;
};

(click to show/hide)

#include "PreciseMusic.hpp"

#include <chrono>

PreciseMusic::PreciseMusic(std::string path) {
    if (not this->openFromFile(path)) {
        throw std::invalid_argument("Could not open "+path);
    }
    position_update_watchdog = std::thread{&PreciseMusic::position_update_watchdog_main, this};
}

void PreciseMusic::position_update_watchdog_main() {
    sf::Time next_music_position = sf::Time::Zero;
    while (not should_stop_watchdog.load()) {
        if (this->getStatus() == sf::Music::Playing) {
            next_music_position = this->getPlayingOffset();
            if (next_music_position != last_music_position.load()) {
                lag.store(lag_measurement.getElapsedTime());
                should_correct.store(true);
                break;
            }
        }
        std::this_thread::sleep_for(std::chrono::milliseconds(1));
    }
};

PreciseMusic::~PreciseMusic() {
    should_stop_watchdog.store(true);
    position_update_watchdog.join();
}

sf::Time PreciseMusic::getPrecisePlayingOffset() const {
    if (should_correct.load()) {
        return last_music_position.load()+last_position_update.getElapsedTime()-lag;
    } else {
        return this->getPlayingOffset();
    }
}

bool PreciseMusic::onGetData(sf::SoundStream::Chunk& data) {
    if (first_onGetData_call) {
        first_onGetData_call = false;
        lag_measurement.restart();
    }
    bool continue_playback = sf::Music::onGetData(data);
    if (continue_playback) {
        last_music_position.store(getPlayingOffset());
        last_position_update.restart();
    }
    return continue_playback;
}

And here's the resulting chart :

(click to show/hide)

We are clearly overshooting on our lag correction.

Method #3.5 : Take less lag into account

Interestingly, only substracting half the measured lag gives this curve :

(click to show/hide)

I then tried adjusting the lag correction to get the same results as Method #1 but I couldn't find a non-magic value based solution, while substracting half the lag seemed like a reasonable thing, I'm not going to try and adjust it even more to get the curves looking nice since that would probably just make my code overfitted to my test file , there has to be some hardware-dependant things that will break any result that's been found by twiddling the amount of lag correction.

I'm unsure about the ideal solution, I like Method 3.5 but I think the correction we are looking for is closer to Method #1

[Laurent]

If I understand correctly, we're tied to the internal precision of OpenAL Soft, which depends on the sample rate and internal buffer size. The sample rate is a property of the loaded sound, so it can be changed easily; the internal buffer size can be changed with a configuration file (see https://github.com/kcat/openal-soft/blob/master/alsoftrc.sample#L74).

I don't like solutions using a separate clock, ie. not relying completely on the played bytes, but I guess there's no other reasonable choice as long as OpenAL Soft doesn't give us more precision.

[Stepland]

If one were able to measure time using the audio hardware clock you could have a simplified Method #1 where you just check for the first step and then measure time from there since there should be no drift ? Or maybe other things I don't know about would come into play, I'm unsure.

Looks like it was in the works at some point for OpenAL ? http://openal.org/pipermail/openal/2017-December/000670.html

[Stepland]

Just found out about AL_SEC_OFFSET_LATENCY_SOFT (It's described here) and figured it might help.

There might be a way to use it to make a better Method #2

I tried it but I couldn't quite make sense of the values it was returning. I'm very new to OpenAL so I'm still struggling a bit ...

[Stepland]

I am getting very close to a solution :



Subtracting lag from the offset reported by OpenAL gives a very smooth result, only two thing to fix :

• It moves back to zero every time sf::SoundStream clears unsed buffers
• it's a bit late compared to the reported offset

I already have ideas to overcome both of these. I'm getting there !

[Stepland]

Well, I completely overlooked that I could just do

getPlayingOffset() - latency

This is almost perfect :

(click to show/hide)

(click to show/hide)

[Stepland]

Alright here's what I came up with so far !

(click to show/hide)

#include <filesystem>
#include <array>

#include <SFML/Audio.hpp>

#include "AL/al.h"
#include "AL/alext.h"

struct PreciseMusic : sf::Music {
    PreciseMusic(const std::filesystem::path& path);
    std::array<sf::Time, 2> alSecOffsetLatencySoft() const;
    sf::Time getPrecisePlayingOffset() const;
    sf::Time lag = sf::Time::Zero;
    void play();
protected:
    LPALGETSOURCEDVSOFT alGetSourcedvSOFT;
};

(click to show/hide)

#include "precise_music.hpp"

PreciseMusic::PreciseMusic(const std::filesystem::path& path) {
    if (not alIsExtensionPresent("AL_SOFT_source_latency")) {
        throw std::runtime_error("Error: AL_SOFT_source_latency not supported");
    }

    alGetSourcedvSOFT = reinterpret_cast<LPALGETSOURCEDVSOFT>(alGetProcAddress("alGetSourcedvSOFT"));

    if (not this->openFromFile(path.string())) {
        throw std::runtime_error("Could not open "+path.string());
    }
}

std::array<sf::Time, 2> PreciseMusic::alSecOffsetLatencySoft() const {
    ALdouble offsets[2];
    alGetSourcedvSOFT(m_source, AL_SEC_OFFSET_LATENCY_SOFT, offsets);
    return {sf::seconds(offsets[0]), sf::seconds(offsets[1])};
}

void PreciseMusic::play() {
    sf::Music::play();
    lag = this->alSecOffsetLatencySoft()[1];
}

sf::Time PreciseMusic::getPrecisePlayingOffset() const {
    if (this->getStatus() != sf::Music::Playing) {
        return sf::Music::getPlayingOffset();
    } else {
        return (
            sf::Music::getPlayingOffset()
            - (this->alSecOffsetLatencySoft()[1] * this->getPitch())
            + lag
        );
    }
}

I just measure the initial lag once per play call and shift by that amount.

Apparently the reinterpret_cast of a void pointer to a function pointer is something very non-standard that could very possibly break at any given time, but it worked for me ! (gcc (Ubuntu 9.3.0-17ubuntu1~20.04))

If a more knowledgeable person could step in and help us find a better way to handle this that would be awesome.

The resulting value occasionally jumps a little, but overall it's pretty clean :

(click to show/hide)

(click to show/hide)

(click to show/hide)

It even seems to work when changing the playback position or the pitch while the song plays, so I don't think I'll bother fixing this any further. It seems to suit my needs for now, and I hope it'll be enough for you as well