[fixed] SFML2: sf::Utf8::ToUtf32 conversion problem

[MarsCoreTeam]

Hello there!

I am trying to load an utf8-encoded text file into a std::vector<sf::String>. When I use this litte, but great library everything works as supposed to. After doing

Code: [Select]


void load(std::vector<sf::String>& lines) {
    // Open the test file (contains UTF-8 encoded text)
    std::ifstream fileStream("unitext.txt");
    std::string line;
    while (std::getline(fileStream, line)) {
        std::basic_string<sf::Uint32> utf32line;
        // this line converts utf8 to utf32 with the help of the metioned library
        utf8::utf8to32(line.begin(), line.end(), back_inserter(utf32line));
        lines.push_back(utf32line);
    }
}


the vector "lines" contains utf-32 encoded sf::Strings.
But since SFML supports utf conversions as well, I tried to do the same, only with SFML. Therefore I changed

Code: [Select]

utf8::utf8to32( ... );
to

Code: [Select]

sf::Utf8::ToUtf32( ... );
which, as I thought, should result in exactly the same behaviour. But it doesn't: When there is a non-ascii character, all trailing characters are dropped.

Am I doing something wrong :? ? Or is it a bug :shock: ?

Greetings, Simon.

[Laurent]

Can you show me a complete source code (with a sample text file similar to the one you load), so that I can try it myself?

[MarsCoreTeam]

Thanks for your reply, Laurent!

There we go... the following example loads an utf8-encoded text file. I used this one (right Click... save as). Since I'm on Linux, it's line endings are simple line feeds. Therefore it will look like one single line on Windows, but this shouldn't be a problem.
The lines of this file are displayed on screen with sf::Text's. The following code does not work for me. Nevertheless, when I include "utf8.h" (from the library mentioned in my first post) and use it's conversion function, it works...

Code: [Select]


# include <SFML/Graphics.hpp>
# include <fstream>
# include <sstream>
# include <iostream>
# include <vector>

//# include "utf8.h"

void load(std::vector<sf::String>& lines) {
    // load text file
    std::ifstream fileStream("unitext.txt");
    std::string line;
    // read line by line
    while (std::getline(fileStream, line)) {
        std::basic_string<sf::Uint32> utf32line;

        //utf8::utf8to32(line.begin(), line.end(), back_inserter(utf32line));
        sf::Utf8::ToUtf32(line.begin(), line.end(), back_inserter(utf32line));

        lines.push_back(utf32line);
    }
}

int main() {
    std::vector<sf::String> strings;
    load(strings);

    sf::RenderWindow App(sf::VideoMode(400, 300), "Unicode Test");
    App.SetFramerateLimit(60);

    // create a sf::Text for each sf::String
    std::vector<sf::Text> lines;
    int y(10);
    for (std::vector<sf::String>::iterator it = strings.begin(); it != strings.end(); ++it, y += 40) {
        sf::Text line(*it, sf::Font::GetDefaultFont(), 30.f);
        line.SetPosition(10.f, y);
        lines.push_back(line);
    }

    while (App.IsOpened())
    {
        sf::Event Event;
        while (App.GetEvent(Event))
        {
            if (Event.Type == sf::Event::Closed)
                App.Close();
        }
        App.Clear();

        // draw them all
        for (std::vector<sf::Text>::iterator it = lines.begin(); it != lines.end(); ++it)
            App.Draw(*it);

        App.Display();
    }
    return EXIT_SUCCESS;
}


[Laurent]

Thank you very much, I'll test it as soon as possible

[Laurent]

It was a bug in SFML, the conversion code was missing a cast to unsigned when the input type was signed.

Thanks for your help

[MarsCoreTeam]

Hey, cool, Thanks ! And you've already committed it to SVN! Awesome!

Greetings, Simon.