String doesn't display accents under Linux

[mateandmetal]

Hi... I'm trying to display spanish strings with accents under Linux.. with no luck 

My GCC version is 4.7.2
Recent SFML 2.0 (compiled by me)
Linux Fedora 18 x64
Codelite IDE

Here is my minimal example:

#include <iostream>
#include <string>
#include <SFML/Graphics.hpp>

using std::cout;
using std::endl;


int main() {


    // Create the main window
    sf::RenderWindow window (sf::VideoMode (800, 600), "String Test");

    // Load font
    sf::Font myFont;
    if (!myFont.loadFromFile("./fuente.ttf")) {
        cout << "error loading font" << endl;
        return -1;
    }


        // Wide string
        const wchar_t *myStringW = L"wide string: áéíóú";
        cout << myStringW << endl;
       
        sf::Text myTextW (myStringW, myFont, 50);
        myTextW.setPosition(20, 100);

        // ANSI string
        const char *myANSIstring = "ANSI string: áéíóú";
        cout << myANSIstring << endl;
       
        sf::Text myANSItext (myANSIstring, myFont, 50);
        myANSItext.setPosition(20, 150);


    // Start the game loop
    while (window.isOpen()) {

        // Process events
        sf::Event event;
        while (window.pollEvent(event)) {

            // Close window : exit
            if (event.type == sf::Event::Closed)
                 window.close();
        }

        // Clear screen
        window.clear();

        // Draw
        window.draw(myTextW);
        window.draw(myANSItext);

        // Update the window
        window.display();

    }

    return EXIT_SUCCESS;

}
 

The wide string displays correctly using sf::Text. The ANSI string does not display accents.
Should I always use wide strings under Linux?


[attachment deleted by admin]

[Laurent]

Should I always use wide strings under Linux?

If you want to avoid problems related to the various encodings involved (source file encoding, compiler encoding, current locale encoding, ...), then yes, definitely.

[mateandmetal]

Same behavior under win7 with MinGW. The wide string works, the ANSI one not 
Thanks Laurent

[eXpl0it3r]

What's the encoding of your file and what's the language you've serup your OSs with?

Personally I wouldn't expect the ANSI (couldn't it be also just ASCII?) version to work.

[Laurent]

There are some limitations with MinGW, it can only use the current globale. And I think that by default, the current globale is the "C" one (which knows only about ASCII characters).

Try the same but change the global locale before,

std::locale::global(std::locale(""));

[mateandmetal]

My goal is to support:
- Windows and Linux platforms
- English and Spanish languages (maybe more)

Testing the same code, Win7 x64 SP1, Region: Spanish (Argentina), IDE: Codeblocks 12, file encoding: UTF-8 (if I leave the default codeblocks encoding, I can´t compile the code)
Inserting the code posted by Laurent at the beginning of the main function




Something interesting I found:
std::wstring VS std::string

1. When I should use std::wstring over std::string?

On Linux? Almost never

On Windows? Almost always

I know there is a lot of info about encodings. I try to understand but its very confusing 
Should I use wide strings and wide file streams to read them from files?
What kind of encoding should I use for my source code files?

[mateandmetal]

Personally I wouldn't expect the ANSI (couldn't it be also just ASCII?) version to work.

Well, using this code (same Win7 OS):

    // Create the main window
    sf::RenderWindow window (sf::VideoMode (800, 600), "String Test");

    // Load font
    sf::Font myFont;
    if (!myFont.loadFromFile("./steelfish_rg.ttf")) {
        cout << "error loading font" << endl;
        return -1;
    }

    // ANSI
    const char *myANSIstring = "Hola áéíóú";
    sf::Text myANSItext (myANSIstring, myFont, 50);

    // Start the game loop
    while (window.isOpen()) {

        // Process events
        sf::Event event;
        while (window.pollEvent(event)) {

            // Close window : exit
            if (event.type == sf::Event::Closed)
                 window.close();
        }

        // Clear screen
        window.clear();

        // Draw
                window.draw(myANSItext);

        // Update the window
        window.display();

    }

    return EXIT_SUCCESS;
 

with codeblocks default system native encoding:


works fine 

[mateandmetal]

Portability, cross-platform interoperability and simplicity are more important than interoperability with existing platform APIs. So, the best approach is to use UTF-8 narrow strings everywhere and convert them back and forth on Windows before calling APIs that accept strings.

UTF-8 Everywhere

I'm using UTF-8 encoding under Linux, however the SFML text doesn't work as expected:

#include <iostream>
#include <string>
#include <locale>

#include <SFML/Graphics.hpp>

using std::cout;
using std::endl;

int main (int argc, char **argv)
{
        std::locale myLocale ("spanish"); // spanish
        std::locale::global (myLocale); // needed for console output only???
        cout << "Locale name = " << myLocale.name() << endl;

        const char *myANSI = "Hola áéíóú";
        cout << "C style ANSI string = " << myANSI << endl;    
       
        sf::String mySFML_String (myANSI, myLocale);
        std::string standardString = mySFML_String.toAnsiString();
       
        cout << "From SFML to std::string = " << standardString.c_str() << endl;
       
        // Graphics Mode xD
        sf::RenderWindow window (sf::VideoMode(600, 400), "Hello Strings!");
       
        sf::Font myFont;
        if (!myFont.loadFromFile("./DYST.ttf")) {
                cout << "Error loading fntfile" << endl;
                return -1;
        }
       
        sf::Text myText (mySFML_String, myFont, 50);
       
        // Loop
        while (window.isOpen()) {
               
                sf::Event myEvents;
                while (window.pollEvent(myEvents)) {
                        if (myEvents.type == sf::Event::Closed) {
                                window.close();
                        }
                } // pollEvent
               
                window.clear();
                window.draw(myText);
                window.display();
               
        } // isOpen
       
       
        return 0;
}
 

Screenshot:

(Font file supports accents)

I don't know what else should I do... 
Please help

[Laurent]

Since your environment is configured to use UTF-8, your literal string is UTF-8 and you must therefore use the corresponding function to convert it to a sf::String (see the sf::Utf8 class), using the "spanish" locale is wrong.

[mateandmetal]

Ok, let me see if I understand

- Should I call Utf8::decode on each character of the ANSI string?
- "Iterator pointing to the beginning of the input sequence" means something like std::string::begin?
- Is this a heavy process?
- Do I need to get/change the C/C++ locale?

[Laurent]

std::string utf8str = ...;
std::basic_string<sf::Uint32> utf32str;
sf::Utf8::toUtf32(utf8str.begin(), utf8str.end(), std::back_inserter(utf32str));
sf::String sfstr = utf32str;

Note that this code is not optimized: it doesn't  preallocate the memory, and uses an extra UTF-32 string rather than filling the sf::String directly. I'd have to tweak sf::String to allow a more optimized code.

And no, it's not heavy.

But if you're only using literal strings, why do you make it so complicated? Just use wide strings.

[mateandmetal]

Thanks for the example code, I will test it.

But if you're only using literal strings, why do you make it so complicated? Just use wide strings.

Because I thought that it was not necessary (at least under Linux)
Ok, thanks again Laurent!

[Laurent]

Nothing is mandatory, but using Unicode directly rather than dealing with locales is always much easier.