Support UTF-8 in String? (->GNU/gettext)

[christoph]

Hi

would it be possible to add UTF-8 support to SFMLs String class so one can use it easily with the gettext library for producing multilingual software?

I can offer you an transfer method to get from std::basic_string<char> holding UTF-8 data to std::basic_string<uint32_t> holding widened data (which should be changeable to an wchar_t implementation) although this piece of code is kind of obscure.

[SirJulio]

Hi Christoph,

I think that String class already handles unicode text (ctor, wstring GetText(), SetText(wstring)), no ?

[christoph]

Yes but with gettext you'll get char* containing UTF-8 encoded data and this does not work currently. You can do a wraper around this transforming this char* / UTF-8 to an std::basic_string<wchar_t> but it would, I guess, be better in SFML and not in the application.

And I think the term multibyte for char* data is really confusing as it does not handle strings with multibyte char*s as far as I can tell

[Laurent]

SFML uses UTF-16 for unicode (in String class and in TextEntered event), if you have any other kind of input encoding it's up to you to write a conversion function. I won't put in SFML functions to convert between all kind of character encodings...

[SirJulio]

Ok christoph, I understand.

If you have a solution, maybe you could create an entry in the wiki. =)

[christoph]

SFML uses UTF-16 for unicode (in String class and in TextEntered event), if you have any other kind of input encoding it's up to you to write a conversion function. I won't put in SFML functions to convert between all kind of character encodings...

OK we'll have to live with this then. But looking at string/font sources the encoding looks more like UCS-2 and less like UTF-16 to me but you should know this better than me anyway

[Laurent]

But looking at string/font sources the encoding looks more like UCS-2 and less like UTF-16

I'm still not an expert in Unicode encodings, so I can make mistakes sometimes
Why do you say so ?

[christoph]

I was thinking about the encoding to not be UTF-16 as you assume that all characters go into an wchar_t. But UTF-16 has some characters (UNICODE Charmap is 24bit AFAIR) that need 2*16bit (wchar_t most probably is 8, 16 or 32 bit in size) so it will not be UTF-16 but (depending of the size of wchar_t) UCS-2 (16bit) or UCS-4/UTF-32 (32bit).

[Laurent]

Ok I see. You're probably right

[Miriam Ruiz]

I think that gettext() and UTF8 is so widespread in POSIX world (and is also supported in Windows world) that it would be a mistake not to support it someway if the idea is to do a multi-platform library.

[Miriam Ruiz]

After thinking a bit about it, there would be 2 satisfying alternative solutions:

1) add a void String::SetText(const std::string& Text, const std::locale& Locale) method
2) make std::wstring myText protected instead of private, and thus solvable through inheritance

What about that? would that be admisible for you? I just don't want to cut&paste sf::String into my app just to be able to add that functionality.

Any other ideas?

Greetings,
Miry

[workmad3]

UTF-16 is UCS2 with support for surrogate pairs (which are 32 bits wide) in order to extend the unicode support of the encoding from the BMP(basic multilingual plane) to the entire unicode set of code points.

[T.T.H.]

Unicode, UTF-8, UTF-16, BOM, etc. - trying to understand all those things somewhat decent took me a serious amount time.

Anyway, four things I want to throw in here:

1) a short but good article about "wchar_t: unsafe at any size"

2) in UTF-16 all characters of the so called "basic multilingual plane" are encoded in 2 bytes (= 16 bit). Those pretty much include all the characters you need to write pretty much every "living" language on this planet. Plus a lot of other symbol like characters, pretty much "Wingdings on steroids". Beyond that are some more characters, a lot of fancy stuff like old Germanic rune symbols and ancient Greek numbers and the music clef. Those characters need 4 bytes to be encoded. So please do not be fooled that "UTF-16" means "2 byte things only" or that "half number of bytes of a UTF-16 string == number of characters" which is simply wrong.

3) UTF-32 instead is "every single defined Unicode character there is in 4 bytes, not more, not less".

4) decodeunicode - a life safer...

[workmad3]

As I haven't posted it here yet, there's also:
http://www.joelonsoftware.com/printerFriendly/articles/Unicode.html
which is nicely described as 'The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)'

That was a very useful document for me when I had to deal with UTF-16 to UTF-8 stuff at work.

[Laurent]

Thanks for the links, I'm going to read them all and try to think about a more robust and flexible implementation for Unicode in SFML