In the olden times, i had a function that would convert a WideString to an AnsiString of the specified code-page:
And everything worked. I passed the function a unicode string (i.e. UTF-16 encoded data) and converted it to an AnsiString , with the understanding that the bytes in the AnsiString represented characters from the specified code-page.
would return the Windows-1252 encoded string:
Note: Information was of course lost during the conversion from the full Unicode character set to the limited confines of the Windows-1252 code page:
- Ŧĥε ｑùíçķ ƀřǭŵņ fôｘ ǰűmpεď ōvêŗ ţħě łáƶÿ ďơǥ (before)
- The qùíçk brown fôx jumped ovêr the lázÿ dog (after)
But the Windows WideChartoMultiByte does a pretty good job of best-fit mapping; as it is designed to do.
Now the after times
Now we are in the after times. WideString is now a pariah, with UnicodeString being the goodness. It’s an inconsequential change; as the Windows function only needed a pointer to a series of WideChar anyway (which a UnicodeString also is). So we change the declaration to use UnicodeString instead:
Now we come to the return value. i have an AnsiString that contains the bytes:
In the olden times that was fine. I kept track of what code-page the AnsiString actually contained; i had to remember that the returned AnsiString was not encoded using the computer’s locale (e.g. Windows 1258), but instead is encoded using another code-page (the CodePage code page).
But in Delphi XE6 an AnsiString also secretly contains the codepage:
- codePage: 1258
- length: 44
- value:The qùíçk brown fôx jumped ovêr the lázÿ dog
This code-page is wrong. Delphi is specifying the code-page of my computer, rather than the code-page that the string is. Technically this is not a problem, i always understood that the AnsiString was in a particular code-page, i just had to be sure to pass that information along.
So when i wanted to decode the string, i had to pass along the code-page with it:
Then one person screws everything up
The problem was that in the olden times i declared a type called Utf8String :
Because it was common enough to have:
and the reverse:
Now in XE6 i have a function that takes a Utf8String . If some existing code somewhere were take a UTF-8 encoded AnsiString , and try to convert it to UnicodeString using Utf8ToWideString it would fail:
Or worse, is the breadth of existing code that does:
The returned string will become totally mangled:
- the function returns AnsiString(1252) ( AnsiString tagged as encoded using the current codepage)
- the return result is being stored in an AnsiString(65001) string ( Utf8String )
- Delphi converts the UTF-8 encoded string into UTF-8 as though it was 1252.
Ideally my UnicodeStringToString(string, codePage) function (which returns an AnsiString ) could set the CodePage inside the string to match the actual code-page using something like SetCodePage :
Except that manually mucking around with the internal structure of an AnsiString is horribly dangerous.
So what about returning RawByteString ?
It has been said, over an over, by a lot of people who aren’t me that RawByteString is meant to be the universal recipient; it wasn’t meant to be as a return parameter:
This has the virtue of being able to use the supported and documented SetCodePage .
But if we’re going to cross a line, and start returning RawByteString , surely Delphi already has a function that can convert a UnicodeString to a RawByteString string and vice versa:
This was a long-winded set of background for a trivial question. The real question is, of course, what should i be doing instead? There is a lot of code out there that depends on the UnicodeStringToString and the reverse.
I can convert a UnicodeString to UTF by doing:
and i can convert a UnicodeString to the current code-page by using:
But how do i convert a UnicodeString to an arbitrary (unspecified) code-page?
My feeling is that since everything really is an AnsiString :
i should bite the bullet, bust open the AnsiString structure, and poke the correct code-page into it:
Then the rest of the VCL will fall in line.
3 Answers 3
In this particular case, using RawByteString is an appropriate solution:
This way, the RawByteString holds the codepage, and assigning the RawByteString to any other string type, whether that be AnsiString or UTF8String or whatever, will allow the RTL to automatically convert the RawByteString data from its current codepage to the destination string’s codepage (which includes conversions to UnicodeString ).
If you absolutely must return an AnsiString (which I do not recommend), you can still use SetCodePage() via a typecast:
The reverse is much easier, just use the codepage already stored in a (Ansi|RawByte)String (just make sure those codepages are always accurate), since the RTL already knows how to retrieve and use the codepage for you:
That being said, I would suggest dropping the helper functions altogether and just use typed strings instead. Let the RTL handle conversions for you:
I think that returning a RawByteString is probably as good as you’ll get. You could do it using AnsiString as you outlined but RawByteString captures the intent better. In this scenario a RawByteString morally counts as a parameter in the sense of the official Embarcadero advice. It is just an output rather than an input. The real key is not to use it as a variable.
You could code it like this:
outputs 1252, 1251, and then 65001 as you would expect.
And you could use LocaleCharsFromUnicode if you prefer. Of course, you need to take its documentation with a pinch of salt: LocaleCharsFromUnicode is a wrapper for the WideCharToMultiByte function. Amazing that text was ever written since LocaleCharsFromUnicode surely only exists to be cross-platform.
However, I wonder if you may be making a mistake in attempting to keep ANSI encoded text in AnsiString variables in your program. Normally you would encoded to ANSI as late as possible (at the interop boundary), and likewise decode as early as possible.
If you simply have to do this then perhaps there is a better solution that avoids the dreaded AnsiString completely. Instead of storing the text in an AnsiString , store it in TBytes . You already have data structures that keep track of encoding, so why not keep them. Replace the record that contains code page and AnsiString with one containing code page and TBytes . Then you would have no fear of anything recoding your text behind your back. And your code will be ready to use on the mobile compilers.
StringToWideChar Преобразовывает строку формата ANSI в Unicode-строку.
WideCharLenToString Преобразовывает указанное количество символов Unicode-строки в ANSI строку.
WideCharLenToStrVar Преобразовывает заданное количество символов Unicode-строки в ANSI формат и копирует результат в указанную переменную.
WideCharToString Преобразовывает длинную строку Unicode в ANSI строку.
WideCharLenToStrVar Преобразовывает строку формата Unicode в ANSI-формат и копирует результирующую строку в указанную переменную.
Я переношу приложение isapi (pageproducers) из delphi 7 в delphi 2009, страницы основаны на html файлах в UTF8.
Все идет хорошо, за исключением случаев, когда Onhtmltag запущен, и я заменяю прозрачный тег любым значением специальными символами, такими как акцентированные символы (éé. ) Эти символы заменяются на выходе символом.
Как часть вашей процедуры отладки, вы должны выяснить, какие байтовые значения, полученные браузером для символа вопросительного знака.
Как вы должны знать, строковый тип Delphi 2009 — Unicode, тогда как все предыдущие версии были ANSI. Delphi 7 представила тип Utf8String , но Delphi 2009 сделал этот тип особенным. Если вы не используете этот тип для хранения строк, которые кодируются как UTF-8, вам следует начать делать это. Значения, хранящиеся в переменных Utf8String , автоматически преобразуются в значения UnicodeString при назначении друг другу.
Если вы сохраняете строки в кодировке UTF-8 в обычных переменных AnsiString , они будут преобразованы в Юникод, используя страницу системного кода по умолчанию, если вы назначили их UnicodeString . Это не то, что вы хотите.
Если вы назначаете литералы с кодировкой UTF-8 для переменных типа string , остановите это. Этот тип ожидает, что его значения будут закодированы как UTF-16, так же как WideString всегда.