Bug 3193 – Support Windows-1251 as a source encoding

If you compile hello.d example with Russian Win1251 charecters in this line: printf("Привет, D!\n"); dmd.exe reports an error: D:\Apps\Prog_D\dmd\samples\d>dmd hello.d hello.d(5): invalid UTF-8 sequence hello.d(5): invalid UTF-8 sequence hello.d(5): invalid UTF-8 sequence hello.d(5): invalid UTF-8 sequence hello.d(5): invalid UTF-8 sequence hello.d(5): invalid UTF-8 sequence If you save hello.d in UTF-8, then anyway dmd.exe compiles it wrong (see http link).

The compiler does not understand Windows-1251, so this is according to spec. However, you say the compiler compiles it wrong if it's in UTF-8; where's the link?

Created attachment 428 This screenshot is from Chris Miller

Sorry, this is invalid. To solve this, you have to do the following: 1) Set cmd.exe's font to Lucida Console. 2) Execute 'chcp 65001'. Then run your program.

Created attachment 429 Correct Russian output Here's an image that shows it working properly.

But Jarrett, almost everybody who codes in Russian needs Windows-1251 codepage by default. If we need to compile small program and we don't have robist IDE we use notapad.exe (or something like this) that saves Russian text in Windows-1251. And nobody will be changing his dafault font in "Command Prompt" to Lucida Console only for my small program - I swear you! Any other compilers (Pascal, C, C++) understand that the Russian text in Windows is in Windows-1251! Currently I dont have any good editor for D whare I can normally edit Russian texts in UTF-8. Entice Designer has a bug confirmed by Chris Miller - you cannot enter Russian text, only copy and paste. Therefore if you build a D compiler for Win32 platform, you have make it work with widely used regional codepages. Because the entire world is not English only and fully not UTF-8!

What you're basically asking for is an enhancement. I'm sorry, but that's the way it works.

Why not make this enhancement request "Write a decent, free, Unicode-compatible code editor that syntax-highlights D properly"?

(In reply to comment #7) > Why not make this enhancement request "Write a decent, free, Unicode-compatible > code editor that syntax-highlights D properly"? Why not be a sarcastic ass _all the time_?

Dear friends, D has really good ideas behind its face, but Unicode support (UTF-16) in the compiler instead of old UTF-8 is "MUST HAVE" feature. Its a great need in non-Latin languages. Windows-1251 codepage is #1 for Russian Windows programmers. Otherwise D compiler will stay an experiment forever.

(In reply to comment #9) > Dear friends, D has really good ideas behind its face, but Unicode support > (UTF-16) in the compiler instead of old UTF-8 is "MUST HAVE" feature. DMD already supports UTF-16. Even UTF-32. Why do you want UTF-8 support removed? > Its a great need in non-Latin languages. Windows-1251 codepage is #1 > for Russian Windows programmers. Otherwise D compiler will stay an > experiment forever. How would supporting codepages work anyway? Would they be converted to UTF-8 at compiletime? In this case, D would need some form of character encoding declaration. Or would they be left as are, and be rejected only in wchar, wchar[], dchar and dchar[] literals? What about all the D features and APIs that rely on char[] being UTF-8? Seriously, if you're going to code in D and need to use non-ASCII characters, it goes without saying that you should have a Unicode-compatible editor. The lack of good D editors may be a real issue at the moment, but AISI it makes little sense to try to work around it. No programming language is born with high-quality development tools. People need to write them. (That said, there have been a few dedicated D IDE projects. What's the highest stage of development any of them is at?)

Stewart, Windows compilers SHOULD understand and correctly convert regional characters for console and dialogs (from resource files). The simplest test for the compiler in Windows is to enter text in notepad.exe in regional language and try to compile the file. MS VCPP compiler, BCC compiler and any other C++ compiler do it. And if DMD supports UTF-16 then how to make it work with UTF-16 Russian text entered in the simplest Notepad editor?

> And if DMD supports UTF-16 then how to make it work with UTF-16 Russian text > entered in the simplest Notepad editor? DMD will automatically detect and work correctly with UTF-16 and UTF-32 encoded source files. The logic to do this is in module.c of the compiler source code. If it does not work with a particular UTF-16 encoded file, please attach that file to this bug report. Note that UTF-16 encoded files are not encoded using a code page. If a source file is encoded with a particular code page, there is no way for the compiler to automatically detect it. C compilers often have a command line flag which is used to tell it what code page to use. Using code pages, therefore, makes your source code completely non-portable which is one of the reasons why D uses Unicode instead.

Created attachment 431 D Windows Unicode text - edit in Notepad and compile result in Console

Created attachment 432 D UTF-8 text - edit in Notepad and compile result in Console

Created attachment 433 D ANSI text with Russian letters - edit in Notepad and compile result in Console

Dear Walter, Please take a close look at my last 3 attachements having "edit in Notepad and compile result in Console" text in descriptions. Note that all Russians have 866 codepage by default in Windows Command Prompt. Nobody will be switching 866 to any other codepage for console application.

Going by your screenshots and their descriptions, DMD is behaving correctly. I do, however, feel that D's stdio ought to support codepages(In reply to comment #16) > Dear Walter, > Please take a close look at my last 3 attachements having "edit in Notepad and > compile result in Console" text in descriptions. Going by your screenshots and their descriptions, DMD is behaving correctly. > Note that all Russians have 866 codepage by default in Windows Command Prompt. You mean it's hard-coded for each language's edition of Windows? That's something else that ought to change. > Nobody will be switching 866 to any other codepage for console application. Console output is an entirely separate issue from source encoding. I feel that D's stdio ought to support codepages, but it doesn't (aside from the fact that printf isn't part of D's stdio). Meanwhile, please check out my utility library http://pr.stewartsplace.org.uk/d/sutil/

I think support for codepages and other character types could be implemented in a library. That was the ambitious purpose behind std.encoding. Yet another great project for someone interested.

As to console output, it's a duplicate of (runtime) bug 2742 or bug 1448. Tango and C API work correctly, phobos doesn't. As to cp1251, this ice age technology is definitely not a way to go, unicode is a future. No, it's the present. Windows works in unicode and you should use it. As to convertion of source from ANSI to OEM codepage, it's valid RFE, but hardly one will implement it. You can try yourself.

(In reply to comment #19) > As to console output, it's a duplicate of (runtime) bug 2742 or bug 1448. This is getting OT for this bug report, but it's 2742 to which what this conversation has drifted into is related. 1448 is a separate issue. > As to convertion of source from ANSI to OEM codepage, it's valid > RFE, but hardly one will implement it. You can try yourself. I already have. See comment 17.

Hmm... your library is just an API, it has nothing to do with source encoding and as far as I see it accepts utf8 text, not ANSI.

(In reply to comment #21) > Hmm... your library is just an API, it has nothing to do with source encoding As has a lot of the discussion here from comment 13 onwards. Maybe, to avoid confusion, we should continue this conversation at bug 2742. Or perhaps even better, on the newsgroup. > and as far as I see it accepts utf8 text, not ANSI. Not quite. It communicates with the console in the console codepage. Application code communicates with it in UTF-8.

I came across this and think, that this can be closed already. Unicode source files works, I don't think other encodings for source files are required anymore. Console output is another story - discussed elsewhere