Bug 16274 – The curses of debugging: short argument passed in 16-bit register, against ABI

On my system (OS X 10.11.5 (15F34), 64-bit), if you compile (with DMD) and run the following program you'll get an executable that behaves incorrectly about half of the executions: extern(C) { void* initscr(); int start_color(); int init_pair(short pair, short f, short b); int attron(int attrs); int mvprintw(int y, int x, const(char)* fmt, ...); int attroff(int attrs); int getch(); int endwin(); int refresh(); } enum { COLOR_BLACK = 0, COLOR_RED = 1, COLOR_GREEN = 2, COLOR_YELLOW = 3, COLOR_BLUE = 4, COLOR_MAGENTA = 5, COLOR_CYAN = 6, COLOR_WHITE = 7 } int COLOR_PAIR(int n) { return n << 8; } void main() { initscr(); start_color(); init_pair(1, COLOR_WHITE, COLOR_BLUE); int pair = COLOR_PAIR(1); attron(pair); mvprintw(1, 1, "Test"); attroff(pair); refresh(); getch(); endwin(); } $ rdmd -L-lcurses test.d If all is well, you'll see the text "Test" in white text on a blue background. If the bug is exercised then you won't see any text. Sometimes it takes quite a few number of executions until it starts or stops exhibiting the problem. This reduced text case was taken very painstakingly from a large codebase, where changing unrelated things around would make the problem come and go --- probably; it was sometimes hard to tell, given the non-deterministic bug. Currently, it seems that this program always works fine when using LDC, although my vague recollection is that this wasn't true when I had last looked at this issue (which was before the latest LDC). That might be because of the following difference in compilation: LDC: __Dmain: subq $0x38, %rsp callq _initscr movq %rax, 0x28(%rsp) callq _start_color movl $0x1, %edi <--- movl $0x7, %esi <--- movl $0x4, %edx <--- movl %eax, 0x24(%rsp) callq _init_pair DMD: __Dmain: pushq %rbp movq %rsp, %rbp subq $0x10, %rsp callq _initscr callq _start_color movw $0x4, %dx <--- movw $0x7, %si <--- movw $0x1, %di <--- callq _init_pair And indeed, if you change the code to the following the problem goes away: void main() { initscr(); start_color(); asm { xor EDX, EDX; xor ESI, ESI; xor EDI, EDI; } init_pair(1, COLOR_WHITE, COLOR_BLUE); int pair = COLOR_PAIR(1); attron(pair); mvprintw(1, 1, "Test"); attroff(pair); refresh(); getch(); endwin(); } I guess the ABI requires the whole register to be used for a 16-bit value? The docs for my system say "The OS X x86-64 function calling conventions are the same as the function calling conventions described in System V Application Binary Interface AMD64 Architecture Processor Supplement, found at http://people.freebsd.org/~obrien/amd64-elf-abi.pdf. See that document for details." In that document I found the following: • Arguments of types (signed and unsigned) _Bool, char, short, int, long, long long, and pointers are in the INTEGER class (...) 2. If the class is INTEGER, the next available register of the sequence %rdi, %rsi, %rdx, %rcx, %r8 and %r9 is used So I guess this is a DMD code generation bug?

(I'm using the latest DMD, v2.071.1, although this problem has existed for quite a while)

A smaller version: extern(C) int init_pair(short pair, short f, short b); void test() { init_pair(1, 2, 3); } which generates: _D4testQfFZv: push RBP mov RBP,RSP mov 8D,3 mov DX,2 mov CX,1 sub RSP,020h call init_pair add RSP,020h pop RBP ret Indeed, you are correct in that the upper 16 bits of the register is not set. But I believe this correct code generation. The problem may be in the implementation of init_pair(), which is assuming the upper 16 bits are zero.

gcc, however, does indeed set the upper 16 bits to zero. I suppose dmd should behave the same way, regardless of the spec.

Curiously, I implemented init_pair() and it turns out that gcc sign extends each parameter before using it. So this problem does depend on what's going on with init_pair's implementation.

@WalterBright created dlang/dmd pull request #11651 "fix Issue 16274 - The curses of debugging: short argument passed in 1…" fixing this issue: - fix Issue 16274 - The curses of debugging: short argument passed in 16-bit register, against ABI https://github.com/dlang/dmd/pull/11651

@ibuclaw created dlang/dmd pull request #11719 "fix Issue 16274 - short argument passed in 16 bit register" fixing this issue: - fix Issue 16274 - short argument passed in 16 bit register https://github.com/dlang/dmd/pull/11719

dlang/dmd pull request #11651 "fix Issue 16274 - short argument passed in 16 bit register" was merged into master: - 00271210bfe8fe9dc4d4bfe17045b17b1621df59 by Walter Bright: fix Issue 16274 - short argument passed in 16-bit register, against ABI https://github.com/dlang/dmd/pull/11651