Bug 8091 – Optimizer generates wrong code when reducing comparisons.

DMD optimizer generates a wrong code for the program below. The second assertion unexpectedly fails if you specify -O flag to DMD, while it passes as expected without optimization. $ dmd -m32 -run test.d # fine $ dmd -m32 -O -run test.d # AssertError@test(8): Assertion failure Tested against DMDv2.060 (git HEAD) on 32-bit Linux. Found by hos_lyric <https://twitter.com/#!/hos_lyric_/status/201270597265793024>. ---- int solve(int n) { int a = (n % 3 == 0) ? 1 : (n % 3 == 1) ? 1 : 0; return (a != 0) ? a : 0; } void main() { assert(solve(0) == 1); assert(solve(1) == 1); // line 8, fails with -O assert(solve(2) == 0); }

more simplification: ---- int solve(int n) { int a = (n == 0) ? 1 : (n == 1) ? 1 : 0; return (a != 0) ? a : 0; } void main() { assert(solve(0) == 1); assert(solve(1) == 1); // fails with -O assert(solve(2) == 0); }

Marginally reduced (this is what the early optimizer passes turn this into): int solve(int n) { int a = n ? (n>=1u) : 1; return a ? a : 0; } void main() { assert(solve(1) == 1); } It looks as after common subexpression elimination of a, the branch address from the >= is out by 1. The generated code is: push EAX cmp dword ptr -4[EBP],0 je L1C mov ECX,1 cmp dword ptr -4[EBP],1 jae L1F mov ECX,0 jmp short L1F <--- wrong address L1C: xor ECX,ECX inc ECX L1F: je L25 <---- shouldn't jump here mov EAX,ECX <---- should jump here instead jmp short L27 L25: xor EAX,EAX L27: mov ESP,EBP

Commit pushed to dmd-1.x at https://github.com/D-Programming-Language/dmd https://github.com/D-Programming-Language/dmd/commit/8a689e26e1ba6565d76b05d045674967e83aef60 fix Issue 8091 - Optimizer generates wrong code when reducing comparisons

Commit pushed to master at https://github.com/D-Programming-Language/dmd https://github.com/D-Programming-Language/dmd/commit/40f81b41931eea6768b38b44802e35cd4f32f419 fix Issue 8091 - Optimizer generates wrong code when reducing comparisons