We encourage use of Ddoc to document functions. But this can result in voluminous comments, which slow down the lexer. Lexing comments can be accelerated by using SIMD vector instructions.
A little inline assembler in the lexer.c dmd source code would implement this.
Comment #1 by briancschott — 2015-06-01T20:17:09Z
The best way to do this that I've found is to skip everything other than a set of bytes that varies based on the comment being lexed:
For /* */ comments:
0x0c (\n)
0x0d (\r)
0x2a (*)
0x2f (/)
x0e2 (Beginning of multi-byte UTF-8 newline)
For /+ +/ comments:
0x0c (\n)
0x0d (\r)
0x2b (+)
0x2f (/)
x0e2 (Beginning of multi-byte UTF-8 newline)
For // comments:
0x0c (\n)
0x0d (\r)
x0e2 (Beginning of multi-byte UTF-8 newline)
The instruction used in libdparse to do this is "pcmpestri", which requires SSE4.2 (First released in 2008 according to wikipedia). My advice is to leave most of the logic intact and implement the assembly code such that it may advance the lexer 0 or more bytes, so that the rest of the algorithm is not disrupted on machines that don't support SSE4.2.
Comment #2 by robert.schadek — 2024-12-13T18:43:08Z