Unused generics (generic functions, classes, structs, templates, and mixins) need initially to be parsed only as balanced curlies - not even tokenized. They get tokenized, parsed, and semantically analyzed only upon first use.
This allows libraries (such as Phobos itself) to provide functionality in templates at virtually no cost to their users.
Comment #1 by deadalnix — 2015-06-11T01:11:23Z
void foo()() {
return "}";
}
It is not as simple as you think it is (but probably doable).
Comment #2 by dlang-bugzilla — 2015-06-11T01:13:45Z
AFAIK the most expensive part of tokenization is identifier lookup (converting an identifier string into a number or pointer).
Comment #3 by andrei — 2015-06-11T02:13:39Z
Yah, the alternate grammar needs to recognize comments, string literals, and newlines (for line counting purposes).
One nice thing about not doing tokenization is a lot of memory is saved.
Comment #4 by bruno.do.medeiros+deebugz — 2015-06-12T14:37:14Z
(In reply to Andrei Alexandrescu from comment #3)
> Yah, the alternate grammar needs to recognize comments, string literals, and
> newlines (for line counting purposes).
>
> One nice thing about not doing tokenization is a lot of memory is saved.
Why does tokenization consume memory? I imagine you can just have the token point to a slice of the source?
Comment #5 by robert.schadek — 2024-12-13T18:43:19Z