Each new file added to the testsuite may as well be adding 5 seconds onto the overall runtime. Logical groupings could be done to improve the overall CI speed.
For example, all fail_compilation tests that check coverage of error messages in `dmd/dinterpret.d` could be put into a single module `fail_compilation/fail_dinterpret.d`
Comment #1 by greensunny12 — 2018-04-05T03:55:30Z
Not saying anything about the logical grouping, but this is a bit exaggerated imho:
> Each new file added to the testsuite may as well be adding 5 seconds onto the overall runtime.
_All_ 1283 fail_compilation tests compile in ~11s on my machine.
The 774 compilable tests are slightly slower with ~25s ... and you don't want to know how long runnable takes :/
(of course the numbers are taken with full parallelism on an i7 4th Gen)
Anyhow, if someone wants to make the testsuite faster, the runnable tests are the real bad boys and should be focussed on first.
Comment #2 by moonlightsentinel — 2021-05-20T18:15:50Z
Grouping test files doesn't achieve much in regard to the general test execution time. The real time sink are the permutations for DMD's optimizer/inliner (compare `./run.d` with `./run.d quick`)
Comment #3 by hey — 2022-01-12T17:25:14Z
Hi! I'm looking to contribute to D and have experience as an SDET so this issue looks perfect. However, from the comments and age of the issue, it's not clear if this is still required. Is it a genuine open issue?
Comment #4 by dkorpel — 2022-01-12T18:20:13Z
(In reply to Eddy Schauman-Haigh from comment #3)
> However, from the comments and age of the issue, it's
> not clear if this is still required. Is it a genuine open issue?
Yes, it's still an open issue, the test suite has only grown since. Contributions are appreciated! You can find the test suite here:
https://github.com/dlang/dmd/tree/master/test
Comment #5 by hey — 2022-01-12T18:36:24Z
(In reply to Dennis from comment #4)
> (In reply to Eddy Schauman-Haigh from comment #3)
> > However, from the comments and age of the issue, it's
> > not clear if this is still required. Is it a genuine open issue?
>
> Yes, it's still an open issue, the test suite has only grown since.
> Contributions are appreciated! You can find the test suite here:
>
> https://github.com/dlang/dmd/tree/master/test
Great! In which case I'll start having a look at this issue. Thanks for the swift reply Dennis!
Comment #6 by ibuclaw — 2022-01-13T09:57:54Z
Just be sure to benchmark, benchmark, and benchmark some more!
Things that have changed since the original raising of this include:
- Running the most expensive tests first had reduced the time it took to complete by about a minute when using parallelism.
- Fail compilation tests no longer run with all permutations.
- Neither do compilable tests.
Comment #7 by robert.schadek — 2024-12-13T18:56:08Z