Bug 18619 – documented unittests extracted by test_extractor should be run by auto-tester
Status
RESOLVED
Resolution
WONTFIX
Severity
enhancement
Priority
P1
Component
tools
Product
D
Version
D2
Platform
x86
OS
All
Creation time
2018-03-15T10:09:34Z
Last change time
2018-03-15T10:38:29Z
Assigned to
No Owner
Creator
Timothee Cour
Comments
Comment #0 by timothee.cour2 — 2018-03-15T10:09:34Z
documented unittests extracted by test_extractor should be run by auto-tester
this would prevent publishing broken unittests in ddoc (eg missing imports)
context: https://issues.dlang.org/show_bug.cgi?id=18488#c5
Comment #1 by greeenify — 2018-03-15T10:16:30Z
I think you misunderstood me. They are already run on CircleCi and there is almost no value in running them on auto-tester as they are already run there, just not extracted. So the problems only amount to accessibility.
Also there are severe disadvantages about the auto-tester: it's a horrible pain to get stuff running there + the pipeline currently takes ~3 minutes which would be added to all ten host runs.
Comment #2 by timothee.cour2 — 2018-03-15T10:27:23Z
so if i understand correctly circleci runs the (unextracted) unittests blocks (regardless of documented or not) and at the moment nothing (except end user on dlang.org) runs the extracted ones; the things that can go wrong in 2nd step are cases of documented unittest relying on private symbols, ie accessibility issues.
These should be caught in code reviews as they're not caught automatically.
> Also there are severe disadvantages about the auto-tester: it's a horrible pain to get stuff running there + the pipeline currently takes ~3 minutes which would be added to all ten host runs.
ok, I understand the tradeoff, that's a good explanation for closing it, thanks!
Comment #3 by greeenify — 2018-03-15T10:38:29Z
(In reply to Timothee Cour from comment #2)
> so if i understand correctly circleci runs the (unextracted) unittests
> blocks (regardless of documented or not) and at the moment nothing (except
> end user on dlang.org) runs the extracted ones; the things that can go wrong
> in 2nd step are cases of documented unittest relying on private symbols, ie
> accessibility issues.
>
> These should be caught in code reviews as they're not caught automatically.
CircleCi runs `make -f publictest -j4`
This is the extraction and run of the public examples.
The normal testsuite with all unittests is run in all CIs is run on auto-tester and CircleCi - the latter is needed for reporting the coverage to CodeCov.
There might be bugs in the extraction step, but apart from that it's almost guaranteed that the examples are runnable via run.dlang.io on dlang.org