Bug 2749 – Make unittests named and nestable

The current unit tests lack some functionality. - The current unit tests are not named. - There is no output that specifically indicates that the tests were run. - A single failing test will prevent all other tests from running. - There is no indication of which test failed. - There is no way to only run a subset of tests. Changing the language to allow nested and named unittest would help. The nesting would allow for the only the tests within the scope to fail and subsequent tests would run. The naming could be used to provide output and to select which tests are run. Example of unittest =================== unittest ("XML") { unittest("elements") { assert(isValidXml("<aaa />")); assert(isValidXml("<aaa/>")); assert(isValidXml("<aaa></aaa>")); ... unittest("case unmatched") assert(!isValidXml("<AaA></aaa>")); unittest("no closing") assert(!isValidXml("<aaa>")); ... } unittest("attributes") { assert(isValidXml("<aaa abc='x'/>")); assert(isValidXml("<aaa abc="\x\"/>")); assert(isValidXml("<aaa abc=\"x\" def=\"y\"/>")); ... unittest("unquoted") assert(!isValidXml("<aaa abc=x />")); unittest("multi attr") assert(!isValidXml("<aaa abc='x' abc='y'/>")); ... } unittest("encoding") { assert(encode("hello") is "hello"); assert(encode("a > b") == "a &gt; b"); ... } }

The functionality you are asking for can be built on top of the existing unittests. If you are expecting that unittest will be something like NUnit then you will be disappointed. As they are set up, they are best run as part of normal program execution during development.

(In reply to comment #1) > The functionality you are asking for can be built on top of the existing > unittests. No. All unittests in one module are combined to create one function, and there is no way to provide a name for them. However, it is possible to change the runtime to continue running unittests after one fails, printing out the name of the modules with failed tests, and offer some useful report at the end.

OK so it takes a little boilerplate. unitests { static const name = "Foo"; if(!RunTestByName(name)) return; try { /// tests } catch(Excption e) { ProcessUnittestError(name,e); } } Or to be even cleaner unittest { RunTests!(Foo, Bar, Baz)(); } void Foo() { /* tests */ } void Bar() { /* tests */ } void Baz() { /* tests */ }

(In reply to comment #3) > OK so it takes a little boilerplate. You might as well say, you can write your unittests in main() and use version blocks to determine whether to run them. It only takes a bit more boilerplate.

(In reply to comment #0) > The current unit tests lack some functionality. > - The current unit tests are not named. > - There is no output that specifically indicates that the tests were run. I often put writefln statements in my unittests, which seem to solve that problem. > - A single failing test will prevent all other tests from running. > - There is no indication of which test failed. If, as in your example, all assert statements are within the unittest code, then how is the line number in the assert error message not an indication? If you're calling functions to do the asserts, and so you can't tell what in the unittest triggered it, then the aforementioned writefln statements are one way around it. > unittest ("XML") { Why make the unittest names strings, rather than identifiers? So if I'm understanding correctly: - Each unittest block would have a pass/fail condition. - When an assert is thrown, the immediately containing unittest fails and exits immediately. - If one unittest fails, the next will be run, but any unittest it's nested within would fail. Have I got this right? Moreover, would it be legal for statements to follow nested unittests within a unittest? How would these be handled?

(In reply to comment #5) > (In reply to comment #0) > > The current unit tests lack some functionality. > > - The current unit tests are not named. > > - There is no output that specifically indicates that the tests were run. > > I often put writefln statements in my unittests, which seem to solve that > problem. > Adding writefln is ok, but it's an implicit way of documenting the unittest. > > - A single failing test will prevent all other tests from running. > > - There is no indication of which test failed. > > If, as in your example, all assert statements are within the unittest code, > then how is the line number in the assert error message not an indication? > > If you're calling functions to do the asserts, and so you can't tell what in > the unittest triggered it, then the aforementioned writefln statements are one > way around it. > > > unittest ("XML") { > > Why make the unittest names strings, rather than identifiers? I agree that finding the failing assertion is not difficult, the difficultly is working out what the failing test is meant to be doing. The reason for naming and nesting, is that it groups related tests together and gives a description to them, this is the reason why strings are preferred to identifiers. > So if I'm understanding correctly: > - Each unittest block would have a pass/fail condition. Yes > - When an assert is thrown, the immediately containing unittest fails and exits > immediately. Yes > - If one unittest fails, the next will be run, but any unittest it's nested > within would fail. Any unittest it nested would not be executed, so they don't fail they are skipped. > ... Moreover, would it be legal for statements to follow > nested unittests within a unittest? How would these be handled? No reason to prevent statements between unittests, if a statement throws an exception that will be treated as a failure of that unittest.

(In reply to comment #6) >> - If one unittest fails, the next will be run, but any unittest it's nested >> within would fail. > > Any unittest it nested would not be executed, so they don't fail they are > skipped. I said "unittest it's nested within" not "unittest it nested". Now try again.

(In reply to comment #7) > (In reply to comment #6) > >> - If one unittest fails, the next will be run, but any unittest it's nested > >> within would fail. > > > > Any unittest it nested would not be executed, so they don't fail they are > > skipped. > > I said "unittest it's nested within" not "unittest it nested". Now try again. > If a nested test fails it also fails the parent unittest(s).

If this is integrated into the language it shouldn't require quotes though. It should be consistent with version() and debug() conditions accepting integers and identifiers.

This clearly needs a DIP. There are various workaround for what you want (such as using a `with` statement for the nesting), but it's not a bug per se.