Bug 8845 – Can't pass immediate or rvalue args to ref function parameters

Consider: float length(ref const(Vector) x); float distBetween = length(p1 - p0); or float speed = length(player.getVelocity()); These don't work, forcing this horrible code: Vector diff = p0 - p1; float distBetween = length(diff); Naturally, if I have a series of these operations, I end up with: Vector diff0 = p0 - p1; Vector diff1 = p1 - p2; Vector diff2 = p2 - p0; And it all starts getting very silly, the local namespace gets polluted very quickly, and sensible temporary names often don't even exist for these intermediate concepts, leading to really stupid names, or just random letters. There's no difference in security. Forcing creation of a function scoped variable just increases the probability of name conflict and inconveniences the programmer. It would be nicer if the temporary was only scoped for life within the functions call, and also un-named, so it doesn't pollute the functions namespace. I've heard arguments about the safety of the operation, but the 'workaround' is just to create a temporary, which has identical security properties. It's also no different than passing any pointer normally (a common suggestion), and no reason to significantly inconvenience the programmer. Perhaps this behaviour could be restricted to ref const, or ref in, if we're getting worried about the safety of the operation? That would perhaps even improve on how behaves now. This is particularly common when working with linear algebra. Vectors, matrices, quaternions are surely the most likely to produce this pattern.

(In reply to comment #0) > This is particularly common when working with linear algebra. Vectors, > matrices, quaternions are surely the most likely to produce this pattern. Yes, it's a common pattern (Example: I have seen it often in a little ray-tracer).

Is there a better version of this with all the auto-ref stuff?

(In reply to comment #2) > Is there a better version of this with all the auto-ref stuff? What do you mean? The solution that we discussed extensively at dconf never made an appearance. This is still precisely where it's always been as far as I know.

(In reply to comment #3) > (In reply to comment #2) > > Is there a better version of this with all the auto-ref stuff? > > What do you mean? > The solution that we discussed extensively at dconf never made an appearance. > This is still precisely where it's always been as far as I know. I was hoping there was a less vague report somewhere with all the dconf stuff in it. On a related note, do you know if there's a final-by-default report out there somewhere?

(In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #2) > > > Is there a better version of this with all the auto-ref stuff? > > > > What do you mean? > > The solution that we discussed extensively at dconf never made an appearance. > > This is still precisely where it's always been as far as I know. > > I was hoping there was a less vague report somewhere with all the dconf stuff > in it. On a related note, do you know if there's a final-by-default report out > there somewhere? I don't think there is. That would have been far too sensible :)

(In reply to comment #5) > I don't think there is. That would have been far too sensible :) Well, if you can remember where the discussion and agreement was, it might be a good idea to make one and link it.

(In reply to comment #4) > On a related note, do you know if there's a final-by-default report out > there somewhere? AFAIK Walter agreed for final-by-default: http://forum.dlang.org/thread/[email protected]?page=28#post-koqkhc:244nn:241:40digitalmars.com

(In reply to comment #6) > (In reply to comment #5) > > I don't think there is. That would have been far too sensible :) > > Well, if you can remember where the discussion and agreement was, it might be a > good idea to make one and link it. IIRC, it was actually you that proposed the winning deprecation path. It was comparable to the introduction of mandatory override. I think you suggested one version introduces a deprecation warning where override is stated but virtual is not present on the base, then the next version enforces it.

(In reply to comment #7) > AFAIK Walter agreed for final-by-default: > > http://forum.dlang.org/thread/[email protected]?page=28#post-koqkhc:244nn:241:40digitalmars.com Is that design decision written in some page of the D wiki? Do you need to add a "virtual" keyword to D?

(In reply to comment #9) > (In reply to comment #7) > > > AFAIK Walter agreed for final-by-default: > > > > http://forum.dlang.org/thread/[email protected]?page=28#post-koqkhc:244nn:241:40digitalmars.com > > Is that design decision written in some page of the D wiki? > Do you need to add a "virtual" keyword to D? I don't know the details. I guess someone should make a DIP and mark it as "approved".

(In reply to comment #9) > (In reply to comment #7) > > > AFAIK Walter agreed for final-by-default: > > > > http://forum.dlang.org/thread/[email protected]?page=28#post-koqkhc:244nn:241:40digitalmars.com > > Is that design decision written in some page of the D wiki? > Do you need to add a "virtual" keyword to D? Not sure. Don't think so. I think most of the conversation is in that thread (and prior ones like it), and verbally at dconf... :/ Yes, a virtual keyword would be required. override would cause a warning at first if used on a function that wasn't either override or virtual, and then it would become an error. final would be used to seal a hierarchy, as is useful in java, C++ and 'sealed' in C#.

(In reply to comment #8) > (In reply to comment #6) > > (In reply to comment #5) > > > I don't think there is. That would have been far too sensible :) > > > > Well, if you can remember where the discussion and agreement was, it might be a > > good idea to make one and link it. > > IIRC, it was actually you that proposed the winning deprecation path. > It was comparable to the introduction of mandatory override. I think you > suggested one version introduces a deprecation warning where override is stated > but virtual is not present on the base, then the next version enforces it. I eventually realized that override implies virtual, and therefore only the introducing methods need to be modified. Up until that point I was pretty skeptical. I've created an enhancement for it: issue 11616 Feel free to CC yourselves if you're interested.

(In reply to comment #12) > (In reply to comment #8) > > (In reply to comment #6) > > > (In reply to comment #5) > > > > I don't think there is. That would have been far too sensible :) > > > > > > Well, if you can remember where the discussion and agreement was, it might be a > > > good idea to make one and link it. > > > > IIRC, it was actually you that proposed the winning deprecation path. > > It was comparable to the introduction of mandatory override. I think you > > suggested one version introduces a deprecation warning where override is stated > > but virtual is not present on the base, then the next version enforces it. > > I eventually realized that override implies virtual, and therefore only the > introducing methods need to be modified. Up until that point I was pretty > skeptical. Yes, it's a much less damaging change than the introduction of 'override', since it only affects the base, rather than all branches/leaves as override did. And now that we have override, introduction of virtual is perfectly safe, we can properly produce accurate warnings/errors where it's missing. > I've created an enhancement for it: issue 11616 > Feel free to CC yourselves if you're interested. I'm just creating a DIP for it... should I bother?

(In reply to comment #13) > > I'm just creating a DIP for it... should I bother? If you want to. I don't think it is necessary though.

(In reply to comment #14) > (In reply to comment #13) > > > > I'm just creating a DIP for it... should I bother? > > If you want to. I don't think it is necessary though. Well if there's no need, I won't. I'm just trying to remember if there were any fine-details that we covered that are worth noting on there. The only detail I remember debate over, was the handling of 'final' for sealing hierarchies; ie, do you need 'override final', or is 'final' enough when specified on a method whose base is virtual? I think 'final' is enough, but it was an argument of clarity.

Addressing Manu problem in first post: you cannot do it directly, but can workaround by void foo(T)(ref const T i) {} ref lvalueOf(T)(T expression) { return [expression][0]; } struct S(T) { T t; ref get() { return t; } //alias get this; } void main() { //foo(0-0); foo((0-0).lvalueOf); foo(S!int(0-0).get); } Second one is ugly but does not consume heap memory as first one, but it also requires that foo() does not save reference somewhere (operation is safe because S!int(0-0) is allocated within main scope, so passing pointer is fine, until it is saved).

(In reply to comment #16) > Addressing Manu problem in first post: you cannot do it directly, but can > workaround by > > void foo(T)(ref const T i) {} > > ref lvalueOf(T)(T expression) > { > return [expression][0]; > } > > struct S(T) > { > T t; > ref get() { return t; } > //alias get this; > } > > void main() > { > //foo(0-0); > foo((0-0).lvalueOf); > foo(S!int(0-0).get); > } > > Second one is ugly but does not consume heap memory as first one, but it also > requires that foo() does not save reference somewhere (operation is safe > because S!int(0-0) is allocated within main scope, so passing pointer is fine, > until it is saved). One allocates, the other is just a subversion of the type system. It's basically the same (and less clear?) than just doing: int temp = 0-0; foo(temp); Either way, it's equally unsafe, and the discussion regarding rvalue->ref parameters was mainly around safety concerns. I was in the camp arguing for a 'scope ref' concept, which would rely on escape analysis to enforce that a 'scope ref' doesn't escape, but we were never going to win that argument. The plan that Walter was most enthusiastic about was, if the function is passed an rvalue, generate an implicit temp in the caller's stack and pass it through. If the function being called returns a ref, it would add an implicit function exit condition which would validate that the ref being returned is not within the function's stack (controlled by -noboundscheck for optimisation). This way, it would be safe to return a ref that was passed in to a function, but not a function local. This would cascade outwards. Frankly, I just want a solution. I like the 'scope ref' concept, but it depends on escape analysis working well, and I'm not that precious about implementation, I just want something that works. It's a major inhibitor to maths intensive code, like image/geometry/physics/audio processing. Particularly where fairly 'primitive' objects like vectors, matrices, quaternions, colours, etc are concerned.

(In reply to comment #17) > > One allocates, the other is just a subversion of the type system. Actually each of them allocates. You either allocate on stack or in heap. Advantage of heap is that you don't care about escaping references and advantage of stack is efficiency. Second example is obviously not a subversion of the type system. > It's > basically the same (and less clear?) than just doing: > int temp = 0-0; > foo(temp); > No, it is not the same. By the way, it was you who complained about writing code without dummy variables. > Either way, it's equally unsafe, and the discussion regarding rvalue->ref > parameters was mainly around safety concerns. > No, first is safe, second is conditionally safe if reference does not escape. By the way, you wrote that creating temporary and passing pointer is acceptable (either you do this or language automatically does). > If the function being called returns a ref, it would add an implicit function > exit condition which would validate that the ref being returned is not within > the function's stack (controlled by -noboundscheck for optimisation). > This way, it would be safe to return a ref that was passed in to a function, > but not a function local. This would cascade outwards. > The idea has flaw, it does not take into account that there are ways for stack pointer to escape other than through return. > Frankly, I just want a solution. I like the 'scope ref' concept, but it depends > on escape analysis working well, and I'm not that precious about > implementation, I just want something that works. It's a major inhibitor to > maths intensive code, like image/geometry/physics/audio processing. > Particularly where fairly 'primitive' objects like vectors, matrices, > quaternions, colours, etc are concerned. Frankly, there is excess of people wanting a solution and shortage of people executing them. I think you need to decide what exactly do you want: allocating temporary by language (which is unlikely to happen), using workaround with the purpose not to have dummy variables (seems to be no), writing DIP for the issue (which having been written and discussed goes to archive), or somebody to fix scope ref (which is unlikely to happen soon).

(In reply to comment #17) > I was in the camp arguing for a 'scope ref' concept, which would rely on escape > analysis to enforce that a 'scope ref' doesn't escape, but we were never going > to win that argument. > > The plan that Walter was most enthusiastic about was, if the function is passed > an rvalue, generate an implicit temp in the caller's stack and pass it through. > If the function being called returns a ref, it would add an implicit function > exit condition which would validate that the ref being returned is not within > the function's stack (controlled by -noboundscheck for optimisation). > This way, it would be safe to return a ref that was passed in to a function, > but not a function local. This would cascade outwards. Related ref safety Issue 9537. I don't think you need to win the 'scope ref' issue because the -noboundscheck version will work fine. Then temporaries will be safe, so long as the called functions don't save their addresses off of the stack somewhere. To ensure safety, I think the compiler needs to keep track of some bits on its parameters and expressions. Some ideas: Each reference variable has a 'mayBeLocal' bit and a 'mayBeRefParam' bit. Each function has a 'mayReturnRefParam' bit. All known references to locals have 'mayBeLocal' set, and all ref parameters have 'mayBeRefParam' set. Any attempt to return a reference to a 'mayBeLocal' causes the appropriate error. Returning a 'mayBeRefParam' reference causes the function to be marked 'mayReturnRefParam', and the result will be the safest of what was passed in. Any time the compiler can't prove that a reference parameter will not be returned, it marks the function as 'mayReturnRefParam' and requires the runtime stack bounds check. Now ref returns are safe. The above scheme is blunt. It will unnecessarily restrict a few cases where more than one ref parameter is passed, but only one is returned: ref T fun(ref T a, ref T b) { return a; } ref T gun(ref T a) { T b; return fun(a, b); // fails unnecessarily, because b is local } If there were a lot of such cases in practice, some addition to this simple scheme may be necessary. Other kinds of unsafe escapes would have to be handled in other ways (e.g. a bit 'mayGetSavedOffStack' on parameters). But the main case of ref returns is handled at least. Then rvalue refs would be safe.

(In reply to Manu from comment #0) > I've heard arguments about the safety of the operation, but the 'workaround' > is just to create a temporary, which has identical security properties. > ... > Perhaps this behaviour could be restricted to ref const, or ref in, if we're > getting worried about the safety of the operation? That would perhaps even > improve on how behaves now. I think rvalues could be passed to const ref parameters so long as that parameter is not return ref. The reason is in case we decide to support local refs initialized from a return ref function.

I came up with a lvalueOf!rvalue template which hopefully is memory-safe. Not tested with Walter's DIP1000 scope branch (yet). enum typeof(v[0])[v.length] staticArray(v...) = [v]; ref const(T) lvalueOf(alias rvalue, T=typeof(rvalue)) (return ref T rv = *staticArray!rvalue.ptr) @trusted { return rv; } auto foo(ref const int i) {assert(i == -1);} @safe @nogc unittest { static assert(!__traits(compiles, foo(0-1))); foo(lvalueOf!(0-1)); }

Resolved by https://github.com/dlang/dmd/pull/9817