Bug 8993 – Implement unique references/isolated memory

'unique' or 'isolated' is a recursive type qualifier which enforces that there is only a single reference to a given piece of memory, and that the memory itself is only pointing to other data that is either immutable, or only referenced within the same memory cluster. Any value/reference that is tagged this way can safely be converted to immutable or can safely be passed (moved) to another thread. In conjunction with (weakly) pure functions it is possible to operate almost freely on a unique value, because it can be guaranteed that no visible additional references can be created. The main advantages of this approach are: - Provides a powerful alternative to safely pass mutable data between threads without requiring locks or synchronization primitives (synchronized/shared) - Construction of immutable objects can be done in a natural procedural fashion and not only within the object constructor or within a pure function - Fully compatible with the current type system, builds nicely on top of the current base - The system is proven both in theory and in practice Microsoft has released a paper describing the system in form of an extension to C#. The paper includes formal proofs for the soundness of the system and mentions a multi-million lines of code project for which it is used: http://research.microsoft.com/pubs/170528/msr-tr-2012-79.pdf D specific and in addition to the Microsoft paper, a solution to support partitioning of a uniqe array with unique slices as the result is desirable. But this can be done as a simple library function if needed.

I think it's a bit too late to add new keywords to the language, other than ones prefixed with "__", and we certainly wouldn't want something this useful to be subject to such annoying-to-type keywords... This seems like a thing for D3 *if* D3 ever becomes a thing.

Writing a detailed DEP requires lot of work, but maybe for a request this complex it's unavoidable.

(In reply to comment #1) > I think it's a bit too late to add new keywords to the language, other than > ones prefixed with "__", and we certainly wouldn't want something this useful > to be subject to such annoying-to-type keywords... > > This seems like a thing for D3 *if* D3 ever becomes a thing. Even with __ it would be better than nothing. I think it's almost safe to say that current D is almost unusable in conjunction with immutable/shared (objects) and that's ironically one of the areas where it tries to shine. I have tried numerous times to cleanly reimplement certain multithreaded functionality using it and _never_ got it running without just so many casts that the result was not only as unsafe/unchecked as the original, but also unreadable und badly usable. I apologize in advance for the following (general) rant: The attitude of implementing ad-hoc solutions, not listening to valid objections and then declaring the language as stable is a very foolish development approach. The way shared was introduced was not the only thing in the past that went this way. Things like these made me really frustrated with D multiple times, to the point where I was seriously looking out for alternatives - the end result was pure disregard/disinterest for the corresponding language parts. And I think that there is quite a number of people who just gave up on either the language or on certain parts, and the resulting silence creates a dangerous deceived feeling of statisfaction. The current mode of development is shizophrenic, stabilizing an obviously unfinished language and, from time to time, introducing new, breaking things anyway. There absolutely needs to be a way to still let innovation and refinement happen, be it in a separate branch (called 3.x or something else doesn't matter) or in the form of allowing new features as long as they are backwards-compatible (e.g. using the __ way). Of course the constant breakage of the past is bad, but stagnation at this point could possibly be the start of a slow, starving death. Sorry again, but I've seen D make so many mistakes in the past and that it has done a lot more things right is no excuse for that.

Yeah, no doubt shared is a disaster in general (I have posted several rants about this in particular on the NG). We're effectively doing a lot of false advertising with that type qualifier in particular, and people are writing code as if it results in atomic operations (and then their code will break on !x86). And immutable/const are only realistically usable with non-class types. I feel you're spot on with the point about perceived satisfaction. But it's not me you have to convince. It's Walter and Andrei...

(In reply to comment #3) > I apologize in advance for the following (general) rant: I agree with what you say.

(In reply to comment #5) > I agree with what you say. But in the end this enhancement request is not enough. A detailed DEP is probably necessary. You don't have to write it down all yourself, of course. Once the DEP is written, you will be in a better position to rant :-)

Just want to chime in to say that I 100% agree with Sönke and Alex.

(In reply to comment #6) > (In reply to comment #5) > > > I agree with what you say. > > But in the end this enhancement request is not enough. A detailed DEP is > probably necessary. You don't have to write it down all yourself, of course. > Once the DEP is written, you will be in a better position to rant :-) Agreed, a DIP is probably in order for this change. It would probably be enough to concentrate on the simple parts first (implicit isolated->immutable conversion and free psasing of isolated values between threads) and then implement further relaxations bit by bit (e.g. implicit scoped isolated->writable->isolated conversion). Allocating time for a proper DIP is difficult for me currently, but I guess it could be marked as work-in-progress in the wiki so that it can be completed step by step?

(In reply to comment #8) > but I guess it could be marked as work-in-progress > in the wiki so that it can be completed step by step? I think this is OK. In the paper they say the Microsoft programmers have implemented that system one piece at a time. So take a look at those implementation steps (avoiding their eventual mistakes).

(In reply to comment #9) > (In reply to comment #8) > > > but I guess it could be marked as work-in-progress > > in the wiki so that it can be completed step by step? > > I think this is OK. > > In the paper they say the Microsoft programmers have implemented that system > one piece at a time. So take a look at those implementation steps (avoiding > their eventual mistakes). Oh, I meant more the process of writing the actual DIP in that sentence (I never did anything on the wiki and don't know the usual process for editing a DIP). So my plan was basically to only include the first step of the implementation into the DIP and then file the rest as enhancement requests should it get implemented at some point.

I recommend you take a look at some blog posts from Bartosz Milewski. He was involved for a while in the D community and had some ideas about race-free multithreading with the help of a type system supporting using "unique" and "lent". Search in the newsgroups archives for "Bartosz Milewski". http://bartoszmilewski.com/2009/05/26/race-free-multithreading/

(In reply to comment #11) > I recommend you take a look at some blog posts from Bartosz Milewski. He was > involved for a while in the D community and had some ideas about race-free > multithreading with the help of a type system supporting using "unique" and > "lent". Search in the newsgroups archives for "Bartosz Milewski". > > http://bartoszmilewski.com/2009/05/26/race-free-multithreading/ I've read those back then. Bartosz' system was very powerful but unfortunately complex to implement and had some implications on performance. The good thing with the MS approach is that it accomplishes an important part of it without requiring explicit owner tracking. It surely doesn't fix 'shared' the way that Bartosz' system would, but at least it alleviates the need for it in quite some places.(In reply to comment #11)

(In reply to comment #8) > implement further relaxations bit by bit (e.g. implicit scoped > isolated->writable->isolated conversion). To clarify for those who didn't read the paper, the implicit conversion of writable->isolated only applies to the return value of a moderately pure function: (a pure function whose arguments are only isolated or immutable) So it is similar to the implicit conversion of the return of a strongly pure function from mutable to immutable.

See also: https://github.com/D-Programming-Language/dmd/pull/1700 http://wiki.dlang.org/DIP29

Hi, I believe that today enhancement requests to the language itself need to be presented as a DIP: https://github.com/dlang/DIPs I see from bearophile's comment that a DIP has already been submitted. As such, and also seeing that this was filed over 4 years ago and the discussion lasted just one day, I'm going to close this, but feel free to reopen if you disagree.