The following code produce some weird behavior when compiled in 32bits on my Windows 7 64b machine. Removing the parallel or compiling in 64b removes the problems.
import std.md5;
import std.stdio;
import std.file;
import std.conv;
import std.getopt;
import std.string;
import std.process;
import std.parallelism;
import std.exception;
struct Entry
{
string name;
ubyte[16] md5;
}
void main(string[] args)
{
string folder1 = args[1];
string folder2 = args[2];
Entry[] entries1;
foreach (name; parallel(dirEntries(folder1, SpanMode.breadth), 1)) // not working
{
if(name.isFile())
{
entries1 ~= Entry(name ,mdFile(name));
}
}
writeln(folder1," has ",entries1.length, " entries");
Entry[] entries2;
foreach (string name; dirEntries(folder2, SpanMode.breadth)) //working fine
{
if(name.isFile())
{
entries2 ~= Entry(name ,mdFile(name));
}
}
writeln(folder2," has ", entries2.length, " entries");
}
/// Digests a file and prints the result.
ubyte[16] mdFile(string filename)
{
MD5_CTX context;
ubyte[16] digest;
context.start();
File f = File(filename,"r");
foreach (buffer; f.byChunk(4096))
context.update(buffer);
context.finish(digest);
try{
f.close();
}
catch(ErrnoException e)
{
writeln(e.errno);
}
writefln("MD5 (%s) = %s", filename, digestToString(digest));
return digest;
}
When executed, the program starts opening files and calculating their MD5. After a short amount of files have been scanned, depending on the WorkUnitSize (between 2 and 10 if set to 1, if default to 100, around 30-40 are processed), the program begins malfunctioning
- Seemingly random files throw an ErrnoException (error n° 0) when closed (not interrupting the rest of the program)
- The program sometime just freeze (need to close it manually)
- The program generates "Object.Error: Access violation"
- The program crashes (program.exe has stopped working).
The culprit seems to be the function byChunk used to calculate the MD5.
Using std.stream like suggested in this question's answer ( http://stackoverflow.com/questions/20888596/opening-files-in-a-parallel-loop-crash-the-program ) makes the program work in 32 and 64bits.
While this issue can be fixed by compiling in 64bit, it's still a major problem.
PS : It's my first time reporting a bug, so if some information is lacking or I've submitted this wrong, please let me know and I'll try to fix it.
Comment #1 by kozzi11 — 2014-01-03T13:11:11Z
Problem seems to be in T[] rawRead(T)(T[] buffer)
version(Win32)
{
immutable fd = ._fileno(_p.handle);
//immutable mode = ._setmode(fd, _O_BINARY); when comment this line
//scope(exit) ._setmode(fd, mode); and this line it seems to works much better
version(DIGITAL_MARS_STDIO)
{
// @@@BUG@@@ 4243
immutable info = __fhnd_info[fd];
__fhnd_info[fd] &= ~FHND_TEXT;
scope(exit) __fhnd_info[fd] = info;
}
}
Comment #2 by nick — 2023-01-14T21:36:08Z
Note that writing to `entries1` here causes a race condition:
```
Entry[] entries1;
foreach (name; parallel(dirEntries(folder1, SpanMode.breadth), 1)) // not working
{
if(name.isFile())
{
entries1 ~= Entry(name ,mdFile(name));
```
See also issue 15129.
Comment #3 by robert.schadek — 2024-12-01T16:19:48Z