Bug 2254 – Size of executable almost triples

I recently had to fix a little bug in an D application (win32). I thought that it would also be a good idea to upgrade to the most current version of the D compiler. When upgrading to 1.033 from 1.025 the executable size almost tripled from 227 kb to 603 kb! After building with all version from 1.025 up I found that this phenomenon occurs when switching from 1.031 to 1.032. I did not change the compiler options (-release, -o) when building with several dmd versions. I saw the many changes/bugfixes from 1.031 to 1.032. However, can that explain this behaviour?

1.032 is the first version where phobos.lib is built using the new -lib switch. That seems the likely culprit. http://www.dsource.org/projects/phobos/changeset/811 I tried building hello world tests with dmd 1.031 and 1.032 to see the size differences: Hello world with writefln (import std.stdio): 1.031: 114.4 k 1.032: 269.5 k Hello world with printf (no imports): 1.031: 78.0 k 1.032: 121.5 k So something bad definitely happened between 1.031 and 1.032. Now, if someone could be bothered to try building phobos 1.032 without the -lib switch and compare, that would be great.

Comparing the two map files shows the difference comes from the first line: 0002:00000000 000193FBH _TEXT CODE 32-bit vs 0002:00000000 0000CC9BH _TEXT CODE 32-bit

(In reply to comment #1) > So something bad definitely happened between 1.031 and 1.032. Now, if someone > could be bothered to try building phobos 1.032 without the -lib switch and > compare, that would be great. Yep, that did the trick. I've used the v1.031 makefile to build phobos from v1.032, using v1.032. Both phobos.lib and the built .exe file have the size from v1.031. It's definately the -lib switch that's causing the bloat.

I rebuilt my application with DMD 1.034 (where it should have been fixed) and it has the same effect as before. It still triples. Can someone (maybe [email protected]) check that.

i can confirm that it even grew again from 2.017 to 2.018

Fixed dmd 1.034 and 2.018

I tried my hello world tests from comment #1 with dmd 1.034. The file sizes are the same or slightly larger. I'm not sure what fix Walter has made, but the original issue is still there.

I just want to comment: The more symbols, the more exports you have...the more file size you get. What i hate the most is that when you execute your program it takes even more memory. I did a simple Direct3D program that initialize device and shows absolutely no stuff, just a blank screen. This program uses 9.5 MB in memory, and it does nothing!

This issue still persist even in the latest versions (it gets more serious with every new version, file size grows even more). I think this is THE bug that prevents newer versions to become the most stable ones in the download page (1.030 has been for a long time) and this bug has been reported a long time ago. I've been thinking in dropping development for now. We are developing a commercial game engine, we use newer versions of DMD1 because it solves many issues that were bothering us, we are fine coding and very happy using D templates but running the engine...it's counter productive, it's using excesive system memory and we're gonna need a huge distribution media if it keeps like this. Walter, i hope you read this, you're the most adecuate person to solve this. This issue must be addresed now before it compromises the DMD development achieved so far. I found out one more thing, one more symptom: Create the most basic hello world program and compile it. Check how large the file is. Now, import as many phobos modules as you can. Don't use any module! just import them. Do this with about 8 modules. Compile and recheck file size...it grews in an extreme manner and not even using new symbols. That's it. All i can do is report and i hope this post change something. Thanxs in advance.

Since I assume this is a long-term bug now that wrapping up D 2 has the main focus, I'll post some tips and tricks. If you are using D 1, you'll probably have better luck with Tango. A hello world program built with Tango is about half the size. If you really want to use phobos you can custom build it, see comment #1 and onwards. I wonder what platform you're on if saving a few hundred kB of disk space is important, though. When that's said, if you use other libraries written in D, make sure you build in one of two ways: A) Build everything in one go, don't use .lib files. Except for tango/phobos and C libs. This means using a build tool or script that hands all files to dmd at once. Very fast and simple build process. Supported by xfbuild/bud/dsss build tools. B) Build .lib files of your library dependencies first, by handing one file at a time to dmd, then running lib.exe to create a .lib file. Supported by DSSS, and possibly xfbuild. Or you could use a makefile or whatever you prefer. The *WRONG* way to build is as follows: C) Build .lib files by handing all files to dmd at once. Then build your app (any way you like) and link to those .lib files. Unless your exe uses absolutely everyting defined in those libs, this causes a bloated executable. You end up getting more then a wanted, because this way of building causes fake dependencies between modules. If you want try A, I suggest having a look at xfbuild, it's very fast and flexible. Ask in the newsgroup D.learn if you need help with setting it all up. http://bitbucket.org/h3r3tic/xfbuild/wiki/Home

The D app is used in production, so rebuilding with tango and/or D2 is simply too risky. And yes, in our scenario size matters because the application gets deployed and updated via the internet. For DSL that wouldn´t be a problem but there are a lot of users that have slower connecions e.g. GPRS mobile connections. However, thanks for the advice of building the app without the lib - plan A I will give it a try.

This problem has just worsened in the last two releases of D2. Executable size of writefln("Hello world"); by version is: 1.00->1.031: 100K-113K. 1.032: 276K 1.034 and later: 324K-340K. 2.00-2.025: 114K-130K 2.027: 189K 2.030-2.047: 217K - 230K 2.048: 264K 2.049: 305K

Please vote for this issue. It may give a higher priority and Walter might take a look at this long term bug.

Those who are affected by this, be sure to check bug 4767. Can pull in unrelated and unused modules from the standard library. There's also bug 3214 which can double the size of debug executables.

I just discovered some interesting results when compiling with -lib and unittests. See bug 5660. -lib definitely needs some work. I hope this is something easy to fix, because the perception is not good for outsiders.

(In reply to comment #15) ... > See bug 5660. Since bug 5660 doesn't exist (yet), I assume you mean bug 5560.

(In reply to comment #16) > (In reply to comment #15) > ... > > See bug 5660. > > Since bug 5660 doesn't exist (yet), I assume you mean bug 5560. Yes, thank you (wishes for comment editing...)

(In reply to comment #17) > (In reply to comment #16) > > (In reply to comment #15) > > ... > > > See bug 5660. > > > > Since bug 5660 doesn't exist (yet), I assume you mean bug 5560. > > Yes, thank you (wishes for comment editing...) (In reply to comment #3) > (In reply to comment #1) > > So something bad definitely happened between 1.031 and 1.032. Now, if someone > > could be bothered to try building phobos 1.032 without the -lib switch and > > compare, that would be great. > > Yep, that did the trick. I've used the v1.031 makefile to build phobos from > v1.032, using v1.032. Both phobos.lib and the built .exe file have the size > from v1.031. It's definately the -lib switch that's causing the bloat. Using V1.030's phobos.lib instead of V1.066, the "hello world" exe size is 116KB, so the issue is all about phobos.lib Using V1.068, the original size of phobos.lib is 1545KB, trying rewrite win32.mak(using lib.exe to make phobos.lib) to get new phobos.lib is 951KB, the "hello world" exe size is 123KB

Hello world on Darwin with writefln(), -release: 1.068: 272 kb 2.053: 954 kb (!) Hello world on Darwin with writefln(), -release, strip: 1.068: 252 kb 2.053: 894 kb (!)

Created attachment 981 Size of phobos.lib across D1 versions

Created attachment 982 Size of "Hello, world" .exe across D1 versions

(In reply to comment #21) > Created an attachment (id=982) [details] > Size of "Hello, world" .exe across D1 versions What's happened in DMD 1.032 and 1.035 to cause that size increase? Is something similar happened to DMD2 too?

As (In reply to comment #22) > (In reply to comment #21) > > Created an attachment (id=982) [details] [details] > > Size of "Hello, world" .exe across D1 versions > > What's happened in DMD 1.032 and 1.035 to cause that size increase? > Is something similar happened to DMD2 too? As Walter always has said, compile something and look at the map file to see what's pulled into the executable.

Created attachment 983 Treemap of map file of D 2.053 "Hello, world" (In reply to comment #23) > As Walter always has said, compile something and look at the map file to see > what's pulled into the executable. Map files are useless. Look at this treemap. Most functions in the large rectangles contain very little code.

Created attachment 984 D 2.053 "Hello, world" executable opened in a file viewer Look at this instead. There are many large areas in the executable filled with zero bytes. If you compress the executables produced by a DMD version before and after this regression, the compressed sizes will differ by very little.

I suggest to compile a hello world with 1.031 and 1.032 and then compare in some way their map files, and show the differences...

(In reply to comment #26) > I suggest to compile a hello world with 1.031 and 1.032 and then compare in > some way their map files, and show the differences... What do you hope to achieve by doing that? As I already said, this will not answer any questions. This is not a dependency bloat problem! How would you do this, anyway? Those old DMD versions don't even have a -map switch - the map file generated by OPTLINK by default is useless, it only lists segments.

Created attachment 985 Side-by-side comparison of cv2pdb+dumpbin-generated map files of DMD 1.031 and 1.032 "Hello, world" This is interesting, but for unexpected reasons: the 5K blocks of fluff have no debugging names. Could be a cv2pdb issue, or something else.

The problem is caused by DMD's -lib option, used since 1.032 to build Phobos. Previous versions used the Digital Mars librarian (lib.exe).

(In reply to comment #28) > the 5K blocks of fluff have no > debugging names. Could be a cv2pdb issue, or something else. Thank you for your work. From the comparison it seems those 5K blocks of fluff are the main cause of binary size increase.

Actually, I should add that the problem might not be with DMD's -lib option. While it certainly is a completely different method of generating phobos.lib, I should note that: 1) The 5K-ish blocks of zeroes are not present in phobos.lib (although we do see a size increase in the chart I attached earlier) 2) UniLink will happily link D's .obj files + phobos.lib and create an executable without these blocks of zeroes (and much smaller size). Thus, the problem could very well be in OPTLINK. So, here's your workaround for the time being: use UniLink. Someone posted the config on the NG, I'll repost it here for convenience (ulink.cfg): -zsnn.lib -LC:\dmd\windows\lib -LC:\dm\lib -Go Adjust paths as necessary.

Is this only a problem on Windows?

For D1 1.073, import std.stdio; void main() { writefln("hello world"); } generates an executable sized 117,276. For D2 2.058, import std.stdio; void main() { writeln("hello world"); } generates an executable sized 146,972. All on Windows compiled with -O -release -inline. I think these sizes are reasonable, especially considering all that the runtime library is expected to do (threads, gc, static construction, etc.).

There are several things at play here, such as Andrei's recent efforts at reducing cross-module dependencies in the case of "Hello, world" programs. However, I'm not seeing the 5K-ish blocks of zero bytes that were present in 2.057. I assume that the underlying DMD -lib bug (or wherever it was) has been fixed. (In reply to comment #33) > For D2 2.058, > > import std.stdio; void main() { writeln("hello world"); } > > generates an executable sized 146,972. For the record, UniLink generates an executable sized 134,752.

(In reply to comment #34) > However, I'm not seeing the 5K-ish blocks of zero bytes that were present in > 2.057. I assume that the underlying DMD -lib bug (or wherever it was) has been > fixed. > Looks like these blocks are related to Issue 7921.

If you're seeing large blocks of zero bytes, make sure first that it is not this issue: http://dlang.org/faq.html#bss