"fatal error C1060: compiler is out of heap space" for lib/edgepaint/lab_gamut.c

magjac · June 16, 2020, 6:04pm

I’m so close (but no cigar) to having a Windows pipeline with all 6 variants of builds, but there’s one tiny (or HUGE) problem left.

It’s the GitLab runners that runs out of memory for the 10MB (800k+ lines) file lib/edgepaint/lab_gamut.c using msbuild. The whole project works fine when I compile it on my 12G RAM laptop. The Shared Windows runners have only 7.5 G.

The error message is:

fatal error C1060: compiler is out of heap space.

I think it’s the difference in available memory that is the problem but an alternative explanation is that it is this problem that was reported fixed Aug 12, 2019. I have Visual Studio Build Tools version 14.26.28801 while the GitLab runner has 14.24.28314. I haven’t been able to find release notes for the build tools so I can’t say if the GitLab runners have this fix or not and I don’t know how to install the GitLab version on my computer.

I’ve made an experimental version of lab_gamut.c, reducing it to just a few lines and GitLab had no problems with it, so I know it’s the size of the file (or rather the size of the data I guess) that is the problem.

Here’s the pipeline with all build jobs, but for some reason it doesn’t show the detailed error message, just Done Building Project "C:\GitLab-Runner\builds\magjac\graphviz\lib\edgepaint\lab_gamut.vcxproj" (default targets) -- FAILED..

Here is a job that builds lab_gamut only that shows the error message.

I’ve read Fatal Error C1060, but got no help from it.

Any help would be greatly appreciated: to pinpoint the problem better, to avoid it or ways to rewrite the file that would consume less memory. It looks like a generated file. Maybe it would be possible to generate it at runtime if we knew how. Perhaps @erg knows?

magjac · June 16, 2020, 7:41pm

I now know it’s not the version difference that is the problem. I managed to install the build tools version 14.24.28314.0 on my laptop through these instructions and it compiled fine.

scnorth · June 16, 2020, 9:50pm

I wonder if someone in GitLab could help: Launch Windows Shared Runner beta (#4815) · Issues · GitLab.org / gitlab-runner · GitLab

smattr · June 16, 2020, 11:36pm

The build log indicates the compiler lives under “C:\Program Files (x86)”. Does that mean this is the 32-bit version of the compiler? In which case, maybe it’s limited to 4GB of RAM. OTOH maybe /3GB will help.

Another guess: maybe there’s something like ulimit on the Gitlab runner that’s artificially limiting how much memory the compiler can allocate.

If we can’t find another work around, I think lab_gamut.c could be refactored into a large string, similar to what is produced by xxd -i, that presumably compilers are more accustomed to seeing.

mark · June 17, 2020, 1:23am

+1 this sounds like a 32/64 bit issue to me. I still think it’s pretty wild that it takes that much memory to compile a 10MB source file though. What the heck is it doing?

smattr · June 17, 2020, 1:58am

C/C++ compilers often aren’t great at large arrays. E.g. GCC has some variant of the same problem.

smattr · June 17, 2020, 2:06am

Just realized I accidentally hit something related to this a week back

smattr · June 17, 2020, 2:45am

Out of curiosity, I ran a little (unscientific) experiment on Linux:

$ gcc-7 --version
gcc-7 (Debian 7.4.0-6) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ /usr/bin/time -f 'system time: %Ss, user time: %Us, RSS: %MKB' gcc-7 -I$(pwd) -c -o /dev/null lab_gamut.c
system time: 0.26s, user time: 4.47s, RSS: 295696KB
$ /usr/bin/time -f 'system time: %Ss, user time: %Us, RSS: %MKB' gcc-7 -O3 -I$(pwd) -c -o /dev/null lab_gamut.c
system time: 0.20s, user time: 4.50s, RSS: 297200KB

$ gcc-8 --version
gcc (Debian 8.3.0-6) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ /usr/bin/time -f 'system time: %Ss, user time: %Us, RSS: %MKB' gcc-8 -I$(pwd) -c -o /dev/null lab_gamut.c
system time: 0.61s, user time: 4.30s, RSS: 394996KB
$ /usr/bin/time -f 'system time: %Ss, user time: %Us, RSS: %MKB' gcc-8 -O3 -I$(pwd) -c -o /dev/null lab_gamut.c
system time: 0.32s, user time: 4.61s, RSS: 396288KB

$ cat - >helloworld.c <<EOT
> #include <stdio.h>
> int main(void) {
>   printf("hello world\n");
>   return 0;
> }
> EOT
$ /usr/bin/time -f 'system time: %Ss, user time: %Us, RSS: %MKB' gcc-7 -I$(pwd) -c -o /dev/null helloworld.c
system time: 0.01s, user time: 0.00s, RSS: 18660KB
$ /usr/bin/time -f 'system time: %Ss, user time: %Us, RSS: %MKB' gcc-7 -O3 -I$(pwd) -c -o /dev/null helloworld.c
system time: 0.01s, user time: 0.04s, RSS: 21480KB
$  /usr/bin/time -f 'system time: %Ss, user time: %Us, RSS: %MKB' gcc-8 -I$(pwd) -c -o /dev/null helloworld.c
system time: 0.00s, user time: 0.01s, RSS: 18416KB
$ /usr/bin/time -f 'system time: %Ss, user time: %Us, RSS: %MKB' gcc-8 -O3 -I$(pwd) -c -o /dev/null helloworld.c
system time: 0.00s, user time: 0.02s, RSS: 21800KB

A quick hack to convert the large array to an xxd-style string yields the following:

$ /usr/bin/time -f 'system time: %Ss, user time: %Us, RSS: %MKB' gcc-7 -I$(pwd) -c -o /dev/null lab_gamut2.c
system time: 0.05s, user time: 0.18s, RSS: 52480KB
$ /usr/bin/time -f 'system time: %Ss, user time: %Us, RSS: %MKB' gcc-7 -O3 -I$(pwd) -c -o /dev/null lab_gamut2.c
system time: 0.07s, user time: 0.17s, RSS: 50864KB

$ /usr/bin/time -f 'system time: %Ss, user time: %Us, RSS: %MKB' gcc-8 -I$(pwd) -c -o /dev/null lab_gamut2.c
system time: 0.05s, user time: 0.18s, RSS: 51320KB
$ /usr/bin/time -f 'system time: %Ss, user time: %Us, RSS: %MKB' gcc-8 -O3 -I$(pwd) -c -o /dev/null lab_gamut2.c
system time: 0.03s, user time: 0.20s, RSS: 52748KB

magjac · June 17, 2020, 9:38pm

On the Windows I’m currenty using there is no /usr/bin/time so I’m relying on the built-in time of MINGW/MSYS bash which does not show memory usage, but I used the Task Manager while it was running and it peaked just above 3G:

Time Elapsed 00:01:13.39

real    1m13.722s
user    0m0.000s
sys     0m0.015s

The command thas was run was:

msbuild -target:lab_gamut graphviz.sln

@smattr If you could supply you quick hack for me as a starting point, I’d appreciate it. If you want do do the whole implementation yourself, I’d appreciate it even more. Swedish midsummer is approaching (Friday is the big day) and the activities that we tend to engage in then does not lend themselves to be combined with programming unfortunately so I’m going to stay off the computer for a couple of days anyway.

smattr · June 17, 2020, 11:10pm

Ah yes, I don’t think I recall seeing any computers in Midsommar

I can’t figure out how to attach a file here, so I posted it on Gitlab: lab_gamut hack ($1987854) · Snippets · GitLab. Implementing a proper version of it wouldn’t be too much work, but we ship lab_gamut, both as a library and with a header that declares the array. I.e. doing this would be an API- and ABI-breaking change. I don’t know how controversial this is considered for Graphviz.

scnorth · June 17, 2020, 11:39pm

What?

I doubt we intend to export this value outside graphviz.

smattr · June 18, 2020, 12:11am

Sorry I was mistaken about the header, but the library is there:

$ git clone https://gitlab.com/graphviz/graphviz
...
$ cd graphviz
$ ./autogen.sh && ./configure --prefix /tmp/tmp.bamfQudTlG && make && make install
...
$ find /tmp/tmp.bamfQudTlG -iname "*lab_gamut*"
/tmp/tmp.bamfQudTlG/lib/liblab_gamut.so
/tmp/tmp.bamfQudTlG/lib/liblab_gamut.so.1
/tmp/tmp.bamfQudTlG/lib/liblab_gamut.so.1.0.0
/tmp/tmp.bamfQudTlG/lib/liblab_gamut.la
/tmp/tmp.bamfQudTlG/lib/pkgconfig/liblab_gamut.pc
/tmp/tmp.bamfQudTlG/share/man/man3/lab_gamut.3
/tmp/tmp.bamfQudTlG/share/graphviz/doc/pdf/lab_gamut.3.pdf

Ellson · June 18, 2020, 12:36am

It’s interesting that there is a .pc file that supports library sharing with pkgconfig, but no .h header installed. That sounds like a bug.

Given this lib is unlikely to be used outside of graphviz I suggest don’t be too concerned about an API change. It would be good to note it in the ChangeLog.

Ellson · June 18, 2020, 1:38am

The header install bug is in lib/edgepaint/Makefile.am where NOINST_HEADERS =
contains lab_gamut.h.
So the documented interface in lab_gamut.3 does not work. Which only goes to prove that liblab_gamut.so is currently unused by others.

smattr · June 18, 2020, 1:46am

Excellent news. I propose we stop shipping liblab_gamut.pc, lab_gamut.3, and lab_gamut.3.pdf. We can probably stop generating the PDF and maybe even delete the manpage. I think we still need to retain installation of the library itself because edgepaint and friends need it, right?

@magjac, this means we’re clear to change the lab_gamut array, but I don’t think it’s worth doing this until we validate it actually helps the Windows compilation. Can you let me know when you’ve tried compiling lab_gamut2.c on Windows and if it reduces memory usage?

magjac · June 18, 2020, 6:21am

@smattr When I click on the lab_gamut2.c link I get 404 and the snippetfile1.txt is not useful. If I’m just being stupid, please educate me.

smattr · June 18, 2020, 2:01pm

Seems Gitlab deleted the attachment for some reason. I’ve committed it to the lab_gamut_hack of my graphviz fork instead: https://gitlab.com/smattr/graphviz/-/commit/3d4b4e37fcdd5126aa6efe22594ea2dbce338847. Can you see the file there? It doesn’t get compiled in the full Graphviz build, so to test you’ll need to call the compiler on it directly like my GCC experiments.

magjac · June 18, 2020, 2:05pm

Yes, now I’ve got it. Thanks. How did you generate it?

smattr · June 18, 2020, 2:17pm

I deleted everything in lab_gamut.c except the array contents itself, ran the below Python, then reinserted the #include and remaining syntax to make it correct into the result. I haven’t rigorously checked my math, and we would need to also add some kind of accessor function to unpack bytes from the string into a lab_struct, but it seemed representative to me of the RAM usage you would see compiling such a thing.

import re
import sys

i = 0
for line in sys.stdin:
  m = re.match(r'{(-?\d+),(-?\d+),(-?\d+)},', line)
  assert m is not None

  for b in (int(m.group(1)), int(m.group(2)), int(m.group(3))):
    if i == 0:
      sys.stdout.write('"')
    if b < 0:
      b = 256 + b
    sys.stdout.write(f'\\x{b:02x}')
    i += 1
    if i == 20:
      sys.stdout.write('"\n')
      i = 0

magjac · June 18, 2020, 2:19pm

I just love Windows software:

C:\Users\magja\graphviz\lib\edgepaint\lab_gamut.c(3279,1): fatal error C1091: compiler limit: string exceeds 65535 bytes in length [C:\Users\magja\graphviz\lib\edgepaint\lab_gamut.vcxproj]
Done Building Project "C:\Users\magja\graphviz\lib\edgepaint\lab_gamut.vcxproj" (default targets) -- FAILED.
Done Building Project "C:\Users\magja\graphviz\graphviz.sln" (lab_gamut target(s)) -- FAILED.