# GitLab runtimes **URL:** \ **Category:** Dev\ **Created:** [October 7, 2021, 5:39pm UTC](https://forum.graphviz.org/t/gitlab-runtimes/868 "2021-10-07T17:39:27Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![magjac](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/magjac/32/16_2.png) [@magjac](https://forum.graphviz.org/u/magjac)\ **Post date:** [October 7, 2021, 5:39pm UTC](https://forum.graphviz.org/t/gitlab-runtimes/868/1 "2021-10-07T17:39:27Z")
Many of the MinGW autotools CI build jobs that I’m currently working on takes more that the allowed 1 hour to build and thus fails with: `ERROR: Job failed: execution took longer than 1h0m0s seconds`. I’ve measured the actual build time in [a job that actually finishes](https://gitlab.com/magjac/graphviz/-/jobs/1656144669) before the deadline and the actual build time (excluding installation) is 45 minutes. The same process takes around 15 minutes on my local Windows 10 laptop with this CPU spec: `Intel(R) Core(TM) i7-4600U CPU @ 2.10GHz 2.69 GHz`. Is it reasonable that the GitLab servers are that slow in comparison? All the jobs can be found in [this pipeline](https://gitlab.com/magjac/graphviz/-/pipelines/383827002). ---
**Author:** ![smattr](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/smattr/32/85_2.png) [@smattr](https://forum.graphviz.org/u/smattr)\ **Post date:** [October 8, 2021, 12:28am UTC](https://forum.graphviz.org/t/gitlab-runtimes/868/2 "2021-10-08T00:28:05Z")
It’s possible. These are backed by GCP, right? My experience with other GCP-backed CI providers is that there is a _massive_ difference in what tier the CI provider are using. Empirically for the same task: - Travis CI: regularly hitting a 50 minute timeout. I had to shard the task into 5 separate jobs, which each individually occasionally hit the 50 minute timeout too. - Cirrus CI: same unsharded task finishes in \<25 minutes. ---
**Author:** ![mark](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/mark/32/84_2.png) [@mark](https://forum.graphviz.org/u/mark)\ **Post date:** [October 8, 2021, 1:11am UTC](https://forum.graphviz.org/t/gitlab-runtimes/868/3 "2021-10-08T01:11:21Z")
Yeah I’m also not very surprised to hear that cloud VMs we get for free might be underpowered, particularly in how much CPU they can access. ---
**Author:** ![schmoo2k](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/schmoo2k/32/29_2.png) [@schmoo2k](https://forum.graphviz.org/u/schmoo2k)\ **Post date:** [October 8, 2021, 6:45am UTC](https://forum.graphviz.org/t/gitlab-runtimes/868/4 "2021-10-08T06:45:35Z")
On internal GitLab server we were able to alter the default timeout, not sure if you can on the public version: [Configuring runners | GitLab](https://docs.gitlab.com/ee/ci/runners/configure_runners.html) ---
**Author:** ![magjac](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/magjac/32/16_2.png) [@magjac](https://forum.graphviz.org/u/magjac)\ **Post date:** [October 8, 2021, 10:32am UTC](https://forum.graphviz.org/t/gitlab-runtimes/868/5 "2021-10-08T10:32:27Z")
There seems to be a hard limit of 3600 seconds on the Windows cloud runners: > **[SaaS runners on Windows (beta) | GitLab](https://docs.gitlab.com/ee/ci/runners/saas/windows_saas_runner.html)** > > Documentation for GitLab Community Edition, GitLab Enterprise Edition, Omnibus GitLab, and GitLab Runner. ---
**Author:** ![smattr](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/smattr/32/85_2.png) [@smattr](https://forum.graphviz.org/u/smattr)\ **Post date:** [October 8, 2021, 3:16pm UTC](https://forum.graphviz.org/t/gitlab-runtimes/868/6 "2021-10-08T15:16:31Z")
I was going to suggest my favorite solution to accelerating any system: do less. We could avoid building Lefty on MinGW. Though looking at ci/build.sh it looks like we already don’t build Lefty. Do you know how many cores these runners have? We could run `make -j …`. It will make the build output a bit unreadable, but I don’t think we’re in the phase of paying down MinGW compiler warnings yet, so maybe this is bearable. ---
**Author:** ![magjac](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/magjac/32/16_2.png) [@magjac](https://forum.graphviz.org/u/magjac)\ **Post date:** [October 9, 2021, 4:30pm UTC](https://forum.graphviz.org/t/gitlab-runtimes/868/7 "2021-10-09T16:30:04Z")
> [@smattr](#): > > Do you know how many cores these runners have? No, and I’m not sure that’s how they work. We might be sharing their capacity with other jobs. > [@smattr](#): > > We could run `make -j …` . I’ve tried that now. It might have given a 5% improvement. The fastest of the problematic jobs that took very close to 60 minutes without `-j`, which sometimes succeeds and sometimes fail, now took 57 minutes, but I’ve done a rebase on main in between so it might not be significant. The other three jobs still failed though. 57-min job: > **[windows-mingw32-build (#1664564481) · Jobs · Magnus Jacobsson / graphviz ·...](https://gitlab.com/magjac/graphviz/-/jobs/1664564481)** > > Graph Visualization Tools All jobs: > **[Pipeline · Magnus Jacobsson / graphviz · GitLab](https://gitlab.com/magjac/graphviz/-/pipelines/385475897)** > > Graph Visualization Tools ---
**Author:** ![smattr](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/smattr/32/85_2.png) [@smattr](https://forum.graphviz.org/u/smattr)\ **Post date:** [October 9, 2021, 4:42pm UTC](https://forum.graphviz.org/t/gitlab-runtimes/868/8 "2021-10-09T16:42:52Z")
What I’m hearing is that you’d like to join me in the dead code removal quest 😉 ---
**Author:** ![magjac](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/magjac/32/16_2.png) [@magjac](https://forum.graphviz.org/u/magjac)\ **Post date:** [October 9, 2021, 4:54pm UTC](https://forum.graphviz.org/t/gitlab-runtimes/868/9 "2021-10-09T16:54:31Z")
Your hearing is extraordinary 😁 ---
**Author:** ![magjac](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/magjac/32/16_2.png) [@magjac](https://forum.graphviz.org/u/magjac)\ **Post date:** [November 14, 2021, 8:08pm UTC](https://forum.graphviz.org/t/gitlab-runtimes/868/10 "2021-11-14T20:08:55Z")
This problem seems to be MinGW specific: > Some of the linked to answers seems to indicate that Cygwin has a better approach. Possibly that’s why we don’t see this problem with Cygwin in CI. ---
**Author:** ![magjac](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/magjac/32/16_2.png) [@magjac](https://forum.graphviz.org/u/magjac)\ **Post date:** [November 21, 2021, 1:57pm UTC](https://forum.graphviz.org/t/gitlab-runtimes/868/11 "2021-11-21T13:57:12Z")
FWIW, here is `/proc/cpuinfo` from the GitLab runners: ```nohighlight processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU @ 2.30GHz stepping : 0 microcode : 0x1 cpu MHz : 2300.000 cache size : 46080 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 17 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht constant_tsc rep_good nopl xtopology cpuid pni pclmuldq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat md_clear arch_capabilities bogomips : 4600.00 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU @ 2.30GHz stepping : 0 microcode : 0x1 cpu MHz : 2300.000 cache size : 46080 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 1 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 17 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht constant_tsc rep_good nopl xtopology cpuid pni pclmuldq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat md_clear arch_capabilities bogomips : 4600.00 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: ``` ---
**Author:** ![smattr](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/smattr/32/85_2.png) [@smattr](https://forum.graphviz.org/u/smattr)\ **Post date:** [November 21, 2021, 4:58pm UTC](https://forum.graphviz.org/t/gitlab-runtimes/868/12 "2021-11-21T16:58:12Z")
My thoughts on what we could try: - `-j2`. May not help as I’ve come across other CI environments where the number of virtual CPUs is basically a lie and there aren’t enough physical CPUs backing them to give you any meaningful parallelism. - persistent `pacman` cache - persistent `ccache` cache I suspect the last one may be the most valuable thing to go after.