Building for Ubuntu 16.04 LTS in CI

Dev
1

@Ellson (or anyone): I’d appreciate any help on this.

Ref: import-convert for cytoscape (#1730) · Issues · graphviz / graphviz · GitLab

Since Ubuntu 16.04 is not EOL until 2024 and this user would need it, I think it’s worth pursuing.

First I got hit by this issue, so I removed ocaml from the Dockerfile.

Then I got:

  CXX      libgv_php_la-gv_php.lo
gv_php.cpp:896:106: error: macro "zend_hash_update" passed 6 arguments, but takes just 3
       zend_hash_update(HASH_OF(z), (char*)"_cPtr", sizeof("_cPtr"), (void*)&resource, sizeof(zval*), NULL);
                                                                                                          ^

which I think is the same problem as this issue, so I removed php-dev from the Dockerfile.

Then I got:

d-devlibdeps \
	--override s/libpathplan4-dev// \
	--override s/libcgraph6-dev//   \
	--override s/libgraph5-dev//    \
	--override s/libxdot4-dev//     \
	--override s/libcdt5-dev//      \
	--override s/libgvpr2-dev//      \
	--override s/libgvc6-dev//      \
	--override s/libexpr4-dev//      \
	--override s/libgts-0.7-5-dev/libgts-dev/ \
	--override s/libexpat1-dev// \
	--override s/libltdl7-dev// \
	--override s/zlib1g-dev// \
	/workdir/graphviz-2.45.20200601.1054/debian/libgraphviz-dev.substvars \
	/workdir/graphviz-2.45.20200601.1054/debian/tmp/usr/lib/*.so
# Move from debian/tmp to the appropriate packages, rename one binary
dh_install --sourcedir=debian/tmp --list-missing
dh_install: libgv-ocaml missing files: usr/lib/graphviz/ocaml
dh_install: libgv-ocaml missing files: usr/share/man/man3/gv.3ocaml
dh_install: libgv-php5 missing files: usr/lib/graphviz/php
dh_install: libgv-php5 missing files: usr/share/man/man3/gv.3php
dh_install: libgv-php5 missing files: usr/share/php/gv.php
dh_install: usr/lib64/tcl8.6/graphviz/libgdtclft.so.0.0.0 exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/libtclplan.so exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/libtcldot_builtin.so.0.0.0 exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/libtcldot.so.0.0.0 exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/libgv_tcl.so exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/libtcldot.so.0 exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/libgdtclft.so.0 exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/libtcldot_builtin.so exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/libtclplan.so.0 exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/libtclplan.so.0.0.0 exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/pkgIndex.tcl exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/libgdtclft.so exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/libtcldot_builtin.so.0 exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/tcl8.6/graphviz/libtcldot.so exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib64/lua/gv.so exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/examples/4elt.gv exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/examples/world.gv exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/bb exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/span exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/cliptree exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/flatten exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/chkedges exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/collapse exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/treetoclust exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/scalexy exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/rotate exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/deghist exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/addedges exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/histogram exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/depath exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/delnodes exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/dijkstra exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/scale exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/color exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/anon exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/group exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/addrings exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/get-layers-list exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/topon exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/binduce exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/bipart exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/cycle exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/dechain exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/path exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/deledges exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/attr exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/chkclusters exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/delmulti exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/indent exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/addranks exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/col exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/maxdeg exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/knbhd exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvpr/bbox exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/doc/cpl1.0.txt exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/doc/addingLayout.txt exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/doc/COPYING exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/doc/fontfaq.txt exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/doc/NEWS exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/graphviz/gvedit/attrs.txt exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/man/man3/gv.3go exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/man/man3/expr.3 exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/man/man3/lab_gamut.3 exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/man/man3/gvpr.3 exists in debian/tmp but is not installed to anywhere
dh_install: usr/share/man/man3/pack.3 exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib/graphviz/ruby/gv.so exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib/graphviz/lua/gv.so exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib/graphviz/go/gv.go exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib/graphviz/go/libgv_go.so exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib/graphviz/go/runtime.h exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib/python3/dist-packages/gv.py exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib/python3/dist-packages/_gv.so exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib/x86_64-linux-gnu/perl5/5.22/gv.so exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib/x86_64-linux-gnu/perl5/5.22/gv.pm exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib/x86_64-linux-gnu/ruby/vendor_ruby/2.3.0/gv.so exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib/python2.7/dist-packages/gv.py exists in debian/tmp but is not installed to anywhere
dh_install: usr/lib/python2.7/dist-packages/_gv.so exists in debian/tmp but is not installed to anywhere
dh_install: missing files, aborting
debian/rules:130: recipe for target 'install' failed

which I don’t think I should, since it’s expected.

Help on any of the above would be appreciated.

2

The Dockerfile is in Commits · wip-build-for-ubuntu-16.04 · Magnus Jacobsson / graphviz · GitLab

3

I don’t know that I’ve ever built for Ubuntu 16.
I’ll see if I can install locally as a VM to take look at these issues. It might take a while.

1 Like
4

You can also use my Dockerfile or pull the image that I just pushed if you don’t want to build it yourself.

5

I still like my VMs for this. Its right here and I can log into it and try things…

I based my apt-gets on your ci/ubuntu-18.04/Dockerfile and encountered three issues:

  1. guile is unavailable
  2. libargon2-0-dev is unavailable (used by php - probably explains later php problems)
  3. libltdl-dev is needed and available, but wasn’t in your 18.04 Dockerfile.

I built graphviz from a fresh git clone using:
./autogen.sh
./configure --disable-ocaml --disable-php
make

Builds with ocaml or php enabled failed, probably not so important why right now. I think we should just build ubuntu-16.04 packages without ocaml and php. The ./configure is in debian/rules, so somehow these rules should be modified for different versions of Ubuntu. I’m not a Debian expert, so not sure how to do that?

6

Those issues are the same as I encountered and led up to the Docker file for Ubuntu 16.04 that I posted above, even though libltdl-dev was implicitly included by something else, so I didn’t have that problem.

This again raises the question how our build system is supposed to work. I would’ve expected that it shouldn’t complain about missing things that it has already detected it cannot build.

My expectation would be that this was done automatically per above, not that a hack for each operating system should be necessary. That could also have negative consequences. Suppose a user has an Ubuntu 16.04 system with ocaml included, they would be prohibited to build with ocaml by the rule modification you suggest.

Sorry if I have misunderstood something. In that case, please educate me.

7

You can do the same with the Docker image, it’s just much easier and faster than spinning up a VM and logging in:

magjac@t440:~/graphviz$ docker run -it graphviz:ubuntu-16.04
root@8715376b4c37:/#

This takes 1 second on my laptop.

8

The configure mechanism is lenient in its requirements consistent with it trying to do the best it can for new users of ./configure;make;make install

The binary builds for specific platforms, on the other hand, are not lenient. Everything possible on a fully loaded instance of the platform should be built, and hard errors should occur if something changes or goes missing.

The strictness override is provided from the spec file (for rpm systems) and the rules file (for deb systems) In both of those mechanisms you will find ./configure args that hard --enable or --disable various features.

I do think I/we have a problem with dealing with variations between distros eg. Ubuntu 16 v 18, and Centos 6 v 8.

At the moment the Redhat version of this is dealt by lots of conditionals in the spec file. This works but is messy. It was done this way so that a single spec file, and a single graphviz.src.rpm, work on all redhat platforms. I no longer consider this important, and in fact we ship separate src.rpm for each distro even though they are internally identical and portable.

I don’t know how to implement similar conditionals in the debian rules file.

I’m starting to think about separate spec files, and rules files, for each distro.
I’m thinking the correct file would be selected from a set of alternates by the .gitlab-ci.yml at the beginning of pipeline stage 2.

Just thinking out loud…

9

“Suppose a user has an Ubuntu 16.04 system with ocaml included, they would be prohibited to build with ocaml by the rule modification you suggest,”

Not at all. The strict requirements only apply to the CI builds.

10

OK, thanks for the detailed explanation.

OK, so you mean that

should only apply when building in CI?

11

I guess I still don’t understand.

In my build for Ubuntu 16.04, the output from configure has a section:

graphviz-2.45.20200605.1617 will be compiled with the following:

which contains

gv_php: No (missing header)

and

gv_ocaml: No (ocaml not available)

This later leads to:

dh_install: libgv-ocaml missing files: usr/lib/graphviz/ocaml
dh_install: libgv-ocaml missing files: usr/share/man/man3/gv.3ocaml
dh_install: libgv-php5 missing files: usr/lib/graphviz/php
dh_install: libgv-php5 missing files: usr/share/man/man3/gv.3php
dh_install: libgv-php5 missing files: usr/share/php/gv.php
...
dh_install: missing files, aborting
debian/rules:130: recipe for target 'install' failed

Why isn’t this wrong?

12

Noob question, but why is debian/rules used for testing? I was under the impression debian/rules was only relevant for package distribution so it only needed to support a single use case: Debian unstable (Sid) with all dependencies installed and all the optional things enabled. Do we package separately for the Ubuntu repositories? Or is debian/rules simply used during testing so we know that we’re really testing the same thing we’re distributing?

13

That was my intention, yes. You could imagine having two kinds of tests though: one kind that you run in order to check if there’s any idea to even produce install packages and one kind to test that you’ve managed to create correct install packages. I’ve chosen the latter only. By having also the former you could fail faster, which could be beneficial in certain situations.

I’m open to change this if there are arguments for doing so.

1 Like
14

Even after explicitly disabling ocaml and php, I get the same error:

diff --git a/debian/rules b/debian/rules
index e7e4630b9..8073ae3ec 100755
--- a/debian/rules
+++ b/debian/rules
@@ -79,8 +79,8 @@ configure-stamp:
        --disable-java \
        --disable-io \
        --enable-lua \
-       --enable-ocaml \
-       --enable-php \
+       --disable-ocaml \
+       --disable-php \
        --enable-python \
        --disable-python23 \
        --disable-python24 \

From the log:

  gv_ocaml:      No (disabled)
...
  gv_php:        No (disabled)
...
dh_install: libgv-ocaml missing files: usr/lib/graphviz/ocaml
dh_install: libgv-ocaml missing files: usr/share/man/man3/gv.3ocaml
dh_install: libgv-php5 missing files: usr/lib/graphviz/php
dh_install: libgv-php5 missing files: usr/share/man/man3/gv.3php
dh_install: libgv-php5 missing files: usr/share/php/gv.php
15

Yes. I think that’s true. Only when we are building binaries that are in principle usable by anyone with a particular distro and distro version.

1 Like
16

So I think I agree with you that debian/rules is only used for distribution. In our case its only our graphviz packages that we distribute, but we have to deal with the variation between the various ubuntu releases.

In my mind it is the equivalent of the spec file for rpms, which similarly have to deal with variations between the rhel, centos, fedora and multiple different releases of each.

I don’t understand your comment about testing. Aren’t the builds from graphviz master branch that are considered deployable to the multiple distros, aren’t these the most important place for regression testing?

17

Never mind. This was a misunderstanding on my part.

18

Following @ellson’s steps, I tried to replicate this in an Ubuntu 16.04 VM. I did not disable ocaml, but based on some old @ellson comments I came across I did something kind of stupid and hacked the SWIG-generated ./tclpkg/gv/gv_ocaml.cpp to find and replace “int64” with “int64_t” and then tweaked the makefile:

diff --git a/tclpkg/gv/Makefile.am b/tclpkg/gv/Makefile.am
index ee2f82a..5167d99 100644
--- a/tclpkg/gv/Makefile.am
+++ b/tclpkg/gv/Makefile.am
@@ -161,7 +161,6 @@ libgv_ocaml_la_LDFLAGS = -module -avoid-version
 libgv_ocaml_la_CPPFLAGS = \
        $(BASECPPFLAGS) \
        $(OCAML_INCLUDES) \
-       -Dint64=int64_t \
        -fpermissive
 gv.ml gv.mli: gv_ocaml.cpp
 gv_ocaml.cpp: gv.i

With this, ./autogen.sh && ./configure --disable-php && make && make install works and the pytest test suite passes.

Maybe what we’re missing in the VM/Docker discrepancy is that my Ubuntu 16.04 VM (and maybe @ellson’s as well?) did a full apt upgrade during installation. Maybe the Docker image is missing an upgrade to some package in the base system that we’re inadvertently using?

Incidentally it seems the root cause for both the ocaml and php failures may be that the version of SWIG in Ubuntu 16.04 is too old. Probably not worth the juggling necessary to get a newer out-of-band SWIG installed.

19

I feel like we’re probably talking at cross purposes here. Too me there are several unanswered questions above, but perhaps it’s because of my lack of knowledge that I think that.

I just realized something (I think) that you may have taken for granted that I already knew; when doing ./configure && make && make install, the binaries are directly installed without any packages being produced (is that correct?), while when doing fakeroot make -f debian/rules binary (for .deb) or rpmbuild -ta graphviz-${GV_VERSION}.tar.gz (for .rpm), packages are produced that can be subsequently installed. It seems you have tested the former path and I the latter.

With that insight (hopefully), I’ll rephrase my question:

I want to build from the portable source and produce installable packages. How can I do that on a system that lacks some packages that are needed for things I don’t care about? I’m willing to sacrifice almost anything other than the core tools themselves, but I’m not willing to hack the portable source.

20

Ah yes, you’re completely right. I misunderstood. I’ll see if I can reproduce your issues with the fakeroot... work flow unless @ellson beats me to it.

1 Like