# Bug report: Clustered subgraph leads to crash in dot

**URL:** <https://forum.graphviz.org/t/bug-report-clustered-subgraph-leads-to-crash-in-dot/3395>\
**Category:** Dev\
**Created:** [August 25, 2026, 10:41am UTC](https://forum.graphviz.org/t/bug-report-clustered-subgraph-leads-to-crash-in-dot/3395 "2026-08-25T10:41:21Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![mmadsen\_cig](https://avatars.discourse-cdn.com/v4/letter/m/f08c70/32.png) [@mmadsen\_cig](https://forum.graphviz.org/u/mmadsen_cig)\
**Post date:** [August 25, 2026, 10:41am UTC](https://forum.graphviz.org/t/bug-report-clustered-subgraph-leads-to-crash-in-dot/3395/1 "2026-08-25T10:41:21Z")

</div>

At my company, we use Graphviz to layout various graphs, including some data flow graphs to show how variables are being used across a number of scripts.

We’ve noticed that in some scenarios, it will simply crash when it tries to layout certain graphs. Checking a debug build of 9.0 (which is the version we’re currently using), it turns out that transpose\_step fails the assertion ND\_order(v) \< ND\_order(w), since both nodes have order 0.

After adding a bunch of debug output and attempting to trace through the bug, it looks to me like the issue is that flat\_reorder never adds nodes to temprank if `local_in_cnt != 0` for a node - but once it actually goes to set ND\_order for the rank, it assumes all of the nodes have been added to temprank.

I also downloaded a debug build of 16.0, and here it asserts an out-of-bounds access. Glancing at the code, the fundamental issue appears to be the same, and it’s just being detected consistently in flat\_reorder because 16.0 does bounds checking. However, I have not done any debugging in this version of the code yet, since I don’t have it building locally yet (that’s the next thing I’ll be looking at).

It feels like it should be safe to just stop processing that rank if a node is skipped, but I’m not entirely confident if that’s the best move, or if it’s better to try to work with what _can_ be done. Skipping the rank makes test\_2368 in the 9.0 test suite pass instead of resulting in an expected failure, but I can’t quite tell if that’s indicative of an actual issue or not.

Here’s the graph, reduced as much as I could without changing the location of the crash in 9.0:

```auto
digraph G {
	graph[ordering = "in"];
	node8->node153
	node211->node8
	node221->node8
	node12->node153
	node12->node205
	node91->node14 // possibly not needed
	node211->node22
	node48->node205
	node91->node48 // possibly not needed
	node118->node48
	node51->node221
	node118->node51
	node72->node221 // possibly not needed
	node8->node205
	subgraph group0 {
		cluster=true;
		node8
		node12
		node14 // possibly not needed
		node22
		node48
		node51
		node72 // possibly not needed
		node91 // possibly not needed
		node118
		node153
		node205
		node211
		node221
	}
}

```

I’ve added comments to the lines that I can still remove and see a crash, but 9.0 crashes or asserts in a different place when I do - so I left those in while doing my testing to be 100% certain I wasn’t suddenly looking at a different bug. But as I mentioned above, I suspect this is really the same issue in flat\_reorder with or without those lines.

---

<div class="post-metadata">

**Author:** ![steveroush](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@steveroush](https://forum.graphviz.org/u/steveroush)\
**Post date:** [August 25, 2026, 2:13pm UTC](https://forum.graphviz.org/t/bug-report-clustered-subgraph-leads-to-crash-in-dot/3395/2 "2026-08-25T14:13:37Z")

</div>

Thanks for reporting this. Here is your bug report: [dot abort (#2854) · Issues · graphviz / graphviz · GitLab](https://gitlab.com/graphviz/graphviz/-/work_items/2854)

Note this odd workaround:  
`nop myfile.gv | dot -Tpng -omyfile.gv`  
All it does is “prettyprint” (reformat & rearrange) the input. Surprising.

---

<div class="post-metadata">

**Author:** ![mmadsen\_cig](https://avatars.discourse-cdn.com/v4/letter/m/f08c70/32.png) [@mmadsen\_cig](https://forum.graphviz.org/u/mmadsen_cig)\
**Post date:** [August 25, 2026, 3:59pm UTC](https://forum.graphviz.org/t/bug-report-clustered-subgraph-leads-to-crash-in-dot/3395/3 "2026-08-25T15:59:39Z")

</div>

Yeah, it feels like it might be dependent on internal ordering of the edges - in the nop image, it routes the long edge on the far right, but if you just remove e.g. the edge node8-\>node153 in my reproduction file, it will generate the rest of the graph properly - but the long edge ends up going between node12 and node22.

From the debug output I’ve been looking at, it appears to be creating an intermediate placeholder node in the layer when it’s routing in-between those nodes. It seems like it does not do that when the edge ends up on the far right, since it doesn’t need to leave room in the middle of the layer.

I’ll investigate if running it through nop first makes a difference in the real scenario, but I kinda suspect that might be more of a coincidence than a guaranteed outcome.

I got 16.0.0 building locally, and with a slight modification to flat\_reorder I could get it to generate the graph without crashing:

```auto
diff --git a/lib/dotgen/mincross.c b/lib/dotgen/mincross.c
index 1aeb25626..265378d1b 100644
--- a/lib/dotgen/mincross.c
+++ b/lib/dotgen/mincross.c
@@ -1325,6 +1325,7 @@ static void postorder(graph_t *g, node_t *v, nodes_t *list, int r) {

 static void flat_reorder(graph_t *g) {
   int i, r, local_in_cnt, local_out_cnt, base_order;
+ bool valid_rank;
   node_t *v;
   nodes_t temprank = {0};
   edge_t *flat_e, *e;
@@ -1334,6 +1335,7 @@ static void flat_reorder(graph_t *g) {
   for (r = GD_minrank(g); r <= GD_maxrank(g); r++) {
     if (GD_rank(g)[r].n == 0)
       continue;
+ valid_rank = true;
     base_order = ND_order(GD_rank(g)[r].v[0]);
     for (i = 0; i < GD_rank(g)[r].n; i++)
       MARK(GD_rank(g)[r].v[i]) = false;
@@ -1362,11 +1364,14 @@ static void flat_reorder(graph_t *g) {
       else {
         if (!MARK(v) && local_in_cnt == 0) {
           postorder(g, v, &temprank, r);
+ } else {
+ valid_rank = false;
+ break;
         }
       }
     }

- if (!LIST_IS_EMPTY(&temprank)) {
+ if (valid_rank && !LIST_IS_EMPTY(&temprank)) {
       if (!GD_flip(g)) {
         LIST_REVERSE(&temprank);
       }

```

The test suite in 16.0.0 gives the same results with and without this patch, except for the aforementioned test\_2368 which now passes when it’s an expected fail.

---

<div class="post-metadata">

**Author:** ![steveroush](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@steveroush](https://forum.graphviz.org/u/steveroush)\
**Post date:** [August 26, 2026, 1:23am UTC](https://forum.graphviz.org/t/bug-report-clustered-subgraph-leads-to-crash-in-dot/3395/4 "2026-08-26T01:23:49Z")

</div>

Whoops, using **nop** does result in a finished graph, but it does so by reordering the nodes and edges, and thereby overriding the **ordering=in** attribute.

---

<div class="post-metadata">

**Author:** ![smattr](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/smattr/32/85_2.png) [@smattr](https://forum.graphviz.org/u/smattr)\
**Post date:** [September 14, 2026, 2:20am UTC](https://forum.graphviz.org/t/bug-report-clustered-subgraph-leads-to-crash-in-dot/3395/5 "2026-09-14T02:20:35Z")

</div>

I posted this to [!5039](https://gitlab.com/graphviz/graphviz/-/merge_requests/5039) but don’t really know how to describe the change. @mmadsen_cig do you have any suggestions?

---

<div class="post-metadata">

**Author:** ![mmadsen\_cig](https://avatars.discourse-cdn.com/v4/letter/m/f08c70/32.png) [@mmadsen\_cig](https://forum.graphviz.org/u/mmadsen_cig)\
**Post date:** [September 14, 2026, 9:35am UTC](https://forum.graphviz.org/t/bug-report-clustered-subgraph-leads-to-crash-in-dot/3395/6 "2026-09-14T09:35:59Z")

</div>

> but mmadsen\_cig has the best understanding of the situation.

Welp, we’re in trouble 🤣

The condition that triggers the bug is when `constraining_flat_edge` returns true for an edge that is ingoing to the rank. That method considers an edge constraining if it has non-zero weight and the head and tail are both in a cluster (the clustering being what causes my test case to break).

So I guess the best way to describe the change would be something like “bail out if incoming edges are constrained”.

Tracing `flat_reorder` back through git history gives me the impression that this was probably always intended, it just didn’t actually work? I can’t be sure since the method predates the first entry in the git history (in 2004), so I have no real way of telling if the cluster checks were added in response to a bug or not.

However, it does occur to me that an edge should probably be considered non-constraining if the cluster for both ends of the edge are the _same_ cluster (`ND_clust(aghead(e)) != ND_clust(agtail(e))`). I don’t see any obvious reason why that shouldn’t work, and the 16.0 test suite at least gives me the same results either way. If that was made part of the same change, then it’s perhaps more accurate to say something like “bail out when encountering cross-cluster connections”, since that’s the scenario that we’re (seemingly) trying to avoid.

---

<div class="post-metadata">

**Author:** ![smattr](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/smattr/32/85_2.png) [@smattr](https://forum.graphviz.org/u/smattr)\
**Post date:** [September 14, 2026, 1:34pm UTC](https://forum.graphviz.org/t/bug-report-clustered-subgraph-leads-to-crash-in-dot/3395/7 "2026-09-14T13:34:25Z")

</div>

Thanks! I’ll update the description.

> [@mmadsen\_cig](#):
>
> However, it does occur to me that an edge should probably be considered non-constraining if the cluster for both ends of the edge are the _same_ cluster (`ND_clust(aghead(e)) != ND_clust(agtail(e))`). I don’t see any obvious reason why that shouldn’t work, and the 16.0 test suite at least gives me the same results either way. If that was made part of the same change, then it’s perhaps more accurate to say something like “bail out when encountering cross-cluster connections”, since that’s the scenario that we’re (seemingly) trying to avoid.

I don’t know enough about the feature to comment on this. We’ll need input from Steve.

---

<div class="post-metadata">

**Author:** ![steveroush](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@steveroush](https://forum.graphviz.org/u/steveroush)\
**Post date:** [September 15, 2026, 6:12am UTC](https://forum.graphviz.org/t/bug-report-clustered-subgraph-leads-to-crash-in-dot/3395/8 "2026-09-15T06:12:54Z")

</div>

Are you asking me to comment about the **ordering** feature or something else?  
Note that this graph seems to have _no_ flat (same rank) edges.

---

<div class="post-metadata">

**Author:** ![mmadsen\_cig](https://avatars.discourse-cdn.com/v4/letter/m/f08c70/32.png) [@mmadsen\_cig](https://forum.graphviz.org/u/mmadsen_cig)\
**Post date:** [September 15, 2026, 8:55am UTC](https://forum.graphviz.org/t/bug-report-clustered-subgraph-leads-to-crash-in-dot/3395/9 "2026-09-15T08:55:59Z")

</div>

In that case, `constraining_flat_edge` is maybe not the best name for that function, because it doesn’t look at rank at all (and going by how it’s used, I don’t think it would want to, either). Totally understandable how it ended up with the name (the edge is constraining a `flat_*` function), doubly so when the name was only added many years later when the check was extracted to a function.

But the question is really more about the “constraining” part. Other than the bounding box from the cluster, it’s not clear to me why a graph that’s fully contained in a single cluster would be laid out any differently from the same graph without any clusters. But because `constraining_flat_edge` only checks if the nodes on either end are _in_ a cluster, not whether they’re in the _same_ cluster, you end up with exactly that difference.

In other words, shouldn’t `constraining_flat_edge` look more like this:

```c
static bool constraining_flat_edge(Agraph_t *g, Agedge_t *e) {
  if (ED_weight(e) == 0)
    return false;
  if (!inside_cluster(g, agtail(e)))
    return false;
  if (!inside_cluster(g, aghead(e)))
    return false;
  if (ND_clust(aghead(e)) == ND_clust(agtail(e)))
    return false;
  return true;
}

```

Of course, this isn’t really important for the bug report or bugfix itself - just an observation I made once @smattr asked for input for the commit message. It can always be addressed at another time.

---

<div class="post-metadata">

**Author:** ![scnorth](https://sea2.discourse-cdn.com/graphviz/user_avatar/forum.graphviz.org/scnorth/32/89_2.png) [@scnorth](https://forum.graphviz.org/u/scnorth)\
**Post date:** [September 16, 2026, 12:30pm UTC](https://forum.graphviz.org/t/bug-report-clustered-subgraph-leads-to-crash-in-dot/3395/10 "2026-09-16T12:30:12Z")

</div>

All I can do is apologize for not having looked at this in probably 5 or 10 years.

The reasonable thing is just take a fresh look at the problem and fix it. Thank you for looking at this, after so long. Clearly there are a few corners in mincross.c and position.c where we didn’t think things through, generally involving flat edges and/or clusters.

Yes I agree that the intent is that a graph contained in a global cluster should be treated the same as if there were no containing cluster.