Debian peaceable having effort with merged /usr

Welcome to LWN.obtain

The following subscription-handiest instruct material has been made on hand to you
by an LWN subscriber. Thousands of subscribers depend upon LWN for the
generous news from the Linux and free gadget communities. Whenever you occur to appreciate this
article, please attach in options subscribing to LWN. Thank you
for visiting LWN.obtain!

By Jake Edge
April 5, 2022

The addition of the “/usr merge” feature has been one thing of longstanding mess
within the Debian world. It looks adore a moderately innocuous alternate, which
is in accordance to the notice of most diversified distributions at this point; it successfully
eliminates the pause-level /bin, /sbin, and /lib*
directories in negate to switch their contents to the corresponding areas
below /usr. However ever since we first lined the feature introduction for
Debian—bigger than six years within the past—it has a been a habitual sequence of
complications inside of that neighborhood. Fresh occasions comprise apparently merely
extended the distress, even when possibly the pause is in peek.

Background

The /usr merge belief modified into as soon as first raised in “The
Case for the /usr Merge” by Lennart Poettering in 2012. It came
out of the systemd neighborhood, nonetheless modified into as soon as no longer supposed to be systemd-particular, as
described in, even when, as the commentary thread
on that LWN publish indicates, the basis modified into as soon as controversial on the least in piece
due to its association with systemd. At its core, it modified into as soon as a
easy idea to switch the recordsdata in /bin to
/usr/bin, then impact a symbolic hyperlink so that /bin would
in finding to the backside of to /usr/bin; the the same will most seemingly be done for the diversified
high-level directories of hobby (/sbin and /lib*).

The principle cause within the support of the merge modified into as soon as for compatibility with diversified Unix
programs and to impact building packages for Linux distributions less complicated;
upstream gadget, in finding tools, and the adore would no longer want to handle
Linux specially. Anything that relied on /bin/foo or
/usr/bin/foo would fetch what they had been buying for, nonetheless all the pieces
would live below /usr. That can possibly possibly possibly allow things adore an immutable,
distribution-equipped /usr, snapshots of the gadget advise will most seemingly be
extra optimistic, and so forth.

Fedora, as is steadily the case, took the lead and implemented merged /usr, for
Fedora 17 in 2012. Rather a lot of distributions followed suit, with most of
the mainstream distributions following the Fedora approach of having a
“flag day” swap; fresh or upgraded versions of the distribution would
either install a merged /usr or swap the gadget to suppose that as
piece of an upgrade. Supporting both the extinct and the fresh structure
concurrently modified into as soon as no longer one thing these distributions chose to provide a enhance to.

Debian took a extra incremental approach, in piece due to it strives no longer to
impact wholesale adjustments to customers’ programs adore these required by a flag-day
upgrade to a merged /usr. In 2016, the flexibility to voluntarily
swap to that draw modified into as soon as added, then some makes an try had been made for more recent
versions of the distribution to be installed with a merged /usr by
default. In 2018, the Debian in finding programs had been switched to suppose that
structure, nonetheless complications cropped up due to
packages built below a merged /usr would most steadily fail to work on
non-merged programs. The attach of living of some recordsdata modified into as soon as being resolved at in finding
time to point into /usr/bin (to illustrate), nonetheless these recordsdata handiest
existed in /bin on the non-merged. programs.

The in finding gadget adjustments had been reverted, so that both kinds of programs may well possibly possibly
be supported. However the difficulty led Ian Jackson to
ask
the Debian technical
committee (generally abbreviated as CTTE or ctte), which
is the final arbiter of technical disputes all around the venture, to
override the decision by the debootstrap developers to
impact merged /usr be the default. Debootstrap is ragged to create a
corrupt gadget correct into a list that can then be ragged as a chroot ambiance
for building additional packages, even when it has diversified makes suppose of as successfully.

The committee declined
to overrule the debootstrap maintainers, nonetheless did give a lengthy summary of
the cases for and against /usr merge. The committee also ragged its advisory draw to
present that the then-below-building Debian 10 (“buster”) would
give a enhance to both gadget kinds in a “broken-down” make, nonetheless
advised that it modified into as soon as honest that Debian 11 (“bullseye”) switch
toward a “middle” ground with higher give a enhance to for merged /usr.

Debian, being Debian, talked about the topic multiple times alongside the contrivance, at
varied levels of depth and rancor. Supporting both kinds of programs
modified into as soon as no longer easy in varied methods, leading some to explain that either
/usr merge wants to be made an important—or scrapped fully.
At the pause of 2020, Ansgar Burchardt asked the
technical
committee to attain to a call
whether or no longer Debian 12 (“bookworm”) may well possibly possibly peaceable handiest give a enhance to merged
/usr programs. At the pause of January 2021, the quiz modified into as soon as
attach
to a vote, and the consequence
modified into as soon as unanimous in desire of doing so:

The Technical Committee resolves that Debian ‘bookworm’ may well possibly possibly peaceable
give a enhance to handiest the merged-usr root filesystem structure, shedding give a enhance to
for the non-merged-usr structure.

Until after the inaugurate of ‘bullseye’, any implementation of this
resolution should always be done within the ‘experimental’ distribution, or
in any other case saved out of the principal paths for the inaugurate of
‘bullseye’.

Symlink farms

That can possibly possibly possibly appear to in finding to the backside of the difficulty, in desire of a flag day for
Debian 12, nonetheless, as will most seemingly be guessed, things didn’t reasonably figure out
that contrivance. All the device thru the dialogue on the vote, committee member Simon
McVittie expressed topic
that there had been two competing visions of what a merged /usr
supposed. He puzzled: “Must peaceable we be extra particular than this in what we
vote on, to protect a long way from
later having to adjudicate between developers who disclose that a particular
implementation is or is now not always genuinely merged-usr?” That commentary would
foreshadow the occasions to attain.

The “standard” contrivance to provide a enhance to /usr merge, as described above, is
most steadily called “merged /usr thru aliased
directories“. It’s miles what the usrmerge tool, which got the ball
rolling for Debian in 2016, does; that mechanism is tracked on the UsrMerge net page on the Debian
wiki. An replacement, which is championed by dpkg maintainer Guillem Jover, makes suppose of
“symlink farms”, which McVittie described as follows:

an association the attach all frequent recordsdata which comprise historically been
in /bin, /sbin, /lib and /lib64 are physically positioned in /usr,
with /bin etc. becoming “symlink farms” containing symlinks adore
/bin/sh -> /usr/bin/sh, /lib/ld-linux.so.2 -> /usr/lib/ld-linux.so.2
and so forth

There modified into as soon as some dialogue within the worm entry about the advantages of the two
diversified arrangements, nonetheless many of the members regarded as if it can possibly possibly judge that
the symlink-farm approach modified into as soon as no longer honest, viable, or both. Alongside the contrivance, McVittie
realized
that the earlier decision on debootstrap regarded as if it can possibly possibly describe that the
committee conception that the aliased-directories mechanism modified into as soon as the singular
approach to solving the /usr merge effort. His vote
mirrored that concept.

The dialogue moved to the mailing lists, the attach a definite quantity of
consensus modified into as soon as stumbled on, on the least in some areas.
McVittie opened a
worm
asking the committee to in finding to the backside of the aliased directories versus symlink farm
quiz alongside with others swirling across the
upcoming /usr merge transition for bookworm. He modified into as soon as asking the
committee to suppose its formal advisory draw to provide steerage on how that
transition may well possibly possibly peaceable proceed.

A lengthy assertion
modified into as soon as attach out for a vote by the committee; as soon as extra, it passed
unanimously. The advice clearly rejected the symlink-farm approach, in desire of
aliased directories, and made it clear that packages may well possibly possibly no longer capture that
programs comprise a merged /usr till Debian 13 (“trixie”)
building starts. Previous that,
upgrading from either form of gadget to Debian 12 may well possibly possibly peaceable work and
packages may well possibly possibly peaceable in finding and efficiency on both kinds of programs (till
after Debian 12), and a few diversified items. The worm modified into as soon as marked as “done” on
October 18 and, as soon as extra, it seemed that the difficulty modified into as soon as resolved.

Dpkg resistance

In the intervening time, even when, the dpkg maintainer modified into as soon as adamantly adverse to the
aliased-directories approach; his complaints had been described in a MergedUsr net page in
the dpkg group‘s piece of the wiki. It presented an extended list of complications
with that approach since the mechanism is handled out of doorways of dpkg
itself:

This approach goes within the support of dpkg’s support, and has induced and [does] now trigger
complications as a result of aliased directories, as multiple pathnames canonicalize
into diversified pathnames that time to the the same dentry, that can mess up
anything else that handles pathnames in databases and identical. Express: dpkg has
supported for a genuinely very long time symlinked directories as a vogue to enable local
admins to administer filesystem size constraints, by shifting directories
contents into diversified partitions, nonetheless has by no device supported aliased directories
thru symlinks pointing to diversified directories already tracked by dpkg. This
approach makes it inconceivable to know what is the canonical (from dpkg’s
PoV) pathname for an object lawful by having a seek recordsdata from on the filesystem, so it’s
trivial for customers suppose the immoral one.

Most of the entries within the list had been marked with an emblem indicating:
“This approach is conception of broken by develop and is unsupported by
dpkg. dpkg-buildinfo marks packages built on these programs as execrable in
the .buildinfo file.” The symlink-farm approach is smartly-behaved, the
net page says, even
even when it peaceable has the aliasing effort for the symbolic hyperlinks that in finding
installed:

This peaceable suffers gentle aliasing complications, nonetheless lawful for pathnames that can
pause up in both areas due to backwards compatibility symlinks, which
may well possibly possibly peaceable protect lowering, and this form of topic will self-heal over time,
at final ending up being lawful a handful of them. The colossal incompatibility is
that within the long flee right here’s a microscopic and lowering bounded attach of living of
pathnames which can possibly possibly pause up inflicting complications, whereas the diversified structure is an
unbounded attach of living affecting all pathnames completely. Every other ingredient that makes
a incompatibility is that the article is a symlink, and no longer the the same object being
accessed from diversified pathnames.

Both that net page and the dpkg
FAQ entry point out the usage of the dpkg-fsys-usrunmess
utility, which “undoes the merged-/usr-thru-aliased-dirs
mess“. So it modified into as soon as possibly unsurprising when Josh Triplett reported in
mid-March that dpkg
had began to emit a warning when it modified into as soon as installed or upgraded:

Putting in place dpkg (1.21.2) ...
dpkg: warning: Gadget unsupported due to merged-usr-thru-aliased-dirs.
dpkg: warning: Consider .

This escalation looks in pronounce contradiction to the tech-ctte decision
in 994388. Furthermore, this looks to successfully suppose kit maintainer
scripts as a technique of directing a complaint at Debian customers that has no longer
gotten traction in diversified boards, after which directing such customers at a wiki
net page that contradicts a previous venture decision.

His file modified into as soon as geared toward the committee worm that resulted within the decision
support in October. As Russ Allbery renowned, that kind
of a warning “will most seemingly be perceived by customers as an official declaration from Debian as
a venture that their gadget configuration is unsupported, whereas
concurrently right here’s the default installation mode for recent programs and
one thing that we now comprise in other places acknowledged is a moral gadget
configuration“. Burchardt acknowledged
that the warning modified into as soon as already inflicting confusion and asked the committee to
attain to a call like a flash on contend with it “to
protect a long way from this becoming yet yet any other vitality drain (we had several
sufficiently long enough threads about this topic already)“.

Bugs and fixes

There is a quiz of what “supported” device within the context of
/usr merge, Helmut Grohne acknowledged; he modified into as soon as added to
the committee on the starting of the year after two members reached their
term restrict and retired from it. He acknowledged:

At this point, neither
merged-/usr nor unmerged-/usr is supported successfully. Both are broken in a single
contrivance or yet any other and no-one steps up to repair the mess. In particular, the
dpkg maintainer does no longer give a enhance to merged-/usr in dpkg (which is his
constitutional moral as long as he does no longer block reasonable patches),
nonetheless neither does somebody else. As such I fetch it no longer easy to disagree
with the instruct material of the warning. I enact stare how it confuses individuals. It
positively does no longer attain individuals that would enact one thing about [it]. Rather it
takes customers as hostages.

As had been talked about in a pair of of the (many) threads surrounding this topic,
Ubuntu has efficiently made the switch to a merged /usr; as Luca
Boccassi attach
it: “And on high of most up-to-date installations, extinct installations of Ubuntu upgrading
to 21.10 and/or the almost at the moment-to-be-released 22.04 had been [forcefully]
migrated too. They in finding no longer appear to be blocked, unsupported, or broken.” There
had been few complications with that transition, on the least in accordance to
/usr merge proponents, so regardless of bugs exist are no longer
describe-stopping. However Allbery cautioned against
taking that too a long way:

We all know that merged-/usr is buggy, in that one can create a
attach of living of kit operations that leave the gadget in an invalid advise. We
comprise a venture incompatibility over how principal these bugs are. No person is
stepping ahead to repair these bugs, which is certainly reasonably wretched. I
in my conception strongly disagree with the realization that merely due to Ubuntu
hasn’t considered many cases of this class of bugs whereas the usage of a kit attach of living
the attach individuals comprise no longer moved recordsdata between packages and out of /lib and
/bin very remarkable if in any respect, it’s acceptable to switch away dpkg in that buggy
advise.

[…] I in my conception am disappointed that the individuals who had been
pushing merged-/usr ahead are willing to switch away dpkg in a identified-buggy
advise without attempting to patch it to repair the final complications. I
worth that there are tons of barriers in efficiently doing that, no longer
all of that are technical, nonetheless I want to imagine that Debian is the form
of venture that can enact the no longer easy work (both technical and social) to repair
edge cases and protect a high level of consistency and correctness.

However Triplett disagreed with that
characterization:

It does no longer
seem in any respect apparent that such patches would had been permitted, given
the repeated vehement objections from the dpkg maintainer about the
chosen approach. These objections didn’t invite contribution; at every
point, the assertion modified into as soon as that usrmerge modified into as soon as broken, no longer that dpkg wanted
support supporting it.

Boccassi acknowledged
that having a seek recordsdata from at Ubuntu’s experience is legit; the complications identified
are reasonably small and “it be believed to be inconceivable
to in finding them fastened” over the maintainer’s objections. He asked what
else will most seemingly be a reasonable contrivance ahead. Allbery spoke back that
proponents of the alternate may well possibly possibly peaceable create a patch to repair the complications that
had been identified in dpkg even supposing it’ll be rejected. In piece, it
comes down to finding one thing that is actionable below the powers of the
technical committee, he acknowledged in a
notice-up: “It be no longer easy, procedurally, for the TC to enact anything else a pair of
theoretical patch that any individual may well possibly possibly write nonetheless hasn’t written.”

Committee chair Sean Whitton agreed
with that prognosis, even when he did disclose that he conception the committee may well possibly possibly
opt action to in finding the warning modified or removed; diversified members concurred
with that gape. Rapidly thereafter,
Triplett pointed
out that the warning message had modified, even when it modified into as soon as no longer genuinely any
higher in his gape.

Grohne also wanted to
stare fixes for dpkg to higher give a enhance to merged /usr. He even
talked about the likelihood of reverting the alternate:

If the resulting bugs enact no longer in finding fastened, we may well possibly possibly want to attach in options
diversified device for limiting their impact. Essentially the most apparent device right here is
revisiting the decision and brooding about whether or no longer the /usr-merge may well possibly possibly comprise
failed. On a project level, it undoubtedly has failed. Sooner or later, we
may well possibly possibly want to seek recordsdata from at a bigger represent than the technical one. If the
individuals driving the alternate are no longer in a build to enact it, then possibly we’d peaceable
no longer comprise that alternate within the first attach and revert support to the identified
working advise. Obviously that route is no longer without put.

A patch arrives

However as the thread stepped forward, Boccassi renowned
that a hyperlink to a dpkg patch had lawful been posted to IRC by a user named “uau”.
Allbery (among others) modified into as soon as contented to
stare the patch and at the moment had some feedback on methods to enhance it.
Committee member Gunnar Wolf conception
that the “patch looks sane from a important, very remarkable
10000m-point-of-gape“; he puzzled if it had “been shared with
Guillem, or incorporated in any relevant
worm file“.

Boccassi followed
up with some extra recordsdata from the author of the patch, again from
IRC. The author acknowledged “that a whereas within the past the patch modified into as soon as presented to the dpkg
maintainer, who rejected it with an reply alongside the traces of the frequent
‘usrmerge is broken by develop’, with no additional commentary“. Boccassi
puzzled if individuals that had been buying for a patch of this nature would “are trying to
opt it ahead themselves“. In step with that conversation with the
author, the license of the patch is the the same as for dpkg, so that should always no longer be
a barrier.

Grohne acknowledged that
he remembered some extra concrete criticism of the patch and that the patch
itself says that it’s incomplete, so it’ll no longer mosey into dpkg in its most up-to-date
make. He lamented the advise of the transition, and as soon as extra raised the
specter of reverting the total ingredient:

The an increasing number of I want to handle the /usr-merge the extra I in finding
disappointed by how badly this transition is deliberate and implemented. In
precept, the technical merits seem solvable to me, nonetheless the total
failure on the project level leaves me want for a revert. I am genuinely
bowled over that rather than improving the project, you protect on with that
detrimental perspective. Given this, it looks unsurprising that Guillem
does no longer want to work alongside with you. Obviously that is no longer an excuse for
imposing the hot adjustments to dpkg. The conversation is clearly
failing on both aspects, which is why we’re right here on the ctte again.

Burchardt asked
that Grohne discontinue suggesting a revert unless he wanted to open the project
of asking the committee (or the developers by contrivance of a conventional resolution)
to enact so. There are many complications with making that alternate at this
point, so: “Whenever you occur to in finding no longer want that solution, please in finding no longer point out it repeatedly:
it be non-motivating to use any additional time.”

Issues went additional off the rails on March 29, with a non-maintainer
add (NMU) for dpkg made by Bastian Easy that removed the warning
and stopped installing
dpkg-fsys-usrunmess. An NMU is one thing of a adverse act for a kit
with an brisk maintainer. As reported by
Triplett, the NMU modified into as soon as like a flash overridden by Jover with a commentary:
“This also clears a bullying NMU.” As Triplett attach it:
“we’re now even additional into chubby-blown ‘fights within the archive’
territory“.

In his most modern
message (as of this writing), Grohne acknowledged that Boccassi modified into as soon as no longer being
constructive and that he modified into as soon as no longer bowled over Jover didn’t want to
keep in touch with him about the complications. He reiterated a pair of of the
complications which comprise came about with the feature, noting that basically the most up-to-date effort modified into as soon as
largely no longer technical:

It has a historical previous of its
proponents no longer fixing the resulting bugs, nonetheless deferring them to others
and/or denying/downgrading them. I’ve positively spent bigger than per week
on fixing /usr-merge breakage excluding for the time discussing it. It’s miles
no longer working magnificent in any respect. Perchance, it’s fixable on a technical level,
nonetheless it’s fully broken socially. Please discontinue this unconstructive
behaviour.

While this will possibly possibly sound single-sided, Guillem’s behaviour wrt /usr-merge
can no longer be described as constructive either. Leisure assured, that aspect of
the represent is no longer being brushed off. That can peaceable even be evident from
dpkg.git at the present.

The dpkg Git
repository does describe a recent
alternate by Jover that will get rid of the offending warning, nonetheless handiest on Debian programs.
The commit message may well possibly possibly undoubtedly be interpreted as non-constructive, as
successfully, nonetheless. To illustrate: “On Debian some individuals seem so offended by a
(correct) warning, that the
apparent recourse has been for them to bully and abuse.” The
conversation between proponents of /usr merge and the
maintainer of the principal kit-manager for the distribution looks utterly broken
at this point—doubtlessly irretrievably so.

It’s miles no longer clear the attach things mosey from right here. The technical committee has
repeatedly made its feelings identified on the course it wants to perceive the venture
opt, nonetheless it’ll no longer negate somebody to enact anything else, precisely. Debian developers are
volunteers and its packages are the non-public fiefdoms of their maintainers.
The Debian
Structure does enable the committee to override developer
decisions—to no longer accept a patch, to illustrate—nonetheless any individual has to voluntarily
enact the work to bring the unhurried-breaking dpkg patch up to snuff.

It looks adore some venture elder (when it involves experience, no longer basically
age) may well possibly possibly steer this roughly alternate thru the rocks. While it can possibly possibly
appear to be
vanishingly no longer going that the committee would mosey into reverse on its three
earlier decisions,
even supposing there are two fresh members since they had been made, it’s no longer
inconceivable, on the least in theory. The bugs that
exist in dpkg with the aliased-directories version of /usr merge
may well possibly possibly merely continue to exist as irritations for a time, since they enact no longer genuinely seem
to rise to the level of a inaugurate-blocking off worm, on the least to this point. We are going to
all want to back and stare how it looks within the next episode of “As the
Debian Turns” …

(Log in to publish comments)