In toughen of single binary executable packages

TLDR

Simply learn the proposed resolution portion,
which boils the entire model down to:

Let’s fall the thought of machine packages,
and let’s undertake the self-contained binary executable
because the fresh unit of machine distribution!
Featured Content Ads
add advertising here
Join Bing with ChatGPT-4
Use the new internet where you get what you ask, Fast and precise using AI.
https://www.bing.com

In model machine distribution

AKA “if truth be told free without a-commercials apps retailer”
of the Linux and BSD worlds.
(Even supposing it remains to gaze for a methodology long the “no-commercials”
statement will retain…)

One of many well-known differentiators between Linux and BSD’s
when put next to Dwelling windows (and to a diploma OSX) is the
bundle manager
that fulfills about a vital duties:

discovery —
shopping for machine packages (mostly start-source)
that are identified to work on one’s machine;
fetching —
downloading (and verifying the signature of)
the entire recordsdata required to run the machine on one’s machine,
in most cases in the invent of bundled packages
(thus the name of “bundle manager”);
and it does this no longer simply for the explicit machine recordsdata
(mainly executables, libraries, sources, or documentation),
but furthermore other dependent recordsdata
(mainly other libraries or sources);
Featured Content Ads
add advertising here
COGNAC Metaverse
Explore World's first metaverse that is as real as our Real world.
https://cognac.knowasiak.com
deployment —
inserting in, updating,
and to a diploma configuring
(mainly thru installation hooks)
each and each the explicit machine
and its (far too many extra in most cases than no longer) dependencies;
but, most importantly, solving Sudoku —
provided that many bundle managers are
SAT solvers,
one would possibly maybe simply cheat at Sudoku
and let the bundle manager clear up the puzzle,
supplied that one is able to particular it by methodology of dependency constraints;

I repeatedly take into accout with enormous alarm
the time after I was utilizing Dwelling windows and I had to reinstall it:
win the “set up kits”,
each and each for machine and drivers,
which extra in most cases than no longer had been
extra cherished than grandmas jewellery
attributable to these had been gripping indispensable executables
chanced on handiest on the CD that came with the enlargement card;
then sit down thru unending sessions of next-next-next wizards;
rinse and repeat each and each 6 months,
or except it crashed bigger than 5 times a day…

And, if it weren’t for Homebrew,
OSX users would suffer from the identical destiny on every fresh set up…

Featured Content Ads

add advertising here

COGNAC Metaverse

Explore World's first metaverse that is as real as our Real world.

https://cognac.knowasiak.com

Also, I was lately made aware that there would possibly maybe be
Chocolatey for Dwelling windows,
that appears to bear ~9K packages accessible.

Sustainability of the gap-quo

Unfortunately, especially as of unhurried, sigh for the last 4 or 5 years,
things bear began to regress on the bundle administration front…
(No, it will not be the Sudoku solving skill, it be aloof there,
and it appears necessary for any non-trivial distribution-wide enhance…)

Or no longer it is that one, especially when working in a developer capability,
can’t repeatedly win the machine packages one is making an strive to search out
(and no longer handiest the most fresh model, but any model thereof).

Granted, for most distributions
— Debian, ArchLinux, Gentoo, FreeBSD, OpenBSD, and plenty others.,
and to a diploma OpenSUSE, Ubuntu, Fedora, and plenty others. —
bundle repositories are maintained completely by volunteers,
who, of their spare time, and with out any compensation
(moreover complaints that they obtain no longer construct adequate),
prefer to assist up with the upstream liberate cycle:
patch, originate, take a look at, normally doc, bundle, liberate, and plenty others.

Unfortunately, their life is no longer made less difficult,
for as soon as as a result of the explosion of a myriad
of start source initiatives that
need to be integrated of their distribution,
but furthermore as a result of about a factors that are extra in the developer’s nook:

increased usage of languages and run-times that
don’t fit the most fresh originate model of ./configure && win && win set up;
they’re written in languages treasure Rust / Scoot/ NodeJS,
and to a diploma Python / Ruby / Haskel / Java,
and even emerging ones treasure Zig / Nim;
these fresh languages no longer handiest come with assorted instruments,
but furthermore with their bear language particular dependency managers
(some of which I wager are themselves Sudoku solving engines);
these kinds of fresh initiatives bear immense numbers
(normally in the tens, hundreds, and even thousands)
of dependencies;
given the most fresh originate model,
each and each of these dependencies need to be in my thought packaged and launched;
(right here is an evaluation of dependencies for NodeJS and Rust😉
on top of that,
many initiatives bear conflicting (recursive) dependency requirements
(I knew that the aptitude of solving Sudoku would bear its cause someplace);
beforehand, distributions managed to
provide a single model of a given library,
which does no longer stand anymore;
no longer handiest initiatives bear conflicting dependency requirements,
some initiatives themselves exercise at the identical time (as a result of deep dependency bushes)
assorted variations of the identical library (e.g. Rust, NodeJS);

Here are about a extra words about the topic:

Numbers supporting the sustainability divulge

OpenSUSE packages statistics

As an instance, on my OpenSUSE Tumbleweed laptop
(with some additional pattern repositories enabled)
there are:

~56K packages in entire;
~13K packages that if truth be told set up binaries underneath
/bin, /sbin, /usr/bin, or /usr/sbin;
(that is ~23%;)
~9K packages that start with ^lib;
(these are “classical” libraries;)
~5K packages that terminate with -devel$;
(these are mostly tied with the classical libraries;)
~10K packages that start with ^python[0-9]*-;
(that is ~17%;)
(these are for Python;)
granted that these kinds of are in triplicates,
as one library is probably to be packaged each and each for Python2, Python3.6, Python3.9, and plenty others.,
but right here’s moreover the level, as each and each is independently packaged and installed;
~8K packages that start with ^texlive-;
(these are for LaTeX and associated;)
~1500 packages that start with ^perl-;
(these are for Perl;)
~1200 packages that start with ^ghc-;
(these are for Haskel;)
~700 packages that possess -rubygem-;
(these are for Ruby;)
~160 packages that start with ^golang-;
(these are for Scoot;)
none appear as if accessible for NodeJS or Java;

In my test these numbers,
particularly the ration between packages that one can if truth be told attain,
and packages that are simply supporting libraries or sources,
will no longer scale with the explosion of start-source pattern…

GitHub initiatives statistics

If one searches for all initiatives tagged cli,
one can win:

~27K initiatives in entire;
~9K initiatives written in JavaScript or written in TypeScript;
~5K initiatives written in Python;
~4K initiatives written in Scoot;
~2K initiatives written in Rust;
handiest ~600 initiatives written in C; (no figures for C++;)

Primarily essentially based on these numbers,
and sigh that handiest 10% of them are edifying (that is ~3K),
sigh that every and each of these initiatives has on common
2 gripping dependencies (no longer shared with other instruments)
(that is an extra ~6K),
we win a entire of ~10K extra machine packages
that need volunteers to patch, take a look at, originate, and liberate…

I specialize in it be obtrusive right here’s no longer sustainable.

Some non-choices

Truly, and this has been occurring for about a years already,
the cracks are starting up to display
given what number of initiatives counsel change set up strategies:

from the “security incident ready to happen” —
curl http://innocent.tool/promise-no longer-to-run-sudo-and-trash-your-machine-and-take-your-passwords-set up-script | bash;
(I do know, I do know… it be unsafe attributable to I’m no longer utilizing https://...😉
alternating with “the cloud is now your laptop” —
docker run simply-but one more-2g-vm-image-running-a-fully-fledged-os-with-all-the-systemd-glory-masked-as-a-container;
(you win bonus ingredients if the machine in put a question to of
is basically a database engine,
that if truth be told would possibly maybe no longer lose any recordsdata need to you purge it by mistake;)
normally sprinkled with the “iOS / Android apps change for Linux”,
particularly one of Flatpack, AppImage, Rapidly, PackageKit, 0install, and plenty others.;
(right here is a tremendous article describing how these are no longer the long term😉
or the “licensed change to ./configure && win && win set up“,
depending on the language:
- mosey set up most-probably-will-be-rapidly; (no sarcasm right here!)
- cargo set up will-no longer-space-the-laptop-on-fire-whereas-building;
- npm set up will-no longer-obtain-the-entire-web-for-colored-messages;
- pip set up hope-you-treasure-debugging-why-native-dependencies-fail-to-originate;
- (I indubitably bear inclined some Ruby, Julia, but my initial estimation locations them in the pip set up category;)
or the fair a shrimp extra “saner” (?) model of the above,
but by utilizing “virtual environments”;
to my most well-most licensed one — no sarcasm right here furthermore!
— simply fetch the damn executable from the GitHub releases page,
replica it someplace to your ${PATH},
and mosey construct whatever you wished to construct in the first location;
(granted right here’s no longer economically sustainable,
attributable to now I will be able to’t bill my consumer
for “8 hours, provisioning the dev atmosphere”;)

A probably “worse is extra healthy” resolution

Which brings me to my level…

Given this day’s machine panorama,
a entire lot dependencies,
a entire lot building complications,
very rapidly liberate cycles,
many distributions to duvet,
extremely anxious users,
I specialize in there is handiest a single methodology out of this:

No, it be no longer Nix, or Guix, or Homebrew…
they’re simply prolonging the sickness…

Let’s fall the thought of machine bundle,
and let’s undertake the self-contained binary executable
because the fresh unit of machine distribution!

How would the workflow uncover treasure?

with a bundle manager, simply treasure it does now,
sans the dependency bother;
(i.e. we can fall our Sudoku solvers from the bundle manager;)
(from right here on, I rob there’ll not be this form of thing as a bundle manager;)
the packager (which extra in most cases than no longer is the developer himself):
- originate the executable for the few platforms on which the tool works;
- upload the executable to the mission’s GitHub releases page
  (which is able to be computerized with GitHub’s gh tool);
  (or anywhere else, if truth be told a centralized repository shall be tremendous😉
the user:
- curl -o ./some-tool https://innocent.tool/some-tool;
- sha256sum ./some-tool and compare with the liberate;
  (right here’s a field for but one more discussion;
  in the intervening time gaze cosign😉
- chmod a=rx ./some-tool
- sudo mv ./some-tool /usr/native/bin/some-tool;
- revel in!

Admire every little thing in life: tradeoffs

Cons

Are there disadvantages: glorious!

Simply don’t reveal the protection other folk,
they’ll express at you about vulnerabilities
which would possibly maybe even simply no longer be with out anguish mounted thru an enhance.

Also, the executable size is enormous, in the tens of MiB,
or normally even in the hundreds…

Plus, what if the binary would no longer construct what it says,
and as a change it be malicious?
What if the packager replaced the tool
with one thing else that does crypto-mining?
Effectively, construct you have confidence the developer
(or the developer of a dependency deep in the dependency tree)
no longer to take from your crypto-pockets?

Here are about a extra words about the disadvantages:

Pros

Are there benefits: glorious, x10!

No longer handiest are you able to with out anguish set up fresh updates
(that will indubitably embody these vulnerabilities fixes
the protection other folk cry about),
but you probably can now:

bear a couple of variations of the identical tool installed and running
at the identical time;
bear it installed and running even need to you bear no longer got
sudo rights;
simply location it in /dwelling/user/.bin (and bear that to your ${PATH}😉
bear it stowed to your .git repository
so that every person to your crew will get the identical abilities
with out onboarding scripts;

Barriers

And for glorious, this is no longer fair correct for every little thing.

I fetch no longer specialize in a grimy rich GUI utility treasure
LibreOffice,
Firefox, Thunderbird,
Slack, Discord, Skype,
GMail, and plenty others.,
would possibly maybe furthermore be delivered as a single executable.

Why, they need sources, and libraries,
and HTML, and JS, and CSS, and sounds,
and half of the fluffy cat photos on the web,
all to place in force
“ingredients that enable users to particular their most legitimate selves
and to elevate them joy whereas”
doing whatever users construct with these
tremendous life-enriching machine jewels of the favored world…
(The citation is from Firefox 94 “Colorways” liberate display.)

Requirements for single binary executables

What shall be the requirements for one thing treasure this to work?

well-known,
bundle all of your code into the executable itself;
obviously supplied out-of-the-box for compiled languages treasure
C / C++ / Rust / Scoot / Zig / Nim,
but it furthermore works for other languages treasure Python / Deno / Erlang / Java
that can exercise a Zip (or change) recordsdata because the container for the code;
2d, but simply as vital,
bundle all of your sources into the executable itself;
works out-of-the-box for Rust / Scoot,
can with out anguish work for C / C++ / Python / Deno / Erlang / Java,
no longer clear for others;
win the tool “set up direction self reliant”;
that is, the code need to no longer care if it be executable
is installed in
/usr/bin/native, /opt/some-tool/v2/bin, or even /dwelling/user/.bin;
win the tool “set up name self reliant”;
simply treasure the code need to no longer care where it be executable is installed,
it furthermore need to no longer care what it be executable file is known as;
it shall be named some-tool, some-tool-v42, whatever, and plenty others.;
(if it desires to re-attain itself, simply uncover at argv[0] or /proc/self/exec on Linux;)
(if it desires to attain but one more tool, that in flip will attain it,
simply export SOME_TOOL_EXEC with the argv[0], and doc that for others to exercise;)
if probably,
especially vital for Linux with it be
glibc variation all the arrangement thru distributions,
strive static linking;
works flawlessly with Scoot (including snide-compiling),
a shrimp bit with Rust / C / C++,
and most-probably it be already fulfilled
with interpreted languages treasure Python / Erlang / Deno / Java;
if probably,
especially to assist the static linking,
but furthermore to clutch away external dependencies,
strive to depend handiest on “pure libraries”
written to your programming language;
that is libraries that don’t depend on any OS supplied libraries;
and where it would no longer work, simply assist their number low;
if it be no longer probably no longer to depend on OS supplied libraries,
as a change,
strive to exercise rpath with $ORIGIN
and provide these libraries for obtain;
(never depend on LD_LIBRARY_PATH😉

No longer a novel discovery, already in exercise

Here’s simply for toy initiatives you sigh?

Here are about a examples:

ClickHouse
— a contemporary SQL server for analytical exercise-cases;
(written in C++;)
(inclined by many companies, from its guardian Yandex to CloudFlare;)
FFmpeg
— the mosey-to-tool for video compression and transcoding;
(written in C;)
firecracker
— Amazon’s light-weight VM change to QEMU,
inclined of their containers and lambda capabilities choices;
(written in Rust;)
deno
— the Deno language runtime (saner change to NodeJS);
(written in Rust;)
Python3
— custom Python3 static builds for Linux
(and dynamic, but with out dependencies, for others);
(written in C;)
bash
— custom bash static builds for Linux
(and dynamic, but with out dependencies, for others);
(written in C;)
Hugo
— the licensed static online page online generator;
(written in Scoot;)
kubectl
— the administration tool for Kubernetes;
(written in Scoot;)
Lego
— ACME (i.e. Let’s Encrypt) consumer;
(written in Scoot;)
DNSControl
— StackExchange’s Git apt DNS records administration tool;
(written in Scoot;)
cloudflared
— CloudFlare’s tool to place Argo tunnels, and other associated duties;
(written in Scoot;)
minio
— S3 compliant self-hosted server;
(written in Scoot;)
gh
— GitHub’s tool for repository associated duties;
(written in Scoot;)
restic
— licensed encrypted backups,
supporting many cloud services from AWS / Azure / Google
to BackBlaze and SFTP;
(written in Scoot;)
age
— licensed encryption tool;
(written in Scoot;)
jq
— put a question to of for JSON;
(written in C;)
minify
— minifier for HTML / CSS / JS;
(written in Scoot;)
hyperfine
— cli tool benchmarker;
(written in Rust;)
rebar3
— Erlang’s bundle manager and originate tool;
ninja
— originate tool change to win;
(written in C++;)

Sidenote about Scoot’s suitability for such exercise-cases

Hmm… Dangle you ever noticed one thing?

most are written in Scoot;
few are written in Rust / C / C++;
none are written in Python / Ruby / NodeJS / Java;

Surprise why?

A memoir for as soon as all all over again…
Nonetheless except then, right here is a hint although…

how easy is it to originate a static binary executable in Scoot?
simply run mosey originate -tags netgo ./cmd/some-tool.mosey;
how easy is it to snide-assemble the above?
simply prepend GOOS=darwin;
how easy is to construct that in Rust / C / C++?
let’s simply sigh that normally,
especially if the code is sufficiently shrimp,
it be less difficult to easily rewrite the damn thing in Scoot…

(Disclaimer: I extra or less abominate Scoot for assorted causes…
Nonetheless I construct exercise it mostly for the reason above…)

Languages fair correct for single binary executables

So, sigh I’ve convinced you about this single binary executable observe,
how can even simply aloof you proceed?

Looking out on the programming language:

Scoot — you are already lined,
simply don’t exercise non-pure libraries that depend on OS supplied libraries;
Rust — sigh goodbye to snide-compiling,
but in the occasion you live away from OS supplied libraries,
you are extra or less lined;
C / C++ — if truth be told probably,
but probabilities are you need to on retainer a UNIX graybeard,
or no no longer up to an autotools-foo grasp;
Python — you are already lined,
simply learn the zipapp documentation;
Erlang — you are already lined,
simply learn the escript documentation;
Deno — you are already lined;
NodeJS — technically, I are aware about it be probably;
but I will be able to almost wager my advantageous arm that given this day’s ecosystem
it be no longer feasible;
Ruby — don’t know;
but given Ruby’s focus (web pattern),
I would sigh it be no longer probably;
Zig / Nim — presumably (?);
Java — probably,
but probabilities are you need to a startup script to easily call java -jar some-tool.jar;
(furthermore no longer a legitimate fit for rapid-lived instruments, mainly as a result of startup times;)

Within the terminate, is it all price it?
Does it clear up the machine distribution downside?

Effectively, it be complicated…

Quick time-frame

I sigh that in the rapid time-frame,
it will probably if truth be told clear up
our on the spot machine distribution complications,
by simply having instruments that are easy to set up,
and simply figure out-of-the-box.

Very long time-frame

On the opposite hand, in the very long time-frame,
although I fetch no longer specialize in it be the advantageous resolution,
I if truth be told specialize in it pushes us in the lovely route…

As a parallel I am going to clutch systemd:
earlier than it used to be adopted, our services and products had been initialized by poorly written
and blunder-inclined bash scripts;
after it used to be adopted, upstream builders started paying extra
consideration to their services and products life-cycles and run-environments
(other service dependencies, start, terminate, restart, reload,
shrimp one processes, logging, and plenty others.).
On the identical time, I fetch no longer specialize in systemd (in its most up-to-date invent)
is the advantageous acknowledge.
On the opposite hand, no no longer up to it compelled us to spruce up
this part of our machine…

Now, coming assist to the single binary executable,
although I fetch no longer specialize in in the very long time-frame it be the advantageous resolution,
I construct specialize in this can force builders to pay extra consideration to their
dependencies, originate / take a look at workflows, and
normally enhance their initiatives from a logistical level of test.

Knowasiak