Running pkgx packages on FROM scratch

This document records what it takes to run pkgx packages on a literally-empty base image (no libc, no shell, no coreutils), the conformance status of common packages, the fixes pkgm maintains to get there, and a proposal for the pkgx maintainers. It is a living document — extend it as new packages are tested.

Why packages don't "just run" on scratch

A pkgx package assumes a host system. Two structural gaps surface on scratch:

1. The interpreter path. Every dynamic ELF has PT_INTERP=/lib/ld-linux-* — an absolute system path that does not exist on a scratch image. pkgm run symlinks the pkgx glibc loader to /lib/ld-linux-* (and /bin/sh to itself) so parent and child ELFs resolve natively.

2. Incomplete runtime dependency graphs. A package's package.yml dependencies: is not its full runtime closure. Examples measured:

package	links (DT_NEEDED)	declares in package.yml
gnu.org/wget	libz	only openssl.org
curl.se	libz, libzstd, libnghttp2	(transitive only)
perl.org	libcrypt (libxcrypt)	—

pkgx compensates with build-time-detected metadata that is not exposed on dist.pkgx.dev (no deps.json; all such URLs 404). So pkgm rebuilds the closure from the packages' own DT_NEEDED — see below.

What pkgm does

pkgm run (and pkgm install --from-scratch) iterate to a fixpoint: scan the installed packages' ELF DT_NEEDED (via Go's debug/elf) and, for every soname not yet provided by the closure, pull the package that provides it:

• glibc set (libc, libm, libpthread, libdl, librt, …) → gnu.org/glibc
• libgcc_s, libstdc++ → gnu.org/gcc/libstdcxx
• libatomic, libgomp → gnu.org/gcc
• common undeclared libraries → a curated soname → project table (see sonameProject in closure.go): libz, libbz2, liblzma, libzstd, liblz4, ncurses/libtinfo, readline, libffi, pcre2, gettext, libiconv, libidn2, libunistring, libpsl, nghttp2, expat, libxml2, sqlite3, gmp, mpfr, libcrypt → libxcrypt, …

Wrapper scripts (#!/bin/sh, e.g. git) are run under the pkgx gnu.org/bash + gnu.org/coreutils, which pkgm installs and points /bin/sh at — pkgm is a package manager, so it uses the real pkgx shell rather than reimplementing one. (An earlier attempt embedded a pure-Go shell, but its set -e semantics diverged from dash/bash and broke git's wrapper; real bash behaves correctly.)

Conformance status (linux/aarch64, image = only the pkgm binary)

package	status	notes
gnu.org/bash	✅
gnu.org/wget	✅	needs libz (undeclared)
curl.se	✅	needs zstd + nghttp2 (undeclared)
sqlite.org	✅
gnu.org/grep	✅
gnu.org/sed	✅
gnu.org/gawk	✅
lua.org	✅
nodejs.org	✅	C++: libgcc_s + libstdc++
python.org	✅
perl.org	✅	needs libcrypt (undeclared)
git-scm.org	✅	#!/bin/sh wrapper → pkgm installs pkgx bash + coreutils
gnu.org/tar, tukaani.org/xz, facebook.com/zstd, lz4.org	✅	compression
rsync.samba.org, gnupg.org, vim.org, htop.dev	✅	incl. gnupg's crypto stack

Across ~20 diverse packages tested, every correctly-named package runs on scratch with no missing-library failures — the DT_NEEDED-driven closure plus the soname → project table cover the implicit graph, and wrapper scripts run under the real pkgx bash.

Pantry-wide audit

An audit of the whole pantry (linux/aarch64):

• Availability — 1818 of 1896 projects (96%) publish a linux package, so nearly the entire pantry is a FROM-scratch candidate.
• Runtime sample — a diverse 100-project sample run under pkgm run … --version: 39 PASS, 38 no runnable bin / no --version (libraries; harness bias), 20 non-zero exit on --version (mostly tools without a --version flag), and only 2 missing-library failures — both libabsl_* (abseil), in grpc.io (grpc_csharp_plugin) and mosh.org.

Those two are not closure gaps pkgm can fill: the packages were built against an abseil whose soname (…so.2501, …so.2401) no declared/available abseil version provides — grpc.io pins abseil.io: ^20250512 (soname 2505). This is a pantry package-vs-declared-dependency drift; pkgx itself would fail there too. No consumer tool can supply a soname no package ships.

Net: the DT_NEEDED-driven closure resolves the implicit graph across the pantry, with the only observed failures being upstream version-pin drift. The soname → project table also gained prefix matching (projectForSoname) for multi-.so projects such as abseil (libabsl_*), protobuf and re2.

Full-pantry conformance sweep (2026-07-28)

The 100-project sample above was extended to the entire set of package-backed projects. This section records the full run.

Provenance

field	value
pantry ref	ef7e60a3a2e6e07f7efeaf1e7c765fa670e66584 (2026-07-28T15:42:19Z)
go-pkgx/pkgm commit	77741ed (2026-07-28)
sweep date	2026-07-28
projects (N)	1818 (every project with a linux package)
method	docker run --rm pkgm-scratch run <project> -- --version on a FROM scratch image containing only the pkgm binary
verdicts	PASS / NOBIN / MISSING:<soname> / DLFAIL / OTHER:<rc> (see phaseB.sh)

Each project is run through pkgm run <project> -- --version inside a FROM scratch container whose only content is the static pkgm binary; pkgm assembles the closure from DT_NEEDED and executes the project's default binary. The verdict is derived from the exit code and stderr.

Results

verdict	count	honest interpretation
PASS	818	ran, exit 0 — genuinely runs FROM scratch ✅
NOBIN	556	run … --version found no matching binary — almost all are libraries with no runnable bin. Not a failure.
OTHER:1	218	binary ran and exited 1 — usually the tool has no --version flag. The binary launched and loaded its closure. Not a FROM-scratch failure.
OTHER:124	58	timeout killed it — the binary ran but --version blocked (REPL / server / waits on stdin). Not a failure.
OTHER:2	48	ran, exit 2 — typically needs a subcommand/args. Ran, non-zero; not a confirmed failure.
OTHER:255, :127, :3, … (misc)	35	ran, other non-zero rc; ambiguous (needs a subcommand/args). Not confirmed failures.
MISSING:<soname>	41	a shared library the closure did not provide (… .so: cannot open shared object). Real, actionable.
DLFAIL	40	pkgm hit a package-fetch failure resolving the closure (GET https://dist… / no package / no version). Real, actionable.
OTHER:139	4	SIGSEGV — a genuine crash. Real.
total	1818

The OTHER:* codes together account for 363 projects (218 OTHER:1 + 58 OTHER:124 + 4 OTHER:139 + 83 other misc rc). Of those, only the 4 SIGSEGV rows are FROM-scratch failures; the remaining 359 launched their closure and returned a non-zero rc or timed out on --version.

Headline: of 1818 package-backed projects run FROM scratch, 818 are confirmed-run (PASS, exit 0); a further 556 are libraries with no runnable binary (NOBIN) and 359 launched their closure but returned non-zero or timed out on --version (mostly tools with no --version flag, REPLs, or servers) — none of these are FROM-scratch failures. The real, actionable gaps are 41 MISSING + 40 DLFAIL + 4 SIGSEGV = 85 projects (4.7%); the other 1733 (95.3%) either ran or launched their closure cleanly.

Fix applied — soname-exact closure completion (pkgm 637f6db, 2026-07-28)

Re-running the 81 MISSING + DLFAIL projects after the fix, 73 now run FROM scratch (25 exit 0, 48 launch their closure and return non-zero/timeout on --version — not failures). All 40 DLFAIL are resolved; MISSING drops from 41 to 8.

Root cause: the closure completer mapped an unsatisfied soname to its provider project and pulled that project's latest package — but a soname carries an ABI version a newer release may have moved past (a binary needs libxml2.so.2, yet libxml2 2.15 ships libxml2.so.16; libssl.so.1.1 is openssl 1.1.x while the latest is 3.x). The provider was often already present as a declared dep at the wrong version, so completion skipped it. The fix walks the provider's versions and installs the newest that ships the exact soname (prioritising versions whose leading number matches the soname's ABI tail, so libssl.so.1.1 finds openssl 1.1.1w directly); the drifted and latest packages coexist on LD_LIBRARY_PATH. Ten soname mappings were also added (libFLAC, libtheora, libgit2, libcbor, libavro, libprotoc, libclang, libtinfow, libgflags, and the libboost prefix).

The 8 still-MISSING:

project(s)	soname	status
open-mpi.org, open-mpi.org/hwloc, openpmix.github.io, fnox.jdx.dev, solana.com	libudev.so.1	FIXED ✅. The only pantry provider was systemd.io (200+ binaries for one client library); a lightweight github.com/eudev-project/eudev recipe was added (pkgxdev/pantry#13922, merged) — its libudev.so.1 needs only glibc — and libudev→eudev + libcap.so.2→kernel.org/libcap are mapped. Verified FROM scratch: all 5 now run (fnox/hwloc/openpmix/solana exit 0; open-mpi launches its closure).
clickhouse.com	librt.so.1	modern glibc merged librt into libc; no standalone librt.so.1 is shipped
ladspa.org	(none)	its analyseplugin treats --version as a plugin name to dlopen — a probe artifact, the tool itself runs
tailwindcss.com	(none)	a Bun single-file binary that self-extracts a native lightningcss .node module to a virtual $bunfs at runtime — a Bun packaging quirk, not a pantry closure gap

So of the 85 original gaps, the fix closes ~73 outright; the 5 libudev projects are addressed by the new eudev recipe + mapping (pending its package publishing); the true residue is 1 librt (glibc absorbed it), 1 Bun quirk, 1 probe artifact, and the 4 SIGSEGV (a separate crash class, not addressed here).

MISSING — unresolved shared library (41)

The closure did not provide a .so the binary needs. Each is actionable: the soname either needs a soname → project mapping in closure.go, or the pantry genuinely ships no package providing that exact soname (version-pin drift, as with the abseil case above — pkgx itself would fail there too). Grouped by soname, most-common first.

libssl.so.1.1 — 10 projects link the OpenSSL 1.1 ABI; the pantry ships OpenSSL 3 (libssl.so.3), so no package provides …so.1.1. Version-pin drift.

• crates.io/cargo-tarpaulin → libssl.so.1.1
• crates.io/get-blessed → libssl.so.1.1
• crates.io/websocat → libssl.so.1.1
• fermyon.com/spin → libssl.so.1.1
• github.com/AppImageCommunity/zsync2 → libssl.so.1.1
• github.com/Virviil/oci2git → libssl.so.1.1
• lychee.cli.rs → libssl.so.1.1
• mise.jdx.dev → libssl.so.1.1
• pkgx.sh/cargox → libssl.so.1.1
• rpm.org/rpm → libssl.so.1.1

libxml2.so.2 — 9 projects (candidate for the soname → project table):

• augeas.net → libxml2.so.2
• freedesktop.org/appstream → libxml2.so.2
• imagemagick.org/v6 → libxml2.so.2
• isc.org/bind9 → libxml2.so.2
• qalculate.github.io → libxml2.so.2
• rpm.org/dnf5 → libxml2.so.2
• sourceforge.net/xmlstar → libxml2.so.2
• wayland.freedesktop.org → libxml2.so.2
• wireshark.org → libxml2.so.2

libudev.so.1 — 5 projects (systemd/udev; no pkgx package provides it):

• fnox.jdx.dev → libudev.so.1
• open-mpi.org → libudev.so.1
• open-mpi.org/hwloc → libudev.so.1
• openpmix.github.io → libudev.so.1
• solana.com → libudev.so.1

libFLAC.so.12 — 4 projects (candidate for the soname → project table):

• breakfastquay.com/rubberband → libFLAC.so.12
• github.com/libsndfile/libsndfile → libFLAC.so.12
• ladspa.org → libFLAC.so.12
• vamp-plugins.org → libFLAC.so.12

Single-project sonames:

• grpc.io → libabsl_die_if_null.so.2501.0.0 (abseil soname drift; see sample above)
• mosh.org → libabsl_log_internal_check_op.so.2401.0.0 (abseil soname drift)
• github.com/edenhill/kcat → libavro.so.23
• gnu.org/source-highlight → libboost_regex.so.1.82.0
• github.com/facebookincubator/fizz → libboost_regex.so.1.88.0
• developers.yubico.com/libfido2 → libcbor.so.0.13
• github.com/pantoniou/libfyaml → libclang.so.22.1
• amp.rs → libgit2.so.1.7
• github.com/protobuf-c/protobuf-c → libprotoc.so.25.7.0
• clickhouse.com → librt.so.1 (modern glibc merged librt into libc; the glibc package ships no standalone librt.so.1)
• xiph.org/libshout → libtheora.so.0
• gnu.org/texinfo → libtinfow.so.6 (wide-char ncurses; distinct from libtinfo.so.6)
• tailwindcss.com → (soname not captured — the sweep detected a cannot open shared object error but its message did not match the soname-extraction regex; needs a manual re-run to identify the library)

DLFAIL — package-fetch failure resolving the closure (40)

pkgm could not fetch a package it needs for the closure (GET https://dist… / no package / no version). Actionable: either a missing soname→project mapping points pkgm at a project/version with no package, or the required dependency genuinely has no published package. Dominated by large C/C++ apps with big plugin closures (gtk, php, ruby, imagemagick, mpv, httpd, …).

• agpt.co
• apache.org/httpd
• appium.io
• autotrace.sourceforge.net
• chiark.greenend.org.uk/puzzles
• crystal-lang.org
• crystal-lang.org/shards
• cython.org
• facebook.com/folly
• facebook.com/watchman
• getcomposer.org
• github.com/AUTOMATIC1111/stable-diffusion-webui
• github.com/libass/libass
• github.com/nat/openplayground
• gnome.org/gdk-pixbuf
• gnome.org/librsvg
• gnu.org/guile
• gtk.org/gtk3
• gtk.org/gtk4
• imagemagick.org
• jenkins.io
• laravel.com
• libvips.org
• llvm.org/clang-format
• mpv.io
• mvdan.cc/gofumpt
• openslide.org
• php.net
• phpmyadmin.net
• pwmt.org/girara
• pwmt.org/zathura
• riverbankcomputing.com/pyqt-builder
• riverbankcomputing.com/sip
• ruby-lang.org
• snaplet.dev/cli
• symfony.com
• symfony.com/cs
• terraform.io/cdk
• wxwidgets.org
• xpra.org

SIGSEGV — genuine crash (4)

The binary loaded and started but crashed (signal 11):

• github.com/marler8997/zigup
• gts.sourceforge.net
• mcgill.ca/lrslib
• pkgx.sh

Interpretation

NOBIN, OTHER:1 and OTHER:124 are not FROM-scratch failures. NOBIN projects are libraries with no runnable binary (an artifact of probing every project with run … --version); OTHER:1 binaries launched their closure and merely lack a --version flag; OTHER:124 binaries ran but blocked because --version dropped them into a REPL, server loop, or stdin wait. In all three the loader resolved PT_INTERP, the closure assembled, and the ELF executed — which is exactly what "runs FROM scratch" tests. Counting them as failures would understate conformance.

The 4.7% of real gaps are concentrated, not random. The MISSING and DLFAIL rows are dominated by large C/C++ applications with big plugin closures — gtk3/gtk4, php, ruby, imagemagick, mpv, httpd, libvips, librsvg, and similar. Those programs assemble much of their runtime graph through dlopen at runtime (image codecs, format plugins, language modules) rather than through DT_NEEDED at link time. A static closure completed from DT_NEEDED cannot see edges that only exist when the program dlopens a plugin by name during execution, so pkgm neither pulls those packages nor knows the soname to map. This is a known, structural limitation of DT_NEEDED-derived closures, and it is the single largest source of the remaining gaps. A second, smaller source is version-pin drift — packages built against a soname (libssl.so.1.1, the abseil …so.2401/2501 variants) that no currently-published package provides; pkgx itself would fail those too. libudev.so.1 and librt.so.1 are a third, minor source: system sonames the pkgx package set does not ship standalone.

This does not change the earlier conclusion; it quantifies it. The DT_NEEDED-driven closure resolves the implicit graph for the large majority of the pantry, and the residual gaps are honest, understood, and — for MISSING and DLFAIL — actionable through soname→project mappings or upstream package fixes.

Proposal to the pkgx / pantry maintainers

The information needed to run a package standalone (its complete runtime closure) exists at build time but is not published. Two complementary asks:

1. Publish the resolved runtime closure per package — e.g. a deps.json next to versions.txt on dist.pkgx.dev, listing the transitive project@version set actually linked (including the implicit libc/gcc/libz/… that package.yml omits). This lets any tool assemble a complete, relocatable environment without re-deriving it from DT_NEEDED.

2. Optionally, an explicit platforms/runtime declaration for the implicit system libraries a package links, so FROM scratch consumers are first-class.

go-pkgx/pkgm is a working proof of concept: a single pure-Go, CGO_ENABLED=0 binary that resolves the closure from DT_NEEDED and runs real packages (bash, curl, node, python, perl, …) on a FROM scratch image with zero system dependencies. See https://github.com/go-pkgx/pkgm.