Composition on a mesh

Composition

Each service, PostgreSQL, MariaDB, Redis or Nginx, is a first-class component that runs standalone or is consumed by another appliance. The stack that used to be LAMP becomes LEMP (Linux, Nginx, PHP-FPM and a database), and LEMP is a recipe built from components, not an image with an identity of its own.

Embedded in a simple installation; discovered on the mesh in an advanced one. A search index is derived data: it is rebuilt when a node joins and never replicated by file. Decisions 0023, 0033 and 0036.

Overlays

An overlay is one component: its packages, its configuration and its first boot scripts. Under decision 0039 each overlay is its own Debian source package, keel-overlay-<name>, released on its own with its own changelog.

Apache is not an overlay: it is retired as the default web server and migrated appliance by appliance. The WordPress test images built today still use it, until Keel PHP exists.

Appliances

Applications

An application appliance declares its runtime, the data services it consumes, the paths of its durable state and its workers. Workers are stateless: they read a queue and write to data services, never to local disk. Planned, Phases 4 and 6

Mastodon is the reference appliance: Ruby, Node.js, PostgreSQL, Redis, Sidekiq and usually object storage and search make it the heaviest of the set, so it is the proof of the composition model. It is built last in its phase, after WordPress, Odoo and Nextcloud (0035).

The manifest

Backup, monitoring, upgrades, directory replication and discovery all read the same facts: which paths hold data, which processes to watch, what a node offers. So the manifest format came first. Version 1 is decided, and its reader is the first step of the implementation (tracker#39). Decided, 0041

• Two kinds, one schema. An overlay manifest ships in each overlay's package and says what it runs. An appliance manifest ships in each appliance's package and says what it is built on and which overlays it adds.
• The manifest owns facts, the spec owns choices. A manifest never names a host, an address, a secret file or a count.
• A state per installation mode. Every overlay is enabled, disabled or ask in each of the three modes. The image is the same; the mode decides what runs.
• Inheritance is additive. Keel Web in a simple installation is one thing for every appliance built on it.
• Derived, never declared. Monit's checks, the firewall, the backup set, the Syncthing folders and the registry entry are rendered from the manifests.
• Versioned by an integer, with unknown keys refused at every level.

# /usr/share/keel/appliances/web.yaml, from keel-web
manifest_version: 1
kind: appliance
name: web
title: Keel Web
summary: Nginx, Coraza and Anubis, the base of every web appliance
base: core
overlays:
  nginx:  {simple: enabled,  cloud_simple: enabled, cloud_advanced: enabled}
  coraza: {simple: disabled, cloud_simple: enabled, cloud_advanced: enabled}
  anubis: {simple: disabled, cloud_simple: enabled, cloud_advanced: enabled}

From the version 1 format. Keel Web has no process of its own; its manifest is three lines of states over Core's.

The spec: one file is the truth

The instance spec, /etc/keel/instance.yaml, declares one machine: its names, IPv6 network, certificate policy, users, secrets by reference, database role and monitoring. It is rendered into the variables TurnKey's first boot hooks already read, so no hook has to know that it exists, and the keel command and the console are both clients of the same library. Working today

version: 1
instance:
  hostname: blog
  fqdn: blog.example.org
network:
  interfaces:
    eth0:
      ipv6:
        method: static
        address: 2001:db8:1::10/64
        gateway: fe80::1
  nameservers:
    - 2001:db8:1::53
secrets:
  root_password:
    file: /etc/keel/secrets/root_password

Decided next (0027, 0041): every installation, simple or advanced, by hand or automated, emits its complete YAML with every default written out, and re-emits it after every upgrade. The spec gains the appliance, the installation mode and the state of every overlay.

IPv6-first

Every appliance is reachable on a routable address of its own, such as 2001:db8:4b1::10, with an ACME certificate and its own host keys. No port mapping. IPv4 is optional and never assumed, and the project's examples and defaults use IPv6.

• Between nodes, native IPv6 goes direct. An IPv4-only node enters the mesh through a rendezvous point that translates (NAT64 with Tayga, in Debian 13).
• On the mesh, keel network wireguard suggest-address prints a random unique local IPv6 address with its /64 for the first node of an overlay.
• In the spec, nameservers are listed IPv6 first, and a database listens on literal addresses such as ::1, because on Debian localhost is an IPv4-only name.

Signed, content-addressed layers

An appliance is assembled from read-only layers, one per appliance in the chain. Each layer is a deterministic tarball with a sha256, listed in a plain-text layer manifest that is signed by the project key and tied to a git commit. The same commit and manifest give the same bytes, and an update downloads only the layer that changed.

• keel pull fetches the layers the cache lacks, from a directory or an https URL, checking every digest.
• keel assemble extracts a cached chain into a root filesystem and packs it as a Proxmox template with its sha512.
• keel verify checks installed layers against their manifests.
• Layers are published on two channels, stable and testing; a third channel would need a decision of its own.

Build measurements, on the build host with unmodified TurnKey 19.0 recipes: core layer 326 MB, LAMP stack delta 80 MB, WordPress app delta 133 MB, WordPress rebuilt on cached layers in 69 s instead of 451 s. On the home page.

Upgrades come from the Keel repository

Decided, 0039

• Everything Keel installs is a .deb in the Keel repository, overlays included, and apt upgrade is the only update mechanism.
• Application packages carry a post-install migration hook, such as Odoo's module update or Mastodon's db:migrate.
• Upgrades are ordered across the mesh: a database primary hands over to its standby, upgrades, and takes the role back after catching up.
• Two tracks, stable and testing, declared in the YAML. A simple installation defaults to stable, with unattended upgrades for security only.
• The archive is built from a dated, pinned pool, so every image rebuilds from the project's own archive.