Skip to main content
Version: 0.1.0-alpha.5 (alpha)

Single-node admission and overload control

latent-admission implements the Phase 1 AdmissionController and local QuotaProvider contracts. The controller rejects invalid, unauthorized, incompatible, infeasible, and overloaded work before execution-cell allocation. It creates no execution cell, cancellation registration, artifact preparation, worker, runtime, socket, service-specific queue, or periodic task.

The embedded deployment catalog supplies immutable typed execution policy via RevisionPolicySource. The fair scheduler consumes the permit and retains its reservation with the assigned cell. The delivered activation manager composes these owners in the standalone node, which serves the authenticated Phase 1 invocation API.

Ownership and trust boundaries​

Create one LocalQuotaProvider per node and share clones with every local admission controller. The provider owns immutable startup policy and one mutex protecting all node, tenant, trust-class, queue-class, and cell-backlog counters. Changing the route view with with_policy_source retains this same ledger. Creating independent providers for services or new route generations would create independent limits and is not a supported node composition.

AdmissionRequest is an internal input from an authenticated adapter, not an untrusted wire identity. The adapter must authenticate the principal, derive its tenant and service identity, and measure the actual payload length. It must not copy an untrusted Content-Length or derive a principal from arbitrary metadata. Admission checks exact tenant equality, a configured subject/kind allowlist, non-anonymous identity, bounded identifiers, and a service identity for service principals. Claims and attributes cannot grant access, priority, trust, or cells. External identity providers and a general policy language are not implemented.

Resolve and admit against the same PinnedRouteResolver:

let pin = std::sync::Arc::new(deployments.pin()?);
let revision = latent_routing::RouteResolver::resolve(
pin.as_ref(), &envelope.target, routing_key,
)?;
let controller = node_admission.with_policy_source(pin);
let permit = controller.admit_now(latent_admission::AdmissionRequest {
activation_id: envelope.activation_id.clone(),
principal: envelope.principal.clone(),
revision,
requested_budget: envelope.budget.clone(),
deadline_unix_millis: envelope.deadline_unix_millis,
payload_bytes: envelope.input.len().try_into().expect("payload length fits u64"),
priority: envelope.priority,
attributes: envelope.metadata.clone(),
})?;

The catalog verifies the entire tenant/service/route/contract/function/revision/ release/generation tuple. It does not reselect a revision with a different routing key. Forged ResolvedRevision::attributes do not provide deployment policy. The returned permit removes these untrusted attributes and exposes typed, immutable obligations instead. A held catalog view continues serving the exact old policy after deployment replacement or deletion; a newer view rejects an old generation. Pins are trusted-local lifetime capabilities, not externally serializable bearer tokens. The embedding owner must not retain unbounded historical pins.

Catalog compilation retains a small charged policy record per revision, not a full artifact, contract tree, or component bytes. Lookup is a bounded immutable index operation with no filesystem access or compilation. Persisted catalog format, checksums, logical revision identities, and route ordering are unchanged; restart rebuilds the same typed policy from verified release metadata.

Configuration and exact limit semantics​

There is no permissive default tenant or wildcard authorization. Configuration is explicit through NodeAdmissionPolicy; LocalQuotaProvider::new validates it before retaining any activation state. Policy is immutable for the ledger's lifetime. Changing a controller's catalog source does not change its node policy.

ConfigurationMeaning
budget_ceilingHard per-activation node budget, including a positive finite relative wall-time ceiling.
limits on node, tenant, and trust classIndependent maximum live activations, queued reservations, reserved CPU fuel, and reserved memory bytes.
maximum_payload_bytes on node and tenantIndependent exact byte ceilings; equality is accepted.
maximum_priority on node and tenantIndependent authorization ceilings. Every allowed priority must map to exactly one configured queue class.
queue_classesNamed, nonoverlapping priority ranges with independent queue bounds.
cell_classesStartup-defined capabilities for a subset of tiny, small, standard, large, and extra-large. Each configured class has positive memory capacity and usable parallelism, supported threading models, and features.
Tenant subject/kind/trust/cell allowlistsExact authorization; empty allowlists deny access. Extra-large requires explicit tenant and trust-class permission.
maximum_identifier_bytesBounds caller-selected activation/target names; empty, whitespace-containing, and control-containing identifiers are rejected. Generated revision/release identities have a separate bounded allowance of at least 83 bytes and must match the pinned catalog exactly.
maximum_metadata_entries / maximum_metadata_bytesAggregate bounds across principal claims, request attributes, and resolved-revision attributes, checked before retaining quota state.
overloadIndependent CPU and memory pressure thresholds and maximum observation age. Pressure at or above its threshold rejects admission.
deadlineEstimated service time, minimum useful execution time, and a safety margin for bounded-backlog feasibility.
Architecture, region, zoneNode placement identity checked against the pinned deployment restrictions.

Every numeric quota is finite and exact. Zero grants no capacity. In particular, zero queue capacity disables admission at that scope; it does not mean an unbounded queue or an immediate-execution-only mode. Every admission initially reserves a queue slot, even when a downstream cell is immediately available.

active_activations means queued plus executing reservations, not only guest execution. CPU fuel and granted memory are reserved at admission for both states. They measure concurrent promised capacity, not CPU billing or a replenishing rate bucket. QuotaSnapshot::reset_at_unix_millis is therefore None. Completion returns the entire reservation, regardless of actual consumption; activation consumption is separately finalized by ActivationBudget from #6.

The budget grant is the bounded intersection of the request, capsule ceiling, deployment ceiling, and node ceiling. CPU or memory zero makes execution infeasible. Later-phase resource dimensions must be zero in Phase 1 requests. An omitted relative wall-time request adds no restriction at that layer; it cannot bypass the node's finite relative ceiling. An absolute caller deadline is separate and can only shorten the grant.

The selected cell is the smallest memory-capable, threading-compatible, feature-compatible class permitted by both the tenant and trust class. Ties use the fixed tiny-to-extra-large class ordering. Admission does not reserve a cell or grow a pool; CellClassPolicy::parallelism must match the downstream usable pool capacity. A node that loses usable capacity must publish conservative queue delay or stop accepting work until configuration and scheduling agree.

Deadline feasibility and node overload​

A grant binds wall time to a monotonic deadline once, at admission. For a class with n existing live reservations and configured usable parallelism p, the local queue estimate is:

local_wait = floor(n / p) * estimated_service_time
required = max(local_wait, observed_queue_delay)
+ minimum_execution_time + safety_margin

Admission rejects when the remaining monotonic duration is less than or equal to required. All arithmetic is checked; overflow rejects instead of wrapping. The quota critical section includes this calculation, so concurrent callers see previously accepted reservations. Live admission resamples monotonic time after policy lookup and quota-lock contention, without moving the original deadline. Expired grants and stale observations cannot become accepted merely because a previous clock sample was valid. admit_at is an explicitly trusted deterministic clock seam; do not expose it to a caller.

This is a conservative configured estimate, not a completion-time guarantee. Use measured service-time assumptions and conservative load input. The scheduler must still check the original deadline and cancellation immediately before cell allocation and handoff. Priority ordering/fairness belongs to #8, not admission.

NodeLoadSource supplies node-wide trusted observations. NodeLoadState is an externally updated shared value, not a monitoring worker. Missing/unavailable, malformed, future-dated, stale, or out-of-order samples fail closed. Sampling is not an atomic transaction with hardware pressure: a new overload after the sample remains a downstream scheduling/execution concern. Never populate load observations from request metadata. Diagnostic errors disclose no load numbers.

Permit lifecycle and downstream obligations​

AdmissionPermit and ExecutionPermit are non-cloneable and cannot be minted by callers. Their identity, effective budget, deadline, and obligations are exposed only through immutable accessors. A compile-fail test protects affine ownership.

  1. Admit before cancellation registration, queue insertion, or cell allocation. The successful permit owns the queue/concurrency/CPU/memory/trust reservation.
  2. Transfer ownership of the permit with the queued work. On enqueue failure, queued cancellation, expiry, panic, or abandoned future, drop that permit.
  3. Immediately before cell allocation, check cancellation and ensure_schedulable_at. After obtaining a lease, consume the queued permit with start_execution. It resamples the live monotonic clock under the quota lock. On success only queue capacity is returned; other reservations remain. On error the permit is dropped, and the scheduler must also release or quarantine its independently owned lease. start_execution_at is the trusted deterministic-clock equivalent, not a caller-supplied timestamp.
  4. Enforce the exact selected cell/queue/trust class, priority, backend, threading/state model, features, call-depth limits, all granted budget dimensions, and the original deadline. Initialize accounting from effective_budget; never reset the relative wall-time ceiling at dequeue.
  5. Retain the execution permit until the guest has stopped, accounting has been finalized, and the cell has been released or quarantined. Then drop it.

The generalized activation owner must retain the permit in the task that actually owns execution. Dropping a transport future while detached execution continues must not release that task's quota prematurely. Admission deliberately does not create a second cancellation registry or attempt to kill execution from Drop. The existing affine cell lease remains the authority for reuse/quarantine.

All reservation checks and checked additions happen before ledger mutation. Duplicate activation IDs cannot replace a live reservation or release another call's capacity. Only configured tenants may create counters, and zero-use entries are removed. Live bookkeeping is bounded by the configured node concurrency limit, identifier sizes, and the fixed configured class/tenant sets. No map or queue grows with rejected service cardinality.

Structured failures​

Admission errors use the existing platform codes and one admission.limit detail with stable scope, dimension, and reason fields. Reasons identify the limiting dimension without echoing principal names, tenant identifiers, request attributes, raw payloads, filesystem paths, or other tenants' counters.

FailureTypical platform code / reason
Unknown/unauthorized principal or tenantpermission-denied / principal-not-authorized
Unauthorized priority or trust/cell classpermission-denied / corresponding *-not-authorized reason
Invalid identifiers, metadata, unsupported request budgetinvalid-argument / dimension-specific reason
Missing or mismatched pinned revisionadmission-rejected / revision-not-available
Unsupported capsule requirements or placementadmission-rejected / unsupported-execution-requirement, placement-not-compatible, or no-compatible-cell
Payload too largeresource-exhausted / payload-too-large (not retryable unchanged)
Reserved concurrency, queue, CPU, or memory exhaustedresource-exhausted / capacity-exhausted
Node pressure threshold reachedresource-exhausted / node-overloaded
Queue cannot reasonably meet deadlineadmission-rejected / queue-deadline-infeasible
Expired deadlinedeadline-exceeded / deadline-exceeded
Load or quota source unavailableunavailable / source-specific sanitized reason
Duplicate live activation IDalready-exists / activation-id-unavailable

Retryability is set for transient unavailability, occupied capacity, node overload, and queue infeasibility. It is not set merely because the platform code is resource-exhausted; increasing retries cannot fix an oversized payload or an intrinsically infeasible zero budget.

QuotaProvider::snapshot and aggregate usage observations are trusted-local administrative surfaces. Their embedding API must separately authorize access to the requested tenant; they are not included in rejection details.

Validation​

Run the feature tests, affine compile-fail test, catalog integration, and existing pool/runner regressions with:

cargo test -p latent-admission --locked
cargo test -p latent-control-store --lib --locked
cargo test -p latent-node -p latent-scheduler --locked
cargo clippy -p latent-admission --all-targets --locked --no-deps -- -D warnings

Admission unit tests cover independent limits, exact boundaries, all unsupported budget dimensions, class selection, sanitized errors, ownership, stale load, deadline arithmetic, and coordinated concurrent reservations. An isolated Linux regression submits 1,000 distinct rejected service targets and completes 100 reservations, checking unchanged thread/descriptor/socket/child counts and empty retained tenant counters. The child has a fixed deadline and is reaped on failure. The larger 100,000-target variant is ignored by ordinary CI and requires explicit selection:

cargo test -p latent-admission --locked \
tests::stress::admission_state_does_not_grow_with_100k_rejected_services -- \
--exact --ignored --nocapture

Catalog tests verify every pinned identity field, deterministic weighted policy selection, generation replacement/deletion, restart, and no artifact access on admission. Additional tests exercise the real Phase0ActivationRunner and FixedCellPool with explicitly injected backend success, declared error, trap, deadline/fuel interruption, platform failure, and uncertain cleanup. They verify cell disposition and quota/cancellation cleanup after terminal outcomes, real queue overflow, queued/running cancellation, and dropped futures. This is not a claim of new Wasmtime execution coverage; the retained Phase 0 runtime suite continues to supply that evidence.

These tests belong to the maintained workspace CI, not a permanent workflow for each completed issue. No WIT, Protobuf, JSON Schema, SDK, or persisted catalog format changes are required. The Rust admission structs intentionally evolve: caller deadlines are explicit, snapshots include queued reservations, and permits become affine capabilities instead of cloneable freely constructible metadata.