What an inference node is
An inference node is a process (or cluster of processes) that exposes an inference endpoint to the gateway and is reachable for a specific set of model identifiers at a specific trust tier. Nodes are tier-scoped: a Tier 1 node serves Tier 1 traffic; a Tier 3 node serves Tier 3 traffic; a node does not span tiers. Node populations:- Tier 1 nodes are operated by frontier providers and by major cloud inference services. Z is a client of these providers; the gateway forwards requests under Z’s contracts with them.
- Tier 2 nodes are operated by partner GPU networks running open-source models under contractual privacy commitments. Z does not operate Tier 2 hardware.
- Tier 3 and Tier 4 nodes run inside hardware-attested trusted execution environments at vetted TEE-providing partners. The active allowlist is published in the gateway documentation and updated when a node’s attestation status changes.
Routing
The gateway routes requests across the available nodes for a given (model, tier) combination. The routing logic operates on several signals: Node health. Each node is tracked through a state machine with three states: Active, Degraded, and Isolated. Only Active nodes receive traffic. A node that fails health checks transitions to Degraded or Isolated. Recovery from Isolation uses aprobe_pending flag; the node is promoted back to Active only after a successful probe.
Latency. Recent latency observations are used to bias routing toward nodes that are responding quickly.
Cost. Where multiple nodes can serve a request, the gateway preferences lower-cost paths unless the client has specified otherwise (for example by setting provider.sort=quality).
Quality. Third-party quality scores influence routing when the client has opted into quality-sorted routing. Quality score staleness degrades routing to health and price sorting but never blocks inference; a 503 fires only when there are zero viable nodes.
Affinity. For Tier 4, sessions are pinned to a specific node for the duration of the session, so that the ephemeral key exchange remains valid. Affinity rerouting is permitted before session-ID issuance; mid-session failover after issuance is prohibited on Tier 4.
Routing decisions are made per request. The gateway does not pin a client to a node across calls except on Tier 4 sessions, where affinity is required for correctness.
Retry and fallback
The gateway will automatically retry a failed request on the next-best available node, up to a maximum of three attempts across the combined retry and fallback chain. Retry triggers only on 5xx responses; 4xx client errors and 402 payment-required responses do not trigger retry or fallback. This budget is shared between the node-retry path and the developer-configurable fallback-model chain. The intent is to make transient node failures transparent to applications without permitting unbounded retry storms.Attestation
For Tier 3 and Tier 4, every node that serves traffic must present a valid hardware attestation before any payload is dispatched to the node. This ordering is mandatory: the request is not transmitted until attestation has been verified. The attestation chain proves that the code running inside the enclave is the code Z approved for that model and tier, that the hardware is genuine, and that the attestation has not been revoked. Attestation results are cached for a configurable TTL per node and code measurement. If a known vulnerability is published against a hardware vendor’s stack, an entry is added to the global revocation list and cached attestations matching that revocation are invalidated immediately. Propagation of the revocation across the gateway fleet completes within seconds via Redis pub/sub. A developer using Tier 3 or Tier 4 can independently verify the attestation: the attestation verification endpoint returns the raw TDX Quote in base64, and a developer with Intel DCAP tooling can verify it without relying on Z’s assertion.Who operates nodes
- Tier 1: frontier providers and major inference services. Z is a customer.
- Tier 2: partner GPU networks under contract. Z is a customer.
- Tier 3 and 4: vetted TEE-providing partners under an allowlist managed by Z. Z is a customer and an integrator.