Fail open vs fail closed

Reference draft

This page comes from our reference drafts and is still being reviewed. Behaviour described here is the current build; defaults can change before launch.

Sometimes your code asks for a decision and Ordainer can’t answer: the call times out, the network fails, or our server has an error. You decide in advance what happens then:

	Fail open	Fail closed
When Ordainer can’t answer	Run the tool.	Don’t run the tool.
Your customers see	Nothing; the tool works.	An error, for as long as the problem lasts.
The risk	Calls run unmetered while it lasts.	Our outage becomes your outage.
Good for	Most tools: availability matters more than a few unmetered calls.	Tools where an unmetered call is expensive or unsafe.

This is a client-side setting. The server can’t fail open for you, because when it can’t be reached, it isn’t there to answer. Nothing is written to the audit log for calls made while there was no decision.

What fail mode covers

• A timeout (your client’s timeout; the Node SDK’s default is 300 ms).
• A network error: DNS, connection refused, TLS.
• A server error (5xx, including 500 internal_error).
• A 200 whose body isn’t a decision.

What fail mode doesn’t cover

A deny is a decision. quota_exceeded, tool_not_in_plan and test_limit_reached are 200 answers, and fail mode doesn’t apply to them. What to do with a deny is the plan’s mode (see Shadow vs enforce), test_limit_reached included: it blocks in enforce and runs in shadow.

An error answer isn’t an outage. When the API answers with a 4xx, it did answer: the key is wrong or revoked, the account has never paid, the request is malformed, the idempotency key conflicts, or a rate limit was hit (unknown_key, revoked_key, subscription_required, invalid_request, invalid_json, invalid_idempotency_key, idempotency_conflict, rate_limited, …). Failing open on these would let a broken setup run with no cap at all, so the Node SDK blocks them in every failMode. Retrying doesn’t help (except after Retry-After on rate_limited); fix the cause. There are two exceptions. Your own lapsed subscription (subscription_inactive) follows the separate billing-lapse setting, not failMode (below). And 429 safety_limit_reached (shadow calls past the safety ceiling) is a safety limit, not a deny: SDK 1.2.0 runs the call in every failMode.

In the Node SDK

• failMode: 'open' is the default. When Ordainer can’t answer, the call runs (except during a sticky deny, below), a warning is logged, and result.failOpen is true. Warnings are throttled: each kind (fail open, fail closed or rejected, per error code) at most once a minute, with “(N more since the last warning)”. Set the gap with warningIntervalMs (default 60000). Count failures in onDecision, which sees every call.
• failMode: 'closed' blocks when Ordainer can’t answer (timeout, network error, 5xx such as 503 db_timeout, unreadable answer). The middleware answers your caller 503 ordainer_unavailable with Retry-After: 5 (“This call wasn’t run. The service couldn’t finish a safety check it does before each call, so it stopped instead of going ahead without it. Nothing is wrong on your side. Please try again in a minute. (ordainer_unavailable)”), and result.failClosed is true. Since SDK 0.2.0 that’s the only time the SDK answers 503.
• Set it on the constructor, or per call on middleware(), wrap() or decide().
• retries (0 to 5, default 1) retries a timeout, a network error or a 503 (such as decide_timeout, db_timeout or db_unavailable) with the same idempotency key, after a 50 ms pause, before falling back to failMode, so a check that reached us is replayed, not counted twice. Other 5xx and all 4xx are never retried. All attempts share retryBudgetMs (default 650).
• Any 4xx, or no agent key or tool configured, is blocked in every failMode: your caller gets plan_check_rejected with no Retry-After, 403 for billing (subscription_required) and 500 for everything else (“The plan check refused this call because of a setup problem on the service’s side, so it was not run. This is not an outage: retrying will not help until the service operator fixes its setup.”), or 429 rate_limited with the server’s Retry-After for the per-key rate limit. The fix is on your side: check the agent key and your setup; retrying won’t help. result.rejected is true (failClosed stays false; at most one of failOpen, failClosed and rejected is true), and a [ordainer] rejected (call blocked in any failMode): <code>: … warning is logged. A 4xx is never retried.
• 403 subscription_inactive (your subscription is inactive) follows the billing-lapse setting billingLapse, not failMode (Terms 10.3(b)): with 'open' (the default, whatever failMode is) every call with your live keys runs, unchecked and uncounted; with 'closed' (opt-in) it’s blocked with 403 plan_check_rejected, no Retry-After and a neutral message for your users (result.rejected). Its own warning says “Calls are being let through, unchecked, because billingLapse is ‘open’ (the default)” or “Calls are being blocked (403 plan_check_rejected) because billingLapse is ‘closed’”. 403 subscription_required (never paid) is blocked under every setting. Both get the onBillingInactive hook on every such call.
• 429 rate_limited (About 100 requests a second, each server counts separately. result.error.limitScope is 'per_key') is blocked with 429 rate_limited and Retry-After. It gets its own “rate limit, not an outage” warning (at most once a minute), result.rateLimited / result.retryAfterSec, and the onRateLimited hook.
• 429 concurrency_limited (the key already has 4 requests in flight at once on that server on our hosted service, with up to 20 more waiting briefly; result.error.limitScope: 'per_key_concurrency') is handled the same way: blocked with 429 rate_limited and Retry-After: 1, never a fail-open, result.rateLimited true, result.reason 'concurrency_limited', onRateLimited runs, and the warning says “concurrency limited … Send fewer calls in parallel”. Agents making parallel tool calls on one key can hit it.
• 429 too_many_failed_auth (your address sent more than 30 missing, malformed or unknown keys in a minute, and this call’s key is missing or unknown too; result.error.limitScope: 'per_address') is a bad key, not a rate limit: rejected like other 4xx (500 plan_check_rejected), with its own warning to check the agent key. Calls with a valid key aren’t refused by it (one rare exception).
• 401 old_key_format (an sk_ agent key from before the rename) is rejected like any other 401 (500 plan_check_rejected). Create a new ord_ key.
• test_limit_reached doesn’t depend on failMode: in enforce it’s blocked (429 test_limit_reached); in shadow (SDK 1.2.0) the call runs as a would-deny.
• 429 safety_limit_reached (SDK 1.2.0; shadow calls past 10 times your monthly test limit) runs the call in every failMode, 'closed' included: result.allowed and result.safetyLimited are true, result.failOpen is false, a warning is logged at most once per warningIntervalMs, and onSafetyLimit(result) runs on every such call.
• If the baseUrl host doesn’t resolve (ENOTFOUND / EAI_AGAIN, e.g. a typo in ORDAINER_BASE_URL), SDK 1.2.0 logs one error-level line the first time, naming the host and saying to set ORDAINER_BASE_URL, not held back by the warning throttle. Otherwise it’s a network error and follows failMode.

Sticky deny

Sticky deny. Once Ordainer has denied an agent key and tool with quota_exceeded, a call for that same pair that would fail open in the next 60 seconds (a timeout, network error, 5xx or unreadable answer) is blocked instead, as a plan-limit deny: your caller gets 429 plan_limit_reached (or your denyStatus) with Retry-After, and result.stickyDeny is true (result.error still says what failed). This stops a customer who is already at its limit from running uncounted calls while Ordainer is slow or unreachable. The window ends early at the plan’s reset or when an allow arrives for the pair; set it with stickyDenyMs (default 60000, max 300000, 0 = off). In shadow mode the call still runs and is reported (stickyDeny, wouldDeny). failMode: 'closed' is unchanged.

Sticky deny lives in memory, per Ordainer instance. A fresh process, a new replica or a cold serverless start that hasn’t received the deny itself still fails open, even for a customer already over its cap. If your caps must hold during an outage and you run on serverless or many short-lived instances, use failMode: 'closed' for those routes. Sticky deny narrows the fail-open gap; it doesn’t close it.

With plain HTTP (any other language)

There’s no setting; it’s the branch you write for “error or timeout”. The Your first deny examples print the error and exit 1, leaving the choice to you. Whatever you pick:

• Set a client timeout that fits your request path. Don’t rely on your HTTP library’s default, which can be minutes.
• Expect the first call from a new process to be slower: make one warm-up request at startup, and on a timeout retry once with the same idempotency key. More in Introduction → When we can’t answer.
• Treat a 4xx as “don’t run”, like the SDK does. Only a timeout, network error or 5xx is a real “Ordainer can’t answer”.
• Log error.code and the X-Request-Id header, and alert on a rising error rate. Support can trace a 4xx from its request ID.
• If you retry, reuse the same idempotency key so the call can’t be counted twice (a key replays for 24 hours). On 429 rate_limited, wait Retry-After seconds first.

Latency and status

We don’t publish a latency number yet; we’ll publish one once it’s measured. There’s no status page yet either. Size your timeout from your own measurements.