Creating an API key is a request you can't safely repeat. If the answer is lost on the way back, the caller doesn't know whether a key was made, and retrying blindly can make a second one: a live credential nobody holds the secret for and nobody knows to revoke.
The usual answer is an Idempotency-Key header. The caller picks a value for the operation and sends it on the first try and every retry. The server remembers what it answered and gives the same answer again instead of creating twice.
Canopy has supported it for a while on the routes that create API keys, OAuth clients, identities and invites. When we read our own code closely, it held up for one kind of retry and not the other.
What already worked
The first request ran, and its response was stored in Redis for 24 hours, under the Application, the calling principal and the key. A later request with the same key got that stored response back, raw secret included, so the caller could finish what it started.
Sending the same key with a different request was refused with 422. We digest the method, the path and the body, so a key reused by mistake can't be answered with another request's response.
The scope matters as much as the replay. The calling principal is part of the cache key, so one API key's stored response, with the secret it minted, is never handed to another caller that happens to pick the same value.
What didn't
The response was stored after the request finished. Nothing marked the key as taken while the request was still running.
That's fine when a retry comes after the first answer is stored. It's not fine for the retry you actually get: a client gives up waiting, the first request is still working, and the retry arrives. Both look in the cache, both find nothing, and both create a key.
A Redis outage had the same effect. The request ran as if no key had been sent, so a retry during the outage could create twice, and nothing told the caller.
Claim the key first
The fix is to take the key before running anything. A request claims it with one atomic Redis SET NX, writing a marker that says "in flight" and a token only that request knows. Then it runs.

What a request with an Idempotency-Key meets now:
nobody holds the key claim it, run, store the response, answer
a finished request replay its response (24 hours)
a request still running 409 idempotency.request_in_progress
a request that vanished 409 idempotency.outcome_unknown
a different request 422 idempotency.key_reused_with_different_request
Redis can't be reached 503 idempotency.unavailableThree details made this correct rather than just shorter:
- The response is stored before the answer goes out. A retry that arrives the instant the first answer lands finds the stored response, not an empty key.
- A failure gives the key back. A
4xx, a5xxor an exception removes the marker, so the caller can fix the request and retry with the same key. - Only the holder can touch the key. Storing and releasing are small Redis scripts that check the token first, so a slow request can never overwrite or delete what a newer one wrote.
The case we chose not to guess
A request can die between claiming the key and recording what happened: a deploy, a crash, a lost connection to Redis. Did it create the key or not? Only that request knew.
The easy option is to let the marker expire and hand the key to the next retry. That's the duplicate the key exists to prevent. So a marker older than five minutes, longer than any request runs, is treated as abandoned, and a retry gets 409 idempotency.outcome_unknown. The caller checks whether the key exists, then sends a new request with a new key.
It's a worse experience for one rare case, and an honest one. We'd rather tell a caller "we don't know" than create a credential they never see.
Redis outages get the same reasoning. A caller who sends the header is asking for a guarantee, so when we can't keep it the request is refused with 503 and can be retried. A request without the header never touches Redis and runs as before.
Sealing the replay
For a create, the stored response holds the raw secret, so it's sealed with AES-256-GCM under its own key, which production refuses to start without. A copy of Redis alone doesn't give up a secret.
One limit stays: if Redis itself loses the entry (a flush, or a restart without persistence), the key is free again. That's a property of where the state lives, and we say so in the code rather than pretend otherwise.
How we know
An end-to-end test fires six identical create requests with one key at the same time. Exactly one API key exists afterwards, every answer that created something carries the same key, and the rest are 409 request_in_progress. Further tests check that a later retry gets the same secret back, that Redis holds no raw secret, that a different request with the key is refused, and that a request whose outcome was lost is never run again.
The short version
- Storing the response after the request finishes protects late retries, not overlapping ones. Claim the key before you run.
- Store the response before you answer, and give the key back on failure.
- When you can't know whether a request did its work, refuse the retry. Don't guess.
- A stored response that carries a secret should be encrypted, and scoped to the caller that made it.
- If the store is down, refuse the keyed request rather than run it unprotected.
Every create that accepts Idempotency-Key says so on its API reference page. How to create and manage keys is in API Keys.