Idempotency in Distributed Systems: Designing Safe APIs

Idempotency is the property of an operation where performing it multiple times produces the same result as performing it once. In distributed systems where networks are unreliable, clients retry requests, and messages can be delivered more than once, idempotency is essential for building safe and reliable systems. Without it, a retried payment request could charge a customer twice.

Understanding Idempotency

A function f is idempotent if f(f(x)) = f(x). In the context of APIs:

Idempotent: PUT /users/123 {name: "Alice"} - Setting a value is idempotent; doing it twice yields the same result
Idempotent: DELETE /users/123 - Deleting is idempotent; deleting twice still results in the resource being gone
Not Idempotent: POST /orders {item: "book"} - Creating a resource is not inherently idempotent; posting twice creates two orders
Not Idempotent: PATCH /accounts/123 {balance: "+100"} - Incrementing is not idempotent; doing it twice adds 200

HTTP Method	Idempotent?	Safe?	Notes
GET	Yes	Yes	Read-only by definition
PUT	Yes	No	Full replacement of resource
DELETE	Yes	No	Second delete is a no-op
POST	No	No	Needs idempotency key
PATCH	No	No	Depends on implementation

Idempotency Keys

The most common pattern for making non-idempotent operations idempotent is the idempotency key. The client generates a unique key for each logical operation and sends it with every request (including retries). The server uses this key to detect duplicates.

import uuid
import hashlib
from flask import Flask, request, jsonify
from datetime import datetime, timedelta

app = Flask(__name__)

# In production, use Redis or a database
idempotency_store = {}

@app.route("/api/payments", methods=["POST"])
def create_payment():
    idempotency_key = request.headers.get("Idempotency-Key")
    if not idempotency_key:
        return jsonify({"error": "Idempotency-Key header required"}), 400

    # Check if we already processed this request
    existing = idempotency_store.get(idempotency_key)
    if existing:
        if existing["status"] == "processing":
            return jsonify({"error": "Request is being processed"}), 409
        return jsonify(existing["response"]), existing["status_code"]

    # Mark as processing
    idempotency_store[idempotency_key] = {"status": "processing"}

    try:
        result = process_payment(request.json)
        response = {"payment_id": result.id, "status": result.status}
        idempotency_store[idempotency_key] = {
            "status": "completed",
            "response": response,
            "status_code": 201,
            "created_at": datetime.utcnow()
        }
        return jsonify(response), 201
    except Exception as e:
        del idempotency_store[idempotency_key]
        raise

Client-Side Implementation

import requests
import uuid
import time

class PaymentClient:
    def __init__(self, base_url):
        self.base_url = base_url

    def create_payment(self, amount, currency, max_retries=3):
        idempotency_key = str(uuid.uuid4())

        for attempt in range(max_retries):
            try:
                response = requests.post(
                    f"{self.base_url}/api/payments",
                    json={"amount": amount, "currency": currency},
                    headers={"Idempotency-Key": idempotency_key},
                    timeout=10
                )
                if response.status_code in (200, 201):
                    return response.json()
                if response.status_code == 409:
                    time.sleep(1)
                    continue
                response.raise_for_status()
            except requests.exceptions.Timeout:
                if attempt < max_retries - 1:
                    time.sleep(2 ** attempt)
                    continue
                raise
        raise RuntimeError("Max retries exceeded")

Database-Level Idempotency Techniques

Unique Constraints

-- Use unique constraint on idempotency key
CREATE TABLE payments (
    id BIGSERIAL PRIMARY KEY,
    idempotency_key VARCHAR(255) UNIQUE NOT NULL,
    amount DECIMAL(10, 2) NOT NULL,
    currency VARCHAR(3) NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'pending',
    created_at TIMESTAMP DEFAULT NOW()
);

-- Insert or return existing
INSERT INTO payments (idempotency_key, amount, currency, status)
VALUES ('key-abc-123', 99.99, 'USD', 'pending')
ON CONFLICT (idempotency_key)
DO NOTHING
RETURNING *;

Conditional Updates (Optimistic Concurrency)

-- Use version column for optimistic concurrency
UPDATE accounts
SET balance = balance - 100.00,
    version = version + 1
WHERE id = 12345
  AND version = 7;  -- Only succeeds if version matches

-- Check rows affected: 0 means concurrent modification

Absolute vs Relative Updates

-- NOT idempotent: relative update
UPDATE accounts SET balance = balance + 100 WHERE id = 123;

-- Idempotent: absolute update with idempotency tracking
INSERT INTO balance_adjustments (idempotency_key, account_id, new_balance)
VALUES ('adj-key-456', 123, 1100.00)
ON CONFLICT (idempotency_key) DO NOTHING;

UPDATE accounts SET balance = 1100.00 WHERE id = 123;

Payment Processing Example: Stripe-Like Flow

Here is how a payment processing system uses idempotency end-to-end:

class PaymentProcessor:
    def process_payment(self, idempotency_key, payment_data):
        # Step 1: Check idempotency store
        existing = self.db.query(
            "SELECT * FROM idempotency_keys WHERE key = %s",
            [idempotency_key]
        )
        if existing:
            if existing.status == "completed":
                return existing.response_body
            if existing.status == "processing":
                raise ConflictError("Request in progress")

        # Step 2: Create idempotency record in same transaction
        with self.db.transaction() as txn:
            txn.execute(
                "INSERT INTO idempotency_keys (key, status, request_body) "
                "VALUES (%s, 'processing', %s)",
                [idempotency_key, json.dumps(payment_data)]
            )

            # Step 3: Create payment intent
            payment = txn.execute(
                "INSERT INTO payments (idempotency_key, amount, currency) "
                "VALUES (%s, %s, %s) RETURNING id",
                [idempotency_key, payment_data["amount"],
                 payment_data["currency"]]
            )

            # Step 4: Call payment gateway
            gateway_result = self.gateway.charge(
                amount=payment_data["amount"],
                card_token=payment_data["card_token"],
                reference=payment.id
            )

            # Step 5: Update payment and idempotency record
            txn.execute(
                "UPDATE payments SET status = %s, gateway_id = %s "
                "WHERE id = %s",
                [gateway_result.status, gateway_result.id, payment.id]
            )

            response = {"payment_id": payment.id,
                        "status": gateway_result.status}
            txn.execute(
                "UPDATE idempotency_keys SET status = 'completed', "
                "response_body = %s WHERE key = %s",
                [json.dumps(response), idempotency_key]
            )

            return response

Idempotency Key Storage with Redis

import redis
import json

r = redis.Redis()
KEY_TTL = 86400  # 24 hours

def check_idempotency(key):
    result = r.get(f"idem:{key}")
    if result:
        return json.loads(result)
    return None

def store_idempotency(key, response, status_code):
    r.setex(
        f"idem:{key}",
        KEY_TTL,
        json.dumps({"response": response, "status_code": status_code})
    )

def mark_processing(key):
    # SET NX ensures only one request processes
    return r.set(f"idem:{key}", json.dumps({"status": "processing"}),
                 nx=True, ex=300)  # 5-min timeout for processing

Best Practices

Key generation: Clients should use UUIDv4 or a deterministic hash of the request content
Key expiration: Store keys for 24-72 hours, then allow re-processing
Fingerprinting: Hash the request body alongside the key to detect misuse (same key, different payload)
Error handling: Do not store idempotency records for failed requests — allow retry
Scope: Idempotency keys should be scoped per user/API-key to prevent cross-user collisions

Idempotency works hand-in-hand with distributed transactions and is an alternative to distributed locks in many scenarios. For more on building reliable APIs, see our rate limiting guide. Use the System Design Calculator to estimate storage requirements for your idempotency key store.

Frequently Asked Questions

Q: How long should I keep idempotency keys?

Stripe keeps idempotency keys for 24 hours. For most systems, 24-72 hours is sufficient. The key expiration should be longer than any realistic retry window. After expiration, the same key can be reused for a new operation.

Q: What if the same idempotency key is sent with different request bodies?

This is a client error. Store a hash of the original request body alongside the idempotency key. When a duplicate key arrives with a different body, return a 422 Unprocessable Entity error to alert the client of the misuse.

Q: Is idempotency the same as exactly-once delivery?

No. Exactly-once delivery means a message is delivered precisely once (very hard to guarantee). Idempotency provides at-least-once delivery with at-most-once processing — messages may arrive multiple times, but the effect occurs only once. This is often called "effectively exactly-once" processing.

Q: How does idempotency relate to the Saga pattern?

In a Saga pattern, each step and its compensating action should be idempotent. If a saga step fails and is retried, idempotency ensures the retry does not create duplicate side effects. The compensating transaction also needs to be idempotent in case it is retried.

Idempotency in Distributed Systems: Designing Safe APIs