Token rate limiting on egress

This guide walks through applying TokenRateLimitPolicy (TRLP) to an Istio egress gateway to control AI token consumption for outbound LLM API calls. It covers global token limits, per-workload budgets using workload identity, and per-tier quotas.

TokenRateLimitPolicy uses the same wasm-shim and Limitador infrastructure as RateLimitPolicy. The gateway extracts usage.total_tokens from AI API responses and counts them against configured limits. No code changes are needed for egress versus ingress.

Prerequisites

• Kubernetes cluster with Kuadrant operator and Istio installed. See the Getting Started guide.
• Egress gateway infrastructure deployed. Run the base setup first:

curl -sL https://raw.githubusercontent.com/Kuadrant/kuadrant-operator/refs/heads/main/hack/setup-egress.sh | bash

See the Egress Gateway Setup guide for details on what this deploys.

Then deploy the mock AI API and test workloads:

curl -sL https://raw.githubusercontent.com/Kuadrant/kuadrant-operator/refs/heads/main/hack/setup-egress-ai-mock.sh | bash

This deploys:

Resource	Value
Mock AI API	llm-sim in ai-mock namespace (no Istio sidecar)
External hostname	api.ai-mock.local
Gateway	kuadrant-egressgateway in gateway-system
Test clients	test-client (default SA), team-gold (team-gold SA) in egress-test

Export the gateway address:

export EGRESS_IP=$(kubectl get gtw kuadrant-egressgateway -n gateway-system \
    -o jsonpath='{.status.addresses[0].value}')

Verify the mock AI API is reachable:

kubectl exec test-client -n egress-test -- \
    curl -s -H "Host: api.ai-mock.local" http://${EGRESS_IP}/v1/models

Mock AI API

The setup deploys llm-d-inference-sim, an OpenAI-compatible simulator, as a cluster-internal service outside the Istio mesh. The egress gateway reaches it via a ServiceEntry with static IP resolution. Responses include the standard usage block:

{
  "choices": [{"message": {"content": "..."}, "finish_reason": "stop"}],
  "usage": {
    "prompt_tokens": 5,
    "completion_tokens": 95,
    "total_tokens": 100
  }
}

TokenRateLimitPolicy extracts usage.total_tokens from this response automatically. In production, replace the ServiceEntry with your actual AI provider endpoint (for example, api.openai.com) and add a DestinationRule for TLS origination.

Basic token rate limiting

Apply a global token limit on all egress traffic to the AI mock:

kubectl apply -f - <<'EOF'
apiVersion: kuadrant.io/v1alpha1
kind: TokenRateLimitPolicy
metadata:
  name: ai-token-limit
  namespace: gateway-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: ai-mock-external
  limits:
    global:
      rates:

        - limit: 100
          window: 1m
EOF

This limits total token consumption across all workloads to 100 tokens per minute.

Wait for the policy to be accepted and enforced:

kubectl wait --timeout=60s tokenratelimitpolicy/ai-token-limit -n gateway-system \
    --for=jsonpath='{.status.conditions[?(@.type=="Accepted")].status}'=True
kubectl wait --timeout=60s tokenratelimitpolicy/ai-token-limit -n gateway-system \
    --for=jsonpath='{.status.conditions[?(@.type=="Enforced")].status}'=True

Testing token counting

Send a chat completion request:

kubectl exec test-client -n egress-test -- \
    curl -s -H "Host: api.ai-mock.local" \
         -H "Content-Type: application/json" \
         -X POST http://${EGRESS_IP}/v1/chat/completions \
         -d '{
               "model": "meta-llama/Llama-3.1-8B-Instruct",
               "messages": [{"role": "user", "content": "What is Kubernetes?"}],
               "max_tokens": 100,
               "stream": false,
               "usage": true
             }'

The response includes usage.total_tokens. The simulator returns approximately 5-15 tokens per request. Each request consumes tokens from the 100/minute budget. After enough requests, the gateway returns HTTP 429:

# Send requests until rate limited (limit is 100 tokens/min, ~10 tokens/request)
for i in $(seq 1 20); do
  CODE=$(kubectl exec test-client -n egress-test -- \
      curl -s -o /dev/null -w "%{http_code}" \
           -H "Host: api.ai-mock.local" \
           -H "Content-Type: application/json" \
           -X POST http://${EGRESS_IP}/v1/chat/completions \
           -d '{
                 "model": "meta-llama/Llama-3.1-8B-Instruct",
                 "messages": [{"role": "user", "content": "Hello"}],
                 "max_tokens": 100,
                 "stream": false,
                 "usage": true
               }')
  echo "Request $i: HTTP $CODE"
  [ "$CODE" = "429" ] && break
done

After cumulative tokens exceed 100, subsequent requests are rejected with 429 until the window resets.

Clean up before the next section:

kubectl delete tokenratelimitpolicy ai-token-limit -n gateway-system

Per-workload token limiting

To give each workload its own token budget, combine TRLP with workload identity via AuthPolicy. This uses the same kubernetesTokenReview pattern as RateLimitPolicy on egress.

Applying workload identity

kubectl apply -f - <<'EOF'
apiVersion: kuadrant.io/v1
kind: AuthPolicy
metadata:
  name: workload-identity
  namespace: gateway-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: ai-mock-external
  rules:
    authentication:
      "workload-sa":
        kubernetesTokenReview:
          audiences:

            - "https://kubernetes.default.svc.cluster.local"
    authorization:
      "allowed-namespaces":
        patternMatching:
          patterns:
            - predicate: auth.identity.user.username.startsWith('system:serviceaccount:egress-test:')
    response:
      success:
        filters:
          identity:
            json:
              properties:
                username:
                  selector: auth.identity.user.username
EOF

The response.success.filters.identity block exposes the authenticated username to downstream policies. Without it, counter expressions referencing auth.identity.username cannot resolve in the wasm-shim.

Workloads must include their SA token in requests. Requests without a valid token are rejected (401). Workloads from unauthorized namespaces are rejected (403).

Applying per-workload token limits

kubectl apply -f - <<'EOF'
apiVersion: kuadrant.io/v1alpha1
kind: TokenRateLimitPolicy
metadata:
  name: ai-per-workload
  namespace: gateway-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: ai-mock-external
  limits:
    per-workload:
      rates:

        - limit: 100
          window: 1m
      counters:
        - expression: auth.identity.username
EOF

Each ServiceAccount now gets an independent 100 tokens/minute budget.

Verifying per-workload limits

Exhaust the test-client budget and confirm that team-gold is unaffected:

# Exhaust test-client (default SA) budget
for i in $(seq 1 20); do
  CODE=$(kubectl exec test-client -n egress-test -- sh -c '
  curl -s -o /dev/null -w "%{http_code}" \
      -H "Host: api.ai-mock.local" \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
      -X POST http://'"${EGRESS_IP}"'/v1/chat/completions \
      -d "{
            \"model\": \"meta-llama/Llama-3.1-8B-Instruct\",
            \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],
            \"max_tokens\": 100,
            \"stream\": false,
            \"usage\": true
          }"
  ')
  echo "test-client request $i: HTTP $CODE"
  [ "$CODE" = "429" ] && break
done

# team-gold (team-gold SA) — independent budget, still allowed
CODE=$(kubectl exec team-gold -n egress-test -- sh -c '
curl -s -o /dev/null -w "%{http_code}" \
    -H "Host: api.ai-mock.local" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    -X POST http://'"${EGRESS_IP}"'/v1/chat/completions \
    -d "{
          \"model\": \"meta-llama/Llama-3.1-8B-Instruct\",
          \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],
          \"max_tokens\": 100,
          \"stream\": false,
          \"usage\": true
        }"
')
echo "team-gold: HTTP $CODE"

Rate limiting one workload does not affect the other. Each SA token counter is tracked independently.

Clean up before the next section:

kubectl delete tokenratelimitpolicy ai-per-workload -n gateway-system

Per-tier token limiting

For differentiated quotas (for example, free versus gold tiers), use when predicates to match workload identity patterns alongside per-identity counters.

Apply a TRLP with tier-based limits:

kubectl apply -f - <<'EOF'
apiVersion: kuadrant.io/v1alpha1
kind: TokenRateLimitPolicy
metadata:
  name: ai-per-tier
  namespace: gateway-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: ai-mock-external
  limits:
    default-tier:
      rates:

        - limit: 100
          window: 1m
      when:
        - predicate: auth.identity.username == 'system:serviceaccount:egress-test:default'
      counters:
        - expression: auth.identity.username
    gold-tier:
      rates:
        - limit: 500
          window: 1m
      when:
        - predicate: auth.identity.username == 'system:serviceaccount:egress-test:team-gold'
      counters:
        - expression: auth.identity.username
EOF

• test-client (using the default SA) gets 100 tokens/minute
• team-gold (using the team-gold SA) gets 500 tokens/minute

Verifying tier-based limits

# Exhaust the default tier (100 tokens, ~10 tokens/request)
for i in $(seq 1 20); do
  CODE=$(kubectl exec test-client -n egress-test -- sh -c '
  curl -s -o /dev/null -w "%{http_code}" \
      -H "Host: api.ai-mock.local" \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
      -X POST http://'"${EGRESS_IP}"'/v1/chat/completions \
      -d "{
            \"model\": \"meta-llama/Llama-3.1-8B-Instruct\",
            \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],
            \"max_tokens\": 100,
            \"stream\": false,
            \"usage\": true
          }"
  ')
  echo "default-tier request $i: HTTP $CODE"
  [ "$CODE" = "429" ] && break
done

# Gold tier (500 tokens) still has budget
CODE=$(kubectl exec team-gold -n egress-test -- sh -c '
curl -s -o /dev/null -w "%{http_code}" \
    -H "Host: api.ai-mock.local" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    -X POST http://'"${EGRESS_IP}"'/v1/chat/completions \
    -d "{
          \"model\": \"meta-llama/Llama-3.1-8B-Instruct\",
          \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],
          \"max_tokens\": 100,
          \"stream\": false,
          \"usage\": true
        }"
')
echo "gold-tier request: HTTP $CODE"

The default tier hits its limit, but the gold tier still has remaining budget.

Clean up before the next section:

kubectl delete tokenratelimitpolicy ai-per-tier -n gateway-system

Streaming responses

TRLP supports streaming OpenAI-style responses. The request must include "stream": true and "stream_options": { "include_usage": true } for usage to be extracted from the final stream event:

kubectl exec test-client -n egress-test -- sh -c '
curl -s \
    -H "Host: api.ai-mock.local" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    -X POST http://'"${EGRESS_IP}"'/v1/chat/completions \
    -d "{
          \"model\": \"meta-llama/Llama-3.1-8B-Instruct\",
          \"messages\": [{\"role\": \"user\", \"content\": \"What is Kubernetes?\"}],
          \"max_tokens\": 100,
          \"stream\": true,
          \"stream_options\": {\"include_usage\": true}
        }"
'

The final SSE event contains the usage data. Because the counter is updated only after the full response completes, a streaming response delivers all its chunks before the token count is recorded. The updated limit applies to subsequent requests, not the stream already in progress.

If stream_options.include_usage is omitted when stream: true, token usage cannot be extracted. Depending on the wasm-shim failure mode, the request may be allowed without counting or rejected.

Deployment considerations

TLS origination and response body access

When using a real external AI API (for example, OpenAI), the egress gateway terminates internal mTLS and originates a new TLS connection to the provider. The response body passes through the gateway unencrypted on the internal side, so token extraction from the response body works normally. The mock used in this guide skips TLS origination for simplicity.

Concurrent request race condition

The Kuadrant CR field spec.tokenRateLimiting.mode controls cluster-wide how TRLP enforces limits: Reservation (the default) or Optimistic. In Optimistic mode, the gateway checks limits before forwarding the request (without consuming tokens), then reports actual usage after receiving the response; between these two phases, concurrent in-flight requests can race past the limit because nothing holds capacity during model processing.

Reservation mode closes this gap by reserving an estimated token amount on request arrival and committing the actual usage once the upstream responds, releasing any unused portion of the reservation. However, reservation.amount defaults to 0, which reserves no capacity — so a TokenRateLimitPolicy that doesn't set amount explicitly behaves like Optimistic even while the cluster is in Reservation mode. See Token Limit Reservations below to enable real protection on this egress gateway, and the Token Rate Limiting Overview for the full explanation of both modes.

Supported response formats

Token extraction works with any back end returning an OpenAI-compatible response body with usage.total_tokens. This includes OpenAI, vLLM, kServe, Ollama, Azure OpenAI, and the Gemini OpenAI-compat endpoint. Anthropic and Gemini native formats are not yet supported. See #1864 for tracking.

Missing token usage in responses

If the external API does not include usage.total_tokens in the response body, or if the field cannot be parsed, the token report phase fails silently and no tokens are counted. Because both the check and report services default to failureMode: allow, the request succeeds but the counter is never incremented. This means TRLP effectively becomes a no-op: no rate limiting is applied.

To reject requests when token extraction fails, set the RATELIMIT_REPORT_SERVICE_FAILURE_MODE environment variable to deny on the operator deployment. This causes the gateway to block any response where usage cannot be extracted. This is a blunt control that affects all TokenRateLimitPolicies in the cluster.

Verify that your AI provider returns usage.total_tokens in every response before relying on TRLP for enforcement. For streaming, verify that stream_options.include_usage is set to true in requests.

Istio-only support

Egress gateway support targets Istio as the Gateway API provider. Envoy Gateway is not supported for egress at this time.

Token limit reservations

The cluster is already in Reservation mode by default (see Concurrent Request Race Condition above), implemented per RFC 0021. But without an explicit reservation.amount on a limit, that limit still behaves like Optimistic mode. This section shows how to turn on real reservation-based protection for the per-workload policy from Per-Workload Token Limiting.

Cluster-wide mode

The Kuadrant CR spec.tokenRateLimiting.mode field controls enforcement behavior cluster-wide, for every TokenRateLimitPolicy:

apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
  namespace: kuadrant-system
spec:
  tokenRateLimiting:
    mode: Reservation   # default; set to Optimistic to use check-then-report instead

• Reservation (default): uses Reserve/Commit for all TokenRateLimitPolicies, per the per-limit reservation settings described below
• Optimistic: uses check-then-report for all TokenRateLimitPolicies, as described in Concurrent Request Race Condition

See the Token Rate Limiting Overview for the full comparison of both modes.

How it works

When a request arrives, the gateway reserves an estimated token amount against the limit. If remaining capacity (accounting for all outstanding reservations) is insufficient, the request is rejected immediately with 429. After the model responds, the actual usage is committed and the unused portion of the reservation is released.

Request arrives → Reserve(estimated amount, TTL) → Forward to model → Commit(actual usage) → Release unused reservation

If the model call fails or times out, the reservation is released by committing 0, reclaiming capacity without waiting for the counter window to roll over.

Set a reservation amount

Add a reservation block to the limit, with an estimated token cost per request and a hold duration:

kubectl apply -f - <<'EOF'
apiVersion: kuadrant.io/v1alpha1
kind: TokenRateLimitPolicy
metadata:
  name: ai-per-workload
  namespace: gateway-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: ai-mock-external
  limits:
    per-workload:
      rates:

        - limit: 100
          window: 1m
      counters:
        - expression: auth.identity.username
      reservation:
        amount: 10
        ttl: 'duration("30s")'
EOF

• reservation.amount: the number of tokens to reserve on request arrival, as a literal integer or a CEL expression. Defaults to 0 (no capacity reserved) when omitted — set a non-zero estimate to get protection against the concurrent-request race.
• reservation.ttl: a CEL expression for how long the reservation is held before it auto-releases. Defaults to the route's backendRequest timeout when available; otherwise, Limitador applies its default.

See the reservation API reference for full details on the field, and the Token Rate Limiting Overview for the Limitador-side caps (spec.reservations.maxFraction, spec.reservations.maxTtl) on how much a single reservation can claim and how long it can be held.

Verifying reservation under concurrency

The reservation effect is only observable when requests overlap in time. The mock AI API answers in a few milliseconds, so the Reserve/Commit cycle closes before the next request arrives and no two requests are ever in flight together. Add response latency so a burst genuinely overlaps:

kubectl patch deployment llm-sim -n ai-mock --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/args","value":["--model","meta-llama/Llama-3.1-8B-Instruct","--port","8000","--time-to-first-token","200","--inter-token-latency","100"]}]'
kubectl rollout status deployment/llm-sim -n ai-mock --timeout=90s

Reset the counters so the burst starts from a clean window (Limitador uses in-memory storage, so restarting it clears all counters):

kubectl rollout restart deployment/limitador-limitador -n kuadrant-system
kubectl rollout status deployment/limitador-limitador -n kuadrant-system --timeout=90s

Fire a burst of 30 concurrent requests from a single identity against the 100-token limit:

kubectl exec test-client -n egress-test -- sh -c '
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
seq 30 | xargs -P 30 -I{} curl -s -o /dev/null -w "%{http_code}\n" --max-time 20 \
    -H "Host: api.ai-mock.local" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $TOKEN" \
    -X POST http://'"${EGRESS_IP}"'/v1/chat/completions \
    -d "{
          \"model\": \"meta-llama/Llama-3.1-8B-Instruct\",
          \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],
          \"max_tokens\": 100,
          \"stream\": false,
          \"usage\": true
        }"
' | sort | uniq -c

With reservation.amount: 10 against a 100-token limit, only ~10 concurrent requests fit before reserved capacity is exhausted, so roughly 10 succeed and the rest are rejected immediately:

     10 200
     20 429

To see the race this prevents, switch the cluster to Optimistic mode, reset the counters again, and repeat the burst:

kubectl patch kuadrant kuadrant -n kuadrant-system --type merge \
    -p '{"spec":{"tokenRateLimiting":{"mode":"Optimistic"}}}'
kubectl rollout restart deployment/limitador-limitador -n kuadrant-system
kubectl rollout status deployment/limitador-limitador -n kuadrant-system --timeout=90s
# (re-run the burst command above)

In Optimistic mode every request checks the limit before any has reported its usage, so all 30 are admitted — the overshoot that reservations are designed to prevent:

     30 200

Restore the cluster mode and remove the mock latency before continuing:

kubectl patch kuadrant kuadrant -n kuadrant-system --type merge \
    -p '{"spec":{"tokenRateLimiting":{"mode":"Reservation"}}}'
kubectl patch deployment llm-sim -n ai-mock --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/args","value":["--model","meta-llama/Llama-3.1-8B-Instruct","--port","8000"]}]'
kubectl rollout status deployment/llm-sim -n ai-mock --timeout=90s

Clean up before the next section:

kubectl delete tokenratelimitpolicy ai-per-workload -n gateway-system

Cleanup

Remove all resources created by this guide:

# Remove policies
kubectl delete tokenratelimitpolicy -n gateway-system ai-token-limit ai-per-workload ai-per-tier --ignore-not-found
kubectl delete authpolicy workload-identity -n gateway-system --ignore-not-found

# Remove AI mock resources
curl -sL https://raw.githubusercontent.com/Kuadrant/kuadrant-operator/refs/heads/main/hack/setup-egress-ai-mock.sh | bash -s cleanup

# Optionally remove the base egress gateway
curl -sL https://raw.githubusercontent.com/Kuadrant/kuadrant-operator/refs/heads/main/hack/setup-egress.sh | bash -s cleanup

References

• RFC 0013: AI Policies
• RFC 0021: Token Rate Limit Reservations
• TokenRateLimitPolicy Overview
• Token Rate Limiting Tutorial
• Egress Gateway Setup
• Credential Injection