ts-llm-gateway

A minimal but real LLM gateway proxy; one unified endpoint routing to multiple providers (Amazon Bedrock, OpenAI) behind production policies: rate limiting, retry/backoff with cross-provider failover, response caching, and SSE streaming with client-driven cancellation.

Endpoints

• GET /health Liveness check → { "ok": true }
• POST /v1/chat Native chat completion. Add "stream": true for an SSE token stream.
• POST /v1/chat/completions OpenAI-compatible. Point the OpenAI SDK here; route with a provider/model id.
• GET /stats Live counters (JSON) powering the dashboard below.

Live stats

Live from /stats, polled every 2s. Backed by Redis when configured, so counters are global and durable across serverless instances; without Redis they fall back to per-instance in-memory state that resets on cold start.

Example

curl -s https://ts-llm-gateway.vercel.app/v1/chat \
  -H 'content-type: application/json' \
  -d '{
    "model": "us.anthropic.claude-haiku-4-5-20251001-v1:0",
    "messages": [{ "role": "user", "content": "Hello" }]
  }'

# OpenAI-compatible: OpenAI wire format, routed to Bedrock via the model prefix
curl -s https://ts-llm-gateway.vercel.app/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0",
    "messages": [{ "role": "user", "content": "Hello" }]
  }'

# Streaming: add "stream": true and read the SSE token stream (curl -N)
curl -sN https://ts-llm-gateway.vercel.app/v1/chat \
  -H 'content-type: application/json' \
  -d '{
    "model": "us.anthropic.claude-haiku-4-5-20251001-v1:0",
    "stream": true,
    "messages": [{ "role": "user", "content": "Hello" }]
  }'

Policies -> production lineage