Portkey
Portkey is an AI gateway and observability platform that sits between applications and LLM providers, offering unified API access, intelligent routing, caching, retries, and cost and usage monitoring across multiple model providers. It…
Definition
Portkey is an AI gateway and observability platform that sits between applications and LLM providers, offering unified API access, intelligent routing, caching, retries, and cost and usage monitoring across multiple model providers. It lets teams manage production LLM traffic through a single control layer regardless of which underlying models they use.
Overview
As organizations began using multiple LLM providers — OpenAI, Anthropic, Google, open-weight models via various hosts — managing reliability, cost, and observability separately for each became a significant engineering burden. Portkey addresses this by acting as an AI gateway: a single API endpoint that applications call, which then routes requests to the appropriate underlying LLM provider based on configurable rules. The gateway layer provides features that are difficult to build reliably in-house, including automatic retries and fallbacks (if one provider is down or rate-limited, requests can fail over to another), load balancing across providers or API keys, semantic and exact-match caching to reduce redundant LLM calls, and request/response logging for full observability. Because all traffic flows through Portkey, teams get a unified view of cost, latency, and error rates across every model and provider they use, without needing custom instrumentation for each one. Portkey also offers guardrails and prompt management features, letting teams version prompts, run A/B tests between models or prompt variants, and enforce policies such as PII redaction or content filtering before requests reach or after responses return from the underlying model. It can be self-hosted as an open-source gateway or used as a managed cloud service. Portkey competes with similar AI gateway and LLMOps platforms such as Helicone, LiteLLM, and OpenRouter, differentiating through its combination of gateway routing, caching, observability, and prompt management in one integrated product aimed at production reliability rather than just logging.
Key Features
- Unified API gateway routing requests across multiple LLM providers
- Automatic retries and fallbacks for provider outages or rate limits
- Semantic and exact-match response caching to cut redundant LLM calls
- Load balancing across providers and API keys
- Centralized cost, latency, and error observability across all traffic
- Prompt versioning and management with A/B testing support
- Guardrails for content filtering and PII redaction
- Available as open-source self-hosted gateway or managed cloud service