TensorZero is an open-source LLMOps platform that unifies:
Gateway: access every LLM provider through a unified API, built for performance (<1ms p99 latency)
Observability: store inferences and feedback in your database, available programmatically or in the UI
Evaluation: benchmark individual inferences or end-to-end workflows using heuristics, LLM judges, etc.
Optimization: collect metrics and human feedback to optimize prompts, models, and inference strategies
Experimentation: ship with confidence with built-in A/B testing, routing, fallbacks, retries, etc.
You can take what you need, adopt incrementally, and complement with other tools. It plays nicely with the OpenAI SDK, OpenTelemetry, and every major LLM provider.
TensorZero is used by companies ranging from frontier AI startups to the Fortune 10 and fuels ~1% of global LLM API spend today.
TensorZero Autopilot is an automated AI engineer powered by TensorZero that analyzes LLM observability data, sets up evals, optimizes prompts and models, and runs A/B tests.
It dramatically improves the performance of LLM agents across diverse tasks:
Learn more →
🌐 LLM Gateway
Integrate with TensorZero once and access every major LLM provider.
Integrate with TensorZero once and access every major LLM provider.
Call any LLM (API or self-hosted) through a single unified API
Infer with tool use, structured outputs (JSON), batch, embeddings, multimodal (images, files), caching, etc.
Create prompt templates and schemas to enforce a structured interface between your application and the LLMs
Satisfy extreme throughput and latency needs, thanks to 🦀 Rust: <1ms p99 latency overhead at 10k+ QPS
Ensure high availability with routing, retries, fallbacks, load balancing, granular timeouts, etc.
Track usage and cost and enforce custom rate limits with granular scopes (e.g. tags)
Set up auth for TensorZero to allow clients to access models without sharing provider API keys
Anthropic, AWS Bedrock, AWS SageMaker, Azure, DeepSeek, Fireworks, GCP Vertex AI Anthropic, GCP Vertex AI Gemini, Google AI Studio (Gemini API), Groq, Hyperbolic, Mistral, OpenAI, OpenRouter, SGLang, TGI, Together AI, vLLM, and xAI (Grok).
TensorZero is an open-source LLMOps platform that unifies:
Gateway: access every LLM provider through a unified API, built for performance (<1ms p99 latency)
Observability: store inferences and feedback in your database, available programmatically or in the UI
Evaluation: benchmark individual inferences or end-to-end workflows using heuristics, LLM judges, etc.
Optimization: collect metrics and human feedback to optimize prompts, models, and inference strategies
Experimentation: ship with confidence with built-in A/B testing, routing, fallbacks, retries, etc.
You can take what you need, adopt incrementally, and complement with other tools. It plays nicely with the OpenAI SDK, OpenTelemetry, and every major LLM provider.
TensorZero is used by companies ranging from frontier AI startups to the Fortune 10 and fuels ~1% of global LLM API spend today.
Need something else? TensorZero also supports any OpenAI-compatible API (e.g. Ollama).
You can use TensorZero with any OpenAI SDK (Python, Node, Go, etc.) or OpenAI-compatible client.
Deploy the TensorZero Gateway (one Docker container).
Update the base_url and model in your OpenAI-compatible client.
base_url
model
Run inference:
from openai import OpenAI # Point the client to the TensorZero Gateway client = OpenAI(base_url="http://localhost:3000/openai/v1", api_key="not-used") response = client.chat.completions.create( # Call any model provider (or TensorZero function) model="tensorzero::model_name::anthropic::claude-sonnet-4-6", messages=[ { "role": "user", "content": "Share a fun fact about TensorZero.", } ], )
See Quick Start for more information.
🔍 LLM Observability
Zoom in to debug individual API calls, or zoom out to monitor metrics across models and prompts over time — all using the open-source TensorZero UI.
Zoom in to debug individual API calls, or zoom out to monitor metrics across models and prompts over time — all using the open-source TensorZero UI.
Store inferences and feedback (metrics, human edits, etc.) in your own database
Dive into individual inferences or high-level aggregate patterns using the TensorZero UI or programmatically
Build datasets for optimization, evaluation, and other workflows
Replay historical inferences with new prompts, models, inference strategies, etc.
Export OpenTelemetry traces (OTLP) and export Prometheus metrics to your favorite application observability tools
Soon: AI-assisted debugging and root cause analysis; AI-assisted data labeling
📈 LLM Optimization
Send production metrics and human feedback to easily optimize your prompts, models, and inference strategies — using the UI or programmatically.
Send production metrics and human feedback to easily optimize your prompts, models, and inference strategies — using the UI or programmatically.
Optimize your models with supervised fine-tuning, RLHF, and other techniques
Optimize your prompts with automated prompt engineering algorithms like GEPA
Optimize your inference strategy with dynamic in-context learning, best/mixture-of-N sampling, etc.
Enable a feedback loop for your LLMs: a data & learning flywheel turning production data into smarter, faster, and cheaper models
Soon: synthetic data generation
📊 LLM Evaluation
Compare prompts, models, and inference strategies using evaluations powered by heuristics and LLM judges.
Compare prompts, models, and inference strategies using evaluations powered by heuristics and LLM judges.
Evaluate individual inferences with inference evaluations powered by heuristics or LLM judges (≈ unit tests for LLMs)
Evaluate end-to-end workflows with workflow evaluations with complete flexibility (≈ integration tests for LLMs)
Optimize LLM judges just like any other TensorZero function to align them to human preferences
Soon: more built-in evaluators; headless evaluations
Run ID: 01961de9-c8a4-7c60-ab8d-15491a9708e4 Number of datapoints: 100 ██████████████████████████████████████ 100/100 exact_match: 0.83 ± 0.03 (n=100) semantic_match: 0.98 ± 0.01 (n=100) item_count: 7.15 ± 0.39 (n=100)
Run ID: 01961de9-c8a4-7c60-ab8d-15491a9708e4 Number of datapoints: 100 ██████████████████████████████████████ 100/100 exact_match: 0.83 ± 0.03 (n=100) semantic_match: 0.98 ± 0.01 (n=100) item_count: 7.15 ± 0.39 (n=100)
🧪 LLM Experimentation
Ship with confidence with built-in A/B testing, routing, fallbacks, retries, etc.
Ship with confidence with built-in A/B testing, routing, fallbacks, retries, etc.
Run adaptive A/B tests to ship with confidence and identify the best prompts and models for your use cases.
Enforce principled experiments in complex workflows, including support for multi-turn LLM systems, sequential testing, and more.
& more!
Build with an open-source stack well-suited for prototypes but designed from the ground up to support the most complex LLM applications and deployments.
Build with an open-source stack well-suited for prototypes but designed from the ground up to support the most complex LLM applications and deployments.
Build simple applications or massive deployments with GitOps-friendly orchestration
Extend TensorZero with built-in escape hatches, programmatic-first usage, direct database access, and more
Integrate with third-party tools: specialized observability and evaluations, model providers, agent orchestration frameworks, etc.
Iterate quickly by experimenting with prompts interactively using the Playground UI
Frequently Asked Questions
How is TensorZero different from other LLM frameworks?
TensorZero enables you to optimize complex LLM applications based on production metrics and human feedback.
TensorZero supports the needs of industrial-grade LLM applications: low latency, high throughput, type safety, self-hosted, GitOps, customizability, etc.
TensorZero unifies the entire LLMOps stack, creating compounding benefits. For example, LLM evaluations can be used for fine-tuning models alongside AI judges.
Can I use TensorZero with ___?
Yes. Every major programming language is supported. It plays nicely with the OpenAI SDK, OpenTelemetry, and every major LLM provider.
Is TensorZero production-ready?
Yes. TensorZero is used by companies ranging from frontier AI startups to the Fortune 10 and powers ~1% of the global LLM API spend today.
Here's a case study: Automating Code Changelogs at a Large Bank with LLMs
How much does TensorZero cost?
TensorZero (LLMOps platform) is 100% self-hosted and open-source.
TensorZero Autopilot (automated AI engineer) is a complementary paid product powered by TensorZero.
Who is building TensorZero?
Our technical team includes a former Rust compiler maintainer, machine learning researchers (Stanford, CMU, Oxford, Columbia) with thousands of citations, and the chief product officer of a decacorn startup. We're backed by the same investors as leading open-source projects (e.g. ClickHouse, CockroachDB) and AI labs (e.g. OpenAI, Anthropic). See our $7.3M seed round announcement and coverage from VentureBeat. We're hiring in NYC.
How do I get started?
You can adopt TensorZero incrementally. Our Quick Start goes from a vanilla OpenAI wrapper to a production-ready LLM application with observability and fine-tuning in just 5 minutes.
Get Started
Start building today. The Quick Start shows it's easy to set up an LLM application with TensorZero.
Questions? Ask us on Slack or Discord.
Using TensorZero at work? Email us at hello@tensorzero.com to set up a Slack or Teams channel with your team (free).
Examples
We are working on a series of complete runnable examples illustrating TensorZero's data & learning flywheel.
Optimizing Data Extraction (NER) with TensorZero This example shows how to use TensorZero to optimize a data extraction pipeline. We demonstrate techniques like fine-tuning and dynamic in-context learning (DICL). In the end, an optimized GPT-4o Mini model outperforms GPT-4o on this task — at a fraction of the cost and latency — using a small amount of training data.
Optimizing Data Extraction (NER) with TensorZero
This example shows how to use TensorZero to optimize a data extraction pipeline. We demonstrate techniques like fine-tuning and dynamic in-context learning (DICL). In the end, an optimized GPT-4o Mini model outperforms GPT-4o on this task — at a fraction of the cost and latency — using a small amount of training data.
Agentic RAG — Multi-Hop Question Answering with LLMs This example shows how to build a multi-hop retrieval agent using TensorZero. The agent iteratively searches Wikipedia to gather information, and decides when it has enough context to answer a complex question.
Agentic RAG — Multi-Hop Question Answering with LLMs
This example shows how to build a multi-hop retrieval agent using TensorZero. The agent iteratively searches Wikipedia to gather information, and decides when it has enough context to answer a complex question.
Writing Haikus to Satisfy a Judge with Hidden Preferences This example fine-tunes GPT-4o Mini to generate haikus tailored to a specific taste. You'll see TensorZero's "data flywheel in a box" in action: better variants leads to better data, and better data leads to better variants. You'll see progress by fine-tuning the LLM multiple times.
Writing Haikus to Satisfy a Judge with Hidden Preferences
This example fine-tunes GPT-4o Mini to generate haikus tailored to a specific taste. You'll see TensorZero's "data flywheel in a box" in action: better variants leads to better data, and better data leads to better variants. You'll see progress by fine-tuning the LLM multiple times.
Image Data Extraction — Multimodal (Vision) Fine-tuning This example shows how to fine-tune multimodal models (VLMs) like GPT-4o to improve their performance on vision-language tasks. Specifically, we'll build a system that categorizes document images (screenshots of computer science research papers).
Image Data Extraction — Multimodal (Vision) Fine-tuning
This example shows how to fine-tune multimodal models (VLMs) like GPT-4o to improve their performance on vision-language tasks. Specifically, we'll build a system that categorizes document images (screenshots of computer science research papers).
Improving LLM Chess Ability with Best-of-N Sampling This example showcases how best-of-N sampling can significantly enhance an LLM's chess-playing abilities by selecting the most promising moves from multiple generated options.
Improving LLM Chess Ability with Best-of-N Sampling
This example showcases how best-of-N sampling can significantly enhance an LLM's chess-playing abilities by selecting the most promising moves from multiple generated options.
Blog Posts
We write about LLM engineering on the TensorZero Blog. Here are some of our favorite posts:
Bandits in your LLM Gateway: Improve LLM Applications Faster with Adaptive Experimentation (A/B Testing)
Is OpenAI's Reinforcement Fine-Tuning (RFT) Worth It?
Distillation with Programmatic Data Curation: Smarter LLMs, 5-30x Cheaper Inference
From NER to Agents: Does Automated Prompt Engineering Scale to Complex Tasks?
TensorZero Autopilot is an automated AI engineer powered by TensorZero that analyzes LLM observability data, sets up evals, optimizes prompts and models, and runs A/B tests.
It dramatically improves the performance of LLM agents across diverse tasks:
Learn more →
🌐 LLM Gateway
Integrate with TensorZero once and access every major LLM provider.
Integrate with TensorZero once and access every major LLM provider.
Call any LLM (API or self-hosted) through a single unified API
Infer with tool use, structured outputs (JSON), batch, embeddings, multimodal (images, files), caching, etc.
Create prompt templates and schemas to enforce a structured interface between your application and the LLMs
Satisfy extreme throughput and latency needs, thanks to 🦀 Rust: <1ms p99 latency overhead at 10k+ QPS
Ensure high availability with routing, retries, fallbacks, load balancing, granular timeouts, etc.
Track usage and cost and enforce custom rate limits with granular scopes (e.g. tags)
Set up auth for TensorZero to allow clients to access models without sharing provider API keys
Anthropic, AWS Bedrock, AWS SageMaker, Azure, DeepSeek, Fireworks, GCP Vertex AI Anthropic, GCP Vertex AI Gemini, Google AI Studio (Gemini API), Groq, Hyperbolic, Mistral, OpenAI, OpenRouter, SGLang, TGI, Together AI, vLLM, and xAI (Grok).
Need something else? TensorZero also supports any OpenAI-compatible API (e.g. Ollama).
You can use TensorZero with any OpenAI SDK (Python, Node, Go, etc.) or OpenAI-compatible client.
Deploy the TensorZero Gateway (one Docker container).
Update the base_url and model in your OpenAI-compatible client.
base_url
model
Run inference:
from openai import OpenAI # Point the client to the TensorZero Gateway client = OpenAI(base_url="http://localhost:3000/openai/v1", api_key="not-used") response = client.chat.completions.create( # Call any model provider (or TensorZero function) model="tensorzero::model_name::anthropic::claude-sonnet-4-6", messages=[ { "role": "user", "content": "Share a fun fact about TensorZero.", } ], )
See Quick Start for more information.
🔍 LLM Observability
Zoom in to debug individual API calls, or zoom out to monitor metrics across models and prompts over time — all using the open-source TensorZero UI.
Zoom in to debug individual API calls, or zoom out to monitor metrics across models and prompts over time — all using the open-source TensorZero UI.
Store inferences and feedback (metrics, human edits, etc.) in your own database
Dive into individual inferences or high-level aggregate patterns using the TensorZero UI or programmatically
Build datasets for optimization, evaluation, and other workflows
Replay historical inferences with new prompts, models, inference strategies, etc.
Export OpenTelemetry traces (OTLP) and export Prometheus metrics to your favorite application observability tools
Soon: AI-assisted debugging and root cause analysis; AI-assisted data labeling
📈 LLM Optimization
Send production metrics and human feedback to easily optimize your prompts, models, and inference strategies — using the UI or programmatically.
Send production metrics and human feedback to easily optimize your prompts, models, and inference strategies — using the UI or programmatically.
Optimize your models with supervised fine-tuning, RLHF, and other techniques
Optimize your prompts with automated prompt engineering algorithms like GEPA
Optimize your inference strategy with dynamic in-context learning, best/mixture-of-N sampling, etc.
Enable a feedback loop for your LLMs: a data & learning flywheel turning production data into smarter, faster, and cheaper models
Soon: synthetic data generation
📊 LLM Evaluation
Compare prompts, models, and inference strategies using evaluations powered by heuristics and LLM judges.
Compare prompts, models, and inference strategies using evaluations powered by heuristics and LLM judges.
Evaluate individual inferences with inference evaluations powered by heuristics or LLM judges (≈ unit tests for LLMs)
Evaluate end-to-end workflows with workflow evaluations with complete flexibility (≈ integration tests for LLMs)
Optimize LLM judges just like any other TensorZero function to align them to human preferences
Soon: more built-in evaluators; headless evaluations
Run ID: 01961de9-c8a4-7c60-ab8d-15491a9708e4 Number of datapoints: 100 ██████████████████████████████████████ 100/100 exact_match: 0.83 ± 0.03 (n=100) semantic_match: 0.98 ± 0.01 (n=100) item_count: 7.15 ± 0.39 (n=100)
Run ID: 01961de9-c8a4-7c60-ab8d-15491a9708e4 Number of datapoints: 100 ██████████████████████████████████████ 100/100 exact_match: 0.83 ± 0.03 (n=100) semantic_match: 0.98 ± 0.01 (n=100) item_count: 7.15 ± 0.39 (n=100)
🧪 LLM Experimentation
Ship with confidence with built-in A/B testing, routing, fallbacks, retries, etc.
Ship with confidence with built-in A/B testing, routing, fallbacks, retries, etc.
Run adaptive A/B tests to ship with confidence and identify the best prompts and models for your use cases.
Enforce principled experiments in complex workflows, including support for multi-turn LLM systems, sequential testing, and more.
& more!
Build with an open-source stack well-suited for prototypes but designed from the ground up to support the most complex LLM applications and deployments.
Build with an open-source stack well-suited for prototypes but designed from the ground up to support the most complex LLM applications and deployments.
Build simple applications or massive deployments with GitOps-friendly orchestration
Extend TensorZero with built-in escape hatches, programmatic-first usage, direct database access, and more
Integrate with third-party tools: specialized observability and evaluations, model providers, agent orchestration frameworks, etc.
Iterate quickly by experimenting with prompts interactively using the Playground UI
Frequently Asked Questions
How is TensorZero different from other LLM frameworks?
TensorZero enables you to optimize complex LLM applications based on production metrics and human feedback.
TensorZero supports the needs of industrial-grade LLM applications: low latency, high throughput, type safety, self-hosted, GitOps, customizability, etc.
TensorZero unifies the entire LLMOps stack, creating compounding benefits. For example, LLM evaluations can be used for fine-tuning models alongside AI judges.
Can I use TensorZero with ___?
Yes. Every major programming language is supported. It plays nicely with the OpenAI SDK, OpenTelemetry, and every major LLM provider.
Is TensorZero production-ready?
Yes. TensorZero is used by companies ranging from frontier AI startups to the Fortune 10 and powers ~1% of the global LLM API spend today.
Here's a case study: Automating Code Changelogs at a Large Bank with LLMs
How much does TensorZero cost?
TensorZero (LLMOps platform) is 100% self-hosted and open-source.
TensorZero Autopilot (automated AI engineer) is a complementary paid product powered by TensorZero.
Who is building TensorZero?
Our technical team includes a former Rust compiler maintainer, machine learning researchers (Stanford, CMU, Oxford, Columbia) with thousands of citations, and the chief product officer of a decacorn startup. We're backed by the same investors as leading open-source projects (e.g. ClickHouse, CockroachDB) and AI labs (e.g. OpenAI, Anthropic). See our $7.3M seed round announcement and coverage from VentureBeat. We're hiring in NYC.
How do I get started?
You can adopt TensorZero incrementally. Our Quick Start goes from a vanilla OpenAI wrapper to a production-ready LLM application with observability and fine-tuning in just 5 minutes.
Get Started
Start building today. The Quick Start shows it's easy to set up an LLM application with TensorZero.
Questions? Ask us on Slack or Discord.
Using TensorZero at work? Email us at hello@tensorzero.com to set up a Slack or Teams channel with your team (free).
Examples
We are working on a series of complete runnable examples illustrating TensorZero's data & learning flywheel.
Optimizing Data Extraction (NER) with TensorZero This example shows how to use TensorZero to optimize a data extraction pipeline. We demonstrate techniques like fine-tuning and dynamic in-context learning (DICL). In the end, an optimized GPT-4o Mini model outperforms GPT-4o on this task — at a fraction of the cost and latency — using a small amount of training data.
Optimizing Data Extraction (NER) with TensorZero
This example shows how to use TensorZero to optimize a data extraction pipeline. We demonstrate techniques like fine-tuning and dynamic in-context learning (DICL). In the end, an optimized GPT-4o Mini model outperforms GPT-4o on this task — at a fraction of the cost and latency — using a small amount of training data.
Agentic RAG — Multi-Hop Question Answering with LLMs This example shows how to build a multi-hop retrieval agent using TensorZero. The agent iteratively searches Wikipedia to gather information, and decides when it has enough context to answer a complex question.
Agentic RAG — Multi-Hop Question Answering with LLMs
This example shows how to build a multi-hop retrieval agent using TensorZero. The agent iteratively searches Wikipedia to gather information, and decides when it has enough context to answer a complex question.
Writing Haikus to Satisfy a Judge with Hidden Preferences This example fine-tunes GPT-4o Mini to generate haikus tailored to a specific taste. You'll see TensorZero's "data flywheel in a box" in action: better variants leads to better data, and better data leads to better variants. You'll see progress by fine-tuning the LLM multiple times.
Writing Haikus to Satisfy a Judge with Hidden Preferences
This example fine-tunes GPT-4o Mini to generate haikus tailored to a specific taste. You'll see TensorZero's "data flywheel in a box" in action: better variants leads to better data, and better data leads to better variants. You'll see progress by fine-tuning the LLM multiple times.
Image Data Extraction — Multimodal (Vision) Fine-tuning This example shows how to fine-tune multimodal models (VLMs) like GPT-4o to improve their performance on vision-language tasks. Specifically, we'll build a system that categorizes document images (screenshots of computer science research papers).
Image Data Extraction — Multimodal (Vision) Fine-tuning
This example shows how to fine-tune multimodal models (VLMs) like GPT-4o to improve their performance on vision-language tasks. Specifically, we'll build a system that categorizes document images (screenshots of computer science research papers).
Improving LLM Chess Ability with Best-of-N Sampling This example showcases how best-of-N sampling can significantly enhance an LLM's chess-playing abilities by selecting the most promising moves from multiple generated options.
Improving LLM Chess Ability with Best-of-N Sampling
This example showcases how best-of-N sampling can significantly enhance an LLM's chess-playing abilities by selecting the most promising moves from multiple generated options.
Blog Posts
We write about LLM engineering on the TensorZero Blog. Here are some of our favorite posts:
Bandits in your LLM Gateway: Improve LLM Applications Faster with Adaptive Experimentation (A/B Testing)
Is OpenAI's Reinforcement Fine-Tuning (RFT) Worth It?
Distillation with Programmatic Data Curation: Smarter LLMs, 5-30x Cheaper Inference
From NER to Agents: Does Automated Prompt Engineering Scale to Complex Tasks?