Model Sets: Balancing Cost and Intelligence (Or: Not Every Test Needs Genius)
Summary
- Published
- Collected
📦 Full-Text Content Snapshot
R2 Archive · Full Article PreservedPublished on: 2026-10-26 · Source: tenzai.com
October 26, 2026
4 min read
Model Sets: Balancing Cost and Intelligence (Or: Not Every Test Needs Genius)
Aner Mazur
Cofounder and Chief Product Officer
Security teams use AI to test their applications continuously, but that puts a critical, and costly, decision in front of them every day: how much intelligence does each test need?
A comprehensive test of a complex, business-critical application benefits from the most capable models available. A fix validation, narrow work confirming a specific patch held, needs far less. In between sit other tests, such as incremental tests on every release or focused tests that validate the exploitability of a new CVE. Running all of them on the same models means overpaying for some and underpowering others.
Not all applications matter the same either. Some are business-critical, internet-facing applications where the potential risk of an unsurfaced exploitable vulnerability is huge. Others are less important internal applications, where you'd still want to test but the overall risk is lower.
That's the mismatch Tenzai’s Model Sets are built to solve - starting today.
Tenzai now curates and optimizes sets of frontier and open-weight models, each tuned to a different balance of intelligence and cost. You choose the set that fits the test and the application, and spend only what it needs.
Match the model to the work
Each model set is named for what it optimizes:
- Efficient gets the most findings per dollar.
- Balanced trades cost for broader coverage, and is the baseline Tenzai prices against.
- Genius puts intelligence first.
Sets come in frontier-only versions, built on models like Anthropic Opus 5 and xAI Grok 4.6, for organizations that haven't adopted open-weight models yet. Other versions bring in open-weight models like Z.ai GLM 5.3. These can be configured per test, with an organization-wide admin setting for defaults.
_Figure 1: Admins choose the allowed models, the default set, and which sets tests can use._
This curation isn't a one-time exercise. Tenzai tests new models as they're released against a rigorous, proprietary benchmark of labs, and updates sets when a model earns its place. Within each set, the harness also matches the right model and effort level to every task the agents perform.
What the numbers show
Each set is run numerous times, nightly, on a wide array of apps in our blackbox benchmark. We measure multiple metrics on each run. Below you can see finding recall, the share of known vulnerabilities a test reports, alongside overall cost. Performance evolves as models evolve, so the results below are a point-in-time snapshot.
_Figure 2: Finding recall vs cost per test, cost is relative to the “Balanced” set._
A few things stand out:
- A light set goes a long way. “Efficient” finds 71% of what “Genius” finds, at 3% of the cost. Many of our users use this for more focused tasks, like validating exploitability of externally reported findings, or for immediate, continuous testing as changes land on the main branch.
- Mixing models cuts costs while improving results. “Balanced” (open weight) matches GLM 5.3 alone in a single run and finds more across three runs, 68% vs 62%, at 41% lower cost.
- Close isn't enough. We tested many, many blends across models, roles and effort levels. We show two because they came close: GLM 5.3 alone and a Grok 4.6 explorer blend landed just under the Pareto curve.
Creativity counts on repeat runs
Each run takes a different path through the app. The more creative freedom and room to explore we give the models, the more a repeat run adds. Over three runs (pass @3), Balanced rose from 59% to 77%, and Balanced”(open weight) from 53% to 68% - a reminder that agentic testing is dynamic, and frequency pays off.
_Figure 3: Vulnerabilities found in one run vs at least once in three runs._
Better models keep coming, and attackers don't wait
Every few weeks a new model raises the bar for offensive work, open-weight models most of all: available to anyone, unregulated and cheap. Attackers adopt them right away.
We keep pace. We tune the harness for each new model and add it only when it fits its role, works with the other agents, and passes our safety testing.
DeepSeek is a recent example. We explored DeepSeek V4.1 Flash 1017 and tuned it with our harness for a further boost. An experimental set involving it reached 97% of “Genius” recall at 6% of its cost. It's still early, and we are rigorously testing it for safety, but it shows how promising these models are and how fast things are moving.
Architecting for safe use of open-weight models
A cheaper model never means weaker safety. Every model set operates under the same data terms and the same harness.
Open-weight models run on Fireworks.ai with zero data retention. Our model and inference providers never use customer data for training, and test data never reaches the labs that built the models. (EU hosting is coming soon)
Every model passes our guardrail labs before it joins a set, and we monitor its behavior across thousands of tests after. Tenzai harness enforces the limits:
- Each test runs in its own sandbox, destroyed when the test stops.
- Your allowlist or blocklist is enforced at the network layer. Blocked requests show in the logs.
- A separate classifier checks risky commands and steers the agent to a non-destructive proof.
Admins stay in control - choosing which sets their organization allows, frontier only or open weight too, and the default for new tests. Users can pick another set per test, and every change lands in the audit log.
With Tenzai’s Model Sets, you can spend on intelligence where it's most needed, and save where a lighter set does the job.
Talk to us. We're listening.
See Tenzai run against an app you own.
Media inquiries and interviews
[[email protected]](https://www.tenzai.com/cdn-cgi/l/email-protection#58282a3d2b2b182c3d36223931763b3735)
Hiring elite hackers and more
📦 Full-Text Markdown Snapshot archived by Phying News · Original source: tenzai.com
--- id: 10340 canonical_url: "https://phying.com/item/10340" title: "Model Sets: Balancing Cost and Intelligence (Or: Not Every Test Needs Genius)" type: "blog" cve: null cvss: null severity: "unknown" vendor: null product: null affected_versions: null source_domain: "tenzai.com" source_url: "https://www.tenzai.com/blog/model-sets-balancing-cost-and-intelligence" published_at: "2026-10-26T15:46:35Z" snapshot_at: "2026-10-27T15:00:58Z" archived_by: "Phying News (https://phying.com/item/10340)" lang: "en" --- # Model Sets: Balancing Cost and Intelligence (Or: Not Every Test Needs Genius) *Published on: 2026-10-26 · Source: [tenzai.com](https://www.tenzai.com/blog/model-sets-balancing-cost-and-intelligence)* October 26, 2026 4 min read # Model Sets: Balancing Cost and Intelligence (Or: Not Every Test Needs Genius) Aner Mazur Cofounder and Chief Product Officer [ ](https://www.tenzai.com/blog/model-sets-balancing-cost-and-intelligence#) [ ](https://www.tenzai.com/blog/model-sets-balancing-cost-and-intelligence#) [ ](https://www.tenzai.com/blog/model-sets-balancing-cost-and-intelligence#)  Security teams use AI to test their applications continuously, but that puts a critical, and costly, decision in front of them every day: how much intelligence does each test need? A comprehensive test of a complex, business-critical application benefits from the most capable models available. A fix validation, narrow work confirming a specific patch held, needs far less. In between sit other tests, such as incremental tests on every release or focused tests that validate the exploitability of a new CVE. Running all of them on the same models means overpaying for some and underpowering others. Not all applications matter the same either. Some are business-critical, internet-facing applications where the potential risk of an unsurfaced exploitable vulnerability is huge. Others are less important internal applications, where you'd still want to test but the overall risk is lower. That's the mismatch Tenzai’s Model Sets are built to solve - starting today. Tenzai now curates and optimizes sets of frontier and open-weight models, each tuned to a different balance of intelligence and cost. You choose the set that fits the test and the application, and spend only what it needs. ### Match the model to the work Each model set is named for what it optimizes: * Efficient gets the most findings per dollar. * Balanced trades cost for broader coverage, and is the baseline Tenzai prices against. * Genius puts intelligence first. Sets come in frontier-only versions, built on models like Anthropic Opus 5 and xAI Grok 4.6, for organizations that haven't adopted open-weight models yet. Other versions bring in open-weight models like Z.ai GLM 5.3. These can be configured per test, with an organization-wide admin setting for defaults.  **_Figure 1: Admins choose the allowed models, the default set, and which sets tests can use._** This curation isn't a one-time exercise. Tenzai tests new models as they're released against a rigorous, proprietary benchmark of labs, and updates sets when a model earns its place. Within each set, the harness also matches the right model and effort level to every task the agents perform. ### What the numbers show Each set is run numerous times, nightly, on a wide array of apps in our blackbox benchmark. We measure multiple metrics on each run. Below you can see finding recall, the share of known vulnerabilities a test reports, alongside overall cost. Performance evolves as models evolve, so the results below are a point-in-time snapshot.  _Figure 2: Finding recall vs cost per test, cost is relative to the “Balanced” set._ A few things stand out: * A light set goes a long way. “Efficient” finds 71% of what “Genius” finds, at 3% of the cost. Many of our users use this for more focused tasks, like validating exploitability of externally reported findings, or for immediate, continuous testing as changes land on the main branch. * Mixing models cuts costs while improving results. “Balanced” (open weight) matches GLM 5.3 alone in a single run and finds more across three runs, 68% vs 62%, at 41% lower cost. * Close isn't enough. We tested many, many blends across models, roles and effort levels. We show two because they came close: GLM 5.3 alone and a Grok 4.6 explorer blend landed just under the Pareto curve. ### Creativity counts on repeat runs Each run takes a different path through the app. The more creative freedom and room to explore we give the models, the more a repeat run adds. Over three runs (pass @3), Balanced rose from 59% to 77%, and Balanced”(open weight) from 53% to 68% - a reminder that agentic testing is dynamic, and frequency pays off.  _Figure 3: Vulnerabilities found in one run vs at least once in three runs._ ### Better models keep coming, and attackers don't wait Every few weeks a new model raises the bar for offensive work, open-weight models most of all: available to anyone, unregulated and cheap. [Attackers adopt them right away.](https://www.tenzai.com/blog/you-cant-ask-attackers-to-slow-down) We keep pace. We tune the harness for each new model and add it only when it fits its role, works with the other agents, and passes our safety testing. DeepSeek is a recent example. We explored DeepSeek V4.1 Flash 1017 and tuned it with our harness for a further boost. An experimental set involving it reached 97% of “Genius” recall at 6% of its cost. It's still early, and we are rigorously testing it for safety, but it shows how promising these models are and how fast things are moving. ### Architecting for safe use of open-weight models A cheaper model never means weaker safety. Every model set operates under the same data terms and the same harness. Open-weight models run on Fireworks.ai with zero data retention. Our model and inference providers never use customer data for training, and test data never reaches the labs that built the models. (EU hosting is coming soon) Every model passes our guardrail labs before it joins a set, and we monitor its behavior across thousands of tests after. Tenzai harness enforces the limits: * Each test runs in its own sandbox, destroyed when the test stops. * Your allowlist or blocklist is enforced at the network layer. Blocked requests show in the logs. * A separate classifier checks risky commands and steers the agent to a non-destructive proof. Admins stay in control - choosing which sets their organization allows, frontier only or open weight too, and the default for new tests. Users can pick another set per test, and every change lands in the audit log. With Tenzai’s Model Sets, you can spend on intelligence where it's most needed, and save where a lighter set does the job. ## Talk to us. We're listening. See Tenzai run against an app you own. [Get a Demo](https://www.tenzai.com/blog/model-sets-balancing-cost-and-intelligence#) Media inquiries and interviews [[email protected]](https://www.tenzai.com/cdn-cgi/l/email-protection#58282a3d2b2b182c3d36223931763b3735) Hiring elite hackers and more [Open roles](https://www.tenzai.com/careers) --- > 📦 **Full-Text Markdown Snapshot** archived by [Phying News](https://phying.com/item/10340) · Original source: [tenzai.com](https://www.tenzai.com/blog/model-sets-balancing-cost-and-intelligence)