How to Narrow the Field in Open Source Model Selection

A valuable reference from Rochelle Turner’s keynote on October 1, 2026, for all participants who attended.

Main Theme

The central theme of the keynote, “How to Narrow the Field When It Comes to Selecting an Open Source Model,” focused on providing a strategic framework for enterprises to navigate the overwhelming “sea of open source models.” The main message is that while open source models offer significant advantages—primarily cost savings up to 70%, data privacy, and control over performance—the selection and deployment process is complex. Successfully transitioning from closed-source to open source models requires a disciplined, iterative approach that balances model accuracy with cost and performance objectives, moving beyond standard benchmarks to real-world evaluation.

Subthemes and Chapters

Rochelle structured the selection process as a funnel, moving from a broad landscape to a specific, production-ready model.

1. Understanding the “Why”: The Drive to Open Source

The initial shift to open source is driven by several key business needs:

• Control: Gaining control over data privacy, latency, and infrastructure.
• Cost Savings: This is the primary driver, with demonstrated potential for up to 70% cost reduction.
• Customization: The ability to fine-tune and optimize models for specific, proprietary use cases.

2. Defining Your Objectives: Accuracy vs. Cost-Performance

Before evaluating any models, it’s critical to define the two core sets of requirements:

• Model Accuracy: This is about performance on your specific data. It involves choosing the right evaluation metrics, covering edge cases, and ensuring the evaluation process mirrors real-world scenarios.
• Cost-Performance: This involves setting hard, technical targets. You must define your dollar-per-token objective, target latency (time to first token, tokens/sec), anticipated traffic (tokens per minute), and input/output sequence lengths. These metrics determine if the model is economically viable at scale.

3. Step 1: Categorizing the Model Landscape

To begin narrowing the field, group models into broad categories to find a set of ~10 candidates. The keynote proposed three main types:

• Frontier Models: High-intelligence models with general reasoning capabilities for a wide range of tasks (e.g., Llama 3 70B, Mixtral).
• Cost-Efficient “Flash” Models: Smaller, faster models optimized for specific, low-latency workflows at a very effective cost (e.g., Qwen2-7B, Llama 3 8B).
• Small, Embeddable Models: Highly specialized models designed to fit into a small memory footprint for on-device or edge applications.

4. Step 2: From 10 to 3 – Using Benchmarks and Real-World Data

Industry benchmarks are a good starting point to narrow your list from ten models down to two or three. However, they don’t tell the whole story. The most effective evaluation method is to use your own data. The speaker strongly advocated for a “replay and shadow” traffic mechanism:

• Example: If you are already using a closed-source model (like from OpenAI or Anthropic) in production, you have a stream of real-world prompts and traffic. The “shadow” strategy involves running this same traffic through your candidate open source models in parallel. This provides a direct, real-world comparison of performance, helps identify edge cases, and aligns evaluation with your actual business needs.

5. Step 3: From 3 to 1 – Assessing Cost and Deployment Strategy

Once you have a top model based on accuracy, you must validate it against your cost-performance targets. This requires understanding your deployment strategy, as performance depends heavily on it. Options include:

• Serverless APIs: Pay-per-token, ideal for variable or bursty workloads.
• Provisioned Throughput: Commit to a certain capacity level for guaranteed performance.
• Dedicated Hardware: Maximum control and performance for high-demand applications.

6. Embracing an Iterative and Hybrid Strategy

The keynote stressed that model selection is not a one-time decision. The best model today will likely be outdated in six months.

• It’s an Iterative Cycle: You must build a continuous evaluation “motion” to quickly test and adopt new, better models as they are released.
• It’s Not One-Size-Fits-All: A single model might not be the answer. Rochelle highlighted hybrid “router” or “cascade” strategies where multiple models are paired together. For instance, a small, efficient model (Model B) can handle simple queries, while complex ones are routed to a more powerful model (Model A), which could be an open source or even a closed-source model.

Inspiring Quotes

• “The model we choose today is probably not the model that we’re going to be using six months from now. This industry is going to survive.”
• “We’ve seen up to 70% cost savings for our customers that have successfully deployed open source models into production.”
• “It’s not a one-size-fits-all when it comes to the strategy… Sometimes I’m going to pair model A and B together. Maybe model A is going to take care of your complex problems and model B is going to take care of your specific problems.”
• “Models just tell you one part of the story. When it comes to actually applying this to production, there are so many other things you also need to think about.”
• “If you get off to a rocky start, these are things we can catch early and ensure a smooth and efficient construction.”

Key Action Points

1. Define Your Hard Requirements First: Before looking at any models, establish concrete, measurable targets for your application. What is your maximum acceptable latency (time to first token)? What is your target cost per 1M tokens? What is your required throughput (tokens/minute)?
2. Use Your Own Traffic for Evaluation: Move beyond generic benchmarks. Implement a “shadow traffic” system to replay your live production prompts through candidate open source models. This is the most effective way to see how they will perform on your specific use cases and data.
3. Build a Multi-Model, Iterative System: Do not assume a “one-and-done” solution. Design your architecture with a “router” or gateway from the start. This allows you to pair different models (open source, closed-source, small, large) and easily swap them out as better ones become available, creating a flexible and future-proof AI stack.