AI Evals Will Outlast AI Architecture

We spent two months building a custom agentic pipeline for a healthcare marketplace client including web search tooling, entity extraction, and structured output. Then GPT-5 was released and we replaced most of it in a week. While it may seem like wasted effort, in a field as fast moving as AI, it is a common occurrence and the key piece we built in the beginning was what allowed us to move quickly even as these developments made some of our original work redundant. In the end, the evals we implemented alongside the original system were exactly what let us make the decision to switch models with confidence and they're the same evals that guided the latest iteration.

The Project

Our client operates a healthcare marketplace that connects buyers to vendors across clinical, operational, and IT categories. The core problem is populating and maintaining structured vendor profiles such as organization names, product lines, pricing models, target specialties from public and private sources at scale.

This is a data enrichment task, but a deceptively hard one. An agent would need to search the web for a given vendor, identify the right entity (not a similarly named company in a different industry), extract the relevant fields, and return structured data. Entity identification was a hard constraint. Confusing a medical device manufacturer with a consumer electronics brand sharing the same parent company name can corrupt the marketplace and reduce trust in its content.

First Approach: Custom Agentic Framework

We built a pipeline using OpenAI Agent SDK with custom web search tooling. The agent would take a vendor name, run targeted searches, parse results, and extract structured profiles. Over two months, we iterated toward something reliable enough for production. What we built worked but there were still some limitations. In particular, it struggled with the normalization of entities and overall became difficult to maintain at the same time as the results reached a plateau. Additionally, there was a problem of hallucinations on links and data extracted from pages. Given that we were using GPT 4o as the primary LLM engine and knowing that the more mature OpenAI Responses API was an alternative, we needed to expand how we could compare models when the time came to switch.

Building Evals Early

In parallel with the pipeline work, we established an evaluation framework. This turned out to be the most important decision of the project. Overall, the evals needed to measure what was important for the product like whether the system returned the correct entity, whether extracted fields were accurate, and whether the output was usable downstream in the marketplace. We used Braintrust as the framework of choice for running the evals. The test cases were developed from a golden dataset built out with SMEs covering 50-100 entries per field that needed to be enriched. We used F1, precision, recall and a confusion matrix with classification report for more manual error analysis.

Re-evaluating After GPT-5

When OpenAI released GPT-5 with built-in web search, we ran an experiment using the same eval suite essentially replacing our search infrastructure and orchestration with the model's native capabilities. The results were better than what we had built over months of iteration which is not uncommon as foundation models rapidly improve. In our case, the classification metrics jumped roughly 10%. The approach was more expensive per query but much simpler to maintain and overall, performed better.

Because we had those evals already implemented and reusable, the decision to switch was supported by clear data. We could see exactly where GPT-5 with built-in search outperformed our custom pipeline, where it matched, and where it fell short.

The Next Iteration

The GPT-5 approach is now in production, but we're reaching its limits. It falls down once the number of processing and validation steps increases, namely needing to crawl, search, scrape, normalize, synthesize and cross-check. This multi-stage process cannot be formed into coherent and sufficiently accurate instructions within one prompt. Additionally, the cost is significant for built-in web search such that it is two times more expensive than using a search provider like Exa or Firecrawl. The other issue is vendor lock-in making it impossible to test other models.

So we're experimenting with a LangGraph-based architecture that gives us back fine-grained control over the search and extraction steps while keeping the stronger base model. The evals, again, are the constant, albeit an evolving one. Without them, comparing this new approach with what is already there would be difficult and vague. They also grew along with the experimentation approaches in that we needed to update our test cases to reflect the latest task implementation and the changing, dynamic nature of web content.

The Pattern

Agent architectures have a short half-life right now with new model versions, frameworks and patterns coming out every other month and new tooling too. Capabilities that used to require custom engineering suddenly become native features. If your evaluation framework is coupled to a specific implementation, you rebuild it every time you change direction, every time something new and more performant appears. However, if the evaluation framework instead measures outcomes independent of how those outcomes are produced, it transfers across every architectural shift.

This means that evals are not merely a testing phase at the end of a project, but in fact, are the infrastructure that makes architectural decisions possible. Without them, switching from a custom pipeline to GPT-5 would have been a leap of faith. With them, it was a data-driven comparison completed in days.

Practical Takeaways

Given the pace of AI foundation model improvements and tooling releases, it’s important to invest in evals even before the architecture is completely decided on and continue to invest in them as the project grows. In order to be portable between model changes, the evals should measure outcomes, rather than intermediate steps. That way, architectural decisions can be treated as experiments, where evals give a clear result to support decisions instead of relying on debates about which approach "seems" better.