So I started by designing scenarios. Every scenario had a different purpose: budget constraints, route optimisation, mid-trip disruptions, remote work scheduling across time zones, and information gathering. Every scenario had its own scoring system with explicit pass/fail criteria, and I wrote all of them before I wrote a single line of agent code.

Then I added what I thought would make everything better: a second pair of eyes... a quality auditor which is commonly called a reflection agent. Its job was simple: critique the itinerary, point out problems, and push the planner to improve it. Generate a plan, critique it, revise it, repeat.

It went a bit off the rails.

But here's the thing... that's not really what this article is about. What I was actually building wasn't a travel app. The eval suite is the product. The travel agent is just the first thing it grades.

The evaluation scenarios were already in my head before I even thought about building an agent. I'd spent months planning this trip myself, so I was constantly thinking about different situations that could actually happen. For example, one of the scenarios came from my own remote work schedule. I have fixed working hours but I also want to experience the city, the culture, and the country. How do I manage both efficiently? That became one of the benchmark scenarios.

The same thing happened with route optimisation. I was traveling from Korea to Japan during autumn, and i wanted to maximise the peak foliage season. Korea reaches its peak a little earlier, while Japan follows from north to south. That wasn't just an interesting benchmark idea; it was a real planning problem I wanted the agent to solve correctly.

That's why writing the evaluation scenarios came naturally. They were based on real problems I had already thought through for my own trip. The agent was almost the easier part. What actually mattered to me was seeing how well the agent responded to those scenarios and constraints. I already had an idea of what a good answer should look like, and i wanted to measure how close the agent could get to it.

There was another reason I built it this way. If I had built the agent first, I would have ended up becoming the reflection loop myself without even realizing it. I would have looked at whatever itinerary the agent produced, thought it looked reasonable, and then started writing scenarios around what the agent already did well. The scenarios would have been shaped by the agent instead of the other way around.

Writing the scenarios first kept the evaluations honest. The agent had to adapt to the scenarios, not the other way around.

Before building the planner, I wanted to define the kinds of problems it should actually be able to solve. These weren't random benchmark scenarios or academic test cases. Every one of them came from a real question I had while planning my own trip. Instead of asking, "Can the agent plan the trip?", I wanted to ask, "Can it solve the problems that actually matter when you're travelling?"

The first scenario was Budget. Almost everyone travels with a fixed budget, so I wanted to see whether the agent could maximize the overall experience within a fixed limit.

The second was Route Optimization, where the planner had to follow the autumn season while avoiding unnecessary backtracking.

The third was Remote Worker, balancing fixed meetings with actually experiencing the country.

The fourth was Mid-trip Replanning, testing whether the planner could adapt to disruptions without throwing away the original itinerary.

The final scenario was Information Gathering, evaluating whether the planner knew when to gather more information before making planning decisions.

Together these five scenarios became the benchmark suite. They weren't designed around the agent's strengths, they were designed around the real problems I wanted solved before leaving for my trip.

What the Benchmarks Revealed

I finally ran the benchmark suite expecting the reflection agent to improve every scenario. That was the whole reason I built it. The idea was simple: let the planner decide the itinerary, let the reflection agent critique it, revise the plan, and end up with something better than the original.

Instead, the original benchmark taught me something much more valuable. The planner itself wasn't really the biggest problem. The reflection loop didn't always know what deserved changing and what should simple be left alone. Without the benchmark suite, I probably would've looked at the itineraries, thought they looked reasonable, and never noticed these weaknesses.

Benchmark Results

Planner-only (v1) vs Planner + Reflection (v2) after introducing the first reflection loop.

The biggest weakness showed up in the Remote Worker scenario. This benchmark came directly from my own trip planning, where I have fixed working hours and meetings that simply can't move. The planner sometimes scheduled sightseeing, train journeys, or other activities during those meeting windows because it wasn't consistently treating timezone-aware work schedules as hard constraints. Reflection improved the score from 21.45 to 35.85, but it was still the weakest-performing scenario by a considerable margin.

The second issue appeared in Mid-trip Replanning. The goal was to see how well the planner could adapt when something unexpected happened without throwing away the rest of the itinerary. Instead, the reflection loop often rewrote parts of the trip that weren't affected at all, sometimes even modifying locked bookings. Rather than making a small, local adjustment, it tried to regenerate large sections of the itinerary. That caused the score to fall from 83.95 to 73.45.

The most interesting result came from Route Optimization. The original planner had already produced a sensible route with very little backtracking. Instead of recognising that it was already good enough, the reflection agent kept suggesting additional changes anyway. It wasn't fixing real problems anymore; it was simply editing for the sake of editing. The score only dropped slightly (83.60 to 82.85), but the lesson was much bigger than the numbers.

That first benchmark run completely changed how I thought about reflection agents. They aren't automatically an improvement. Without clear stopping criteria and well-defined constraints, a reflection loop can easily become an overcritical editor that keeps changing things simply because it can.

The Fixes

The benchmark results made the next steps pretty obvious. I wasn't trying to redesign the planner from scratch anymore. I was trying to fix the specific weaknesses that the evaluation framework had exposed.

The first fix focused on the Remote Worker scenario. I updated the reflection agent so it performs a strict check for timezone conversions and calendar conflicts before approving an itinerary. Working hours and meeting slots shouldn't be treated as suggestions, they're hard constraints that the planner has to respect.

The second fix addressed Mid-trip Replanning. I changed the revision prompt so that locked bookings became immutable, and the planner was only allowed to modify the part of the itinerary that was actually affected by the disruption. If a typhoon affects Day 12, there shouldn't be a reason to rewrite Day 18.

The final fix was for Route Optimization. I added a clear stopping condition to the reflection loop. If an itinerary already satisfies the important constraints, the reflection agent simply returns ITINERARY APPROVED instead of continuing to look for problems that don't really exist.

Those were relatively small changes, but they raised a much more interesting question: would they actually improve across different models, or had I just optimized for one benchmark?

Did the Fixes Actually Generalize?

It wasn't enough for the fixes to work on just one model. If they only improved the benchmarks on the model I had been testing with, then I probably hadn't fixed the underlying architecture... I had just optimized the prompts for that particular model. To test that, I reran the entire benchmark suite across four different LLMs: GPT 5.6 Terra, Gemini 3.1 Pro, Gemini 3.5 Flash, and Llama 3.1 8B.

The results were encouraging. The biggest improvements appeared in the Remote Worker and Mid-trip Replanning benchmarks. On Gemini 3.1 Pro, the Remote Worker scored improved by 36.95 points, showing that treating work hours and meeting slots as hard constraints made a significant difference. GPT 5.6 Terra also showed a 16.90 point improvement in the Replanning benchmark after I introduced immutable bookings and localized revisions instead of rewriting the itinerary.

Not every result improved, and I think that's just as important to mention. Llama 3.1 8B produced noticeably weaker scores in the Budget and Information Gathering benchmarks. After looking into the results, most of that variation came from limited factual database coverage rather than changes in the planning logic itself. I decided to keep those results instead of removing them because they represent the system's actual behaviour, not the outcome I was hoping for.

Overall, I wasn't trying to get perfect scores across every benchmark. What mattered more was seeing the same architectural changes produce consistent improvements across different model families. That gave me much more confidence that I was improving the system itself rather than accidentally overfitting prompts for a single model.

Beyond Travel

The more I worked on this project, the more I realised it wasn't really about travel anymore. I picked travel because it's something I already had real experience with. I had already spent months planning this trip, so it felt like the perfect domain to start learning AI evaluations. But somewhere along the way, the project stopped being about itineraries and started being about something much bigger.

One of the biggest things I learned is that building the agent isn't the difficult part. You can always improve prompts, switch models, or add another agent into the workflow. The difficult part is knowing whether those changes actually made the system better. That's what good evaluations give you. They give you an objective way to measure whether your changes are genuine improvements or just different outputs that happen to look convincing.

This project completely changed how I think about building AI agents. Before this, I mostly thought about generating better responses. Now I think much more about designing better evaluations. If the evaluation itself isn't honest, then it doesn't matter how good the agent looks, you have no reliable way of knowing whether it's actually improving.

I also realised that these problems aren't specific to travel planning. A reflection agent that keeps changing an already-good itinerary isn't very different from a coding agent that keeps rewriting working code, or a scheduling agent that keeps moving confirmed meetings. The underlying problem is the same. Good constraints and good evaluations matter regardless of what the agent is trying to do.

I started building a travel planning agent because I wanted to work with AI evaluations. In the end, the travel planner became the example. Learning how to design honest evaluation frameworks became the real project.

Limitations and What's next

Like any early-stage project, this one still has its limitations. The research agent currently works with a limited mock knowledge base rather than live APIs, so some benchmark scores, especially factual ones can fluctuate depending on the model being evaluated. Right now, the framework is primarily testing planning logic, constraint satisfaction, and decision making rather than real-time data retrieval. That was a conscious trade-off while building the evaluation framework itself.

The next step is to make those evaluations more realistic. I'm currently expanding the framework with Model Context Protocol (MCP) integrations so the agents can interact with real tools and live data instead of mock datasets. After that, I want to build a second agent in a completely different domain to see whether the same evaluation-first approach generalizes beyond travel planning.

The project started because I wanted a better way to plan my own trip across Korea and Japan. Along the way, it became a way to learn how to build and evaluate AI agents more systematically. The travel planner will continue to evolve, but the evaluation framework is what I'm most excited to keep building.

If you're working on agent systems or AI evaluations, I'd genuinely love to hear your thoughts or feedback. The repository includes the evaluation scenarios, benchmark reports, and the complete agent implementation, and I'll continue publishing new experiments and improvements as the project evolves.

🔗 GitHub: Agent Eval Platform

💼 LinkedIn: Linkedin

If you spot something I could evaluate better, or evaluate differently... I'd genuinely love to hear your perspective.