The problem

At Odyseek, I was building our career agent as the founding engineer, while our CEO, Cat, held the domain expertise. As an experienced hiring manager, she knew right away when the agent's advice sounded right and when it missed the mark.

The problem was how she was testing prompts: she was tweaking a prompt in a single chat window, reading the reply, and tweaking it again. Nothing was saved, there was no easy way to compare different models side by side, and every prompt change still needed an engineer to test properly and ship.


What we did

Instead of just handing her a login, I sat down with her and set up a simple workflow in Vellum so she could test and compare prompts herself:

  1. Version every prompt: We moved prompts out of chat windows and into versioned workflows with clear inputs, so every change was saved and easy to roll back.
  2. Test against real examples: We pulled a small set of real user cases so she could test every prompt change against the exact same examples instead of guessing from a single chat reply.
  3. Compare models side by side: She could run the same test cases across OpenAI, Anthropic, and Gemini models in one view instead of copying and pasting across browser tabs.
  4. Score what "good" looks like: We turned the things she naturally checked as a hiring manager into simple evaluation rules so she could see which model and prompt actually scored best.

The result

  • No more waiting on engineering to test prompts: Cat went from relying on an engineer for every prompt tweak to running tests, comparing models, and shipping better prompts on her own.
  • Real confidence in every change: Every prompt update was backed by side-by-side test results instead of gut feel.