The frustration
Every time I built an agent, I’d write a one-off script to evaluate it instead of reaching for an eval library.
I tried the obvious ones — ragas, deepeval, promptfoo. Each came with framework complexity I didn’t want or need at the start of a project. One came with a class hierarchy to learn, another a strange config DSL to fill in, and they all pushed me towards a hosted UI to log into. So much setup before I’d even answered “does my agent answer the question correctly?”
Evaluating an agent isn’t complicated. Call your agent with a question, get the response, use an LLM judge to score the response against an expected value. Three lines of pseudocode. With a coding agent you can have a working script in a minute.
The problem with this, though, is the script grows unwieldy. You add a second metric. Then a way to cache judge calls because they cost money. Then a way to save results because you want to compare runs. Then a way to detect when the agent’s topology changed so old results aren’t compared against new ones. At some point you’ve built half an eval library, badly, and now you’re maintaining it.
This library is the middle ground between ease of use and growing pains.
Decisions
Metrics are functions
Python developers, for better or worse, reach for functions before classes. So a metric is just a function:
def my_metric(row: dict) -> float:
return 1.0 if row["answer"] == row["expected_answer"] else 0.0
evaluation = Evaluation(metrics=[my_metric, exact_match])The cost is discoverability. A user finds metrics by reading the README, not by something like agentstax_eval.<metrics>.
Judges are functions
Same idea, different signature:
def my_judge(prompt: str) -> str:
return openai_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
).choices[0].message.content
metric = llm_correctness(llm=my_judge)The built-in providers — openai_provider(), anthropic_provider(), google_provider() — are written this way too. No Provider base class means no shared retry logic, no shared backoff, no shared rate-limit handling. Most users will eventually hit a 429 and have to write their own. I think this is fine — retry logic should be handled carefully and explicitly because it costs money each try.
Zero core dependencies
pip install agentstax-eval pulls down agentstax-eval and nothing else. The openai, anthropic, and google-genai packages live in optional extras. If you only use one of them, you don’t get the other two.
The trade-off is that the providers are lazy — openai_provider() only imports openai inside the function body. If you haven’t installed the package, you don’t find out until you call the function:
>>> openai_provider()
ImportError: The openai package is required for openai_provider.
Install it with: pip install openaiThe message tells you exactly what to do, but discovering a dependency at runtime isn’t a great experience. My reasoning is that the provider dependency should normally already be installed from the main agent source code such that this friction is not met often.
Four objects, named for what they do
Dataset, Task, Evaluation, Pipeline. Each does one thing:
pipeline = Pipeline(
dataset=Dataset([{"question": "...", "expected_answer": "..."}]),
tasks=[Task(call_my_agent)],
evaluation=Evaluation(metrics=[exact_match]),
)Most people who’ve done agent evaluation already know the shape of it. You have a dataset, a way to call your agent on each row, and a way to score the output. I named the objects after them. Pipeline is the fourth — the thing that runs them in order, the word the user would already use.
Pipeline is also just a convenience wrapper. A user can reach for it and never see the underlying Task.run or Evaluation.run.
Pass the agent directly
When trying to enrich evaluations with agent metadata, other libraries I’d used, like ragas, wanted me to describe my agent to them — a config dict, a yaml file, a list of tool names and a model string. The problem with this is you had to keep your real agent in sync with these descriptors, which is one of those small ongoing taxes that adds up.
agentstax-eval takes the agent object directly:
pipeline = Pipeline(
dataset=dataset,
tasks=[Task(call_my_agent)],
evaluation=Evaluation(metrics=[llm_correctness(llm=judge)]),
agent=my_langgraph_agent,
)The library walks the agent and figures out the rest — name, model, system prompt, tools, sub-agents, the edges between them. Six framework extractors live behind that one keyword argument: LangGraph, Google ADK, OpenAI Agents SDK, CrewAI, LlamaIndex, Microsoft Agent Framework. Each one represents “an agent” differently. Each extractor normalizes its framework’s shape into the same AgentGraph.
Five of those six were actually straightforward. The frameworks expose what you need as attributes — agent.model, agent.handoffs, agent.tools. You read them, you map them, you’re done.
LangGraph doesn’t work that way. The model isn’t an attribute on the agent — it lives in a closure cell of a function bound to a node. To find sub-agents, the extractor has to read the bytecode-level co_names of the tool function and resolve them through __globals__, looking for a CompiledStateGraph. That’s a fairly invasive read of someone else’s library internals, and it’s the only way to handle this extraction logic, so I wrote it.
The LangGraph extractor is roughly three times the size of any other one in the codebase. It’s coupled to internals that aren’t part of LangGraph’s stable interface, so any time LangGraph refactors those internals, the extractor breaks. It’s the most fragile part of the library and I knew it would be before I wrote it. The alternative was telling the user to describe their LangGraph agent in a config dict, and the whole point of this design was not making the user do that.
What I left out
Async-first was tempting. To me, the default execution model is synchronous. You write pipeline.run() and it runs. Async is there if you want it (pipeline.run_async(concurrency=N)), but it’s an explicit layer on top, not the standard path. Most users running an eval on a small dataset don’t need it. Most users running an eval on a large dataset do, and when they reach for it, it’s there.
What it cost
The lived cost of most of my design decisions is discoverability. I love intellisense and use it everywhere — I’ll define complex terraform object types just to get autocomplete I can rely on. agentstax-eval doesn’t give that back to its users. They have to know what to import and read the README to find out what’s available.
Dependency injection is also absent, which is an incredibly powerful construct. Java Spring and NestJS both use it to power enterprise-grade applications. Agentstax-eval is probably not for that hardcore enterprise user. That niche has already been met by others. This library is for the scrappy custom agent creator looking to bootstrap a testing suite quickly, and I made it less powerful so it could be more inviting.
What I’d do differently
I’d revisit how LLM providers are installed. Zero core dependencies is a clean install story and I still believe in it, but the way it plays out — call openai_provider(), hit an ImportError, realize you needed to install the package — is worse than I’d like. The fix is probably to ship the SDKs as core dependencies and eat the dependency-tree weight. The clean install was a small vanity. The late ImportError is a real issue.
Close
What I have at version 0.1.0 is a library that asks the user to learn almost nothing. It fails them with two specific kinds of pain — a late ImportError if they pick the wrong provider, and a fragile coupling to LangGraph internals that I can’t make go away. Those trade-offs are the price of the shape I wanted. I’m watching to see which one I get tired of first.