EvalCore
Snapshot testing for AI behaviorRecord AI behavior offline and replay it in CI for free
Highlights
What you can do with it
- A chatbot records responses, then CI replays them after each update to check quality.
- A recommendation system records outputs and verifies stability across data and environments.
- A customer service bot is tested so updates do not change how it answers.
How to use it
- 1 Write a YAML eval file with the target app, dataset and scoring criteria.
- 2 Run the real model once with the run command to record requests and responses to a local SQLite file.
- 3 Commit the SQLite record to version control for CI.
- 4 Replay the recorded cases in CI on every code change with the cache replay command.
Who it is for
Pros
- Free to use in CI, with no API keys or network needed for replay.
- Deterministic results make tests stable and repeatable.
- Works with apps in any language and allows custom scoring.
Limitations
- It tests recorded behavior, so it will not catch brand-new failure modes you have not captured.
- You must maintain the eval file and recorded baselines as the app changes.
- Setup assumes comfort with YAML and CI workflows, so there is some learning cost.
What EvalCore is
EvalCore is a snapshot testing tool for AI behavior. Its purpose is to catch a drop in AI performance before your users notice. It works by recording the behavior of an LLM application or agent in detail, saving the request and response of every run, so you have an accurate baseline. Then, on every code change, it reruns those recorded cases in your CI environment to make sure the AI has not changed in unexpected ways. It runs offline, so there is no network dependency, and it is deterministic, so the same input always gives the same result. It needs no API keys and avoids unstable criteria, which keeps tests reliable and cheap to run. It is free to use in CI.
What you can do with it
- Record the requests and responses of an LLM app or agent as a local snapshot.
- Replay recorded cases in CI to detect unexpected behavior changes after edits.
- Run tests offline and deterministically, with no API keys and no flaky criteria.
- Set custom scoring, such as checking whether an answer is on topic or factual, for any language.
Who it is for
- AI developers who need to see how code changes affect model behavior.
- Data scientists who want to record and reproduce AI performance over time.
- AI teams that need a shared, reliable testing framework for collaborative work.
- Anyone running LLM apps or agents in CI who wants stable, repeatable checks.
What to watch out for
Snapshot testing only catches changes against cases you have already recorded, so it will not find brand-new failure modes until you add them. You must maintain the eval file and the recorded baselines as your app evolves, or the tests will drift from reality. Setup assumes some comfort with YAML and CI workflows, so there is a learning cost, and custom scoring needs care to avoid misleading results.
FAQ
Is it free in CI?
Yes, it is free to use in CI.
Does it need API keys?
No, replay runs offline from recorded responses, so no API keys are needed.
What languages does it support?
It supports applications written in any programming language.
What does it test?
It records AI behavior as snapshots and replays them to detect unexpected changes.
Why it is free
What the official page says： free to use
Free limits, trial length and features are decided by the official site and can change at any time.
Product preview
Screenshot archived from the listing page, for preview only.
