The safety net you lose when you go on-device
Cloud agents fail in public, but they also get fixed in public: someone watches a dashboard, spots the bad answer, and pushes a patch the same afternoon. An on-device companion is different. Once a LyboAI Edge companion is signed and running on someone's phone or laptop, it does its work locally — often offline, often with no telemetry watching over its shoulder. That's the whole point of on-device AI, and it's also why you can't treat testing as something you do after launch.
This is the problem evals exist to solve in Edge Studio. Before a companion moves from a Canvas draft to a signed build, you run it against a battery of scenarios and see, in black and white, how it actually behaves. Not how you hope it behaves — how it behaves.
What an eval actually is (no mystique required)
Strip away the jargon and an eval is three things: a scenario, an expectation, and a verdict. A scenario is a realistic input — a customer asking for a refund outside the policy window, a patient describing symptoms that need escalation, a user trying to talk a retail agent into a discount it shouldn't give. An expectation is what a good response looks like: the facts it must contain, the tone it should hold, the lines it must never cross. The verdict is the comparison between the two.
In Edge Studio, evals live alongside the project they test. You build a suite the way you'd build a set of unit tests: start with the happy paths, then add the awkward cases you've seen in real conversations, then add the adversarial ones you hope never happen. Each time the companion changes — a new prompt, a new tool, a new knowledge source — the same suite runs again, so you know immediately whether the change helped or quietly broke something three features away.
Where evals sit in the Edge Studio pipeline
Edge Studio's workflow runs from Canvas, where you sketch the companion's behaviour and wiring, through projects, where it becomes a versioned thing you can iterate on, to signing, where a build is sealed for distribution to a device. Evals are the gate between iteration and signing. The flow below shows the loop most teams settle into: design, run the suite, read the results, fix what failed, and only then sign.
The discipline this enforces is simple but valuable: a signed build is always a build with receipts. When someone asks “how do you know it handles refund disputes correctly?”, the answer isn't “we tried it a few times in chat” — it's a suite of named scenarios with pass marks attached to that exact version.
Reading the results like an operator, not a researcher
Eval output is only useful if a busy team can act on it. The workspace view below is the shape we aim for: scores over time, so you can see whether this week's version is better than last week's; breakdowns by scenario group, so a dip in “billing questions” doesn't hide inside a healthy overall average; and the individual transcripts behind every failure, because the fix almost always lives in the specifics of one bad answer.
Two habits make the numbers trustworthy. First, treat a falling score as a stop signal, not a discussion point — if the suite got worse, the change doesn't ship. Second, keep growing the suite from production reality: every odd conversation your companion has in the field is a candidate scenario for the next run.
The on-device wrinkle: test where it will actually run
There's one more layer that cloud-only teams never have to think about. An Edge companion doesn't run on your infrastructure — it runs on whatever hardware your users hold, under Lybo OS on the device side. Memory pressure, thermal throttling and offline stretches are part of its real operating environment, so a suite that only ever runs in ideal conditions is telling you a comfortable half-truth.
Practically, that means two tiers of testing. Behavioural evals answer “does it say and do the right things?” and can run quickly on every iteration. Device-profile checks answer “does it still do those things on the low-end tablet, with the network off?” and run before a release candidate is signed. The diagram below shows how the tiers stack.
Start smaller than you think
Teams sometimes stall on evals because they imagine a hundred-scenario suite as the entry price. It isn't. Ten scenarios that reflect your ten most common real conversations will catch more regressions than zero scenarios ever will, and the suite grows naturally from there. If you've already run a VALUE-AI assessment, you have a head start: the risks and success measures it surfaces map almost one-to-one onto your first eval expectations.
If you're building a companion in Edge Studio — or deciding whether an on-device agent fits your organisation at all — start at lyboai.app. Sketch it in Canvas, give it a report card before you give it a signature, and ship with receipts.