a task-trained open source model vs the frontier

Part IV

A digital twin the model can practice in

A model can only practice a workflow it can actually perform, fail at, and retry. So we
rebuilt the catalog workflow as a digital twin of an e-commerce platform: a simulated
environment with the same listings, the same tools, and the same stakes as the real
thing. The raw material is the
Amazon Berkeley Objects
dataset of real product images and listing data, which we turned into
177,767 review episodes: one listing each, with image, title, description,
claimed brand, and region. Into that stream we planted controlled policy cases,
mismatched images, conflicting brand claims, and deliberately legitimate
claims as hard negatives, so every episode has a known correct answer to score against.

Inside the twin, the model works the way an analyst would. It can search a taxonomy of
roughly 13,000 categories, check whether a brand is registered and protected, and
retrieve the required attributes for a chosen category; then it must commit a category,
the attributes, and a policy decision. A scorer grades every episode, rewarding correct
outputs and penalizing missed violations, unsupported attributes, invalid categories,
and wasted tool calls. The penalties encode business priorities directly: a missed
violation costs 7× more than a false alarm.