App Bench is how we test Framer 4. It checks whether the apps the agent builds do everything we want, and whether they get better or worse as the agent changes.
Each time we run it, the agent builds the same 20 apps from scratch. These are real products, not tiny to-do apps: a course platform, a booking page, a Shopify store, a team CRM, a forum and more, with sign-in, payments, saved data and email.
A run starts 20 customers at once, one per app. Each customer is an outside agent, separate from Framer 4. It reads two files, opens a new Framer 4 project, and asks the Framer agent for the app the way a real customer would. The agent never sees these files:
- spec.md: what the app does and the checks it must pass
- persona.md: how the customer writes, from short to long
How a run works:
- Customer: sends one prompt at a time
- Agent: builds from those prompts only
- Judge: tries the live app and runs the checks
- Repeat: until every check passes, or 40 prompts
An app passes only if every check passes. If the judge can't run a check, it shows as Not checked, and the app can't pass until it is checked. Not checked is a gap in our setup, not the agent's fault; the gaps are listed in the Blockers tab.
Each customer writes in one of four styles, based on real prompts in Raindrop. Each style is used for 5 apps.
- Tweak: short commands
- Describe: a sentence or two
- Specify: long, precise instructions
- Brief: one long brief, then small fixes