App Bench
App Bench
Fail · 11
Booking pagePractitioner email could not be checkedCourse sitePurchase and new lessons unverifiedEmail digestDaily email not sent from the previewEvent RSVPConfirmation email could not be checkedLead quizQuote email could not be checkedPaid AI toolSubscription could not be checkedPaid membershipCard payment could not be checkedReview toolManager feedback could not be checkedShopify storeShopify order could not be completedTeam dashboardClosing a ticket was not checkedTemplate shopNo sale possible without a file upload
Pass · 9
AI chatScrappy bike shop with a chat helperClient portalAn editorial studio portalFile converterStatements become a CSVForumA heritage forum for watch restorersListings siteA coffee-magazine roaster directoryReturns trackerA big-screen returns logSite with loginPlayful podcaster site with gated readingTeam CRMA light, shared sales pipelineTeam whiteboardA live weekly board for a small team

spec.md

We write one spec.md per app, the way its owner would describe it. After every prompt, the judge reads its checks and tries each one in the live app. The agent never sees it.

  • Goal: who it's for and what it does
  • Accounts: the test logins the judge uses
  • Checks: things a person can try in the app
  • Out of scope: what the app shouldn't have

Some checks carry a weight:

  • Must pass: the app fails without it
  • Serious: a big flaw
  • Minor: a small flaw

A real check

From Client portal:

  • Check: a client can't open another client's project
  • Evidence: saw "This project isn't available"
  • Result: Pass