How to score your AI teammate’s performance

In an illustrative support case, a customer asks why an API import has stopped. An AI teammate finds a troubleshooting note and drafts a reassuring answer: “The import will retry automatically.” It reads well. But another, newer guide says the team must investigate before making that promise.

Would you send the draft? If the answer is no, fluency was the wrong thing to grade.

A founder needs a way to judge the work an AI teammate produces, find out why it missed, and see whether a correction holds. Start with one task, define what you would accept from a teammate, and review actual outputs against that standard. A stop-worthy mistake cannot be canceled out by a lovely tone or a faster draft.

Define the job before you score it

“Be helpful with support” gives you nothing concrete to assess. A job statement should say what the AI may read, what it should prepare, and where it must stop.

For the import question, a usable assignment might be:

Prepare a reply using the current approved import guidance and this customer’s permitted support history. Show where each factual claim comes from. If sources conflict or the cause is unknown, flag that for the support owner. Do not promise a retry or fix date. Leave the reply unsent.

Now the reviewer knows what success would look like: a response that deals with the question, uses the right guidance, states uncertainty plainly, and gives the customer a next step the team can own. A draft is the deliverable. A sent message, a changed ticket, or a repaired import would be different jobs with separate authorization and checks.

Choose more than the easy case. Try a request with complete guidance, one with an outdated or conflicting note, and one where a necessary source isn’t available to the person running the task. Record a case you couldn’t test as not run. That small discipline keeps the score from becoming a story you wish were true.

Score the work a founder would actually use

Score both the outcome and the path to it. One cannot rescue the other.

Use this editorial scorecard, not a universal AI rating. For each case, mark the result accept for this task, revise, stop, or not run. A stop means the team must hold the output or action; it is not a low point in an average.

A case can satisfy one row and fail another. Keep the row-level reasons. If you later compare effort or cost with the team’s current process, include setup, source review, corrections, and the work a person still completes. A quicker first draft alone is not a savings result.

This distinction matters beyond a founder’s own worksheet. Enterprise-Bench examines how agents perform under realistic conditions such as data scale, access, reliability, and cost. Its findings are not a score for your customer’s import question; you still need to review the work in front of you.

A convincing answer can still fail

In the fictional example, the AI chooses the convenient source and tells the customer to expect a retry. That is a stop – for truth and for the unsupported promise. The message has not been sent; the support owner must check which guidance applies and what the team can honestly say.

Notice what the failure is not. It is not simply “the AI needs a warmer tone.” It found a plausible source and drew the wrong conclusion about its authority. Editing “will retry” to “should retry” would make the sentence softer without fixing the underlying problem.

Improve the cause, then test again

Feedback helps when it points to a change somebody can make and verify. Diagnose the miss before changing the instructions.

• Outdated or conflicting source: Have the knowledge owner review the current guidance, retire or mark the obsolete material, and preserve the approved source. Treat this as a knowledge-management problem, not something the AI should settle by confidence alone.
• Right source, wrong conclusion: Clarify the task instructions: when guidance conflicts, show the conflict and ask for owner review instead of producing a customer promise.
• Missing access or unsupported action: Stop the task and route it to someone authorized. Better wording cannot create permission or a tool that the setup lacks.
• Incomplete but accurate reply: Add a reviewed example of the answer shape the customer needs – acknowledgment, confirmed fact, open question, and owned next step – without copying a real customer’s private details.

Call this improving the AI teammate’s work, not training a model on every interaction. A correction in one chat does not, by itself, update a shared customer agent or its underlying model. Personal writing instructions, reviewed knowledge, and a configured customer-facing agent are different controls, with different owners.

After one approved change, rerun the failed case. Then try a fresh import question that wasn’t used to write the correction, plus a case that worked before.

Did the AI now surface the right source? Did it avoid the unsupported promise without becoming vague on a routine question? Keep the actual outputs and reviewer edits. Until those tests are done, the fix remains a proposal. A broader AI agent evaluation guide can help the team build a repeatable test practice.

Where Computer fits in the review

Computer, by DevRev, is built to work with relevant, permission-aware context through Computer Memory. For this task, ask it to show the source behind a proposed reply so a person can check the guidance and the customer’s situation. If the approved material isn’t available or two sources disagree, a useful result should make that gap visible for review rather than act as if it knows the answer.

If your organization has an enabled Agent Studio setup and an authorized builder, preview tests and structured tests can compare expected and actual outputs, surface errors, and help examine a changed agent version. Those test results still need a person to decide whether the customer-facing work is acceptable. Confirm the exact features and access in your startup’s configuration before relying on them.

A founder does not need a perfect global score to make the next call. Keep this task under review, narrow it to answers the team can verify, or pause it until the source problem is fixed. Name who will make the change and who will judge the retest. That is how trust grows – one accepted piece of work at a time, with the exceptions still in view.

Explore Computer for Startups and check the benefits and eligibility for your stage.