Define a useful AI task
Turn an ambitious idea into a testable specification with a sensible fallback.
- Specify a target and constraints
- Identify the cost of a wrong answer
- Write acceptance criteria
Write the contract
“Make a smart assistant” is too broad to evaluate. “Given a public help article and a question, return a relevant passage or ask for clarification” specifies inputs, outputs, and an allowed fallback. Add language, input size, response time, and data boundaries.
Choose a metric that reflects the actual decision. A routing tool may need high recall for urgent messages, while an automatic action may need high precision. A model can score well on a convenient benchmark and still fail your product requirements.
Define failure before success
A fallback is part of the design. Missing input, unsupported language, uncertain predictions, and unavailable services should have explicit outcomes. Returning “needs review” can be more useful than a confident but unsupported answer.
Acceptance criteria must be checked on examples not used to shape the rules. Track a baseline, a target, and a release threshold. Keep a human review path for consequential decisions and do not present a probability as a guarantee.
A small experiment you can run.
A contract can be checked without any AI. Validation catches malformed outputs but cannot prove that a well-formed prediction is correct.
spec = {"max_characters": 500, "allowed_labels": {"billing", "account", "review"}}
def validate(message, prediction):
return (isinstance(message, str) and 0 < len(message.strip()) <= spec["max_characters"]
and prediction in spec["allowed_labels"])
checks = [("Invoice question", "billing"), ("", "account"), ("Hello", "unknown")]
for text, label in checks:
print(validate(text, label))
Save the file, open your terminal in that folder, and run python define-a-useful-ai-task.py. Use python3 or py if required by your installation. Setup guide
The script prints True, False, False. Your contract should include an uncertainty path.
Write a release contract for a study assistant.
- Specify input and output types.
- Choose three measurable acceptance checks.
- Describe what happens when a source does not contain the answer.
Compare with a suggested solution
Use nonempty questions, bounded document sizes, and a response containing a passage ID or an explicit no-answer status. Check relevance on held-out questions, passage existence, and response time independently.
One idea to take with you.
Make it part of your progress.
Finish the practice and answer the knowledge check to mark this lesson complete.
Go deeper with primary documentation
Optional references for further study. This lesson and its examples were written for Artificials.
Python: virtual environments