How to evaluate an AI product idea before building it

Problem fit

Define the product decision before discussing a model

Write the product thesis in operational language: for a named user in a named context, improve a specific decision or artifact from the current baseline to a measurable target without exceeding defined risk and cost. Observe the workflow rather than relying only on stakeholder description. Capture inputs, handoffs, waiting, rework, exceptions and the action taken from the output. A summarization feature has little value if the actual bottleneck is missing data or approval. A recommendation can be dangerous if the user cannot inspect the basis before acting.

Measure the baseline as a distribution. Include cases per period, active handling, elapsed time, error categories, downstream correction, abandonment and consequence. Segment by complexity and user group. Do not turn an uncertain observation into a single precise figure. Identify who experiences the pain, who pays, who bears failure and who must change behavior. A product can save one team minutes while transferring review or liability to another. The target should represent system improvement, not local output speed.

Intervention

Choose the smallest useful AI role and compare simpler alternatives

Name what the system will do. Retrieval finds sources. Extraction turns content into fields. Classification routes cases. Drafting creates a proposed artifact. Recommendation offers a ranked option. Action changes an external system. These roles carry different evaluation and control needs. Start with the smallest unit that creates user value. A draft with citations may remove blank-page effort while retaining review. Autonomous action may add little benefit if the process already requires authorized approval.

Compare the idea with structured data capture, search, deterministic validation, workflow changes, templates, policy clarification and added human capacity. AI can be part of the winning design without being the whole solution. A hybrid may use rules for mandatory constraints, a model for ambiguous content and humans for consequential exceptions. Record why each alternative fails or succeeds against the same baseline, not why AI feels more innovative. This comparison often exposes product prerequisites that matter regardless of model choice.

• Name the role as retrieval, extraction, classification, drafting, recommendation or action.
• Minimize the unit of behavior that must be trusted.
• Compare structured, deterministic, process and staffing options.
• Use hybrid boundaries based on ambiguity and consequence.
• Evaluate every alternative against the same target and constraints.

Feasibility

Prove that the idea can be evaluated before selecting a model

Inventory production-like inputs, their source, permission, retention, sensitivity, languages, formats, rarity and expected change. A large document archive is not automatically usable training or evaluation data. Determine whether the team can obtain representative cases and whether some groups or failures are systematically absent. If historical decisions contain inconsistent policy or bias, imitating them is not a valid target. Define a reference policy and adjudication route for ambiguous examples.

Build an evaluation design around the decision. Include ordinary cases, difficult cases, out-of-scope inputs, adversarial or malformed content and high-consequence segments. Define correctness, completeness, grounding, calibration, latency or other task-specific measures. Create a failure taxonomy and identify errors that demand separate thresholds. Test reviewer agreement; if qualified humans cannot agree, the product may need clearer policy, different scope or an assistive design. Aggregate model quality alone cannot answer whether the product is safe or useful.

Product viability

Include human control, exceptions and ongoing evaluation in the business case

Map consequence and reversibility. Low-consequence suggestions may allow lightweight review. Decisions affecting access, safety, rights, money or contractual commitments need stronger authority, evidence, logging, override and appeal. Define what the system does when confidence is low, input is out of scope or a dependency fails. Abstention and escalation are product outcomes, not defects to hide. Give users enough context to exercise judgment and avoid interfaces that imply certainty the system does not possess.

Model end-to-end economics at realistic volume and concurrency. Include data preparation, integration, model calls, storage, retrieval, latency engineering, monitoring, evaluation refresh, human review, exception queues, support and model or policy change. Compare saved handling and improved outcomes with new work, not only inference cost. Then stage evidence: manual concierge test, technical spike, offline benchmark and bounded pilot. Each gate should retire a specific uncertainty and have advance, revise and stop conditions set before results. A good discovery can conclude that a narrower non-AI product is the better investment.

• Scale human authority with consequence and reversibility.
• Design abstention, fallback, override and appeal explicitly.
• Calculate the full operating system, not token cost alone.
• Make every discovery stage answer one risky assumption.
• Permit stop or scope reduction as a successful evidence outcome.