How to do AI analysis you can actually trust
Key Takeaways
Define clear quote selection rules to avoid ambiguity in AI outputs.
Always verify AI-generated quotes against original data to ensure accuracy.
Segment user responses by context to enhance analysis depth.
Iterate on your prompting techniques to refine the quality of insights.
Utilize different LLMs like Claude and Gemini for varied analytical strengths.
The Problem with AI Analysis
AI tools often produce outputs that appear confident but can be misleading, leading to poor decision-making. As Caitlin Sullivan highlights, outputs can contain fabricated quotes, generic insights, and contradictory information that can skew results. This is particularly problematic in user research where decisions based on flawed insights can lead to significant financial repercussions.
Understanding AI's Limitations
AI struggles with unstructured data like interviews and surveys. Interviews can be messy, with participants providing contradictory information or going off-topic. AI tends to simplify this complexity, missing critical nuances. Similarly, survey data, while structured, can be misleading due to lack of context. For instance, a response like "It wasn't for me" lacks the depth needed for meaningful analysis. Understanding these limitations is crucial for effective AI use.
Four Common Failure Modes
- Invented Evidence: AI may generate fictional quotes or combine multiple responses inaccurately. This can happen due to ambiguous prompting.
- False or Generic Insights: AI can produce insights that sound plausible but lack depth or specificity.
- “Signal” That Doesn’t Guide Decisions: AI might highlight patterns that don’t lead to actionable insights, causing confusion.
- Contradictory Insights: Different outputs from AI can present conflicting information, making it difficult to draw conclusions.
Effective Prompting Techniques
To mitigate these failure modes, implement specific prompting techniques. For example, when requesting quotes, specify the need for verbatim responses and provide clear rules for what constitutes a valid quote. Instead of asking for a summary, prompt the AI to highlight contradictions or nuances in the data. This will help maintain the integrity of the customer’s voice and ensure that insights are grounded in actual responses.
Choosing the Right LLM for Analysis
Different LLMs excel in various aspects of analysis. Claude is best for thorough analysis with depth, while Gemini offers strong evidence-based themes and can analyze non-verbal cues in video data. ChatGPT, while creative, is less reliable for evidence. For comprehensive analysis, Claude is recommended, but be prepared to verify themes and quotes rigorously.
Implementation Steps for Reliable Insights
- Define Quote Selection Rules: Establish clear criteria for what a valid quote looks like to avoid ambiguity.
- Verify Quotes: Always cross-check AI-generated quotes against the original data to ensure accuracy.
- Segment Data: When analyzing, segment user responses by need or context to avoid oversimplification.
- Iterate on Prompts: Continuously refine your prompts based on the outputs you receive to improve the quality of insights.
- Use Multiple Models: If possible, run the same analysis across different LLMs to compare outputs and identify discrepancies.
Why it matters
Mastering these techniques will enhance your ability to derive actionable insights from AI, significantly improving your decision-making skills. This expertise not only boosts your analytical capabilities but also positions you as a trusted resource in your organization.
Get your personalized feed
Trace curates the best articles, videos, and discussions based on your interests and role. Stop doom-scrolling, start learning.
Try Trace free