UX Research Case Study
How to design an AI data-analysis tool for human verifiability and control?
Exploratory evaluation of the human-AI interaction design of a findings-generation module for pre-structured qualitative data.
Overview
Impact
The layout and one new feature identified as a success driver were shipped in the production version. Three early-stage features are being considered for further exploration.
Challenge
Users of AI tools for qualitative data analysis want to control and verify AI-generated findings. It was unclear how the company’s prototype performed on these core needs and how it could be improved. There was also no suitable evaluation framework defining what to measure and how.
Key findings
Basic features drive Control B and Verifiability
Control A is addressed too narrowly by the design
Speculative AI-native features are promising differentiators but need more exploration
What I did
Ran a user study with six commercial user researchers as participants (think-aloud sessions + questionnaires). Developed an evaluation framework consisting of core user needs, questionnaire items, and scales. Designed an interactive concept mockup as stimulus for the sessions.
Client
UseTree GmbH (in-house)
Timeframe
May–Oct 2026
My role
UX Researcher, Interaction Designer
Team
PM, Senior UX Designer
Methods
Mixed-methods user study, Research through Design
Tools
Condens, Claude, Excel, Figma
Context and background
The company wanted to turn its in-house prototype of a structured multi-step AI tool for qualitative data analysis (QDA) into a commercial product. Through a pre-study and a review of recent academic studies, I identified three core user needs that critically matter to users of AI applications for QDA and that are sensitive to human-AI interaction design – especially in the findings-generation step. I conceptualized the needs as three measurable dimensions: Control A, Control B, and Verifiability.
Core user needs
Control A
Control B
Verifiability
Control A/B denote different areas of human control over AI output production; Verifiability refers to the ability to check AI output against the source data
Timeline and collaboration
After a handoff from the Senior UX Researcher previously on the project, I ran the studies end-to-end and secured buy-in from the PM and management. I also defined and created the concept mockup, which included facilitating a group ideation session with seven researchers and designers.
Timeline
May
Pre-study
Concept user study with seven researchers
June
Ideation session
With seven researchers and designers
Concept and mockup
Study stimulus, iterated four times
Study design
Measures, session guide, questionnaires
July
Recruitment and pilots
Two pilot sessions
Sessions
Six remote think-aloud sessions
Analysis and report
Descriptive stats, thematic analysis
September
Presentation
Report and recommendations to the designer, PM, and management
October
Design feedback
On the implemented production version
Upcoming
MVP launch
Planned
Problem and research questions
I had established three critical user needs – Control A, Control B, and Verifiability – but we did not know how well our concept met them or which aspects of the design were decisive for success. This knowledge would be important for targeted optimization.
The goal was to understand how MVP-ready the concept is, what needs to be done, and what additional short- and mid-term improvement opportunities exist.
Research question 1
How does the developed concept perform on the user needs?
Research question 2
Which features and properties of the interface drive the performance?
Method
Exploratory mixed-methods user study
I chose an exploratory approach because the project was at an early stage, with some fundamental understanding yet to be established. Combining quantitative and qualitative data showed both how well the concept performed and why.
Research through Design (RtD)
Combined with the mixed-methods study, I developed a dedicated interactive mockup to address the research questions optimally. Alongside the findings, it also served as a reference for developing the production version. Additionally, it allowed me to include speculative AI-native features for initial exploration.
The mockup – initial state
Analysis question selected, finding generation not yet started; confidential details blurred
Sample
Convenience sample of six external commercial user researchers – generally sufficient for an early-stage, mostly exploratory study.
Sessions
60-minute interactive and semi-structured user sessions (remote, English or German):
Pre-questionnaire: to control for extreme pre-existing attitudes regarding the core user needs.
Interactive think-aloud: to collect qualitative data while participants explored or performed tasks in the mockup.
Post-questionnaire: to collect quantitative data via 7-point Likert scales per measure and multiple-response items on intended feature use.
Discussion: to collect qualitative data specifically on participants’ ratings and responses.
Instruments
Operationalization of the core user needs
User need
Measure / item
Tested feature
Control A
Ease of Control A
Feature sufficiency for Control A
Feature set associated with Control A
Control B
Ease of Control B (aspect 1)
Feature sufficiency for Control B (aspect 1)
Ease of Control B (aspect 2)
Feature sufficiency for Control B (aspect 2)
Feature set associated with Control B
Verifiability
Ease of Verifiability (aspect 1)
Feature sufficiency for Verifiability (aspect 1)
Ease of Verifiability (aspect 2)
Feature sufficiency for Verifiability (aspect 2)
Feature set associated with Verifiability
Making core user needs and associated feature sets measurable through questionnaire items
I made performance on the user needs measurable through questionnaire items capturing two types of perceived quality: ease and feature-set sufficiency.
Example of a questionnaire item
This item represents one of the measures for Verifiability; confidential details are blurred
In addition, participants could select which features of each feature set they would use.
Analysis and synthesis
Descriptive statistics
For the quants: mean, median, and spread, plus frequencies for the nominal scales. Given the small sample, I used a table that included both the statistics and the number of individual cases for a nuanced interpretation.
Deductive thematic analysis
For the quals, I derived codes from the three dimensions, as the goal was to understand given variables rather than to discover new issues. I still kept an eye out for relevant findings outside this frame and synthesized the coded segments into a thematic matrix.
Triangulation
I triangulated the questionnaire stats with the themes. The ratings showed how the concept performed and signaled where to look closer; the themes explained why.
Tooling
Condens for session-video transcription and cross-checks. Claude and Excel for a structured thematic analysis, with dedicated tables for the codebook, coded segments, and synthesized themes. This leveraged AI for efficiency while keeping the analysis transparent and under human control.
Analysis pipeline
Questionnaire data
Statistics
Transcripts
Codebook
Segments
Thematic matrix
Triangulation
Report
Key findings
Finding 1
The basic feature concepts drive Control B and Verifiability
Associated features would be used by 5–6 of 6 participants
Means of 6.2–7.0 out of 7 for the measures associated with these user needs and features
Triangulation showed no contribution of the speculative features to these ratings
“We already have it all in there: feature X, feature Y and feature Z. I don’t need any more than that.”
P6
Post-questionnaire results
Measure
M
Mdn
Min–Max
Control A
Ease of Control A
5.50
5.0
5–7
Feature sufficiency for Control A
5.00
5.0
3–6
Control B
Ease of Control B (aspect 1)
7.00
7.0
7–7
Feature sufficiency for Control B (aspect 1)
7.00
7.0
7–7
Ease of Control B (aspect 2)
6.83
7.0
6–7
Feature sufficiency for Control B (aspect 2)
6.17
6.0
6–7
Verifiability
Ease of Verifiability (aspect 1)
7.00
7.0
7–7
Feature sufficiency for Verifiability (aspect 1)
6.50
7.0
5–7
Ease of Verifiability (aspect 2)
6.83
7.0
6–7
Feature sufficiency for Verifiability (aspect 2)
6.50
6.5
6–7
Stats of the post-questionnaire ratings on a 7-point scale, n = 6; Control A/B and aspect 1/2 are placeholders for confidential details
Finding 2
Control A is not sufficiently addressed – the user need covers more aspects than the design accounts for
Received clearly lower ratings, with means around 5.3
Participants missed features and capabilities needed to fully address their need for Control A
Finding 3
The speculative AI-native features are promising candidates for differentiating the product but need more exploration
Despite some resistance, they raised interest and would be used by 4–6 of 6 participants
Unprompted, two participants independently discovered an iterative check-and-improve workflow based on three features
Discovered check-and-improve workflow
Step 1
Check
Check AI output quality with [Feature X]
Step 2
Identify issues
If the result is weak, identify issues with [Feature Y]
Step 3
Improve
Improve output with [Feature Za] or [Feature Zb]
Step 4
Re-check
Check the improved output again with [Feature X]
Final step
Approve
Set status to approved once the output is good enough
Repeat until the result is good enough
Outputs and influence
Sharing findings and supporting adoption
I shared the results and recommendations in a report and a presentation to the designer, PM, and management. Once the production version was implemented, I was asked to give feedback on it.
Design and evaluation framework
The three identified core user needs proved to capture the relevant problem space well and can serve as design guidance and an evaluation framework for future iterations – or even related products.
The definition of Control A, where the design fell short, should be refined – I suggested how.
Impact and outcomes
The layout and one basic feature not yet in the company’s prototype, identified as a success driver, were adopted and implemented in the production version.
Further exploration and implementation of additional features are being considered for the product backlog and next phase of development.
Shipped features
Tested mockup in a state after finding-generation; confidential details blurred
“Shipped (minor)”: smaller additions not identified as success drivers
Next steps and recommendations
My recommendations, in order of priority:
Build the remaining basics.
Adapt the concept where it fell short on Control A.
Further explore the speculative features, as they could be impactful product differentiators, starting with the iterative check-and-improve workflow.
Reuse the three user needs – with a refined definition of Control A – to design and evaluate future iterations.
For follow-up exploration, I recommend building the speculative features into a fully functional prototype and testing them in real work scenarios.
Reflections
What went well: Framing the problem with the three dimensions gave the team a shared language and kept design, study, and analysis aligned.
What I’d change: I’d define Control A more broadly from the start.
I built some features only as placeholders, which weakened participants’ ability to evaluate them. Next time, I’d implement them more deeply so participants can react to real behavior instead of imagining it.
Limits: A sample of six participants means the results show tendencies, not proof. Still, for an early-stage study meant to shape the design rather than validate it, that’s the right level of evidence, and it gave the team a clear direction for the MVP.