top of page
UI Design Elements

When Research Doesn't Scale: Rebuilding the Usability Process

NDA Notice
This case study has been anonymized to protect proprietary information. Company name, internal tools, and identifying details have been generalized. The work, the process, and outcomes accurately reflect my contributions and responsibilities. 

📌 TL;DR 

Role: UX Researcher (End-to-End Ownership)

Focus: Rebuilding and scaling voice-based usability testing

Impact: Reduced research cycle time from 6 to 2 weeks while preserving methodological integiry and enabling single-researcher execution

Blank Green Chalkboard

Rebuilding Research Infrastructure Under Constraint

The Problem

Usability testing existed - but it was fragile. 

​

Voice-based studies relied on a legacy Wizard-of-Oz (WoZ) tool inherited through acquisition. Workflows were fragmented across multiple platforms, required coordination between multiple people, and were increasingly difficult to scale as demand grew.

​​​

The risk wasn't just about inefficiency. 

It was inconsistency.

​

As usability testing became a part of the company's service offering, we needed a process that could scale without sacrificing research validity.

The Core Tradeoff
 

As AI tools matured and automation became tempting, the central question wasn't "How do we move faster?".

​

It was: 

"How do we increase velocity without compromising experimental control?"

​

This became the guiding principle behind every decision.

​​

Sketch On Notebook

The Process

Step 1: Diagnosing Structural Bottlenecks

I audited the existing workflow across recruitment, moderation, scripting, execution, and synthesis.

​

Key issues: 

  • Tool fragmentation created handoff delays

  • Script sprawl increased session variance

  • WoZ access constraints created dependency risks

  • AI automation lacked reliability safeguards

​

Rather than optimizing isolated tasks, I focused on the research systems as a whole.

​

Platform Evaluation

I evaluated five moderated research platforms against criteria grounded in research methodology:

  • Experimental control

  • Flexibility across study types

  • AI maturity (support vs replacement)

  • Scalability and cost

  • Insight quality

 

AI features were assessed based on whether they reduced cognitive load for the researcher without introducing validity risks.

​

Outcome: A moderated research platform was selected that consolidated recruitment, execution, and analysis - reducing fragmentation and improving repeatability.

Step 2: Reducing Variance in Moderation

​

The existing moderator script had evolved into a lengthy, redundant document. Sessions often spend ~10 minutes on setup before tasks began, increasing time, pressure, and inconsistency.

​

Drawing from principles of cognitive load and experimental control, I: 

  • Removed redundancy

  • Standardized task framing

  • Reorganized content into a single editable moderation view

  • Designed for real-time note-taking

​

Impact

  • Pre-task setup reduced to ~5 minutes

  • Tasks consistently completed within session time

  • Reduced session variance

  • Improved data comparability across participants

​

This wasn't about speed alone - it was about improving signal quality.

Step 3: A Deliberate Decision About AI

​

I explored automating the Wizard-of-Oz using an internal AI-powered audio tool. 

​

Internal testing suggested potential time savings.

​

However, live simulations revealed: 

  • Response variability

  • Script drift

  • Hallucinated outputs

​

These issues posed a direct threat to study validity.

​

I made a deliberate decision not to deploy AI in this layer of the researcher process.

​

Instead, I prioritized controlled execution over automation. 

​

This preserved experimental integrity - even at the cost of convenience.

Step 4: Rebuilding Capability Under Constraint

​

Three days before a scheduled usability study, access to the existing WoZ tool was unavailable.

 

Canceling would have delayed deliverables and undermined stakeholder trust.

​

Rather than reschedule, I rebuilt the capability.

​

Using HTML/CSS and iterative AI-assisted coding, I developed a lightweight WoZ soundboard tool that enabled:

  • Step-by-step controlled audio playback

  • Visual prompts to reduce moderator error

  • Time tracking per clip

  • Clear visual hierarchy for live execution

 

The tool was designed, built, tested, and deployed within 48 hours.

​

Post usability study, it was refined for accessibility and potential broader use.

Technical Constraint Resolution

​

The moderated testing platform did not support direct system audio sharing.

​

Initial workarounds degraded audio quality.

​

I diagnosed the technical limitation and implemented an audio-routing solution using OBS and VoiceMeeter, enabling direct desktop audio playback into live sessions.

​

Result: Improved professionalism and reduced technical risk during moderation.

Impact

​

This work transformed voice-based usability testing from a fragile, multi-person effect into a scalable, researcher-owned capability.

​

Results

  • Reduced study cycle time from 6 weeks to 2.

  • Enabled single-researcher execution of moderated voice studies.

  • Increased session consistency and task completion rates.

  • Preserved methodological integrity while selectively leveraging AI.

  • Established a foundation for scalable research infrastructure.

Judgement In Practice 

​​​​

This project required more than operational efficiency. It required discernment.

  • When to consolidate platforms

  • When to simplify workflows

  • When to reject automation

  • When to build instead of buy

​

Each decision balanced speed, scalability, and research validity.

What This Project Demonstrates

​​​

  • System-level research thinking
  • Application of experimental control principles
  • Cognitive load awareness in study design
  • Large-scale workflow analysis
  • Responsible AI integration
  • Constraint-driven problem solving
bottom of page