Benchmark usability testing

Zapier

In the fast-paced world of SaaS, it’s easy to get caught up in feature-driven development. At Zapier, we discovered that while A/B testing individual features was valuable, it often led to a myopic view of our product. Teams became hyper-focused on optimizing local metrics, potentially at the expense of the overall user experience. This challenge was particularly acute when multiple teams were simultaneously implementing changes across the platform.

Recognizing the need for a holistic view of our product’s usability, I introduced and implemented benchmark usability testing. This approach helped create a comprehensive understanding of our user experience, even as our product rapidly evolved.

The power of benchmark testing

Benchmark usability testing monitors macro-level changes in your product’s user experience over time. Think of it as a recurring health check for your SaaS app. By conducting the same usability test script multiple times a year, you create a consistent baseline for comparison. The outcome is a scorecard and detailed report that allows you to track the evolution of your user experience. This data-driven approach enables you to:

  1. Identify trends in user behavior and satisfaction
  2. Measure the cumulative impact of multiple feature releases
  3. Detect unintended consequences of changes across different parts of your product
  4. Provide clear, actionable insights to guide product strategy

Implementing benchmark testing at scale

Benchmark testing requires repeatability to compare results from one period to the next and observe changes. This involved:

  1. Automation: We leveraged UserTesting.com to create fully unmoderated tests, eliminating facilitator bias and reducing the time investment required from our team.
  2. Standardized Protocol: We crafted a set of tasks that any new user could complete without special accounts or domain knowledge, ensuring consistency across test sessions.
  3. Objective Analysis: We developed a comprehensive rubric and data dictionary, allowing multiple researchers to analyze sessions using consistent criteria, thus minimizing interpretation bias.
  4. Efficient Reporting: We created tools to automatically generate benchmark report metrics, streamlining the process of creating and sharing insights across the organization.

Below are some artifacts from the process.

Task definitions
Task definitions

Analysis & tagging manual
Analysis & tagging manual

Collection and reporting tool
Collection and reporting tool

Outcome

Implementing benchmark usability testing provides several strategic advantages:

  1. Holistic View: It offers a bird’s-eye view of your product experience, complementing feature-specific metrics with a broader understanding of user satisfaction and efficiency.
  2. Early Warning System: Regular testing helps identify emerging issues before they become significant problems, allowing for proactive rather than reactive product management.
  3. Cross-Team Alignment: The shared benchmark data fosters collaboration between teams, encouraging a unified approach to improving the overall user experience.
  4. Data-Driven Decision Making: Concrete, longitudinal data on user experience trends empowers leadership to make informed decisions about resource allocation and product direction.
  5. Competitive Edge: By consistently refining your product based on comprehensive user feedback, you stay ahead of competitors who may be focused solely on feature parity.

In today’s competitive SaaS landscape, delivering an exceptional user experience is paramount. Benchmark usability testing provides the insights needed to continuously improve your product, align your teams, and ultimately drive user satisfaction and retention.