Sagar Patil

sauce labs

Designing a unified web and app testing UX.

challenge Design a scalable user experience and design language that integrates all Sauce Labs products - present and future - into a single testing platform.

role Product design consultant and UX framework architect.

Sauce Labs is a cloud-hosted, web and mobile application automated testing platform company based in San Francisco, allowing users to run tests in the cloud on more than 700 different browser platform, operating system and device combinations, providing a comprehensive test infrastructure for automated and manual testing of desktop and mobile applications using Selenium, Appium and JavaScript unit testing frameworks.

In April 2019, Terri Avnaim (CMO), Charles Ramsey (former CEO) and Brad Edelberg (former CTO) contacted me to help audit and redesign the existing Sauce Labs UX, and to create a unified experience across all products, some of which resided on other domains and URLs. During this process, I worked with a talented team of four UX designers, two UI/IxD designers and two UX researchers, under the leadership of Michael Wiekrykas, the Director of Design. I was the Principal UX Designer for designing the product unification experience.

Identifying Opportunities

Between June and August 2019, I met with different teams - Design, Engineering, Product, Sales and Support - located across San Francisco, Berlin and Warsaw, to get their thoughts and insights about the product, process and user experience. The interviews also gave me a chance to ask questions around data and analytics, company health, team structure, and collaboration dynamics within the company.

After meeting the teams, I compiled the feedback and presented my findings to the management team. Below is some of the feedback and primary concerns that came up during a few of the interviews.

Sauce Labs - Screen-by-screen audit and interview feedback during my audit of the product A screen-by-screen audit along with interviews with different teams across the SF and Berlin offices helped uncover many of the UX and UI issues along with identifying opportunities for improvement and growth.
# Product/Feature Issues Desired outcome
1 Live Testing UX
  • 100% of first time users run a Live test. If they have a bad experience, they do not move forward.
  • Users have to go to another site (TestObject) in order to test on real devices leading to a fragmented user experience.
  • Real device selection UI is confusing.
  • Live Testing experience is often compared to BrowserStack.
  • Quick and easy ways to get started with Live Testing.
  • Design a unified experience for users to run live tests on browsers and apps in a single UI.
  • Simplify the device selection UX.
2 Product navigation
  • Does not accurately represent product heirarchy.
  • Does not link to acquired products.
  • Does not follow a standard naming convention for products.
  • Non-responsive on smaller screens.
  • Make it easy for users to find the products they are looking for.
  • Build an accurate product hierarchy model that matches the user's expectations.
  • Should be responsive and usable at all screen sizes.
3 Design process
  • No standardized UI or component library.
  • Lack of design guidelines and design delivery processes.
  • Lack of ownership.
  • Teams unaware of what design is doing (no transparency).
  • Create a unified design langauge and guidelines to determine who and when the library gets updated.
  • Develop a design and asset delivery protocol with other teams. Become part of the scrum process.
4 Documentation
  • Deeply nested links are hard to find.
  • Search functionality does not work accurately.
  • Documentation is overly technical and verbose.
  • Simplified documentation with short format content like explainers, videos and PDFs.
  • Accurate and smarter search fucntionality.
  • At most, three levels of nesting.
  • Shareable segments that can be incorporated into the product directly.
  • Remove comments feature.
5 Sauce Connect / Tunnels
  • Does not educate users about the value of tunnels.
  • Difficult to understand and use, even for technical users.
  • Only 10% of users get a 100% score for completely enabling tunnels without any help.
  • Can't start a tunnel directly from the UI.
  • Simplify process of getting started.
  • Simplify documentation around advanced tunnel concepts.
  • Allow creation of tunnels from the UI.
6 Analytics implementation and Data Tracking
  • Very little data currently available about user journeys or metrics, in order to make accurate product decisions.
  • Lack of data makes it hard to determine health of key success metrics.
  • Makes it hard to truly understand user behavior.
  • Many disparate data sources exist without documentation about what is being tracked.
  • Implement a system of analytics to start tracking and understanding user behvaior.
  • Teams implement analytics that are relevant to their success metrics.
  • Refine persona profiles and identify common user patterns.

Redesigning the UX of the product would require major changes to both, the way users navigate the product and the individual product experiences themselves. We broke down the redesign into two parts that could be worked on incrementally and independently - Navigation and the Product Experience.

Sauce Labs - Project phases
Phase 1: Navgation; Phase 2: Product experience

Redesigning the Navigation

The navigation experience was split between the header and a sidebar. The experience was broken and disjointed and did not accurately communicate what the website did. The information architecture did not make sense.

Sauce Labs - Information Architecture (IA) for the navigation
Problems and opportunities for improving the navigation

We first refined the navigation and broke it down into individual components to reflect the mental model users had when using the navigation. UX researchers Joan and Steve, used card sorts and wireframes to determine how users would use the UI. This included features and tools that users desired, or were generally missing or inaccessible from the UI.

Sauce Labs - Competititive and comparitive navigation teardowns
Sauce Labs - Berlin Product Design workshop - Restructure the navigation Sauce Labs - Berlin Product Design workshop - Because it's not all work Product Design Organization workshop, Berlin, October 2019

Competitive and comparative analysis

We reviewed a number of competitors to understand what features they offered, how they promoted these features, how the application UI was laid out and the different ways in which users could use the app. We also did teardowns for common productivity apps that users used regularly.

Sauce Labs - Competititive and comparitive navigation teardowns
Navigation teardowns helped us get a quick, block-level view of how UI elements are grouped toegther, based on their importance in the user flow.

Using the teardowns, we reviewed and learned about how the navigation could be better organized, communicate product hierarchy accurately, and provide convinience functions that allow users to achieve their tasks quicker, without having to browse through the product first.

Sauce Labs - Information Architecture (IA) for the navigation
Information Architecture (IA) and information heirarchy model for the navigation
Sauce Labs - Information Architecture (IA) for the navigation

Wireframes and ideas

Sauce Labs - Horizontal header navigation concept
We tried design explorations with a platform first approach, where users would first choose if they wanted to perform mobile or web testing. Based on their selection, a subnavigation would be presnted with the different testing tools displayed. Ultimately, we scrapped the idea as users found it cubmersome and confusing.
Sauce Labs - Wireframes for different states of the product navigation
Sidebar navigation design explorations
Sauce Labs - Wireframes for different states of the header navigation
Header navigation design explorations

Redesigning the Product Experience

Only 48% of users regularly visited the Sauce Labs UI, primarily to view the results of their tests and particularly if their tests had failed or had an error. This told us, that a majority of users were running automated tests from their IDE or CI/CD. However, based on my initial interviews with the Sales and CSM teams, we also knew that a 100% of first time users ran a manual test.

As part of the new product UX, we wanted to create a common design pattern/template that could be used across all product UIs, while still giving individual teams to customize their products based on user feedback and data. We decided to adopt an architecture that was at most 3 levels deep, starting with a customizable dashboard and drilling down into a test details page.

Sauce Labs - Berlin Product Design workshop - Paper wireframes that the team mocked up rapidly Paper wireframes that the team mocked up in 30 second sketch exercises to determine how diferent designers prioritized information and user actions.

Dashboard

We wanted users to have a customizable home page that would provide a summary of, and insights into the data that a user would be interested in. Since the importance of data varies from user to user and based on role, we proposed augmenting the existing card interface with usage rules and guidelines.

Sauce Labs - Anatomy of a dashboard card Anatomy of a card
Sauce Labs - Dashboard cards can be organized in different layouts in order to create a customized dashboard
Cards and rows can be of three sizes - single column, double column or full row width. These cards can then be organized in different layouts that allows users to customize their dashboard to organize data in a manner easy for them to consume.

Use case:
An engineering manager or a senior software developer is interested in purchasing more devices for their private device cloud but does not have the budget to buy additional devices and can only replace existing devices.

The user would be interested in learning more about the cost benefits and coverage of their existing test setup and how it can be improved. By providing concise information about their usage and testing costs, users build trust and eventually derive more value from the platform.

Sauce Labs - Custom dashboard for an Engineering Manager
Custom dashboard for a Senior SDET or Engineering Manager, displaying performance and test data that can help identify problematic tests and devices, concurrency requriements, etc.
Sauce Labs - Custom dashboard for an SDET
Custom dashboard for an SDET, showing all jobs run by the team
Sauce Labs - Conceptual design for displaying the job details on the dashboard, within a expandable and collapseable window
Conceptual design for displaying the job details on the dashboard, within a expandable and collapseable window.

Quick start

Since most users usually test on a handful of browsers and devices, we wanted to provide them with the ability to quickly test recently tested websites and apps without having to go through the initial selection process each time.

After speaking to customers and reviewieng support tickets, we wrote down a few user stories to isolate how users construct their sentences when communicating their testing requirements. On analysing these statements, we were able to identify a pattern and simplify the query into a template.

Sauce Labs - Designing the user flow when choosing to test
Sauce Labs - Anatomy of the Quick Start toolbar Anatomy of the Quick Start toolbar
Sauce Labs - Wireframe options for the Quick Start toolbar

After speaking to customers and reviewieng support tickets, we wrote down a few user stories to isolate how users construct their sentences when communicating testing requirements.

Sauce Labs - Wireframes for the browser and device selection menus when the Quick Start menu is open and closed
Sauce Labs - Wireframes for the browser and device selection menus when the Quick Start menu is open and closed
Refined UI for the browser and device selection menus showing the evolution of the browser selection menu
Sauce Labs - Design variations of the Quick Start bar for different products
Final design concepts for the Quick Start toolbar for the different product types - Live, Automated and Visual testing

Quick Start Browser/Device selection menu

Designs for the browser and device selection menus across Live, Automated and Visual testing. Automated and Visual testing menus allow users to save custom groups of devices and test multiple builds across these sets.

Sauce Labs - Design variations of the device and browser listing menus for the Quick Start component

Job/Test Details

The most visited page of the product, we wanted to update the layout of the page, without changing too much of the structure that users were used to. In particular, we wanted to de-emphasize the video, and create a standardized navigation format that could be used as a template to display all types of test results - Selenium, Appium, CYpress, etc.

Sauce Labs - Conceptual designs for the test results page
Conceptual designs for the test results page. This template would be used across all products to display test results.
Sauce Labs - Users can view the video for a test diectly from the results page, without having to go deeper into the details page
Users can view the video for a test diectly from the results page, without having to go deeper into the details page.
Sauce Labs - Conceptual design option for displaying screenshots in the test details
Conceptual design option for displaying screenshots in the test details. In this option, the navigation is laid out horizontally above the main content area. The third column acts as a tertiary content area to display links, promotions or other relevant, non-critical information.
Sauce Labs - Conceptual design option for displaying test logs in the test details
Conceptual designs for displaying logs for the selected test. In this design, the navigation is placed on the left of the main content area, representing the flow of information.

Cross browser testing

The manual, cross browser testing experience was updated to include the new product UI template and provide a simpler user experience for all users.

Sauce Labs - Live Testing - Cross browser testing default UI
Live Testing - Cross browser testing default UI
Sauce Labs - Live Testing - Cross browser testing UI with browser/device selection menu
Live Testing - Cross browser testing UI with browser/device selection menu visible when Internet Explorer 11 is selected

App testing

A large part of the unification project was to complete the integration of TestObject into the Sauce Labs family, and allow users to test their apps directly on Sauce Labs, rather than having to go to the TestObject UI.

Sauce Labs - Live Testing - Cross browser testing default UI
Live Testing - App testing default UI
Sauce Labs - Live Testing - App testing UI with app versions being displaeyd
Live Testing - App testing UI with different versions and builds of the app being displayed
Sauce Labs - Live Testing - App testing UI with app versions being displayed
Live Testing - Choose a device to test the app on
Sauce Labs - Live Testing - App settings UI
Live Testing - App settings UI

Sauce Connect

Sauce Connect allows users to test their local, development or staging servers on Sauce's infrasturtcure directly. 50% of all Free Trial users try to connect their local setup using Sauce Connect. However, the process of getting started for both, self-serve and CSM-assisted users, was technical and difficult. Moreover, tbe UI provided very little information for users to understand the features, benefits and implementation of Sauce Connect.

We suggested moving the tunnels UI into the header, from where it can be viewed across all product UIs.

Sauce Labs - Before and After comparison of the Tunnels UI
TOP: Tunnels are part of the product navigation in the current UI; BOTTOM: Tunnels are part of the header and accessible across all products at any point in the proposed redesign.

Global file uploads

Individual product teams were given the freedom and encouraged to use the new design library components to build out the pages they required, based on usage and customer data. As an example, we proposed a new global, floating file upoad component, that would allow users to track their uploads across all products without blocking the UI.

Sauce Labs - Designs for a new file upload component that would make the product feel more like an app, and less like a website
The new file upload component would allow users to track current and past uploads in a single location. The floating UI also makes the fiile upload component feel more like an app

We wanted to ensure that users could continue using the UI and perform other testing tasks while not being blocked by large uploads in the queue.

Sauce Labs - Designs for a new file upload component that would make the product feel more like an app, and less like a website
File uploads in progress on the Live testing screen for testing apps.

Learnings

Know your success metrics.

Success metrics help the team determine if the product and it's features have been successful in achieving the teams' goals. Always have a way to track key success metrics over short, medium and long term periods.

Do a few things but do them well.

Nothing will demotivate a team more than constantly changing product objectives that result in incomplete or half-baked features. Focus on what you do well, understand why users prefer you over the competition and build a plan to deliver value that YOUR users want.

Deliver value, not features.

Having a large feature set excites users initially, but less than 10% will use all the features. It's also possible that users would be overwhelmed users and make the product seem bulky Supporting a large feature set properly puts a strain on the entire team and takes resources away from more critial user flows and bugs that need attention. Always determine user interest and value in a feature before building it out. For example, how many users have shown interest in paying for the feature and shown urgency in requesting the feature.

Know your product thoroughly.

Understand all aspects of your product, not just the vertical you are responsible for. Invest in data to research and learn which user flows are successful and which need more work. Identify cross-product growth and revenue oportunties within the product. By learning how users use the product in it's entirety, where they come from and why they are using your product, you will be better informed when making meaningful changes to the product.

2020.

Be greatful, gracious and thankful for the things we take for granted.

Acknowledgements

I am grateful to Charles Ramsey, Terri Avnaim, and Brad Edelberg for giving me the opportunity to work with the Sauce Labs team. I enjoyed working with, and learning from a very talented team of designers - Michael Wiekrykas, Seth Kornfeld, Joan Jasak, Josh King, Steve Leslie, Wanda Seto, Anna Nawrocka and Marek Pohl - and look forward to having the chance to work with them again.

I would also like to thank Marcia Foley for helping me get around and get familiar with who's who, while staying laser focussed on the job at hand.