Sagar Patil

Browserstack

Designing a responsive testing tool.

challenge Build a responsive testing tool using screenshots of websites taken from actual mobile and desktop devices.

role Product and user research, wireframing, UX/UI and visual design, coding the front end.

In 2013, the responsive testing landscape was underdeveloped, with most freely available responsive testing tools only offering testing of layouts based on viewport. Screen features such as aspect ratio and pixel density were often not considered and hence results were partially accurate. BrowserStack wanted to build a free responsive tool that allowed users to get screenshots of websites from actual mobile and desktop browsers.

Opportunities

Existing responsive testing tools simply resized a window to trigger layout changes. These results however, were sometimes inaccurate because:

  • No testing service provided results from real devices or official emulators.
  • People were unaware of which devices to test on.
  • Users could not test for specific features such as aspect ratio, pixel density, content above the fold, and more.
  • For companies maintaining device labs for testing, costs could quickly escalate due to procurement, transportation, warehousing and maintenance.

Goals & Research

We wanted to build a responsive testing tool that provided results from real desktop and mobile devices. We also wanted to:

  • Sell more screenshots plans
  • Guide users to test on the right set of devices
  • Allow users to test devices at full scale

Numerous, free testing tools already existed, though most only provided viewport testing, without accounting for differences in OSes, browsers, screen sizes, device sizes, aspect ratio and more. While these responsive tools worked for quick and dirty tests, testing on real devices or emulators was essential for production quality results.

Studying usage data of Screenshots over 12 weeks, we observed that users tested iOS and Samsung devices more than any desktop browsers. This helped us understand how users select devices to test on, the duration of a test session and the frequency of tests run. Based on this, we were able to determine an approximate list of devices that users tend to use the most.

By limiting the devices to a curated set, we defined the relevant devices to test on. Higher usage of iOS devices meant that we needed different iOS versions in the mix. We added Samsung and Amazon devices due to their popularity. For the sake of completeness, we also included large desktop resolutions.

Solutions

test on the right browsers

Curate a set of latest devices

Users were unaware of which devices to test on. Data from Live testing sessions showed us that iOS devices were tested for at least 2 generations. More Android devices were tested than iOS due to the large number of Android device manufacturers. Samsung, HTC and Google devices were tested the most and each user session on mobile tested at least 3 Android devices. We built a rule set, that could automatically update the device list each time new OSes or browsers were added to the BrowserStack family.

  • popularity
    2 Phones
    2 Tablets
  • variety
    4 Phones
    4 Tablets
  • completeness
    2 Desktop
    Resolutions
natural testing experience

Show results at the same size as the actual device

We wanted the experience of viewing the results to be as natural as viewing the website on their phone in their hands. We designed skins for all devices and created an algorithm to determine how to scale the results based on the user's monitor size. This helped us determine how to position the screenshot within the device viewport.

We ported the Wikipedia display size matrix into a JSON object that allowed us to look up and convert device widths and heights from pixels to millimeters, grouped by aspect ratio. For example, for an aspect ratio of 5:4, a sample entry in the hash map for a 3.5" screen would be,

{
"aspectRatio": "5:4",
"displays": [{
"diag-in": 3.5,
"diag-cm": 9,
"width-cm": 7,
"height-cm": 6,
"area-cm": 39
}],
"aspectRatioValue": 1.25
}

Using the millimeter values above, we calculated pixel values for each device based on their aspect ratio. The aspect ratio was required in order to finally determine the correct screen diagonal size and position of the screenshot.

function convertMMtoPixels (size_in_mm, offset) {

var wd_in = size_in_mm / (2.54*10)
wd_screen_px = wd_in * screenPPI;
return wd_screen_px;

var resolutionX = screen.width;
screenDiagnol_calc_inches = screenDiagnol_calc / screenPPI;
var aspectRatio = screenAspectRatio;
if (aspectRatio == 1.33) {
var screenDiagonal = screenDiagnol_calc_inches * 25.4;
var angle = 36.86989764584402;
} else if (aspectRatio == 1.6) {
var screenDiagonal = screenDiagnol_calc_inches * 25.4;
var angle = 32.0053832080835;
} else if (aspectRatio == "ipad") {
var screenDiagonal = 246.38;
var angle = 53.8;
} else {
var screenDiagonal = screenDiagnol_calc_inches * 25.4;
var angle = 29.357753542791272;
}
return ((1 + (offset / 100)) * (Math.round(width * (resolutionX / ((Math.cos((angle / 180) * Math.PI)) * screenDiagonal)) * 10) / 10));
};

The output would be a JSON file with the final pixel values used for the mapping. In case the scaling was incorrect, we allowed users to set their monitor size based on a pre-populated list, or by manually entering the size of their screen. The scaling algorithm was then adjusted by this amount.

"iphone6" : {
device:"iphone 6",
port:67.1, land:138.1,
leftOffset:4.544483333333333, topOffset:19.90725,
width:57.912, height:100.34058333333333,
land_left:16.891, land_bottom:4.4238333333333335,
land_width:104.16116666666666, land_height:58.71633333333333,
urlcss: {
  top: 0, left: 3.4572222222222226, width: 50,
  height: 4.494388888888889, font: 2.592916666666667, textalign: 'center'
},
urlcss_land: {
  top: 0.60325, left: 23, width: 58
}
}
Changing the monitor size changes the size of the device. Larger screen sizes make devices smaller and vice versa.
Comparing actual devices to the results generated
Comparing actual devices with the results generated from Responsive
sell screenshots plans

Generate screenshots using the Screenshots API

The Screenshots API allowed users to request screenshots from BrowserStack's remote devices and integrate the results into their projects. The Responsive tool was the best way to exemplify how the Screenshots API could be used if users bought a Screenshots plan with API access.

BrowserStack Screenshots uses the same Screenshots API for Responsive

Results

The tool generated screenshots from real devices and emulators, allowing users to test all screen features accurately. Responsive was well received by users, helping BrowserStack see a 50% increase in registrations over 2 weeks. However, most users ran a few tests and eventually dropped off. Majority of usage came from API users that were testing on the responsive set of devices. Adoption plateaued and Responsive gave way to the new Live product which offered interactive responsive testing by default.

Final result in a real testing session

Customer reactions

Learnings

Understand the market requirements before building a product.

Responsive was built to complement Screenshots. While Free Tier usage was high, the increase in subscriptions was marginal and in the end, we hadn’t really addressed any crucial market requirements. We made responsive testing more accurate and simpler, but did not improve the testing process enough to warrant a paid subscription. Early validation would have helped us understand the use cases and if Responsive even needed to be built.

While I was happy with the early responses from customers, since the goal of increasing Screenshots subscriptions was not met, I don’t consider the project to be a success.

Acknowledgements

I worked with a small but effective team, that was able to design, build and deploy iteratively. Many thanks to Ahmed Abbas for all the JS work and helping me build the scaling algorithm. I'd also like to thank the Rails team - Rahul Nawani and Utsav Kesharwani - for building a great API that helped us reuse much of the existing code from Screenshots.

Case studies