Designing a responsive testing tool.
In 2013, the responsive testing landscape was underdeveloped, with most freely available responsive testing tools only offering testing of layouts based on viewport. Screen features such as aspect ratio and pixel density were often not considered and hence results were partially accurate. BrowserStack wanted to build a free responsive tool that allowed users to get screenshots of websites from actual mobile and desktop browsers.
Opportunities
Existing responsive testing tools simply resized a window to trigger layout changes. These results however, were sometimes inaccurate because:
- No testing service provided results from real devices or official emulators.
- People were unaware of which devices to test on.
- Users could not test for specific features such as aspect ratio, pixel density, content above the fold, and more.
- For companies maintaining device labs for testing, costs could quickly escalate due to procurement, transportation, warehousing and maintenance.
Goals & Research
We wanted to build a responsive testing tool that provided results from real desktop and mobile devices. We also wanted to:
- Sell more screenshots plans
- Guide users to test on the right set of devices
- Allow users to test devices at full scale
Numerous, free testing tools already existed, though most only provided viewport testing, without accounting for differences in OSes, browsers, screen sizes, device sizes, aspect ratio and more. While these responsive tools worked for quick and dirty tests, testing on real devices or emulators was essential for production quality results.
Studying usage data of Screenshots over 12 weeks, we observed that users tested iOS and Samsung devices more than any desktop browsers. This helped us understand how users select devices to test on, the duration of a test session and the frequency of tests run. Based on this, we were able to determine an approximate list of devices that users tend to use the most.
By limiting the devices to a curated set, we defined the relevant devices to test on. Higher usage of iOS devices meant that we needed different iOS versions in the mix. We added Samsung and Amazon devices due to their popularity. For the sake of completeness, we also included large desktop resolutions.
Solutions
test on the right browsersCurate a set of latest devices
Users were unaware of which devices to test on. Data from Live testing sessions showed us that iOS devices were tested for at least 2 generations. More Android devices were tested than iOS due to the large number of Android device manufacturers. Samsung, HTC and Google devices were tested the most and each user session on mobile tested at least 3 Android devices. We built a rule set, that could automatically update the device list each time new OSes or browsers were added to the BrowserStack family.
-
popularity
2 Phones
2 Tablets -
variety
4 Phones
4 Tablets -
completeness
2 Desktop
Resolutions
Show results at the same size as the actual device
We wanted the experience of viewing the results to be as natural as viewing the website on their phone in their hands. We designed skins for all devices and created an algorithm to determine how to scale the results based on the user's monitor size. This helped us determine how to position the screenshot within the device viewport.
We ported the Wikipedia display size matrix into a JSON object that allowed us to look up and convert device widths and heights from pixels to millimeters, grouped by aspect ratio. For example, for an aspect ratio of 5:4, a sample entry in the hash map for a 3.5" screen would be,
{ "aspectRatio": "5:4", "displays": [{ "diag-in": 3.5, "diag-cm": 9, "width-cm": 7, "height-cm": 6, "area-cm": 39 }], "aspectRatioValue": 1.25 }
Using the millimeter values above, we calculated pixel values for each device based on their aspect ratio. The aspect ratio was required in order to finally determine the correct screen diagonal size and position of the screenshot.
function convertMMtoPixels (size_in_mm, offset) { var wd_in = size_in_mm / (2.54*10) wd_screen_px = wd_in * screenPPI; return wd_screen_px; var resolutionX = screen.width; screenDiagnol_calc_inches = screenDiagnol_calc / screenPPI; var aspectRatio = screenAspectRatio; if (aspectRatio == 1.33) { var screenDiagonal = screenDiagnol_calc_inches * 25.4; var angle = 36.86989764584402; } else if (aspectRatio == 1.6) { var screenDiagonal = screenDiagnol_calc_inches * 25.4; var angle = 32.0053832080835; } else if (aspectRatio == "ipad") { var screenDiagonal = 246.38; var angle = 53.8; } else { var screenDiagonal = screenDiagnol_calc_inches * 25.4; var angle = 29.357753542791272; } return ((1 + (offset / 100)) * (Math.round(width * (resolutionX / ((Math.cos((angle / 180) * Math.PI)) * screenDiagonal)) * 10) / 10)); };
The output would be a JSON file with the final pixel values used for the mapping. In case the scaling was incorrect, we allowed users to set their monitor size based on a pre-populated list, or by manually entering the size of their screen. The scaling algorithm was then adjusted by this amount.
"iphone6" : { device:"iphone 6", port:67.1, land:138.1, leftOffset:4.544483333333333, topOffset:19.90725, width:57.912, height:100.34058333333333, land_left:16.891, land_bottom:4.4238333333333335, land_width:104.16116666666666, land_height:58.71633333333333, urlcss: { top: 0, left: 3.4572222222222226, width: 50, height: 4.494388888888889, font: 2.592916666666667, textalign: 'center' }, urlcss_land: { top: 0.60325, left: 23, width: 58 } }
Generate screenshots using the Screenshots API
The Screenshots API allowed users to request screenshots from BrowserStack's remote devices and integrate the results into their projects. The Responsive tool was the best way to exemplify how the Screenshots API could be used if users bought a Screenshots plan with API access.
Results
The tool generated screenshots from real devices and emulators, allowing users to test all screen features accurately. Responsive was well received by users, helping BrowserStack see a 50% increase in registrations over 2 weeks. However, most users ran a few tests and eventually dropped off. Majority of usage came from API users that were testing on the responsive set of devices. Adoption plateaued and Responsive gave way to the new Live product which offered interactive responsive testing by default.
Customer reactions
@browserstack the first #ResponsiveDesign tester I found that shows what a website truly looks like on all devices - http://t.co/mKpcCGu4Dk
— Michael Hanton (@mickojh) May 12, 2014
@raamjoshi Another really great browser / responsive test tool we tend to use is @browserstack -- check it out!
— David Smith (@MResponsively) February 3, 2014
@browserstack drives value to us & our Fortune 500 clients by increasing precision of dozens of responsive web launches a month.
— Centric Digital (@CentricDigital) October 22, 2014
@browserstack Responsive testing across devices was a great addition to your tools. Bravo.
— Roni Lagin (@RoniLagin) June 26, 2013
@browserstack is awesome ! Making responsive web development and testing so much easier !
— Cory Rylan (@SplinterCode) May 3, 2013
Wow, really impressed by @browserstack responsive tests. http://t.co/bDS6jBvOsq
— Erick Galassi (@etimesg) March 27, 2014
Responsive web design testing. Really helpful. http://t.co/rqovoSvy5U#UX @browserstack
— Sudipta Basumallik (@sunshine_n_me) February 26, 2014
The @BrowserStack innovate again, by creating a tool to generate screenshots to across a full stack of mobile devices http://t.co/yAFlfTjhlu
— Thomas Zilliox (@iamtzi) June 17, 2013
Just tried @browserstack's Responsive Testing tools. To say I am impressed would be an understatement. Very, VERY useful. #awesome
— Martyn Morrish (@martynmorrish) June 19, 2013
Learnings
Understand the market requirements before building a product.
Responsive was built to complement Screenshots. While Free Tier usage was high, the increase in subscriptions was marginal and in the end, we hadn’t really addressed any crucial market requirements. We made responsive testing more accurate and simpler, but did not improve the testing process enough to warrant a paid subscription. Early validation would have helped us understand the use cases and if Responsive even needed to be built.
While I was happy with the early responses from customers, since the goal of increasing Screenshots subscriptions was not met, I don’t consider the project to be a success.
Acknowledgements
I worked with a small but effective team, that was able to design, build and deploy iteratively. Many thanks to Ahmed Abbas for all the JS work and helping me build the scaling algorithm. I'd also like to thank the Rails team - Rahul Nawani and Utsav Kesharwani - for building a great API that helped us reuse much of the existing code from Screenshots.
Case studies
-
zealth ai
UX Design for reporting and managing cancer symptoms.
-
hdfc bank
Redesigning HDFC Bank’s consumer banking app.
-
sauce labs
Designing a unified, web and app testing UX.
-
browserstack
Redesigning the Live user experience.
-
browserstack
Rebranding and building a design system.
-
jawe
A website builder for everybody.