New York Life Case Study
— Starting, not quite from from scratch, I gradually put in place a set of tools that enabled a rich view of the customer’s behavior and thought processes. I say not quite because we did have analytics from SiteCatalyst so we could see, at least one dimensionally, where the customer might be struggling.Collectively, these tools and services provide, I submit, a robust solution in the form of a UX testing framework.
What’s in your toolkit?
Over the course of my career, and doubtless like may of you, I have spent countless hours watching from behind a mirror with a small group of design strategists, clients, and researchers — as a moderator conducted roundtable discussions, focus groups, and one-on-one user testing sessions. They were always incredibly valuable, and the collaboration behind the glass provided a genuine synergy that always improved our understanding of the client’s needs, frustrations, delights and feeling of accomplishments.
But I have, on occasion, had difficulty getting this to happen as often as I felt was necessary. Budget constraints, agency SOW timelines and allocations, and launch date commitments were often obstacles. I wanted something in place to allow us to run very quick tests at any point in the design cycle.
Online User Testing
Usertesting.com, and UserZoom
I put both of these online video capturing user testing platforms through their paces for 1 year each. Performing usability tests on existing content and Sketch/inVision prototypes, benchmarking studies to capture qualitative reactions to existing sites before launching the next site, and comparing our experiences to the competition. We performed both moderated and unmoderated testing. The Enterprise versions allow the ability to recruit your own pool or use theirs, as well as a pool of research hours where their experts can advise an assist, or set-up and analyze the tests. It is very easy to set up screener questions to recruit test subjects matching your segment criteria, set up individual tasks, and than have them talk out loud while performing the tasks you assign them. When they are done with one task, they click to the next task until complete, and then fill out an exit questionnaire. The SUPR-Q survey was used to provide a consistent quality score.
UserZoom seemed to have better capabilities reporting quantitive results if you wanted to use a large enough sample size, and their research team was very helpful in teaching us how to process spreadsheet downloads into pivot tables for reporting purposes.
The video output was a set of videos flagged by task, that you could add comments to at any point, and create clips and combine them into highlight reel.
Amazingly fast results. Like within hours. Generally good quality, serious test-takers. Did I mention fast? The speed is great for early tests where you just need validation for a concept, or a quick prototype.
Watching the hours of video running these tests will generate is an effort that needs to be understood, should be done as a team. And while their is the ability to create individual clips and combine them into highlight reels, that has the capability of being a cheat, in my view, either in favor of your own bias or easily perceived that way by someone with the opposite bias.
Adobe Target, like Optimizely and any other a/b testing tools demonstrate the winner in a head to head test by tracking confidence and significance
Session Recording Services
Session recording is a must have, in my view. You are capturing the real behaviors of real site visitors, hundreds of thousands of them, and seeing them interact with your content and forms in a way that your analytics just can’t capture. Did they skim past that long copy that would have inspired them to convert? Get the same form validation error multiple times? Abandon the sign up process the moment you told them the password rules after you made them enter their new password twice? (no, not us, but you know who you are). The session data can be aggregated into heapmaps, as below, or prioritized individually by conversion, specific funnel, abandonment, or struggle.
We evaluated several vendors that offered this service: SessionCam, Clicktale, Decibel Insights, and Content Square. The PoC we ran with SessionCam was an eye opener, but I’m still eager to try the others, as well.
We began a/b testing with Adobe Target more than a year before we adopted a broad suite of Adobe Marketing Cloud/Experience Cloud products, adding Adobe Experience Manager, Adobe Analytics, Adobe Audience Manager to the mix.
This opened up some interesting possibilities, and we began designing a/b tests that targeted specific audience segments revealed by 1st and 3rd party data as well as our analytics
Targeted A/B testing AT + AA + AAM
I moved New York Life’s support for WCAG standards from unevaluated, to automatically scanned to WCAG level A (by the creative agency that designed the new site), to manually validated by a professional Accessibility Auditing firm. And I’m glad I did. Even starting down this path has exposed me to some of the most patient people on earth.
Whenever possible, we looked for reliable industry research to help predict the potential advantages or risks of our design decisions.
Is the hamburger menu enough? Or should you try to keep some nav exposed on mobile? Nielsen Norman published a study last year about hamburger vs combo menus. Hamburger Menus and Hidden Navigation Hurt UX Metrics, based upon which we can predict that the version on the right, below, could significantly outperform the existing version (+29% in the NN/g study). Plus, it even supports the brand message better. A win all around.
Jakub Linowski has a great set of A/B tests to study. GoodUI Evidence. We used this resource to guide decisions on what to test next, adding a layer of external support for our predictions on what would have the greatest impact.