Charles Schwab: Research program management

 

problem

As lead researcher for two product families, I needed to increase cadence of testing to get our products up to speed. To accelerate the process, we established a working relationship with AnswerLab, an independent research agency.

goal

Our primary goal was to combine in-house research with the efforts of AnswerLab in order to produce research insights that would help us to improve our product offerings more quickly. I managed our relationship with AnswerLab and provided generative and iterative in-house research support. AnswerLab provided recruiting, research plan, moderation, and reporting services.

 

phase 1: establishing testing cadence

 

  • I connected with project managers to propose a structure and schedule for testing.

    • Day 1: Usability testing

    • Day 2: Reiteration of prototype

    • Day 3: Follow-up usability testing to gauge efficacy of changes

  • Each product team prioritized projects according to urgency.

  • Design gauged how often they could realistically provide prototypes.

  • Research and design management discussed realistic testing frequency and possible issues related to rapid pace.

  • I proposed a flexible testing schedule with AnswerLab so that we could pivot if cadence did not match production.

  • Our solution was to aim for 3 weeks of usability testing every month. Each week consisted of testing on Tuesday and Thursday with Wednesdays free for reiteration.

 

phase 2: establishing round timelines

 

  • Each round of testing would last approximately 6 weeks and include 4 session days of testing.

  • AnswerLab created research timelines for a 6-month period.

  • I coordinated with product and management to determine which timelines would work for our product families and UX team.

 

phase 3: populating the schedule

 

  • I worked with the product teams to determine order of projects for fielding.

  • Time slots were populated in Confluence.

 

phase 4: scheduling meetings according to the timeline

 

  • Each project began with a kickoff meeting. Walkthroughs, recruiting, and readout meetings were to be scheduled as needed.

  • I sent AnswerLab a list of key attendees per project. AnswerLab created the calendar invites and WebEx links for each meeting or research session.

  • As scheduling needs shifted, we shifted meeting and fielding times.

 

phase 6: preparation for testing

 

  • Kick-off: I evaluated each product as the in-house researcher and conducted generative research as needed. We held workshops, journey mapping exercises, and meetings as needed leading up to our kick-off with AnswerLab. In preparation for kick-off, I used the personas I created to define recruiting criteria. Stakeholder interviews and group meetings allowed me to create a full project brief and preliminary task flow, which I shared with AnswerLab.

  • Recruiting: AnswerLab either recruited directly or used clients from our database list pull to create participant lists. Participants were recruited according to location if we were doing on-site testing and from all over the country if we wanted AnswerLab to test remotely in their own lab.

  • Preliminary Walkthrough: During the preliminary walkthrough, the designer shares the prototype with AnswerLab. I coordinated with AnswerLab to define task flows and to strategize theoretical scenarios for testing.

  • Final Walkthrough: Prior to testing, we reworked prototypes and refined task flows to match research goals. Final prototypes were shared with Answerlab as Invision links.

  • Usability Testing Day 1: AnswerLab conducted usability testing and our team observed via WebEx. I transcribed 2-3 pages of notes and gathered quotes from participants

  • Testing Day 2: I worked with our internal designers, content strategist, and product teams to improve elements that did not test successfully on the first day

  • Testing Day 2: AnswerLab conducted usability testing with improved prototypes.

 

phase 7: summary reports

 

  • Ten Key Takeaways: AnswerLab produced a list of key takeaways immediately after fielding.

  • Final Report: AnswerLab provided a final report complete with findings, visuals, quotations, and recommendations.

 

phase 8: repeat & conclude

 

  • Each product team completed one round of testing every 3 weeks.

  • Eventually, our product teams were up-to-date and an in-house researcher took over testing internally.