Monday, 6 February 2017

On data blind and data informed

This post is the story of a personal journey. It starts with a legacy product. Both powerful and flexible having grown to meet needs over time. But it is my view on steps to improve the process in my context - so your mileage may vary.

Phase 1: Transactional DB as source

Started with looking at the transactional data. Some things were obvious, one option in the UI is recorded as true/false in the DB. Some were more complicated and required more investigation. To back this up we conducted a user survey. Followed up by a workshop to find the most important user journey to concentrate on. This highlighted three related journeys. The key characteristic of activity in this phase was mining different structures of data. The format of the data coming from custom reports.

Getting to the questions to ask was guided by a user survey to almost the whole user base, a smaller workshop looking at the "job to be done" for the platform. Other than that the skills needed where accessible to most development companies or teams - SQL and Excel.

Phase 2: Audit metrics

During developing these we put in place metrics to track the different activities. This was across all clients in on pace. Even though these were basic metrics they did allow patterns of usage to emerge. For example, actions being triggered and then canceled. Repeatedly. We were able to follow up and check why this happened. Before it became a support or account management issue. This phase had standardised event data that was easily aggregated and sorted. But like the previous phase still tabular in nature. A key benefit here over the transactional data was explicit time stamping of events and users.

In this phase, we also started to use Google Analytics to make the collection of browser user statistics easier. One advantage that we gained over web server log file analysis (other than having to do log file analysis ;) was the browser window size and native screen size. This was useful for guiding the constraints on page design, and tracking how this changed over time.

Phase 3: Users behaviour

The next phase came in adding in more user journeys and variations on the three that already existed. the different combinations made it more difficult to see what was happening and who the interrelated functions were used in tabular data. So the next step was to use session recording. This also helps speed up doing usability testing by opening up a way of recording sessions and playing back. This is invaluable as the pace of development in the new framework speeds up. The feedback on how that solves the problem also needs to scale and improve in quality.

This does bring interesting challenges on workflow. Previous phases have ultimately ended up in some for of document - Excel spreadsheet or table in Word. So a factor in evaluating tools is "how can I make sure this interesting behaviour can be referenced". Can I get a permalink to go to a point in the playback? Will it integrate with our chat tool to let people know something has been found? ... will they then use it?

Phase 4+: Infinity and beyond...

What comes next very much depends on which process problem we are looking to solve next. One thing in our context that probably won't make sense is A/B testing, as in our B2B environment consistency and predictability are valued. Some of the things that it might include are:
  • ROI - Taking into account factors like whole-life costing of the new product development, revenue generation/protection and feature usage to get the most bangs per buck.
  • Process mining - looking at user behaviour to find correlations between tasks to form new packages or products.
  • In app feedback - fine grained/easy feedback, like the MSDN "was this article useful" to flag issues.
  • Visulisation - how information flows through the system, maybe animated to really tell a story.
  • Heat maps - Another way to tell a story about feature usage.
  • Further development of customer types and tailoring products for them.

Further reading