Problem validation & preventing expensive mistakes


TL;DR I saved a company $300,000 by conducting research that proved that the problem they were trying to solve didn't actually exist.

Contact me to learn more

Due to the nature of my NDA, I can't share too much detail about this project. Please reach out to me if you would like to know a bit more about my process for this client.

I spent two months while I was working at Airteam working on this research project for this client. I was approached to design the plan for this research, and execute it. You can see Airteam's post about this project here.

Summary

Problem

The client had conceptualised an app that they believed solved a tangible problem for their customers. The app was based on assumptions they had about users as well as research that they had conducted years ago. They wanted to know if they should go forward with developing it (which would have been an initial $300,000 commitment).

Approach

I conducted research focusing more on understanding problems users were facing, but also included testing the concept they had come up with. Here's a high level overview of some things I did:

  • Stakeholder workshops to debunk assumptions about perceived user needs and journeys, and discuss their business goals
  • Guerilla interviews to explore current user needs and journeys
  • Guerilla concept validation of new app idea
  • Survey to support qualitative findings and
  • Workshops with users to dive deeper into insights from interviews + look into parallel journeys

-

Outcome & impact

My research proved that the perceived user pain points were not that prevalent, and therefore the client should not continue exploring this concept.

  • The client was able to make an informed decision about whether or not to invest money into an expensive concept.
  • Despite not going through with the app idea, the client now has insight into what the user's journey is really like, and are able to to explore and ideate for the new problems I discovered.

Takeaways

💡 Define the right problem before coming up with solutions

I acknowledge that:

  1. Not all teams can afford time or money to do thorough domain research
  2. It's sometimes tempting and more tangible to jump to “making” things rather than doing problem space research first (as highlighted by this anti-research blogger in this Medium article)
  3. Some teams can afford the risk of not doing problem space research first (which is fine!)
Despite this, I stand by the opinion that if you don’t look at the right problems first, or if you assume there is a problem in the first place, there is higher risk of you building an idea that is not useful.

In this scenario, I didn’t have time or money to do thorough domain research, but I could have easily made the mistake of only testing the prototype of the concept. The outcome of this would have been identifying usability issues, and making usability improvements to the prototype.

Instead, I acknowledged the uncertainty that the clients thought people had, and focused on giving them the ability to make confident decisions. I didn't have time or money to do "explore" completely, but I focussed on validating the perceived problem. The outcome of this was seeing if the perceived problems existed, discovering what problems actually existed, and seeing if the concept brought value to users.

The interesting part of this project is that the outcome was cancelling the development of the product. I was even nervous to put this in my portfolio because of that but I realise there is a lot to be learned from this sort of thing. Even if the development of the product stopped, what’s worse; making a usable product that doesn’t solve any problems or saving a bunch of money by not going in the wrong direction?

As a leaving gift, UX Strategist @dav_hamill brings up an interesting point in his talk Why User Testing Isn't Enough: "Stop throwing shit at walls and start studying real behaviour." The rest of his slides are pretty spot on and reiterate the essence of what I learned project.