Evaluating Qual Data Analysis abilities of AI powered platforms

 “AI is going to save you time and money!”

About five years ago, the first rumblings of AI were being mentioned in marketing research circles. Like most other people in my industry, I filed those rumblings away in my brain. Over time, those mentions of AI in marketing research collected dust on those proverbial bookshelves of my mind.

But roughly two years ago, I had to suddenly dust off those memories because AI came rushing in with a thunderous boom. I’d gone to an industry conference and was surprised to see the marketing research conference schedule included several sessions dedicated to AI. I was fascinated and interested, but also lost; I didn’t know how to interpret what I was hearing, nor imagine how AI could be applied in my line of work. There were whispers in the convention hallways about AI. “Did you see that session on AI?” “Wasn’t that cool?” There was a lot of buzz about the potential of AI in marketing research.

The following year, AI was unavoidable; it was everywhere. It came crashing into my world with the gale-force winds of a hurricane! It seemed like every vendor in our industry had launched a new AI-powered market research platform. And there were seemingly 100s of new AI entrants jumping into our industry that had never been there before.

So, it’s no surprise that many of these vendors started reaching out to me and courting me for my business. They were offering free or discounted trials of their new AI-powered tools. I began scheduling calls with some of the vendors so I could better understand the tools that were now being offered. And it was during my very first vendor call that the guy on the other end of the line said something akin to: “this AI tool is going to save you time by doing your analysis and reporting for you.” My ears perked up!  But what I heard in the back of my brain was “Artificial Intelligence is coming for my job.” This has been an unfounded refrain for decades in the marketing research industry; every new technology implies that “it’s the end of qualitative research.” And history has shown us that previous tech advances merely gave us more tools for qualitative researchers to do our jobs.

But this time seemed potentially different. If these AI-powered tools could really do what the vendors were saying they can do, then was I right to be truly worried?

Skeptical Optimism

As much as I enjoy sci-fi and have probably seen every movie Hollywood has ever produced about AI, there was something nagging at me. I’ve always been an optimistic skeptic or a skeptical optimist. I just couldn’t fathom that some fancy algorithm could replicate what millions of years of evolution had accomplished and culminated in the form of the human mind. In my estimation, the human brain is one neuron short of a miracle! It’s been estimated that the adult human brain has upwards of 100 billion neurons. And each neuron is connected to thousands of other neurons. Before synaptic pruning occurs in the adult brain, there are upwards of 150 quadrillion synaptic connections in a human child!  That’s a mind-numbingly large number. Yes, computers are faster than humans, but the human mind is immeasurably complex. In my humble opinion, it was hard for me to see how any computer could do what the human brain can do. Until science proves otherwise, the human brain is quite literally the most complex and powerful creation in the universe.

In my call with the vendor, he said his AI tool was going to revolutionize qualitative research and save me countless hours of analysis. I needed to see the receipts. I asked the vendor to show me how the AI tool worked and more importantly, I wanted to see the output—the AI-powered analysis. I listened to his pitch intently, took notes, and watched his demo of the AI-generated “reports.”  By the end of his pitch, he asked me for my thoughts.  I took a beat, collected myself, and calmly thanked him for the demo, told him that I was impressed with how far AI has come as a technological advancement. But more than anything else, I was truly relieved. I mentioned that I was relieved because based on the demo he had just shown me, AI was nowhere close to taking my job as a qualitative researcher. I explained that AI’s ability to summarize main themes is truly and amazingly impressive. I could see the benefit of using a tool like this to identify main themes of a research project. But his “report” contained zero synthesis of new ideas, or implications, or recommendations based on solid analysis. He agreed, that yes, perhaps his sales pitch was overpromising a bit.

But the drumbeat of the AI-powered market research platforms only got louder and louder. I was being contacted weekly, if not daily, by new vendors trying to tell me how their platform was truly different than the competition. They weren’t. 

I wasn’t only hearing that drumbeat from afar either—I was hearing the same drumbeat from my colleagues as well. And their evangelizing call for AI was getting louder and more widespread. I was starting to feel like an outsider—a naysayer of sorts.

The Experiment

So, I decided to run an experiment. I wanted to see if any of the market research platforms were living up to their promises. I had questions I wanted answered:

1.      Will these AI-powered tools really save me time?

2.      Can these AI-powered tools really create solid and accurate summaries of main themes?

3.      Can they synthesize the data into relevant implications and actionable insights?

4.      Are these AI-powered tools going to put me out of a job?

There were other very important questions that needed to be answered, but those questions are far more complex, more nuanced, and ethically sticky. Questions no one individual is qualified to answer—questions that the marketing research industry as a whole needs to answer. Questions like: 

1.      What do these AI tools do with the data we provide them?  And what should they do with our data?

2.      Should we divulge to our clients when we are using AI tools? 

3.      Who’s responsible if poor business decisions are made based on the results provided by these AI tools?

4.      How do we identify hallucinations?

5.      What do we do about hallucinations?

6.      Who’s responsible if a hallucination makes it through to the end client?

7.      Is good enough really good enough? 

8.      Do we want to make decisions or recommendations based on good enough?

9.      AI tools skew toward the mean.  Do we as professionals only want to focus on or report on the mean?  Is the mean what our clients really want? Or need?

To conduct my experiment, I reached out to six different AI-powered market research platforms. I purposely picked a range of brands from very well-known brands to lesser-known brands. The one criteria they each had to have:  they all had to claim that their AI platforms provided qualitative analysis and/or reporting with the promise that they would save researchers time.

I asked each vendor for free, unfettered access on a limited trial basis to their AI tools. I was upfront with the vendors about my intentions. And I explained that my experiment was not about pitting vendor against vendor, but rather solely about human vs. machine. I explained I was running an experiment to uncover two things:

1.      Are they living up to their product claims? (e.g., saving me time, saving me money, saving me cognitive resources)

2.      How does their AI-powered analysis and reporting stack-up against human-powered analysis and reporting?

I promised to share the unvarnished results of my experiment with them. And each vendor agreed to the terms of my experiment.

The Methodology

My goal was to compare machine-generated insights and analysis to human-generated insights and analysis. So, I decided to use a completed project of my own for one of my cybersecurity clients. I picked a report that I knew the client was very happy with the results. And then I set some parameters of the experiment:

1.      To avoid giving access to any non-relevant data that might confuse the AI or increase chances of the AI hallucinating, I opted to limit the experiment to a single, straightforward question the client wanted answered:

“What are the top 3 challenges you face in securing your organization?”

2.      Each transcript only included the answer to this single question.

3.      No PII was included except for the respondents’ industry.

4.      And all AI platforms were given the exact same queries or prompts.

The Results of My Experiment

After uploading the edited transcripts, I began running the queries on each platform.  The queries were not complex questions; they were very simple questions, which I’ve illustrated below.  The results of my experiment were mixed. Overall, I found that AI platforms are really, really good at one thing—summarizing main themes. And abysmal on most other analysis and reporting tasks.

1.      Generate a report of all pertinent and relevant insights.

For query #1 above, this is where AI truly shined. Nearly all of the AI-powered marketing research platforms excelled at summarizing main themes. Some were better than others, but overall they all did a good job; however, there was one disconcerting finding. A few of the platforms emphasized themes or insights that I had deemed less relevant, and more importantly, at least two of the six platforms actually hallucinated insights. They reported a theme that was never mentioned in the transcripts.

2.      Write a 1-2 sentence summary of each respondent's answer to the question: “What are the top 3 challenges you face in securing your organization?”

For query #2, it’s unsurprising that the AI-powered platforms did well on this task as well. They provided mostly accurate summaries of each respondent’s viewpoint. But like before, a few of the platforms hallucinated insights that were never said by any respondent.

3.      Write a single, short, concise, pithy headline that encapsulates the research findings.

For query #3, the results were almost laughable. In my human-generated report, I wrote a powerful, yet succinct, action-oriented, four-word headline:  “People are the problem.”  In my prompt that I gave to each AI tool, I requested a “single, short, concise, pithy headline that encapsulates the research findings.” The AI-generated responses were too long, unfocused, misleading, and sprinkled with baseless clichés. Here’s just a sample of a few of the AI-generated headlines:

“Internal Employees and Budget Constraints: The Dual Achilles' Heels of Cybersecurity Across Industries”

“Navigating Cybersecurity Challenges:  Staying Ahead of Threats, Selecting Suitable Solutions, and Empowering the Workforce”

“User Awareness and Legacy Systems: Key Cybersecurity Challenges for Organizations”

4.      Create a list of recommendations for the client if they asked what they should do to address the IT decision-makers' challenges in securing their organizations.

And finally for query #4, this is another area where the AI-powered platforms performed poorly. To illustrate how poorly the AI-generated responses were, I need to briefly summarize the challenges the IT decision-makers mentioned. First, their biggest problem was people—both internal and external users. Their other constraints were small IT budgets, an under-skilled talent pool, no money to train them, and exponential company growth which was stretching their IT staffs to the brink.

To understand the gravity of what the AI-platforms did, you have to recall that I only uploaded transcripts to a single question and that question asked what challenges IT decision-makers experience in securing their organizations. And keep in mind that the question or prompt I asked the AI to answer was to provide recommendations taking into account the challenges IT decision-makers faced.

In this instance, all of the platforms flat out ignored my instructions. Every platform provided recommendations that cost time, money, and lots of effort—the very three challenges the IT decision-makers expressly said were problematic for them! I will concede that all of the recommendations were very good recommendations—absent taking into account their challenges.

The Successes and Pitfalls of My Experiment

Granted this is only my one-man, small-scale, qualitative experiment.  Anecdotally, I’ve heard from several colleagues of quite similar experiences they’ve encountered with AI-powered analysis platforms.   But what successes and pitfalls can you glean from this experiment?

I am in favor of the use of AI-powered analysis tools to suss out main themes and insights quickly—but keep in mind, these are tools to be utilized by researchers; they are not final arbiters of research findings. In my view, we can use these powerful AI tools in two ways:  1) either we can use them as a starting point for further human analysis; 2) or alternatively, we can use them as a double-check of our own human-generated insights.  And what makes them especially useful is if we use tagging or coding of segments, then we can explore the differences and similarities in the main themes on a per segment basis. And that’s a true time-saver.

As for the pitfalls, there are many. As my experiment has shown, they can hallucinate and add in false insights—even when these AI agents were fed small, edited bits of data they still hallucinated. I’m left to imagine what would have happened if I had uploaded unedited transcripts?  My worry is that if we researchers cannot easily and quickly identify an AI-generated hallucination, then we become part of the problem. We become the facilitators of unreliable insights based on falsehoods. Additionally, it’s concerning that AI can actually ignore instructions in a query/prompt. And this leads me to an even bigger worry…these AI-powered tools are designed to provide answers. But we as qualitative researchers know, sometimes there is no answer.

For some valuable tips-n-tricks and best practices on using AI-powered marketing research platforms, head over to my blog post:  https://www.happytalkresearch.com/post/tips-n-tricks-for-using-ai-powered-qualitative-research-platforms