ONLINE WEBINAR : Applying AI for effective report-writing in Qualitative Research [August 28, 2024]

Moderator: Jiten Madia, Founder, flowres.io

Experts:

Introductions

Jiten: Let me just quickly introduce Dean, Kayte and Doug. So Dean has been naturally inquisitive about human behavior, with a degree in Humanities and a 13-year long stint in Advertising. He loves talking to people and understand the whys. Dean is also a great data analyst, qualitative data analyst, runs a highly demanded report-writing course for QRCA,

Moving to Kayte... so, Kayte brings an interesting perspective to research. She started her research journey with Idea Fairy and today, combines her passion for research and social media, through her firm Social Question. Kayte also helps with influencer marketing guided by consumer insights, is a vocal advocate for inclusivity and accessibility and also developing ScribeQ, an app for social questions.

Okay, and moving to Doug. Doug is a market research veteran with 30 years of MR experience, across Qualitative and Quantitative. He loves finding stories in the data and helps clients in making the decisions that are consumer-centric. He is also passionate about helping people find their career path through ICN, Insight Career Network, which is a fabulous career networking website. He also co-leads ‘MRX Pros’, a very lively MR community.

A little bit about myself… I’ve been a qualitative researcher, turned into an entrepreneur. I have been on both sides of table that is, Nielsen/ Ipsos and also John Deere/ Bajaj Auto. Right now, we operate two brands... we have a service business, which is myTranscriptionPlace, a transcription service tailored to MR projects. And we have flowresAI… our (product) web platform, which basically turns your Zoom account into a full-fledged qualitative research platform, through various features

Webinar agenda

Jiten: This entire seminar is going to be tool-agnostic. We are going to talk more about how to use AI and to start with, we have a quick poll... everyone can just quickly answer these two questions. Just give it a minute.

The first question is – Where you are on your AI journey? I think a lot of people are using a lot of AI, so it's possible that I might have oversimplified here.

And the second one is – Where all you have used AI in your qual research process?

State of AI x Qual

Jiten: Okay, so what do we have here? We have a lot of AI curious people. We have a lot of AI dabblers and practitioners. People have used it for proposal writing, data analysis, followed by transcription. Synthetic moderation is interesting. So, Kayte, Dean and Doug, I think, let me launch the question and maybe you can decide to pick it up. 

Doug: First of all, thank you so much for the opportunity to be here today. I just want to emphasize that when I look at those poll results, they really line up with what I've heard… which is something like around 30% of businesses are actually using AI to do anything. So we think it's an all-pervasive tool, yet 70% of businesses are not using it. Once I started using it, I realized I had wasted a lot of time not using it, but and it's proven to be incredibly valuable.

Kayte: Yeah, a lot of wasted time before getting started, but also the hindsight of wishing that I had access to some of these tools like a decade ago. And I remember there was a time that the second or third report that I was working on… I was adamant that a particular phrase/ topic should belong in the report, since we had X amount of people talking about it. And the rest of the team was like, no, that really wasn't like talked about that often. So I physically went back to the transcripts myself and manually highlighted each time this word/ phrase/ topic was discussed… it came up 14 times in 6 interviews. It came out later that it needed to be de-prioritized in the report for other reasons, but the idea of hearing something in a focus group and not having a tool to, like, quickly synthesize what that high point was a point of tension between me and my team. In the present day. I could have uploaded it and asked for that to be synthesized.

I am still not fully in AI. I don't think any of us are truly AI experts yet. But I needed to learn the probes to send in to the chat feature, how to craft the instructions/ guidance/ directions. I wish I had started when I had just a little bit of data to earn those skills to figure out how to actually use the platform.

Jiten: And Dean, how about you? Did you start using AI out of compulsion like Doug and Kayte, or was it a different situation?

Dean: My path was a little bit different than Kayte and Doug's. I basically was interested because I've been hearing so much about AI at conferences… the subtext was – AI is coming for your job and so that was a little bit scary to me, and I wanted to understand it more. I was teaching a class on report writing. And the second year it was run, I decided to add in AI for qualitative analysis. I wanted to evaluate how good AI was in comparison to human analysis. And so I did an apple-to-apple comparison with 6 different platforms. And the results were interesting. All the vendors were telling me that AI is going to be doing all these wonderful things. And I just wanted to understand the reality of it.

Challenges when using AI for market research purposes

Jiten: Any challenges you faced while trying to use AI, anything that you had to overcome, anything that you had to put in a lot of effort for

Dean: I'll quickly say, because I looked at several different platforms, the challenge was is that every platform does it differently. And you have to decide which platform is right for you. That's a little bit of a challenge. So you have to kind of feel them all out and decide what works best for your style, your qualitative analysis style.

Doug: The main thing I knew was you have to ask good questions of it. I had really started thinking, well, you're doing the same thing that you're doing with a respondent. You have to ask a good question initially to get an initial response, and then you have to ask good follow-up questions to get more detail. And I'd read an article in the New York Times about ‘golden’ prompts… they are the ones that elicit the best response across whatever subject you're asking about. And it's even more important when you are asking questions about research that you've done because AI does not really understand what you're asking or why you're asking it. So you even have to explain that piece to it… “I am a qualitative researcher. I am researching this topic”. The first time I did it, I actually just copied all of the objectives out of the proposal and put them into the description. This is what I am trying to learn and why I want to learn it.

But then the second piece is when you finally start getting information back, which is very, very quick. What I've realized is, no, people actually count on me, the human, to come back with great information. You're the one who's going to whittle all of that information down. So I think one issue you can run into is if you say – “Give me a summary with five points that I can use as my executive summary”. And then you just copy and paste that and put it in. A… someone's going to figure that out because it's written like a 9^th-grader would write a book report. B… it's going to miss things that you clearly know that it doesn't know. So get used to that idea that I'm not trying AI to just save time. I'm actually trying to do better work and get more detail. I think once I understood that, then I could just figure out the nuts and bolts of using the tool.

Kayte: I would say that's one of the things that was challenging for me getting started because of that mountain of data that I needed to like push through, is that I neglected the whole training element. And I was has this perception that okay, this AI is going to do all of the thinking for me… I just need to give it the data and it's going to spit out The Magic Exec Summary, the perfect quote. However, I realized that I needed to give it that background, some boundaries in which to analyze the data.

I had to learn that those instructions are really, really important to whatever GenAI engine you're using. Another one of the challenges that I've had is I'm using AI most frequently, to code open-ended responses. I'm using Instagram stories where I partner with an influencer and that influencer or creator hosts questions with their audience. So now I have potentially thousands of data points that have come back from just hosting one or two questions on social media.

Don't wait until it's like really pivotal when you need AI's help because you're going to face similar challenges and then you're going to be up against the time block and getting more frustrated. You really do need to like slowly train yourself or do something as investigative as Dean did. Because if you wait until it's highly critical, then you're just setting yourself up for failure.

Doug: I'll tell you one of the great challenges I've had with ChatGPT… because it's not a tool that's designed for research, you have to do a lot of data prep. You can't just say – “Here are 10 transcripts. Tell me the themes across all of them”. It just comes back with a response that says – “I can't do that”.

So then you have to do something like – “Summarize Respondent 1's transcript”. Or, go down to the question level – “Tell me xxx (type of information) for Respondent 1”. I take all that information, I cut and paste it into a document, then I upload that document and say – “Summarize this information”. If you've done 8 interviews, it's not a big deal. But when there's 32 interviews, it really can be a big deal.

So one of the great advantages of the market research specific tools now, and flowres as an example… I did a project with 32 interviews, and I was able to upload all 32 interviews but say – “Only analyze interviews in x/ y/ z segment”.

AI-enabled Qualitative Research tools/ platforms used

Jiten: Absolutely. Absolutely. In fact, I read an article somewhere. I think by Renee Hopkins, in a QRCA where she advises that not to wait for the winner. Instead, just get started with the AI. I think Kayte mentioned tools. Dean, you have tried some tools... maybe you want to talk about what all tools you have tried, anything on your wish list.

Dean: Okay, so let me start with what I think they do really well. And I'm talking specifically about market research platforms that have AI inside them. They do a very, very decent job of summarizing main themes. I had really good results across all of them. It's when you go deeper than that, where I started to see some of the qualities start to vary among platforms. In terms of tips and tricks, I would say, choose your AI platform carefully. Don't commit right away to any one platform… instead, experiment with a couple of different platforms. I experimented with 6 and now I've gone through 10 or 12. Once you decide on a platform, ask for a demo, have them show you how to use it. Because they will show you things that you wouldn't think of. And just know that some of they are going to hallucinate. Not a lot, but they do hallucinate.

Jiten: We are seeing the infancy stage right now for almost every tool in the market, at least from what we know. But what all tools have you tried? I mean, Kayte mentioned about Quad...

Dean: Okay, so in my original test, I tested flowres, which is Jiten's company. I tested Yabble, CoLoop, Quillit.AI, and EasyThemes. EasyThemes is a little bit different in that it actually produces a PowerPoint document. So their system is a little bit different. The others just give you Word or Excel documents as outputs. In terms of quality, and I don't want to show favorites, but Quillit.ai and flowres produced the highest quality themes; compared to human analysis.

How to explore AI-enabled Qualitative Research tools/ platforms

Doug: I want to emphasize what Dean said about trying to get demos, if you can. I think as an independent researcher, and I think there are a number of independent researchers here listening in today, the biggest challenge is how am I going to pay for this? So it could be the coolest tool in the world, but if it's going to cost you $1,000 a month to use it, are you going to get $1,000 a month worth of value out of it? So when I started, I started with ChatGPT and as I mentioned, I got the paid version.

I just want to stress for anybody that's new to it, data privacy is a huge concern. So you just want to be sure that you're using a tool that at least promises that this information is not being used to train their LLM. One of the great values of using a tool that was developed for Research is that there's already some understanding built into the tool, that we're talking about Research. So, the tool doesn't start speculating on broad topics.

And just one side note, if transcripts are your input, you have to point out to AI – “Please only review the attached material”… because otherwise, it'll just go out on the worldwide web and just pull back anything across all industries. 

When I did my first ChatGPT analysis, I first ran a Google Search – “What are the best AI prompts for doing this type of analysis”? And I found an entire article about it, with prompts written out already. And I started cutting pasting those prompts (into ChatGPT) and just shaping what I was doing. So, I didn't start from scratch.

If you find that somebody's started using these tools to do what you're doing, then start figuring out how you can advance yourself to the next level. How do you make yourself more valuable? This sounds really broad and easy to do, right? It's not. But you have to say, okay, how can I use this tool to stay ahead of the people that need to use it? Because they're probably like any bad researcher. They're asking the wrong questions of the thing. So you have to tell them… hey, a little knowledge can be dangerous. Let me tell you how to use this information correctly.

AI-enabled Qualitative Research tools/ platforms : What works?

Jiten: Absolutely. So changing gears a little bit. What are the advantages that you have observed of using these tools Some of it you already I know sort of talked about but and I'm just trying to understand is it time-saving? Is it adding more rigor? Is it making the process better? What would you say is really how specifically it is helping you in the data analysis process?

Kayte: One of the things that I have found really beneficial is once I have my draft/ final slides ready, I can actually upload each of those slides independently or as an entire PDF and be like – “Please critique this. Does this make sense? Is the story flow the right way?” Or send it text-heavy slides and ask for an infographic. So, use it as an extension of my creative brain – to either proofread or give me creative ideation, so I can get to that final stage.

But it still always takes human intelligence to overlay, which is something I don't think we've really discussed yet as a group here. AI is not always going to give you the right information or the right information for your project, for your objectives.

I never take the advice or the suggestions or the headline rewrite. It's just blindly copying and pasting it into my PowerPoint. I still need to make sure it has the voice of the social question or the voice of the product or the right tone usage and those kinds of things. And that's also what I tell my undergraduate students when it comes to AI as well – that I am okay if they use it for a couple of their assignments, but they still need to proof it, because I promise them – the information might not be accurate.

Doug: I've seen presentations and talks where time-saving is presented as the greatest feature of AI. I don't really see it so much as a time-saving tool as helping me do reporting.

I remember years ago, A friend of mine said… “So do you do what I do? Which is – you remember quotes off the top of your head and those are the ones you type into your deck?”. I think that used to be kind of an approach, like I jotted down some notes and I'll just use that. But I would get to a point where I'm trying to remember some specifics that people said about a certain topic. Now, I can ask AI – “Hey, can you summarize what people said about this particular question or this theme?”. So would you call that a time saver? Maybe it is, because then you don't have to go back and try and recreate your own memory. There are some things you can do, I think, that will be time saving. A friend of mine had written code to generate slides. I'm not quite that advanced. But he put it into ChatGPT and it spit out like 20 PowerPoint slides. That's a time saver. But in terms of doing analysis, it's I think you have to see it as a helper that's going to help you do a more thorough job.

Dean: Yeah, 100% agree with everybody there. The claims that these vendors are making about time savings aren’t true. I see the biggest value of these AI platforms in two ways – one, they can give you a really good starting place for human analysis. Two, you've done your human analysis, and now you want to just double check it. And I think right now, none of these platforms are replacing human synthesis and analysis and report writing. There are tools for us to do a better job, like Doug said.

Role of AI vs Human Intervention, in Qualitative Research analysis

Jiten: Just taking this little further. So how would you really What advice you will give in terms of maintaining the human component versus using the AI help? What is the right balance or where all you think human has to keep intervening or as a researcher, whereas you give some work to AI and be sure that it will deliver well. Does that make sense?

Kayte: I mean, I think you have to lean into your qualitative moderator training to know if it's the right output or the wrong output. You have to trust your relationship with the client and how close you are to the project because the AI is not replacing you. You know your project, the data, you were there in conversations.

Doug: I thought about this a lot in the last couple of years… Where do I actually add value? There are a lot of great moderators, great qualitative researchers. And I think it's in pulling together threads of information that maybe the client hadn't even considered. So when a client comes to you with a request and they say, I have these 3 objectives, and you provide insights and recommendations for each objective. But did you also notice this overarching theme that people talked about, which could change the way that your client does their business?

So you get 5 themes summarized, from the data. But then you have to say – What was it that I learned that I can take this research to a different level, help this client look like a hero? And if you ever think about CEOs of companies that are doing talks like this, they aren't walking on to the stage cold and just winging it. There's a team of people behind them that says, we should think about this. You can think about this. We can summarize it this way. And they're well prepared when they're answering the types of questions we're talking about today. And then they can build on that because they've already kind of thought through it. And that's what AI is, I think. Where it's really valuable is that you've been giving a whole bunch more information that you might have had, that was kind of sketchy in your brain. And now you can say, wow, look at this, I can pull something from this that I never thought about.

Dean: I just want to quickly build on what Doug was saying because I had a thousand percent agree with what he said. Like, for example, we're all experts, right? Like Kayte's as an expert in social analytics. I've done a thousand automotive and pharmaceutical and cybersecurity type stuff. And I have institutional knowledge, 20 years of automotive experience and pharmaceutical and all that stuff, right. And that's all and I have like, I have stuff from previous research projects. All this, the internet doesn't have and AI doesn't have. And that's the value that I bring to my clients. And I know how to string all that together.

Q&A

1.     Attendee 1: This is so great. I am currently working for a company where we're kind of late to the party with AI. It has been around for 40 plus years. So we have a lot of the older generation who is fear based about AI. So we slowly introduced AI. I'm still new to AI, but I'm even fearful of it. I just wanted to put it out there because there are still many of us that are fearful. I talked to a lot of screenwriters, producers. We had a long strike last summer, and a lot of protesting about being replaced and losing their jobs. So that's my case. Thank you.

Kayte: I would say – from a resume/ job-replacement perspective – it needs to be part of your skill set, part of that continual education/ learning. I was once paid as an undergraduate intern, to learn how to use PhotoShop.

2.     Jiten: I can read some questions – What are the golden prompts that have helped you. Some tips and tricks to refine the prompts?

Doug: I think the simplest way to explain it like you were talking to a junior researcher. You'd say – Let me tell you about the objectives of this study, explain what I'm trying, who I am. I'm a senior researcher, and I have to write this report for a project I did where I talked to people about billing errors. And the problem with the billing errors is blah, blah, blah. So essentially, you're telling the tool to think like a researcher, not a consumer. So you're having a chat with it and then we're doing your refinements. When you see the thing that's wrong, just tell it, what's wrong. Think of it like I'm having a conversation with this thing.

Kayte: It's going to respond to every input that you send it. So I have to give it a really long set of prompts and then the task. Because as soon as I hit enter, it's going to respond with something.

3.     Maniish: Attendee 2 has asked about some examples of AI Chatbots being used on websites.

Doug: It's actually a very interesting question. And just from a Research perspective, I've talked to a few companies that have developed chatbots to do research. It has progressed a good bit. One of the companies actually developed a chat bot for businesses to use for Customer Service, but we can utilize it for Research too. And it is really a challenge because people ask nuanced questions and then the thing doesn't have a nuanced response. It just gives you like, here's three directions you can go. There are these organizations I think that represents companies, to talk about their use in these chatbots. You might want to search kind of in that direction.

Closing comments

Doug: Well, I can tell you what was really the best way to start. It just was coincidental, but I'd done an online bulletin board. So therefore there'd be a question followed by responses, then the next question followed by responses. I could just paste all those responses in and say – Summarize this for me. It is more challenging if you've done interviews and the questions are not in any particular order. Take one question, take responses that I see use the paid version of ChatGPT, to protect your data. You can attach to the responses in a document and say – Please, only reviewing this document, write a summary of 3 to 5 bullet points. See what you get back and just start experimenting with it. Don't wait until 11 o'clock the night before the reports are due to say, I'm going to use this thing. It could be a little frustrating.

Dean: If you have one tip or trick, so to speak, is – When you're choosing a MR platform, really ask about the output. Because I found that there was a lot of differences among platforms. Some don't allow any kind of download at all, which I find unfathomable. Others will only download in Excel. Others will only download in Word document, or PDF. So think about what you need from the AI tool and really investigate the output that it gives you.

Jiten: And the formatting part. I mean, they need to keep the same format as the results, so that you don't have to put effort in formatting.

Kayte: I think in the grand theme of AI, we tend to forget that AI exists on a spectrum and there's different components. Not all chatbots are created equally, not all websites are created equally. I don't truly think we're in the artificial intelligence era. I think we're in the advanced intelligence era. And that's one of the reasons why there's so many platforms to experiment and try with and probably why Dean has tested so many of them because there are so many that are out there. All AI is not all AI. I had this debate with somebody at the Insights Association Conference and they were very adamant that we were in the era of AI and all of the examples they were giving me were actually could be scaled back to like purely input-output decision-making. It has the appearance of being AI, but it's not truly that. It doesn't mean I'm opposed to learning about it or trying it, but there also has to be some realization that the word choice for the platform (“AI”) being used, isn't truly accurate.

Jiten: Super. Totally great. Thanks, Dean, Doug, Kayte, for this really high engagement. And happy that this has been a value-added experience for everyone. Thank you so much for your time today.