Generic LLMs : The “One-Size-Fits-All” AI : Boon or Curse?

Generic LLMs (Large Language Models) are created to execute knowledge and language-based tasks. Think Open AI’s ChatGPT, Google’s Gemini, Microsoft’s Copilot or Anthropic’s Claude. These models have been trained on Internet-scale data and can ‘discuss’ virtually any topic. In the use-case of Market Research, one might use them via a chat interface or an Application Programming Interface (API), to analyze raw primary data. For example, a Qualitative researcher might paste interview text into ChatGPT and ask for key themes, or use an API to batch-process open-ended comments from a Quantitative survey, for sentiment analysis.

What advantages do generic LLMs offer, in Market Research

• Two-way, iterative analysis: Generic LLMs are built to be inherently Conversational with the user. This means one can literally, “talk to” one’s data. For market researchers, this is a valuable trait of Generic LLMs. Because it allows them to “feed” their data in and “query” it from various angles; allowing for iterative and thorough analysis.

• Versatility and broad knowledge: Because they’ve seen all sorts of text, generic LLMs can handle a wide range of subjects. This makes them an immensely useful tool, when you need to “go wide.” For instance, ChatGPT can analyze text on almost any topic, making it a general-purpose tool​. This broad training means it might catch unexpected references or general consumer trends; which is helpful if your data is likely to throw up wider cultural themes.

• Out-of-the-box convenience: Generic models are immediately available via web interfaces or APIs. There’s no need to train a model yourself. This makes them easy to experiment with. A team can start using a generic LLM in their workflow, with minimal setup. If a particular generic LLM does not seem to fetch relevant outputs, teams can pivot to other LLMs and experiment with applying those LLMs; to choose a generic LLM that best delivers to their specific use-case.

• Scalability and speed: These models can process text much faster than a human possibly can. For instance, if you have hundreds of open-ended responses, ChatGPT-4 can summarize them in seconds. This scalability allows researchers to analyze data at a volume that would be impractical to process, manually​.

• Integration potential: Major LLM providers offer APIs, so you can integrate a generic LLM into your systems/ tools one is already working with. For instance, a developer can connect an LLM to your research repository, to auto-tag new entries.

• Low upfront cost: With pay-as-you-go pricing or even free research previews, using a generic LLM can be cost-effective. You don’t pay to develop the model. Instead, you pay just for using it. This lowers the barrier to trying AI on your Qualitative data.

So what’s the fine print?

• Lack of domain context and nuance: A generic model isn’t tailored to the language of your specific industry or research context. It might produce answers that come across as surface-level when analyzing Qualitative data​. For example, if participants use slang or brand-specific jargon, the model might misinterpret it or give a boilerplate summary.

• Inconsistent or unpredictable outputs: Because these models are so general, the same prompt can yield different answers on different attempts​. In that sense, there is indeed an element of Randomness. This can be problematic in research, where one wants consistent, repeatable analysis methods. In other words, a ChatGPT response might be useful; but if you run it again you might get a slightly different take, which makes validation hard.

• Hallucinations and inaccuracies: LLMs sometimes make up information that wasn’t in the input. This phenomenon has been popularly labelled as ‘hallucination’. In a research context, this could mean the AI wrongly identifies a theme or attributes a quote… one that never occurred (in the data). This limitation has been addressed to a large extent by early adopters of generic LLMs; by introducing human intervention for data-checking. Without such intervention, a generic LLM might confidently provide an insight that is actually false.

• Data privacy concerns: This is a big one for corporate researchers. When you use a public LLM (like a free ChatGPT account) or even some APIs, the data you input could be stored or used to further train the model. Sensitive data might be used to further train it, unless privacy settings are carefully controlled. If you’re analyzing confidential customer feedback or new product development (NPD) concepts, uploading those to a third-party (LLM) server needs to be handled carefully. Many LLM providers now offer opt-outs or enterprise agreements; but one has to know how to actively manage these.

• New learning curve: Generic LLMs don’t automatically know what you want. Researchers must craft prompts to get useful outputs, often through trial and error. One might have to write a detailed instruction like, “Summarize this interview, focusing on pain points with the product and any suggestions the interviewee made.” Designing effective prompts and adjusting them when the output isn’t right – is a skill in itself. In initial stages, this repetitive prompt engineering can be time-consuming.

• Lack of built-in workflow integration: Using a generic LLM in isolation is very different from having analysis integrated into your existing research workflow. If you use a generic LLM, you often have to copy-paste data or write custom scripts. There’s no structured process or interface that’s specific to research – it’s up to you to carve out an analysis methodology. AI won’t automatically know concepts like code frames, sentiment scales or your research objectives; unless you prompt it every time. Such ad-hoc use can lead to inconsistent analysis methods across projects​.

• Limited validation or guardrails: Out-of-the-box, a generic AI won’t tell you if its answer is dubious. The onus of verifying accuracy lies on the researcher. Without additional checks, there’s a risk of an error slipping through. For instance, AI might mis-summarize a quote and you might not catch it immediately. In that sense, generic LLMs are a black box, with no standard safety net.

What this means for Qualitative researchers interested in exploring LLMs