Top 3 AI advances that matter for Qual Data Analysis: and where do we go from here

1. You can take advice on what research to do, what methods to use and in general ask Generative Ai to create a draft research proposal.  
2. Gen AI can write draft screeners and discussion guide even moderate as well as act as participant. 
   
3. It can transcribe the data, code it and create presentations. 
   

Advances that impact Qualitative Data Analysis 

This is where the rubber hits the road, in our opinion. For any technology to be truly impactful, it should impact both existing processes, as well as create new pathways for innovation. Let’s take a closer look at 3. 

1. LLM’s (Large Language models) 

This is where the significant development has been seen in the last year or so. ChatGPT, Llama, HuggingFace and their ilk are all taking over conversations in the development world. Flowres.io has been testing out various aspects of LLM’s and how they can greatly enhance content analysis, for instance (you will soon be able to tes  this yourselves, in our CA Experience Hub). LinkedIn is awash with webinars on the subject of data analysis and the catharsis being experienced as AI advances in leaps and bounds (Llama 2, GPT 4.0). 

So what has changed? Simply put, the training data sets, computing power and better algorithms and code have come together to continuously move the needle. 

2. Natural Language Processing and Generation (NLP/NLG) 

Natural language processing (NLP) uses statistical and algorithmic techniques to find the meaning of words and sentences in context of the whole body of text. 

Natural language generation (NLG) uses algorithms and models techniques to create text that resembles human language. 

Both of these models are getting better and understanding patterns in a body of text and rewriting them to    

Here are some of the ways that NLP and NLG can impact qualitative data analysis in the immediate term, according to a research paper titled "The Impact of Natural Language Processing and Natural Language Generation on Qualitative Data Analysis" by Chen et al. (2023): 

1. Automated coding: NLP can be used to automate the process of coding qualitative data, which is often a time-consuming and labor-intensive task. This could free up qualitative researchers to focus on more creative and analytical tasks. 
2. Theme identification: NLP can be used to identify themes in qualitative data, which can be difficult and time-consuming to do manually. This could help qualitative researchers to better understand the data and to generate new insights. 
   
3. Text summarization: NLG can be used to summarize qualitative data, which can be helpful for communicating the findings of qualitative research to a wider audience. 
   
4. Visualization of qualitative data: NLP and NLG can be used to create visualizations of qualitative data, which can help to make the data more accessible and understandable. 
   
5. Translation of qualitative data: NLG can be used to translate qualitative data into different languages, which can help to make the data more accessible to a wider audience. 
   
6. Discovery of hidden patterns: NLP can be used to discover hidden patterns in qualitative data that would be difficult or impossible to find using traditional methods. This could lead to new insights into the data and could help to improve decision-making. 

(from "The Impact of Natural Language Processing and Natural Language Generation on Qualitative Data Analysis" by Chen et al. (2023) 

3. Increased availability of data 

Both of the above are powered by the data that is available to learning and training. And as more types, sources and volumes of data get digitized, 

More theories and hypotheses can be tested from a conceptual point of view, leading to greater contextual awareness of models and therefore better output in general. As we all continue to test applications and chatbots and image generators, we are asked to rate the quality of the output. All this is further meta-data generated that helps models become sharper and more capable. 

If you want more information on how Generative AI is being used and can be used in the immediate term, do check the webinar we conducted with #lazresearch by Lazada  

View Webinar - Harnessing Generative AI for Qualitative Data Analysis

Challenges and Mitigation 

History is littered with examples of how unmindful technological advancement raises significant ethical conundrums as to the consequences of applying these. So it is with Generative AI. But it is not just those that one needs to be aware of. At the risk of repeating what many, more qualified experts are saying, here is a list of things one has to be mindful of. 

1. Ethics: There are ethical concerns about the use of NLP and NLG for qualitative data analysis, such as the potential for privacy violations and the use of the technology to manipulate or deceive people. 
   
2. Bias: NLP and NLG models can be biased, which could lead to inaccurate or misleading results. 
   
3. Interpretability: It can be difficult to interpret the results of NLP and NLG models. 

At this juncture, the one thing that will help us tame the beast is expert intervention and augmentation. Experts have a historical and cultural perspective that thus far cannot easily be replicated. Applying this implicit wisdom to the process, and the outputs, can help mitigate these challenges to a certain extent. 

For the ethical aspects, self-regulation, coupled with oversight from central agencies can potentially bring in the necessary transparency and accountability. 

Conclusion 

The advances in generative AI promise to change how we approach and execute qualitative data analysis. However,  

In terms of the immediate term impact of qualitative data analysis, here are the 2 areas we feel are going to see a lot of development and progress over the next few months. 

It is also equally obvious that humans will have a part of play not just the process and output related aspects of qualitative data analysis, but will also play a huge role in maintaining appropriate oversight. And of course maintaining the right balance in benefitting from advancement while addressing negative consequences. 

Maniish Karve 

In next week's blog, let's take a look at some AI generated output to demonstrate what are talking about.

PS: If you’d like to know about how myMRPlace is building tools to greatly improve efficiency while dealing with some of the conundrums, ethical and procedural, do get in touch. Happy to discuss and discover together.  

Send Email