Rethinking Qualitative Data Analysis. Do we truly want a faster horse?May 23, 2023
You may have come across the quote: "If I had asked people what they wanted, they would have said a faster horse." Although often attributed to Henry Ford, there is no evidence that he actually said it. The essence of the quote is that true innovation goes beyond customer input.
While customer input is generally important for product development, there are instances where customers may be limited by their current perspective. Consider the example of the PC:
When asked in the early 1940s how many computers the world needed, IBM's president, Thomas J. Watson, responded, "I think there is a world market for about five computers."
Innovations such as the Mac, iPod, or iPhone wouldn't exist if a few individuals hadn't thought beyond the existing solutions and generated fresh ideas.
So, how does this relate to qualitative data analysis?
Recently, we have witnessed the integration of generative AI in various tools, including software designed to support qualitative data analysis. In my previous blog post, I extensively discussed two of these tools, and you can find related videos on my YouTube channel.
Generative AI has been implemented in three primary ways:
- Summarizing coded data.
- Summarizing categorized data and enabling users to ask the built-in AI assistant questions about the data.
- Coding qualitative text data, with the promise of replacing time-consuming manual coding.
Earlier this year, I wrote about the use of machine learning tools for coding in qualitative research and explored the potential usefulness of Chat-GPT for qualitative researchers. My conclusion was that machine learning-generated codes based on noun phrases or NLP learning could not fully replace human coding in qualitative research. For more details, you can refer to the post.
However, I did find Chat-GPT useful for summarizing coded data. At that time, this feature was not integrated into any tools, so I conducted my own tests by copying and pasting content between applications.
Subsequently, some tools integrated this feature, likely having already been in the process of implementation while I was conducting my experiments in a rudimentary manner.
Then, ATLAS.ti released an update announcing the integration of Open-AI for coding, claiming a potential 90% reduction in overall analysis time. I tested this feature and was disappointed with the results, as detailed in this post. Alternatively, you can watch a video summarizing my findings.
My pursuit of this subject is primarily academic. Throughout my professional career, I have been dedicated to advancing qualitative data analysis through computing and writing about it. Now, I believe we are on the cusp of a new era, prompting the need to reconsider qualitative data analysis methodologies.
Promising a faster horse wouldn't have propelled us into the future 115 years ago, just as the utilization of generative AI for coding does today.
Envisioning the Future
Recently, I was approached by researchers from MIT who are developing a new AI-based tool for qualitative data analysis called AILYZE. With their technical background, they sought to understand how social scientists work with qualitative data and came across my YouTube videos, leading to our connection.
As I began experimenting with the tool, albeit in its rudimentary stage, I could already glimpse the future.
Given these new possibilities, is it not time to thoroughly reconsider qualitative data analysis?
Do we still need to rely solely on coding data? While this approach has served us well over the past 30 years, advancing from index cards and stacks of papers, there may be room for more sophisticated approaches, such as those implemented in Cody. Developed at KIT (Karlsruhe Institute of Technology) several years ago, Cody currently lacks generative AI capabilities but offers AI-assisted coding by emulating human coding. Human coders implement coding rules, and based on these rules, the machine learning system learns and suggests coding for similar data segments. Human coders can continuously refine the rules, and the machine learning system improves accordingly.
If you're interested in delving deeper into AI-assisted analysis, I will be conducting a free seminar on June 7, 2023, at 12 pm CEST in German and 9 am EDT in English. This seminar is part of the monthly seminar series organized by the Qualitative Research Community ALL THINKS QUALITATIVE. You can join here.
For those interested in trying out these tools and engaging in a constructive discussion about the quality of the outcomes, their impact on the analysis process, qualitative methods, ethical implications, and more, I invite you to join my two-day workshop on September 12-13, 2023.