How do you recognize good qualitative data? One of the main challenges newcomers to qualitative research face is the fact that everything can be used as data. I liked this book a lot because it provides some simple but effective answers to the question above. It is a short read and it comes from authors who know what they are talking about.
The book Qualitative Literacy A Guide to Evaluating Ethnographic and Interview Research was written by Mario Luis Small (Harvard University) and Jessica McCrory Calarco (University of Wisconsin-Madison) and was published by University of California Press in 2022. I read an eBook version (230 pages, ISBN: 9780520390676).
The authors’ main argument is that there is a lack of clarity about how quality is defined for qualitative data. The authors claim that we need conceptual tools to enable us to unambiguously say the differences in quality between interview and observation data from one study to another, even when both studies report on an equal amount of data. For this reason, qualitative researchers sometimes do a less than optimal job of data collection and reporting and often use quality measures adopted from quantitative data –e.g., increasing the number of interviews or looking for generic patterns in data. In particular, qualitative data depend heavily on and are created by the researcher, which makes it challenging to have well-established standards for what is mainly a craft.
The authors set out to answer the question: How do we tell good from poor qualitative data? If we were to read a paper with loads of interview and observation data, would it mean that we were reading a methodologically strong paper? Or is there something about those data themselves that tells whether the study is rigorous? Having a non-sociology technology background, these are questions that arise every time I read a qualitative paper. Therefore, the promise of this book resonated well with me. I have peer reviewed papers where the authors claim to have done 50 or more interviews, while I have felt they did not do a good job of reporting those data. This book will help me with my next peer review.
The authors have organized the book around a few principles and five quality indicators (my term for them). For instance, qualitative researchers should strive to increase their exposure to the phenomena they study, they should avoid abstract data and collect palpable data, they should try to demonstrate diversity instead of patterns, and they should fill holes in their data by following up, i.e., by going back to their case and collecting missing data. There are also other indicators in addition to these. The chapters in the book discuss these quality indicators one by one through examples.
In the book’s introduction the authors say the book is mainly for peer reviewers. I also found it relevant as teaching material for my methods courses and master and PhD supervision. In fact, I gave a short presentation of the quality indicators to our research group after I read the book, trying to challenge our PhD students to think about their own data collection activities in terms of these indicators.
The book fills a gap in methods literature by focusing on qualitative data quality and developing an explicit, concise, and applicable quality framework. The authors have a strong background in the field and have written numerous papers and books about this topic.
Early in the book, the authors make it very clear what this book is not about. It is not about qualitative analysis such as coding, nor is it about access to the case. It is about how we can tell good from poor empirical qualitative data. The book mainly focuses on data collected through interviews and ethnographic observations. However, the general indicators that are provided work well for most qualitative data. For instance, one can employ the principle of exposure to the phenomenon by spending sufficient time on reading and analyzing documents, news articles, and literature to get a more profound and nuanced understanding of the phenomenon.
In my view, the book lacks is a discussion of where the quality indicators come from and what alternatives exist. The introduction does an excellent job of motivating the need for such a book but remains generic and does not discuss related research about qualitative data quality. It is, therefore, difficult to see the novel contribution of the book without trusting the authors that there is one.
For instance, while it is discussed explicitly in the book, I would argue that the book is more inline with the interpretative research paradigm. In particular, several indicators are directly related to interpretivism, such as heterogeneity, empathy, and self-awareness. While I do not mind this inclination, the authors don’t say whether it was intentional, i.e., whether they intentionally left out indicators for, e.g., positivist paradigms, or whether it is done without self-reflection –the authors seem to have an ethnography background –and other quality indicators that were not relevant for their perspective were left out unintentionally.
A strength of the book is that it is concise and well-written. The chapters follow the same structure and develop real-world examples. Each chapter develops one example of low-quality data through several steps of making them high-quality, providing excellent illustration of what each indicator is about. The examples are realistic and often based on real-world studies. Each chapter also concludes with a well-known study that applies the indicator in an ideal way. The book is pedagogically well-written and can be used as teaching material in a course.
Overall this is a good book on this narrow but important –and often neglected –topic of quality in qualitative data. I recommend it as a first introduction to the topic. The book achieves its goal of defining what quality is in qualitative data. At the same time, I believe the authors could have done a better job mapping the territory by reporting on other research and how they arrived at the set of indicators they present in the book.
What do others say?
- Iddo Tavory has written an excellent review of this book for the Sociological Methods & Research Journal. One of his main concerns is whether we can standardize the craft of qualitative data creation. The “tension between literacy and a beginners’ orienting manual” that the book creates sometimes glosses over nuances that may not be important to beginners but that constitute central conversations in building a standard literacy in the field (as an example, Tavory uses the terms variation and heterogeneity introduced by the authors). In other words, “the question of internal versus external standards haunts the project as a whole.” Tavory claims that this should not be seen as methods book but “a better way to understand this book is as a Rosetta Stone of sorts, a work of simplified translation.” He concludes by announcing the book somehow as a necessary evil: “At the end of the day, we [social scientists] don’t have the luxury of ignoring the question of standards.”
- There is a review of this book Sarah Damaske in Social forces. It is behind a paywall, so I have not read it. The extract is full of praise.