The acceptance of empirical studies in software engineering and their contributions to increasing knowledge is continuously growing. The analytical research paradigm is not sufficient for investigating complex real life issues, involving humans and their interactions with technology. However, the overall share of empirical studies is negligibly small in computer science research; Sjøberg et al. (2005), found 103 experiments in 5,453 articles Ramesh et al. (2004) and identified less than 2% experiments with human subjects, and only 0.16% field studies among 628 articles. Further, existing work on empirical research methodology in software engineering has a strong focus on experimental research; the earliest by Moher and Schneider (1981), Basili et al. (1986), the first methodology handbook by Wohlin et al. (2000), and promoted by Tichy (1998). All have a tendency towards quantitative approaches, although also qualitative approaches are discussed during the later years, e.g. by Seaman (1999). There exist guidelines for experiments’ conduct (Kitchenham et al. 2002; Wohlin et al. 2000) and reporting (Jedlitschka and Pfahl 2005), measurements (Basili and Weiss 1984; Fenton and Pfleeger 1996; van Solingen and Berghout 1999), and systematic reviews (Kitchenham 2007), while only little is written on case studies in software engineering (Höst and Runeson 2007; Kitchenham et al. 1995; Wohlin et al. 2003) and qualitative methods (Dittrich 2007; Seaman 1999; Sim et al. 2001). Recently, a comprehensive view of empirical research issues for software engineering has been presented, edited by Shull et al. (2008).
The term “case study” appears every now and then in the title of software engineering research papers. However, the presented studies range from very ambitious and well organized studies in the field, to small toy examples that claim to be case studies. Additionally, there are different taxonomies used to classify research. The term case study is used in parallel with terms like field study and observational study, each focusing on a particular aspect of the research methodology. For example, Lethbridge et al. use field studies as the most general term (Lethbridge et al. 2005), while Easterbrook et al. (2008) call case studies one of five “classes of research methods”. Zelkowitz and Wallace propose a terminology that is somewhat different from what is used in other fields, and categorize project monitoring, case study and field study as observational methods (Zelkowitz and Wallace 1998). This plethora of terms causes confusion and problems when trying to aggregate multiple empirical studies.
The case study methodology is well suited for many kinds of software engineering research, as the objects of study are contemporary phenomena, which are hard to study in isolation. Case studies do not generate the same results on e.g. causal relationships as controlled experiments do, but they provide deeper understanding of the phenomena under study. As they are different from analytical and controlled empirical studies, case studies have been criticized for being of less value, impossible to generalize from, being biased by researchers etc. This critique can be met by applying proper research methodology practices as well as reconsidering that knowledge is more than statistical significance (Flyvbjerg 2007; Lee 1989). However, the research community has to learn more about the case study methodology in order to review and judge it properly.
Case study methodology handbooks are superfluously available in e.g. social sciences (Robson 2002; Stake 1995; Yin 2003) which literature also has been used in software engineering. In the field of information systems (IS) research, the case study methodology is also much more mature than in software engineering. For example, Benbasat et al. provide a brief overview of case study research in information systems (Benbasat et al. 1987), Lee analyzes case studies from a positivistic perspective (Lee 1989) and Klein and Myers do the same from an interpretive perspective (Klein and Myers 1999).
It is relevant to raise the question: what is specific for software engineering that motivates specialized research methodology? In addition to the specifics of the examples, the characteristics of software engineering objects of study are different from social science and also to some extent from information systems. The study objects are 1) private corporations or units of public agencies developing software rather than public agencies or private corporations using software systems; 2) project oriented rather than line or function oriented; and 3) the studied work is advanced engineering work conducted by highly educated people rather than routine work. Additionally, the software engineering research community has a pragmatic and result-oriented view on research methodology, rather than a philosophical stand, as noticed by Seaman (1999).
The purpose of this paper is to provide guidance for the researcher conducting case studies, for reviewers of case study manuscripts and for readers of case study papers. It is synthesized from general methodology handbooks, mainly from the social science field, as well as literature from the information systems field, and adapted to software engineering needs. Existing literature on software engineering case studies is of course included as well. The underlying analysis is done by structuring the information according to a general case study research process (presented in Section 2.4). Where different recommendations or terms appear, the ones considered most suited for the software engineering domain are selected, based on the authors’ experience on conducting case studies and reading case study reports. Links to data sources are given by regular references. Specifically, checklists for researchers and readers are derived through a systematic analysis of existing checklists (Höst and Runeson 2007), and later evaluated by PhD students as well as by members of the International Software Engineering Research Network and updated accordingly.
This paper does not provide absolute statements for what is considered a “good” case study in software engineering. Rather it focuses on a set of issues that all contribute to the quality of the research. The minimum requirement for each issue must be judged in its context, and will most probably evolve over time. This is similar to the principles by Klein and Myers for IS case studies (Klein and Myers 1999), “it is incumbent upon authors, reviewers, and exercise their judgment and discretion in deciding whether, how and which of the principles should be applied”. We do neither assess the current status of case study research in software engineering. This is worth a study on its own, similar to the systematic review on experiments by Sjøberg et al. (2005). Further, examples are used both to illustrate good practices and lack thereof.
This paper is outlined as follows. We first define a set of terms in the field of empirical research, which we use throughout the paper (Section 2.1), set case study research into the context of other research methodologies (Section 2.2) and discuss the motivations for software engineering case studies (Section 2.3). We define a case study research process (Section 2.4) and terminology (Section 2.5), which are used for the rest of the paper. Section 3 discusses the design of a case study and planning for data collection. Section 4 describes the process of data collection. In Section 5 issues on data analysis are treated, and reporting is discussed in Section 6. Section 7 discusses reading and reviewing case study report, and Section 8 summarizes the paper. Checklists for conducting and reading case study research are linked to each step in the case study process, and summarized in the Appendix.
The purpose of a paper in the social sciences designed around a case study is to thoroughly investigate a subject of analysis in order to reveal a new understanding about the research problem and, in so doing, contributing new knowledge to what is already known from previous studies. In general, the structure of a case study research paper is not all that different from a standard college-level research paper. However, there are subtle differences you should be aware of. Here are the key elements to organizing and writing a case study research paper.
As with any research paper, your introduction should serve as a roadmap for your readers to ascertain the scope and purpose of your study. The introduction to a case study research paper, however, should not only describe the research problem and its significance, but you should also succinctly describe why the case is being used and how it relates to addressing the problem. The two elements should be linked. With this in mind, a good introduction answers these four questions:
- What was I studying? Describe the research problem and describe the subject of analysis you have chosen to address the problem. Explain how they are linked and what elements of the case will help to expand knowledge and understanding about the problem.
- Why was this topic important to investigate? Describe the significance of the research problem and state why a case study design and the subject of analysis that the paper is designed around is appropriate in addressing the problem.
- What did we know about this topic before I did this study? Provide background that helps lead the reader into the more in-depth literature review to follow. If applicable, summarize prior case study research applied to the research problem and why it fails to adequately address the research problem. Describe why your case will be useful. If no prior case studies have been used to address the research problem, explain why you have selected this subject of analysis.
- How will this study advance new knowledge or new ways of understanding? Explain why your case study will be suitable in helping to expand knowledge and understanding about the research problem.
Each of these questions should be addressed in no more than a few paragraphs. Exceptions to this can be when you are addressing a complex research problem or subject of analysis that requires more in-depth background information.
II. Literature Review
The literature review for a case study research paper is generally structured the same as it is for any college-level research paper. The difference, however, is that the literature review is focused on providing background information and enabling historical interpretation of the subject of analysis in relation to the research problem the case is intended to address. This includes synthesizing studies that help to:
- Place relevant works in the context of their contribution to understanding the case study being investigated. This would include summarizing studies that have used a similar subject of analysis to investigate the research problem. If there is literature using the same or a very similar case to study, you need to explain why duplicating past research is important [e.g., conditions have changed; prior studies were conducted long ago, etc.].
- Describe the relationship each work has to the others under consideration that informs the reader why this case is applicable. Your literature review should include a description of any works that support using the case to study the research problem and the underlying research questions.
- Identify new ways to interpret prior research using the case study. If applicable, review any research that has examined the research problem using a different research design. Explain how your case study design may reveal new knowledge or a new perspective or that can redirect research in an important new direction.
- Resolve conflicts amongst seemingly contradictory previous studies. This refers to synthesizing any literature that points to unresolved issues of concern about the research problem and describing how the subject of analysis that forms the case study can help resolve these existing contradictions.
- Point the way in fulfilling a need for additional research. Your review should examine any literature that lays a foundation for understanding why your case study design and the subject of analysis around which you have designed your study may reveal a new way of approaching the research problem or offer a perspective that points to the need for additional research.
- Expose any gaps that exist in the literature that the case study could help to fill. Summarize any literature that not only shows how your subject of analysis contributes to understanding the research problem, but how your case contributes to a new way of understanding the problem that prior research has failed to do.
- Locate your own research within the context of existing literature [very important!]. Collectively, your literature review should always place your case study within the larger domain of prior research about the problem. The overarching purpose of reviewing pertinent literature in a case study paper is to demonstrate that you have thoroughly identified and synthesized prior studies in the context of explaining the relevance of the case in addressing the research problem.
In this section, you explain why you selected a particular subject of analysis to study and the strategy you used to identify and ultimately decide that your case was appropriate in addressing the research problem. The way you describe the methods used varies depending on the type of subject of analysis that frames your case study.
If your subject of analysis is an incident or event. In the social and behavioral sciences, the event or incident that represents the case to be studied is usually bounded by time and place, with a clear beginning and end and with an identifiable location or position relative to its surroundings. The subject of analysis can be of a rare or critical event or focus on a typical or regular event. The purpose of studying a rare event is to illuminate new ways of thinking about the broader research problem or to test a hypothesis. Critical incident case studies must describe the method by which you identified the event and explain the process by which you determined the validity of this case to inform broader perspectives about the research problem or to reveal new findings. However, the event does not have to be a rare or uniquely significant to support new thinking about the research problem or to challenge an existing hypothesis. For example, Walo, Bull, and Breen conducted a case study to identify and evaluate the direct and indirect economic benefits and costs of a local sports event in the City of Lismore, New South Wales, Australia. The purpose of their study was to provide new insights from measuring the impact of a typical local sports event that prior studies could not measure well because they focused on large "mega-events." Whether the event is rare or not, the methods section should include an explanation of the following characteristics of the event: when did it take place; what were the underlying circumstances leading to the event; what were the consequences of the event
If your subject of analysis is a person. Explain why you selected this particular individual to be studied and describe what experience he or she has had that provides an opportunity to advance new understandings about the research problem. Mention any background about this person which might help the reader understand the significance of his/her experiences that make them worthy of study. This includes describing the relationships this person has had with other people, institutions, and/or events that support using him or her as the subject for a case study research paper. It is particularly important to differentiate the person as the subject of analysis from others and to succinctly explain how the person relates to examining the research problem.
If your subject of analysis is a place. In general, a case study that investigates a place suggests a subject of analysis that is unique or special in some way and that this uniqueness can be used to build new understanding or knowledge about the research problem. A case study of a place must not only describe its various attributes relevant to the research problem [e.g., physical, social, cultural, economic, political, etc.], but you must state the method by which you determined that this place will illuminate new understandings about the research problem. It is also important to articulate why a particular place as the case for study is being used if similar places also exist [i.e., if you are studying patterns of homeless encampments of veterans in open spaces, why study Echo Park in Los Angeles rather than Griffith Park?] and, if applicable, what type of human activity involving this place makes it a good choice to study [prior research reveals Echo Park has more homeless veterans].
If your subject of analysis is a phenomenon. A phenomenon refers to a fact, occurrence, or circumstance that can be studied or observed but with the cause or explanation to be in question. In this sense, a phenomenon that forms your subject of analysis can encompass anything that can be observed or presumed to exist but is not fully understood. In the social and behavioral sciences, the case usually focuses on human interaction within a complex physical, social, economic, cultural, or political system. For example, the phenomenon could be the observation that many vehicles used by ISIS fighters are small trucks with English language advertisements on them. The research problem could be that ISIS fighters are difficult to combat because they are highly mobile. The research questions could be how and by what means are these vehicles used by ISIS being supplied to the militants and how might supply lines to these vehicles be cut? How might knowing the suppliers of these trucks from overseas reveal larger networks of collaborators and financial support? A case study of a phenomenon most often encompasses an in-depth analysis of a cause and effect that is grounded in an interactive relationship between people and their environment in some way.
NOTE: Evidence that supports the method by which you identified and chose your subject of analysis should be linked to the findings from the literature review. Be sure to cite any prior studies that helped you determine that the case you chose was appropriate for investigating the research problem.
The main elements of your discussion section are generally the same as any research paper, but centered around interpreting and drawing conclusions about the key findings from your case study. Note that a general social sciences research paper may contain a separate section to report findings. However, in a paper designed around a case study, it is more common to combine a description of the findings with the discussion about their implications. The objectives of your discussion section should include the following:
Reiterate the Research Problem/State the Major Findings
Briefly reiterate the research problem you are investigating and explain why the subject of analysis around which you designed the case study were used. You should then describe the findings revealed from your study of the case using direct, declarative, and succinct proclamation of the study results. Highlight any findings that were unexpected or especially profound.
Explain the Meaning of the Findings and Why They are Important
Systematically explain the meaning of your case study findings and why you believe they are important. Begin this part of the section by repeating what you consider to be your most important or surprising finding first, then systematically review each finding. Be sure to thoroughly extrapolate what your analysis of the case can tell the reader about situations or conditions beyond the actual case that was studied while, at the same time, being careful not to misconstrue or conflate a finding that undermines the external validity of your conclusions.
Relate the Findings to Similar Studies
No study in the social sciences is so novel or possesses such a restricted focus that it has absolutely no relation to previously published research. The discussion section should relate your case study results to those found in other studies, particularly if questions raised from prior studies served as the motivation for choosing your subject of analysis. This is important because comparing and contrasting the findings of other studies helps to support the overall importance of your results and it highlights how and in what ways your case study design and the subject of analysis differs from prior research about the topic.
Consider Alternative Explanations of the Findings
It is important to remember that the purpose of social science research is to discover and not to prove. When writing the discussion section, you should carefully consider all possible explanations for the case study results, rather than just those that fit your hypothesis or prior assumptions and biases. Be alert to what the in-depth analysis of the case may reveal about the research problem, including offering a contrarian perspective to what scholars have stated in prior research.
Acknowledge the Study's Limitations
You can state the study's limitations in the conclusion section of your paper but describing the limitations of your subject of analysis in the discussion section provides an opportunity to identify the limitations and explain why they are not significant. This part of the discussion section should also note any unanswered questions or issues your case study could not address. More detailed information about how to document any limitations to your research can be found here.
Suggest Areas for Further Research
Although your case study may offer important insights about the research problem, there are likely additional questions related to the problem that remain unanswered or findings that unexpectedly revealed themselves as a result of your in-depth analysis of the case. Be sure that the recommendations for further research are linked to the research problem and that you explain why your recommendations are valid in other contexts and based on the original assumptions of your study.
As with any research paper, you should summarize your conclusion in clear, simple language; emphasize how the findings from your case study differs from or supports prior research and why. Do not simply reiterate the discussion section. Provide a synthesis of key findings presented in the paper to show how these converge to address the research problem. If you haven't already done so in the discussion section, be sure to document the limitations of your case study and needs for further research.
The function of your paper's conclusion is to: 1) restate the main argument supported by the findings from the analysis of your case; 2) clearly state the context, background, and necessity of pursuing the research problem using a case study design in relation to an issue, controversy, or a gap found from reviewing the literature; and, 3) provide a place for you to persuasively and succinctly restate the significance of your research problem, given that the reader has now been presented with in-depth information about the topic.
Consider the following points to help ensure your conclusion is appropriate:
- If the argument or purpose of your paper is complex, you may need to summarize these points for your reader.
- If prior to your conclusion, you have not yet explained the significance of your findings or if you are proceeding inductively, use the conclusion of your paper to describe your main points and explain their significance.
- Move from a detailed to a general level of consideration of the case study's findings that returns the topic to the context provided by the introduction or within a new context that emerges from your case study findings.
Note that, depending on the discipline you are writing in and your professor's preferences, the concluding paragraph may contain your final reflections on the evidence presented applied to practice or on the essay's central research problem. However, the nature of being introspective about the subject of analysis you have investigated will depend on whether you are explicitly asked to express your observations in this way.
Problems to Avoid
One of the goals of a case study is to lay a foundation for understanding broader trends and issues applied to similar circumstances. However, be careful when drawing conclusions from your case study. They must be evidence-based and grounded in the results of the study; otherwise, it is merely speculation. Looking at a prior example, it would be incorrect to state that a factor in improving girls access to education in Azerbaijan and the policy implications this may have for improving access in other Muslim nations is due to girls access to social media if there is no documentary evidence from your case study to indicate this. There may be anecdotal evidence that retention rates were better for girls who were on social media, but this observation would only point to the need for further research and would not be a definitive finding if this was not a part of your original research agenda.
Failure to Document Limitations
No case is going to reveal all that needs to be understood about a research problem. Therefore, just as you have to clearly state the limitations of a general research study, you must describe the specific limitations inherent in the subject of analysis. For example, the case of studying how women conceptualize the need for water conservation in a village in Uganda could have limited application in other cultural contexts or in areas where fresh water from rivers or lakes is plentiful and, therefore, conservation is understood differently than preserving access to a scarce resource.
Failure to Extrapolate All Possible Implications
Just as you don't want to over-generalize from your case study findings, you also have to be thorough in the consideration of all possible outcomes or recommendations derived from your findings. If you do not, your reader may question the validity of your entire analysis, particularly if you failed to document an obvious outcome from your case study research. For example, in the case of studying the accident at the railroad crossing to evaluate where and what types of warning signals should be located, you failed to take into consideration speed limit signage as well as warning signals. When designing your case study, be sure you have thoroughly addressed all aspects of the problem and do not leave gaps in your analysis.
Case Studies. Writing@CSU. Colorado State University; Gerring, John. Case Study Research: Principles and Practices. New York: Cambridge University Press, 2007; Merriam, Sharan B. Qualitative Research and Case Study Applications in Education. Rev. ed. San Francisco, CA: Jossey-Bass, 1998; Mills, Albert J., Gabrielle Durepos, and Eiden Wiebe, editors. Encyclopedia of Case Study Research. Thousand Oaks, CA: SAGE Publications, 2010; Putney, LeAnn Grogan. "Case Study." In Encyclopedia of Research Design, Neil J. Salkind, editor. (Thousand Oaks, CA: SAGE Publications, 2010), pp. 116-120; Simons, Helen. Case Study Research in Practice. London: SAGE Publications, 2009; Kratochwill, Thomas R. and Joel R. Levin, editors.Single-Case Research Design and Analysis: New Development for Psychology and Education. Hilldsale, NJ: Lawrence Erlbaum Associates, 1992; Swanborn, Peter G. Case Study Research: What, Why and How? London : SAGE, 2010; Yin, Robert K. Case Study Research: Design and Methods. 6th edition. Los Angeles, CA, SAGE Publications, 2014; Walo, Maree, Adrian Bull, and Helen Breen. “Achieving Economic Benefits at Local Events: A Case Study of a Local Sports Event.” Festival Management and Event Tourism 4 (1996): 95-106.