Old and New Methods in Police Research¹

Introduction

Police research is an expanding field. Høigaard (2009) found 215 publications in her review of Scandinavian² police research between 1990 and 2005, and Valland (2011) found 172 publications in the following six-year period. In a bibliography from 2010, Knutsson (2010) covers government affiliated police research in Sweden, and counts almost 80 studies, 21 of them from the period 2000–2008. Subsequent research has not been formally reviewed, but there are no signs of slowing in the pace of research in this area.³ In this paper, I discuss the methods in Scandinavian police research since the year 2000 and suggest new methods that may improve future police research and address new questions about the police. Examples of new methods are given, especially from studies of other professions in which the questions and challenges often have clear parallels to police research.

The choice between different research methods and data depends, of course, on the research question we ask. Qualitative methods will often be the best choice when you are interested in “how” questions rather than “how many” questions, and for exploring the construction of narratives (Silverman, 2013). The purpose of qualitative research is often to analyse meanings and interpretation, trying to understand processes and see reality from the point of view of the people who are studied (Järvinen & Mik-Meyer, 2017, p. 10).

Both previous and future police research will imply many questions where the new methods and data discussed in this paper, cannot provide answers. To use recent examples, both Nordberg (2018) and Haake, Rantatalo, and Lindberg (2017) describe gendered norms among police leaders, norms that would have been hard to grasp using quantitative methods. However, the methods and data suggested in this paper can be suitable when the goal is to increase the likelihood of being able to make causal claims and/or produce results that are externally valid (being transferable to a wider population and other settings). Both qualitative research and traditional surveys have limitations when it comes to detecting causal effects and providing external validity. Possibilities for making causal claims and external validity are increasingly asked for across the social sciences (Tufte, 2013). Although this by no means will or should be the goal of all future police research, increased possibilities for studies of causal mechanisms and studies with high external validity can strengthen the position of police research.

Current methods in police research

Although a complete review of Scandinavian police research is beyond the scope of this paper, a brief overview of the research field is needed to discuss the methods in police research. The research discussed in this paper is limited to studies on police officers, police organization and police work from Denmark, Sweden and Norway in the period between 2000 and 2018. Since the volume of police research is higher in Sweden and Norway than in Denmark (Holmberg, 2014a; Høigaard, 2009; Valland, 2011), there are more Swedish and Norwegian studies cited in this paper, but several Danish studies are included. Because this paper mainly discusses new possibilities for quantitative studies, more quantitative than qualitative studies will be discussed, although many references to qualitative police research are mentioned. In line with Holmberg (2014a), the aim of this paper is to comment upon some trends in Scandinavian police research, not to provide a complete, systematic review of Scandinavian police research. Of course, this implies that some studies will be left out, and limits the possibilities for drawing firm conclusions on the nature of all published Scandinavian police research. However, given that the aim is to comment upon some of the methods widely used and to suggest new methods, the goal has been to ensure a wide representation of studies providing examples of methods being used in recent studies in this research field.

Many of the most influential works in Scandinavian police research are based on qualitative methods. The list includes, but are by no means restricted to, much cited works by Holgersson and Knutsson (2011), Finstad (2000), Granér (2004) Holmberg (2002),⁴ Balvig (2005), Larsson (2010) and K. F. Aas and Gundhus (2015).

Among the highly cited police researchers we also find examples of quantitative police research, such as Strömwall and Granhag (2003) and Glomseth, Gottschalk, and Solli-Sæther (2007). Other examples of quantitative police research can be found in the quite extensive body of work based on police student surveys (including, but not restricted to Annell, Lindfors, & Sverke, 2015; Bäck, 2015; Fekjær, 2014; Lagestad, 2013; Petersson, 2015), surveys of police officers (e.g. Ask, 2010; Burke & Mikkelsen, 2006; Dean, Fahsing, & Gottschalk, 2006; Ekenvall, 2003; Hansson, Hurtig, Lauritz, & Padyab, 2017; Hellesø-Knutsen, 2013) and some studies based on population surveys (Holmberg, 2005; Thomassen, 2017; Thomassen, Strype, & Egge, 2014). Among the quantitative studies that do not rely on survey methods, we find some studies that use natural variation designs/field experiments (Fahsing & Ask, 2016; Holmberg, 2005; Jørgensen, 2010; Pedersen, 2013; Phelps, Strype, Le Bellu, Lahlou, & Aandal, 2016) and some that use register data (J. Y. Dahl & Lomell, 2016; Kruize, 2001; Lindström, 2011; Shannon, 2008; Skardhamar, Fekjær, & Pedersen, 2016; Winnæss & Helland, 2014).

In the following sections, I review three of the challenges associated with the methods that are typically used in police research before discussing possible new methods.

Challenges of traditional methods

In this paper, three challenges associated with the often-used methods in Scandinavian police research will be discussed: the Hawthorne effect, a lack of representativeness and limited possibilities for making causal claims. These challenges are neither mutually exclusive nor omnipresent. All of them may be present in the same study, or none of them, and they may be present to varying degrees. For example, the Hawthorne effect may influence the findings to a limited degree in some studies, whereas such observer effects may completely compromise the results in other studies.

The Hawthorne Effect

The Hawthorne effect refers to situations in which research participants are influenced by knowing that they are being studied (McCambridge, Witton, & Elbourne, 2014). The traditional example of this stems from research at the Hawthorne Works, a factory where researchers discovered that workers were more productive when working conditions (such as lighting) changed, regardless of how they changed (increased or decreased). Of course, the reason for the increased productivity was that the workers put in extra effort when the researchers were present, and hence all new changes in working conditions seemed to work. This type of observer effect may be highly relevant in police research. Do police officers behave differently with a researcher in the back seat of the patrol car? Finstad (2000) reports that all officers in her study answered “no” to this question and claimed that the presence of a researcher did not alter their behaviour. However, as noted by both Finstad (2000) and McCambridge et al. (2014), it seems highly unlike that the presence of a researcher does not affect the behaviour and answers of police officers. This is of special concern when the research focuses on sensitive subjects such as police violence or ethnic discrimination (Sollund, 2007).⁵ The Hawthorne effect may decrease if the persons under study gradually get used to the presence of the researcher and “forget” that they are being observed. Still, it is quite likely that the informants wish to present a certain picture to the public, and hence will downplay some aspects and highlight others (Finstad, 2000).

The risk of considerable observer effects is potentially heightened when the researcher is in the dual role as police researcher and police educator, which is the case in several of the Scandinavian police studies (e.g. Granér, 2004; Hoel & Christensen, 2016; Lander, 2013; Winnæss & Helland, 2014). When respondents regard the researcher as a teacher, there may be an “interviewer effect”: the interviewer’s observable characteristics may suggest to the respondents which answers the interviewer considers desirable (Krosnick & Presser, 2010). For example, the respondents may downplay strong negative opinions about police education and their attitudes towards non-legalistic police methods when the interviewer is seen as a representative of police education.

The Hawthorne effect is also a challenge in survey research. There is reason to believe that respondents may change their answers when they know they are being studied, especially with respect to topics and situations for which some answers are seen as more “correct” than others. In the methodological literature, a social desirability bias occurs when respondents give a more favourable image of themselves because they want to make a good impression on others (Krosnick & Presser, 2010). This is a particular concern in studies of aspiring police recruits, who are well aware that certain answers will sound better in the application interview. Hence, surveys conducted during the recruitment process to measure the attitudes and values of those who did not make the cut for the police academy are probably of limited interest. Surveys distributed to new recruits shortly after admission may also be prone to the social desirability bias on subjects such as political attitudes and reasons for choosing the police profession (Larsson, Strype, & Thomassen, 2006), attitudes towards non-legalistic police methods (Fekjær & Petersson, 2018) and opinions about female police officers (Bloksgaard, Fekjær, & Møberg, 2018). However, finding small changes in attitudes between the time when recruits enter the police academy and when they graduate (Bloksgaard et al., 2018; Fekjær & Petersson, 2018) might indicate that the initial answers were not too influenced by the recruitment process. These studies generally offer anonymity, which reduces social pressure and hence the social desirability bias. Nevertheless, investigating other measures to reduce the social desirability bias would be a welcome contribution to police research (see Krosnick & Presser 2010: 285–287 for suggestions on how to detect and reduce social desirability bias).

Lack of Representativeness

Lack of representativeness is a well-known challenge when using qualitative methods. How do we know whether the relatively small group of people that we have observed or interviewed represents the rule or the exception? The importance of representativeness depends upon the research question. Representativeness is important if we want to describe how widespread a phenomenon or attitude is, or differences between groups. It is less important if the focus of our research is to provide in-depth descriptions and suggest possible mechanisms.

How and if we can generalize based upon qualitative data is an ongoing debate (Corbin & Strauss, 1990; Katz, 2015; King, Keohane, & Verba, 1994), and a full account of the nature of and solutions to this problem is beyond the scope of this paper. However, we do note that an interesting angle to this question is to consider alternative understandings of generalization, different from the traditional understanding of representativeness in quantitative studies. Tjora (2017) discusses three forms of generalization in qualitative studies. Naturalistic generalization means that the readers themselves should be the judges of the studies’ validity beyond the specific case, while moderate generalization implies that the researcher is responsible for describing in which cases the results will be valid. Tjora himself defends the third alternative, conceptual generalization, where the goal of qualitative studies is to develop concepts, typologies and theories which can be valid beyond the concrete case studied.

Nevertheless, lack of representativeness still can leave us with unsolved questions when reading existing research. For example, Egge, Ganapathy, and Runhovde (2008) and Lander (2013) provide compelling stories about the experience of being a police student who does not represent the white, heterosexual, male majority, but they cannot tell us how widespread different attitudes are and whether these experiences are shared by all minority police students. If the goal is to answer questions like these, we need to use other research methods than qualitative interviews.

Although a lack of representativeness is generally a critique of qualitative methods, falling response rates have increasingly threatened the representativeness of quantitative studies. Declining interest in answering surveys is a problem in all research areas, and it has accelerated rapidly in recent decades (Johnson & Wislar, 2012). Several factors determine whether people will answer a survey, but one important factor in police studies is the target audience. Surveys of police students still have high response rates in all the Scandinavian countries (>85%), although the Swedish police students seem to be the most inclined to answer surveys (Bloksgaard et al., 2018). Surveys of working police officers are more of a challenge, with response rates at, e.g. 27% (Wathne, 2015, survey 2), 34% (Dean et al., 2006; Hansson, Ghazinour, & Padyab, 2017) and 43% (Hellesø-Knutsen, 2013). The highest response rates among police officers seem to be obtained by actively tracking up individuals or police districts to get the respondents to participate and/or offer rewards for participation (Petersson, 2015, 58 %, Wathne, 2015 survey 1, 83 %), whereas, unsurprisingly, some of the lowest response rates are found in studies that ask working police officers to do physical tests (Lagestad, Jenssen, & Dillern, 2014, 28 %). Despite these numbers, police officers are generally more willing to answer surveys than are members of the general population. The lowest response rates in police research are found in studies of the general population that measure such attitudes as trust in the police. These studies report response rates of e.g. 12 % (G. Aas, Runhovde, Strype, & Bjørgo, 2010) and 10 % (Thomassen, 2017) of the original sample.

What are the consequences of falling response rates and what can we do about it? Low response rates are seldom a problem per se; the serious challenges occur if the non-responders differ from those who answer the survey. This will be the case when the causes of participation are correlated with the survey variables. The extent to which nonresponse causes nonresponse bias is contested, and probably varies between surveys. This means that the level of nonresponse is not necessary a trustworthy indicator of the level of biased estimates in the survey (Davern, 2013; Groves & Peytcheva, 2008; Hellevik, 2015; Peytchev, 2013). However, nonresponse makes it hard to prove the representativeness of the sample. Analyses of the non-response may make up for this to some extent and such analyses probably should be standard in most survey research, at least in research areas with low response rates. If you can prove that your sample is representative on variables such as gender, age, rank, etc., it increases the reliability of the study (see Petersson, 2015: 137–143 for an example of how analysis of non-response can be performed in police research).

Although the sample may be representative on certain background characteristics that are easy to measure, it does not ensure that the sample also will be representative on other central features, such as attitudes and values. Hence, falling response rates are not an easy problem to solve. Judging from the low and decreasing response rates among police officers, one solution would be to mainly survey police students, for whom response rates are still quite high, and primarily rely on other methods for information about working police officers and the population in general. This will considerably limit the questions we can ask, but the new response climate may force us to rethink our traditional methods of police research. Other research fields have addressed falling response rates by limiting longitudinal surveys to students and relying on other methods such as register data to examine questions about professionals in the workforce (Abrahamsen, 2015, 2016).

Limited Possibilities for Making Causal Claims

The final disadvantage of current methods is that they offer limited possibilities for making causal claims. Although we may find a clear and statistically significant correlation between two variables, we cannot be certain that there is a causal relationship between the variables. Correlations can also be caused by spurious effects, selection bias, coincidence and reverse causation (Fekjær, 2015). For example, when declining crime rates led many people to believe that the New York Police Department’s so-called “Broken Windows” policy was a success, further scrutiny indicated that other factors that occurred at the same time (e.g. improved economic conditions and abortion policy) may have caused the declining crime rates. This means that statistical correlations between declining crime rates and changing police practices could be more of a coincidence than a causal relationship (Corman & Mocan, 2005; Harcourt & Ludwig, 2006). Similarly, population surveys show that those who have had contact with the police have less trust in the police (Thomassen, 2017). However, it is difficult to judge whether this is a causal effect (that contact with the police has made them more sceptical) or a selection effect (people who are likely to come in contact with the police are also more sceptical initially). Spurious effects are also difficult to rule out with traditional methods. For example, Knutsson and Strype (2003) convincingly showed that there are more shooting incidents in Denmark and Sweden, where the police are routinely armed, compared with Norway, where the police normally have been unarmed. Nevertheless, it is difficult to prove that this is a causal effect (arming the police has led to more shootings) and not a reflection of other differences between these countries. One alternative explanation is that the correlation is spurious, that is, high crime levels have led both to the police being armed and to a high number of shooting incidents.

To a certain extent, alternative statistical techniques that are seldom used in police research could help us determine causality. For example, individual-level fixed effects models or propensity score matching could reduce the bias caused by omitted confounding variables (Angrist & Pischke, 2009). Heggebø and Elstad (2017) convincingly demonstrated how statistical methods affect empirical findings by comparing ordinary least squares regression, individual-level fixed effects models and propensity score matching. Using the widespread method OLS-regression, the unemployed seem to be in worse health than the rest of the population, regardless of setting. However, the use of more sophisticated methods reveals that being unemployed probably has smaller health effects in countries with high unemployment rates. In these countries, the experience of being unemployed is widely shared and probably involves less stigma and self-blame. Heggebø and Elstad (2017) demonstrate how non-standard statistical methods may be needed to uncover complex empirical patterns. However, as we shall see, new research designs will also be necessary if the goal is to make more definitive claims about causality in Scandinavian police research.

New methods

The new methods discussed in this paper are not necessarily “new”, as several of them have been used for a number of years. For example, psychological research has a long tradition of using experiments (Orne, 1962), and vignette studies have been tested in several research fields such as the social sciences, medicine and history (Ejrnæs & Monrad, 2012; Mutz, 2011). However, these methods are new in the sense that they are increasingly being used in many research areas, often sparked by technological development that makes the methods more easily accessible (see, e.g. Mutz, 2011). More importantly, these methods are quite new in Scandinavian police research. As we will see, exceptions exist, especially with respect to natural variation designs and employing register data. However, existing studies have only just started to tap into the opportunities these methods provide, and several suggestions for police research based on these new methods are discussed below.

Natural Variation Designs

Natural variation designs involve comparing a group that has been exposed to an intervention (suspected causal factor) with a control group (which has not been exposed to the intervention). However, the distribution into groups is not arbitrary in natural variation designs (no randomization), and the suspected causal factor is not controlled by the researcher. Quasi-experimental design shares many similarities with natural variation designs, except that the researcher controls the suspected casual factor. The main challenges due to lack of randomization are common to both natural variation designs and quasi-experiments, and will be further discussed below.

Police reforms provide a topic of high interest and heated debates among police researchers, policy makers and police employees alike (see e.g. Degnegaard, 2010; Holmberg, 2014b; Lindström, Pauloff, & Granath, 2001). Changing policies and reforms provide ample opportunities for interesting natural variation designs. Examples of this from other professions include research by E. S. Dahl and Lima (2016), who showed how a social welfare reform that required more activation from the recipients of social welfare seemed to have a strong deterrent effect, and Torsvik and Vaage (2014), who showed how a policy reform in how to report sickness absences reduced sickness absences. Similarly, police reforms, such as those we have seen in recent years in all of the Scandinavian countries, present many possibilities for natural variation designs. How have important issues such as trust in the police, police reports, crime and arrests changed after these reforms? Both survey and register data, analysed by difference-in-difference models (Angrist & Pischke, 2015), for example, can provide interesting answers to these questions after important police reforms.

Ideally, such natural variation designs would include random assignment to control and effect groups. Such situations are rare, but they do occur. For example, cases are distributed randomly to appeal judges, and some judges are more lenient than others. G. B. Dahl, Kostøl, and Mogstad (2014) used this random variation to estimate the causal link between having parents on disability insurance and outcomes for the next generation. Similarly, random assignment of cases to police investigators and police attorneys could provide very interesting opportunities for determining which factors increase the likelihood of a reported case being dropped or leading to a conviction. In addition, one could estimate whether the background characteristics and attitudes of law enforcement officers influence the outcome. For example, do the chances of legal proceedings increase if the investigators handling rape reports are female?

The lack of randomization represents a challenge in most natural variation designs, as discussed further below. This means that natural variation designs would be more interesting if it were possible to control for pre-intervention differences between the control and effect groups, and/or if these groups were quite similar to begin with. One research possibility is to examine reforms and/or new efforts that are introduced in some police districts and not in others. This approach has been used to assess the effect of different closing hours for bars on violence. Cities with extended closing hours experience more violence, and this result remains when violence outside the city centre is used as a proxy for potential cofounding variables (Rossow & Norström, 2012).

Vignette Studies

In vignette studies, respondents are asked to respond to a case story that describes a person or a social situation (Ejrnæs & Monrad, 2012). This design allows researchers to manipulate potential causal factor(s) within the framework of a traditional survey. For example, a researcher who is interested in the effects of ethnicity on attitudes can construct two identical versions of a vignette to describe a person, with the exception that one version identifies the person as a member of an ethnic minority and the other version describes the person as a member of the ethnic majority. The two versions of the vignette are then randomly assigned to respondents to see whether people respond differently to the descriptions that vary only on the ethnicity of the person being described.

Vignette studies offer many opportunities to vary what are believed to be crucial effect variables (e.g. ethnicity, gender, motivation, appearance) and to explore how the background characteristics of respondents affect their answers. For example, one could examine whether some groups of police officers are influenced by the characteristics of the offender more than other groups; for example, are female officers more forgiving or harsher on female offenders than their male colleagues, and do the two groups respond in the same way when the offender is described as male?

The clear advantage of vignette studies is randomization. All respondents have the same chance of receiving the different vignettes. Hence, we can assume (and to a certain extent show) that the groups are similar except for the factor we have altered. This means that claims of causality will be clearly more valid compared with traditional survey research. At best, vignette studies can be said to combine the internal validity of experiments and the external validity of representative population surveys (Mutz, 2011), although the external validity can be questioned (Barabas & Jerit, 2010).

Vignette studies have produced valuable results in research from other professions. For example, Terum, Torsvik and Øverbye (2017) showed that the ethnic background of the claimants has a very limited impact on social workers’ decisions. Similarly, Bye, Horverak, Sandal, Sam, and van de Vijver (2014) found a limited effect of applicants’ ethnic background on hiring decisions, although the applicants’ degree of cultural fit influenced their chances of being hired. In contrast, Jakobsson, Kotsadam, Syse, and Øien (2016) found that gender influences decisions in elderly care; clients who have an adult daughter receive less support than those who have an adult son. Similar methodological designs could be introduced into police research. For example, it would be interesting to do a vignette study based on the qualitative results reported by Holmberg (2003). To what extent do the social characteristics of the offender influence police officers’ judgements about how to handle public drinking; would a tramp be treated differently than an ordinary Joe if the offence were the same?

Field Experiments

An experiment is characterized by the manipulation of a factor that is believed to have a causal effect. In randomized experiments, the subjects are randomly divided in two groups, one of which is exposed to the believed causal factor. One could also divide the subjects into several groups that were exposed to varying conditions. Because randomization ensures that the groups are equal, any differences between the groups can be attributed to the causal factor. Field experiments refer to experiments that take place in real life or “field” settings rather than in a laboratory.

For example, in one field experiment, employers were much less likely to respond positively to a job application with a typical Pakistani name (Saera Ahmed) on it than they were to an identical application with a typical ethnic Norwegian name (Silje Johansen) (Birkelund, Rogstad, Heggebø, Aspøy, & Bjelland, 2014). Field experiments have also shown that job applicants with names that are typical of the social elite are more likely to get a call back from potential employers (Jackson, 2009) and that taxpayers report higher incomes when they are informed that the tax authorities have information about their previous foreign income (Bott, Cappelen, Sørensen, & Tungodden, 2014). Similarly, one could perform field experiments to see whether social characteristics affect the decisions of police officers. Are reports to the police handled differently depending on the ethnicity of the suspected perpetrator, and do the address and status of the person making the report matter? Field experiments can supply valid answers to important questions such as these.

The effect of field training is debated in police research (Fekjær & Petersson, 2018; Moskos, 2008; Sun, 2002); field experiments could thus be used to determine the effects of field training in different environments. For example, police students could be randomly assigned to field training in rural/urban areas with different crime patterns, and male-dominated or more gender-balanced departments, to be supervised by different types of field officers. These types of experiments would provide clearer answers about how different field training conditions affect students. A similar field experiment in a military setting showed how the characteristics of roommates and fellow squad members affect the attitudes of soldiers. Recruits who shared a room with female recruits or recruits with ethnic minority backgrounds became more positively disposed towards female leaders and ethnic minorities (Finseraas, Hanson, Johnsen, Kotsadam, & Torsvik, 2016; Finseraas, Johnsen, Kotsadam, & Torsvik, 2016).

Register Data

In addition to the new methods discussed, another alternative is to use new sources of data – or more precisely, extended use of already existing data sources. Two types of registers in particular hold great potential for police research: police registers and population registers. Police registers include detailed information on crimes, investigations and interrogations. Police registers also provide ample opportunities for text analyses, as they include detailed descriptions of such documents as police reports and interviews (see Bjelland & Dahl, 2017 for a detailed description of research possibilites with police registerdata).

If utilized properly, police registers yield possibilities for exploring, for example, changes in crime rates after police reforms, police effectiveness using different methods, geographical crime patterns, and the prevalence of different methods of investigation and interrogation. To date, the few studies that have used data from police registers in Scandinavian police research cover limited areas (J. Y. Dahl & Lomell, 2016; Kruize, 2001; Lindström, 2011; Shannon, 2008; Skardhamar et al., 2016; Sætre & Grytdal, 2012), and the possibilities for further research are almost countless. Many of these studies could be very interesting for police authorities and public policymakers, as they have the potential to tap into the key question of the effectiveness of different police methods and organizations.

Population registers can also be used in police research, especially for gathering information about police officers and their careers. The population registers that are maintained by state authorities in the Scandinavian countries (Statistics Sweden, Statistics Denmark and Statistics Norway) include detailed information on, for example, socio-economic background, education, wages, careers, fertility, health and sickness absence. In many cases, register data provide more reliable information than do survey data, such as on the social background of police officers (Winnæss & Helland, 2014). Register data are also better when studying small groups. For instance, register data would be a good source of information about police officers with minority backgrounds, as such groups are often too small to be explored in single-cohort surveys. Finally, register data are the most promising data for studying police officers’ careers after graduation. Most surveys of police officers are limited by low and selective response rates, while register data include the whole cohort. An interesting approach that has been tested with other professions is to combine survey data from the worker’s in-school training period with register data (Abrahamsen, 2015, 2016). This presents the opportunity for a range of interesting research questions, including how attitudes, professional commitment and work–life balance preferences influence the careers of police officers.

Challenges of new methods

It is important to recognize the challenges associated with the methods that have been proposed in this paper as models for future police research. The challenges of natural variation designs include the lack of randomization into control and treatment groups and the fact that the researchers cannot manipulate the suspected causal factor. This has been the case for most Scandinavian police studies that have used natural variation designs (e.g. Holmberg, 2005; Skardhamar et al., 2016), which can make it difficult to make causal attributions. For example, Pedersen (2013) found that youth who had participated in preventive youth–police dialogues had more subsequent criminal offences than other youth. However, she acknowledged that a causal relationship between these variables was highly questionable given that the distribution of such youth–police dialogues is not random: the youth who participated in these dialogues were considered to be youth at risk initially. Hence, this finding does not prove that this crime prevention strategy actually increases the risk of continued delinquency. The observed correlation between participating in youth–police dialogues and delinquency may be spurious, because having a long criminal record increases the risk of both being called to a dialogue meeting with the police and later delinquency.

A more methodologically valid approach to using natural variation designs is provided by Jørgensen (2010). Jørgensen compared the level of violence on weekends when the Copenhagen police made special efforts to reduce violence with weekends when such efforts were cancelled. The cancellations were due to external factors such as personnel scheduling that are unlikely to be related to the level of violence. Under such circumstances, the natural variation design comes close to being a “true” experimental design, because the groups are randomized.⁶ Another opportunity for increasing the validity of natural variation designs is the control of pre-treatment conditions, which has been lacking in some of the natural variation designs used in police studies (e.g. Holmberg, 2005), but (partly) provided in others (Pedersen, 2013; Skardhamar et al., 2016).

External validity is a main concern in vignette studies (Barabas & Jerit, 2010), as researchers cannot be sure that the participants’ responses to a vignette reflect their real-life attitudes and actions. For example, the public debate on alleged racism by police officers may make officers especially sensitive to such accusations and, thus, they may provide “correct” answers when they see a typical ethnic minority name on a report in a vignette. This suggests that the Hawthorne effect is also a concern in vignette studies and that it needs to be addressed when such studies are designed and interpreted. External validity is especially difficult to achieve when researchers use vignettes and other experimental designs on complex cases (e.g. training within areas with different crime patterns). Such designs may be better suited to explore more simple cases, e.g. when an officer makes a judgement of one specific case.

As is the case with other surveys, vignette studies may also be plagued by falling response rates and hence a possible lack of representativeness. However, this will be less problematic in vignette studies if the goal is to see whether alterations in the vignette influence the results rather than how widespread certain attitudes and characteristics are in the population.

Using true experimental designs with randomization and interventions controlled by the researchers will make it easier to draw causal inferences, but such designs may still create challenges. Internal validity can be questioned also in experimental design: Is the study conducted in such a way that systematic bias is minimalized, and causal conclusions can be drawn? Treating randomized controlled trials (RCTs) as the gold standard has been contested. Sampson (2010) discusses several challenges of RCTs, including sample selection bias, non-compliance with the treatment among the participants, low policy relevance of the results and treatment effect heterogeneity, which means that the treatment effect will vary across people or settings. More specifically, the possibilities for drawing causal conclusions will be reduced if the experiment and the control group are not sufficiently balanced, if the subjects sense how they are expected to respond and if the variables are unreliably measured (Dunoff & Pollack, 2017). However, strong voices defend the use of RCTs: This design gives the strongest empirical evidence of causal inference since it does not require that all confounding variables are accounted for (Dunoff & Pollack, 2017; Weisburd, 2010). The widespread influence of RCTs in current police research outside of the Scandinavian countries (see e.g. Ariel, Farrar, & Sutherland, 2015; Mazerolle, Antrobus, Bennett, & Tyler, 2013; Ratcliffe, Taniguchi, Groff, & Wood, 2011) gives reason to ask whether this potential should be further explored also in Scandinavian police research.

Ethical considerations are another main concern when using experimental designs. Some field experiments, such as examining how police reports are handled by manipulating the sender’s characteristics, involve misleading the officers who are handling the reports and using valuable police resources on false reports. In addition, field experiments that involve the public have the same problems associated with obtaining informed content that we have seen in observational police research (Holmberg, 2011) and in studies involving vulnerable groups. In addition to the serious ethical considerations, this means that obtaining permission from both the police authorities and research committees for field experiments in police research probably will prove to be difficult.

Researchers in other fields have managed to gain permission for field experiments, often arguing for the importance of finding the answers to these questions that can only be addressed through field experiments (Birkelund et al., 2014; Jackson, 2009). Riach and Rich (2004) discussed the ethical considerations of using deception in field experiments that examine discrimination, and they concluded that the deception of subjects is justified in such research because of the potential social harm of discrimination, the superiority of this technique to uncover discrimination, and the limited inconvenience imposed upon the subjects. Hence, well-grounded and carefully considered field experiments probably are possible in police research. It is also worth noting that some field experiments may be easier to implement and have less serious ethical hurdles. For example, a field experiment on the effects of different types of field training, modelled on the work of Finseraas et al. (2016), could be implemented in co-operation with the police academies.

Extended use of register data also implies challenges, for instance requiring permission to access such data. To use police registers, researchers must apply to several different institutions (e.g. the Office of the Public Prosecutor, the Police Directorate and the National Centre for Research Data). Access to public registers such as the population register requires costly and time-consuming applications to the bureaus in charge of the official statistics. These types of requirements necessitate larger research projects with substantial funding and multiple collaborators. Up to now, such projects have been rare exceptions in Scandinavian police research, but they could also facilitate the establishment of closely co-ordinated, highly effective research groups.

The well-known challenge of missing data can be present also in registers. For example, background information on immigrants is often imperfect, and non-residents, illegal immigrants and other persons without a residence permit are totally missing in many registers. An obvious problem when mapping crime is that offences will not be registered if they are not detected or reported. Researchers and policy makers run the risk of oversimplifying when register data are used to estimate the effect of different police strategies and reforms, since the outcome measures in these registers often have limitations. For example, a decrease in the reported crime level following a police reform may be interpreted as a positive consequence of the reform. In reality, the decrease may have been caused by several other factors like decreased tendency to report crimes among the public, new registration routines in the police or economic growth.

Register data generally provide limited information. Because registers (justifiably) do not contain information about values and attitudes, for example, the research questions we can examine based on these data are limited (Fekjær, 2011). Nevertheless, research on other professions has convincingly shown that many interesting questions can be addressed using register data, and when combined with the use of survey data, register data open up unexplored possibilities in police research to examine both attitudes and values and further career development (Abrahamsen, 2015; I. Drange, 2011).

Conclusion

Although several weaknesses of current methods in police research have been discussed in this paper, it is also important to note the strengths of these methods. Qualitative studies have been important in the development of our knowledge about the police, and they will continue to make valuable contributions to police research in the future. Without qualitative studies, our possibilities for understanding meaning, interpretations and processes would be severely limited (Järvinen & Mik-Meyer, 2017). Among other strengths, qualitative research is useful for describing and understanding underlying mechanisms, and for developing new research questions. Surveys have provided a valuable overview of the characteristics and attitudes of police students and police officers. Much of the survey research in this area is also characterized by a closeness to the field, partly because police research often is conducted “from within” by researchers who are situated in police academies. However, we should note that this can have both positive and negative consequences: Although closeness to the field may provide inside knowledge and potentially generate research that is useful to the police, it also involves the risk of losing the important critical perspective that an outside researcher can provide (Holmberg, 2014a). Hartmann, Hestehave, Høgh, and Rønn (2018) contest this and claim that embeddedness does not necessarily lead to entanglement. On the contrary, the rich knowledge gained from police research from within may be exactly what is needed to achieve critical research.

The methods and data discussed in this paper are of course not the only alternatives for developing police research. Other methods and data of interest for police researchers include, but are by no means restricted to, systematic social observation (SSO), combining quantitative and qualitative techniques (Mastrofski, Parks, & McCluskey, 2010); social network analysis used for example to map organized crime based on police intelligence data (Campana & Varese, 2013); program theory evaluation to assess whether the program design makes the intended outcomes possible (Brousselle & Champagne, 2011), and the use of new available data sources, like information from body-worn cameras or Twitter (Crump, 2011). Although all methods and data of course will have weaknesses, they also provide interesting opportunities for observing the police from a different angle.

Another alternative strategy is to use mixed methods, combining different data sources and methods to learn more about a specific research topic. Studies combining quantitative and qualitative methods include for example Holgersson and Knutsson (2012) and several of the Ph.D. theses in Scandinavian police research from recent years (Hansson, 2017; Hellesø-Knutsen, 2013; Petersson, 2015). These studies display the advantages of using mixed methods, e.g. developing concepts through qualitative studies that can be explored further in quantitative studies. However, many of the methodological challenges discussed in this paper will not be automatically solved by using mixed methods. For example, low response rates in surveys of police officers may limit the external validity even when the surveys are based upon solid qualitative studies, and selection issues limit the possibilities for causal claims. E.g.: Although it is highly interesting to note that police officers handling the deportation of unaccompanied refugee children do not report reduced mental health (Hansson, 2017), we cannot out rule the possibility that these officers may have been a positively selected group prior to the experience of working with deportation.

In this paper, I have discussed new methods for future Scandinavian police research: natural variation designs, vignette studies and field experiments. In addition, extended use of register data has been discussed. As we have seen, these methods and data are not flawless; they have challenges that must be addressed if they are to improve future research. However, the challenges of the new methods must be compared with the limitations of other methods frequently used in Scandinavian police research: the Hawthorne effect, a lack of representativeness and limited possibilities for making causal claims. The proposed new methods have clear advantages on certain points. Field experiments and vignette studies provide stronger possibilities for making causal claims. Because both of these methods use randomization, it is possible to single out causal factors that often are of great interest to researchers, policy makers and the police themselves. The use of register data allows for increased external validity. Because registers cover almost all possible cases, we can make claims about the entire population under study. This is not possible with qualitative methods or surveys with selective response. Designs based on natural variation have the potential to both use a kind of randomization that makes causal claims possible and maintain external validity because the research is conducted in natural settings, although “perfect” natural variation designs are rare.

Ensuring the possibility to generalize and make causal claims has neither been the goal of all police research up to now, nor should be the goal of all future police research. However, to answer also the research questions which do call for external validity and/or causal explanations, our methodological repertoire needs to be broad. As I have argued here, we need new methodological approaches to ensure the future of Scandinavian police research as a vital, high-quality research field that communicates both with the practice field and with the international research community. The development of methods that are quite new to Scandinavian police research may lead to improved connections and citations by other research fields and reduce the risk of police research becoming an isolated field characterized by self-citation and methodological stagnation. Greater methodological variation can help us address new questions and broaden the horizons of Scandinavian police research.