The Inter-Rater Reliability of Violence Risk Assessment Tools Used by Police Employees in Swedish Police Settings

1. Introduction

Violence risk assessment tools have been used in the Swedish police organization since the late 1990s (National Swedish Police Board, 2010). Their use began with the introduction of an intimate partner violence (IPV) risk assessment tool, the Spousal Assault Risk Assessment guide (SARA; Kropp, Hart, Webster & Eaves, 1994, 1995, 1999), in a couple of police authorities. Thereafter, a number of other tools have been introduced in more or less all Swedish police authorities. Most of these tools are variants of structured instruments developed in the forensic psychiatric context (e.g. SARA). In addition, there is a general violence risk assessment tool denoted the Police Screening Tool for Violent Crimes (PST-VC), which has been designed at one of Sweden’s regional police authorities. This tool is different, in that it assigns more weight to subjective judgements than the structured instruments.

The number of studies regarding violence risk assessment and management in police settings has increased in the recent years. For instance, areas of interest have been the applicability of psychiatric tools in police settings (e.g. Belfrage & Strand, 2009; Kropp, 2008; Kropp & Hart, 2004; Storey & Strand, 2012a); the development/adaptation of assessment tools (e.g. Belfrage, 2008; Belfrage & Strand, 2008; Belfrage, Strand, Ekman & Hasselborg, 2012; Dayan, Fox & Morag, 2013); the predictive validity of IPV assessments (e.g. Belfrage & Strand, 2012; Belfrage et al., 2012a; Storey, Kropp, Hart, Belfrage & Strand, 2014); police officers’ decision making in the risk assessment and management procedure (e.g. Robinson, Pinchevsky & Guthrie, 2016; Trujillo & Ross, 2008), and the inter-rater reliability/consistency of police officers risk assessments (e.g. Sebire & Barling, 2016).

The main findings of these studies can be summarized in the following statements. The majority of risk- and victim vulnerability factors employed in various assessment tools are useful in police settings, in particular in male offender samples (Belfrage, 2008; Belfrage & Strand, 2008, 2009). However, police officers have difficulty assessing psychiatric risk factors, because information regarding these factors is difficult to access and police officers in general lack training in psychiatry (Belfrage, 2008). This was one of the problems that motivated the development of the Brief Spousal Assault Form for the Evaluation of Risk (B-SAFER; Kropp, Hart & Belfrage, 2005, 2010), which is a police version of the SARA guide (see Belfrage, 2008 and Kropp, 2008). Another assessment tool developed to guide professionals in the law enforcement and social authorities is the PATRIARCH (Belfrage, 2005). It is a structured instrument, like the SARA and the B-SAFER, and focuses on the assessment of honor-based crime (see Belfrage et al., 2012b). Studies on risk assessment and management in female offender samples are still limited. However, a recent study examined police officers’ use of the B-SAFER instrument in a female offender sample (Storey & Strand, 2012a¹). Some of the B-SAFER factors were found to be useful in assessing female IPV offenders, but there seem to be some gender-related factors also, that are not included in the tool. Additional studies are required to identify these factors.

Further, the number of risk factors/the total score of the risk factors has been found to be positively associated with the summarized risk assessment (Belfrage et al., 2012a; Kropp & Hart, 2004; Storey et al., 2014). A relationship between the summarized risk assessment and recommended protective actions has also been found in a number of studies. Taken together, these studies show that a higher number/level of protective actions or specific protective actions are more often recommended in cases assessed as being high risk (Belfrage & Strand, 2012; Belfrage et al., 2012a; Kropp, 2008; Storey et al., 2014). The predictive validity of IPV assessments was found to be low to moderate, which was mainly thought to be a consequence of effective protective actions preventing repeat IPV in high-risk cases (Belfrage & Strand, 2012; Belfrage et al., 2012a; Storey et al., 2014). However, Trujillo and Ross (2008) found police officers to be biased in their summarized IPV risk assessments. They also found, in line with Robinson et al. (2016) that police officers tend to put the greatest emphasis on a small number of risk factors in their assessments. This suggests that the poor predictive accuracy might also be due to the quality of the summarized risk assessments themselves, and thus not merely a result of the effect of interventions. Some other possible reasons for the low accuracy, such as an underestimation of the risk, are also addressed in Belfrage and Strand (2012).

Moreover, it has been found that there is a relationship between the inter-rater agreement and the predictive validity of violence risk assessments (McNeil, Lam & Binder, 2000). Clinical assessments administered by nurses at a short-term psychiatric inpatient unit were compared with assessments administered by physicians. The cases were also followed up in terms of violent behavior. The main results showed that predictive validity was higher in cases in which the nurse and the physician agreed on the level of risk. In line with these results, Kropp and Hart (2000) state the importance of a high inter-rater agreement as follows: “If raters cannot agree on the presence of individual risk factors or the implications that can be drawn from them, there is little point in conducting risk assessments” (p. 109).

According to the National Swedish Police Board (2010), inter-rater agreement and the quality of risk assessments conducted in police settings increase with the use of assessment tools, which thus creates a firmer foundation for the recommendation of protective actions across different cases. This assumption is not motivated by any results from previous studies and to our knowledge, it has not yet been tested in a police setting. However, the stability of police officers’ SARA, B-SAFER and DASH (Domestic Abuse, Stalking, Harassment and Honor-Based Violence) ratings has been studied previously (Belfrage et al., 2012a; Storey et al., 2014; Sebire & Barling, 2016). In the first two studies, the first and the second assessment in cases of repeat IPV were compared. The stability of the total score of the risk assessment was high (ICC= .76² in both studies) and for the global risk assessment it was fair in the first study (SARA, ICC= .45) and low in the second study (B-SAFER, ICC= .32). Both measures increased over time and the changes were statistically significant. Belfrage et al. (2012a) also measured the stability of the police officers’ assessments of items related to symptoms of mental disorder (recent suicidal or homicidal ideation/intent, recent psychotic and/or manic symptoms and personality disorder with anger, impulsivity, or behavioral instability), and found the ICC coefficients (.56, .68, and .59) to be of moderate size. The stability of 38 specialist investigating police officers’ domestic violence risk assessments using the DASH risk assessment tool was recently examined (Sebire & Barling, 2016). All participants assessed the risk (low, medium or high) in four cases at two different points in time. Between the two occasions, the participants were reminded of the national risk grading definitions. The consistency of the first-time risk assessments was weak (ICC= .182, p < .05). The second assessments were somewhat more consistent (ICC= .283, p < .001). Further, 66 percent of the police officers’ assessments were similar at the two occasions. Officers working in areas with a comparatively high rate of domestic violence cases more often used lower risk steps in their assessments compared to officers working in areas with a lower rate of such cases.

The current risk assessment tradition dates back to the new conception of psychopathy advanced by Hare and Schalling (1978), and the construction of a psychopathy checklist (Levander, 1979). Such a checklist was used in the 1980s (the Penetanguishene project) as part of the VRAG risk assessment instrument (Harris, Rice & Quinsey, 1993), which out-performed contemporary actuarial risk assessment instruments with respect to predictive accuracy. Most of the follow-up work of this ground-breaking study was done in North American correctional settings, by psychologists/behavioral scientists. HCR-20 (Webster, Eaves, Douglas & Wintrup, 1995) marks the beginning of the current generation of risk assessments, the items of which are strongly loaded with psychopathy-related personality/behavioral factors. Used in correctional settings by properly educated and trained staff, the inter-rater reliability is impressive (around .90).

Kropp and Hart (2000), for example, compared IPV assessments (SARA; Kropp et al., 1994, 1995, 1999) conducted by staff from correctional settings with assessments conducted by a clinical psychologist. The clinical significance of the SARA total score was good to excellent (the total score, ICC= .84, Part 1, ICC= .68 and Part 2, ICC= .87). The inter-rater reliability for the summary risk assessment was somewhat lower (ICC= .63). Grann and Wedin (2002) presented similar results in a study in which the SARA instrument was used in an actuarial manner by a B.A.-level psychology student and a psychology researcher. In this case, however, the intra-class correlation score for part 1 of the instrument was higher than that for part 2 (total score, ICC= .85, Part 1, ICC= .88 and Part 2, ICC= .74). Finally, a violence risk assessment tool called the Spouse Violence Risk Assessment Inventory (SVRA-I) was recently evaluated (Dayan et al., 2013). As a part of the evaluation, the inter-rater reliability was examined, including 19 cases assessed by one investigator and two psychology graduate students per case. Assessments conducted by the students were compared both to each other and to the assessments conducted by investigators (correlations were computed). The results were fair (r= .68, .73 and .75 (p < .01).

Given these results and the police authorities’ duty to prevent violence and protect victims, the use of violence risk assessment and management methods in police settings is reasonable. The fact that these methods were implemented widely in the Swedish police organization without being fully tested in this specific context may be questioned. Thus, in the present study we aim to fill this knowledge gap by examining two violence risk assessment tools used in the Swedish police, the B-SAFER and the PST-VC. The first one is the tool that is used to the greatest extent among Swedish police, and the latter differs from this and most other tools used by the police, in that it is a general tool, allows more weight for subjective judgements and is developed by police employees. The tools will be evaluated separately with some comparisons when possible.

The present study is based on a broader evaluation regarding the use of violence risk assessment tools in the Swedish police authority, which was conducted on behalf of the National Swedish Police Board between the years 2012 and 2014.³ The overall aim of the evaluation was to examine police employees’ use of violence risk assessment tools and particularly whether these tools can be useful in identifying high-risk cases for repeat crime, and if different assessors examine the same cases coherently. All empirical data used in this study was collected for the evaluation.

2. Aims

The main aim of this study was to examine the inter-rater reliability of the Police Screening Tool for Violent Crimes (PST-VC) and the Brief Spousal Assault Form for the Evaluation of Risk (B-SAFER), as used by police employees in two different police authorities in Sweden.

3. Methods

4. Results

5. Discussion

The level of inter-rater agreement varied substantially for different types of factors in both samples. Overall, the assessors agreed regarding the global assessments, particularly when the percentage agreement was examined allowing for a one-step difference in the assessments. Even though one of the tools was a general violence risk assessment tool and the other was an IPV assessment tool, the overall results were similar. Sebire and Barling (2016) on the other hand, found a low consistency in police officers’ summarized risk assessments. However, it is problematic to compare the results since the tools under study were different, as were the procedures of the evaluations. There was a high rate of missing values and a low inter-rater agreement for most of the risk- and vulnerability factors, which may be explained by two different shortcomings: lack of training and lack of access to information. The overall level of training regarding violence risk assessment, management and key risk/victim vulnerability factors, was low among the assessors. Some of them had not had any training in how to assess and manage violence risk, meaning they might not even know which factors to include in the PST-VC assessment, or how to assess such factors. The latter has also been found in a previous study. Belfrage (2008) found that police officers had difficulty assessing psychiatric risk factors in IPV cases; they chose to omit risk factors rather than including information that was uncertain. One way to improve the way these instruments are used in a police setting would thus be to provide a basic education with respect to personality and mental health factors: a recommendation that is justified by the fact that male offenders who commit lethal violence on a female intimate partner suffer from a psychiatric disorder in almost 90 percent of the cases¹² (Rying, 2007).

The level of omitted factors varies greatly in previous studies (from less than 1% in -Storey et al. (2014) up to 39% in Belfrage (2008)). We can only speculate on the reasons for the large differences. However, the level of omitted factors was reduced in Belfrage’s (2008) study, after the assessors had received additional information on how to conduct the assessments. The assessors in the present study did not have any training related to the study, which may be one of the reasons to the large number of omitted factors in the study.

Storey, Gibas, Reeves and Hart (2011) evaluated a violence risk assessment training program for staff working in criminal justice settings, about 50 percent of whom were police officers. Overall, the results were promising, e.g. the participants’ global risk assessments and recommended protective actions were more consistent with the experts’ assessments when the training had been completed. They also improved their skills in identifying the presence of risk factors, and the level of agreement among the participants’ assessments increased. On the negative side, the number of irrelevant risk factors included in the participants’ assessments did not decrease after training and an even greater number of irrelevant protective actions were recommended in the assessments following training as compared to the situation prior to the training program.

Aas (2014) highlights whether working with cases of domestic violence should be a specialty in Norwegian police settings, due to the complexity in such cases. The findings of the present study motivate consideration of the corresponding question, regarding the work with violence risk assessments in Swedish police settings. Should violence risk assessment be a task for police employees in general or a specialty for a dedicated group?

The lack of access to information noted in the current study was due to both case-specific factors and to factors that were common to all of the assessors. In one case, neither the police nor the victim had been able to identify the offender and in some other cases, the offender was unknown to the victim and this limited the level of access to information. Another case-specific factor relates to the victims’ participation in the legal process. The victim is a key informant in risk assessments, which means that crucial information may be lost if the victim refuses to participate. Thus, the decision not to gather information for the B-SAFER assessments by speaking to victims probably contributed to the high rate of missing values and the low inter-rater agreement found in relation to those B-SAFER factors that could not be answered by only checking the police records/reports. The same explanation could probably be applied to the PST-VC results, since victims were not involved in the new assessments and rarely, as it seems, in the original assessments. The other reason to the lack of information was related to the lack of access to information regarding psychiatric risk factors among police employees in general (see Belfrage, 2008).

The fact that the assessors did not use the structured variables in the risk assessment process leaves too much space for subjective judgements, which in previous research has shown to be associated with heuristics and biases (Tversky & Kahneman, 1974). The researchers describe three different heuristics that people often use in doing such judgements: representativeness (comparisons with similar cases), availability (associations to instances close at hand) and adjustment and anchoring (adjustments from a starting point). Of these, primarily the first two can be linked to the violence risk assessments described in this study. For instance, the representativeness heuristic is used in classifying different cases to certain groups and the availability heuristics in assessing the probability of an event occurring. However, both representativeness and availability are affected by biases. E.g. in the use of representativeness, people seldom take the base-rate of the phenomenon under study into account and the base-rate is not related to the representativeness (called insensitivity to prior probability of outcomes by Tversky & Kahneman [1974, p. 1124]). In the case of subjective violence risk assessments this can be exemplified by an assessor’s use of his/her professional experiences regarding similar cases/situations. Thus, a case that resembles a previous high-risk case may be assessed as high risk, without taking the base-rate of such cases into account.

The availability heuristics are associated with a kind of bias called illusory correlation (Tversky & Kahneman, 1974, p. 1128), which is described as the influence of our perceptions regarding the association between different events. If we consider two events to be closely linked, these will more often be assessed as co-occurring. In the context of violence risk assessments, illusory correlations between for instance offender or victim characteristics and the risk of repeat IPV could lead to incorrect judgements.

The fairly high reliability for the global risk assessments was most likely due to shared assumptions (heuristics, tacit knowledge) among the assessors rather than being based on the information obtained by the tools. One reason for this among the PST-VC assessors was probably that they meet occasionally to discuss and compare their risk assessments. Although one of the tools (B-SAFER) is commonly used by police forces within and outside Europe the assessments seem to be flawed. Still, the actual assessments might nonetheless be relevant for risk predictions and selection of protective measures – an issue which we have not addressed in this study.