Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
In response to WIRED’s requests for information, Avon and Somerset Police provided detailed information on 13 systems used between 2017 and 2024 – including those used to predict missing persons, anti-social behaviour, and those likely to commit or be caught in crime. WIRED passed this, along with other information about Avon and Somerset Police’s data science program, to the independent AI firm Eticas for review. The verdict was worse.
“Many of these models produce low-accuracy estimates, meaning that many of the people who put them at risk are misdiagnosed,” the data analysis found. According to police findings, a model used to help predict burglaries appears to be more than 10 percent better over three years. According to Eticas, this means that at least one in ten of those identified as being at high risk will experience disappointment. Other concerns include the performance metrics of various brands changing dramatically. “This is not the same as the well-controlled sample used,” the study noted.
A spokesperson for Avon and Somerset Police told WIRED that the force chose not to use some of the footage it produced, including footage related to the robbery. When asked why the group had years of analysis and performance of the models they did not use, the spokesperson said the analysis process was “self-explanatory” and used data from a “standard file that was not deleted when the decision was made not to use the model.”
Police declined requests for information about his data science work and did not respond to a detailed list of questions. “Each type is assigned based on their performance, and when information is identified, it is changed or deactivated,” a spokesman for Avon and Somerset Police said in a statement, adding that samples are reviewed by a forensic scientist before being sent to police.
It is not clear what steps Avon and Somerset Police took to deal with the risks that its ethics committee started in the early days of its data science career. The committee did not appear to discuss the forecast after 2017, according to record requests. And while Avon and Somerset Police say on its website that “every item and project” that is tracked as part of its data science work is reviewed by a dedicated team, a spokesperson told WIRED that “so far there has been no meeting,” because “no pattern has been made that has been identified.”
In response to one public records request, Avon and Somerset Police released a picture of a “bias monitoring program” that appears to monitor and compare the rate of crime between whites and blacks, concluding that “there is no significant difference between the two.” The Eticas review said: “Simply including race as an assessment is not the same as testing whether the sample produces discriminatory results,” citing the absence of detailed testing of race, gender, and socioeconomic status as a “serious failure.”
Asked if he believes that predictive analytics has an important role to play in policing or social work, Davies says more work is needed. “When we tried to do it, we tried to do it for the right reasons, in the right way, but we didn’t have the capacity that was needed.” Another part of the project should look at how vulnerable people can inform employees without forcing them to think about what happened in the past, he says. “There is a risk that workers will see the computer say something and then not use their imagination.”
Predictive analytics continues to play a major role in police and community services. Bristol City Council is still using a risk assessment model to assess whether a child is likely to leave education, work or training. Avon and Somerset Police recent research data, presented in July last year, shows that the model used by the Offender Management App correctly predicts one in three people who actually offend, while one in four people indicated that maybe the offenders do not.
