Several high-profile cases of law enforcement officers using deadly force against civilians within the past year have politicians, police and researchers looking for ways to prevent such incidents. This search includes a closer look at the computerized early warning systems that many large police departments have used for years to identify officers who are most likely to overreact violently during stressful situations. The main challenge: it is difficult to say with certainty how well or even if these systems actually work.

Early warning systems debuted in large police departments—those with more than 1,000 officers—decades ago as a way to identify those officers whose unprofessional behavior could cause problems in the communities they served. Departments programmed these systems to flag recurring complaints against officers and notify supervisors when certain thresholds were reached, such as a certain number of use-of-force complaints over a given period of time. Early systems’ predictive abilities were crude, primarily because they were capable of basing their analyses only on individual data sources—such as formal complaints—rather than combining information from various police databases that could provide context for an officer’s behavior. This might include the officer’s level of experience, whether the officer responded to an incident alone as well as the time and location of the event.

Pres. Barack Obama’s recently announced Police Data Initiative seeks to fill this gap via a research program to study the efficacy of law enforcement early warning systems—also referred to as early intervention systems—and to determine how they might be improved. The goal is to more effectively apply statistical tools, machine learning and other predictive analytics that take current data and look for trends that might continue into the future. When a system identifies an officer whose performance records and behavior suggest the need for some kind of intervention, supervisors can step in to arrange counseling, reassignment or additional training.