BPD looks to build ambitious public visualization site for crime data, but first it hopes to restore a simpler spreadsheet-like feed of the information

Boston Police have started looking at building a Web site that would let the public see data from arrests and other incidents in a variety of ways, including on maps.

But first, BPD Supt. James Hasson and city CIO David Elges said today, the city hopes to restore a long-standing data set of the information on the city's Analyze Boston public data portal.

That data has been offline since last month, after City Councilor Andrea Campbell (Dorchester, Mattapan, Roslindale) called for a hearing into complaints from Citizens for Juvenile Justice that the current system was running amok - entries would be duplicated 30 or 40 times.

Also, the number of categories used to categorize incidents had been collapsed from several dozen different types to just a few - making it impossible, for example, to tell when an arrest was for minor marijuana possession and when it was for running a major crack ring - Citizens for Justice said today. And even the number of arrests and other events would vary on almost every download - even in 2021 for 2020 data - the group said, adding all the problems made it impossible to run any sort of meaningful analysis on the data.

Other groups were also relying on the city data set, such as the ACLU, which had to build its own entire database for its daily downloads of BPD data - because in addition to the duplication and category issues, the city system often deleted the geographic coordinates of incidents.

Campbell said that, especially now, in an era of "unchecked police power used against its citizens," most recently in Minnesota the other night, transparency and accurate police data are vital for police accountability.

"Nothing nefarious was going on here," Hasson said. Hasson said problems began in September, 2019, when BPD switched on a new internal data system, by a company called Mark43, as part of a federally required change in the way police departments organize data and submitt it to the feds.

The new system was designed to handle everything from evidence and arrest tracking to case management and the creation of daily "public journals" that had been posted on the department Web site - and to feed daily updates to Analyze Boston, set up under the directive of former Mayor Walsh to let the public use and analyze public information from a collection that now includes more than data sets, from building permits and violations to the salaries of city workers.

The "pipeline" from the new BPD system to the Analyze Boston "data wareouse" began doing all sort of odd things, such as duplicating entries, he said.

He continued that whoever programmed it failed to take into account the new way BPD officers entered incident information into the system - first by checking off a broad crime category, such as "Drugs," which would then bring up a dropdown list with additional, more detailed descriptions of specific types of drug incidents. Whatever part of the system was collecting the information for the public data was only using the initial top category, rather than the more detailed types of crimes, he said.

Hasson and Elges said a fix for the current city site, which has rudimentary search tools but which lets visitors download all the data for their own analysis and mapping, is just days away.

However, Hasson said his staff has begun preparing a formal request to Mark43 to build a separate Web site that would not require users to download the data and then figure out on their own how to map or analyze it, but would instead give them all sorts of online tools to consider the data That site, which might also let people download detailed police incident reports, rather than making a visit to BPD headquarters, will take six months to a year to build.

"It's going to be a great thing," he said.

Councilor Julia Mejjia (at large) urged BPD to create a citizen task force to help design the new system, to ensure the new system gives visitors the information they need in a format they'd want.

Lauren Chambers, staff technologist at the ACLU, said one thing she'd hope to see in the new system is the race, age and sex of people arrested, so that groups such as hers could begin to look at possible issues related to those.