Winners of the Hidden Signals Challenge designed systems that use open data to detect biothreats.
by Mahala Pagán
At what point does a data puzzle become a human behavior puzzle? When using nontraditional data and technology for public safety, even the best analysis tool will fail if it’s not part of a system designed for the behavioral and operational realities of the people using it, as well as the people affected by it.
Last fall, on behalf of the Department of Homeland Security (DHS) Science & Technology Directorate and in partnership with NASA Tournament Lab, we asked a big question: Can you help identify biothreats in real time, and can you do so using data we already have? The Hidden Signals Challenge — a $300,000 open innovation prize competition powered by Luminary Labs — searched for answers to that question, welcoming original ideas from a diverse array of perspectives, industries, and expertise. This week, DHS announced the grand prize winner (Boston Children’s Hospital) and runner-up (Daniel Neill and Mallory Nobles). How did these teams arrive at winning system designs? Here’s a look at how they worked their way from original concept to viable system.
An open call for creative concepts
The call for submissions appealed to a broad cross-section of experts. Stage 1 submissions came in from universities and academia, artificial intelligence and machine learning technology startups, research think tanks, city administrators, health innovation teams, and more. Their concepts explored a wide variety of existing data sources and analysis methodologies, including closed-circuit television and satellite imagery, transportation and commuter pattern data, social and news monitoring, as well as semantic analysis and natural language processing. A panel of judges selected the five finalists with the most promising concepts, awarding $20,000 in seed money to each team as they advanced to the Stage 2 Virtual Accelerator and turned their focus to developing system designs.
Balancing theory and practice through system design
Even the most innovative or algorithmically sound systems cannot succeed without thoughtful, user-centered design, a realistic implementation plan, and sustainable operational projections in place. We’ve all experienced the results of ineffective notification systems — whether it’s unplugging a smoke alarm that won’t stop beeping for no clear reason, or tuning out the many beeps and buzzes of smartphone notifications — and on an individual level, these potential miscommunications are typically not a major cause for concern. But large-scale historical examples like the Three Mile Island and Deepwater Horizon incidents show how system design decisions can be the critical difference between warnings that work in theory and warnings that work in practice. In the case of Three Mile Island, user-centered design was a fundamental gap. In a paper on “The Design of Interactions in a High-Stakes Environment,” Axel Roesler explained, “the design and arrangement of information displays and controls in the control room confused operators about what was going on and negatively affected decision-making during critical phases of the accident.”
With this balance of theory and practice in mind, the eight-week Virtual Accelerator included targeted webinar sessions, office hours, reading lists, field exercises, and ongoing expert mentorship. The curated selection of learning modules pushed finalists to advance their concepts beyond the outline of an analytical and technical approach to form a comprehensive, human-centered, actionable system design.
The need for an interdisciplinary perspective was top of mind during the Virtual Accelerator, and finalists were encouraged to step outside their areas of expertise and explore, talk to relevant experts and end users, conduct testing and experimentation, look at their systems through different lenses, and learn by doing. By the time they submitted their final system designs, teams had formed partnerships with local city administrators and tech startups and placed end users at the center of both algorithmic design and UX design. The teams had developed greater expertise and a greater understanding of where they would need additional support, testing, and input.
The results
Pandemic Pulse, the $150,000 grand prize-winning submission from the Computational Epidemiology Lab at Boston Children’s Hospital, provides a dashboard for integrating Twitter and Google Search data with infectious disease monitoring tools (Flu Near You and HealthMap) to detect biothreat signals. The tool filters data based on pathogen category, information source, and transmission mode, using a tiered evaluation method. When asked about the insights that helped the team center their system design around city-level operators, they explained that “through the Hidden Signals Virtual Accelerator, we have come to realize that within our single, intended end-user group, the needs and priorities may vary based on position, responsibilities, or even personality. We need our platform to be flexible enough to allow for zooming in and panning out at any given moment, as this will increase understanding and therefore trust among our end-users.” Learn more about Pandemic Pulse and their winning system design.
Pre-syndromic Surveillance, the $50,000 runner-up submission from Daniel B. Neill and Mallory Nobles, integrates emergency department chief complaints with data from health clinics and social media to discover outbreaks that do not correspond with known illnesses. The team is piloting a working prototype with New York City’s Department of Health and Mental Hygiene and other city agencies. “We are developing a ‘human in the loop’ interface that allows public health officials to define events of interest and distinguish between events of high interest (‘notify me of any meningitis exposures’) and low interest (‘don’t show me clusters from motor vehicle accidents’). The system will learn continually from this feedback, enabling it to better identify emerging threats, while limiting false positives.” Learn more about Pre-syndromic Surveillance and their winning system design.
Learn more about open innovation and how Luminary Labs’ methodology closes the gap between concept and viability.
Photo by Amos Bar-Zeev on Unsplash.