1
To Ignore or Not to Ignore? Follow-up to
Statistically Significant Signals"
Reflections from San Diego County
Biosurveillance Information Exchange Working Group
2/23/06
Jeffrey Johnson, MPH San Diego County Health
Human Services Agency
2
SAN DIEGO COUNTY

• Nearly 3 million population
• International border
• Large military presence
• Biotechnology Hub
• 21 Emergency Departments

3
Early Event Detection in San Diego

• Evolving effort since pre - 9/11
• Data sources ER Visits, Paramedic transports,
  911 calls, school surveillance, OTC sales
• Systems Local SAS/Minitab system, ESSENCE, and
  BioSense
• Statistical
• Methods Descriptive, time series, CUSUM,
  EWMA, process control methods (PU Charts)
• Multiple syndromes
• Visualization and alerting
• Incident Characterization
• Follow-up to signals

County of San Diego Health Human Services Agency
4
If We Ignore A Signal

• We take no action or follow-up
• Save staff resources
• Avoid bothering hospital staff yet again
• Another data source may signal
• The Feds may pick it up
• Might lose an earlier start to a response
• We might be dead wrong to ignore

5
If We Do Not Ignore a Signal

• Will it be another false alarm
• May detect an event earlier
• Earlier response
• Continued interaction with the medical community
• Gain experience with follow-up
• Increased situational awareness

6
Characterization of Detections

• Detection Method
• Syndrome group
• Admitted
• Deaths?
• Geographic cluster?
• Prior days level?
• Recent level?
• Age groups?
• Severe syndrome?
• Detections in other data sources?
• Other epidemiological intelligence?
• Other diagnostic information

Follow-up? Action or No Action or Watch
7
Detection Follow-up with Medical Community
What is the final diagnosis of Patients A, B,
C? Is there a common pattern among admitted
patients? Did any have lab test results that
might suggest a larger event? Among patients
with a common zip code, was there a shared
living setting or common exposure? Can we send
someone out to review medical charts? What is
your facilitys assessment of the situation?
County of San Diego Health Human Services Agency
8
ABERRATION DETECTION RESPONSE GUIDELINES
Routine Surveillance Activities
Ignore?
IDENTIFY
NO
Aberration detected
YES
Rule out system error
Potential false positive
VERIFY
Preliminary evaluation
Ignore?
Describe initial results
Ignore?
True Positive
YES
NO
False Positive
NOTIFY
Inform key divisional staff
Inform key departmental staff
Intensive monitoring surveillance
Evaluate other data sources
Ignore?
County of San Diego Health Human Services Agency
Cluster check
9
GI Syndrome Over Time (10/31/04 8/24/05)
ED
911
Paramedic Runs
10
The Significant Aspects of Syndromic
Surveillance

• Statistical Significance
• Public Health Significance
• Significant Event
• Significant Public Awareness
• Significant Biological Agent Detection

11
Statistical significance vs. public health
significance
HAZMAT FLAG 12/04/2004
County of San Diego Health Human Services Agency
12
Statistical significance vs. public health
significance
County of San Diego Health Human Services Agency
13
County of San Diego Health Human Services Agency
14
Significant event with statistically significance
outcomes
Syndromic Surveillance for Natural Disasters
San Diego Wild Fires, 2003
San Diego County
15
Significant event with statistically significance
outcomes
Syndromic surveillance for natural disasters
16
Significant Public Awareness
The Clinton Effect September 4, 2004
While spikes in both datasets are apparent,
normalized counts show a relatively larger
increase in ED visits on Sept. 6, 2004.
17
Significant Public Awareness
7/7/05 London Bombings
San Diego County Paramedic Transports for
Chest Pain
18
Significant BT Agent Detection
Biowatch

• BioWatch Detection
• Tells us agent, sensor site and date
• Plume plot may help us narrow surveillance on a
  geographic area

Application of Syndromic Surveillance
Agent Syndrome categories Specific word
search in CC or DX fields Sensor site Zip
codes, population (schools) Date Temporal
based surveillance New pre-detection baselines
19
Anatomy of a Detection (a case example)
20
Daily Email Report
Feb 5, 2006
911 Call Data
Attached Table
21
911 Call Center - GI Syndrome Signal
22
(No Transcript)
23
Line listing for review
Non-specific call complaints
24
911 Call Center - GI Syndrome Signal
The count for the signals include a consistent
range
21 Signals since 07/01/03
Various statistical signals
25
What did we do?

• Magnitude of cases
• Which method(s) signaled?
• Check the other call centers
• Check the other data sources
• (ED data, EMS transports)
• Review the line listing
• Our conclusion..

... 14 vs mean of 7.8 ... CUSUM (2),
P-Chart, U-Chart No signals No
Signals . No apparent pattern gtgtgtgtgt

• Super Bowl Sunday
• Fewer trauma calls
• Smaller denominator (P-Chart)
• Traditional increase in GI on this day
• Watch next days results

26
Case Example 2 Hospital 9 ED Data Respiratory
Syndrome
27
Hospital 9 - Daily Results Table
28
(No Transcript)
29
(No Transcript)
30
Hospital 9 Respiratory Syndrome
01/01/04 - 02/03/06
Many signals. So whats the context? Do we ever
ignore the signals?
31
Hospital 9 Respiratory Syndrome

• 24 signals over a 37 day period
• Count range 11 34
• Over time an increasing mean

32
Greater Syndrome Specificity
Hospital 9 Influenza-like-illness (ILI) Syndrome

• ILI syndrome has greater syndrome specificity
  than Respiratory syndrome
• 16 signals over a 37 day period

33
What We Have Learned

• Sgnls Happen!
• Make sure you see flames before yelling Fire
• CUSUM 2 3 STD may be too sensitive
• We lose precision with non-specific syndromes
• Everyone wants to know whats going on all the
  time
• Increasing focus on situational awareness
• Further evaluation and testing required

34
Hype Cycle of Emerging Syndromic Surveillance
Technologies
Adapted from the Gartner Hype Cycle
Dual use, situational awareness, appropriate
signals
Too many signals?, IT Costs, poor syndrome
specificity, evaluation results
The magic bullet
9/11, Anthrax attacks
Prioritized data sets, protocols in place,
Event or Technology Trigger
Peak of Inflated Expectations
Trough of Disillusionment
Slope of Enlightenment
Plateau of Productivity
35
Considerations

• More work in all areas of syndromic surveillance
  is needed
• Knowledge requires responsibility
• The enemy is studying our efforts
• Current/future funding levels require
  reliability, efficiency and sustainability of
  systems and approaches
• The Future
• Neural networks and Artificial Intelligence
  (AI)?
• Are we ready?

36
Contact Information
Jeffrey Johnson 619-531-4945 jeffrey.johnson_at_sdcou
nty.ca.gov
Thank You