Personal Assistant System: Use of Computer Technologies to Facilitate Elderly People with Assisted L

1
Personal Assistant System Use of Computer
Technologies to Facilitate Elderly People with
Assisted Living

• Presenter Jennifer C. Hou
• Collaboration with Linda Ball, Stanley Birge,
  Marco Caccamo, Carl Gunter, Karrie G.
  Karahalios, Narasimhan Nitya , Lui Sha, Yang
  Yu
• Dept of Computer Science University of Illinois
  at Urbana Champaign
• College of Medicine, Washington University in
  Saint Louis
• Pervasive Platforms and Architecture Lab,
  Motorola Labs
• jhou_at_cs.uiuc.edu, 217-265-6329

2
Population Aging

• Aging of the baby boomer has become a social and
  economical issue.
• In the United States alone, the number of people
  over age 65 is expected to hit 70 million by
  2030, almost doubling from 35 million in 2000.

• Table compiled by the U.S. Administration on
  Aging based on data from the U.S. Census Bureau.

3
Percentage of People of 65 and 85

• People over age 65 are expected to constitute 20
  of the population in 2030.
• Similar increases are expected worldwide.

• Table compiled by the U.S. Administration on
  Aging based on data from the U.S. Census Bureau.

4
Similar Expenses Worldwide
2002
SOURCE United Nations ? Population Aging ? 2002
5
Similar Expenses Worldwide
2030
SOURCE United Nations ? Population Aging ? 2002
6
Consequences Are ..

• Along with the increase of elderly people
  population, the expenditures of the United States
  for health care will project to rise to 15.9 of
  the GDP (2.6 trillion) by 2010.
• -- Health care industry study, Digital
  Foresight
• Many elderly people will stay at home, rather
  than being consigned to expensive retirement
  homes.
• Even today, only 10 of elderly people of age
  65-85 and 25 of those of age gt 85 are
  institutionalized.
• Many elderly people choose to stay at home also
  for privacy/dignity issues.

7
Overview of the Talk

• Why Do We Care About Senior Care?
• Overview of Personal Assistant System (PAS)
• Underlying Research that Makes Things Work
• Systems Architecture
• Wireless communications
• Tracking and localization
• Security and privacy
• Software safety, reliability, and availability
• Human computer interface
• Do Elderly People Accept This Technology?
• Pilot studies
• Where Do We Go From Here?

8
Why Do We Care About Senior Care

• Because your parents are the next in line to be
  qualified as seniors, and you are next to the
  next in line.
• Can advances in sensing, object localization,
  wireless communications technologies
• enable elderly people to regain their capability
  of independent living?
• make possible unobtrusive supervision of basic
  needs of frail elderly and thereby replicate
  services of on-site health care providers?
• We believe the answers are Yes!

9
We Believe Technology Can Help

• Time Driven Reminders of Daily Activities
• Home PC
• Serves as the intelligence.
• Sends reminder messages to wireless-enabled
  appliances.
• Closes the loop with HCI/ localization
  techniques.
• Takes action in the lack of response A reminder
  can be sent more times, after which a designated
  on-site personnel or a healthcare provider is
  notified.

Jennifer, It is 830am. Time to take your Insulin
injection before breakfast.
10
Overview of PAS

• A drop-box architecture in which
• A security-enhanced, assisted living device
  called Authentication Manager for You (AMY)
  co-exists with a home PC, called the
  Assisted-Living Hub (ALH).
• The ALH is equipped with multiple wireless
  interfaces and serves as the local intelligence.
  It also communicates with the Assisted Living
  Service Provider (ALSP).
• Through web interfaces, ALSP allows healthcare
  providers and clinicians to retrieve/analyze data
  and give instructions.

11
A More Technical View of PAS
Home Environment
ALSP
ALH
Monitoring
IEEE 802.11 WLAN
Internet

.
AMY
Service
Clinicians

• Easy to deploy.
• Few entities that are subject to security
  attacks.
• Numerous applications can be built on top of it.

12
What Applications Are Most Critical
Factors Contributing to Loss of Independence and
Institutionalization
 

of Residents
of Residents as Need

With Need Primary
Cause ____________________________________________
____________________________ Needs prompting to
take medications
95 42 Risk
of injury due to falls
42
17 Unable to get up after a fall

20
17 Monitoring of vital signs too labor intensive
12
20 Needs physical assistance with
Activities of Daily Living 90
67 Needs prompting to
toilet on a schedule
67 17 Needs
prompting to go to meals
33
10 Needs prompting to bathe
75
0 Gets lost in
apartment
17
0 May wander out of facility

12 10 Needs
monitoring of blood sugar frequently
20
8 Needs monitoring of weight daily/weekly
25
0
Respondents include 8 geriatricians, 10 nurse
administrators of assisted living facilities, and
6 home health clinicians at Washington University
in Saint Louis
13
Application I Time-based Reminder Services

• PAS can help reminding residents of daily
  activities.
• ALH obtains from the ALSP updated prescription
  and appointment records of a resident.
• When it is time for the resident to carry out
  their time driven routines, the ALH locates
  active wireless-enable devices and sends reminder
  messages to one or more devices that are in the
  proximity of the resident.
• Whether or not these routines are followed as
  advised is detected in a non-intrusive manner by
  exploiting sensor localization techniques.

14
Application II Monitoring of Physiological
Functions

• A number of physiological functions critical to
  maintaining homeostasis for different medical
  conditions can be measured by Bluetooth-enabled
  medical devices, transmitted to the ALH and then
  to the ALSP to be evalauted by healthcare
  providers.
• Measures will have a prescribed desired range and
  deviations from that range will generate an alert
  from the ALSP to the health care provider.
• This enables prompt intervention before the
  situation deterioates to a point requiring
  hospitalization and may simply be in the form of
  additional instructions to the resident.

ALSP
Home Environment
IEEE 802.11 WLAN
Monitoring
Internet
.
.
AMY
Service
15
Application III Non-intrusive Monitoring of
Daily Activities and locations

• Detection of early warning signs for depression
  and/or other chronic diseases
• The location of a resident and the duration in
  which he/she stays at that location are profiled
  by Motes-based (or Ubisense-based) sensing and
  tracking in a privacy-preserving manner.
• Movement profiles are transmitted to the ALSP and
  analyzed (if necessary) for
• Early warning signs for severe depression (e.g.,
  not taking medicine, not eating, staying in bed
  for very long time)
• Preventive measures (behavior changes) for
  chronic diseases commonly seen in elderly people
  (such as Parkinsons disease and/or Alzheimers
  disease).

16
Application IV Fall Detection

• Residents wear sensors equipped with
  accelerometers (with fall detection algorithms
  that detect falls with the combination of speed
  and orientation changes).
• In the case of fall detection, the sensor device
  beeps and an alert message is sent to the ALSP
  and to a designated healthcare provider.
• In the case of false alarm, the resident can
  press a button and disable the message sending.

17
Overview of the Talk

• Why Do We Care About Senior Care?
• Overview of Personal Assistant System (PAS)
• Underlying Research that Makes Things Work
• Systems Architecture
• Wireless communications
• Tracking and localization
• Security and privacy
• Software safety, reliability, and availability
• Human computer interface
• Do Elderly People Accept This Technology?
• Pilot studies
• Where Do We Go From Here?

18
Software Infrastructure Needed

• A software infrastructure that integrates
  sensing, communication, and event/information
  management.
• Understand, analyze, structure and control the
  complex interactions across the layers of
  computing, communication and sensing along the
  dimensions of robustness, reliability, QoS,
  security and privacy.

Emergency event processing Telemedicine
Reminders/social interaction
Monitoring
Interface mgmt
Dependability
Real-time
Security Privacy
Evolvability
Events/data management
Communication
Sensing
19
ServerSocket ss new ServerSocket(90) ss.accept(
)
Networked Software Architecture
Socket s new Socket(myaslp.com90)
Active Application A
ALSP Server
Under Normal Situation When the Gateway is
Available
J2ME API for AccessingBT Nodes and Internet
TCP/IPComm. API
ALSPServerOS
FixedALHOS
BT Stack
TCP/IP Stack
TCP/IP Stack
Bluetooth
Cellphone Network Internet
Vital Sign Meters
IP Routing Service
802.11
WirelineMAC
802.11MAC
Gateway
20
System Architecture of the ALH
21
OS Layer in the ALH Architecture

• The OS layer
• Is equipped with various communication stacks and
  corresponding platform-dependent APIs
• Depending on the family of peripherals to
  support, other stacks and APIs may include
  Bluetooth, Zigbee, and Infrared.

22
Middleware Layer in the ALH Architecture

• Middleware
• Device monitoring daemons monitor the join/leave
  of peripheral devices in the environment and
  registers/de-registers the devices in the Device
  Registry Services.
• The register-de-register process includes
  creating/destroying proxy or singular proxy stub
  objects for the device.

• The proxy/proxy stub allows applications to be
  built upon well-known device APIs instead of
  vendor specific APIs/semantics. It also allows
  off-the-shelf devices to be integrated into the
  PAS system, as long as the vendor provides the
  semantics specifications on how to communicate
  with the device.

23
Middleware Layer in the ALH Architecture

• Device registry service maintains a database of
  peripherals available in the environment. Each
  entry of the database is a proxy (or proxy stub)
  object created for a specific peripheral device.
• Applications will query this service for
  appropriate proxies. When an appropriate proxy is
  found, a clone of the proxy migrates to the
  application.

24
Middleware Layer in the ALH Architecture

• The Unified Peripheral Communication APIs
• Abstracts different network stack programming
  interfaces into a consistent paradigm, which
  basically follows java.net. APIs.
• Allows users to specify (in XML) their QoS
  demands when establishing communication links,
  and the middleware returns approved QoS
  guarantees.
• This allows proxies to be developed independently
  of specific network stack programming interfaces.
  

25
Middleware Layer in the ALH Architecture

• Internet Heartbeat Daemon
• periodically checks the availability of Internet
  access through the gateway router.
• When the gateway router fails/recovers, this
  daemon activates/deactivates the Bluetooth cell
  phone to access to the ALSP.
• ALH Main Daemon is in charge of managing (start,
  suspend, stop, restart etc.) all the application
  daemons and middleware daemons on the ALH.

26
Overview of the Talk

• Why Do We Care About Senior Care?
• Overview of Personal Assistant System (PAS)
• Underlying Research that Makes Things Work
• Systems Architecture
• Wireless communications and networking
• Tracking and localization
• Security and privacy
• Software safety, reliability, and availability
• Human computer interface
• Do Elderly People Accept This Technology?
• Pilot studies
• Where Do We Go From Here?

27
Networked Software Architecture
TV
ALSP
Peripheral Bluetooth Network
ALH
earplug
Internet
Gateway
WLAN(e.g. IEEE 802.11)
Internet-capable Medical Meters
28
ServerSocket ss new ServerSocket(90) ss.accept(
)
Networked Software Architecture
Socket s new Socket(myaslp.com90)
Active Application A
ALSP Server
Under Normal Situation When the Gateway is
Available
J2ME API for AccessingBT Nodes and Internet
TCP/IPComm. API
ALSPServerOS
FixedALHOS
BT Stack
TCP/IP Stack
TCP/IP Stack
Bluetooth
Internet
Vital Sign Meters
IP Routing Service
802.11
WirelineMAC
802.11MAC
Gateway
29
ServerSocket ss new ServerSocket(90) ss.accept(
)
Networked Software Architecture
Socket s new Socket(128.174.11.1190)
Passive Application A
ALSP Server
Under Normal Situation When the Gateway is
Available
J2ME API for AccessingBT Nodes and Internet
TCP/IPComm. API
ALSPServerOS
BT Stack
FixedALHOS
TCP/IP Stack
TCP/IP Stack
PC IP 128.174.11.11
Bluetooth
Cellphone Network Internet
Dumb Dev
IP Routing Service
802.11
WirelineMAC
802.11MAC
Gateway
30
What If the Internet Gateway Is Down?

• One major deficiency of PAS is that all the
  traffic is transported through the gateway AMY,
  which becomes unavailable when it fails or when
  the resident is away from home.
• To enhance robustness and ubiquity of PAS, we
  have used cell phones (transparent to users) as
  both a backup AMY and the local intelligence for
  data aggregation and acquisition.
• we have leveraged the programming capability of
  Motorola EZX platforms (mainly the A780, E680 and
  E680i phones) to incorporate the following novel
  features that are not currently present in any
  cell phones
• Enabling cell phones to serve as local data
  storage/fusion intelligence.
• Enabling cell phones as a delivery endpoint for
  reminder messages

31
Networked Software Architecture
TV
mobile medical devices
ALSP
Peripheral Bluetooth Network
ALH
cellphone
earplug
Internet
WLAN(e.g. IEEE 802.11)
GPRS network or WiFi
Base Station
Internet-capable Medical Meters
When the Gateway is Not Available
32
ServerSocket ss new ServerSocket(90) ss.accept(
)
Socket s new Socket(myaslp.com90)
The cellphone modem is activated
Active Application A
ALSP Server
Cellphone DUN Modem(service provided by most BT
enabled cellphones)
APIs (TAPI/NAPI, File system, etc.)
J2ME Bluetooth Dial-Up Networking Profile
(BT-DUN) for Internet Access
TCP/IPComm. API
ALSPServerOS
Service Layer (Network, Connectivity, Telephony,
etc)
BT Stack
TCP/IP Stack
FixedALHOS
GPRS
BT Stack
Cellphone OS
Bluetooth
Cellphone Network Internet
Bluetooth Medical Meter
IP Routing Service
802.11
WirelineMAC
802.11MAC
A. L. Device
33
ServerSocket ss new ServerSocket(90) ss.accept(
)
The cellphone modem is activated
Socket s new Socket(128.174.11.1290)
Active Application A
ALSP Server
Cellphone DUN Modem(service provided by most BT
enabled cellphones)
J2ME Bluetooth Dial-Up Networking Profile
(BT-DUN) for Internet Access
TCP/IPComm. API
APIs (TAPI/NAPI, File system, etc.)
ALSPServerOS
Service Layer (Network, Connectivity, Telephony,
etc)
BT Stack
TCP/IP Stack
FixedButlerOS
GPRS
BT Stack
Cellphone OS
Bluetooth
Cellphone Network Internet
Bluetooth Medical Meter
IP Routing Service
802.11
WirelineMAC
802.11MAC
Gateway
34
Networked Software Architecture
mobile medical devices
ALSP
Peripheral Bluetooth Network
Cellphone
earplug
GPRSor WiFi
Internet
Base Station
WLAN(e.g. IEEE 802.11)
Internet-capable Medical Meters
When cell phone takes the roll of ALH
35
ServerSocket ss new ServerSocket(90) ss.accept(
)
Socket s new Socket(myaslp.com90)
The cellphone is used as a smart device
Application 2 (Sending data to ALSP Server)
Active Application A
ALSP Server
Application 1 (Listening to PC)
QoS based
APIs (TAPI/NAPI, File system, etc.)
J2ME
TCP/IPComm. API
ALSPServerOS
Service Layer (Network, Connectivity, Telephony,
etc)
BT Stack
TCP/IP Stack
FixedButlerOS
GPRS Or WiFi
BT Stack
Cellphone OS
Bluetooth
Cellphone Network Internet
Bluetooth Medical Meter
IP Routing Service
802.11
WirelineMAC
802.11MAC
A. L. Device
36
ServerSocket ss new ServerSocket(90) ss.accept(
)
Networked Software Architecture
Socket s new Socket(myaslp.com90)
Active Application A
ALSP Server
When cell phone takes the roll of ALH
J2ME API for AccessingBT Nodes and Internet
TCP/IPComm. API
ALSPServerOS
FixedALHOS
BT Stack
TCP/IP Stack
GPRS Or WiFi
Bluetooth
Vital Sign Meters
Internet
IP Routing Service
GPRS network or 802.11
Base Station
37
Overview of the Talk

• Why Do We Care About Senior Care?
• Overview of Personal Assistant System (PAS)
• Underlying Research that Makes Things Work
• Systems Architecture
• Wireless communications and networking
• Tracking and localization
• Security and privacy
• Software safety, reliability, and availability
• Human computer interface
• Do Elderly People Accept This Technology?
• Pilot studies
• Where Do We Go From Here?

38
Real-Time Tracking and Localization

• We have built a system that uses a combination of
  ultrasonic and RFID technologies as the
  underlying sensing mechanism for real-time
  tracking of both human and objects.

• Human Tracking
• Ultrasonic transmitters, called beacons and
  denoted by Bi, occasionally sends out an
  ultrasonic pulse and a short RF signal at the
  same time.
• Due to the speed difference between an RF signal
  and an ultrasonic signal, the listener denoted by
  Lj on the resident can infer its distance from
  the beacon using Time Difference Of Arrivals of
  the two signals.
• This distance measurement can be reported to the
  ALH for real-time tracking of the use (marked as
  (2) and (3)).

39
Real-Time Tracking and Localization

• Object tracking
• The RFID (radio frequency identification) reader
  attached on the wristband can read RFID tags of
  objects touched by the user.
• The objects can be tracked, whenever the user
  touches and/or carries them.
• The whereabouts of the objects are logged.

40
Overview of the Talk

• Why Do We Care About Senior Care?
• Overview of Personal Assistant System (PAS)
• Underlying Research that Makes Things Work
• Systems Architecture
• Wireless communications and networking
• Tracking and localization
• Security and privacy
• Software safety, reliability, and availability
• Human computer interface
• Do Elderly People Accept This Technology?
• Pilot studies
• Where Do We Go From Here?

41
How Medical Data Can Be Securely Transmitted ?
Sign. on Msg.
Sign. on Key
Michael et al. 06
42
How Medical Data Can Be Securely Transmitted ?
Sign. on Msg.
43
How Medical Data Can Be Securely Transmitted ?
44
How Medical Data Can Be Securely Transmitted ?
SOAP Envelope
Header
Medical Information
Medical Information
45
Implementation
46
Implementation

• H/W
• Device Digital Pulse Oximeter
• AL Hub A labtop, Windows
• ALSP A desktop, Linux
• Clinicians computer A labtop, Windows
• S/W
• Language Java
• Database MySQL
• Web service Apache web server, tomcat, axis 1.4
• Libraries XML-SEC, SAAJ, avetanaBluetooth, etc.

47
Security tokens

• The following security tokens are set up
• Doctors public key certificate
• The ALSPs public key certificate
• A secret key shared among AP and APs family
  members or friends
• APs and family/friends password to ALSP
• Access control matrix (ALSP), DataID table (AL
  Hub)

GDoc, GAS, KAP, U V, PPA , PFA , ACLAP , ?AP

• GDoc Doctors offices public key certificate.
• GAS The ALSP's public key certificate.
• U V URL pointers to the ALSP's public key
  certificate. U V are the ones which AP and
  clinician have, respectively.
• KAP A secret key shared the AP and her family
  members.
• PPA PFA AP's and Family's secret passwords
  (respectively) shared with the ALSP.
• ACLAP An access control list for the AP's
  records at the ALSP.
• ?AP A mapping table from a serial number of a
  medical device to DataID

48
Bootstrapping

• Clinician sets up relationship with ALSP /
  Exchange URL pointers
• Assisted person (AP) subscribes to an ISP / Set
  up a gateway for WPA network
• AP visits clinician. Clinician recommends a list
  of devices, hubs, and ALSPs. AP buys devices and
  hub and adds them to home network
• AP subscribes to ALSP. ALSP sends a URL pointer
  and a password token. AP loads them into the
  hub.
• AP creates a shared family key for family
  members and friends access to data on the ALSP.
  Family members register passwords to ALSP.
• Hub generates ?AP , a set of DataID. A dataID is
  mapped to a device serial number and recipients
  URL pointers. Each entry is a tuple (serial
  number, dataID, URL pointer)
• With the recipient information (clinicians and
  family members), ALSP creates access control list
  ACLAP.

?
?
?
?
?
?
?
?
?
Gdoc, GAS, KAP, U, V, PPA, PFA, ACLAP, ?AP
Clinician
Family
AP
ALSP
49
Overview of the Talk

• Why Do We Care About Senior Care?
• Overview of Personal Assistant System (PAS)
• Underlying Research that Makes Things Work
• Systems Architecture
• Wireless communications and networking
• Tracking and localization
• Security and privacy
• Software safety, reliability, and availability
• Human computer interface
• Do Elderly People Accept This Technology?
• Pilot studies
• Where Do We Go From Here?

50
Pilot Study at Nazareth Living Center

• With the help of geriatricians at Washington
  University in Saint Louis, we have carried out a
  3-week pilot study at the Nazareth Living Center
  for Assisted Living in June-July 2006.
• This facility houses 110 well-educated,
  predominantly female residents, whose average age
  was 88. Of 30 residents who attended a
  presentation, 14 agreed to participate.
• After administering a standard cognitive
  assessment, two residents were consented to test
  the PAS prototypes (application 1 and
  applicatoin2) and ten residents to carry/wear a
  placeholder device for a period of two weeks.

Nazareth Living Center is a Catholic, nonprofit
skilled nursing and assisted living community,
sponsored by the Sisters of St. Joseph of
Carondelet.
51
Pilot Study at Nazareth Living Center

• Residents found PAS useful and were willing to
  wear the sensing apparatus
• The two residents using the PAS prototype found
  it to be quite useful.
• Residents not chosen to use the working prototype
  expressed their desire to use the working version
  as well.

• Residents lack in confidence in PAS when it did
  not work properly
• Wireless connectivity is made possible by a
  low-end Linksys WRT54G wireless router. This,
  coupled with concrete walls between residents'
  rooms and the nurse station, led to intermittent
  connectivity.
• Residents, as a result, were not confident in
  relying solely on PAS for medical monitoring.
• To address this problem for the time being, we
  will add a wireless repeater (i.e., an additional
  Linksys WRT54G router configured in the
  client-mode and forwarding all packets to
  another, master router) between residents' rooms
  and the nurse station.

52
Pilot Study at Nazareth Living Center

• Nurses/caretakers desire interfaces that provide
  security/privacy.
• With a high resident-to-nurse ratio, nurses were
  usually very busy and could not ensure that the
  information being displayed would not be viewed
  by unauthorized personnel. The need for privacy
  should be addressed by designing adequate access
  control to PAS.
• Nurses inquired whether or not medical data could
  be securely transmitted via wireless technology.
  This implies they also had concerns about PAS
  security.

53
Overview of the Talk

• Why Do We Care About Senior Care?
• Overview of Personal Assistant System (PAS)
• Underlying Research that Makes Things Work
• Systems Architecture
• Wireless communications and networking
• Tracking and localization
• Security and privacy
• Software safety, reliability, and availability
• Human computer interface
• Do Elderly People Accept This Technology?
• Pilot studies
• Where Do We Go From Here?

54
Where Do We Go From Here?

• Technology enhancement with respect to software
  robustness/reliability, security/privacy, and
  HCI.
• Extensive, hypothesis-driven clinical trials with
  Washington University in Saint Louis and Buehler
  Center on Aging, Health Society, Northwest
  University
• The average cost of nursing home stay is about
  8,300/month. Can PAS delay transfers of
  elderly people to skilled nursing facilities
  (SNFs) and improve the quality of their lives (by
  preserving independence)?
• Delaying institutionalization by simply one month
  would cover the cost of deploying PAS.
• Phibbs CS, Holtz J-EC, Goldstein MK et al. The
  effect of geriatrics evaluation and management on
  nursing home use and health care costs Results
  from a randomized trial. Med Care 20064491-5.

55
Concluding Remark

• With most of the PIs being baby boomers, it is
  our sincere hope that we will be the first
  generation to live independently and with dignity
  with the help of PAS when we age, and the project
  will benefit the healthcare system and the
  society as a whole.