PCbased Telerehabilitation System with Force Feedback

1

Electrical and Computer Engineering Dept.
Traditional Applications of VR
2

Application (training, education, etc.)
System architecture
3

Types of VR Applications

• VR is having a resurgence in traditional and new
  markets
• Traditional markets are medical,
  entertainment/arts/education and
  defense/aerospace
• Emerging markets are manufacturing, oil/gas
  exploration, data visualization

4

CyberEdge Information Services Survey 2002
5

MEDICAL APPLICATIONS OF VR

• Are needed in order to better train medical
  students and doctors alike
• Allow errors to be made on virtual rather than
  real patients
• Allow rare cases to be modeled and studied
• Alleviate the animal rights controversy and the
  shortage of cadavers problem
• Permit more enjoyable and efficacious treatment
  for patients (example rehab)
• Allow more realistic examination (board exams)
• Allow reduction in medical costs (less liability
  suits, cheaper training emergency response).

6


MEDICAL VR Virtual Anatomy

• Everett Koop, former US Surgeon General observed
  in 1993 that
• The medical knowledge we have today is 20 times
  as great as when I went to medical school, but in
  that time, the teaching methods havent changed
  much..
• Interactive anatomy teaching is needed since
  cadaver training is non-repetitive and scarce
• Visible human dataset is the first completely
  realistic model for both male and female Is a
  de-facto standard for algorithm comparisons
• It is however huge, so cannot be rendered in
  real time. Needs model segmentation and
  decimation. Such models are sold by Visible
  Productions Co. (http//visiblep.com/)

7


MEDICAL VR Virtual Anatomy
8

MEDICAL VR Virtual Anatomy
9

MEDICAL VR Anatomic VisualizeR
10

Medical VR applications
Diagnostic and triage
Anesthesia (IV, epidural)
Surgery (emergency, MIS, open)
Rehabilitation
11

Training in Emergency Medical Response to
Bio-terrorism

• Training in bio-warfare response is not
  practical with current live exercise methods.
  No true casualties, very expensive to organize
  rarely done.
• VR allows casualties to die, is much cheaper
  to organize, can be done locally, without
  requiring EMT personnel to travel away from their
  township.

Real Response to Bio-terrorism
Avatar in protective suit
(Stansfield et al., 2000)
12

Training in Emergency Medical Response to
Bio-terrorism

• Training scenario airport explosion with
  casualties and propagation of Staphilococus
  Enterotoxin (germ warfare)
• Highly stressful situation awareness training
  where timing counts. Tasks have to be done
  correctly and fast.

Simulation view of airport
Propagation of toxin
(Stansfield et al., 2000)
13

Training in EMR - system block diagram
(Stansfield et al., 2000)
14
Training in EMR - system communication diagram
Adapted from (Stansfield et al., 1998)
15

Training in EMR -continued

• Intelligent objects help in grasping operations
  as the user/trainee lacks dexterity (no finger
  position measurement)
• Intelligent objects can be queried by the
  simulation about their status

(Stansfield et al., 2000)
16

Training in EMR -continued

• Virtual patient is driven by a state machine
  (finite state automata)
• It changes state based on disease progression,
  or as a response to users input
• Responses are specific to disease, example eye
  reflex is absent for head trauma, skin color
  changes for pneumothorax, etc.

(Stansfield et al., 2000)
17

Training in EMR Human Factors study

• Human factors evaluation of the simulator was
  performed in 1999 at Texas AM University on 23
  EMT subjects
• All subjects were volunteers, and screened for
  simulation sickness, medication and poor vision
• Subjects had no experience with VR

18

• Had to fill in a subjective evaluation
  questionnaire (score 1 lowest 5 highest)

Training in EMR Human Factors study
19

Training in EMR Human Factors study
(Stansfield et al., 2000)
20

Diagnostic of Prostate Malignancies

• Prostate cancer second leading cause of death
  among men 25 of patients die
• Detection of palpable nodules through digital
  rectal exam (DRE) is a strong indication of
  cancer (50 of palpable prostate nodules are
  malignant)
• Prostate can be in one of four stages normal,
  enlarged, incipient malignancy (single nodule),
  advanced malignancy (cluster of nodules)
• DRE is taught in medical schools in a primitive
  way (a turret with rubber models that the student
  can palpate and blindly diagnose)
• Limited training on patients, no follow up, no
  online data on student proficiency, no confidence
  after graduation, discomfort for patient, etc.

21

Rutgers DRE trainer

• A prostate DRE trainer was developed at Rutgers
  using a PHANToM, an SGI High Impact workstation,
  and a mechanical board to restrict movement

(Burdea et al., 1999)
22

Rutgers DRE trainer - continued

• The system was programmed using OpenGL for
  graphics and GHOST for haptics

(Burdea et al., 1999)
23

Rutgers DRE trainer - continued

• Prostate model was simplified to 200 vertices,
  with each malignancy constructed of 100 vertices.
  Malignancies were detected based on optimized
  vertex search approach, due to the 1000Hz
  requirement of the PHANToM

(Burdea et al., 1999)
24

Rutgers DRE trainer - continued

• Virtual nodules could be placed randomly on the
  north-south, east-west lobes of the prostate.

(Burdea et al., 1999)
25

DRE Human factors trials

• In order to ascertain efficacy as a trainer, a
  number of human-factor trials were performed in
  1998
• There were three groups
• First group was 22 non-medical student
  volunteers (16 male and six female)
• Second group were 4 tired urology residents
  (after 24 hours emergency room rotation)
• Third group were other volunteer urology
  residents from UMDNJ.
• Subjects in the first two groups used the
  PHANToM for force feedback, while the third group
  (control) used traditional rubber models
  (turret)
• Only subjects in the second group used the
  mechanical restricting board.

26

DRE Human factors trials - continued

• First subjects were trained for five minutes on
  the use of the PHANToM and then they palpated
  virtual prostates that were visible on the
  screen. The prostates were in wire-frame mode,
  such that malignancies were visible
• During examination mode the screen was blank
  (they had only force feedback to based the
  diagnosis on)
• Each subject was given 12 random cases to
  diagnose, with three-minute rest periods between
  groups of four cases
• Variables recorded were length of time to
  diagnose (sec.), the diagnosis given, and the
  correct diagnosis

27

Diagnosis accuracy vs. case type
(Burdea et al., 1999)
28

Time to diagnose vs. case type
(Burdea et al., 1999)
29

DRE task learning
30

Endoscopic Examinations

• Routinely done to detect cancer and other
  diseases Done by inserting a flexible viewing
  device (the endoscope), which projects an image
  of the body on a monitor
• Depending on the region of interest these
  procedures are called bronchoscopy (for the
  lungs), angioplasty (for the circulatory
  system), or colonoscopy for the colon.
• The are invasive procedures, require patient
  anesthesia, and can lead to injury, if done
  badly
• So endoscopic procedures require training, as
  well as a minimum of procedures done annually, to
  maintain skill.

31

Endoscopic Examinations - continued

• One training system is the PreOp bronchoscopy
  simulator developed by HT Medical (now part of
  Immersion Co.).

VC 8.1
32

PreOp Study

• Study done to determine efficacy in training
  novice physicians
• An experimental group of five physicians-in-train
  ing with no endoscopy experience and a control
  group of four experienced physicians with more
  than 200 endoscopic procedures each
• The experimental group had 4 hours of training
  on PreOp the control group had only 30 minutes
  of familiarization with the simulator

33
PreOp Study - continued

• Then both groups had to inspect the airways of a
  normal virtual patient.
• The variables measures were speed (duration of
  bronchoscopy), dexterity (number of contacts with
  the virtual bronchial wall), and accuracy (number
  of segments missed).
• The speed of experts was better, but their
  accuracy was poor (29 missed segments vs. only
  4.5 for the novices).
• Results were confirmed on a mannequin (17 missed
  segments vs. no miss).

34

Endoscopic procedures Virtual Colonoscopy

• Colonoscopy is a screening procedure that can
  save lives (90 of colon cancers found early are
  treatable)
• Advantages of virtual colonoscopy Non invasive,
  does not require anesthesia, is much faster (15
  minutes vs. hours)
• An algorithm extracts the colon wall, performs
  electronic cleansing, and then reconstructs the
  3D profile of the colon
• Another algorithm then allows a virtual camera
  to fly through the reconstructed colon looking
  for polyps.

35

Endoscopic procedures Virtual Colonoscopy

• The virtual camera can travel the whole length of
  the colon, which is not possible with real
  flexible endoscopes
• Has the same detection rate as real colonoscopy
  for polyps larger than 5 mm.
• Found 3-mm polyps which were missed by regular
  colonoscopy

36

Endoscopic procedures Virtual Colonoscopy
Virtual colonoscopy movie www.viatronix.com
37

Medical VR applications
Diagnostic and triage
Anesthesia (IV, epidural)
Surgery (MIS, open)
Rehabilitation
38

Intravenous Procedures Trainer

• Insertion of IV needle is high-volume procedure
  which is also a leading cause of infections and
  discomfort for patients.
• Present teaching methods are inadequate and
  nurses learn on patients
• This leads to low retention of nurses (high
  turnover rates) and
• patient complications.

39

I.V. Trainer

• HT Medical developed the
• CathSim simulator
• 1-DOF needle simulator with
• force feedback.
• The case library has six types
• of patients from drug user to
• geriatric female.

VC 8.2
40

Medical VR applications
Diagnostic (palpation)
Anesthesia (IV, epidural)
Surgery (MIS, open)
Rehabilitation
41

Open surgery - Anastomosys

• Open surgery has advantages and disadvantages
  Requires skill for suturing of two blood vessels
  anastomosys
• Boston Dynamics developed a pilot system for
  training in VR

(www.bdi.com/Anastomosis.html)
Stereo glasses
Half-mirror
PHANToM arm
PHANToM arm
42

Anastomosys - continued

• The blood vessels are modeled using the TELEOS
  spline-based toolkit
• Vessels are textured to increase realism
• Trainee holds real surgical tools (tweezers,
  needle holder, etc) for increased task realism.

(www.bdi.com/Anastomosis.html)
Needle insertion task
Blood vessel being deformed
43

Anastomosys Human factors study

• An experiment was conducted to compare the
  medical students and vascular surgeons using the
  VR surgical trainer
• The task consisted of repeated insertion of a
  curved needle through a simulated vessel
• The two groups were 12 students (Harvard Medical
  School), and 9 Boston area surgeons
• Each group performed the task nine times
• Results showed that the trainer was able to
  measure surgical skill differences and thus may
  be a tool for (future) board/certification
  examinations.

(www.bdi.com/Anastomosis.html)
44


Anastomosys human factors - continued

• The variables used to gauge performance were
• Time
• Accuracy
• Angle error
• Peak force
• Tissue damage
• Surface damage
• Overall score.

Screen view of the trainer
(www.bdi.com/Anastomosis.html)
45

Anastomosys human factors study - continued

• Surgeons did better than students by having less
  tissue damage
• There was also less learning for the surgeons,
  and more uniformity as a group.

(www.bdi.com/Skills_test.html)
46

Anastomosys human factors study - continued

• Individual task performance (normalized over
  surgeons performance)

47

Anastomosys Human factors study

• Subjects learning process exemplified by
  overall score

(www.bdi.com/Skills_test.html)
Surgeons
Overall score ()
Medical
students
Trial Number
48

Minimally-Invasive Surgery (MIS)

• Many current surgical procedures are done
  minimally invasive
• MIS has advantages for the patient
• less hospital stay (from one week to less than
  one day)
• less scarring (three 1-2 cm cuts)
• faster recovery.
• MIS has disadvantages for the surgeon
• loss of direct (3D) view of the cutting area
  has to look at a 2D monitor
• loss of tactile feedback, which is filtered by
  the laparoscopic instrument (stick with a
  handle)
• the fulcrum effect due to the orifice in the
  body

49

MIS - continued

• The fulcrum effect cannot be compensated by
  increased attention
• Requires training to automate the
  proprioception
• Training should not be done on animals, much
  less on patients!

50
MIST VR

• The Minimally Invasive Surgical Trainer in
  Virtual Reality (MIST VR) developed in UK is a
  computerized system to train and asses MIS skills

• In consists of a PC coupled with an Immersion
  Laparoscopic Interface (dual hand version)
• No force feedback is provided

51
MIST VR

• The simulation consists of six 3-D
  manipulation/cutting/burning tasks of graduated
  difficulty
• Task time, motion length, errors, are measured
  transparently and available remotely to the
  instructor.

(www.vrweb.com/docs/news/mist.htm)
Equivalent MIST VR task
Real laparoscopic task (tissue burning)
52
MIST VR - GUI
(www.vrweb.com/docs/news/mist.htm)
53
MIST VR - Evaluation

• The goal is to have objective measures of
  surgical skill, and to train MIS surgeons in an
  uniform way.
• MIST VR was evaluated to determine
• Does training on MIST VR make a difference?
• Does it work as a skill evaluator (with possible
  use in surgical license examinations)? I.e. is
  the system sensitive enough to detect skill
  differences between experienced and inexperienced
  subjects?

54
MIST VR - The Gallager study

• Used two groups of novice MIS surgeons, with
  eight subjects each
• The experimental group got training on MIST VR
  doing two complete (6 tasks each) sessions in 24
  hours. The control group got no training
• Following training, both groups were evaluated
  on their surgical skills on a non-VR (real) task.
  Task consisted of cutting correctly a pattern of
  lines on a piece of paper, using the laparoscopic
  instruments (with fulcrum effect)
• There were 10 trials of 26 cuts each

(Gallager et al., 1999)
55
MIST VR - The Gallager study
Adapted from (Gallager et al., 1999)
Maximum number of correct incisions in a trial
26 STD in Trial 1 was 1.75/3.16 STD in
Trial 10 was 1.99/3.16)
56
MIST VR - The Taffinder study

• Done at the Imperial College of Medicine in
  London looking at the use of MIST VR as an
  evaluation tool
• Three experimental groups of 10 subjects each
  consisted of
• experienced surgeons (gt 100 laparoscopic cases)
• trainee surgeons
• non-surgeons
• All groups were put on MIST VR and performed the
  first five tasks as training
• The last MIST VR task, requiring two-hand
  manipulation was then used to assess surgical
  skill.

(Gallager et al., 1999)
57
The Taffinder study - continued
(Taffinder et al., 1998)
58
MIST VR - continued

• It is in current use at the European Surgical
  Institute (Germany)
• In 2000 over 10,000 trainees took classes
  there, and reported a 30 improvement in basic
  skills after using the MIST-VR trainer
• Most reported that haptics needed to be added

59
Laparoscopic Surgical Workstation

• Immersion Co. has recently introduced the
  Laparoscopic Surgical Workstation, with force
  feedback for insertion, pitch, yaw, handle twist
  and handle grip
• Works with a PC, being connected over a PCI
  card.
• Has more realistic graphics (through LapSym
  software).

http//www.immersion.com/medical/docs/LSW_data_she
et.pdf
60
Trauma Center simulation using the Wii
Series of surgery games produced by ATLUS
(www.atlus.com) for Nintendo game systems
61
Tools used in the game
The Wii Remote allows for precision controlling
of the tools. The action is also felt with the
built-in Rumble feature. Each surgical tool is a
different direction on the Control Stick. The
player operates with both hands
Stitches use the needle to suture incisions and
other delicate areas.
Surgical Laser the laser incinerates tumors and
viruses.
Bandages end an operation by applying tape to
the sutured area.
Scanner it's used to find concealed tumors or to
magnify affected areas.
Syringe this is necessary for injecting various
medications.
Antibiotic Gel a potent medication used for
disinfection that can also heal small wounds.
Forceps extract harmful materials and pick up
delicate objects like synthetic membranes.
Scalpel a blade to make incisions and
excise/remove tumors and other objects.
Hand take a hands-on approach to various
actions, such as heart-massage and membrane
application.
Drain a long tube for removing fluid from the
area of operation, primarily blood.
62

Medical VR applications
Diagnostic (palpation)
Anesthesia (IV, epidural)
Surgery (emergency, MIS, open)
Rehabilitation
63
Orthopedic rehabilitation- Ankle

• 1.3 million people visited emergency rooms in US
  in 1998 because of ankle problems
• 1.2 million visits to physicians offices for
  ankle sprains and 675,000 visits for ankle
  fractures
• 25,000 people sprain their ankle every day
• Source American Academy of Orthopedic Surgeons
  (http//orthoinfo.aaos.org/fact)

64
Orthopedic rehabilitation- Ankle
The Rutgers Ankle rehabilitation system(2000)
65
Orthopedic rehabilitation- Ankle
Remote monitoring station
The Rutgers Ankle rehabilitation system(2002)
Dual platforms
VC 8.3
Telerehab
66
2003-2004 study

• 8 patients chronic post-stroke exercised on the
  Rutgers Ankle for 12 sessions (4 weeks)
• The first three weeks the therapist was present
  in the room. The last week the therapist was in a
  different room, controlling at a distance.

Movement accuracy
Ankle movement repetitions
67
2005 study at Harvard

• 4 patients chronic post-stroke exercised on the
  Rutgers Ankle, 2 with VR, 2 without VR (acting
  as controls)
• Significant differences in walking speed and
  distance walked in a 6-minute walk endurance test
  (Mirelman et al., 2006)

68
Stroke rehabilitation

• There are 4 million Americans that survived a
  stroke, and 500,000 are added each year. In the
  chronic phase they do not receive rehab
• Holden and colleagues at MIT developed the
  teaching by imitation method of rehab

• Tests on 9 patients underwent 30 one-hour rehab
  sessions, three times/week
• Significantly higher shoulder flexion and grip
  strength post VR-rehab.

69
Stroke patient trials at Rutgers (2000)

• The group at Rutgers/UMDNJ/NJIT developed
  game-like exercises
• Each VR-based rehabilitation session consisted
  of four exercises of N trials each. Each
  individual exercise concentrates on one
  particular parameter of hand movement
• RANGE
• SPEED
• FRACTIONATION (Individual Movement)
• STRENGTH

(Jack et al., 2000)
70
Stroke patient trials at Rutgers (2000)
VC 8.4
VC 8.5
71
Stroke rehabilitation - Rutgers
72
The good news...
73

VR Psychological rehabilitation

• Used to treat eating disorders, post-traumatic
  stress disorders, and phobias
• Patients with phobias may be afraid of heights,
  confined spaces, animals, speaking in public or
  fear of flying.
• Classical treatment of phobias is exposure
  therapy in which patient gets controlled
  amounts of stimulus, in order to desensitize
• Classical treatment is expensive, risky, and
  poses problems with patients confidentiality.
• VR therapy has advantages
• on a PC in doctors office (privacy)
• safer (like fear of spiders or snakes)
• cheaper (no need to take patient on an actual
  flight).

74

VR Psychological rehabilitation

• Fear of flying therapy developed at Georgia Tech
  and Emory University
• Uses a PC and HMD The software has a state
  machine

Fear of flying
75

VR Psychological rehabilitation

• Patient controls progression through a set of
  progressively harder situations taxiing, take
  off, level flight, aborted landing and landing
• Controlled study on 45 subjects showed VR to be
  as effective as standard exposure with a 92
  retention of gains after a year.

76

VR Cognitive Rehabilitation

• For patients with Alzheimers disease, traumatic
  brain injury, and children with attention
  deficit/hyperactivity disorder

• For AD/HD trackers are used to register
  restlessness. 3-D sound is provided as
  distractions. Subjects see a virtual classroom. A
  study showed that the system can diagnose ADHD.

77

VR for Post-Traumatic Stress Disorder (PTSD)

• It is found is war veterans as well as those
  exposed to acts of terrorism (such as the World
  Trade Center destruction)
• Studies at USC use Full Spectrum Warrior game
  engine to adapt it for training of Iraq veterans
• Research in New York at the Cornell Medical
  Center train survivors of 9/11

Full-spectrum warrior
World Trade Center
78

VR Education Applications

• Learning is constructivist, constructionist,
  and situated.
• Constructivist learning involves exploration of
  pre-built world
• Example is CyberMath developed in Sweden for
  college students

79

VR Education Applications

• Another example of exploration-based learning is
  the Virtual Physics Laboratory to teach
  high-school students Newtonian and Quantum
  Physics
• Students interact with the simulation using a
  sensing glove and virtual control panels.
  Students are more motivated and understand 3D
  concepts better compared to other methods of
  instruction.

80

VR Education Applications

• Constructionist learning involves active
  building of a world
• An example is the NICE project to teach
  gardening to elementary school children they
  used a CAVE and ImersaDesk to interact with VEs
  showing a virtual garden.

81

VR Education Applications

• The system was tested on 52 second-grade
  students
• They were divided in groups with a leader that
  interacted with the simulation the garden had
  avatars to represent team leaders and
  intelligent agents (plants).
• The percentage of children understanding
  gardening concepts went from 12 (before) to 35
  (after), most being the team leaders.

82

VR Arts Applications

• VR is a medium for artists to create in
• It is also a new way to experience and explore
  art very useful for art historians
• It allows preservation of cultural heritage in
  VR
• It increases access to art for people living in
  remote locations, or for the handicapped through
  virtual museums.

83

• Original statue created by Michelangelo in the
  XVI-th Century as his tomb monument
• Recreated in VR by a team at IBM over a period
  of two weeks, using stereo scanners and
  co-registered color digital photos to produce a
  mm-level of detail.

Virtual Florentine Pietà
Virtual statue
Real statue
84

VR Arts Applications

• A 3-D viewer (low resolution) allowed
  interactive frame rates on a laptop for art
  historians
• Views not possible in a museum
• The statue was restored in VR adding missing
  parts
• It was placed in context in a virtual
  mausoleum.

85

Virtual Heritage

• Groups replicas of famous archeological,
  architectural and natural sites In 2000 the
  Virtual Heritage Network (www.virtualheritage.net)
  was formed as a UNESCO affiliate.
• The recreation of a virtual heritage model
  starts with architectural plans, historical
  documents and visit to the actual site to acquire
  photos (used later for textures).
• Light measurements need to be made for input to
  radiosity computations.
• Care for cultural sensitivities, as models are
  placed online..

86

Virtual Heritage SS Sergius and Bachus
(Istambul)

• Created by a group at University of Geneva
• Consists of a 18,000 polygon exterior model and
  a 59,000 polygon interior model created using 3D
  Studio Max Uses light maps.

Model viewed in VRML 97
87

Virtual Heritage SS Sergius and Bachus
(Istambul)

• The VRML viewer assured 30 frames/sec with 2-cm
  geometrical accuracy
• A custom VRML extension (to allow
  multi-texturing) was subsequently used to
  increase realism, but frame rate dropped to 10-26
  fps.

Custom VRML 97
88

Virtual Heritage

• Virtual Notre Dame the best known church in
  Europe (took 200 years to complete).
• Modeled using the Epic Unreal engine (normally
  used in game creation).

89

• Virtual tours with a guide who is a friar avatar
  (1200 textured polygons). Had predetermined
  motion sequences.
• An AI engine drives his behavior, based on input
  from proximity sensors placed at various
  locations inside the virtual cathedral

Virtual Notre Dame
90

VR Entertainment Applications

• PC games pose a serious threat to the
  traditional arcade-based VR entertainment
  applications

Adventure games Lara Croft
Strategy games - StarCraft
91
How about haptics and wind and ambient light?

(amBX, Phillips, 2007)
92
Games are migrating to cell phones

• Mobile phones have cameras, which means that
  GestureTeks vision gesture recognition runs on
  them too.
• Tilt a World is the fastest selling video game
  for cell phones. It was developed by a team
  headed by Manjula Kuttuva (former researcher in
  my lab at Rutgers).

93

• Disney has constructed Disney Quest a building
  full of arcades
• Allow feedback effects not possible at home, and
  multiplayer games. An example Pirates of the
  Caribbean motion platform
• Passive tactile feedback, stereo graphics (SGI)

VR Entertainment Applications
94

VR Entertainment Applications

• Disney also developed the Virtual Jungle
  Cruise on inflatable rafts. Pneumatic motion
  platform, sensorized oars and water sprays add to
  the feeling of immersion.

95

VR Military Applications

• The military has traditionally been a proponent
  of VR since it offers many advantages
• Ability to train remotely as a team (SIMNET)
  reduced transportation/housing costs and reduced
  environmental impact
• Aircraft simulators that are programmable and
  modular, and allow a shorter life cycle
• Ability to simulate missions ahead of execution,
  as well as an advanced debriefing (after mission)
  modality
• Ability to visualize enemy weapon capabilities
  in order to reduce casualties
• Ability to train individual solders in new
  weaponry, as well as evaluate trainee performance

96
Army use of VR
De-mining trainer
Standard military probe
Head Mounted Display
HMD view
PHANToM arm
97
Army use of VR
Mockup of rocket launcher
Head Mounted Display
Stinger trainer
Sensing switch
Stinger rocket trainer developed by TNO (Holland)
in early 90s.
98

Army use of VR

• The system is in use in Germany 1999
• It allows team training with two users, a gunner
  and a commander
• Each trainee wears a high-resolution HMD (1280 x
  1024) due to the task of spotting airplanes
  remotely
• Each trainee is tracked and their position is
  mapped to avatars

Stinger trainer in Germany
Gunner
Commander
(Reichert, 2000)
99

100

Small Arms Trainer (CST 300D)

• Configuration
• Standard LCD projector
• Standard Hit Camera
• Optional Night-Vision Filter
• Optional Ceiling Mount
• Joystick controls movement during tactical
  scenarios.
• Each PSC supports 8 weapons

• Provides
• Individual marksmanship skills training
• Team and squad level tactical training
• Enhances Instructors ability to analyze
  shooters performance

101

Weapons Devices
FATS utilizes live weapons converted into
simulated weapons. This provides students with
the best, realistic training.
Rifles M16, M203, SA80, MP5, SA80 - Isw
Machine Guns M249 SAW, M240
Pistols 9mm, Magnum, Sig, M9
Shotguns
Crew served AT-4, M72 LAW
Kits MK-19
Small arms
102
Army use of VR Platoon leadership training

• Needed to improve decision making by young
  officers, in unusual situations Creates mission
  rehearsal exercises
• Example an accident between a military jeep and
  a civilian car in Bosnia
• The only real character is the trainee

VC 8.6
103

Army VR Applications Battalion-level
simulations

• The Simulation Network (SIMNET), to train tank
  commanders in a virtual battlefield Uses dead
  reckoning, intelligent agents (Semi-autonomous
  forces SAFOR)

• Has stealth vehicles to look at a battle from
  any angle and at any time like a 4D history
  book
• (used in the first Gulf War)
• Limited terrain database 50x75 km2

104

Army VR Applications

• The Close Combat Tactical Trainer (CCTT) is a
  replacement of earlier SIMNET, with a larger
  terrain database, several weather conditions, and
  communication with supporting elements
• Example Aviation CCTT (or AVCCTT)

105

Army VR Applications

• The Close Combat Tactical Trainer (CCTT) system
  consists of fixed or mobile sites which are
  modular to allow various helicopter missions to
  be simulated

AVCATT
106

Navy VR Applications

• The VESUB project to train officer of the
  deck. Consists of a station that recreates the
  command station, a suspended and tracked HMD
  (CRT-based) which works as virtual binoculars,
  voice communication, and AI agents. System is now
  in use at the Submarine School in Grotton,
  Connecticut.

VC 8.7
107
VESUB Simulation

• The VESUB system was tested in 1998 on 41
  subjects participating in a 3-hour simulation
  (familiarization, training, examination)
• Results showed significant task learning on
  skills like
• issuing correct commands
• checking range markers
• using correct commands during emergencies (like
  man overboard).
• Most learning occurred in novice users (Officers
  of the Deck)
• Experienced users had no learning (or negative
  learning), probably due to conflicts with prior
  mental image or due to lack of interest.

108
Naval Artillery Trainer in the British Royal Navy

• Needed to alleviate environmental concerns and
  to save on training costs

Close-range naval gun
Traditional shore-based trainer
109
Naval Artillery Trainer

• System consists of tracked HMD, real gun with
  sensorized trigger, and a PC running the
  simulation. There is also a commander station.

110
Naval Artillery Trainer

• A third PC is dedicated to the Instructors
  Station System was placed in service in 2001

Gunner Director view
Scenario Control Interface view
111
Air Force Applications

• Need for smaller, more mobile trainers. Two
  approaches The A10 trainer and the Mako virtual
  cockpit.
• A10 uses real cockpit and side-by-side displays
• Mako uses HMD, and virtual instruments, glove
  interaction.

A10 trainer
Mako virtual cockpit
112
Air Force Applications

• Needs to allow remote team training distribute
  mission training (DMT) trains in dissimilar Air
  Combat Tactics