Imagineering Creative Technologies

1
Imagineering Creative Technologies

• Professor Paul Mc Kevitt
• Intelligent Systems Research Centre
• School of Computing Intelligent Systems
• Faculty of Computing Engineering
• University of Ulster, Magee
• Derry/Londonderry
• Northern Ireland

2
Background

• Intelligent MultiMedia MultiModal Systems
• Integration of natural language, speech/audio,
  graphics/VR, vision
• Semantic representation, fusion synchronisation
• Affective computing
• Integration reuse of existing software
  (Thórisson, 2007)
• Relations with cognitive science
• Deployment on mobile devices
• (Dell Axim, Smartphones, Nintendo DS)
• Applications in arts music, drama, dance,
  design, computer games, film, TV/radio, art,
  storytelling

3
(No Transcript)
4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
CHAMELEONa platform for developing intelligent
multimedia applications

• Tom Brøndsted, Paul Dalsgaard, Lars Bo Larsen,
• Mike Manthey, Paul Mc Kevitt
• (CPK, Aalborg University, Denmark)
• Speech input/output, image processing laser
• IntelliMedia Workbench campus information system
  application
• Semantic representation with frames (Minsky,
  1975)
• Blackboard for semantic representation
• Bielefeld DACS distributed communications
  platform (Fink et al., 1996)
• Implemented with C, C, DACS, Infovox TTS,
  grapHvite
• Applications in multimodal systems development

10
Architecture of CHAMELEON
11
IntelliMedia Workbench
12
CHAMELEON Semantic Representation

• Semantic Frames (Minsky, 1975)

13
CONFUCIUSautomatic conversion of natural
language to 3D animation

• Minhua (Eunice) Ma, Paul Mc Kevitt
• Lexical Visual Semantic Representation (LVSR)Use
  of Jackendoffs (1990) Lexical Conceptual
  Structure (LCS)
• Schanks (1977) scripts
• Use of Fellbaums (1998) semantic classes in
  WordNet
• Implementation with Java, VRML, WordNet, Connexor
  parser, FreeTTS (Java)
• Applications in education, computer games,
  animation production,
• digital storytelling

14
Architecture of CONFUCIUS
15
MultiModal semantic representation
16
CONFUCIUS examples

• Contact verb example
• (e.g. hit, collide, scratch, touch)
• The car collided with a wall.
• Collision detection
• Nonspeech sound effects
• No human role involved
• Human action verb examples
• Ipke gave Brigitte a loaf of bread.
• Bernie put a cup of coffee on the table.
• H-Anim Site node
• locative tags of object
• (on tag for table object)
• Paul pushed the door.
• Gert ate the bread.

17
Paul sat on the chair. Teri was waving
goodbye. Helge went to the gym. Gert put on
his hat. States Anke is sad. Anke is
ashamed.
18
(No Transcript)
19
SceneMakerautomatic visualisation of
screenplays

• Eva Hanser, Joan Condell, Tom Lunney, Paul Mc
  Kevitt
• Automatically generates affective virtual scenes
  from screenplays/play scripts
• Realistic visualisation of emotional aspects
• Extraction of genre/moods from screenplays
• Use of OCC (Ortony-Clore-Collins) affective
  model,
• ConceptNet, Open Mind Common Sense (OMCS),
  WordNet-Affect
• Enhance believability of virtual actors and scene
  presentation
• Multimodal representation with 3D animation,
  speech,
• audio cinematography
• Applications in assisting directors, actors,
  drama students, scriptwriters

20
Architecture of SceneMaker
21
MemoryLanea mobile digital storytelling
companion

• Sheila Mc Carthy, Heather Sayers, Mike McTear,
  Paul Mc Kevitt
• Multimodal stories (speech, text, video) from
  life-cached
• memory data (poems, songs, photos, home videos)
• Extensive HCI requirements analysis with focus on
  older people
• Multimodal preferences and abilities of users
  determines
• story modalities and user interface
  configuration
• Prudent on sensitive life-cached data
• Creates new stories on the fly
• Applications in medical informatics and digital
  care
• Implemented with Microsoft Visual Studio
• Deployed on Dell Axim X51v PDA

22
Architecture of MemoryLane
23
MemoryLane Prototype
24
AmbiLearnan ambient intelligent multimodal
learning environment for children

• Jennifer Hyndman, Tom Lunney, Paul Mc Kevitt
• Educational games virtual learning environments
• Child - Computer Interaction (ChiCI)
• AmbiLearn architecture TreasureLearn
  application
• Implementation with DevKit Pro (Nintendo
  homebrew)
• Deployment on mobile device (Nintendo DS)
• Application to collaborative learning for children

25
Architecture of AmbiLearn
26
PlayPhysicsan emotional games learning
environment for teaching Physics

• Karla Munoz Esquivel, Tom Lunney, Paul Mc Kevitt
• Virtual Learning Environment (VLE) for teaching
  Physics
• Incorporating serious games intelligent
  tutoring systems (ITS)
• Modelling of emotions personality of student
• Use of Bayesian Networks for modelling student
  knowledge
• Affective feedback through audio-visual effects
• On-line web application with Student, Lecturer,
• Head of Department, Administrator views
• Implemented with Elvira Bayesian Network tool,
  Flash, Java, Tomcat Apache Server, MySQL
  database, ConceptNet, 3D Studio Max Maya
• Applications in educational software computer
  games

27
Olympia architecture
28
PlayPhysics interface (astronaut game)
29
TeleMorph TeleTurasbandwidth determined
mobile multimodal presentation

• Tony Solon, Kevin Curran, Paul Mc Kevitt
• Multimedia limitations of mobile networks,
• mobile device capabilities
• User preferences on modalities
• Transmoding between modalities with
  cross-modality adaptation
• Use of Fuzzy (logic) Inference system (FIS)
• TeleMorph platform TeleTuras tourist
  application
• Implemented with MATLAB
• Applications in streaming media, computer
  networks,
• multimodal presentation

30
Architecture of TeleMorph Fuzzy Inference System
(FIS)
31
TeleTuras GUI
32
Available media qualities in 6 sample test
scenarios
33
Adaptation trace data from test scenario 6
34
MediaHubBayesian decision-making in an
intelligent multimedia distributed platform hub

• Glenn Campbell, Tom Lunney, Aiden McCaughey, Paul
  Mc Kevitt
• Interpret/generate semantic representations of
  multimodal input/output
• Fusion synchronisation of multimodal data
• Decision-making with Bayesian
  Causal Probabilistic
  Networks (CPNs)
• Implemented with Psyclone platform,
  HuginLite Bayesian modelling
  tool, Java, NetBeans IDE
• Applications in decision-making for key problems
  in
• multimodal systems
  anaphora resolution,
  domain knowledge awareness, multimodal
  presentation, turn-taking, dialogue-act
  recognition parametric learning

35
Architecture of MediaHub
36
Decision-making layers
37
psyProbe view of MediaHub Whiteboard
38
Alternative Turn-taking' Bayesian network in
Hugin
39
SoFISOng Form Intelligence for repairing
streaming musicacross wireless bursty networks

• Jonathan Doherty, Kevin Curran, Paul Mc Kevitt
• Large packet loss dropouts of 5 - 15 s.
• Use audio pattern recognition matching to
  repair gaps
• K-means clustering classifier determines
  self-similarity matches
• Match current sections of song received with
  previous
• sections played
• Typicality of Western Tonal Format (WTF)
• Use of MPEG-7 Audio Spectrum Envelope (ASE)
  multimedia content description standard giving
  features for extraction
• Implemented with G-streamer for steaming media,
  Ices2, Icecast2 for audio broadcast MPEG-7
  Audio Spectrum Envelope (ASE)
• Applications in repair of streaming music

40
Architecture of SoFI
41
Spectrogram representation
42
(No Transcript)
43
(No Transcript)
44
Steganoflagestrengthening steganography in
digital images

• Abbas Cheddad, Joan Condell, Kevin Curran, Paul
  Mc Kevitt
• Hiding/embedding data in transmission medium
• Use of 1D hash algorIthm coupled with
  2D iFFT (irreversible
  Fast Fourier Transform)
• Encryption of digital documents in 2D spatial
  domain
• Use of steganography to add security layer
  resistant to
• noise JPEG compression
• Exploitation of Jarvis kernel (Jarvis et al.,
  1976) to generate error diffusion signal
• Use of Wavelet-based Inverse Halftoning via
  De-convolution (WInHD) to recover approximate
  original signal
• Implemented with MATLAB PHP
• Applications in security, forensics, forgery,
  digital watermarking

45
Architecture of Steganoflage
46
Securing confidential data
Application
The traditional method of securing individuals
data with the notion of security by obscurity
that Steganography introduces
47
National international collaboration

• InstitutoTecnológico de Estudios Superiores de
  Monterrey, Campus Ciudad de México
  (ITESM-CCM), Mexico
• Aalborg University, Denmark
• CADIA, Reykjavik University, Iceland
• University of Bielefeld, Germany
• LIMSI (CNRS), Orsay, Paris, France
• Sheffield University, England
• Stanford University, California, USA

48
(No Transcript)
49
(No Transcript)
50
Noel Sharkey thinks AI is an illusion
51
(No Transcript)
52
Intellectural Property (IP)

• Steganoflage
• United Kingdom Patent Application No.
  0819982.0
• Method for Skin Tone Detection,
• University of Ulster,
• Abbas Cheddad, Joan Condell, Kevin Curran, Paul
  Mc Kevitt.
• SoFI
• United Kingdom Patent Application No.
  0908879.0
• A system and method for streaming music repair
  and error concealment,
• University of Ulster,
• Jonathan Doherty, Kevin Curran, Paul Mc Kevitt.

53
Industrial collaboration

• InterTradeIreland Fusion project, 2008/10
• Nvolve Ltd., Letterkenny, Co. Donegal, Ireland,
• Michael McNeill,
• School of Computing Information Engineering,
  Coleraine
• Prof. Paul Mc Kevitt.
• Enterprise Ireland Innovation Voucher, IV-2008
  1047, 2008/09
• Navigatour'',
• Backseat Tour Guides Ltd., Letterkenny, Co.
  Donegal, Ireland,
• Anton Hutton, School of Creative Arts EyeSpyFX
  
• Prof. Paul Mc Kevitt.

54

• University of Ulster, Office of Innovation,
• Proof-of-Principle (PoP), 2008/09
• Identity cards employing steganography,
• Abbas Cheddad, Joan Condell, Kevin Curran, Paul
  Mc Kevitt.
• University of Ulster, Office of Innovation,
• Proof-of-Principle (PoP), 2009
• System and method for streaming music repair
• and error concealment,
• Jonathan Doherty, Kevin Curran, Paul Mc Kevitt.
• Invest NI Proof-of-Concept (PoC) project,
  2009/10
• Secure digital watermarks (SDW)
• secure data hiding in image frames using
  steganography,
• Joan Condell, Kevin Curran, Paul Mc Kevitt.

55
Creative People
56
Conclusion future work

• Imagineering creative technologies
• MultiModal semantic representation
• Affective computing
• Deployment on mobile devices
• Applications in creative industries