The Leeds Met Centre for Creative Technology

1
The Leeds MetCentre for Creative Technology

• Research, Development and Practise
• 1. November 2007
• Reinhold Behringer
• Lecture at Danube University Krems

2
About Presenter

• Physicist, Software Engineer/Scientist.
• Past work and interest
• Autonomous system (road vehicles).
• Real-time computer vision applications.
• Augmented Reality (AR) concept demonstrations.
• Mobile and Wearable Computing.

3
Curriculum Vitae

• 1990-1996 computer vision for automated driving
  (FP-3 EU project PROMETHEUS).
• 1996-2005 Rockwell Scientific (RSC), US (now
  Teledyne Scientific)
• Since Sept. 2005 Professor of Creative
  Technology at Leeds Metropolitan University.
• http//www.leedsmet.ac.uk/inn/ReinholdBehringer.ht
  m

4
About Leeds Metropolitan University

• 52,000 students
• 3,500 members of staff
• Annual expenditure 200 M (2006).
• Leading the Regional University Network (RUN),
  with total of 300,000 students
• Partners DUK, Fachhochschule Vorarlberg, NY
  State University at Farmingdale, Hongkong
  Vocational Training College.

5
History

• 1824 Leeds Mechanics Institute.
• Mergers with other educational institutions in
  Leeds.
• 1907 Headingley Campus built for Leeds Training
  College.
• 1970 Leeds Polytechnic.
• 1992 Leeds Metropolitan University becomes
  full university, with right to award PhD degree.

6
Universities in UK

• Russell Group of Universities
  http//www.russellgroup.ac.uk/ 20
  research-intensive universities.
• Founded in 1994, at a meeting in Russell Hotel.
• Oxbridge U. of Oxford, U. of Cambridge.
• 65 (over 1.8billion) of UK Universities'
  research grant and contract income, 56 of all
  doctorates awarded in the United Kingdom.
• The other 106 Universities
• Post-1992 Universities.

7
The Russell Group

• The University of BirminghamThe University of
  BristolThe University of CambridgeCardiff
  UniversityThe University of EdinburghThe
  University of GlasgowImperial College
  LondonKing's College LondonThe University of
  LeedsThe University of LiverpoolLondon School
  of Economics and Political ScienceThe University
  of ManchesterNewcastle University The
  University of NottinghamThe University of
  OxfordQueen's University BelfastThe University
  of SheffieldThe University of SouthamptonUnivers
  ity College LondonThe University of Warwick

8
Funding for Research

• From Higher Education Funding Council for England
  (HEFCE) http//www.hefce.ac.uk/
• Funding depends on rating of research
• Research Assessment Exercise (RAE) is conducted
  every 7-8 years http//www.rae.ac.uk/
• Leads to rating, which translates into funding
  for research activities.
• Funding Councils
• Specific proposals need to be written for
  research projects.

9
Leeds Met Striving to Excellence

• 2006 Shortlisted for Times Higher University of
  the Year
• Honorary Mention.
• Award for best contribution to Community.
• 2007 Award by People and Planet for most
  environmentally friendly UK university.
• Vice Chancellor Prof. Simon Lee
• Since 2003, implemented many changes.
• Research activities are high on the agenda
  supported by foundation of Research Centres.

10
Centres at Leeds Met

• Purpose of Centres
• Coordinate research activities,
• Foster interdisciplinary cross-faculty
  cooperation,
• Collaborate with external partners.
• Are organised around the interest of professors.
• Examples of other Centres
• Centre for Mobile and Converging Technology
• i-Centre (Technology-Enhanced Learning)
• PRAXIS Centre (Peace and Conflict Resolution)
• Centre for Research into Higher Education

11
This Centre

• Creative Technology
• Tools for the creation of artworks (visual,
  acoustic).
• Enable human creativity.
• Lower the threshold of interacting with computing
  devices.
• Ways of interaction with and engaging the
  audience.
• Works, created with these tools
• Multimedia installations,
• Computer music,
• Computer graphics,
• Computer games.
• From software development to artistic expression.
• From graphics (visual) to audio/music (acoustic).
• http//creativetech.inn.leedsmet.ac.uk

12
Launch Event

• 9. July 2007 launch event evening
• With demonstrations by members of centre.

13
Creative Technology

• Is linked to Entertainment, but has serious
  engineering behind it
• Visual and acoustic processing,
• Analogue and digital processing,
• Software development,
• Sensors, networking,
• Collaboration, interaction.
• Intelligent systems, artificial
  intelligence/creativity.

14
Graphics and Video
Creative Technology
Games
Augmented/Mixed Reality
Music and Audio
Human-Computer Interaction Technology
Artificial Intelligence
Robotics
15
Vision for the Centre

• Develop novel technologies for use in the
  creative realm.
• Create novel multimedia art installations.
• Improve human-computer interaction systems.
• Stimulate interdisciplinary collaboration
• Across different work areas,
• Across faculties,
• Beyond the boundaries of our university, with
  external partners.
• Apply Creative Technology in other areas.

16
Leeds Metropolitan University
Research Centre for Creative Technology
Innovation North
Other Centres
Other External Institutions
17
Members

• Membership is open to everyone.
• Is defined by work topic, interest in
  collaboration, and concrete joint projects.
• Can be from
• Innovation North,
• Other faculties at Leeds Met,
• External institutions.
• External Partner collaboration envisioned.

18
Activities

• Many individual research and work topics, worked
  on by individual researchers.
• Goal
• To combine work of several researchers into
  larger-scale projects.
• To leverage different capabilities and interests
  of members of the centre.

19
Research, Development, Practise

• Three core pillars
• Research
• Uncovering basic principles in creativity,
  interaction, aesthetics.
• Development
• Develop novel ways of interaction with computing
  systems, by improving intelligence of these
  systems.
• Practice
• Creation of art, music, multimedia.
• PostGraduate Education
• Centre will provide topics for Master and PhD
  theses, from repository of its projects.

20
Activity Clusters

• Computer Music.
• Computer Graphics.
• Machine Intelligence.
• There are many areas of overlap between these
  clusters.

21
Activity Matrix
Research Development Practise
Computer Music Aesthetics, sound creation, automatic composition Interfaces to instruments Compositions Performances
Computer Graphics Mixed/augmented reality Animation, simulation, computer games Artistic graphics, interactive installations
Machine Intelligence Language understanding, computer vision Human-computer interface Applications in learning and teaching
22
Computer Music

• Novel interfaces to music instruments.
• Electro-acoustic compositions.
• Music and sound in computer games.
• Automatic music following and beat tracking.
• Automated composition (artificial creativity).
• Digital Music Research Network.

23
Computer Graphics

• Animation.
• Photography, Video, Film.
• 360-degree photography.
• Immersive virtual reality.
• Mixed and augmented reality.

24
Machine Intelligence

• Artificial intelligence.
• Automated language understanding.
• Computer vision.
• Intelligent Human-Computer Interfaces.

25
General Topics

• Software development
• Methods and technology.
• Sensor integration.
• Non-linear narrative for games and film.
• Many applications of these technologies also in
  non-creative areas.

26
Contacts with other Faculties

• Carnegie Faculty of Sport and Education
• Joint project for simulation of physiological
  processes Virtual Runner game.
• Consideration of application of virtual reality
  in context of psychology.
• Faculty of Health
• Automatic wheelchair for collision avoidance.

27
Facilities

• Music Studios
• Installed in James Graham, 2nd floor, Dec. 2006
• Motion Capture room for capturing human movements
  into digital animation.
• Blue screen facilities.
• Foley Room for recording sound effects.
• Computer laboratories.

28
Music Studios

• 5 control rooms/studios and 5 production rooms.
• Open since December 2006.

29
Music Studios Equipment

• Audient 8024 36 channel desk complete with ASP510
  controller and Genelec 1038 and 1032s monitors
  for 5.1 mixing and a Protools HD3 system with
  Mackie controller.
• Soundcraft Ghost desks and Dynaudio BM15A
  monitoring, equipped for 5.1 monitoring.
• Production rooms are equipped with Protools 002s
  and Mackie controllers and a Mackie DXB console
  and a Protools HD3 system.
• All rooms are equipped with outboard
  effects/dynamics from TC Electronics, Lexicon,
  Yamaha, TL Audio and Drawmer.

30
Motion Capture System

• Optical Motion Capture System developed by Vicon.
• 10 infrared cameras.
• Full body motion capture, face capture.

31
Examples of Projects

• Research
• Development
• Practice

32
Structural Playability

• Applying psychological facts to analyse
  playability of computer games.
• Study suitability for eLearning.
• Contact
• Hazel Bradshaw,
• H.Bradshaw_at_leedsmet.ac.uk

33
Medical Simulation

• Visualisation of hand surgery.
• Usable for
• Patient education.
• Training for surgeon.
• Contact
• Steve Wilkinson
• S.Wilkinson_at_leedsmet.ac.uk

34
Computer Graphics

• Genetic Art
• Organic forms, based on algorithmic renditions.
• Contact
• William Latham, Goldsmiths College
• mas01whl_at_gold.ac.uk
• Visiting Professor at Leeds Met.

35
Mobile Games

• Games on PDAs or mobile phones
• Limited hardware resources.
• Contact
• Ben Dalton
• b.dalton_at_leedsmet.ac.uk

36
Bluetooth Multiplayer Game

• Uses Java to Micro-Edition (J2ME).
• Suitable for mobile phones.
• Contact
• Muthu Ramachandran M.Ramachandran_at_leedsmet.ac.uk

37
Mobile Games

• Games on PDAs or mobile phones
• Limited hardware resources.
• Contact
• Ben Dalton
• b.dalton_at_leedsmet.ac.uk

38
Augmented Reality

• Goal
• To develop a framework for seamless operation
  of Augmented Reality applications.
• Issues
• Transition from cooperative infrastructure into
  non-cooperative environment.
• Portability and scalability on various hardware
  platforms.
• Large market near future mobile camera phones.

39
Geographic Visualisation of Photos

• Fusion of individual photographs with location
  Geo-tagging
• Services Flickr, Panoramio
• Current state overlay on map.
• Future integrate photo with 3D model
• Either by texture mapping onto 3D model
• Or by overlay into the users view.

40
(No Transcript)
41
Location of Picture in Map
With Google Maps display
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
VibraChair

• Student project.
• Contacts
• Matthew Ian Smith
• M.Smith4794_at_student.leedsmet.ac.uk
• Lionel Myers
• L.Myers5892_at_student.leedsmet.ac.uk

47
interactiveVIDEO approaches to developing an
interactive movie system
Brian Larkman

• the entertainment limelight
• CGI influenced movies increasingly computer
  generated vs. 3D real-time video games
  increasingly cinematic.
• convergence is still some way off largely for
  commercial and philosophical differences - not
  technological.

research direction Interactive Video and
especially Panoramic Video, is the major topic of
the presentation.
Contact Brian Larkman, b.larkman_at_leedsmet.ac.uk
Jonathan Wilkinson 360 video Paris
Rana Gangari Interactive Video The Car Journey
48
Eye Tracking

• Monitoring the players eye motion during game
  sequences with desktop Tobii eye tracking system.
• Commercialisation in Usability North.
• Interactive installation Re-Possessed
• Analysing Hitchcocks Vertigo.
• Contacts
• Tony Renshaw T.Renshaw_at_leedsmet.ac.uk
• Richard Stevens R.Stevens_at_leedsmet.ac.uk

http//www.re-possessed.com/
49
Repossessed explores the evolution of
spectatorship in the digital era. An
interdisciplinary team of writers, artists and
technologists have collaborated to produce
artworks in which the audience actively
participates. Whilst under electronic
surveillance visitors register their responses
and continually re-make the content of the
exhibits. Inspired by Alfred Hitchcocks
Vertigo questions of power, freedom and desire
are never far from the surface, emerging as
metaphors for the relationships between audience,
artist and medium.
50
Creative Technology
RePossessed ReViewed ReSearch
51
Andy Brennan Clap n smack
Masters student on the Sound and Music for
Interactive Games course.
New interfaces for interacting with sound and
music
Features interactive music, motion detection, and
fun !!
52
Real-time Motion Capture

• This demonstration will show real time motion
  capture using the Vicon mocap system and
  AutoDesks Motionbuilder

Contacts Duncan Folley d.folley_at_leedsmet.ac.uk
Anthony Partington a.partington_at_leedsmet.ac.uk

53
Instant Messaging with Expressive Avatars

• two people can meet and chat over a distance
• communication via
• text chat
• emoticons
• avatar eye-gaze
• facial expressions of emotion
• Contact
• Marc Fabri
• m.fabri_at_leedsmet.ac.uk

54
So what?

• facial expressions can get you really involved
• but you may forget about what you originally
  intended
• facial expressions can make you more likeable
• if you use them in a trust-building, empathic
  way
• facial expressions can get you what you want
• if you use them to deliberately influence the
  other person
• so potentially a very powerful tool for
  creating trust relationships over a distance!

Key Question How do these expressive avatars
affect user experience and user behaviour?
55
Sensing for Interaction

• Sensing of user interaction with virtual object
  through physical simulation
• Inclination of virtual world includes virtual
  gravity.
• Blowing towards virtual object.
• Contact
• Bal Singh
• b.singh_at_leedsmet.ac.uk

56
Audio and Music

• Linking audio with games
• Acoustic hotspots.
• Head-related binaural audio.
• Contact
• Richard Stevens, r.stevens_at_leedsmet.ac.uk
• Computer-generated music
• Algorithmic composition.
• Electro-acoustic experimental music performance
  and composition.
• Contacts
• Nikos Stavropoulus, n.stavropoulos_at_leedsmet.ac.uk
• Ian Gibson, i.gibson_at_leedsmet.ac.uk
• Adam Stansbie, a.stansbie_at_leedsmet.ac.uk

57
Interfaces to Music Instruments

• Virtual Theremin
• Uses tracking methodology to emulate RF
  interaction of real theremin.
• Contact Paul Conway, p.conway_at_leedsmet.ac.uk
• Touchpad interface for playing instruments
• Pressure sensitive 2D pad for creating sounds.
• Contact Dave Raybould, d.raybould_at_leedsmet.ac.uk
  

58
The Call Centre Installation

• Ian Gibson, Chris Newell, Alistair Edwards

• Collaborative project with University of York and
  University of Hull
• An adapted phone with audio speakers, speech
  input and button input. Proximity sensors and
  floor pressure pads used to generate ambient
  effects
• Speech system exploring the role of pauses and
  breaths with actors
• Artistic intent To recreate the feelings of
  frustration and isolation often associated with
  call centres
• Alan Ayckbourn to provide future guidance with
  scripts for a related project

59
Sound Diffusion The Performance of
Electroacoustic Music

• Deployment of multiple loudspeakers in the
  concert hall
• Facilitates the performance of electroacoustic
  music
• Enables the articulation of spatial cues and
  trajectories within the music

• Adam Stansbie a.stansbie_at_leedsmet.ac.uk

60
Playing Music

• The original problem
• When creating electronic renditions of music
  (e.g. in MIDI notation), it is very difficult to
  denote and enter music tempo.
• Advanced problem
• Simultaneous play of orchestra / band with an
  autonomous computer music instrument (playing
  from a given score) the computer as an
  instrumentalist.

61
Example of Tempo Flow
Heuristically determined timeline of the
beginning of Gustav Mahlers Adagietto (5.
Symphony, 4th movement)
Beats per minute
time
62
Conducting Music

• Tempo flow is felt by a musician
• If playing, tempo is done intuitively.
• For orchestra, conductor provides tempo reference
  to players.

63
Proposed Solution

Visual tracking of baton / hand
t
Create tempo map
Bridge the latencies by prediction and
extrapolation, based on music score.
Digital score in sequencer.
64
Possible Applications

• Automated events for performance, e.g. lights,
  stage events, sounds, synchronous to music and
  determined by expression of conductor.
• Additional accompanying electronic instrument,
  playing synchronously with orchestra.
• Shaping time-flow of digital synthesizer
  rendition of music.

65
Visual Tracking

• Hand tracking
• Determine colour blobs.
• Follow the motion.
• For robustness
• Employ internal predictive model.

66
System Concept
2D point sequence 50 or 60 fps
Visual Tracking
Interpolation of motion timeline
Spline curve of motion
Acoustic Tracking
Analysis of motion timeline
Tempo and beat position Possible expression
Notes from MIDI file
Extrapolation to current time
Play synthesizer
67
Computer Vision and Music

• Interact with synthesizer through gesture
  interface, using computer vision.

68
Vertical Baton Motion
69
Vertical Baton Speed
70
Hurdles

• Overall hurdle real-time computation performance
  is necessary.
• Robustness stay on target (hand), do not get
  distracted by visual clutter.
• Automated reset / detection in case of tracking
  loss.

71
Output in Timeline

• Rhythm from motion pattern
• Tempo is determined by distance of peaks.
• Location in score
• bouncing motion determines beat.
• Allows time synchronization of digital score.
• Expression / musical phrasing
• From motion amplitude or speed.

72
Envisioned Capabilities

• Future system to be used in real performance
  needs the following capabilities
• Automatic re-synchronisation during rehearsal
  phase, where parts of the music are repeatedly
  rehearsed.
• Speech recognition for capturing conductor
  commands (repeat from bar 90).
• Potential extension
• Recognition of facial expression of conductor.

73
Acoustic Tempo Adaptation

• Microphone captures audio of playing band /
  orchestra.
• Computing system determines tempo and beat.
• Allows automatic accompaniment or playing as a
  computer band / orchestra member.
• PhD studentship
• Michael Ward, mjward_at_vodafone.net

74
Intelligent Systems

• Computer vision
• Application-oriented use as an interface to
  computing systems.
• Natural language processing
• Contacts
• Elizabeth Guest, e.guest_at_leedsmet.ac.uk
• John Elliott, j.elliott_at_leedsmet.ac.uk

75
Detecting Languageness
Dr John Elliott Centre for Creative Technology
Innovation North - Faculty of Information and
Technology Leeds Metropolitan University
76
What do I do? (John Elliott)
In a nutshell, I model complex systems.

• This has included and includes
• Language Discovery Decipherment
• SETI computational modelling and identification
  of signals naturally occurring phenomena
• Interstellar message design and construction for
  SETI
• Modelling Universal Language Structures
• Calculating the Intelligence of Dolphins
• Information Visualisation of Complex Data
  Structures e.g., Visualising the Cohesion of
  Linguistic Objects
• An Ethical Engine / Moral Compass for Robotic
  Soldiers.

77
The Language Machine What is languageness? Is
there something unique to natural language that
distinguishes itself in the 'signal universe'?
Can unsupervised computational analysis of a
signal's surface structure, detect the lingua ex
machina signatures of cognitive, orthotactic and
ontological constraints in which a natural
language operates? The problem goal is therefore
to separate language from non-language without
dialogue, and learn something about the structure
of language in the passing. The language may
not be human (animals, aliens, computers...), the
perceptual space can be unknown, and we cannot
assume human language structure but must begin
somewhere. We need to approach language from a
naive viewpoint, in effect, increasing our
ignorance and assuming as little as possible.
78
Using an approach, which draws from the areas of
computational linguistics, corpus linguistics,
information theory, computer visualisation,
psychology, neuroscience and statistics, we will
endeavour to isolate computational linguistic
universals by analysing a representative sample
set of the human chorus. Methods developed
are designed to work without any specific
in-built prior knowledge of an individual system,
for the filtration of inter-galactic objets
trouvés and the identification of language
structure at its varying levels of abstraction
from the base physical level to the
parts-of-speech, where behavioural syntax meets
semantics.
79
Towards Automatic Language Understanding

• Visualisation of language semantics.
• Contact
• John Elliott, j.elliott_at_leedsmet.ac.uk
• Elisabeth Guest, e.guest_at_leedmet.ac.uk

80
Future Perspective

• Goal
• To establish external partnerships
• To obtain funding for research activities
• Centre for Creative Technology gives a branding
  to our research activities.

81
Summary

• Centre for Creative Technology has many different
  topics.
• We offer collaboration with external partners
• Joint bids for funding.
• Consultation.
• Technology portfolio goes well beyond classical
  creative domain
• Many possible applications.
• Exciting perspectives for future work.

82
Contact

• Reinhold Behringer
• Running Stream Professor of Creative Technology
• 101 Caedmon Hall
• Headingley
• Leeds, LS6 3QS
• 0113-812-3716
• r.behringer_at_leedsmet.ac.uk
• http//creativetech.inn.leedsmet.ac.uk