Audio and Video Communication in Multiplayer Games through Generic Networking Middleware

1
Audio and Video Communication in Multiplayer
Games through Generic Networking Middleware

• Maarten Wijnants
• Wim Lamotte

2
Outline

• Introduction
• NetworkIntelligence Middleware
• Application and Network Awareness
• Generic but Extensible
• Integration in Existing NVE Application
• Experimental Results
• Conclusions Future Work
• Questions

3
Introduction

• Advancements in many different areas of game
  design and development
• However still little support for the real-time
  streaming of multimedia content in on-line
  multiplayer games
• Nonetheless has a number of interesting
  applications
• In-game audiovisual communication
• Dynamic streaming of multimedia content based on
  user location/interest, (e.g. video
  advertisements)

4
Introduction

• Audio and video communication
• Easier to input and less awkward than textual
  chat
• Inherently transfers emotional state of players
• Attaches a distinct voice and face to nicknames
• Enables players to communicate in a natural and
  immersive way
• Current on-line multiplayer games
• Still support textual chat
• Some also support voice communication
• Virtually no support for video communication

5
Introduction

• The lack of support for multimedia streaming in
  games can be attributed to
• The high bandwidth requirements of these streams
• Game developers need to implement it efficiently
• Client network bandwidth is a scarce resource
• Games should be playable on both
• low-bandwidth dial-in modems
• high-bandwidth DSL or cable modem connections
• requires extra code to be written
• We present our networking middleware that
  facilitates inclusion of multimedia streaming

6
NetworkIntelligence Middleware

• Our networking middleware consists of a number of
  interconnected proxy servers
• Proxies are positioned at edge of the network
• Clients need to connect to a proxy
• Proxy is responsible for
• forwarding relevant network streams to the client
• routing packets sent by client to their
  destination
• Each proxy is network aware
• Has knowledge of the transportation network
• Proxy periodically probes the network links of
  connected clients ? throughput, delay,
• Proxy records bandwidth usage of streams

7
NetworkIntelligence Middleware

• Each proxy is also application aware
• Has high-level knowledge of the application
• Provided by the client application
• E.g. client could keep proxy up-to-date regarding
  the relative importance of the different
  multimedia streams that are currently used in the
  application
• Developers can utilize a provided
  NetworkIntelligence layer to take care of the
  client/proxy communication
• Only needs to be interfaced with the clients
  awareness manager
• Automatically responds to probing packets
• Continuously queries the awareness manager
• Differentiates our middleware from other proxy
  systems!

8
NetworkIntelligence Middleware

• Based on its compound awareness, the middleware
  intelligently manages the bandwidth of connected
  clients
• E.g. players move around in the virtual world
• Relative importance of individual streams changes
• Proxy can detect this due to its application
  awareness
• Changes allocation of client's bandwidth
  accordingly
• E.g. middleware can intelligently react to
  fluctuations in a client's downstream capacity
• Proxy can detect this due to its network
  awareness
• When bandwidth drops, the middleware determines
  which streams should be reduced in quality or
  blocked ? client bandwidth no longer exceeded

9
NetworkIntelligence Middleware

• Middleware should be applicable in a wide range
  of networked applications
• We therefore opted for a generic design
• However, many applications require specific
  middleware functionality
• Proxy plug-in mechanism ensures extensibility
• Application developers can implement own plug-in
• E.g. video transcoding plug-in
• It should cost developers minimal effort to
  integrate our middleware in their application
• Implementing a plug-in deriving from a base
  class and adding desired functionality
• NILayer prevents that the client application
  needs to be completely recoded

10
Integration in Existing NVE App

• We integrated our middleware in our in-house
  developed NVE framework
• Supports video-based avatars
• Video streams need to be exchanged between users
  in real-time
• Supports voice communication
• Closely resembles a multiplayer game

11
Integration in Existing NVE App

• Some technical details of the framework
• The awareness manager of the NVE is region-based
• Each region has a MC address for distributing
  events
• AM determines which regions this client should be
  aware of
• Only the multicast groups of these regions are
  joined
• Uses 3 distinct video qualities
• For reasons of scalability (see table)
• Each region has 3 distinct video multicast
  addresses
• Video awareness manager selects the video quality
• Separate multicast group per region for audio
  streams

12
Integration in Existing NVE App

• Actual integration of the middleware in the NVE
  proved to be straightforward. It sufficed to
• Insert the NILayer in the client software
• Link the NILayer with the different awareness
  managers
• The NILayer
• Extracts information about the currently selected
  regions
• Monitors the virtual distance between the local
  user and the other clients
• Informs the proxy of relative importance of the
  different multimedia streams, based on this
  positional information
• More priority is given to streams coming from
  nearby clients
• This is where the middleware gets its application
  awareness

13
Experimental Results

• Experiment
• 3 unmodified NVE clients C1, C2 and C3
• Video
• Audio
• 2 clients connected through our middleware PA
  and PB
• No audio or video
• These 5 clients were located in the same region
  of the virtual world (see figure for exact client
  positioning)
• Client PA
• Fixed downstream bandwidth (320 Kbps)
• Moved around in the virtual world
• Client PB
• Fluctuating bandwidth 440-gt320-gt600 Kbps
• Remained stationary in the virtual world

14
Experimental Results

• Multimedia network packets received by client PA

15
Experimental Results

• Multimedia network packets received by client PB

16
Conclusions

• What does this experiment learn us?
• The current downstream capacity of a client's
  network connection is always respected
• This is due to the middleware's network awareness
• Stream playback at client-side will normally
  improve
• The middleware exploits its application awareness
  to intelligently distribute a client's downstream
  capacity over the different multimedia streams
• Less important streams are reduced in quality or
  blocked before more significant streams are
  altered
• Clients receive the streams that are most
  important to them at an as high as possible
  quality
• Developers need not spend time on these issues
• Our middleware facilitates the integration of
  efficient real-time streaming functionality in
  networked applications

17
Future Work

• The networking middleware presented here is a
  work in progress
• In the future we would like to
• Add device awareness to the middleware
• Develop some additional plug-ins for our
  networking middleware
• E.g. a plug-in which adds 3D audio mixing
  functionality
• Integrate our work into a shared workspace
  application
• Goal is to efficiently support real-time meetings
  between both collocated and distributed team
  members
• Audio and video communication will play an
  important role

18
Questions?
19
Additional Results

• C1 stops transmitting audio ? C2 video boosted