Video Adaptation: Concept, Technologies, and Open Issues

1
Video Adaptation Concept, Technologies, and Open
Issues

• SHIH-FU CHANG
• Presented by Jun-Cheng Chen
• 03/17/2005

2
Outline

• Introduction
• Unified Conceptual Framework and Technology
  Taxonomy
• Active Research Areas
• Open Issues
• Support of Adaptation in International Standards
• Conclusion

3
Introduction(1/3)

• Video adaptation
• Emerging field in pervasive media applications
  (such as PC, TV, PDA, or cellular phone).
• Transform the input video to an output in video
  or augmented multimedia.
• utilize manipulations at multiple levels
• Signal, structural, semantic.
• constrained optimization
• Its objective is to maximize the utility of final
  presentation while satisfying various constraints
  (such as bandwidth).

4
Introduction(2/3)

• Video adaptation differs from video coding in its
  scope and intended application locations.
• signal level vs structural level vs semantic
  level, tanscoding vs selection vs summarization,
  bandwidth vs power vs
  time-constrained.
• Often In the intermediate location, such as proxy
  between server and client.
• Video adaptation is still a relatively less
  defined field.
• No coherent set of concepts, terminologies, or
  issues defined over well-formulated problems

5
Introduction(3/3)
6
(No Transcript)
7
Unified Conceptual Framework and Technology
Taxonomy
8
Unified Conceptual Framework and Technology
Taxonomy

• Entity
• Defined to refer to the basic unit of video that
  undergoes the adaptation process.
• Different levels, such pixel, object, frame,
  shot, scene, syntactic components, and semantic
  components
• Each entity is associated with certain resource
  requirements and utility values.

9
Unified Conceptual Framework and Technology
Taxonomy

• Utility
• It represents the quality or users satisfaction
  of the video content (such as PSNR).
• Adaptations space
• The space of feasible adaptation for a given
  video entity.
• Different adaptation operators can be defined for
  different types of entities.
  (ex a
  video frame can be reduced in resolution, spatial
  quality, or skipped to reduce bandwidth cost.)

10
Unified Conceptual Framework and Technology
Taxonomy

• Systematic Procedure for Designing Video
  Adaptation Technologies
• Video Adaptation Taxonomy
• Format transcoding
• Selection/Reduction
• Replacement
• Synthesis

11
Systematic Procedure for Designing Video
Adaptation Technologies (1/3)

• Identify the adequate entities for adaptation.
• Identify the feasible adaptation operators.
• Develop models for measuring and estimating the
  resource and utility values associated with video
  entities undergoing identified operators.

12
Systematic Procedure for Designing Video
Adaptation Technologies(2/3)

• Given user preferences and constraints on
  resource or utility, develop strategies to find
  the optimal adaptation operator(s) satisfying the
  constraints.

Problem formulation Given a content entity E,
user preferences, and resource constraints Cr,
find the optimal adaptation operations Aopt
within the feasible adaptation region so that the
utility of the adapted entity e is maximized.
13
Systematic Procedure for Designing Video
Adaptation Technologies(3/3)
10.Y. Wang, J.-G. Kim, and S.-F. Chang,
Content-based utility function prediction for
real-time MPEG-4 transcoding, presented at
the IEEE Int. Conf. Image Processing, Barcelona,
Spain, 2003.
14
Video Adaptation Taxonomy

• Format transcoding
• To transcode video from one format to another, in
  order to make the video compatible with the new
  usage environment.
• Selection/Reduction
• Select some components of the entity and reduce
  them for saving resources.
• Example We can change the bit rate, frame rate
  or resolution for shots and frames in a video
  clip,

15
Video Adaptation Taxonomy

• Replacement
• Replace selected elements in a video entity with
  less expensive counterparts, while aiming at
  preserving the overall perceived utility.
• Example a video sequence may be replaced with
  key frames.
• Synthesis
• Synthesize new content presentations based on
  analysis results.
• The goal is to provide a more comprehensive
  experience or a more efficient tool for
  navigation.

16
Video Adaptation Taxonomy
17
Active Research Areas

• Semantic Event-Based Adaptation
• Structural-Level Adaptation
• Transcoding
• Rapid Fast-Forward Drastic Temporal Condensation

18
Semantic Event-Based Adaptation

• Doing video analysis for events and boundaries
  detection.
• By using the information of video content, such
  as the scoring points in sports video, and the
  breaking news in broadcast programs.
• Results of video event analysis can be utilized
  to produce different forms of adaptation.

19
Semantic Event-Based Adaptation
In this way, we can save bandwidth or the total
viewing duration.
20
Semantic Event-Based Adaptation

• Example The percentage of important segments in
  the whole stream (such as sports broadcast).
• They found non-important segments occupy more
  than 50 of duration.
• Their system which focuses on sports can reach
  higher than 90 accuracy

6S.-F. Chang, D. Zhong, and R. Kumar,
Real-time content-based adaptive streaming of
sports video, presented at the IEEE Workshop
Content-Based Access to Video/Image Library, IEEE
CVPR Conf., Honolulu, Hawaii, Dec. 2001.
21
Structural-Level Adaptation

• The structures in video are caused by event
  occurrence orders, camera control patterns, and
  the final editing process.
• Exploration of relations of structural elements
  provides great potential for video adaptation.
• Example
• Key frame extraction
• Mosaicing

22
Structural-Level Adaptation
23
Transcoding

• Signal level adaptation
• Involving various manipulations of coded
  representations and issues of bit allocation
• Manipulation of video signals
• Spatial change spatial resolution, i.e., frame
  size.
• Precision change the bit plane depth, color
  depth, or the step size for quantizing the
  transform coefficients.
• Temporal change the frame rate
• Object transmit a subset of objects

24
(No Transcript)
25
Rapid Fast-Forward Drastic Temporal Condensation

• Video skimming
• Bad ways
• Increase the frame rate of the player.
• Make the audio track unrecognizable.
• Uniformly sample the frames in the original
  sequence.
• Important video frames may be skipped and audio
  content may be unrecognizable.
• Extract keyframes to form shorter image sequence.
• Lose the synchronization between video and the
  associated audio track.

26
Rapid Fast-Forward Drastic Temporal Condensation

• 14.H. Sundaram, L. Xie, and S.-F. Chang, A
  utility framework for the automatic generation of
  audio-visual skims, presented at the
  ACMMultimedia Conf., Juan Les Pins, France, 2002.
• Adaptation entities video shots.
• Adaptation operations length trimming or
  dropping of individual shots.
• The problem was formulated as constrained
  optimization.
• Constraints viewing time, dialogs, key phrases,
  key audio, etc.

27
Open Issues

• Define Utility Measures and User Preferences
• Resolve Ambiguity in Specifying Adaptation
  Operation
• Relations Among Adaptation, Utility, and Resource
• Search Optimal Solutions in Large Spaces
• Design End-to-End Integrated Systems

28
Define Utility Measures and User Preferences

• It is difficult to define a universal measure
  for different levels or dimensions.
• Levels include Perceptual, semantic, and
  comprehensiveness.
• Signal-level measures are often inadequate m
• many high-level operations such as shot removal,
  modality replacement, etc.
• These operations also cause complex changes to
  content at other levels.
• Users preferences often vary with content, task,
  and usage environment.

29
Define Utility Measures and User Preferences

• Some possible alternatives
• Infer user preferences based on the usage
  history.
• Correlate subjective preferences with content
  characteristics.

30
Resolve Ambiguity in Specifying Adaptation
Operation

• Some adaptation operations are not unambiguously
  defined.
• remove the second half of each shot
• drop 10 of transform coefficients
• Some possible ways
• Restrict adaptation operation only on unambiguous
  representation formats, such as JPEG 2000 and
  MPEG-4 fine grained scalable schemes.
• Estimate the bound of variations in resource and
  utility.

31
Relation Among Adaptation, Utility, and Resource

• Relations among adaptation, resource, and utility
  are often complex.
• The complexity is especially high when the
  dimensionality of each space is high.
• Potential approaches
• Sample the adaptation space and store the
  corresponding resource and utility values.
• Decompose the adaptation space into
  low-dimensional spaces and sample each subspace
  separately.
• These schemes may lose the chance of exploring
  correlations among different dimensions.

32
Search Optimal Solution in Large Spaces

• Exploration of the above multi-space relations
  often leads to formulation of constrained
  optimization.
• Analytical solutions may exist for some cases.
• example rate-distortion model (low dimensional
  cases)
• Adaptation space quantization
• Resource space bit rate
• Utility space SNR
• In general, each space may have high
  dimensionality and the relations across spaces
  may be complex.

33
Design End-to-End Integrated Systems

• Difficulties
• Require joint consider joint consideration of the
  adaptation subsystem with other subsystems.
• Inconsistent and imperfect content analysis
  subsystem
• Rights management
• Content owners impose many restrictions on video
  content altering.

34
Design End-to-End Integrated Systems

• Possible solutions
• Adopt modular designs of subsystems and provide
  well-defined abstraction of requirements and
  performance of each subsystem.
• Follow the international standard which are
  needed for describing information related to
  media rights management.

35
Support of Adaptation in International Standards

• Mpeg-7 Content Descriptions
• Mpeg-21 Digital Item Adaptation
• Standardized Adaptation Framework

36
Mpeg-7 Content Descriptions

• Descriptors (Ds) Description schemes (DSs)
• XML
• Usage history DS
• UserPreferences DS (creators, time periods,
  locations, etc.)
• Summary descriptions
• Variation descriptions
• Transcoding hints
• Motion hints (for guiding motion-based
  transcoding methods)
• Semantic importance hints (for guiding rate
  control)
• Etc

37
Mpeg-21 Digital Item Adaptation

• Digital Item Adaptation(MPEG21 part7)
• Address an extended scope of issues related to
  adaptation of digital multimedia content.
• Usage environment descriptions (UEDs)
• Used to describe a wide array of user, terminal
  capabilities, network, and natural environment
  characteristics.
• Universal constraints description (UCD) tool
• similar to UEDs
• more explicit
• AdaptationQos tool
• Relations between constraints
• Feasible adaptation operations and associated
  utilities

38
Standardized Adaptation Framework
39
Conclusion

• Despite the burgeoning activities and advances,
  this field is in need of an analytical foundation
  and solutions to many challenging open issues.
• It is worthwhile to note that solutions to most
  of the above identified open issues require joint
  consideration of adaptation with several other
  closely related issues, such as analysis of video
  content, rights management of digital content,
  etc.