Trajectory Analysis of Broadcast Soccer Videos

1
Trajectory Analysis of Broadcast Soccer Videos
by Prof. Jayanta Mukherjee jay_at_cse.iitkgp.ernet.i
n
Computer Science and Engineering
Department Indian Institute of Technology,
Kharagpur
2
Collaborators

• V. Pallavi --- research scholar.
• Prof. A.K. Majumdar, CSE
• Prof. Shamik Sural, SIT

3
OUTLINE

• Motivation and Objective
• State Based Video Model
• Extraction of Features
• Trajectory Detection
• States and Event Detection

4
Motivation

• Increasing availability of soccer videos
• Soccer videos appeal to a large audience
• Processing of soccer videos to deliver it over
  narrow band networks
• Relevance of soccer videos drops significantly
  after a short period of time
• Therefore soccer video analysis needs to be made
  automatic and the results must be semantically
  meaningful

5
State based Video Model
Video data model representation of
information contained in the unstructured video
in order to support users queries.
State based model states of soccer video
objects and their transitions (due to some event).
6
State Chart Diagram for Ball Possession
7
Immediate Goal
Our objective is to identify these states and
their transitions by analyzing the unstructured
video.
8
Detection of States and Events (contd..)
In a soccer match, the ball possession states
may be any of the following

• possession of Team A
• possession of Team B
• both the teams fighting to possess the ball
• ball in possession of none during a break

9
Features Used

• Cinematic Features
• Shot Transitions
• Shot Types
• Shot Durations
• Object Based Features
• Players
• Ball
• Billboards
• Field Descriptors

10
Cinematic Features
Feature Extraction (contd ..)?

• Shot is a continuous sequence of frames captured
  from the same camera in a video.
• Shot detection algorithms segment videos into
  shots automatically.
• Shot classification algorithms partitions a video
  stream into a set of meaningful and manageable
  segments.

11
Shot classification
Shots can be classified into

• Long shot
• Captures a global view of the field
• Medium shot
• Shows close up view of one or more players in a
  specific part of the field
• Close shot
• Shows an above-waist view of a single player

12
Cinematic Features
Shot Classification (contd..)?

• A soccer field has one distinct dominant color
  i.e. green which varies from
• Stadium to stadium
• Lighting conditions
• In long views it has been observed that either
  grass dominates the entire frame or the crowd
  covers upper part of the frame

13
Typical long views in soccer videos
Grass covering entire frame Grass covering
partial frame
14
Shot Classification (contd..)?
Soccer Video Sequence
If dominant color is green
Dominant color ratio gt0.75 and lt1.0
Dominant color ratio gt0.25 and lt0.5
Dominant color ratio gt0.5 and lt0.75
Medium Shot
Long Shot
Close Shot
15
(No Transcript)
16
(No Transcript)
17
Cinematic Features
Shot Detection
Shots in sports videos can be

• Wipe
• Dissolve
• Hard cut
• Fade

18
Proposed Shot Detection Method

• Extends the approach proposed by Vadivel et al.
  for broadcast soccer videos
• Combines the shot detection method by Vadivel et
  al. with the proposed shot classification method.

Limitations of Vadivel et als method for
broadcast soccer videos Hard cuts are missed
19
Proposed Shot Detection Method
Each frame in a shot is classified with the shot
classification algorithm If a long shot is
segmented into a sequence of long and medium view
frames If the number of frames in the sequence is
above a certain threshold Hard cut exists within
the shot
20
Proposed Shot Detection Results
Overall Recall and Precision by Vadivel et als
method 85.43, 89.02 Proposed method 91.76,
93.65
21
Shot detection improved by shot classification
22
Object Based Features
Feature extraction for grass pixels
Each frame is processed in YIQ color space. It is
found experimentally that grass pixels have I
values ranging between 25 and 55 while Q values
range between 0 and 12.
23
Playfield region detected
Grass pixels detected for a long view frame
24
Object Based Features (contd..)?
Playfield Line Detection
A playfield line separates playfield from the non
playfield background which are usually the
billboards (also called advertisement
boards). Hough transform is used to detect the
playfield line.
25
Object Based Features (contd..)?
Midfield Line Detection
Midfield line is the line that divides the
playfield in half along its width. Hough
transform is applied to detect the midfield line.
26
Ball Detection
Object Based Features (contd..)?
Challenges

• Features of the ball (color, size, shape) vary
  with time
• Relative size of the ball is very small
• Ball may not be an ideal circle because of fast
  motion and illumination conditions
• Objects in the field or in the crowd may look
  similar to a ball
• Field appearance changes from place to place and
  time to time

No definite property to uniquely identify ball in
a frame
27
Object Based Features (contd..)?
Detecting Ball Candidates in Long Shots

• Obtain ball candidates by detecting circular
  regions by using circular Hough Transform
• Filter the non ball candidates by
• Removing candidates from channels logo
• Removing candidates from gallery region
• Removing candidates from midfield line
• Filtering out the candidates moving against the
  camera

28
Object Based Features (contd..)?
Ball candidates before and after filtering
Ball candidates before filtering
Ball candidates after filtering
29
Detecting Players in Long Shots
Object Based Features (contd..)?
Challenges

• Features of the players (color, texture, size,
  motion) are neither static nor uniform
• Players appear very small in size
• Size of players changes with their position and
  zooming of cameras
• Color and texture of the jersey and shorts vary
  from team to team
• Players in the field do not have constant motion

30
Object Based Features (contd..)?
Detecting Player Regions

• Obtain player pixels by removing non player
  pixels
• Removing grass pixels
• Removing the broadcasting channels logo
• Removing the extra field region (billboards and
  gallery)?
• Removing pixels from the midfield line
• Segment the image containing player pixels to
  isolated player regions by
• Region growing algorithm
• Center of the bounding rectangle of each region
  is said to be the location of the player

31
Object Based Features (contd..)?
A Long Shot View
32
Object Based Features (contd..)?
Player pixels detected
33
Object Based Features (contd..)?
Players detected in long shot views
34
Team Identification in Soccer Videos
Feature Detection (Contd.)
Players in a soccer videos are classified using a
supervisory classification method. Mean I and Q
values of the player regions are obtained by
randomly selecting a few frames The minimum and
maximum I and Q values are set as the range for
classifying player regions
35
Team Classification in Soccer Videos
Feature detection (contd.)

• Experiments were performed on two different
  matches
• Real Madrid and Manchester United (UEFA
  Champions League 2003)?
• Chelsea and Liverpool (UEFA Champions League
  2007)?

36
(No Transcript)
37
(No Transcript)
38
Camera Related Feature
Object Based Features (contd..)?
Camera Direction Estimation

1. Optical Flow velocities and their directions are
   computed using Horn and Shuncks method.
2. Based on the sign of the horizontal component of
   the majority pixels in a frame, the direction of
   movement (left or right) of the camera is
   estimated.

39
Camera Direction Estimation (contd..)?
Optical flow velocities for the camera moving
towards right
40
Tracking of Broadcast Video Objects
Challenges

• Camera parameters are unknown
• Cameras are not fixed
• Cameras are zoomed and rotated
• Broadcast video is an edited video

41
Construction of a Directed Weighted Graph
Objects in a frame form nodes. Between two
correlated objects in two different frames an arc
(edge) is formed. The measure of correlation or
similarity provide the weight. Temporal
direction provides the direction of the edge.
42
Directed Weighted Graph (contd..)
Tracking of Broadcast Video Objects (contd..)?
43
Object Trajectory Detection
Tracking of Broadcast Video Objects (contd..)?
Given a source node, longest path of the graph
obtained by dynamic programming gives the path of
the object.
44
(No Transcript)
45
Results for ball detection in long shots
(contd..)?
46
Tracking a Single Player
Tracking of Broadcast Video Objects (contd..)?
Given a source node (player in the first frame),
longest path of the graph obtained by dynamic
programming gives the path of the player in the
whole sequence.
Player being tracked
47
Tracking of Broadcast Video Objects (contd..)?
Tracking Multiple Players
Longest path from each node (represented by
players in the first frame) of the graph obtained
by dynamic programming gives the trajectories of
the players for the sequence of frames.
Limitations

• Occlusion between players
• Players in contact
• Similarity between players belonging to same team

48
Resolving Conflicting Player Trajectories
Tracking Multiple Players (contd..)?

• If more than one player has more than two common
  nodes in its trajectory then only one amongst
  them is true.
• The path having maximum weight is said to be the
  true trajectory
• Nodes constituting the paths of correctly
  detected players are removed and a graph is again
  constructed
• Mistracked players are again tracked

49
(No Transcript)
50
(No Transcript)
51
(No Transcript)
52
Multi - Player Tracking Results
53
Multi - Player Tracking with Occlusion Results
54
Multi - Player Tracking with Occlusion Results
(contd..)?
55
Multi - Player Tracking with Occlusion Results
(contd..)?
56
Multi - Player Tracking with Occlusion Results
(contd..)?
57
Tracking the Mistracked Player (contd..)
58
Tracking the Mistracked Player (contd..)
59
Tracking the Mistracked Player (contd..)?
60
Tracking the Mistracked Player (contd..)?
61
Detection of States and Events
The features extracted and the trajectories
detected are used to detect states and events
based on the proposed state based video model.
States identified - Ball possession
states Events detected - Ball passing events
62
Detection of States and Events (contd..)
State Chart Diagram for Ball Possession
63
Play Break Detection
Detection of States and Events (contd..)
64
State Detection

• Ball possession states are obtained based on
• Spatial proximity analysis
• Distance between nearest player and second
  nearest player to the ball
• Spatial arrangements between the players and the
  ball

65
Ball Possession State Detection
Ball in possession of player 1s team
Ball in possession of player 1s team
66
Ball Possession State Detection
Ball in possession of player 1s team
Ball in a fight state
67
(No Transcript)
68
Edit Distance as performance measure for ball
possession states

• If the actual state sequence for a sequence of
  frames is
• AAAAFFFFFFFFFFBBBB
• And if the state sequence obtained by the
  proposed algorithm is
• AAAAFFFFFFFFBBBBBB
• Both the sequences are represented as strings S1
  and S2.
• Edit distance D(S1, S2 ) is defined as the
  minimum number of point mutations required to
  change S1 to S2 where a point mutation is one of
• replacing an alphabet
• inserting an alphabet
• deleting an alphabet
• Edit distance for the above sequence is 2. While
  normalized edit distance is
• D(S1, S2 )/ S1

69
Shot wise ball possession results
70
Event Detection

• The event detected in this work is the ball
  passing event. It can be
• Forward pass
• Reverse pass

71
Event Detection (contd..)?

• The ball passing event cannot be detected
• from state transition graphs because
• Ball is usually passed between players of the
  same team
• State transition graphs show the change in ball
  possession states from Team A-Team B, Team B -
  Team A, Team B Fight , Fight Team B, Team A
  Fight or Fight Team A

72
Schematic diagram for ball passing events

• Ball is said to be passed in a sequence of
  frames, if
• Nearest player in the initial frames of the
  sequence is the second nearest player to the ball
  in the subsequent frames
• Nearest and the second nearest players to the
  ball belong to the same team

73
Example of a ball passing event
74
Example of a ball passing event (contd..)?
75
Example of a ball passing event (contd..)?
76
Example of a ball passing event (contd..)?
77
Classifying ball passing events
Forward pass
Direction of camera motion is towards the goal
post of the team opposite to that of the nearest
player
Reverse pass
Direction of camera motion is towards the goal
post of the team of the nearest player
78
Results for ball passing events
Average Recall 100 and Precision 60
79
(No Transcript)
80
Graphs for ball possession and ball passing
Graphs illustrating ball possession states and
ball passing events for Sequence 7
81
Graphs for ball possession and ball passing
Graphs illustrating ball possession states and
ball passing events for Sequence 10
82
Publication

• V. Pallavi, A. Vadivel, Shamik Sural, A.K.
  Majumdar, Jayanta Mukherjee, Identification of
  moving objects in a Soccer video, Workshop on
  Computer Vision, Graphics and Image Processing
  2006, Hyderabad, India, pp. 13-18.
• V. Pallavi, J. Mukherjee, A.K. Majumdar and
  Shamik Sural, Shot classification in Soccer
  videos, Proceedings of National Conference on
  Recent Trends in Information Systems 2006,
  Kolkata, India, pp. 216-219.
• V. Pallavi, J. Mukherjee, A.K. Majumdar and
  Shamik Sural, Identification of team in
  possession of ball in a soccer video using static
  and dynamic segmentation, Proceedings of Sixth
  International Conference on Advances in Pattern
  Recognition 2007, Kolkata, India, pp. 249-255. 
• V. Pallavi, J. Mukherjee, A.K. Majumdar and
  Shamik Sural, Ball detection from broadcast
  soccer videos using static and dynamic features,
  Journal of Visual Communication and Image
  Representation, (Accepted for a second review).

83
Thank You