TeamTrack: An Algorithm and Benchmark Dataset for Multi-Sport Multi-Object Tracking in Full-pitch Videos

1Nagoya University 2University of Tsukuba 3Tokai University 4AIST


This paper presents TeamTrack, a new benchmark dataset and algorithm for multi-object tracking (MOT) in team sports, specifically in full-pitch videos. MOT in team sports is a challenging task due to object occlusions, similar appearances, and complex movements. Existing methods often struggle to accurately track objects in these scenarios. To address this challenge, the proposed TeamTrack dataset captures diverse object appearances and movements in soccer, basketball, and handball games using high-resolution fisheye and drone cameras. The dataset includes over 4 million annotated bounding boxes and provides a comprehensive resource for developing and evaluating MOT algorithms. The paper also introduces a new MOT approach that incorporates trajectory forecasting using a graph neural network (GNN) to model complex group movement patterns. The experiments demonstrate the effectiveness of the proposed algorithm on the TeamTrack dataset.


We present TeamTrack, a novel MOT dataset comprising over 150 minutes of high-resolution video from multiple team sports. The main characteristics of TeamTrack are the following:

  • Large-scale: TeamTrack introduces over 4 million annotated bounding boxes across various tracklets, making it one of the largest datasets of its kind.
  • Multiple Sports: TeamTrack includes matches from three team sports; soccer, basketball, and handball.
  • Similar Appearance and Dynamic Movement: The dataset features targets with similar appearance, dynamic movements, and frequent occlusions, offering a robust challenge for tracking algorithms.
  • Full Pitch Multi Angle View: Videos are recorded from two angles, top view via drones and side view with fisheye lenses, both covering the entire playing field.

Beyond its significant size and the inherent challenges it presents, TeamTrack’s unique value lies in its multi-view and full-pitch capture settings. These features enable experimentation with multi-view tracking and the use of prior information (e.g., player count, pitch dimensions) not feasible with traditional broadcast videos. Additionally, we hope the adversarial-cooperative nature of team sports can also be studied to further refine tracking techniques.

Table 1: Comparison of MOT datasets and TeamTrack.
Dataset Frames BBoxes Domain
MOT16 11,235 292,733 Pedestrians
MOT20 13,410 2,102,385 Pedestrians
KITTI-T 10,870 65,213 Autonomous
DanceTrack 105,855 - Dance
SSET 12,000 12,000 Soccer
SN-Tracking 225,375 3,645,661 Soccer
SportsMOT 150,379 1,629,490 Soccer
SoccerTrack 82,800 2,484,000 Soccer
279,900 4,374,900 Soccer
Table 2: Camera and video details of the multi-object tracking dataset for various sports.
Sport Camera Type Location Device Resolution Minutes Bounding Box Count
Soccer Fisheye A University, Outdoor Z CAM E2-F8 8K 30 1,242,000
Soccer Drone A University, Outdoor DJI Mavic 3 4K 30 1,242,000
Basketball Fisheye B University, Indoor Z CAM E2-F6 6K 17.5 346,500
Basketball Drone (Side View) C University, Outdoor DJI Mavic 3 4K 24 475,200
Basketball Drone (Top View) C University, Outdoor DJI Mavic 3 4K 24 475,200
Handball Fisheye B University, Indoor Z CAM E2-F6 6K 30 594,000
Total 155.5 4,374,900

Video Preview

The videos are shown are compressed and resized for the website. You can see more video previews on Kaggle.

Tracking Benchmarks

Below we present a direct evaluation of our proposed TeamTrack dataset by applying two state-of-the-art tracking algorithms, ByteTrack and BoT-SORT. Additionally, we compare these results with earlier benchmarks performed on the MOT17 and DanceTrack datasets to highlight the challenges posed by our TeamTrack dataset. We use the implementations from Ultralytics for both ByteTrack and BoT-SORT.

Performance of the various MOT models.
Table 3: Soccer SideView
Table 4: Soccer TopView
Table 5: Basketball SideView
Table 6: Basketball SideView2
Table 7: Basketball TopView
Table 8: Handball SideView
Performance of the various MOT models on different datasets.
Table 9: Comparison of TeamTrack with other datasets.
TeamTrack Ave.61.964.

Change Log

  • 2024-04-22: Released the dataset and updated arXiv links.


To cite this work in your academic research, please use the following BibTeX entry:

  title={TeamTrack: An Algorithm and Benchmark Dataset for Multi-Sport Multi-Object Tracking in Full-pitch Videos},
  author={Scott, Atom and Uchida, Ikuma and Ding, Ning and Umemoto, Rikuhei and Bunker, Rory and Kobayashi, Ren and Koyama, Takeshi and Onishi, Masaki and Kameda, Yoshinari and Fujii, Keisuke},
  journal={arXiv preprint arXiv:submit/5550700},