HORUS: multimodal large language models framework for video retrieval at VBS 2025

Tai Nguyen; Vo Ngoc Minh Anh; Duc Dat Pham; Tran Quang Vinh; Nhu Duong Thi Quynh; Le Anh Tien; Tan Duy Le; Binh T. Nguyen

doi:10.1007/978-981-96-2074-6_34

HORUS: multimodal large language models framework for video retrieval at VBS 2025

Tai Nguyen, Vo Ngoc Minh Anh, Duc Dat Pham, Tran Quang Vinh, Nhu Duong Thi Quynh, Le Anh Tien, Tan Duy Le, Binh T. Nguyen^*

^*Corresponding author for this work

School of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In the dynamic field of video retrieval, precise and effective search methods are crucial for managing complex datasets. We present HORUS, a novel approach based on multimodal Large Language Models (mLLMs) that advances video retrieval capabilities through two key innovations: (1) advanced multi-modal feature aggregation, integrating text-to-image search with CLIP, free-text search from captions generated by Video-LLaMA2, and visual features from Video-LLaMA to capture temporal dynamics; and (2) GPT-based query expansion, combined with an advanced filter, addresses issues with low-quality open-ended text queries and refines item searches based on type and location within a scene. This work provides cutting-edge solutions for the VBS 2025 challenge and offers valuable insights into enhancing video search techniques.

Original language	English
Title of host publication	MultiMedia modeling
Subtitle of host publication	31st international conference on multimedia modeling, MMM 2025, Nara, Japan, January 8–10, 2025, proceedings, part V
Editors	Ichiro Ide, Ioannis Kompatsiaris, Changsheng Xu, Keiji Yanai, Wei-Ta Chu, Naoko Nitta, Michael Riegler, Toshihiko Yamasaki
Place of Publication	Singapore
Publisher	Springer Nature Singapore
Pages	286-293
Number of pages	8
ISBN (Electronic)	9789819620746
ISBN (Print)	9789819620739
DOIs	https://doi.org/10.1007/978-981-96-2074-6_34
Publication status	Published - 1 Jan 2025

Publication series

Name	Lecture notes in computer science
Publisher	Springer Singapore
Volume	15524
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Keywords

Video browser showdown
Video retrieval
Multi-modal feature aggregation

Access to Document

10.1007/978-981-96-2074-6_34

Handle.net

Persistent link

Cite this

Nguyen, T., Anh, V. N. M., Pham, D. D., Vinh, T. Q., Quynh, N. D. T., Tien, L. A., Le, T. D., & Nguyen, B. T. (2025). HORUS: multimodal large language models framework for video retrieval at VBS 2025. In I. Ide, I. Kompatsiaris, C. Xu, K. Yanai, W.-T. Chu, N. Nitta, M. Riegler, & T. Yamasaki (Eds.), MultiMedia modeling: 31st international conference on multimedia modeling, MMM 2025, Nara, Japan, January 8–10, 2025, proceedings, part V (pp. 286-293). (Lecture notes in computer science; Vol. 15524). Springer Nature Singapore. https://doi.org/10.1007/978-981-96-2074-6_34

Nguyen, Tai ; Anh, Vo Ngoc Minh ; Pham, Duc Dat et al. / HORUS: multimodal large language models framework for video retrieval at VBS 2025. MultiMedia modeling: 31st international conference on multimedia modeling, MMM 2025, Nara, Japan, January 8–10, 2025, proceedings, part V. editor / Ichiro Ide ; Ioannis Kompatsiaris ; Changsheng Xu ; Keiji Yanai ; Wei-Ta Chu ; Naoko Nitta ; Michael Riegler ; Toshihiko Yamasaki. Singapore : Springer Nature Singapore, 2025. pp. 286-293 (Lecture notes in computer science).

@inproceedings{4c3513171bbc42a0a40889138b7c21c7,

title = "HORUS: multimodal large language models framework for video retrieval at VBS 2025",

abstract = "In the dynamic field of video retrieval, precise and effective search methods are crucial for managing complex datasets. We present HORUS, a novel approach based on multimodal Large Language Models (mLLMs) that advances video retrieval capabilities through two key innovations: (1) advanced multi-modal feature aggregation, integrating text-to-image search with CLIP, free-text search from captions generated by Video-LLaMA2, and visual features from Video-LLaMA to capture temporal dynamics; and (2) GPT-based query expansion, combined with an advanced filter, addresses issues with low-quality open-ended text queries and refines item searches based on type and location within a scene. This work provides cutting-edge solutions for the VBS 2025 challenge and offers valuable insights into enhancing video search techniques.",

keywords = "Video browser showdown, Video retrieval, Multi-modal feature aggregation",

author = "Tai Nguyen and Anh, \{Vo Ngoc Minh\} and Pham, \{Duc Dat\} and Vinh, \{Tran Quang\} and Quynh, \{Nhu Duong Thi\} and Tien, \{Le Anh\} and Le, \{Tan Duy\} and Nguyen, \{Binh T.\}",

note = "Funding: Acknowledgments. The work was partially supported by Mr. Hoang Anh Le from OCANY VIET NAM VENTURE CO., LTD.",

year = "2025",

month = jan,

day = "1",

doi = "10.1007/978-981-96-2074-6\_34",

language = "English",

isbn = "9789819620739",

series = "Lecture notes in computer science",

publisher = "Springer Nature Singapore",

pages = "286--293",

editor = "Ichiro Ide and Ioannis Kompatsiaris and Changsheng Xu and Keiji Yanai and Wei-Ta Chu and Naoko Nitta and Michael Riegler and Toshihiko Yamasaki",

booktitle = "MultiMedia modeling",

}

Nguyen, T, Anh, VNM, Pham, DD, Vinh, TQ, Quynh, NDT, Tien, LA, Le, TD & Nguyen, BT 2025, HORUS: multimodal large language models framework for video retrieval at VBS 2025. in I Ide, I Kompatsiaris, C Xu, K Yanai, W-T Chu, N Nitta, M Riegler & T Yamasaki (eds), MultiMedia modeling: 31st international conference on multimedia modeling, MMM 2025, Nara, Japan, January 8–10, 2025, proceedings, part V. Lecture notes in computer science, vol. 15524, Springer Nature Singapore, Singapore, pp. 286-293. https://doi.org/10.1007/978-981-96-2074-6_34

HORUS: multimodal large language models framework for video retrieval at VBS 2025. / Nguyen, Tai; Anh, Vo Ngoc Minh; Pham, Duc Dat et al.
MultiMedia modeling: 31st international conference on multimedia modeling, MMM 2025, Nara, Japan, January 8–10, 2025, proceedings, part V. ed. / Ichiro Ide; Ioannis Kompatsiaris; Changsheng Xu; Keiji Yanai; Wei-Ta Chu; Naoko Nitta; Michael Riegler; Toshihiko Yamasaki. Singapore: Springer Nature Singapore, 2025. p. 286-293 (Lecture notes in computer science; Vol. 15524).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - HORUS: multimodal large language models framework for video retrieval at VBS 2025

AU - Nguyen, Tai

AU - Anh, Vo Ngoc Minh

AU - Pham, Duc Dat

AU - Vinh, Tran Quang

AU - Quynh, Nhu Duong Thi

AU - Tien, Le Anh

AU - Le, Tan Duy

AU - Nguyen, Binh T.

N1 - Funding: Acknowledgments. The work was partially supported by Mr. Hoang Anh Le from OCANY VIET NAM VENTURE CO., LTD.

PY - 2025/1/1

Y1 - 2025/1/1

N2 - In the dynamic field of video retrieval, precise and effective search methods are crucial for managing complex datasets. We present HORUS, a novel approach based on multimodal Large Language Models (mLLMs) that advances video retrieval capabilities through two key innovations: (1) advanced multi-modal feature aggregation, integrating text-to-image search with CLIP, free-text search from captions generated by Video-LLaMA2, and visual features from Video-LLaMA to capture temporal dynamics; and (2) GPT-based query expansion, combined with an advanced filter, addresses issues with low-quality open-ended text queries and refines item searches based on type and location within a scene. This work provides cutting-edge solutions for the VBS 2025 challenge and offers valuable insights into enhancing video search techniques.

AB - In the dynamic field of video retrieval, precise and effective search methods are crucial for managing complex datasets. We present HORUS, a novel approach based on multimodal Large Language Models (mLLMs) that advances video retrieval capabilities through two key innovations: (1) advanced multi-modal feature aggregation, integrating text-to-image search with CLIP, free-text search from captions generated by Video-LLaMA2, and visual features from Video-LLaMA to capture temporal dynamics; and (2) GPT-based query expansion, combined with an advanced filter, addresses issues with low-quality open-ended text queries and refines item searches based on type and location within a scene. This work provides cutting-edge solutions for the VBS 2025 challenge and offers valuable insights into enhancing video search techniques.

KW - Video browser showdown

KW - Video retrieval

KW - Multi-modal feature aggregation

UR - https://www.scopus.com/pages/publications/85215955292

U2 - 10.1007/978-981-96-2074-6_34

DO - 10.1007/978-981-96-2074-6_34

M3 - Conference contribution

SN - 9789819620739

T3 - Lecture notes in computer science

SP - 286

EP - 293

BT - MultiMedia modeling

A2 - Ide, Ichiro

A2 - Kompatsiaris, Ioannis

A2 - Xu, Changsheng

A2 - Yanai, Keiji

A2 - Chu, Wei-Ta

A2 - Nitta, Naoko

A2 - Riegler, Michael

A2 - Yamasaki, Toshihiko

PB - Springer Nature Singapore

CY - Singapore

ER -

Nguyen T, Anh VNM, Pham DD, Vinh TQ, Quynh NDT, Tien LA et al. HORUS: multimodal large language models framework for video retrieval at VBS 2025. In Ide I, Kompatsiaris I, Xu C, Yanai K, Chu WT, Nitta N, Riegler M, Yamasaki T, editors, MultiMedia modeling: 31st international conference on multimedia modeling, MMM 2025, Nara, Japan, January 8–10, 2025, proceedings, part V. Singapore: Springer Nature Singapore. 2025. p. 286-293. (Lecture notes in computer science). doi: 10.1007/978-981-96-2074-6_34

HORUS: multimodal large language models framework for video retrieval at VBS 2025

Abstract

Publication series

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this