A METHOD FOR INTELLIGENT PROCESSING AND RETRIEVAL OF MULTIMEDIA DATA
Keywords:
multimedia data, semantic retrieval, vector database, multimodal neural networks, spatial indexingAbstract
To overcome the limitations of traditional media archives in natural language search, the study proposes a multimodal Retrieval-Augmented Generation architecture for indexing and retrieving visual data. The proposed method combines dense vector search with lexical Best Matching 25 ranking by decomposing automatically generated image captions into atomic sentences. The entire software complex is implemented within a single PostgreSQL instance. Experiments demonstrate that this hybrid approach significantly outperforms baseline vector search, particularly for complex queries with rare proper nouns and vague spatio-temporal hints.
References
Lewis P., Perez E., Piktus A. et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems (NeurIPS 2020). 2020. Vol. 33. P. 9459–9474.
Reimers N., Gurevych I. Sentence-BERT: sentence embeddings using Siamese BERT-networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP 2019). Stroudsburg, PA : ACL, 2019. P. 3982–3992.
Robertson S., Zaragoza H. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval. 2009. Vol. 3, No. 4. P. 333–389.
Malkov Yu. A., Yashunin D. A. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020. Vol. 42, No. 4. P. 824–836.
Cormack G. V., Clarke C. L. A., Buettcher S. Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’09). New York : ACM, 2009. P. 758–759.
Carbonell J., Goldstein J. The use of MMR, diversity-based reranking for reordering documents and producing summaries. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’98). New York : ACM, 1998. P. 335–336.
pgvector: open-source vector similarity search for PostgreSQL : вебсайт. URL: https://github.com/pgvector/pgvector (дата звернення: 29.04.2026).
