Computer Science › Computer Vision and Pattern Recognition

Advanced Image and Video Retrieval Techniques

Works Count: 66310

Wikipedia

Description

This cluster of papers focuses on the development and evaluation of techniques for extracting, matching, and utilizing local image features for tasks such as object recognition, image retrieval, and scene classification. It covers a wide range of methods including local descriptors, deep learning approaches, binary codes, and cross-modal retrieval techniques.

Keywords

Local Descriptors; Feature Matching; Object Recognition; Image Retrieval; Deep Learning; Binary Codes; Scene Classification; Interest Point Detectors; Bag-of-Features; Cross-Modal Retrieval

BRIEF: Binary Robust Independent Elementary Features

2010-01-01

Michael Calonder , Vincent Lepetit , Christoph Strecha +1 more

Machine Learning for High-Speed Corner Detection

2006-01-01

Edward Rosten , Tom Drummond

Improving the Fisher Kernel for Large-Scale Image Classification

2010-01-01

Florent Perronnin , Jorge Sánchez , Thomas Mensink

Visual categorization with bags of keypoints

2004-01-01

G. Csurka

We present a novel method for generic visual categorization: the problem of identifying the object content of natural images while generalizing across variations inherent to the object class. This bag … We present a novel method for generic visual categorization: the problem of identifying the object content of natural images while generalizing across variations inherent to the object class. This bag of keypoints method is based on vector quantization of affine invariant descriptors of image patches. We propose and compare two alternative implementations using different classifiers: Naive Bayes and SVM. The main advantages of the method are that it is simple, computationally efficient and intrinsically invariant. We present results for simultaneously classifying seven semantic visual categories. These results clearly demonstrate that the method is robust to background clutter and produces good categorization accuracy even without exploiting geometric information.

FAST APPROXIMATE NEAREST NEIGHBORS WITH AUTOMATIC ALGORITHM CONFIGURATION

2009-01-01

Marius Muja , David Lowe

For many computer vision problems, the most time consuming component consists of nearest neighbor matching in high-dimensional spaces.There are no known exact algorithms for solving these high-dimensional problems that are … For many computer vision problems, the most time consuming component consists of nearest neighbor matching in high-dimensional spaces.There are no known exact algorithms for solving these high-dimensional problems that are faster than linear search.Approximate algorithms are known to provide large speedups with only minor loss in accuracy, but many such algorithms have been published with only minimal guidance on selecting an algorithm and its parameters for any given problem.In this paper, we describe a system that answers the question, "What is the fastest approximate nearest-neighbor algorithm for my data?"Our system will take any given dataset and desired degree of precision and use these to automatically determine the best algorithm and parameter values.We also describe a new algorithm that applies priority search on hierarchical k-means trees, which we have found to provide the best known performance on many datasets.After testing a range of alternatives, we have found that multiple randomized k-d trees provide the best performance for other datasets.We are releasing public domain code that implements these approaches.This library provides about one order of magnitude improvement in query time over the best previously available software and provides fully automated parameter selection.

View PDF Chat

SURF: Speeded Up Robust Features

2006-01-01

Herbert Bay , Tinne Tuytelaars , Luc Van Gool

A survey of image registration techniques

1992-12-01

Lisa Gottesfeld Brown

Registration is a fundamental task in image processing used to match two or more pictures taken, for example, at different times, from different sensors, or from different viewpoints. Virtually all … Registration is a fundamental task in image processing used to match two or more pictures taken, for example, at different times, from different sensors, or from different viewpoints. Virtually all large systems which evaluate images require the registration of images, or a closely related operation, as an intermediate step. Specific examples of systems where image registration is a significant component include matching a target with a real-time image of a scene for target recognition, monitoring global land usage using satellite images, matching stereo images to recover shape for autonomous navigation, and aligning images from different medical modalities for diagnosis. Over the years, a broad range of techniques has been developed for various types of data and problems. These techniques have been independently studied for several different applications, resulting in a large body of research. This paper organizes this material by establishing the relationship between the variations in the images and the type of registration techniques which can most appropriately be applied. Three major types of variations are distinguished. The first type are the variations due to the differences in acquisition which cause the images to be misaligned. To register images, a spatial transformation is found which will remove these variations. The class of transformations which must be searched to find the optimal transformation is determined by knowledge about the variations of this type. The transformation class in turn influences the general technique that should be taken. The second type of variations are those which are also due to differences in acquisition, but cannot be modeled easily such as lighting and atmospheric conditions. This type usually effects intensity values, but they may also be spatial, such as perspective distortions. The third type of variations are differences in the images that are of interest such as object movements, growths, or other scene changes. Variations of the second and third type are not directly removed by registration, but they make registration more difficult since an exact match is no longer possible. In particular, it is critical that variations of the third type are not removed. Knowledge about the characteristics of each type of variation effect the choice of feature space, similarity measure, search space, and search strategy which will make up the final technique. All registration techniques can be viewed as different combinations of these choices. This framework is useful for understanding the merits and relationships between the wide variety of existing techniques and for assisting in the selection of the most suitable technique for a specific problem.

A Comparison of Affine Region Detectors

2005-10-14

Krystian Mikolajczyk , Tinne Tuytelaars , C. Schmid +5 more

View PDF Chat

PatchMatch

2009-07-27

Connelly Barnes , Eli Shechtman , Adam Finkelstein +1 more

This paper presents interactive image editing tools using a new randomized algorithm for quickly finding approximate nearest-neighbor matches between image patches. Previous research in graphics and vision has leveraged such … This paper presents interactive image editing tools using a new randomized algorithm for quickly finding approximate nearest-neighbor matches between image patches. Previous research in graphics and vision has leveraged such nearest-neighbor searches to provide a variety of high-level digital image editing tools. However, the cost of computing a field of such matches for an entire image has eluded previous efforts to provide interactive performance. Our algorithm offers substantial performance improvements over the previous state of the art (20-100x), enabling its use in interactive editing tools. The key insights driving the algorithm are that some good patch matches can be found via random sampling, and that natural coherence in the imagery allows us to propagate such matches quickly to surrounding areas. We offer theoretical analysis of the convergence properties of the algorithm, as well as empirical and practical evidence for its high quality and performance. This one simple algorithm forms the basis for a variety of tools -- image retargeting, completion and reshuffling -- that can be used together in the context of a high-level image editing application. Finally, we propose additional intuitive constraints on the synthesis process that offer the user a level of control unavailable in previous methods.

Aggregating local descriptors into a compact image representation

2010-06-01

Hervé Jeǵou , Matthijs Douze , Cordelia Schmid +1 more

We address the problem of image search on a very large scale, where three constraints have to be considered jointly: the accuracy of the search, its efficiency, and the memory … We address the problem of image search on a very large scale, where three constraints have to be considered jointly: the accuracy of the search, its efficiency, and the memory usage of the representation.We first propose a simple yet efficient way of aggregating local image descriptors into a vector of limited dimension, which can be viewed as a simplification of the Fisher kernel representation.We then show how to jointly optimize the dimension reduction and the indexing algorithm, so that it best preserves the quality of vector comparison.The evaluation shows that our approach significantly outperforms the state of the art: the search accuracy is comparable to the bag-of-features approach for an image representation that fits in 20 bytes.Searching a 10 million image dataset takes about 50ms.

View PDF Chat

SUN database: Large-scale scene recognition from abbey to zoo

2010-06-01

Jianxiong Xiao , James Hays , Krista A. Ehinger +2 more

Scene categorization is a fundamental problem in computer vision. However, scene understanding research has been constrained by the limited scope of currently-used databases which do not capture the full variety … Scene categorization is a fundamental problem in computer vision. However, scene understanding research has been constrained by the limited scope of currently-used databases which do not capture the full variety of scene categories. Whereas standard databases for object categorization contain hundreds of different classes of objects, the largest available dataset of scene categories contains only 15 classes. In this paper we propose the extensive Scene UNderstanding (SUN) database that contains 899 categories and 130,519 images. We use 397 well-sampled categories to evaluate numerous state-of-the-art algorithms for scene recognition and establish new bounds of performance. We measure human scene classification performance on the SUN database and compare this with computational methods. Additionally, we study a finer-grained scene representation to detect scenes embedded inside of larger scenes.

View PDF Chat

Learning Hierarchical Features for Scene Labeling

2012-10-24

Clément Farabet , Camille Couprie , Laurent Najman +1 more

Scene labeling consists of labeling each pixel in an image with the category of the object it belongs to. We propose a method that uses a multiscale convolutional network trained … Scene labeling consists of labeling each pixel in an image with the category of the object it belongs to. We propose a method that uses a multiscale convolutional network trained from raw pixels to extract dense feature vectors that encode regions of multiple sizes centered on each pixel. The method alleviates the need for engineered features, and produces a powerful representation that captures texture, shape, and contextual information. We report results using multiple postprocessing methods to produce the final labeling. Among those, we propose a technique to automatically retrieve, from a pool of segmentation components, an optimal set of components that best explain the scene; these components are arbitrary, for example, they can be taken from a segmentation tree or from any family of oversegmentations. The system yields record accuracies on the SIFT Flow dataset (33 classes) and the Barcelona dataset (170 classes) and near-record accuracy on Stanford background dataset (eight classes), while being an order of magnitude faster than competing approaches, producing a 320×240 image labeling in less than a second, including feature extraction.

View PDF Chat

The Pascal Visual Object Classes (VOC) Challenge

2009-09-08

Mark Everingham , Luc Van Gool , Christopher K. I. Williams +2 more

View PDF Chat

The Pascal Visual Object Classes Challenge: A Retrospective

2014-06-24

Mark Everingham , S. M. Ali Eslami , Luc Van Gool +3 more

View PDF Chat

CNN Features Off-the-Shelf: An Astounding Baseline for Recognition

2014-06-01

Ali Sharif Razavian , Hossein Azizpour , Josephine Sullivan +1 more

Recent results indicate that the generic descriptors extracted from the convolutional neural networks are very powerful. This paper adds to the mounting evidence that this is indeed the case. We … Recent results indicate that the generic descriptors extracted from the convolutional neural networks are very powerful. This paper adds to the mounting evidence that this is indeed the case. We report on a series of experiments conducted for different recognition tasks using the publicly available code and model of the OverFeat network which was trained to perform object classification on ILSVRC13. We use features extracted from the OverFeat network as a generic image representation to tackle the diverse range of recognition tasks of object image classification, scene recognition, fine grained recognition, attribute detection and image retrieval applied to a diverse set of datasets. We selected these tasks and datasets as they gradually move further away from the original task and data the OverFeat network was trained to solve. Astonishingly, we report consistent superior results compared to the highly tuned state-of-the-art systems in all the visual classification tasks on various datasets. For instance retrieval it consistently outperforms low memory footprint methods except for sculptures dataset. The results are achieved using a linear SVM classifier (or L2 distance in case of retrieval) applied to a feature representation of size 4096 extracted from a layer in the net. The representations are further modified using simple augmentation techniques e.g. jittering. The results strongly suggest that features obtained from deep learning with convolutional nets should be the primary candidate in most visual recognition tasks.

View PDF Chat

Vlfeat

2010-10-25

Andrea Vedaldi , Brian Fulkerson

VLFeat is an open and portable library of computer vision algorithms. It aims at facilitating fast prototyping and reproducible research for computer vision scientists and students. It includes rigorous implementations … VLFeat is an open and portable library of computer vision algorithms. It aims at facilitating fast prototyping and reproducible research for computer vision scientists and students. It includes rigorous implementations of common building blocks such as feature detectors, feature extractors, (hierarchical) k-means clustering, randomized kd-tree matching, and super-pixelization. The source code and interfaces are fully documented. The library integrates directly with MATLAB, a popular language for computer vision research.

Selective Search for Object Recognition

2013-04-01

Jasper Uijlings , Koen E. A. van de Sande , Theo Gevers +1 more

View PDF Chat

Linear spatial pyramid matching using sparse coding for image classification

2009-06-01

Shuicheng Yan , Kai Yu , Yihong Gong +1 more

Recently SVMs using spatial pyramid matching (SPM) kernel have been highly successful in image classification. Despite its popularity, these nonlinear SVMs have a complexity O(n 2 ∼ n … Recently SVMs using spatial pyramid matching (SPM) kernel have been highly successful in image classification. Despite its popularity, these nonlinear SVMs have a complexity O(n 2 ∼ n 3 ) in training and O(n) in testing, where n is the training size, implying that it is nontrivial to scaleup the algorithms to handlemore than thousands of training images. In this paper we develop an extension of the SPM method, by generalizing vector quantization to sparse coding followed by multi-scale spatial max pooling, and propose a linear SPM kernel based on SIFT sparse codes. This new approach remarkably reduces the complexity of SVMs to O(n) in training and a constant in testing. In a number of image categorization experiments, we find that, in terms of classification accuracy, the suggested linear SPM based on sparse coding of SIFT descriptors always significantly outperforms the linear SPM kernel on histograms, and is even better than the nonlinear SPM kernels, leading to state-of-the-art performance on several benchmarks by using a single type of descriptors.

ImageNet: A large-scale hierarchical image database

2009-06-01

Jia Deng , Wei Dong , Richard Socher +3 more

The explosion of image data on the Internet has the potential to foster more sophisticated and robust models and algorithms to index, retrieve, organize and interact with images and multimedia … The explosion of image data on the Internet has the potential to foster more sophisticated and robust models and algorithms to index, retrieve, organize and interact with images and multimedia data. But exactly how such data can be harnessed and organized remains a critical problem. We introduce here a new database called "ImageNet", a large-scale ontology of images built upon the backbone of the WordNet structure. ImageNet aims to populate the majority of the 80,000 synsets of WordNet with an average of 500–1000 clean and full resolution images. This will result in tens of millions of annotated images organized by the semantic hierarchy of WordNet. This paper offers a detailed analysis of ImageNet in its current state: 12 subtrees with 5247 synsets and 3.2 million images in total. We show that ImageNet is much larger in scale and diversity and much more accurate than the current image datasets. Constructing such a large-scale database is a challenging task. We describe the data collection scheme with Amazon Mechanical Turk. Lastly, we illustrate the usefulness of ImageNet through three simple applications in object recognition, image classification and automatic object clustering. We hope that the scale, accuracy, diversity and hierarchical structure of ImageNet can offer unparalleled opportunities to researchers in the computer vision community and beyond.

LabelMe: A Database and Web-Based Tool for Image Annotation

2007-10-30

Bryan Russell , Antonio Torralba , Kevin Murphy +1 more

Face recognition: features versus templates

1993-01-01

Roberto Brunelli , Tomaso Poggio

Two new algorithms for computer recognition of human faces, one based on the computation of a set of geometrical features, such as nose width and length, mouth position, and chin … Two new algorithms for computer recognition of human faces, one based on the computation of a set of geometrical features, such as nose width and length, mouth position, and chin shape, and the second based on almost-gray-level template matching, are presented. The results obtained for the testing sets show about 90% correct recognition using geometrical features and perfect recognition using template matching.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">></ETX>

ORB: An efficient alternative to SIFT or SURF

2011-11-01

Ethan Rublee , Vincent Rabaud , Kurt Konolige +1 more

Feature matching is at the base of many computer vision problems, such as object recognition or structure from motion. Current methods rely on costly descriptors for detection and matching. In … Feature matching is at the base of many computer vision problems, such as object recognition or structure from motion. Current methods rely on costly descriptors for detection and matching. In this paper, we propose a very fast binary descriptor based on BRIEF, called ORB, which is rotation invariant and resistant to noise. We demonstrate through experiments how ORB is at two orders of magnitude faster than SIFT, while performing as well in many situations. The efficiency is tested on several real-world applications, including object detection and patch-tracking on a smart phone.

ImageNet Large Scale Visual Recognition Challenge

2015-04-10

Olga Russakovsky , Jia Deng , Hao Su +7 more

View PDF Chat

Speeded-Up Robust Features (SURF)

2008-01-07

Herbert Bay , Andreas Ess , Tinne Tuytelaars +1 more

View PDF Chat

Face recognition by elastic bunch graph matching

1997-07-01

Laurenz Wiskott , Jean‐Marc Fellous , N. Kuiger +1 more

We present a system for recognizing human faces from single images out of a large database containing one image per person. Faces are represented by labeled graphs, based on a … We present a system for recognizing human faces from single images out of a large database containing one image per person. Faces are represented by labeled graphs, based on a Gabor wavelet transform. Image graphs of new faces are extracted by an elastic graph matching process and can be compared by a simple similarity function. The system differs from the preceding one (Lades et al., 1993) in three respects. Phase information is used for accurate node positioning. Object-adapted graphs are used to handle large rotations in depth. Image graph extraction is based on a novel data structure, the bunch graph, which is constructed from a small get of sample image graphs.

View PDF Chat

Product Quantization for Nearest Neighbor Search

2010-03-19

H. Jegou , Matthijs Douze , Calvin F. Schmid

This paper introduces a product quantization-based approach for approximate nearest neighbor search. The idea is to decompose the space into a Cartesian product of low-dimensional subspaces and to quantize each … This paper introduces a product quantization-based approach for approximate nearest neighbor search. The idea is to decompose the space into a Cartesian product of low-dimensional subspaces and to quantize each subspace separately. A vector is represented by a short code composed of its subspace quantization indices. The euclidean distance between two vectors can be efficiently estimated from their codes. An asymmetric version increases precision, as it computes the approximate distance between a vector and a code. Experimental results show that our approach searches for nearest neighbors efficiently, in particular in combination with an inverted file system. Results for SIFT and GIST image descriptors show excellent search accuracy, outperforming three state-of-the-art approaches. The scalability of our approach is validated on a data set of two billion vectors.

View PDF Chat

Automatic Panoramic Image Stitching using Invariant Features

2006-12-18

Matthew Brown , David Lowe

View PDF Chat

Object retrieval with large vocabularies and fast spatial matching

2007-06-01

James Philbin , Ondřej Chum , Michael Isard +2 more

In this paper, we present a large-scale object retrieval system. The user supplies a query object by selecting a region of a query image, and the system returns a ranked … In this paper, we present a large-scale object retrieval system. The user supplies a query object by selecting a region of a query image, and the system returns a ranked list of images that contain the same object, retrieved from a large corpus. We demonstrate the scalability and performance of our system on a dataset of over 1 million images crawled from the photo-sharing site, Flickr [3], using Oxford landmarks as queries. Building an image-feature vocabulary is a major time and performance bottleneck, due to the size of our dataset. To address this problem we compare different scalable methods for building a vocabulary and introduce a novel quantization method based on randomized trees which we show outperforms the current state-of-the-art on an extensive ground-truth. Our experiments show that the quantization has a major effect on retrieval quality. To further improve query performance, we add an efficient spatial verification stage to re-rank the results returned from our bag-of-words model and show that this consistently improves search quality, though by less of a margin when the visual vocabulary is large. We view this work as a promising step towards much larger, "web-scale" image corpora.

Photo tourism

2006-07-01

Noah Snavely , Steven M. Seitz , Richard Szeliski

We present a system for interactively browsing and exploring large unstructured collections of photographs of a scene using a novel 3D interface. Our system consists of an image-based modeling front … We present a system for interactively browsing and exploring large unstructured collections of photographs of a scene using a novel 3D interface. Our system consists of an image-based modeling front end that automatically computes the viewpoint of each photograph as well as a sparse 3D model of the scene and image to model correspondences. Our photo explorer uses image-based rendering techniques to smoothly transition between photographs, while also enabling full 3D navigation and exploration of the set of images and world geometry, along with auxiliary information such as overhead maps. Our system also makes it easy to construct photo tours of scenic or historic locations, and to annotate image details, which are automatically transferred to other relevant images. We demonstrate our system on several large personal photo collections as well as images gathered from Internet photo sharing sites.

View PDF Chat

Locality-sensitive hashing scheme based on p-stable distributions

2004-06-08

Mayur Datar , Nicole Immorlica , Piotr Indyk +1 more

We present a novel Locality-Sensitive Hashing scheme for the Approximate Nearest Neighbor Problem under lp norm, based on p-stable distributions.Our scheme improves the running time of the earlier algorithm for … We present a novel Locality-Sensitive Hashing scheme for the Approximate Nearest Neighbor Problem under lp norm, based on p-stable distributions.Our scheme improves the running time of the earlier algorithm for the case of the lp norm. It also yields the first known provably efficient approximate NN algorithm for the case p<1. We also show that the algorithm finds the exact near neigbhor in O(log n) time for data satisfying certain "bounded growth" condition.Unlike earlier schemes, our LSH scheme works directly on points in the Euclidean space without embeddings. Consequently, the resulting query time bound is free of large factors and is simple and easy to implement. Our experiments (on synthetic data sets) show that the our data structure is up to 40 times faster than kd-tree.

View PDF Chat

Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories

2006-07-10

Svetlana Lazebnik , C. Schmid , Jean Ponce

This paper presents a method for recognizing scene categories based on approximate global geometric correspondence. This technique works by partitioning the image into increasingly fine sub-regions and computing histograms of … This paper presents a method for recognizing scene categories based on approximate global geometric correspondence. This technique works by partitioning the image into increasingly fine sub-regions and computing histograms of local features found inside each sub-region. The resulting "spatial pyramid" is a simple and computationally efficient extension of an orderless bag-of-features image representation, and it shows significantly improved performance on challenging scene categorization tasks. Specifically, our proposed method exceeds the state of the art on the Caltech-101 database and achieves high accuracy on a large database of fifteen natural scene categories. The spatial pyramid framework also offers insights into the success of several recently proposed image descriptions, including Torralba's "gist" and Lowe's SIFT descriptors.

View PDF Chat

Scale & Affine Invariant Interest Point Detectors

2004-05-13

Krystian Mikolajczyk

A performance evaluation of local descriptors

2005-08-24

Krystian Mikolajczyk , C. Schmid

In this paper, we compare the performance of descriptors computed for local interest regions, as, for example, extracted by the Harris-Affine detector. Many different descriptors have been proposed in the … In this paper, we compare the performance of descriptors computed for local interest regions, as, for example, extracted by the Harris-Affine detector. Many different descriptors have been proposed in the literature. It is unclear which descriptors are more appropriate and how their performance depends on the interest region detector. The descriptors should be distinctive and at the same time robust to changes in viewing conditions as well as to errors of the detector. Our evaluation uses as criterion recall with respect to precision and is carried out for different image transformations. We compare shape context, steerable filters, PCA-SIFT, differential invariants, spin images, SIFT, complex filters, moment invariants, and cross-correlation for different types of interest regions. We also propose an extension of the SIFT descriptor and show that it outperforms the original method. Furthermore, we observe that the ranking of the descriptors is mostly independent of the interest region detector and that the SIFT-based descriptors perform best. Moments and steerable filters show the best performance among the low dimensional descriptors.

View PDF Chat

Learning Deep Features for Discriminative Localization

2016-06-01

Bolei Zhou , Aditya Khosla , Àgata Lapedriza +2 more

In this work, we revisit the global average pooling layer proposed in [13], and shed light on how it explicitly enables the convolutional neural network (CNN) to have remarkable localization … In this work, we revisit the global average pooling layer proposed in [13], and shed light on how it explicitly enables the convolutional neural network (CNN) to have remarkable localization ability despite being trained on imagelevel labels. While this technique was previously proposed as a means for regularizing training, we find that it actually builds a generic localizable deep representation that exposes the implicit attention of CNNs on an image. Despite the apparent simplicity of global average pooling, we are able to achieve 37.1% top-5 error for object localization on ILSVRC 2014 without training on any bounding box annotation. We demonstrate in a variety of experiments that our network is able to localize the discriminative image regions despite just being trained for solving classification task1.

View PDF Chat

Pyramid Scene Parsing Network

2017-07-01

Hengshuang Zhao , Jianping Shi , Xiaojuan Qi +2 more

Scene parsing is challenging for unrestricted open vocabulary and diverse scenes. In this paper, we exploit the capability of global context information by different-region-based context aggregation through our pyramid pooling … Scene parsing is challenging for unrestricted open vocabulary and diverse scenes. In this paper, we exploit the capability of global context information by different-region-based context aggregation through our pyramid pooling module together with the proposed pyramid scene parsing network (PSPNet). Our global prior representation is effective to produce good quality results on the scene parsing task, while PSPNet provides a superior framework for pixel-level prediction. The proposed approach achieves state-of-the-art performance on various datasets. It came first in ImageNet scene parsing challenge 2016, PASCAL VOC 2012 benchmark and Cityscapes benchmark. A single PSPNet yields the new record of mIoU accuracy 85.4% on PASCAL VOC 2012 and accuracy 80.2% on Cityscapes.

View PDF Chat

Places: A 10 Million Image Database for Scene Recognition

2017-07-04

Bolei Zhou , Àgata Lapedriza , Aditya Khosla +2 more

The rise of multi-million-item dataset initiatives has enabled data-hungry machine learning algorithms to reach near-human semantic classification performance at tasks such as visual object and scene recognition. Here we describe … The rise of multi-million-item dataset initiatives has enabled data-hungry machine learning algorithms to reach near-human semantic classification performance at tasks such as visual object and scene recognition. Here we describe the Places Database, a repository of 10 million scene photographs, labeled with scene semantic categories, comprising a large and diverse list of the types of environments encountered in the world. Using the state-of-the-art Convolutional Neural Networks (CNNs), we provide scene classification CNNs (Places-CNNs) as baselines, that significantly outperform the previous approaches. Visualization of the CNNs trained on Places shows that object detectors emerge as an intermediate representation of scene classification. With its high-coverage and high-diversity of exemplars, the Places Database along with the Places-CNNs offer a novel resource to guide future progress on scene recognition problems.

View PDF Chat

YOLOv3: An Incremental Improvement.

2018-04-08

Joseph Redmon , Ali Farhadi

We present some updates to YOLO! We made a bunch of little design changes to make it better. We also trained this new network that's pretty swell. It's a little … We present some updates to YOLO! We made a bunch of little design changes to make it better. We also trained this new network that's pretty swell. It's a little bigger than last time but more accurate. It's still fast though, don't worry. At 320x320 YOLOv3 runs in 22 ms at 28.2 mAP, as accurate as SSD but three times faster. When we look at the old .5 IOU mAP detection metric YOLOv3 is quite good. It achieves 57.9 mAP@50 in 51 ms on a Titan X, compared to 57.5 mAP@50 in 198 ms by RetinaNet, similar performance but 3.8x faster. As always, all the code is online at this https URL

Color indexing

1991-11-01

Michael J. Swain , Dana H. Ballard

Caffe: Convolutional Architecture for Fast Feature Embedding

2014-01-01

Yangqing Jia , Evan Shelhamer , Jeff Donahue +5 more

Caffe provides multimedia scientists and practitioners with a clean and modifiable framework for state-of-the-art deep learning algorithms and a collection of reference models. The framework is a BSD-licensed C++ library … Caffe provides multimedia scientists and practitioners with a clean and modifiable framework for state-of-the-art deep learning algorithms and a collection of reference models. The framework is a BSD-licensed C++ library with Python and MATLAB bindings for training and deploying general-purpose convolutional neural networks and other deep models efficiently on commodity architectures. Caffe fits industry and internet-scale media needs by CUDA GPU computation, processing over 40 million images a day on a single K40 or Titan GPU ($\approx$ 2.5 ms per image). By separating model representation from actual implementation, Caffe allows experimentation and seamless switching among platforms for ease of development and deployment from prototyping machines to cloud environments. Caffe is maintained and developed by the Berkeley Vision and Learning Center (BVLC) with the help of an active community of contributors on GitHub. It powers ongoing research projects, large-scale industrial applications, and startup prototypes in vision, speech, and multimedia.

Unsupervised Visual Representation Learning by Context Prediction

2015-12-01

Carl Doersch , Abhinav Gupta , Alexei A. Efros

This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation. Given only a large, unlabeled image collection, … This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation. Given only a large, unlabeled image collection, we extract random pairs of patches from each image and train a convolutional neural net to predict the position of the second patch relative to the first. We argue that doing well on this task requires the model to learn to recognize objects and their parts. We demonstrate that the feature representation learned using this within-image context indeed captures visual similarity across images. For example, this representation allows us to perform unsupervised visual discovery of objects like cats, people, and even birds from the Pascal VOC 2011 detection dataset. Furthermore, we show that the learned ConvNet can be used in the R-CNN framework [19] and provides a significant boost over a randomly-initialized ConvNet, resulting in state-of-the-art performance among algorithms which use only Pascal-provided training set annotations.

View PDF Chat

ImageNet: A large-scale hierarchical image database

2009-06-01

Jia Deng , Wei Dong , R. Socher +3 more

The explosion of image data on the Internet has the potential to foster more sophisticated and robust models and algorithms to index, retrieve, organize and interact with images and multimedia … The explosion of image data on the Internet has the potential to foster more sophisticated and robust models and algorithms to index, retrieve, organize and interact with images and multimedia data. But exactly how such data can be harnessed and organized remains a critical problem. We introduce here a new database called “ImageNet”, a large-scale ontology of images built upon the backbone of the WordNet structure. ImageNet aims to populate the majority of the 80,000 synsets of WordNet with an average of 500–1000 clean and full resolution images. This will result in tens of millions of annotated images organized by the semantic hierarchy of WordNet. This paper offers a detailed analysis of ImageNet in its current state: 12 subtrees with 5247 synsets and 3.2 million images in total. We show that ImageNet is much larger in scale and diversity and much more accurate than the current image datasets. Constructing such a large-scale database is a challenging task. We describe the data collection scheme with Amazon Mechanical Turk. Lastly, we illustrate the usefulness of ImageNet through three simple applications in object recognition, image classification and automatic object clustering. We hope that the scale, accuracy, diversity and hierarchical structure of ImageNet can offer unparalleled opportunities to researchers in the computer vision community and beyond.

YOLOv3: An Incremental Improvement

2018-01-01

Joseph Redmon , Ali Farhadi

View PDF Chat

Edge Boxes: Locating Object Proposals from Edges

2014-01-01

C. Lawrence Zitnick , Piotr Dollár

The use of object proposals is an effective recent approach for increasing the computational efficiency of object detection. We propose a novel method for generating object bounding box proposals using … The use of object proposals is an effective recent approach for increasing the computational efficiency of object detection. We propose a novel method for generating object bounding box proposals using edges. Edges provide a sparse yet informative representation of an image. Our main observation is that the number of contours that are wholly contained in a bounding box is indicative of the likelihood of the box containing an object. We propose a simple box objectness score that measures the number of edges that exist in the box minus those that are members of contours that overlap the box's boundary. Using efficient data structures, millions of candidate boxes can be evaluated in a fraction of a second, returning a ranked set of a few thousand top-scoring proposals. Using standard metrics, we show results that are significantly more accurate than the current state-of-the-art while being faster to compute. In particular, given just 1000 proposals we achieve over 96% object recall at overlap threshold of 0.5 and over 75% recall at the more challenging overlap of 0.7. Our approach runs in 0.25 seconds and we additionally demonstrate a near real-time variant with only minor loss in accuracy.

Unsupervised Multimodal UAV Image Registration via Style Transfer and Cascade Network

2025-06-24

Xue Bi , Rongkai Qie , Chengyang Tao +2 more | Remote Sensing

Cross-modal image registration for unmanned aerial vehicle (UAV) platforms presents significant challenges due to large-scale deformations, distinct imaging mechanisms, and pronounced modality discrepancies. This paper proposes a novel multi-scale cascaded … Cross-modal image registration for unmanned aerial vehicle (UAV) platforms presents significant challenges due to large-scale deformations, distinct imaging mechanisms, and pronounced modality discrepancies. This paper proposes a novel multi-scale cascaded registration network based on style transfer that achieves superior performance: up to 67% reduction in mean squared error (from 0.0106 to 0.0068), 9.27% enhancement in normalized cross-correlation, 26% improvement in local normalized cross-correlation, and 8% increase in mutual information compared to state-of-the-art methods. The architecture integrates a cross-modal style transfer network (CSTNet) that transforms visible images into pseudo-infrared representations to unify modality characteristics, and a multi-scale cascaded registration network (MCRNet) that performs progressive spatial alignment across multiple resolution scales using diffeomorphic deformation modeling to ensure smooth and invertible transformations. A self-supervised learning paradigm based on image reconstruction eliminates reliance on manually annotated data while maintaining registration accuracy through synthetic deformation generation. Extensive experiments on the LLVIP dataset demonstrate the method’s robustness under challenging conditions involving large-scale transformations, with ablation studies confirming that style transfer contributes 28% MSE improvement and diffeomorphic registration prevents 10.6% performance degradation. The proposed approach provides a robust solution for cross-modal image registration in dynamic UAV environments, offering significant implications for downstream applications such as target detection, tracking, and surveillance.

An Empirical Study on Training Paradigms for Deep Supervised Hashing

2025-06-23

Yang Shen , Peng Wang , Xiu-Shen Wei +1 more | International Journal of Computer Vision

Correction: Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spaces

2025-06-23

Georgii Mikriukov , Gesina Schwalbe , Korinna Bade | International Journal of Computer Vision

PR-CLIP: Cross-Modal Positional Reconstruction for Remote Sensing Image–Text Retrieval

2025-06-20

Jihong Guan , Yulou Shu , Wengen Li +2 more | Remote Sensing

With the development of satellite technology, remote sensing images have become increasingly accessible, making multi-modal remote sensing retrieval increasingly important. However, most existing methods rely on global visual and textual … With the development of satellite technology, remote sensing images have become increasingly accessible, making multi-modal remote sensing retrieval increasingly important. However, most existing methods rely on global visual and textual features to compute similarity, ignoring the positional correspondence between image regions and textual descriptions. To address this issue, we propose a novel cross-modal retrieval model named PR-CLIP, which leverages a cross-modal positional information reconstruction task to learn position-aware correlations between modalities. Specifically, PR-CLIP first uses a cross-modal positional information extraction module to extract the complementary features between images and texts. Then, the unimodal positional information filtering module filters out the complementary information from the unimodal features to generate embeddings for reconstruction. Finally, the cross-modal positional information reconstruction module reconstructs the unimodal embeddings of the images and texts based on the complete embeddings of the opposite modality, guided by a cross-modal positional consistency loss to ensure reconstruction quality. During the inference stage of retrieval, PR-CLIP directly calculates the similarity between the unimodal features without executing the modules of the reconstruction task. By combining the advantages of dual-stream and single-stream models, PR-CLIP achieves a good balance between performance and efficiency. Extensive experiments on multiple public datasets demonstrated the effectiveness of PR-CLIP.

Unsupervised cross-modal hashing retrieval with multi-level interaction

2025-06-20

Zhenqiu Shu , Jie Zhang , Kailing Yong +4 more | Engineering Applications of Artificial Intelligence

A lightweight CNN for LULC classification: an alpha-cut fuzzy geometric normalization approach

2025-06-19

V. Anitha , S. Kalaiselvi , V. Gomathi +1 more | Signal Image and Video Processing

Similarity-Based Decision Support for Improving Agricultural Practices and Plant Growth

2025-06-19

Iulia Bărăian , Honoriu Vălean , Oliviu Matei +1 more | Applied Sciences

Similarity-based decision support systems have become essential tools for providing tailored and adaptive guidance across various domains. In agriculture, where managing extensive land areas poses significant challenges, the primary objective … Similarity-based decision support systems have become essential tools for providing tailored and adaptive guidance across various domains. In agriculture, where managing extensive land areas poses significant challenges, the primary objective is often to maximize harvest yields while reducing costs, preserving crop health, and minimizing the use of chemical adjuvants. The application of similarity-based analysis enables the development of personalized farming recommendations, refined through shared data and insights, which contribute to improved plant growth and enhanced annual harvest outcomes. This study employs two algorithms, K-Nearest Neighbour (KNN) and Approximate Nearest Neighbour (ANN) using Locality Sensitive Hashing (LSH) to evaluate their effectiveness in agricultural decision-making. The results demonstrate that, under comparable farming conditions, KNN yields more accurate recommendations due to its reliance on exact matches, whereas ANN provides a more scalable solution well-suited for large datasets. Both approaches support improved agricultural decisions and promote more sustainable farming strategies. While KNN is more effective for smaller datasets, ANN proves advantageous in real-time applications that demand fast response times. The implementation of these algorithms represents a significant advancement toward data-driven and efficient agricultural practices.

F-ResPoint: a multi-modal fusion object detection algorithm based on frustum culling

2025-06-18

Jinjun Rao , Kai Yang , Qing‐Tai Zhao +5 more | Displays

A Unified Generative Hashing for Cross-Modal Retrieval

2025-06-18

Junfeng Tu , Xueliang Liu , Yanbin Hao +1 more | ACM Transactions on Multimedia Computing Communications and Applications

Cross-modal hashing is a highly effective and efficient method for information retrieval, enabling the search for correlated data across different modality databases using compact hash codes. Conventional cross-modal hashing typically … Cross-modal hashing is a highly effective and efficient method for information retrieval, enabling the search for correlated data across different modality databases using compact hash codes. Conventional cross-modal hashing typically uses separate model structures for each modality and aligns approximate continuous representations of the final hash codes. These approaches not only require specialized models for each modality but also introduce a gap between the discrete hash codes and their continuous features, yielding only approximate alignment. To address these issues, we propose a unified generative cross-modal hashing method that leverages a single Uniform Mixture-of-Expert Decoder (UMoED) for both image and text modalities. UMoED streamlines cross-modal hash learning by integrating two key design elements: (1) a cross-modal representation unification module that employs unified queries to consolidate modality-specific features into a common space, and (2) an adaptive expert enhancement module that adaptively enhances feature modeling based on the input modality. Furthermore, our decoder-based hashing method outputs hash codes in a generative manner, producing precise representations of discrete codes to bridge the gap between the discrete and continuous space, thus ensuring precise alignment during similarity learning. Extensive experiments on three benchmark datasets demonstrate that the proposed method achieves state-of-the-art performance in cross-modal hashing retrieval.

Efficient Image Retrieval using Dense-SIFT for Enhanced Object Segmentation and Key Point Localization

2025-06-18

D. Jeyakumari , A. Nesarani , B. Manojkumar +2 more | International Journal of Computational and Experimental Science and Engineering

In this paper a new technique for image retrieval is used. Traditional methods used for image retrieval are not supported for large set of databases. By using the features of … In this paper a new technique for image retrieval is used. Traditional methods used for image retrieval are not supported for large set of databases. By using the features of the image such as color, shape and texture image can be retrieved efficiently. The Content Based image retrieval (CBIR) technique is the traditional technique used for image retrieval . Several kinds of detectors and descriptors such as SIFT, SURF, FAST, BRIEF, ORB, BRISK, FREAK are used for image retrieval. Among these techniques SIFT is quite powerful. The main drawback of the existing system is that it computes only at the interest points. The proposed system addresses about the D- SIFT algorithm in which the SIFT is computed at every pixel, or every kth pixel. The Density- Scale Invariant Feature Transform (D-SIFT) is a stand out amongst the most locally feature detector and descriptors which is utilized as a part of the majority of the vision programming. The main advantage of using Dense SIFT over SIFT is speed. The main goal is to segment the similar object from the two images. As the result of the proposed system the poorly localized points are removed by key point localization.

MIRAGE-ANNS: Mixed Approach Graph-based Indexing for Approximate Nearest Neighbor Search

2025-06-17

Sairaj Voruganti , M. TAMER ÖZSU | Proceedings of the ACM on Management of Data

Approximate nearest neighbor search (ANNS) on high dimensional vectors is important for numerous applications, such as search engines, recommendation systems, and more recently, large language models (LLMs), where Retrieval Augmented … Approximate nearest neighbor search (ANNS) on high dimensional vectors is important for numerous applications, such as search engines, recommendation systems, and more recently, large language models (LLMs), where Retrieval Augmented Generation (RAG) is used to add context to an LLM query. Graph-based indexes built on these vectors have been shown to perform best but have challenges. These indexes can either employ refinement-based construction strategies such as K-Graph and NSG, or increment-based strategies such as HNSW. Refinement-based approaches have fast construction times, but worse search performance and do not allow for incremental inserts, requiring a full reconstruction each time new vectors are added to the index. Increment-based approaches have good search performance and allow for incremental inserts, but suffer from slow construction. This work presents MIRAGE-ANNS ( M ixed I ncremental R efinement A pproach G raph-based E xploration for Approximate Nearest Neighbor Search) that constructs the index as fast as refinement-based approaches while retaining search performance comparable or better than increment-based ones. It also allows incremental inserts. We show that MIRAGE achieves state of the art construction and query performance, outperforming existing methods by up to 2x query throughput on real-world datasets.

DIGRA: A Dynamic Graph Indexing for Approximate Nearest Neighbor Search with Range Filter

2025-06-17

Mengxu Jiang , Zhi Yang , Fangyuan Zhang +5 more | Proceedings of the ACM on Management of Data

Recent advancements in AI have enabled models to map real-world entities, such as product images, into high-dimensional vectors, making approximate nearest neighbor search (ANNS) crucial for various applications. Often, these … Recent advancements in AI have enabled models to map real-world entities, such as product images, into high-dimensional vectors, making approximate nearest neighbor search (ANNS) crucial for various applications. Often, these vectors are associated with additional attributes like price, prompting the need for range-filtered ANNS where users seek similar items within specific attribute ranges. Naive solutions like pre-filtering and post-filtering are straightforward but inefficient. Specialized indexes, such as SeRF, SuperPostFiltering, and iRangeGraph, have been developed to address these queries effectively. However, these solutions do not support dynamic updates, limiting their practicality in real-world scenarios where datasets frequently change. To address these challenges, we propose DIGRA, a novel dynamic graph index for range-filtered ANNS. DIGRA supports efficient dynamic updates while maintaining a balance among query efficiency, update efficiency, indexing cost, and result quality. Our approach introduces a dynamic multi-way tree structure combined with carefully integrated ANNS indices to handle range filtered ANNS efficiently. We employ a lazy weight-based update mechanism to significantly reduce update costs and adopt optimized choice of ANNS index to lower construction and update overhead. Experimental results demonstrate that DIGRA achieves superior trade-offs, making it suitable for large-scale dynamic datasets in real-world applications.

Optimizing YOLO-Based Algorithms for Real-Time BISINDO Alphabet Detection Under Varied Lighting and Background Conditions in Computer Vision Systems

2025-06-17

Lilis Nur Hayati , Anik Nur Handayani , Wahyu Sakti Gunawan Irianto +3 more | International Journal of Engineering Science and Information Technology

This research explores the optimization of YOLO-based computer vision algorithms for real-time recognition of Indonesian Sign Language (BISINDO) letters under diverse environmental conditions. Motivated by the communication barriers faced by … This research explores the optimization of YOLO-based computer vision algorithms for real-time recognition of Indonesian Sign Language (BISINDO) letters under diverse environmental conditions. Motivated by the communication barriers faced by the deaf and hearing communities due to limited sign language literacy, the study aims to enhance inclusivity through advanced visual detection technologies. By implementing the YOLOv5s model, the system is trained to detect and classify correct and incorrect BISINDO hand signs across 52 classes (26 correct and 26 incorrect letters), utilizing a dataset of 3,900 images augmented to 10,920 samples. Performance evaluation employs k-fold cross-validation (k=10) and confusion matrix analysis across varied lighting and background scenarios, both indoor and outdoor. The model achieves a high average precision of 0.9901 and recall of 0.9999, with robust results in indoor settings and slight degradation observed under certain outdoor conditions. These findings demonstrate the potential of YOLOv5 in facilitating real-time, accurate sign language recognition, contributing toward more accessible human-computer interaction systems for the deaf community.

Multi-Scene Recognition in Single Aerial Images Using CNN

2025-06-17

G. Ananthi , S. Mehala Shevani , Priyanka Devi | IGI Global eBooks

Scene recognition from aerial images is a primary application in utilizing high resolution satellite images. In the recent past, many studies were carried out in this field to classify an … Scene recognition from aerial images is a primary application in utilizing high resolution satellite images. In the recent past, many studies were carried out in this field to classify an image to one scene category. But the real scenarios are different, they often contain more than one scene. Therefore, we explore an approach to recognize multiple scenes from a single aerial image. To do this, three different datasets namely UCM dataset, AID dataset and MAI dataset consisting of 2100, 10000, 3923 images respectively were used. Out of the three datasets, UCM and AID dataset are Single scene datasets and the MAI dataset is a multi-scene dataset. These datasets are utilized in two configurations namely UCM2MAI and AID2MAI. Prototype based memory network using different CNN baseline models is used as the backbone. First, the single scene datasets were used to extensively train different baseline models and store them as prototypes in the external memory. Experiments were conducted to analyse the performance of different baseline models.

ReSTIR BDPT: Bidirectional ReSTIR Path Tracing with Caustics

2025-06-17

Trevor Hedstrom , Markus Kettunen , Daqi Lin +2 more | ACM Transactions on Graphics

Recent spatiotemporal resampling algorithms (ReSTIR) accelerate real-time path tracing by reusing samples between pixels and frames. However, existing methods are limited by the sampling quality of path tracing, making them … Recent spatiotemporal resampling algorithms (ReSTIR) accelerate real-time path tracing by reusing samples between pixels and frames. However, existing methods are limited by the sampling quality of path tracing, making them inefficient for scenes with caustics and hard-to-reach lights. We develop a ReSTIR variant incorporating bidirectional path tracing that significantly improves the sampling quality in these scenes. Combining bidirectional path tracing and ReSTIR introduces multiple challenges: the generalized resampled importance sampling (GRIS) behind ReSTIR is, by default, not aware of how a path was sampled, which complicates reuse of bidirectional paths. Light tracing is also challenging since light subpaths can contribute to all pixels. To address these challenges, we apply GRIS in a sampling technique-aware extended path space, design a bidirectional hybrid shift mapping, and introduce caustics reservoirs that can accumulate caustics across frames. Our method takes around 50ms per frame across our test scenes, and achieves significantly lower error compared to prior unidirectional ReSTIR variants running in equal time.

Scalable Complex Event Processing on Video Streams

2025-06-17

Chenxia Han , Chaokun Chang , Saurish Srivastava +2 more | Proceedings of the ACM on Management of Data

The rapid expansion of video streaming content in our daily lives has rendered the real-time processing and analysis of these video streams a critical capability. However, existing deep video analytics … The rapid expansion of video streaming content in our daily lives has rendered the real-time processing and analysis of these video streams a critical capability. However, existing deep video analytics systems primarily support only simple queries, such as selection and aggregation. Considering the inherent temporal nature of video streams, queries capable of matching patterns of events could enable a wider range of applications. In this paper, we present Bobsled, a novel video stream processing system designed to efficiently support complex event queries. Experimental results demonstrate that Bobsled can achieve a throughput improvement over state-of-the-art ranging from 2.4× to 11.6×, without any noticeable loss in accuracy.

Efficient Dynamic Indexing for Range Filtered Approximate Nearest Neighbor Search

2025-06-17

Fangyuan Zhang , Mengxu Jiang , Guanhao Hou +5 more | Proceedings of the ACM on Management of Data

Given a set O of objects consisting of n high-dimensional vectors, the problem of approximate nearest neighbor (ANN) search for a query vector q is crucial in many applications where … Given a set O of objects consisting of n high-dimensional vectors, the problem of approximate nearest neighbor (ANN) search for a query vector q is crucial in many applications where objects are represented as feature vectors in high-dimensional spaces. Each object in O often has attributes like popularity or price, which influence the search. Practically, searching for the nearest neighbor to q might include a range filter specifying the desired attribute values, e.g., within a specific price range. Existing solutions for range filtered ANN search often face trade-offs among excessive storage, poor query performance, and limited support for updates. To address this challenge, we propose RangePQ, a novel indexing scheme that supports efficient range filtered ANN searches and updates, requiring only linear space. Our scheme integrates seamlessly with existing PQ-based index---a widely recognized, scalable index type for ANN searches---to enhance range-filtered ANN queries and update capabilities. Our indexing method, supporting arbitrary range filters, has a space complexity of (O(n log K)), where K is a parameter of the PQ-based index and log K scales with O(log n). To reduce the space cost, we further present a hybrid two-layer structure to reduce space usage to O(n), preserving query efficiency without additional update costs. Experimental results demonstrate that our indexing scheme significantly improves query performance while maintaining competitive update performance and space efficiency.

Practical and Asymptotically Optimal Quantization of High-Dimensional Vectors in Euclidean Space for Approximate Nearest Neighbor Search

2025-06-17

Jianyang Gao , Yutong Gou , Yuexuan Xu +3 more | Proceedings of the ACM on Management of Data

Approximate nearest neighbor (ANN) query in high-dimensional Euclidean space is a key operator in database systems. For this query, quantization is a popular family of methods developed for compressing vectors … Approximate nearest neighbor (ANN) query in high-dimensional Euclidean space is a key operator in database systems. For this query, quantization is a popular family of methods developed for compressing vectors and reducing memory consumption. Among these methods, a recent algorithm called RaBitQ achieves the state-of-the-art performance and provides an asymptotically optimal theoretical error bound. RaBitQ uses 1 bit per dimension for quantization and compresses vectors with a large compression rate. In this paper, we extend RaBitQ to compress vectors with flexible compression rates - it achieves this by using B bits per dimension for quantization with B = 1, 2, ... It inherits the theoretical guarantees of RaBitQ and achieves the asymptotic optimality in terms of the trade-off between space and error bounds as to be proven in this study. Additionally, we present efficient implementations of the extended RaBitQ, enabling its application to ANN queries to reduce both space and time consumption. Extensive experiments on real-world datasets confirm that our method consistently outperforms the state-of-the-art baselines in both accuracy and efficiency when using the same amount of memory.

Accelerating Graph Indexing for ANNS on Modern CPUs

2025-06-17

Mengzhao Wang , Haotian Wu , Xiangyu Ke +3 more | Proceedings of the ACM on Management of Data

In high-dimensional vector spaces, Approximate Nearest Neighbor Search (ANNS) is a key component in database and artificial intelligence infrastructures. Graph-based ANNS methods, particularly HNSW, have emerged as leading solutions, offering … In high-dimensional vector spaces, Approximate Nearest Neighbor Search (ANNS) is a key component in database and artificial intelligence infrastructures. Graph-based ANNS methods, particularly HNSW, have emerged as leading solutions, offering an impressive trade-off between search efficiency and accuracy. Many vector databases utilize graph indexes as their core algorithms, benefiting from various optimizations to enhance search performance. However, the high indexing time associated with graph algorithms poses a significant challenge, especially given the increasing volume of data, query processing complexity, and dynamic index maintenance demand. This has rendered indexing time a critical performance metric for users. In this paper, we comprehensively analyze the underlying causes of the low graph indexing efficiency on modern CPUs, identifying that distance computation dominates indexing time, primarily due to high memory access latency and suboptimal arithmetic operation efficiency. We demonstrate that distance comparisons during index construction can be effectively performed using compact vector codes at an appropriate compression error. Drawing from insights gained through integrating existing compact coding methods in the graph indexing process, we propose a novel compact coding strategy, named Flash, designed explicitly for graph indexing and optimized for modern CPU architectures. By minimizing random memory accesses and maximizing the utilization of SIMD (Single Instruction, Multiple Data) instructions, Flash significantly enhances cache hit rates and arithmetic operations. Extensive experiments conducted on eight real-world datasets, ranging from ten million to one billion vectors, exhibit that Flash achieves a speedup of 10.4× to 22.9× in index construction efficiency, while maintaining or improving search performance.

Fast Approximate Similarity Join in Vector Databases

2025-06-17

Jiadong Xie , Jeffrey Xu Yu , Yingfan Liu | Proceedings of the ACM on Management of Data

Recent advancements in deep learning, particularly in embedding models, have enabled the effective representation of various data types such as text, images, and audio as vectors, thereby facilitating semantic analysis. … Recent advancements in deep learning, particularly in embedding models, have enabled the effective representation of various data types such as text, images, and audio as vectors, thereby facilitating semantic analysis. A large number of massive vector datasets are maintained in vector databases. Approximate similarity join is a core operation in vector database systems that joins two datasets, and outputs all pairs of vectors from the two datasets, if the distance between such a pair of two vectors is no more than a specified value. Existing approaches for similarity join are selection-based such that they treat each data point in a dataset as an individual query point to search data points by an approximate range query in another dataset. Such methods do not fully capitalize on the inherent properties of the join operation itself. In this paper, we propose a new join algorithm, SimJoin. Our join algorithm aims at boosting join processing efficiency by leveraging relationships between partial join results (e.g., join windows). In brief, our join algorithm accelerates the join processing to process a join window by utilizing the join windows from the processed data points. Then, we discuss optimizing join window order to minimize join costs. In addition, we discuss how to support k -similarity join, and how to maintain proximity graph index based on k-similarity join. Extensive experiments on real-world and synthetic datasets demonstrate the significant performance superiority of our proposed algorithms over existing state-of-the-art methods.

PDX: A Data Layout for Vector Similarity Search

2025-06-17

Leonardo Kuffò , Elena Krippner , Peter Boncz | Proceedings of the ACM on Management of Data

We propose Partition Dimensions Across (PDX), a data layout for vectors (e.g., embeddings) that, similar to PAX [6], stores multiple vectors in one block, using a vertical layout for the … We propose Partition Dimensions Across (PDX), a data layout for vectors (e.g., embeddings) that, similar to PAX [6], stores multiple vectors in one block, using a vertical layout for the dimensions (Figure 1). PDX accelerates exact and approximate similarity search thanks to its dimension-by-dimension search strategy that operates on multiple-vectors-at-a-time in tight loops. It beats SIMD-optimized distance kernels on standard horizontal vector storage (avg 40% faster), only relying on scalar code that gets auto-vectorized. We combined the PDX layout with recent dimension-pruning algorithms ADSampling [19] and BSA [52] that accelerate approximate vector search. We found that these algorithms on the horizontal vector layout can lose to SIMD-optimized linear scans, even if they are SIMD-optimized. However, when used on PDX, their benefit is restored to 2-7x. We find that search on PDX is especially fast if a limited number of dimensions has to be scanned fully, which is what the dimension-pruning approaches do. We finally introduce PDX-BOND, an even more flexible dimension-pruning strategy, with good performance on exact search and reasonable performance on approximate search. Unlike previous pruning algorithms, it can work on vector data ''as-is'' without preprocessing; making it attractive for vector databases with frequent updates.

Leveraging 3DCNN and Weighted Similarity Metrics forEnhanced Content-Based Video Retrieval

2025-06-16

Farooq Shaik , Ashu Abdul , Jatindra Kumar Dash | Research Square (Research Square)

<title>Abstract</title> With the widespread adoption of high-speed networks such as 4G and 5G, along with the explo-sive growth of social media platforms, video content is now frequently captured and shared … <title>Abstract</title> With the widespread adoption of high-speed networks such as 4G and 5G, along with the explo-sive growth of social media platforms, video content is now frequently captured and shared onlinewithout accompanying metadata such as tags or descriptions. This absence of textual annotationspresents a significant challenge for indexing and retrieving relevant video content. Content-BasedVideo Retrieval systems address this issue by analyzing the visual content of videos rather than rely-ing on external metadata. However, only limited efforts in the literature have jointly explored both thespatial and temporal context of video data for retrieval. To address this gap, we propose a Content-Based Video Retrieval framework that leverages a 3D Convolutional Neural Network, specificallythe R(2+1)D architecture enhanced with transfer learning. This model decomposes spatiotemporalconvolutions to more effectively capture both spatial and temporal video features. In addition, weintroduce a novel classification-similarity-based weighted distance approach, which overcomes the lim-itations of traditional distance-based and classifier-based retrieval methods. Experimental evaluationon the UCF101 dataset demonstrates that the proposed system achieves a significant improvementin retrieval performance, with over a 20% increase in AUC compared to baseline techniques.