IEEE Transactions On Image Processing[JOURNAL]
Sun
200 papers
GeoStyler: A Generalizable Geometry-Aware Diffusion-Based Approach for Direct 3D Gaussian Style Transfer.
Hu Q, Zhang Y, Dang J … +3 more , Chen M, Wang L, Guo Y
IEEE Trans Image Process · 2026 · PMID 42308065 · Publisher ↗
Direct 3D scene stylization from sparse views remains a significant challenge, as existing optimization-based methods are prohibitively slow and require dense inputs to prevent geometric corruption. While recent direct m...
Direct 3D scene stylization from sparse views remains a significant challenge, as existing optimization-based methods are prohibitively slow and require dense inputs to prevent geometric corruption. While recent direct methods accelerate this process, their rigid decoupling of a static geometry from appearance often leads to visual artifacts, where stylistic textures conflict with and distort the underlying scene structure. To address these limitations, we introduce GeoStyler, a direct framework that generates high-fidelity, multi-view consistent stylized 3D scenes in seconds. Our approach reformulates the conventional pipeline by first leveraging a diffusion model to generate a set of geometrically consistent stylized 2D images. The core of this stage is a novel hybrid query formulation for the self-attention mechanism. Specifically, cross-view geometric information is directly embedded into the query to enforce 3D consistency, while style information is independently injected via the key and value to preserve scene structure. This process is further stabilized by a geometrically-aware latent initialization that provides a coherent starting point for the denoising process. Subsequently, a decoupled reconstruction network lifts these 2D stylized images to 3D Gaussians. A geometry branch predicts a robust 3D scaffold from the original content images, while a parallel style branch predicts the final appearance from our generated stylized images, ensuring structural integrity is not compromised. Extensive experiments on large-scale benchmarks, including RealEstate10K and ACID, demonstrate that GeoStyler significantly outperforms prior arts in stylization quality and multi-view consistency, achieving state-of-the-art performance with a dramatic speedup. Our project page: https://huhuhuxiao.github.io/Geo-Styler/.
Lighted-SAM: Lightening Open-World SAM for Low-Light Segmentation.
Jia Y, Duan L, Li W … +1 more , Lv F
IEEE Trans Image Process · 2026 · PMID 42301844 · Publisher ↗
Segment Anything Model (SAM) has achieved impressive segmentation performance in an open-world setting. However, SAM relies heavily on high-quality input images and usually struggles in low-light conditions. This is main...
Segment Anything Model (SAM) has achieved impressive segmentation performance in an open-world setting. However, SAM relies heavily on high-quality input images and usually struggles in low-light conditions. This is mainly caused by the pre-training dataset, SA-1B, in which low-light samples constitute a relatively small fraction of the data. This lack of presence leads to a noticeable weakness when SAM is applied in real-world dark environments. With the motivation of improving SAM's performance under low-light conditions while retaining its strong zero-shot capability, this work proposes an alignment stage between the pre-training stage and testing stage. Unlike existing low-light studies that mainly focus on task-specific and close-set settings, our work further emphasizes pursuing the segmentation ability under low-light conditions for open-world models. To this end, we construct DarkSeg58K, a realistic and diverse dataset, which serves as the alignment dataset to support this stage. We further introduce Lighted-SAM as the lightweight repair strategy to fix SAM's performance in low-light conditions. Different from existing methods focusing on introducing spectral adapters into the model design and training this model end-to-end, Lighted-SAM introduces the Spectral Information Resonance (SIR) mechanism to harmoniously integrate the spectral enhancement module into SAM, which is usually kept frozen due to its large-scale parameters. Based on our lightweight repairing strategy, Lighted-SAM can improve SAM's ability in low-light conditions while preserving its zero-shot ability. Experiments on different benchmarks validate the superiority of our approach. Code is available at: https://github.com/Jaaaahan/LightedSAM.
Contour Field-Based Elliptical Shape Prior for the Segment Anything Model.
Zhao X, Wang F, Cui L … +2 more , Duan Y, Liu J
IEEE Trans Image Process · 2026 · PMID 42301843 · Publisher ↗
The elliptical shape prior information plays a vital role in improving the accuracy of image segmentation for specific tasks in medical and natural images. Existing deep learning-based segmentation methods, including the...
The elliptical shape prior information plays a vital role in improving the accuracy of image segmentation for specific tasks in medical and natural images. Existing deep learning-based segmentation methods, including the Segment Anything Model (SAM), often struggle to produce segmentation results with elliptical shapes efficiently. This paper proposes a new approach to integrate the prior of elliptical shapes into the deep learning-based SAM image segmentation techniques using variational methods. The proposed method establishes a parameterized elliptical contour field, which constrains the segmentation results to align with predefined elliptical contours. Utilizing the dual algorithm, the model seamlessly integrates image features with elliptical priors and spatial regularization priors, thereby greatly enhancing segmentation accuracy. By decomposing the SAM into four mathematical subproblems, we integrate the variational ellipse prior to design a new SAM network structure, ensuring that the segmentation output of the SAM consists of elliptical regions. Experimental results on some specific image datasets demonstrate an improvement over the original SAM. The codes are available in https://github.com/zhaoxinyum/SAM-ESP.
Progressive Fusion of Multi-Scale Mamba Context and Local Detail Priors for Infrared Small Target Detection.
Zhu X, Qin F, Wang C … +5 more , Fan J, Lin F, Bai J, Zhang C, Zhang D
IEEE Trans Image Process · 2026 · PMID 42301842 · Publisher ↗
Infrared Small Target Detection (IRSTD) requires strong target-level detection capability, which depends on effective modeling of long-range global dependencies. This demand has driven the transition from CNN-based appro...
Infrared Small Target Detection (IRSTD) requires strong target-level detection capability, which depends on effective modeling of long-range global dependencies. This demand has driven the transition from CNN-based approaches to Transformer-based architectures. Although Transformers improve global context modeling, their high computational cost limits practical deployment. Recent advances in Mamba enable efficient long-range dependency modeling with reduced complexity, offering a promising alternative that alleviates the efficiency limitations of Transformers while preserving target-level detection performance. However, Mamba is not inherently tailored for IRSTD, as it lacks explicit mechanisms for capturing fine-grained local details and modeling background variations across multiple spatial scales. To address these limitations, we propose MCFNet, an encoder-decoder framework that integrates Mamba to enhance target-level detection performance with moderate computational cost. MCFNet introduces a Detail-Capturable Convolution Block to strengthen local detail perception and a Multi-scale Contextual Mamba Block to improve background modeling across different scales. While the resulting dual-branch design enhances both global semantics and local details, it also introduces challenges in feature fusion. To this end, a Feature Fusion Decoding Module is further proposed to enable effective collaboration between global and local representations. Extensive experiments on multiple public IRSTD benchmark datasets demonstrate that MCFNet consistently outperforms existing methods in both pixel-level and target-level metrics, achieving higher detection accuracy with reduced false alarms. The code of our model is available at: https://github.com/Fihven/MCFNet.
Underwater Image Enhancement via Intelligent Optimized Multi-Exposure Image Fusion.
Zhang W, Yu B, Zhao W … +3 more , Liang Z, Zhuang P, Zhu K
IEEE Trans Image Process · 2026 · PMID 42301841 · Publisher ↗
Underwater images often suffer from visual degradation due to varying light absorption at different wavelengths and scattering from suspended particles. To tackle these issues, we present an intelligent optimized multi-e...
Underwater images often suffer from visual degradation due to varying light absorption at different wavelengths and scattering from suspended particles. To tackle these issues, we present an intelligent optimized multi-exposure image fusion method called IMIF. Specifically, we propose an adaptive color transfer strategy that employs a colorless reference image to correct the color distortion issue by transferring the mean and standard deviation of the reference image to adjust a color-balanced image. Subsequently, we introduce a particle swarm optimization algorithm that intelligently selects the optimal set of exposure image sequences by employing information entropy and edge intensity of the image as fitness metrics. Meanwhile, we leverage a guided filtering strategy to decompose the exposure image sequences into basic and detailed layers, taking into account the exposure characteristics of each layer to generate corresponding weight maps. Finally, we employ a multi-exposure fusion strategy to adaptively fuse the exposed image sequences with weight maps, producing an enhanced result. Extensive experiments conducted on three datasets demonstrate that our IMIF method outperforms state-of-the-art (SOTA) methods in both qualitative and quantitative evaluations. Additionally, the enhanced results produced by our proposed IMIF method significantly improve the accuracy of object detection and keypoint detection. The is available at https://www.researchgate.net/publication/403951386_2026-IMIF.
Slide Deformable Transformer for High-Precision LiDAR Point Cloud Compression.
Li H, Xu L, Xie L … +4 more , Gao W, Ren Z, Li G, Guo Y
IEEE Trans Image Process · 2026 · PMID 42301840 · Publisher ↗
Dynamic LiDAR point cloud compression with range images aims to reduce storage and transmission costs while preserving both spatial accuracy and temporal consistency across frames. Vision Transformers (ViTs) are commonly...
Dynamic LiDAR point cloud compression with range images aims to reduce storage and transmission costs while preserving both spatial accuracy and temporal consistency across frames. Vision Transformers (ViTs) are commonly used for cross-frame dependency modeling. However, they suffer from feature misalignment under cross-frame displacement due to fixed patch partitioning, and their global attention across all patches is costly yet ineffective for local motions. High-precision sequences also face precision loss when 16-bit range data are quantized in a single channel. To address these limitations, we propose a Slide Deformable Transformer framework for high-precision dynamic LiDAR point cloud compression, termed SDT-PCC. At its core, the proposed SDT layer restricts attention to local sliding windows, capturing fine-grained correspondences across consecutive frames. It integrates deformable convolution into cross-frame attention to adaptively sample motion-offset locations, thereby enhancing temporal alignment and motion modeling. We also propose a Radix-Decomposition Multi-Channel Quantizer (RDMCQ), which decomposes range values into multiple channels and progressively refines precision across radix levels. Consequently, these designs can produce more temporally-coherent, accurate and stable reconstructions. Experiments on the SemanticKITTI dataset show that SDT-PCC achieves high efficiency in dynamic point cloud compression. The code is available on https://github.com/SYSU-SAIL/SDT-PCC.
Learning to Super-Resolve Face Images via Dual-Domain Multi-scale Feature Interaction.
Liu L, Liu J, Zhang Q … +2 more , Xie T, Chen CLP
IEEE Trans Image Process · 2026 Jun · PMID 42301839 · Publisher ↗
Face Super-Resolution (FSR), aiming to improve the quality of Low-Resolution (LR) facial images, has been greatly propelled by the deep learning techniques. However, existing approaches, whether based on Convolutional Ne...
Face Super-Resolution (FSR), aiming to improve the quality of Low-Resolution (LR) facial images, has been greatly propelled by the deep learning techniques. However, existing approaches, whether based on Convolutional Neural Networks (CNNs) or Transformers, are either inherently damaging facial structures limited by their architectures or failing to capture essential multi-scale textures due to the rigid receptive fields. To address these concerns, we propose a novel dual-domain feature interaction method called Spatial-frequency Multi-scale feature Learning Network (SMLNet) for FSR by employing a dual-branch architecture. Specifically, the frequency branch captures high-quality global structures and fine high-frequency details, while the spatial branch operates complementarily to preserve fine-grained local texture patterns. Moreover, we further introduce a Multi-scale Spatial-frequency feature Interaction Module (MSIM), which combines a Multi-scale Feature Extraction Block (MFEB) and a Spatial-Frequency feature Interaction Module (SFIM) to interact and aggregate multi-level complementary features from the dual branches. Extensive quantitative experiments and qualitative analyses across multiple datasets, together with evaluations on real-world images, demonstrate that the proposed SMLNet significantly outperforms other state-of-the-art methods.
Large Language Model Guided Progressive Feature Alignment for Multimodal UAV Object Detection.
Wu W, Li C, Wang X … +1 more , Luo B
IEEE Trans Image Process · 2026 · PMID 42301838 · Publisher ↗
Existing multimodal UAV object detection methods often overlook the impact of semantic gaps between modalities, which makes it difficult to achieve accurate semantic and spatial alignments and ultimately limits detection...
Existing multimodal UAV object detection methods often overlook the impact of semantic gaps between modalities, which makes it difficult to achieve accurate semantic and spatial alignments and ultimately limits detection performance. To address this problem, we propose a Large Language Model (LLM) guided Progressive feature Alignment Network called LPANet, which leverages the semantic features extracted from a large language model to guide the progressive semantic and spatial alignment between modalities for multimodal UAV object detection. To employ the powerful semantic representation of LLM, we generate the fine-grained text descriptions of each object category by ChatGPT and then extract the semantic features using the large language model MPNet, providing high-level semantic priors to guide multimodal alignment. Based on the semantic features, we guide the semantic and spatial alignments in a progressive manner as follows. First, we design the Semantic Alignment Module (SAM) to pull the semantic features and multimodal visual features of each object closer, alleviating the semantic differences of objects between modalities. Second, we design the Explicit Spatial Alignment Module (ESM) by integrating the semantic relations into the estimation of feature-level offsets, alleviating the coarse spatial misalignment between modalities. Finally, we design the Implicit Spatial alignment Module (ISM), which leverages the cross-modal correlations to aggregate key features from neighboring regions to achieve implicit spatial alignment. Comprehensive experiments on two public multimodal UAV object detection datasets demonstrate that our approach outperforms state-of-the-art multimodal UAV object detectors. The source code will be released on https://github.com/Vehicle-AHU/LPANet.
DiMuS: Disentangled Multi-Signal Learning for Weakly Supervised Point-Based 3D Object Detection.
Zhang W, Zhuge Y, Zhang L … +3 more , Hu P, Wang D, Lu H
IEEE Trans Image Process · 2026 · PMID 42301837 · Publisher ↗
Weakly supervised 3D object detection has emerged as a promising paradigm to reduce the reliance on costly 3D annotations. Existing methods often rely on 2D projection constraints or heuristic priors to supervise 3D box...
Weakly supervised 3D object detection has emerged as a promising paradigm to reduce the reliance on costly 3D annotations. Existing methods often rely on 2D projection constraints or heuristic priors to supervise 3D box regression with inexpensive 2D labels. However, they still suffer from projection ambiguity and geometry inconsistency due to the entangled optimization of 3D parameters. In this paper, we propose DiMuS, a Disentangled Multi- $\boldsymbol {S}$ ignal learning framework that integrates complementary supervision from 2D boxes, LLM-derived semantic prior, and 3D geometric alignment to enhance distinct 3D properties of position, dimension, and orientation, respectively. Specifically, DiMuS incorporates three key components: (i) a Centerness-enhanced Projection Constraint (CPC) that improves position estimation through a centerness weighting strategy, (ii) a Semantic Prior Anchoring (SPA) module that leverages LLM-derived category-specific priors for robust dimension decoding, and (iii) a Rotation-aware Consistency Regularization (RCR) mechanism that enforces orientation consistency through synthetic rotations and self-supervised invariance learning. Additionally, an Adversarial Geometric Alignment (AGA) module is proposed to build attraction/repulsion forces between LiDAR points and box edges for dynamic boundary refinement. Extensive experiments on the KITTI dataset demonstrate that DiMuS outperforms previous weakly supervised methods, achieving 96.82% of fully supervised performance on car detection while maintaining robustness across different categories.
DynSUP: Dynamic Gaussian Splatting From an Unposed Image Pair.
Li W, Chen W, Qian S … +3 more , Busam B, Cremers D, Li H
IEEE Trans Image Process · 2026 · PMID 42301836 · Publisher ↗
Recent advances in 3D Gaussian Splatting have shown promising results. Existing methods typically assume static scenes and/or multiple images with prior poses. Dynamics, sparse views, and unknown poses significantly incr...
Recent advances in 3D Gaussian Splatting have shown promising results. Existing methods typically assume static scenes and/or multiple images with prior poses. Dynamics, sparse views, and unknown poses significantly increase the problem complexity due to insufficient geometric constraints. To overcome this challenge, we propose a method that can use only two images without prior poses to fit Gaussians in dynamic environments. To achieve this, we introduce two technical contributions. First, we propose an object-level two-view bundle adjustment. This strategy decomposes dynamic scenes into piece-wise rigid components, and jointly estimates the relative camera motion and dynamic object motions for dynamic Gaussian initialization. Second, we design an SE(3) field-driven Gaussian training method. It enables fine-grained motion modeling through learnable per-Gaussian transformations. Our method leads to high-fidelity novel view synthesis of dynamic scenes while accurately preserving temporal consistency and object motion. Experiments on both synthetic and real-world datasets demonstrate that our method significantly outperforms state-of-the-art approaches designed for the cases of static environments, multiple images, and/or known poses. Our project page is available at https://colin-de.github.io/DynSUP/.
Deep Error-Aware Iterative Optimization Network for Broadband Mosaiced Hyperspectral Imaging.
Xie Y, Wang N, Dian R … +2 more , Tan L, Li S
IEEE Trans Image Process · 2026 · PMID 42301835 · Publisher ↗
Snapshot hyperspectral imaging based on narrowband mosaic array encoding suffers from limitations such as low signal-to-noise ratio, limited spectral range, and low spatial resolution. To address these challenges, we pro...
Snapshot hyperspectral imaging based on narrowband mosaic array encoding suffers from limitations such as low signal-to-noise ratio, limited spectral range, and low spatial resolution. To address these challenges, we propose a novel hyperspectral imaging system that integrates broadband mosaic image with high-resolution (HR) panchromatic (PAN) image of the same scene, establishing a new paradigm for HR hyperspectral image (HSI) acquisition. To fully leverage the complementary information from multi-source images, we introduce a Deep Error-aware Iterative Optimization Network (EIONet), which iteratively reduces reconstruction errors to successfully reconstruct images with both high spatial resolution and high spectral quality. Specifically, we design a Hierarchical Error-aware Cube Updating Mechanism (HECUM) that dynamically partitions image regions based on their reconstruction difficulty during iterations. By prioritizing the enhancement of feature representation in high-difficulty areas, it effectively suppresses the accumulation and propagation of errors. Meanwhile, we employ a Physics-Based Spectral Degradation Modeling approach, constructing a spectral response function with well-defined physical meaning to accurately model the degradation process from the target domain to the observation domain. Experimental results on two public datasets demonstrate that EIONet achieves state-of-the-art performance across multiple evaluation metrics. The related code is available at: https://github.com/Xiexieiii/EIONet.
Contextual Style Coherence Network for X-Ray Prohibited Item Image Synthesis.
Wang H, Jia T, Chen D … +1 more , Deng S
IEEE Trans Image Process · 2026 · PMID 42301834 · Publisher ↗
Prohibited item detection in X-Ray baggage images plays a crucial role for preventing the social security and stability. Well annotated X-Ray prohibited item training samples show necessity in achieving high detection pe...
Prohibited item detection in X-Ray baggage images plays a crucial role for preventing the social security and stability. Well annotated X-Ray prohibited item training samples show necessity in achieving high detection performance for X-Ray inspection system. While collection of massive samples is extremely laborious and costly, especially for those X-Ray images, which need professional inspection machine. Synthesizing X-Ray images through Threat Image Projection (TIP) is a promising solution to overcome the data insufficient limitation in prohibited item detection. However, TIP based methods rarely consider the contextual style coherence between the foreground prohibited items and background images, resulting in generating low realistic X-Ray security images. For improving image quality and diversity, we propose a Contextual Style Coherence Network for X-Ray Prohibited item Image Synthesis. Specifically, we first propose a style fusion module to guarantee the style coherence and consistency between the foreground prohibited items and background images. We transfer the threat image projection from image space to feature space, and an affine transformation matrix is applied to uniformly sample the location, ratio and scale of the prohibited items to improve the sample diversity. We further normalize the features of the foreground prohibited item by implementing the style transfer through Gram matrix. Then, a mask partial convolution is designed for inpainting the non-object regions of the foreground prohibited items to achieve a better style transition, especially for the boundary parts. The whole network follows the adversarial training pipeline in an unsupervised manner guided by the incorporation of adversarial loss and total variation regularization. We evaluate the synthetic images generated by our method from different evaluating metrics including image quality and object detection performance on various prohibited item detection datasets. The results verify that our method can effectively generate realistic X-Ray prohibited item images and improve the detection performance.
Text-Driven Relation Manipulation of Diffusion Imagery.
Li Y, Zhou P, Hu H … +3 more , Qin X, Sun J, Xu Y
IEEE Trans Image Process · 2026 Jun · PMID 42284165 · Publisher ↗
Text-guided image manipulation has recently attracted significant attention. Prevailing algorithms predominantly focus on modifying the appearances of existing instances, such as texture and attribute editing, while they...
Text-guided image manipulation has recently attracted significant attention. Prevailing algorithms predominantly focus on modifying the appearances of existing instances, such as texture and attribute editing, while they often fail to address the interactions between different instances or achieve fundamental structural changes, such as multi-object editing. This paper introduces a novel text-guided manipulation task named "relation manipulation", aimed at fundamentally altering the structure of images. This task is capable of modifying the quantity of instances and, more importantly, enhancing the understanding and editing of interactions among diverse instances. Our approach comprises two main components: relation customization and multi-region guided diffusion. Relation customization fine-tunes specific relationships using a compact dataset of exemplary relations, facilitating nuanced understanding and implementation of instance interactions. Multi-region guided diffusion employs gradient optimization to update the generation process across multiple regions, integrating a fine-grained attention control strategy to minimize regional interference and conflict. Additionally, the demonstrated applications of our method in multi-region inversion underline its potential in practical scenarios, such as relation manipulation of real images and consecutive image manipulation. Compatible with different variants of Stable Diffusion models, our approach seamlessly integrates into the Stable Diffusion WebUI, enabling high-quality image generation and exceptional control over extensive manipulation. This makes it a robust tool for both academic research and creative industries.
WSformer: Wavelet-Based Sparse Transformer for Blind Image Restoration.
Sun Z, Zhang C, Li J … +2 more , Zhang M, Gao X
IEEE Trans Image Process · 2026 · PMID 42275338 · Publisher ↗
As a fundamental task in image processing, blind image restoration (BIR) faces significant challenges due to the unknown nature of the degradation process. While transformer-based methods have shown promise in various ap...
As a fundamental task in image processing, blind image restoration (BIR) faces significant challenges due to the unknown nature of the degradation process. While transformer-based methods have shown promise in various applications, they encounter difficulties in BIR. One key challenge is that the complexity of degradation easily leads to incorporate irrelevant information into their attention mechanisms, thereby hindering restoration performance. To address this challenge, sparsification strategies have been commonly adopted. However, existing sparse transformer-based methods typically determine sparse members through fixed patterns such as constant thresholds or predefined sources, making their sparsification strategies too rigid. To tackle this issue, we propose WSformer, a Wavelet-based Sparse transformer tailored for BIR, which offers three key advantages. First, we design a Sparse Reciprocal Multi-head Self-Attention (SR-MSA) mechanism in the attention layer. This mechanism employs sparse and reciprocal strategies to adaptively select reliable information, while operating across channels to reduce computational complexity. Second, recognizing that feed-forward networks in existing transformer blocks fail to effectively leverage global information, we develop a Recalibrated Feed-Forward Network (RFFN). It fully exploits the fusion of local and global information, enhancing the robustness of feature learning. Finally, to mitigate the increased computational burden introduced by these innovations, we equip WSformer with wavelet transform. Combined with a U-shaped architecture, it enables WSformer to achieve an optimal balance between performance and inference time. Extensive experiments on multiple BIR tasks validate WSformer's effectiveness in both quantitative metrics and visual quality. The code is available at https://github.com/CanZhang01/WSformer.
Visual-Textual Information-Driven Tactile Data Generation Method.
Song R, Xu Y, Tu Z … +3 more , Wang H, Tan H, Lu H
IEEE Trans Image Process · 2026 · PMID 42275337 · Publisher ↗
Tactile data can enhance the environmental perception and interaction capabilities of intelligent agents, serving as a foundational component for the development of embodied intelligence. Despite its critical role, tacti...
Tactile data can enhance the environmental perception and interaction capabilities of intelligent agents, serving as a foundational component for the development of embodied intelligence. Despite its critical role, tactile data acquisition remains cost-prohibitive and labor-intensive, resulting in severe data scarcity. Cross-modal generation offers a promising solution by leveraging abundant visual and textual data. However, effectively aligning heterogeneous visual-textual modalities under data-scarce and sparsely-annotated conditions remains a significant challenge. To address these challenges, a visual-textual information-driven tactile data generation (VTTac) framework is proposed, which features three key innovations. First, a multi-granularity text enhancement strategy is introduced to mitigate annotation sparsity through hierarchical semantic enrichment. Second, a cascaded dual cross-attention mechanism is designed to ensure cross-modal alignment. Third, a condition adapter injects a low-frequency background prior, enabling the generative backbone to focus on high-frequency texture synthesis. Subsequently, a wavelet transform seamlessly fuses these synthesized details with the real background. Extensive evaluations across three datasets demonstrate that VTTac consistently outperforms representative baselines. Furthermore, downstream tasks validate the physical faithfulness of the synthesized data for material classification and semantic reasoning, and zero-shot experiments confirm generalization to unseen objects.
Multi-Label Image Classification via Contrastive Co-Occurrence Learning.
Zhu X, Liu J, Tang D … +4 more , Liu W, Ge J, Liu B, Cao J
IEEE Trans Image Process · 2026 · PMID 42275336 · Publisher ↗
Multi-label image classification is an essential task in computer vision that aims to identify multiple objects in images. Recently, there has been growing research interest in modeling the relationships between labels t...
Multi-label image classification is an essential task in computer vision that aims to identify multiple objects in images. Recently, there has been growing research interest in modeling the relationships between labels to enhance label representation learning. An intuitive approach is to train a network to estimate label co-occurrence probabilities in a supervised manner, which are then leveraged to guide the interactions between label representations. However, the extreme sparsity of label co-occurrence signals poses substantial challenges. To address this issue, we commence by examining the potential interaction behaviors between label representations under the guidance of ground-truth label co-occurrence signals. Inspired by our findings, a novel contrastive learning mechanism is crafted to mimic and enhance such behaviors, facilitating effective label representation interactions without relying on explicit supervision from label co-occurrence signals. Based on this, we develop a pioneering contrastive co-occurrence learning framework, which operates on the instance-level label co-occurrence graph for multi-label image classification. This framework involves sequential processes of label representation learning followed by co-occurrence perception learning. Cross-entropy loss for label classification learning and contrastive loss for co-occurrence perception learning are used to jointly optimize the entire framework end-to-end. In this way, label representations can interact effectively, fully perceiving their co-occurrence relationships at the instance level, thereby significantly improving the performance in label recognition. Extensive experiments on public benchmarks demonstrate the superiority of the proposed framework in multi-label image classification. Codes are available on https://github.com/jasonseu/CoCo.
Explainable Action Form Assessment by Exploiting Multimodal Chain-of-Thoughts Reasoning.
Qi M, Wu Y, Yun W … +2 more , Zhang X, Ma H
IEEE Trans Image Process · 2026 · PMID 42275335 · Publisher ↗
Evaluating whether human action is standard or not and providing reasonable feedback to improve action standardization is very crucial but challenging in real-world scenarios. However, current video understanding methods...
Evaluating whether human action is standard or not and providing reasonable feedback to improve action standardization is very crucial but challenging in real-world scenarios. However, current video understanding methods are mainly concerned with what and where the action is, which is unable to meet the requirements. Meanwhile, most of the existing datasets lack the labels indicating the degree of action standardization, and the action quality assessment datasets lack explainability and detailed feedback. Therefore, we define a new Human Action Form Assessment (AFA) task, and introduce a new diverse dataset CoT-AFA, which contains a large scale of fitness and martial arts videos with multi-level annotations for comprehensive video analysis. We enrich the CoT-AFA dataset with a novel Chain-of-Thought explanation paradigm. Instead of offering isolated feedback, our explanations provide a complete reasoning process-from identifying an action step to analyzing its outcome and proposing a concrete solution. Furthermore, we propose a framework named Explainable Fitness Assessor, which can not only judge an action but also explain why and provide a solution. This framework employs two parallel processing streams and a dynamic gating mechanism to fuse visual and semantic information, thereby boosting its analytical capabilities. The experimental results demonstrate that our method has achieved improvements in explanation generation (e.g., + 16.0% in CIDEr),action classification (+ 2.7% in accuracy) and quality assessment (+ 2.1% in accuracy), revealing great potential of CoT-AFA for future studies. Our dataset and source code are available at https://github.com/MICLAB-BUPT/EFA.
Holistic Invariant Retracing for Distortion-Resilient Multi-Modal Learning in Spatial Transcriptomics.
He X, Zhao H, Wang D … +2 more , Tao D, Du B
IEEE Trans Image Process · 2026 · PMID 42275334 · Publisher ↗
Spatial transcriptomics provides a multi-modal perspective by simultaneously capturing gene expression profiles, spatial coordinates, and histological images. While existing methods focus on maintaining view consistency...
Spatial transcriptomics provides a multi-modal perspective by simultaneously capturing gene expression profiles, spatial coordinates, and histological images. While existing methods focus on maintaining view consistency to handle distribution shifts, they frequently neglect semantic conflicts introduced by distorted views-a common limitation arising from technical data acquisition and processing constraints. These conflicts lead to distorted consensus representations. To address this challenge, we propose Holistic Invariant RetrAcing for mitigating representation distortion (HiraST). Our framework explicitly corrects distorted multi-view representations through two complementary mechanisms: 1) Cross-view invariant retracing, which jointly aligns instance-level features and pseudo-label distributions to retrace invariant information. This dual alignment ensures that semantically similar cells or tissue regions remain consistent across heterogeneous modalities, even in the presence of acquisition-induced distortions; and 2) holistic prototype learning, which leverages low-frequency structural components to recalibrate corrupted views and enhance robustness against noise. Extensive experiments on spatial transcriptomics datasets and incomplete multi-view clustering benchmarks demonstrate our framework's state-of-the-art performance. Meanwhile, HiraST demonstrates strong capability across various downstream tasks. The demo code of this work is publicly available at https://github.com/hexiao0275/HiraST.
A Geometric Framework for Absolute Pose and Velocity Estimation With Event Cameras.
Liu Z, Liang S, Guan B … +3 more , Shang Y, Yu Q, Zhao J
IEEE Trans Image Process · 2026 · PMID 42275333 · Publisher ↗
Despite the rapid advancements in event-based motion estimation, current geometric methods primarily focus on velocity estimation. However, absolute pose estimation, which is equally crucial for key applications such as...
Despite the rapid advancements in event-based motion estimation, current geometric methods primarily focus on velocity estimation. However, absolute pose estimation, which is equally crucial for key applications such as robotic navigation and augmented reality, remains relatively underexplored. Consequently, the simultaneous recovery of absolute pose and velocity from event streams remains an open and challenging problem. To address this gap, we propose a geometric framework for absolute pose and velocity estimation by leveraging 3D lines in the scene and the events they trigger. At the core of the framework lie two key geometric constraints: the orthogonality between a 3D line and the normal vector of its corresponding event plane, and the collinearity of an event with the 2D projection of its associated line. Based on these constraints, we present both linear and polynomial solvers for absolute pose estimation. The former enables efficient computation, while the latter provides a globally optimal solution for rotation. For velocity estimation, we develop an efficient linear solver and a more accurate optimization-based solver to recover both angular and linear velocities. Notably, our methods require a minimum of three event-line correspondences to determine the 6-DoF absolute pose or velocities independently. Extensive experiments in simulation and on real-world datasets demonstrate that our methods achieve state-of-the-art performance, with significant improvements in accuracy and computational efficiency compared to existing methods. The demo code is publicly available at https://github.com/Zibin6/EventPoseVelocity.
CSVSUF: A Deep Unfolding Framework for Compressive Spectral Video Sensing.
Li Z, Wang H, Duan J … +2 more , Li B, Liu Y
IEEE Trans Image Process · 2026 · PMID 42275332 · Publisher ↗
Spectral videos (SVs) capture spatio-temporal-spectral information from dynamic scenes, but their acquisition traditionally requires expensive and complex systems, motivating the development of compressive spectral video...
Spectral videos (SVs) capture spatio-temporal-spectral information from dynamic scenes, but their acquisition traditionally requires expensive and complex systems, motivating the development of compressive spectral video sensing (CSVS). It typically employs the coded aperture snapshot spectral imager (CASSI) to acquire compressed measurements, from which SVs are reconstructed via model-driven or learning-based algorithms. However, two major limitations remain in current CASSI-based reconstruction methods: 1) conventional model-driven algorithms rely on iterative optimization, which limits their representational capacity in complex scenes and results in slow reconstruction; 2) existing deep learning-based approaches overlook the joint modeling of spatial, temporal, and spectral correlations, failing to fully exploit the multi-dimensional dependencies. Hence, we propose a principled compressive spectral video sensing unfolding framework (CSVSUF) in a CASSI system for spectral video reconstruction. Moreover, we develop a novel spatio-temporal-spectral prior-learning Transformer (STS-PLT) to capture the multi-dimensional correlations within each unfolding stage. By treating STS-PLT as a Gaussian denoiser for the prior term in CSVSUF, we establish a deep unfolding-based method for CSVS. Extensive experiments demonstrate that our method consistently outperforms existing approaches in both reconstruction accuracy and visual quality, validating the benefit of combining physics-guided modeling with deep prior learning in CSVS. Code is available at https://github.com/zli1024/CSVSUF.
About
- Frequency
- Sun
- Papers found
- 200
- RSS feed
- Subscribe