IEEE Transactions On Image Processing[JOURNAL]
Sun
200 papers
Leveraging Feature Alignment in Grassmannian Manifold for Multi-Output Regression Tasks.
Kong L, Zdrazil J, De Diego N … +4 more , Ignacio Jasso Robles F, Snasel V, Das S, Pan JS
IEEE Trans Image Process · 2026 · PMID 42202189 · Publisher ↗
Despite notable progress in domain adaptation for classification, regression-based domain adaptation remains challenging, particularly in terms of handling complex data structures, ensuring cross-domain generalization, a...
Despite notable progress in domain adaptation for classification, regression-based domain adaptation remains challenging, particularly in terms of handling complex data structures, ensuring cross-domain generalization, and maintaining the precision and mathematical rigor required to validate model effectiveness. Unlike classification tasks, which are more resilient to variations in feature scaling, regression tasks are notably more sensitive, making their performance more vulnerable in domain adaptation scenarios. In this paper, we propose a generalized regularization technique grounded in the Grassmannian manifold to address the feature alignment problem. This approach leverages the underlying manifold structure of the data while preserving mathematical bounds, thereby enhancing the precision and efficiency of problem-solving. To demonstrate the effectiveness of the proposed algorithm, we apply it to estimate multi-output parameters in two distinct domains: 1) the Arabidopsis thaliana plant dataset, collected from a high-throughput phenotyping platform at Palacký University, and 2) the publicly available dSprites shape recognition with six adaptation tasks. These tasks are critical to advance agricultural research and address generalization challenges in multi-output regression. Accurate predictions provide deeper insights into plant growth and health, thereby supporting more effective crop management strategies. We evaluate the effectiveness of our framework by comparing it with state-of-the-art regression alignment techniques that are independent of the underlying backbone and adaptable to transfer learning tasks. Experimental results show that our framework consistently outperforms existing methods,results description. The source code will be made publicly available upon acceptance at https://github.com/lingping-fuzzy.
Neural Wave Propagation for Surgical Video Action Recognition: A New Dataset and Baseline.
Chen T, Wang W, Tan Z … +6 more , Zhou R, Wu Y, Wang Z, Lu L, Wang M, Ye Z
IEEE Trans Image Process · 2026 · PMID 42202188 · Publisher ↗
Accurate and efficient recognition of surgical actions in videos is critical for advancing AI-driven surgical robotics. However, current surgical video action recognition (SVAR) datasets suffer from limitations such as s...
Accurate and efficient recognition of surgical actions in videos is critical for advancing AI-driven surgical robotics. However, current surgical video action recognition (SVAR) datasets suffer from limitations such as small scale, low resolution, inconsistent annotations, and insufficient action coverage. Most latest video recognition models are trained on large-scale common datasets and underperform in SVAR due to architectures that suppress high-frequency visual details (crucial for recognizing surgical tools and motions) and lack a strong spatial inductive bias, requiring extensive training data for good convergence. This is particularly challenging in the surgical domain, where data access is limited. Therefore, a new baseline is required. To address these issues, we introduce LapSurg-230K, an SVAR dataset of 7,569 high-resolution laparoscopic surgical video clips with 230,246 frames, well-annotated for 11 key actions across 9 surgery types. It supports both full and progressive data volume evaluation settings. We further propose WaveR, an attention-free baseline based on physical wave propagation. WaveR embeds an innate physical inductive bias: each video patch acts as a wave source that propagates waves toward action-critical regions (e.g., instrument tips), adaptively aggregating spatial-temporal context while preserving high-frequency surgical cues. This mechanism eliminates dependency on massive training data. Experiments demonstrate WaveR's robustness under extreme data scarcity ( $\leq 30\%$ training samples), achieving state-of-the-art accuracy on both surgical video action recognition and phase recognition tasks. The complete dataset, licensed under CC-BY 4.0, is available at https://doi.org/10.6084/m9.figshare.32237319. Our code is available at https://github.com/yezizi1022/WaveR_TIP.
Correlation-Guided Recursive Pyramid Network for Deformable Brain MRI Registration.
Zhang W, Liu Y
IEEE Trans Image Process · 2026 · PMID 42202187 · Publisher ↗
As a key preprocessing technique in medical image analysis, deformable image registration has remained a research focus over the past decade. Recently, deep learning-based registration methods have become mainstream. Nev...
As a key preprocessing technique in medical image analysis, deformable image registration has remained a research focus over the past decade. Recently, deep learning-based registration methods have become mainstream. Nevertheless, simultaneously handling large-scale deformations and accurate feature matching remains a persistent challenge. While pyramid architectures are widely employed to mitigate large-scale deformations, existing methods often exhibit an unbalanced focus. One group emphasizes iterative refinement to handle large deformations but relies on implicit, coarse feature interactions. Conversely, the other group concentrates on explicit matching techniques, but such static matching is often unreliable in regions with significant anatomical discrepancies. To bridge this gap, we propose a novel Correlation-Guided Recursive Pyramid Network (CRPNet). Unlike previous approaches, CRPNet addresses these challenges in a unified manner by embedding explicit correlation modeling directly into the recursive optimization. Specifically, we propose a Correlation-Guided Intra-layer Recursive Strategy (CGIRS), which enables the network to continuously refine matching accuracy through recursive feedback while preventing cross-scale error propagation. To facilitate this, we design a Spatial Correlation Module (SPCM) for accurate spatial correspondence and a Semantic Correlation Module (SECM) for high-level semantic alignment. Extensive experiments on three brain imaging datasets demonstrate that our method achieves state-of-the-art performance, particularly exhibiting exceptional robustness under extreme deformations, proving the efficacy of our method for deformable brain MRI registration. The code is available at https://github.com/ZhangWH0129/CRPNet.
SimMTC: Simple Multi-View Tensor Clustering.
Xin H, Hao Z, Cao Z … +3 more , Zhao Z, Wang R, Nie F
IEEE Trans Image Process · 2026 · PMID 42202186 · Publisher ↗
Tensor-based multi-view clustering algorithms have attracted considerable attention due to their superior clustering performance. However, these algorithms typically treat each view independently, failing to utilize the...
Tensor-based multi-view clustering algorithms have attracted considerable attention due to their superior clustering performance. However, these algorithms typically treat each view independently, failing to utilize the complementary information across all views, thus lacking globality. Additionally, employing low-rank tensor constraints to extract consistent information among views may result in the loss of important information due to weak consistency constraints. These limitations significantly hinder the clustering performance. To address these issues, we propose Simple Multi-view Tensor Clustering (SimMTC), which achieves globality and strong consistency. SimMTC first applies Fast Fourier Transform (FFT) to the anchor graphs to obtain high-frequency and low-frequency information, which encode similarities between samples and anchors from all views, thereby capturing global information. Orthogonal tensor factorization is then conducted in the frequency domain. Moreover, a novel strong consistency constraint based on FFT is introduced, which enhances the extraction of consistent information in the frequency domain. What's more, an efficient alternating optimization algorithm is designed to solve the optimization problem in SimMTC. Finally, extensive experiments on real-world datasets demonstrate that SimMTC achieves state-of-the-art clustering performance. The code has been made publicly available on GitHub at: https://github.com/haonanxin/SimMTC_code.
Robust Fine-Grained Oriented Ship Detection for Remote Sensing imagery via Controllable Generative Pretraining.
He D, Liu H, Li Z … +8 more , Zhao Y, Hu X, Zhong P, Li W, Shi Q, Liu X, Zhong Y, Zhang L
IEEE Trans Image Process · 2026 May · PMID 42184176 · Publisher ↗
Fine-grained ship recognition in remote sensing imagery is essential for maritime applications. However, its development is hindered by two challenges: (1) the limited granularity of existing ship detection datasets, and...
Fine-grained ship recognition in remote sensing imagery is essential for maritime applications. However, its development is hindered by two challenges: (1) the limited granularity of existing ship detection datasets, and (2) the disturbance of complex maritime conditions as well as the arbitrary ship orientations and distributions. To address the first issue, we annotated a large-scale fine-grained ship instance detection dataset (LAFI), comprising 48,717 ship instances worldwide with 49 categories. To tackle the challenges of marine disturbance and diverse ship status, we proposed a controllable generative knowledge-driven ship detection framework (COSD). It employs a controllable generative model guided by ship-marine knowledge to generate millions of synthetic images that not only preserve ship structures but also cover diverse sea and weather conditions for robust pretraining. Furthermore, a heterogeneous feature alignment decoder is designed to align multi-modal metrics of orientation and distribution features in the latent space, allowing for accurate representation of diverse ship status. Extensive experiments on two benchmark datasets showed that our method respectively increased 0.093 and 0.129 mean average precision (mAP) over SOTA methods, particularly in scenarios involving small, densely packed and arbitrary oriented ships.
TVRN: Invertible Neural Networks for Compression-Aware Temporal Video Rescaling.
Feng X, Li L, Liu D … +1 more , Wu F
IEEE Trans Image Process · 2026 · PMID 42184175 · Publisher ↗
To fit diverse display and bandwidth constraints, high-frame-rate videos are temporally downscaled to low-frame-rate (LFR) and later upscaled, requiring joint optimization for effective frame-rate rescaling. However, exi...
To fit diverse display and bandwidth constraints, high-frame-rate videos are temporally downscaled to low-frame-rate (LFR) and later upscaled, requiring joint optimization for effective frame-rate rescaling. However, existing methods typically link the two operations via training objectives, without fully exploiting their reciprocal nature, which may cause high-frequency information loss. Moreover, they overlook the impact of lossy codecs on LFR videos, limiting real-world applicability. In this work, we propose an end-to-end framework for compression-aware frame-rate rescaling, named TVRN. To regularize high-frequency information lost during frame-rate downscaling, TVRN adopts an invertible architecture that combines a Multi-Input Multi-Output Temporal Wavelet Transform with a high-frequency reconstruction module. To enable end-to-end training through non-differentiable lossy codecs, we design a surrogate network that approximates their gradients. Finally, to improve robustness under various compression levels, we extend TVRN to an asymmetric architecture by incorporating compression-aware features learned via a learning-to-rank strategy. Extensive experiments show that TVRN outperforms existing methods in reconstruction quality under industrial video compression settings. Source code is publicly available at https://github.com/fengxinmin/TVRN_public.
ISTASTrack: Bridging ANN and SNN via ISTA Adapter for RGB-Event Tracking.
Liu S, Wang Z, Zheng H … +6 more , Hu Y, Wang X, Yang Q, Wu J, Guo H, Deng L
IEEE Trans Image Process · 2026 · PMID 42172158 · Publisher ↗
RGB-Event tracking has become a promising trend in visual object tracking to leverage the complementary strengths of both RGB images and dynamic spike events for improved performance. However, existing artificial neural...
RGB-Event tracking has become a promising trend in visual object tracking to leverage the complementary strengths of both RGB images and dynamic spike events for improved performance. However, existing artificial neural networks (ANNs) struggle to fully exploit the sparse and asynchronous nature of event streams. Recent efforts toward hybrid architectures combining ANNs and spiking neural networks (SNNs) have emerged as a promising solution in RGB-Event perception, yet effectively fusing features across heterogeneous paradigms remains a challenge. In this work, we propose ISTASTrack, the first transformer-based ANN-SNN hybrid Tracker equipped with ISTA adapters for RGB-Event tracking. The two-branch model employs a vision transformer to extract spatial context from RGB inputs and a spiking transformer to capture spatio-temporal dynamics from event streams. To bridge the modality and paradigm gap between ANN and SNN features, we systematically design an ISTA adapter for bidirectional feature interaction between the two branches. The ISTA adapter is derived from the sparse representation theory by unfolding the iterative shrinkage-thresholding algorithm. Additionally, we incorporate a temporal downsampling attention module within the adapter to align multi-step SNN features with single-step ANN features in the latent space. Experimental results on RGB-Event tracking benchmarks, such as FE240hz, VisEvent, COESOT, and FELT, have demonstrated that ISTASTrack achieves state-of-the-art performance while maintaining high energy efficiency. This work highlights the effectiveness and practicality of hybrid ANN-SNN designs for robust visual tracking. The code is publicly available at https://github.com/lsying009/ISTASTrack.git.
COMBINER: Composed Image Retrieval Guided by Attribute-Based Neighbor Relations.
Li Z, Hu Y, Chen Z … +3 more , Wen H, Song X, Nie L
IEEE Trans Image Process · 2026 · PMID 42172157 · Publisher ↗
Composed Image Retrieval (CIR) represents a challenging retrieval task that targets locating specific images through multimodal inputs. Despite recent progress in CIR techniques, prior approaches often overlook cases whe...
Composed Image Retrieval (CIR) represents a challenging retrieval task that targets locating specific images through multimodal inputs. Despite recent progress in CIR techniques, prior approaches often overlook cases where images appear visually alike yet differ in attributes, potentially undermining both multimodal feature fusion and similarity modeling. To mitigate this limitation, we design a unified representation of cross-modal features based on attribute prototypes. Nevertheless, the task is far from straightforward, owing to three core issues: (1) entanglement in attribute-level semantics, (2) inconsistency across modalities, and (3) supervised signal missing. To tackle the above obstacles, we introduce a COMposed image retrieval network guided By attrIbute-based NEighbor Relations (COMBINER). Specifically, we first design an Adaptive Semantic Disentanglement module, which is capable of disentangling attribute features based on multimodal primitive features. Secondly, we propose a Unified Prototype-based Composition module, which can construct cross-modal unified prototypes (CUP) and facilitate multimodal feature composition. Finally, we introduce a Dual Relations Modeling module, which can mine pairwise and neighbor relations based on attribute similarity. Compared to traditional neighbor relations modeling CIR methods, COMBINER represents the first study addressing the phenomenon of visually similar but attribute-unrelated samples. It achieves a more accurate understanding of the semantic relations among samples by employing an attribute prototype-based similarity metric. Comprehensive experiments conducted on three benchmark datasets confirm the effectiveness of our proposed COMBINER. The implementation of our method will be accessed at https://github.com/Lee-zixu/COMBINER.
PolarGS: Polarimetric Cues for Ambiguity-Free Gaussian Splatting With Accurate Geometry Recovery.
Guo B, Wen S, Zhao Y … +2 more , Li J, Zheng Z
IEEE Trans Image Process · 2026 · PMID 42172156 · Publisher ↗
Recent advances in surface reconstruction for 3D Gaussian Splatting (3DGS) have enabled remarkable geometric accuracy. However, their performance degrades in photometrically ambiguous regions such as reflective and textu...
Recent advances in surface reconstruction for 3D Gaussian Splatting (3DGS) have enabled remarkable geometric accuracy. However, their performance degrades in photometrically ambiguous regions such as reflective and textureless surfaces, where unreliable cues disrupt photometric consistency and hinder accurate geometry estimation. Reflected light is often partially polarized in a manner that reveals surface orientation, making polarization an optical complement to photometric cues in resolving such ambiguities. Therefore, we propose PolarGS, an optics-aware extension of RGB-based 3DGS that leverages polarization as an optical prior to resolve photometric ambiguities and enhance reconstruction accuracy. Specifically, we introduce two complementary modules: a polarization-guided photometric correction strategy, which ensures photometric consistency by identifying reflective regions via the Degree of Linear Polarization (DoLP) and refining reflective Gaussians with Color Refinement Maps; and a polarization-enhanced Gaussian densification mechanism for textureless area geometry recovery, which integrates both Angle and Degree of Linear Polarization (A&DoLP) into a PatchMatch-based depth completion process. This enables the back-projection and fusion of new Gaussians, leading to a more complete reconstruction. PolarGS is framework-agnostic and achieves superior geometric accuracy compared to state-of-the-art methods.
RUMPL: Ray-Based Transformers for Universal Multi-View 2D to 3D Human Pose Lifting.
Ghasemzadeh SA, Alahi A, De Vleeschouwer C
IEEE Trans Image Process · 2026 · PMID 42172155 · Publisher ↗
Estimating 3D human poses from 2D images remains challenging due to occlusions and projective ambiguity. Multi-view learning-based approaches mitigate these issues but often fail to generalize to real-world scenarios, as...
Estimating 3D human poses from 2D images remains challenging due to occlusions and projective ambiguity. Multi-view learning-based approaches mitigate these issues but often fail to generalize to real-world scenarios, as large-scale multi-view datasets with 3D ground truth are scarce and captured under constrained conditions. To overcome this limitation, recent methods rely on 2D pose estimation combined with 2D-to-3D pose lifting trained on synthetic data. Building on our previous MPL framework, we propose RUMPL, a transformer-based 3D pose lifter that introduces a 3D ray-based representation of 2D keypoints. This formulation enables a model agnostic to camera parameters that can be universally deployed across arbitrary camera configurations in a given area without retraining or fine-tuning. A new View Fusion Transformer leverages learned fused-ray tokens to aggregate information along rays, further improving multi-view consistency. Evaluation on standard benchmarks shows that RUMPL significantly outperforms existing methods, yielding a 56.6% MPJPE (All KP) reduction on Human3.6M over triangulation-based methods and exceeding 70% improvement on the CMU Panoptic dataset when compared to transformer-based image-representation approaches. Results on new benchmarks, including in-the-wild multi-view and multi-person datasets, confirm its robustness and scalability.
Point-RMAE: Reinforcement Masked Autoencoder for 3D Representation Learning.
Cheng H, Wei L, Wang W … +5 more , Yan W, Chen J, Lu J, Yue K, Zhu J
IEEE Trans Image Process · 2026 · PMID 42172154 · Publisher ↗
The Mainstream 3D masked point modeling representation learning community typically employs predefined, fixed-ratio random or block masking strategies, aiming to obtain optimal representations and achieve high downstream...
The Mainstream 3D masked point modeling representation learning community typically employs predefined, fixed-ratio random or block masking strategies, aiming to obtain optimal representations and achieve high downstream performance. However, these empirical designs overlook the significant geometric information and structural importance differences that are inherent among different 3D points, leading to a suboptimal trade-off between the representation capture capabilities and reconstruction difficulty of such masking strategies. To address this issue, we are the first to present this decision-making problem to a reinforcement learning agent and propose a Reinforcement Masked Autoencoder for 3D representation learning, named Point-RMAE. Guided by geometric features as state factor, this method leverages the Masking Strategy Analyzer and the Dynamic Masking Generator to adaptively decide and apply the masking strategy during pretraining. The Masking Ratio Scheduling module dynamically adjusts the masking ratio based on the optimal strategy. Subsequently, the analyzer is updated by multiscale rewards derived from reconstruction quality level, distribution-aware feedback, and policy exploration. Notably, to enrich the Reward Function with distribution-aware signals and avoid decision collapse issue, we propose a Flow Matching Point Cloud Fast Generator that guides the selected masking decisions. Our method achieves outstanding performance across downstream tasks such as shape classification, medical diagnosis, object detection, action recognition, denoising and multiscale scene segmentation on ten popular 3D and 4D datasets. More importantly, Point-RMAE pioneers the application of reinforcement learning in 3D self-supervised representation learning.
3D-UIR: 3D Gaussian for Underwater 3D Scene Reconstruction via Physics-Based Appearance-Medium Decoupling.
Yuan J, Li Y, Zhang Y … +4 more , Guo C, Tang X, Wang R, Li C
IEEE Trans Image Process · 2026 · PMID 42172153 · Publisher ↗
Novel view synthesis for underwater scene reconstruction presents unique challenges due to complex light-media interactions. Optical scattering and absorption in water body bring inhomogeneous medium attenuation interfer...
Novel view synthesis for underwater scene reconstruction presents unique challenges due to complex light-media interactions. Optical scattering and absorption in water body bring inhomogeneous medium attenuation interference that disrupts conventional volume rendering assumptions of uniform propagation medium. While 3D Gaussian Splatting (3DGS) offers real-time rendering capabilities, it struggles with underwater inhomogeneous environments where scattering media introduces artifacts and inconsistent appearance. In this study, we propose a physics-based framework that disentangles object appearance from water medium effects through tailored Gaussian modeling. Our approach introduces appearance embeddings, which are explicit medium representations for backscatter and attenuation, enhancing scene consistency. In addition, we propose a depth-guided optimization strategy that leverages pseudo-depth maps as supervision with depth regularization and scale penalty terms to improve geometric fidelity. By integrating the proposed appearance and medium modeling components via an underwater imaging model, our approach achieves both high-quality novel view synthesis and physically accurate scene restoration. Experiments demonstrate our significant improvements in rendering quality and restoration accuracy over existing methods. The project page is available at https://bilityniu.github.io/3D-UIR.
OutDreamer: Video Outpainting With a Diffusion Transformer.
Zhong L, Li F, Huang Y … +3 more , Liu J, Pei R, Song F
IEEE Trans Image Process · 2026 · PMID 42172152 · Publisher ↗
Video outpainting is a challenging task that generates new video content by extending beyond the boundaries of an original input video, requiring both temporal and spatial consistency. Many existing methods utilize laten...
Video outpainting is a challenging task that generates new video content by extending beyond the boundaries of an original input video, requiring both temporal and spatial consistency. Many existing methods utilize latent diffusion models with U-Net backbones but still struggle to achieve high quality and adaptability in generated content. Diffusion transformers (DiTs) have emerged as a promising alternative because of their superior performance. We introduce OutDreamer, a DiT-based video outpainting framework comprising two main components: a video control branch and a conditional outpainting branch. The video control branch effectively extracts masked video information, while the conditional outpainting branch generates missing content based on these extracted conditions. Additionally, we propose a mask-driven self-attention layer that dynamically integrates the given mask information, further enhancing the model's adaptability to outpainting tasks. Furthermore, we introduce a latent alignment loss to maintain overall consistency both within and between frames. For long video outpainting, we employ a cross-video-clip refiner to iteratively generate missing content, ensuring temporal consistency across video clips. Extensive evaluations demonstrate that our OutDreamer outperforms existing video outpainting methods on widely recognized benchmarks.
Pseudo Sentences Evaluation and Quality-Aware Robust Learning for Unsupervised Text-Based Person Search.
Niu K, Chen J, Han K … +2 more , Song X, Zhang Y
IEEE Trans Image Process · 2026 · PMID 42172151 · Publisher ↗
Unsupervised Text-Based Person Search (TBPS) eliminates the need for costly manual sentence annotations by generating pseudo sentences via Multi-modal Large Language Models (MLLMs). However, these pseudo sentences often...
Unsupervised Text-Based Person Search (TBPS) eliminates the need for costly manual sentence annotations by generating pseudo sentences via Multi-modal Large Language Models (MLLMs). However, these pseudo sentences often face the quality defect issues, resulting in semantic misalignment across modalities, which will hinder discriminative representation learning. To address this problem, we propose the PSE-QRL (Pseudo Sentences Evaluation and Quality-aware Robust Learning), a unified framework that enhances robustness to pseudo sentences for unsupervised TBPS. The PSE-QRL dynamically couples an evolving TBPS model with MLLMs to assess pseudo sentences' reliability, and adaptively leverages high-quality ones during training. It consists of three key components: 1) Multi-granularity Sentence Augmentation, for enriching pseudo sentences with multiple granularities to broaden the diversity of image-sentence pairs; 2) Hybrid Quality Evaluation, to combine MLLM's cross-modal reasoning knowledge with TBPS model's person-specific distinguishing capabilities for effective sentence quality assessment; and 3) Quality-aware Robust Learning, for selecting and re-weighting samples based on quality scores to emphasize reliable sentence annotations while suppressing low-quality ones. Extensive experiments on CUHK-PEDES, ICFG-PEDES, and RSTPReid benchmarks demonstrate the effectiveness of PSE-QRL for improving learning robustness, achieving state-of-the-art (SOTA) retrieval performance for unsupervised TBPS.
Pose Guided Unsupervised Domain Adaptation for Human Body Part Segmentation.
Dutta A, Lal R, Garg Y … +3 more , Ta CK, Raychaudhuri DS, Roy-Chowdhury AK
IEEE Trans Image Process · 2026 · PMID 42160254 · Publisher ↗
Existing algorithms for human body part segmentation have shown promising results on challenging datasets, primarily relying on end-to-end supervision. However, these algorithms exhibit severe performance drops in the fa...
Existing algorithms for human body part segmentation have shown promising results on challenging datasets, primarily relying on end-to-end supervision. However, these algorithms exhibit severe performance drops in the face of domain shifts, leading to inaccurate segmentation masks. To tackle this issue, we introduce POSTURE: Pose Guided Unsupervised Domain Adaptation for Human Body Part Segmentation - an innovative pseudo-labelling approach 0designed to improve segmentation performance on the unlabeled target data. Distinct from conventional domain adaptive methods for general semantic segmentation, POSTURE stands out by considering the underlying structure of the human body and uses anatomical guidance from pose keypoints to drive the adaptation process. This strong inductive prior translates to impressive performance improvements, averaging 8% over existing state-of-the-art domain adaptive semantic segmentation methods across three benchmark datasets. Furthermore, the inherent flexibility of our proposed approach facilitates seamless extension to source-free settings (SF-POSTURE), effectively mitigating potential privacy and computational concerns, with negligible drop in performance.
Degradation-Adaptive Denoising: Aligning Diffusion Models With Physics of Video Snapshot Compressive Imaging.
Zhang M, Li M, Guo J … +1 more , Li Y
IEEE Trans Image Process · 2026 · PMID 42154694 · Publisher ↗
Video Snapshot Compressive Imaging (SCI) captures multiple video frames in a single exposure, enabling efficient reconstruction of high-speed scenes for motion analysis and event detection. Existing SCI in coded aperture...
Video Snapshot Compressive Imaging (SCI) captures multiple video frames in a single exposure, enabling efficient reconstruction of high-speed scenes for motion analysis and event detection. Existing SCI in coded aperture compressive temporal imaging (CACTI) methods predominantly rely on feedforward deep networks with fixed denoising strategies. However, they lack alignment with the SCI physical inverse model and struggle to balance motion detail recovery and static background smoothing. In this paper, we propose PCD-Diffusion for Video SCI, the first diffusion-based reconstruction framework for Video SCI, which reformulates the inverse problem as a progressive denoising process. Specifically, we design a Physically-Constrained Dynamic Diffusion (PCD-Diffusion) model, introducing a region-adaptive diffusion schedule and spatiotemporal residual estimation. This method explicitly aligns the denoising process with SCI's spatially non-uniform and temporally evolving residual distribution. Additionally, a motion prior-guided diffusion schedule and a Gauss-guided spatiotemporal adaptive residual estimation dynamically steer the denoising trajectory, ensuring accurate motion detail restoration and physically consistent reconstructions. Extensive results on simulated and real datasets verify the superior reconstruction fidelity and temporal coherence of the proposed PCD-Diffusion framework over existing approaches. Code will be released upon publication.
Open-Set Anomaly Segmentation in Complex Scenarios.
Xia S, Yu Y, Ding H … +4 more , Yang W, Liu S, Kot AC, Jiang X
IEEE Trans Image Process · 2026 · PMID 42154693 · Publisher ↗
Precise segmentation of out-of-distribution (OoD) objects, herein referred to as anomalies, is crucial for the reliable deployment of semantic segmentation models in open-set, safety-critical applications, such as autono...
Precise segmentation of out-of-distribution (OoD) objects, herein referred to as anomalies, is crucial for the reliable deployment of semantic segmentation models in open-set, safety-critical applications, such as autonomous driving. Current anomalous segmentation benchmarks predominantly focus on favorable weather conditions, resulting in untrustworthy evaluations that overlook the risks posed by diverse meteorological conditions in open-set environments, such as low illumination, dense fog, and heavy rain. To bridge this gap, this paper introduces the ComSAmy, a Complex Scenarios Anomaly segmentation benchmark. ComSAmy encompasses a wide spectrum of adverse weather conditions, dynamic driving environments, and diverse anomaly types to comprehensively evaluate the model performance in realistic open-world scenarios. Our extensive evaluation of several state-of-the-art anomalous segmentation models reveals that existing methods demonstrate significant deficiencies in such challenging scenarios, highlighting their serious safety risks for real-world deployment. To solve that, we propose a novel energy-entropy learning (EEL) strategy that integrates the complementary information from energy and entropy to bolster the robustness of anomaly segmentation under complex open-world environments. Additionally, a diffusion-based anomalous training data synthesizer is proposed to generate diverse and high-quality anomalous images to enhance the existing copy-paste training data synthesizer. Extensive experimental results on both public and ComSAmy benchmarks demonstrate that our proposed diffusion-based synthesizer with energy and entropy learning (DiffEEL) framework serves as an effective and generalizable plug-and-play method to enhance existing models, yielding an average improvement of around 4.96% in AUPRC and 9.87% in $\rm {FPR}_{95}$ .
Simpler is Better: Feature Guard and Interaction for Semantic Correspondence.
Wang Z, Du S, Xiao G
IEEE Trans Image Process · 2026 · PMID 42154692 · Publisher ↗
Semantic correspondence establishes keypoint correspondences between different instances of the same category. Fusing texture and semantic features from vision foundation models like stable diffusion (SD) and DINO signif...
Semantic correspondence establishes keypoint correspondences between different instances of the same category. Fusing texture and semantic features from vision foundation models like stable diffusion (SD) and DINO significantly improves matching performance. However, we found an unnoticed yet essential problem: current feature fusion enhances the edge and semantic information in SD features with fine textures and DINOv2 features with fine semantics, but it destroys the semantic and structural information in SD features with weak and coarse semantics. We propose guard features (GuFT), a simple yet efficient method, to prevent feature degradation. Moreover, matching methods designed for traditional deep neural networks can be simplified based on two key insights: 1) vision foundation models provide rich visual knowledge; and 2) GuFT yields high-quality feature descriptors. We propose a bottleneck-style non-shared aggregation and backward interaction (NABI) module to efficiently capture intra- and inter-feature relationships, instead of common self- and cross-attention. The resulting framework, SimBetter, embodies a "simpler is better" design philosophy. It achieves state-of-the-art results with lower computation on SPair-71k, AP-10K, and PF-PASCAL, excelling in geometry-aware, cross-species, cross-family, and cross-dataset tasks. SimBetter also shows excellent potential in the applications of image-video semantic correspondence and sticker editing. Code is available at https://github.com/wzhlearning/SimBetter.
GarmentRec: Towards Individual Garment Reconstruction From a Monocular Human Image.
Xu Z, Gao Z, Cheng S … +4 more , Fei W, Zhang Q, Li W, Gao X
IEEE Trans Image Process · 2026 · PMID 42154691 · Publisher ↗
Reconstructing high-quality garment models from monocular images is important as it provides a practical and effective solution for human digitization and virtual try-on etc. Recent implicit function-based garment recons...
Reconstructing high-quality garment models from monocular images is important as it provides a practical and effective solution for human digitization and virtual try-on etc. Recent implicit function-based garment reconstruction methods recover free-form geometry but struggle to reconstruct individual garment meshes from human images and tend to produce disembodied limbs or degenerate shapes for novel views. In contrast, explicit parametric garment template models can be utilised to construct separate meshes and constrain the shape reconstruction robustly. However, this limits the reconstruction of garment details and shape variations, such as the wrinkles and pockets etc. To address this problem, in this paper, we introduce a novel explicit garment template that is designed for both closed and open garment topology. Powered by our new garment template, we further propose a detailed garment reconstruction method based on a monocular view that can process both the closed and open types for shape recovery. To capture those challenging parts with unknown geometry and topology, we predict displacement maps on the parameterization domain for the target garment from the monocular image and elaborate it to the 3D garment surface via the UV coordinates, achieving realistic details on the 3D garment shape. Extensive experiments demonstrate the accuracy and robustness of our method and show that realistic details like garment wrinkles and pockets can be faithfully recovered in an explicit way. The code and dataset are available at https://github.com/worryDes/GarmentRec.
DiCLIP: Diffusion Model Enhances CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation.
Yang Z, Song P, Meng Y … +3 more , Fu K, Wang S, Song Z
IEEE Trans Image Process · 2026 · PMID 42139125 · Publisher ↗
Weakly Supervised Semantic Segmentation (WSSS) with image-level labels typically leverages Class Activation Maps (CAMs) to achieve pixel-level predictions. Recently, Contrastive Language-Image Pre-training (CLIP) has bee...
Weakly Supervised Semantic Segmentation (WSSS) with image-level labels typically leverages Class Activation Maps (CAMs) to achieve pixel-level predictions. Recently, Contrastive Language-Image Pre-training (CLIP) has been introduced to generate CAMs in WSSS. However, previous WSSS methods solely adopt CLIP's vision-language paired property for dense localization, neglecting its inherently limited dense knowledge across both visual and text modalities, which renders CAM generation suboptimal. In this work, we propose DiCLIP, a novel WSSS framework that leverages the generative diffusion model to enhance CLIP's dense knowledge across two modalities. Specifically, Visual Correlation Enhancement (VCE) and Text Semantic Augmentation (TSA) modules are proposed for dense prediction enhancement. To improve the spatial awareness of visual features, our VCE module utilizes diffusion's reliable spatial consistency to mitigate the over-smoothing issue in CLIP's attention. It designs the Attention Clustering Refinement (ACR) module to reliably extract diverse correlation maps from the diffusion model. The correlation maps act as a diversity bias for CLIP's self-attention, recursively pushing its visual features towards a more discriminative dense distribution. To augment the semantics of text embeddings, our TSA module argues that a single text modality is insufficient to encompass the variability of visual categories. Thus, we leverage diffusion's generative power to maintain a dynamic key-value cache model, shifting CAM generation from a patch-text matching mechanism to a novel visual knowledge retrieval paradigm. With these enhancements, DiCLIP not only outperforms state-of-the-art methods on PASCAL VOC and MS COCO but also significantly reduces training costs. Code is publicly available at https://github.com/zwyang6/DiCLIP.
About
- Frequency
- Sun
- Papers found
- 200
- RSS feed
- Subscribe