IEEE Transactions On Image Processing[JOURNAL]
Sun
200 papers
Toward Robust Alignment for Video Dehazing With Temporal Lookup Table.
Deng H, Li Z, Zhang F … +4 more , Xu B, Lu Q, Gao C, Sang N
IEEE Trans Image Process · 2026 · PMID 42102082 · Publisher ↗
Video dehazing aims to restore clean scenarios from a sequence of hazy frames, where frame alignment is a critical stage for leveraging temporal information. However, haze degrades contrast and obscures details, making a...
Video dehazing aims to restore clean scenarios from a sequence of hazy frames, where frame alignment is a critical stage for leveraging temporal information. However, haze degrades contrast and obscures details, making alignment challenging. Existing methods ignore the impairment of haze on alignment and thus struggle to align frames accurately. To address this challenge, we propose an alignment network with the temporal lookup table (temporal-LUT), which effectively enhances the haze-degraded frames and provides vivid cues for precise alignment. Specifically, to tackle the color degradation of haze, we employ a learnable lookup table (LUT) to enhance hazy color. The color mapping nature of LUT favorably preserves the naturalness of enhanced outcomes. Besides, we introduce a temporal weight prediction strategy to strengthen inter-frame interaction, which ensures temporal consistency across enhanced results and thereby benefits alignment. Extensive experimental results on two widely used benchmarks and real-world scenes demonstrate the superiority of our method.
Adaptive 3D Convolution for Remote Sensing Image Fusion.
Peng S, Zhu X, Deng SQ … +2 more , Deng LJ, Lei Z
IEEE Trans Image Process · 2026 · PMID 42102081 · Publisher ↗
Remote sensing image fusion aims to create a high-resolution multi/hyper-spectral image from a high-resolution image with limited spectral information and a low-resolution image with abundant spectral data. Recently, dee...
Remote sensing image fusion aims to create a high-resolution multi/hyper-spectral image from a high-resolution image with limited spectral information and a low-resolution image with abundant spectral data. Recently, deep learning (DL) techniques have shown significant effectiveness in this area. Most DL-based methods approach image fusion as a 2D problem by encoding spectral information into feature map channels. However, our research suggests that this strategy introduces notable spectral distortions. In contrast, some methods consider spectral data as an additional dimension, utilizing standard 3D convolutions to preserve spectral information. Nevertheless, in a standard 3D convolutional layer, the same set of kernels is applied across all input regions, which we have found to be sub-optimal for image fusion. Furthermore, standard 3D convolutions necessitate substantial computational resources. To address these challenges, we propose a novel convolutional paradigm called Adaptive 3D Convolution (Ada3D) for remote sensing image fusion. Ada3D applies a unique set of 3D kernels to each input voxel, enabling the capture of fine-grained details. These adaptive kernels are generated through a two-step process: 1) spatial and spectral kernels are derived from their respective image sources and 2) these two types of kernels are then combined to form content-aware 3D kernels that effectively integrate spatial and spectral information. Additionally, adaptive biases are introduced to enhance the convolutional outcome at the voxel level. Furthermore, we incorporate the group convolution technique to reduce computational complexity. As a result, Ada3D offers full adaptivity in an efficient manner. Evaluation results across five datasets demonstrate that our method achieves state-of-the-art (SOTA) performance, underscoring the superiority of Ada3D. The code is available at https://github.com/PSRben/Ada3D.
Prompting Rain Off: Evolving Compact Dual Prompts for Continual De-Raining.
Liu M, Yang W, Liu J
IEEE Trans Image Process · 2026 · PMID 42096397 · Publisher ↗
In recent years, there has been notable progress in single-image rain removal, particularly focusing on static data distributions in these approaches. When dealing with data that constantly changes, the challenge of cata...
In recent years, there has been notable progress in single-image rain removal, particularly focusing on static data distributions in these approaches. When dealing with data that constantly changes, the challenge of catastrophic forgetting arises, which is quite common and critical in real-world scenarios. To address this, we propose Evolving COmpact Dual Prompt Learning (EcoDPL), an efficient rehearsal-free continual learning deraining framework designed specifically for low-level vision tasks. Specifically, we design two prompt pools at both image and feature levels and insert these prompts into images and embedding tokens, for better knowledge transfer across tasks. Our adaptive weight generation module, P-Fuser, attaches an attention map to each prompt, to adaptively pay attention to different inputs, and get different weights to fuse prompts, making the inserted prompts more flexible with various inputs. Also, we introduce Grad-Tuner, a dictionary learning strategy, to compress knowledge into fewer prompts. This makes the knowledge more compact and provides more space for new prompts to learn new tasks. Our method stands out by leveraging small, learnable prompts for efficient knowledge retention across tasks, not increasing training time or parameters. Furthermore, we present an augmented method that upgrades the distance function $\gamma $ from simple cosine distance to a more advanced weight generation network. We also employ a fine-tuned dictionary learning technique, compressing knowledge into a more compact form, and enhancing the ability of prompts to learn new tasks. With our new designs, the model becomes more flexible with various inputs and it compresses knowledge into fewer prompts to free up spaces to learn new tasks. Through extensive experiments on various rain removal datasets, our EcoDPL method consistently outperforms previous continual learning techniques. Notably, although EcoDPL is designed for continual learning with changing data, it also performs well with stationary data, proving its robustness and versatility. Our website is available at: https://starymoon.github.io/Prompting-Rain-Off.
SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition.
Hu X, Wang Y, Fan L … +6 more , Luo C, Fan J, Lei Z, Li Q, Peng J, Zhang Z
IEEE Trans Image Process · 2026 · PMID 42096396 · Publisher ↗
3D Gaussian Splatting has emerged as an alternative 3D representation for novel view synthesis, benefiting from its high-quality rendering results and real-time rendering speed. However, the 3D Gaussians learned by 3D-GS...
3D Gaussian Splatting has emerged as an alternative 3D representation for novel view synthesis, benefiting from its high-quality rendering results and real-time rendering speed. However, the 3D Gaussians learned by 3D-GS have ambiguous structures without any geometry constraints. This inherent issue in 3D-GS leads to a rough boundary when segmenting individual objects. To remedy these problems, we propose SAGD, a conceptually simple yet effective boundary-enhanced segmentation pipeline for 3D-GS to improve segmentation accuracy while preserving segmentation speed. Specifically, we introduce a Gaussian Decomposition scheme, which ingeniously utilizes the special structure of 3D Gaussians, finds out, and then decomposes the boundary Gaussians. Moreover, to achieve fast interactive 3D segmentation, we introduce a novel training-free pipeline by lifting a 2D foundation model to 3D-GS. Extensive experiments demonstrate that our approach achieves high-quality 3D segmentation without rough boundary issues, which can be easily applied to other scene editing tasks. Our code is publicly available at https://github.com/XuHu0529/SAGS.
OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving.
Wang L, Zheng W, Ren Y … +4 more , Jiang H, Cui Z, Yu H, Lu J
IEEE Trans Image Process · 2026 · PMID 42096395 · Publisher ↗
Understanding the evolution of 3D scenes is crucial for autonomous driving. While conventional methods describe scene development through individual instance motions, world models provide a generative framework for model...
Understanding the evolution of 3D scenes is crucial for autonomous driving. While conventional methods describe scene development through individual instance motions, world models provide a generative framework for modeling overall scene dynamics. However, most existing approaches rely on autoregressive next-token prediction, which suffers from error accumulation and limited global spatiotemporal reasoning, leading to degraded long-term consistency. To address these issues, we propose a diffusion-based 4D occupancy generation model, OccSora, to simulate 3D world evolution for autonomous driving. A 4D scene tokenizer is introduced to obtain compact spatiotemporal representations and enable high-quality reconstruction of long occupancy sequences. We then train a diffusion transformer on these representations to generate 4D occupancy conditioned on trajectory prompts. Experiments on the nuScenes dataset with Occ3D annotations show that OccSora can generate 16s videos with authentic 3D layout and strong temporal consistency. With trajectory-aware 4D generation, OccSora has the potential to serve as a world simulator for autonomous driving decision-making. Project page: https://wzzheng.net/OccSora.
Hierarchical Causal Learning for Face Age Synthesis.
Wang Y, Sun P, Zhou X … +4 more , Shen L, Leng J, Wang G, Yu H
IEEE Trans Image Process · 2026 · PMID 42096394 · Publisher ↗
Face age synthesis (FAS) predicts a person's future or past facial appearance. In FAS, modifying one facial attribute usually affects the generation of other attributes during face image generation. Current models direct...
Face age synthesis (FAS) predicts a person's future or past facial appearance. In FAS, modifying one facial attribute usually affects the generation of other attributes during face image generation. Current models directly learn entangled representations of age-related features, resulting in insufficient feature disentanglement, which consequently impairs their causal reasoning capability for FAS tasks. To this end, we propose a hierarchical causal learning model for face age synthesis (HCFace), which integrates hierarchical structures and causal relationships into the facial generative model. Specifically, we propose to leverage hierarchical causal relationships to align with facial features for feature disentanglement. Furthermore, we design a novel nonlinear mapping function that captures the true patterns of facial attribute changes with age, enhancing the disentanglement of these attributes. We conduct extensive experiments to validate the superiority of our proposed model. Compared to other advanced baseline methods, HCFace improves overall accuracy by 2.47%, with improvements of 9.75% and 9.69% in certain age-related attributes, such as skin and hair. Our source code is available at https://github.com/SE-hash/HCFace.
SCASeg: Strip Cross-Attention for Efficient Semantic Segmentation.
Xu G, Chen J, Huang W … +3 more , Jia W, Gao G, Qi GJ
IEEE Trans Image Process · 2026 · PMID 42090527 · Publisher ↗
The Vision Transformer (ViT) has achieved notable success in computer vision, with its variants widely validated across various downstream tasks, including semantic segmentation. However, as general-purpose visual encode...
The Vision Transformer (ViT) has achieved notable success in computer vision, with its variants widely validated across various downstream tasks, including semantic segmentation. However, as general-purpose visual encoders, ViT backbones often do not fully address the specific requirements of task decoders, highlighting opportunities for designing decoders optimized for efficient semantic segmentation. This paper proposes Strip Cross-Attention (SCASeg), an innovative decoder head specifically designed for semantic segmentation. Instead of relying on the conventional skip connections, we utilize lateral connections between encoder and decoder stages, leveraging encoder features as Queries in cross-attention modules. Additionally, we introduce a Cross-Layer Block (CLB) that integrates hierarchical feature maps from various encoder and decoder stages to form a unified representation for Keys and Values. The CLB also incorporates the local perceptual strengths of convolution, enabling SCASeg to capture both global and local context dependencies across multiple layers, thus enhancing feature interaction at different scales and improving overall efficiency. To further optimize computational efficiency, SCASeg compresses the channels of queries and keys into one dimension, creating strip-like patterns that reduce memory usage and increase inference speed compared to traditional vanilla cross-attention. Experiments show that SCASeg's adaptable decoder delivers competitive performance across various setups, outperforming leading segmentation architectures on benchmark datasets, including ADE20K, Cityscapes, COCO-Stuff 164k, and Pascal VOC2012, even under diverse computational constraints.
ADAN: Adversarial Distribution Alignment Network for Multi-View Semi-Supervised Classification.
Huang S, Fu L, Chen Z … +3 more , Zhang T, Li X, Cui Z
IEEE Trans Image Process · 2026 · PMID 42090526 · Publisher ↗
Multi-view learning aims to integrate multi-source information for a comprehensive data representation, which has gained widespread attention in image processing. Each view contains view-specific noise and joint features...
Multi-view learning aims to integrate multi-source information for a comprehensive data representation, which has gained widespread attention in image processing. Each view contains view-specific noise and joint features associated with other views, and thus exploring the specificity and consistency among views is a typical solution to deal with multi-view data for learning discriminative representations. In this paper, we present a theory-induced model, termed Adversarial Distribution Alignment Network (ADAN), which learns view-invariant features and alleviate the negative impact of view-specific noise. We first demonstrate the necessity of suppressing view-specific noise and capturing view-invariant features inspired by the theory of view generalization, and then derive two collaborative modules: a feature disentangler and an adversarial alignment module. In detail, the feature disentanglement separates view-specific noise and view-invariant features by minimizing the mutual information between them. Following this, a negative entropy is proposed to suppress the negative impact of view-specific noise. Meanwhile, the adversarial module uses the adversarial technique that can fit more complex data conformed to different distributions to adaptively align cross-view features so that features encoded in different views converge. Substantial experiments are constructed on multi-view datasets, demonstrating that ADAN can achieve more promising performance compared to other superior methods. Code is available at https://github.com/huangsuj/ADANet.
HRMamba: A Hybrid Retinex and State-Space Model for Underwater Image Enhancement.
Fan Y, Gao L, Zhou F … +2 more , Li N, Huang Y
IEEE Trans Image Process · 2026 · PMID 42090525 · Publisher ↗
Underwater light absorption and scattering lead to severe color distortion, reduced visibility, contrast loss, and a significant degradation in image quality, thereby impeding both human visual analysis and machine visio...
Underwater light absorption and scattering lead to severe color distortion, reduced visibility, contrast loss, and a significant degradation in image quality, thereby impeding both human visual analysis and machine vision tasks. Although considerable progress has been achieved in improving image quality, existing deep learning-based methods for underwater image enhancement (UIE) remain constrained by high computational complexity and insufficient modeling of global dependencies, which restricts their practical deployment in resource-limited underwater environments. To tackle these issues, we propose a novel hybrid framework integrating Retinex theory and state-space models (SSMs) for underwater image enhancement, named HRMamba. Different from existing Transformer-based approaches constrained by quadratic complexity, HRMamba attains computational efficiency through linear-complexity state-space operations while maintaining global dependency modeling capabilities. Moreover, to achieve comprehensive feature fusion, an Illumination Feature Fusion Module (IFFM) is proposed, which synergizes the global dependency modeling of SSMs with the local adaption capability of convolutional neural networks (CNNs). For context-sensitive noise suppression with illumination awareness, we propose an Illumination-Guided Denoising Module (IGDM) that employs directional-scanning Vision State Space Module (VSSM) blocks. Experiments demonstrate that HRMamba achieves state-of-the-art enhancement quality via an efficient architecture, significantly improving color fidelity and visibility restoration while substantially reducing computational demands. The code is available at https://github.com/YeFan-web/HRMamba/.
Open-Set Domain Adaptation via Target-Relaxed Optimal Transport.
Ren CX, Huang ZX, Yan H
IEEE Trans Image Process · 2026 · PMID 42090524 · Publisher ↗
Open set domain adaptation (OSDA) aims to transfer classification-oriented knowledge from a labeled source domain to an unlabeled target domain, which faces the challenges from unseen knowledge in open-set scenarios, i.e...
Open set domain adaptation (OSDA) aims to transfer classification-oriented knowledge from a labeled source domain to an unlabeled target domain, which faces the challenges from unseen knowledge in open-set scenarios, i.e., unknown classes privileged to the target domain. Existing methods usually identify unknown classes from classifier prediction directly, which are sensitive to the intrinsic clustering structure and cluster numbers of the unknown class data. In this paper, inspired by the sample relation characterization ability of Optimal Transport (OT), we propose a new type of OT method for OSDA, namely, Target-relaxed Optimal Transport (TROT). Compared with existing OT with strict marginal constraints, TROT imposes a single-side relaxation to the mass requirement on the open-set target domain. Theoretically, we prove that such a relaxation can reduce mis-matches between known and unknown classes, which indicates the transport plan of TROT is promising to identify unknown classes. Methodologically, TROT can identify unknown classes adaptively and map the cross-domain shared data with a sparse plan assignment, which improves both the effectiveness and robustness of known class alignment; besides, a graph embedding with multi-cluster structure of unknown classes is designed to learn a discriminative metric space for open-set classification. Empirically, extensive evaluations are conducted on several image datasets, where TROT achieves significant performance improvements compared with existing techniques for visual recognition in open-set scenarios.
MIST: A Benchmark and Baseline for Multi-Frame Infrared Small Target Detection in Complex Motion.
Gao R, Zhang M, Li G … +4 more , Li G, Zhao K, Zhang X, Zeng D
IEEE Trans Image Process · 2026 · PMID 42090523 · Publisher ↗
Motion cues play a vital role in multi-frame infrared small target detection (MISTD). However, most targets in existing datasets exhibit regular and slow motion, which cannot reflect the complex and diverse motion patter...
Motion cues play a vital role in multi-frame infrared small target detection (MISTD). However, most targets in existing datasets exhibit regular and slow motion, which cannot reflect the complex and diverse motion patterns in real-world scenarios. This biased data distribution makes recent data-driven methods highly rely on simplified motion assumptions that tend to fail in irregular or fast motion, resulting in noisy feature representations cluttered with target-irrelevant factors. Hence, we stress that methods for MISTD should also work when targets are in complex motion. To enable this research, we propose a large-scale dataset called MIST for airborne infrared detection scenarios. The dataset is built on a synthetic data engine that models variations in pose, size, and intensity of moving targets while seamlessly blending them into real backgrounds for physical, geometric, and visual realism. Targets in MIST exhibit low signal-to-clutter ratios and complex motion, making it a promising yet challenging benchmark for developing algorithms focused on motion analysis. To tackle the challenges of MIST, we develop MISTNet, a robust baseline based on the Information Bottleneck theory. To handle irregular and fast motion, we propose a shifted neighborhood compensation block to efficiently model multi-scale correspondences for implicit motion compensation. To distill compact representations free from irrelevant cues, we design a progressive distillation decoder to hierarchically filter out redundancy while preserving target-relevant information. We benchmark 31 state-of-the-art methods and find that their performance on MIST drops significantly compared with that on the widely used NUDT-MIRSDT dataset. Our MISTNet outperforms all other methods by a large margin, with an over 6% gain in the IoU metric, demonstrating its superiority. The dataset, code, and model weights are available at https://github.com/GR-ray/MIST.
Context-Infused Trajectories: Enhancing Context and Frame Consistency in Reasoning Video Object Segmentation.
Zhuge Y, Gong S, Zhang L … +4 more , Xu Q, Zhao W, Zhan J, Lu H
IEEE Trans Image Process · 2026 · PMID 42090522 · Publisher ↗
Reasoning video object segmentation (ReaVOS) aims to segment referred objects in video sequences based on implicit and complex linguistic queries. Existing methods typically compress limited video frames into pooled repr...
Reasoning video object segmentation (ReaVOS) aims to segment referred objects in video sequences based on implicit and complex linguistic queries. Existing methods typically compress limited video frames into pooled representations and prompt multimodal large language models (MLLMs) to generate a single global segmentation token. However, this strategy lacks explicit contextual guidance and causes substantial loss of spatial details, limiting capability and segmentation consistency. To overcome these limitations, we introduce Context-infused Consistent Video Segmentor (CiCVS), a novel framework leveraging contextual information to guide generation of temporally coherent and accurate mask trajectories. CiCVS incorporates a Hierarchical Frame Sampling (HFS) module, which globally samples support frames across the entire video to ensure broad temporal coverage, and then uniformly selects target frames within the support set. It also employs a Contextual Token Prompting (CTP) module, which utilizes contextual cues from support frames to guide the MLLM in generating specialized tokens for various target frames, enabling the model to capture intricate temporal patterns and ensure consistency across long-range sequences. At the core of CTP is the Multimodal Injection Compressor (MIC) block, which efficiently integrates support frame features and textual semantic information into a compact set of latent queries, enhancing temporal-level object perception. To further advance the ReaVOS field, we introduce the CoCoRVOS benchmark, which features more temporally intricate reasoning instructions and a diverse set of video scenarios. Extensive experiments demonstrate that CiCVS establishes a new state-of-the-art on multiple benchmarks, achieving significant improvements in $\mathcal {J}\& \mathcal {F}$ scores, including +2.7 on CoCoRVOS, +1.4 on ReVOS, and +7.0 on ReasonVOS, underscoring its superior contextual reasoning and segmentation capabilities.
Infrared and Visible Image Fusion Using Bimodal Neuron and Dynamic Receptive Field Mechanisms.
Gao S, Tan M, Lv S … +2 more , Liu Y, Li Y
IEEE Trans Image Process · 2026 · PMID 42090521 · Publisher ↗
Infrared and visible image fusion (IVIF) significantly enhances scene interpretation by integrating broad-spectrum information. Drawing inspiration from specific snakes that possess an evolutionarily optimized bimodal se...
Infrared and visible image fusion (IVIF) significantly enhances scene interpretation by integrating broad-spectrum information. Drawing inspiration from specific snakes that possess an evolutionarily optimized bimodal sensory system capable of parallel processing infrared and visible radiation, we propose a novel IVIF framework incorporating two key elements: nonlinear cross-modal interactions across six distinct classes of snake bimodal neurons and dynamic center-surround receptive field organization. These biological principles are mathematically formalized and integrated within a deep neural network (DNN), optimized through an object detection region-guided loss and a frequency-dependent fusion loss that enable data-driven fusion strategy learning. Experimental results demonstrate that the optimized model effectively emulates the infrared-visible information integration observed in snake bimodal neurons. Critically, the nonlinear bimodal neurons capture a significantly greater amount of edge information and finer mid-to-high-frequency details, which are essential for the subsequent reconstruction of the fused image. Furthermore, a comprehensive evaluation of visual quality, encompassing both qualitative and quantitative assessments on six datasets, along with extensive object detection and semantic segmentation experiments using the fused images in both daytime and nighttime scenarios, demonstrates that our model outperforms traditional biologically-inspired IVIF algorithms, achieving performance comparable to SOTA DNN-based methods. The code and weights are available at https://github.com/rwerwer2024/SBNF.
FinePruner: Unbiased Attention-Head-Level Fine-Grained Token Reduction for Efficient Inference of Large Vision-Language Models.
Wang Z, Zheng X, Peng Y
IEEE Trans Image Process · 2026 · PMID 42085416 · Publisher ↗
Large Vision-Language Models (LVLMs) suffer from the high computational cost of the attention mechanism caused by the large number of visual tokens. Token reduction has emerged as a promising approach to reduce the compl...
Large Vision-Language Models (LVLMs) suffer from the high computational cost of the attention mechanism caused by the large number of visual tokens. Token reduction has emerged as a promising approach to reduce the complexity by eliminating redundant visual tokens. However, existing token reduction methods struggle to preserve task-relevant tokens and eliminate irrelevant ones. This is due to the attention biases of LVLMs, where tokens with high attention scores are not always the critical ones. Such biases force existing methods into a dilemma: they face either high performance degradation or limited inference acceleration. This issue becomes more severe in fine-grained perception tasks, which rely heavily on the fine-grained information stored in specific visual tokens. To address the above issue, we propose an unbiased fine-grained token reduction method named FinePruner, which explores the attention patterns of LVLMs at the attention-head-level to mitigate the interference of attention biases. Concretely, we first conducted comparative studies to validate the impact of tokens corresponding to visual objects on final task performance, which established the conclusion that these tokens should be preserved while others can be pruned. Also, a series of visualizations unveils the changing patterns of LVLMs' attention biases across layers and attention heads. Based on the patterns of attention biases, the pipeline of FinePruner is divided into two stages. The first stage, named Instruction-Agnostic Clustering, clusters visual tokens into groups according to their embeddings to exclude the attention biases. The second stage, named Attention-Refined Pruning, selects attention heads with less bias by the divergence, which are used to identify the preserved tokens. Experiments on VQA benchmarks and fine-grained benchmarks demonstrate that our FinePruner achieves better accuracy-efficiency tradeoffs than state-of-the-art methods. The code is available at https://github.com/PKU-ICST-MIPL/FinePruner_TIP2026.
Scale-Aware Prompting With Optimal Transport for Remote Sensing Image Captioning.
Zhang C, Ren Z, Hou B … +4 more , Ning J, Wang K, Li W, Jiao L
IEEE Trans Image Process · 2026 · PMID 42081398 · Publisher ↗
Remote sensing image captioning is a multimodal foundation task for fine-grained understanding of remote sensing images. However, remote sensing images contain complex scenes and rich objects, it is very challenging to a...
Remote sensing image captioning is a multimodal foundation task for fine-grained understanding of remote sensing images. However, remote sensing images contain complex scenes and rich objects, it is very challenging to accurately describe the objects in the scene with their attributes and dependencies. To address these issues, the article proposes a novel scale-aware prompting with optimal transport (SPOT) to learn effective multiscale features under diverse scenes, and to build fine-grained cross-modal alignment between semantic features and linguistic words during caption generation. Specifically, a scale-aware prompt extractor is constructed to explore feature integrations in complex scenes through learning prompts that query multi-scale features, and to enhance the representation of attributes and dependencies for objects by embedding positional relations. Besides, a fine-grained cross-modal alignment is designed to dynamically match image feature representations and textual semantics through optimal transport. Through the above manner, the model can learn effective language-aligned feature representations for caption generation. Finally, a caption Transformer with causal self-attention is introduced to generate accurate captions for remote sensing scenes. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance on three public datasets, with the superiority of the proposed method further demonstrated by ablating the role of each component.
ConsDreamer: Advancing Multi-View Consistency for Zero-Shot Text-to-3D Generation.
Zhou Y, Jin S, Hua L … +3 more , Lv W, Duan H, Han J
IEEE Trans Image Process · 2026 · PMID 42081397 · Publisher ↗
Recent advances in zero-shot text-to-3D generation have revolutionised 3D content creation by enabling direct synthesis from textual descriptions. While state-of-the-art methods leverage 3D Gaussian Splatting with score...
Recent advances in zero-shot text-to-3D generation have revolutionised 3D content creation by enabling direct synthesis from textual descriptions. While state-of-the-art methods leverage 3D Gaussian Splatting with score distillation to enhance multi-view rendering through pre-trained text-to-image (T2I) models, they suffer from inherent prior view biases in T2I Models. These biases lead to inconsistent 3D generation, particularly manifesting as the multi-face Janus problem, where objects exhibit conflicting features across views. To address this fundamental challenge, we propose ConsDreamer, a novel method that mitigates view bias by refining both the conditional and unconditional terms in the score distillation process: (1) a View Disentanglement Module (VDM) that eliminates viewpoint biases in conditional prompts by decoupling irrelevant view components and injecting precise view control; and (2) a similarity-based partial order loss that enforces geometric consistency in the unconditional term by aligning cosine similarities with azimuth relationships. Extensive experiments demonstrate that ConsDreamer can be seamlessly integrated into various 3D representations and score distillation paradigms, effectively mitigating the multi-face Janus problem.
EyeKey: Self-Supervised Keypoint Detection and Description Network Based on Local Feature Saliency for Retinal Image Global Registration.
Liang Y, Ma D, Wu X
IEEE Trans Image Process · 2026 · PMID 42081396 · Publisher ↗
Retinal image registration (RIR) plays an important role in the diagnosis and long-term monitoring of retinal diseases. Retinal image global registration (RIGR) is usually the first step of RIR. Traditional methods often...
Retinal image registration (RIR) plays an important role in the diagnosis and long-term monitoring of retinal diseases. Retinal image global registration (RIGR) is usually the first step of RIR. Traditional methods often struggle to achieve robust keypoint detection and description when faced with high-resolution, fine-textured retinal images. Deep learning-based methods for this task have not been widely developed. Therefore, we propose a keypoint detection and description network based on local feature saliency, EyeKey, for RIGR. EyeKey uses the "Detect While Describing (DWD)" design. Specifically, two proposed UDPAM++ modules are embedded into the feature description network to enhance its feature description capability. Concurrently, these modules detect distinctive keypoints based on local feature saliency, combined with a Mapping Module featuring only three learnable parameters. Moreover, we achieve self-supervised feature description network training on high-resolution, fine-textured retinal images through the Random Local Hardest Example Mining strategy. Additionally, we realize robust unsupervised keypoint detection network training based on the High Matching Probability Defines Keypoints strategy and the proposed Cumulative Salient Keypoint Expansion, which, together with the DWD design, mutually reinforce the training of the keypoint detection and description network. Finally, combined with the feature-based RIGR pipeline, our method achieves outstanding performance while maintaining excellent inference speed on monomodal and multimodal RIGR evaluation datasets.
A Cross-Modal Network for Facial Expression Recognition.
Tian C, Xie J, Zhang Q … +3 more , Li C, Zuo W, Zhang S
IEEE Trans Image Process · 2026 · PMID 42081395 · Publisher ↗
Deep neural networks enriched with structural information have been widely employed for facial expression recognition tasks. However, these methods often depend on hierarchical information rather than face property to fi...
Deep neural networks enriched with structural information have been widely employed for facial expression recognition tasks. However, these methods often depend on hierarchical information rather than face property to finish expression recognition. In this paper, we propose a cross-modal network with strong biological and structural information for facial expression recognition (CMNet). CMNet can respectively learn expression information via face symmetry on a whole face, left and right half faces to extract complementary facial features. To prevent negative effect of biological and structural information fusion, a salient facial information refinement module can obtain salient facial expression information to improve stability of an obtained facial expression classifier. To reduce reliance on unilateral facial features, a half-face alignment optimization mechanism is designed to align obtained expression information of learned left and right half faces. Our experimental results demonstrate that CMNet outperforms several novel methods, i.e., SCN and LAENet-SA for facial expression recognition. Codes can be obtained at https://github.com/hellloxiaotian/CMNet.
Toward Comprehensive Information-Theoretic Multi-View Learning.
Shi L, Ye Y, Wang W … +4 more , Lei T, Zhao Y, Kou G, Chen B
IEEE Trans Image Process · 2026 · PMID 42081394 · Publisher ↗
Information theory has inspired numerous advancements in multi-view learning. Most multi-view methods incorporating information-theoretic principles rely an assumption called multi-view redundancy which states that commo...
Information theory has inspired numerous advancements in multi-view learning. Most multi-view methods incorporating information-theoretic principles rely an assumption called multi-view redundancy which states that common information between views is necessary and sufficient for down-stream tasks. This assumption emphasizes the importance of common information for prediction, but inherently ignores the potential of unique information in each view that could be predictive to the task. In this paper, we propose a comprehensive information-theoretic multi-view learning framework named CIML, which discards the assumption of multi-view redundancy. Specifically, CIML considers the potential predictive capabilities of both common and unique information based on information theory. First, the common representation learning maximizes Gács-Körner common information to extract shared features and then compresses this information to learn task-relevant representations based on the Information Bottleneck (IB). For unique representation learning, IB is employed to achieve the most compressed unique representation for each view while simultaneously minimizing the mutual information between unique and common representations, as well as among different unique representations. Importantly, we theoretically prove that the learned joint representation is predictively sufficient for the downstream task. Extensive experimental results have demonstrated the superiority of our model over several state-of-art methods. The code is released on CIML.
Dual-Masked and Discriminative Reconstruction for Unified Vision Anomaly Detection.
Gao BB
IEEE Trans Image Process · 2026 · PMID 42065980 · Publisher ↗
Unsupervised reconstruction networks have shown promise for unified vision anomaly detection, i.e., image-level anomaly classification and pixel-level anomaly segmentation, where a single model trained on multi-class nor...
Unsupervised reconstruction networks have shown promise for unified vision anomaly detection, i.e., image-level anomaly classification and pixel-level anomaly segmentation, where a single model trained on multi-class normal images can detect various anomalies. This is more challenging than most existing separate methods, i.e., one model for one class, as it requires handling a more complex data distribution. Notably, pure reconstruction networks often suffer from overfitting due to "identity shortcut", where both normal and anomaly images may be well recovered and thus fail in detecting anomalies. Recent efforts have focused on developing specific modules for different network architectures, e.g., Convolutions and Transformers. However, it is still unclear how to essentially and effectively prevent learning from this shortcut in a simpler and more general manner. Furthermore, most existing methods consider anomaly detection solely as unsupervised classification, resulting in inaccurate anomaly segmentation due to "weak discrimination", where normal and anomaly features may be entangled. To address these challenges, we propose a simple yet general Dual-masked and Discriminative Reconstruction (D2Rec) for unified vision anomaly detection. First, we propose a general dual-masked reconstruction, i.e., using a pair of complementary masks, resolving the "identity shortcut" so that all masked positions are reconstructed by unmasked original features. Second, we propose a self-supervised discriminator, which refines reconstruction errors with synthesized anomaly images to enhance the discrimination ability between normal and abnormal features. The dual-masked reconstruction and self-supervised discriminator can serve as universal plugins, easily integrated into reconstruction-based anomaly detection methods of any architecture. Despite its simplicity, D2Rec outperforms previous methods on three industrial benchmarks (MVTec, BTAD, and VisA), and three medical datasets (Brain MRI, Liver CT and Retinal OCT). The code for D2Rec is available at https://github.com/gaobb/D2Rec.
About
- Frequency
- Sun
- Papers found
- 200
- RSS feed
- Subscribe