IEEE Transactions On Image Processing[JOURNAL]
Sun
200 papers
Expert-Guided Cross-View Fusion With Self-Derived Lesion Proposals for Multi-View Diabetic Retinopathy Grading.
Wong WK, Liu W, Zhou X … +3 more , Hou J, Lin Y, Wen J
IEEE Trans Image Process · 2026 · PMID 42258682 · Publisher ↗
Recent advances in multi-view fundus imaging show great promise for automated diabetic retinopathy (DR) grading. However, mainstream end-to-end CNN/Transformer pipelines rely on striding or tokenization that compresses s...
Recent advances in multi-view fundus imaging show great promise for automated diabetic retinopathy (DR) grading. However, mainstream end-to-end CNN/Transformer pipelines rely on striding or tokenization that compresses spatial detail, causing small, low-contrast lesions (e.g., microaneurysms) to be under-represented and creating performance ceilings. Prior efforts have mitigated this by incorporating external lesion- or vessel-level annotations into models. However, such labels are costly to acquire, break the end-to-end training, and make performance over-reliant on the annotation quality. To reduce dependence on expensive annotations, we propose an end-to-end framework that generates lesion proposals on the fly during training and inference, providing self-derived cues for grading. First, we introduce a Grade-Activated Lesion Proposal (GALP) module that derives grade-conditioned evidence maps (GEMs) from stage-wise auxiliary classifiers and selects the top-K high-evidence regions per view as lesion proposals. Second, we propose a Cross-View Lesion Expert Guided Regional Fusion (LGRF) module, which selectively activates experts for a view's lesion proposals based on contextual guidance from other views, ensuring that only the most relevant feature extractors contribute to fusion. Experimental results on two multi-view DR datasets show that our method matches or surpasses strong baselines without external annotations, confirming that self-generated proposals can substantially reduce annotation needs.
EDA-DM: Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models.
Liu X, Li Z, Xiao J … +3 more , Chen M, Li J, Gu Q
IEEE Trans Image Process · 2026 · PMID 42258681 · Publisher ↗
Diffusion models have achieved great success in image generation tasks. However, the lengthy denoising process and complex neural networks hinder their low-latency applications in real-world scenarios. Quantization can e...
Diffusion models have achieved great success in image generation tasks. However, the lengthy denoising process and complex neural networks hinder their low-latency applications in real-world scenarios. Quantization can effectively reduce model complexity, and post-training quantization (PTQ), which does not require fine-tuning, is highly promising for compressing and accelerating diffusion models. Unfortunately, we find that due to the highly dynamic activations, existing PTQ methods suffer from distribution mismatch issues at both calibration sample level and reconstruction output level, which makes the performance far from satisfactory. In this paper, we propose EDA-DM, a standardized PTQ method that efficiently addresses the above issues. Specifically, at the calibration sample level, we extract information from the density and diversity of latent space feature maps, which guides the selection of calibration samples to align with the overall sample distribution; and at the reconstruction output level, we theoretically analyze the reasons for previous reconstruction failures and, based on this insight, optimize block reconstruction using the Hessian loss of layers, aligning the outputs of quantized model and full-precision model at different network granularity. Extensive experiments demonstrate that EDA-DM significantly outperforms the existing PTQ methods across various models and datasets. Our method achieves a $1.83\times $ speedup and $4\times $ compression for the popular Stable-Diffusion on MS-COCO, with only a 0.05 loss in CLIP score. Code is available at http://github.com/BienLuky/EDA-DM.
SpecEStop: Self-Supervised Hyperspectral Mixed Noise Removal via Deep Spectral Prior.
Zhang R, Ng MK, Gao L … +2 more , Ljubenovic M, Zhuang L
IEEE Trans Image Process · 2026 · PMID 42258680 · Publisher ↗
Hyperspectral remote sensing images often suffer from mixed noise-Gaussian, stripe, and impulse-due to atmospheric interference, solar variability, and sensor imperfections. These noises are typically band-dependent and...
Hyperspectral remote sensing images often suffer from mixed noise-Gaussian, stripe, and impulse-due to atmospheric interference, solar variability, and sensor imperfections. These noises are typically band-dependent and diverse in distribution, making unified denoising particularly challenging. Existing deep denoising methods rely on clean/noisy pairs, which are unavailable in real-world remote sensing, while traditional approaches require manual tuning and lack adaptability. We propose SpecEStop, a fully self-supervised spectral vector denoising framework that requires only a single noisy HSI for training. Leveraging a novel deep spectral prior, SpecEStop exploits the spectral bias of neural networks, which tend to learn low-frequency (clean) signal components with Gaussian noise before overfitting to non-Gaussian noise ones. An adaptive early stopping strategy halts training before non-Gaussian noise is learned, enabling effective suppression of complex noise patterns. To address remaining Gaussian noise, we purposely design the network architecture to preserve its statistical properties in the latent space, allowing the use of off-the-shelf Gaussian denoisers during inference. Without any clean supervision, SpecEStop achieves effective, stage-wise removal of mixed noise, as validated across diverse real-world scenarios. Code will be released at https://github.com/ruobing-Zhang and the permanent code repository maintained by the corresponding author at http://github.com/LinaZhuang.
Flow-Matching Posterior Sampling: A Training-Free Conditional Generation for Flow Matching.
Song K, Lai H, Pan Y … +2 more , Yue K, Yin J
IEEE Trans Image Process · 2026 · PMID 42241270 · Publisher ↗
Training-free conditional generation based on flow matching aims to leverage pre-trained unconditional flow matching models to perform conditional generation without retraining. Recently, a successful training-free condi...
Training-free conditional generation based on flow matching aims to leverage pre-trained unconditional flow matching models to perform conditional generation without retraining. Recently, a successful training-free conditional generation approach incorporates conditions via posterior sampling, which relies on the availability of a score function in the unconditional diffusion model. However, flow matching models lack an explicit score function, rendering this strategy inapplicable. Approximate posterior sampling for flow matching has been explored, but it is limited to linear inverse problems. In this paper, we propose Flow Matching-based Posterior Sampling (FMPS) to broaden its scope of application. We introduce a correction term by steering the velocity field. This correction term can be reformulated to incorporate a surrogate score function, thereby bridging the gap between flow matching models and score-based posterior sampling. Hence, FMPS enables posterior sampling to be adjusted within the flow-matching framework. Furthermore, we propose two practical implementations of the correction mechanism: one to improve generation quality and the other to enhance computational efficiency. Experimental results on diverse conditional generation tasks demonstrate that our method achieves superior generation quality compared to existing state-of-the-art approaches, validating the effectiveness and generality of FMPS.
Unveiling the Power of Multi-Modal Template Update in RGBT Tracking.
Liu L, Li C, Lu A … +4 more , Zhu Y, Han S, Cai X, Li C
IEEE Trans Image Process · 2026 · PMID 42241269 · Publisher ↗
Template update is essential for improving the adaptability of tracking algorithms to target appearance variations. While previous methods have leveraged the spatio-temporal complementarity of multi-modal templates for R...
Template update is essential for improving the adaptability of tracking algorithms to target appearance variations. While previous methods have leveraged the spatio-temporal complementarity of multi-modal templates for RGBT tracking, a comprehensive analysis of the template update mechanism remains underexplored. In this work, we propose a novel prototype-based framework that decomposes the multi-modal template update process from the perspective of prototype learning into four key components: multi-modal prototype, prototype integration, prototype evaluation, and prototype update algorithm. Our findings highlight that the multi-modal prototype is the most critical factor in enhancing tracking adaptability to appearance variations, leading to more robust target representations. While prototype integration is less crucial when the target representation is already robust, it still contributes to learning a more discriminative representation. Additionally, the accuracy of template updates is strongly influenced by prototype evaluation, which controls the accuracy of the update process. Finally, the prototype update algorithm, which determines when and how template updates occur, is key to maintaining tracking robustness. Building on these insights, we introduce the Multi-modal Prototype RGBT Tracker (MPTrack), which adapts dynamically to appearance variations through prototype learning. MPTrack combines a fixed template from the first frame with both modality-shared and modality-specific templates, forming a robust multi-modal prototype representation. It incorporates a prototype evaluation module that guides updates based on template reliability, and an adaptive update algorithm to manage templates effectively. Additionally, a prototype-guided cross-modal integration module enhances the discriminative power of multi-modal relation modeling. Experimental results on five challenging RGBT tracking benchmarks demonstrate that MPTrack consistently outperforms state-of-the-art methods, setting new performance records. The experimental data and source code will be made publicly available at: https://github.com/mmic-lcl/Datasets-and-benchmark-code.
Marine Saliency Segmenter: Object-Focused Conditional Diffusion With Region-Level Semantic Knowledge Distillation.
Chang L, Wang Y, Huang J … +3 more , Deng L, Du B, Xu C
IEEE Trans Image Process · 2026 · PMID 42241268 · Publisher ↗
Marine Saliency Segmentation (MSS) plays a pivotal role in a wide range of vision-based marine exploration tasks. However, existing techniques often face the dilemma of imprecise boundaries due to the interference-rich n...
Marine Saliency Segmentation (MSS) plays a pivotal role in a wide range of vision-based marine exploration tasks. However, existing techniques often face the dilemma of imprecise boundaries due to the interference-rich nature of underwater environments, where suspended particles, low contrast, and color distortion hinder accurate segmentation. Although diffusion models have shown impressive performance in visual tasks, their potential to incorporate contextual semantics for enhancing feature learning of region-level salient objects remains underexplored, thereby hindering segmentation outcomes. Building on this insight, we propose DiffMSS, a novel marine saliency segmenter based on the diffusion model, which utilizes semantic knowledge distillation to guide the detection of marine salient objects. Specifically, we design the Word-level Semantic Saliency Extraction module that identifies salient terms at the word level from the captions by computing region-word similarity. These high-level semantic features are distilled into the Conditional Feature Learning Network to generate accurate and semantically informed diffusion conditions. The Object-Focused Conditional Diffusion module then leverages these conditions to iteratively generate fine-grained segmentation masks of marine instances, while a Consensus Deterministic Sampling scheme is further employed to suppress overconfident mis-segmentations and enhance structural fidelity. Extensive experiments demonstrate the superior performance of DiffMSS over state-of-the-art methods in both quantitative and qualitative evaluations. Our code and pre-trained models will be released on GitHub.
Embodied Spatial Affordance: Spatial-Aware Affordance Learning for Embodied Navigation and Manipulation.
Hao X, Tang Y, Zhang L … +5 more , Chen L, Zhou W, Han J, Ding W, Zhang XP
IEEE Trans Image Process · 2026 · PMID 42241267 · Publisher ↗
Embodied navigation and manipulation are fundamental capabilities for embodied agents operating in physical environments. A key challenge in this process is understanding the spatial context and the affordances of the en...
Embodied navigation and manipulation are fundamental capabilities for embodied agents operating in physical environments. A key challenge in this process is understanding the spatial context and the affordances of the environment, which involves recognizing how objects can be interacted with (object affordance) and identifying suitable locations for movement and object placement (free space affordance). While Vision-Language Models (VLMs) have shown promise in high-level task planning, their ability to translate reasoning into precise executable actions remains limited, particularly in image-based spatial understanding and precise affordance localization-a critical gap in image processing for robotics. To bridge this gap, we propose EspA, a novel image-to-keypoint model that leverages spatial-aware affordance learning to predict actionable affordances directly from 2D image inputs. Built on a hierarchical vision-language architecture, EspA jointly reasons about object affordances and free space affordances, enabling pixel-level localization of both types of interactions. Crucially, EspA translates language instructions into precise 2D affordance keypoints from observed images, which are then projected into 3D actionable coordinates using depth information. To support this unified affordance reasoning, we introduce the Embodied Spatial Affordance (ESA) dataset, which captures both object-centric interactions and free space contexts. By jointly modeling these affordances in a shared representation space, EspA overcomes the limitations of prior works that treat them independently. The dataset's fine-grained annotations enable our model to learn the intricate relationship between object functionality and spatial feasibility, significantly enhancing the spatial understanding in embodied tasks. Extensive experimental results demonstrate that EspA outperforms existing state-of-the-art Vision-Language Models (VLMs), both open-source and closed-source, in object and free space affordance prediction. Furthermore, it exhibits superior performance in real-world embodied navigation and manipulation experiments. Our work advances the field of image-based spatial reasoning by providing a scalable solution for translating high-level instructions into low-level actionable affordances. We believe this work paves the way for more robust and versatile embodied agents capable of effectively interacting with complex environments. The dataset, benchmark, and evaluation code will be publicly available to facilitate future research. Project website: https://embodied-spatial-affordance.github.io/.
3D Segment Anything Model With Visual Mamba for Diagnosing Placenta Accreta Spectrum.
Zhang Y, He F, Peng L … +5 more , Yan T, Zhang P, Song T, Du L, Chen D
IEEE Trans Image Process · 2026 · PMID 42241266 · Publisher ↗
Placenta Accreta Spectrum (PAS) is a rare but highly dangerous obstetric disease. Early and accurate PAS diagnosis is critical for maternal health. Traditional PAS diagnosis relies on experienced doctors by analyzing the...
Placenta Accreta Spectrum (PAS) is a rare but highly dangerous obstetric disease. Early and accurate PAS diagnosis is critical for maternal health. Traditional PAS diagnosis relies on experienced doctors by analyzing the cesarean history and Magnetic Resonance Imaging (MRI) data. However, district-level hospitals often lack the expertise and resources for accurate PAS diagnosis. To address these challenges, we establish the first MRI-based PAS dataset, which includes both fine-grained segmentation and classification annotations. Meanwhile, diagnosing PAS can be significantly enhanced by segmenting lesion areas from MRI images of the uterus. To achieve automatic PAS diagnosis, we propose 3DSAMba, a novel feature learning framework for effective lesion segmentation. More specifically, we first design a 3D Segment Anything Model (SAM) and incorporate medical domain information into the model through an efficient adapter mechanism. In addition, we introduce a Multi-Level Aggregation Mamba (MLAM) to aggregate feature maps across different levels and a Fusion State Space Model (FSSM) to fuse multi-scale features from both the encoder and decoder. Finally, we apply segmentation masks to the original MRI images through element-wise multiplication, effectively isolating lesion areas for more accurate PAS diagnosis. Extensive experiments validate that our framework significantly improves the PAS diagnostic performance. To facilitate further research in PAS diagnosis, we have released the dataset and source code at https://github.com/Drchip61/PASD.
Detail++: Training-Free Detail Enhancer for T2I Diffusion Models.
Chen L, Wang J, Pan Z … +3 more , Zhu B, Yang X, Zhang C
IEEE Trans Image Process · 2026 · PMID 42241265 · Publisher ↗
Recent advances in text-to-image (T2I) generation have led to impressive visual results. However, these models still face significant challenges when handling complex prompts-particularly those involving multiple subject...
Recent advances in text-to-image (T2I) generation have led to impressive visual results. However, these models still face significant challenges when handling complex prompts-particularly those involving multiple subjects with distinct attributes. Inspired by the human drawing process, which first outlines the composition and then incrementally adds details, we propose Detail++, a training-free framework that introduces a novel Progressive Detail Injection (PDI) strategy to address this limitation. Specifically, we decompose a complex prompt into a sequence of simplified sub-prompts, guiding the generation process in stages. This staged generation leverages the inherent layout-controlling capacity of self-attention to first ensure global composition, followed by precise refinement. To achieve accurate binding between attributes and corresponding subjects, we exploit cross-attention mechanisms and further introduce a Centroid Alignment Loss at test time to reduce binding noise and enhance attribute consistency. Extensive experiments on T2I-CompBench and a newly constructed style composition benchmark demonstrate that Detail++ significantly outperforms existing methods, particularly in scenarios involving multiple objects and complex stylistic conditions.
Visual Camera Localization by Globally Consistent Descriptor Learning and Combined Bundle Adjustment.
Wang J, Wang Y, Wang C … +1 more , Qi Y
IEEE Trans Image Process · 2026 · PMID 42241264 · Publisher ↗
In this paper, we introduce a brand-new localization pipeline designed to comprehensively leverage both hand-crafted and learned features, operating at two distinct levels (point-level and object-level) while simultaneou...
In this paper, we introduce a brand-new localization pipeline designed to comprehensively leverage both hand-crafted and learned features, operating at two distinct levels (point-level and object-level) while simultaneously recovering scene scale from the RGB input. The pipeline integrates a learned globally consistent descriptor matching process for initial camera pose estimation, followed by a pose optimization phase that synergistically combines various features. To generate learned descriptors, we propose a siamese Globally Consistent Feature Descriptor Network (GCFDNet), which accepts a pair of images, Inertial Measurement Unit (IMU) data, and pose sequences as inputs, producing both the image descriptors and the relative camera pose as outputs. The strengths of GCFDNet manifest in two key aspects. First, by incorporating a spatial-to-temporal feature fusion module, GCFDNet enhances relative pose regression, meanwhile enabling accurate scene scale estimation. Second, we devise a loss function that balances descriptor similarity and distance, thereby improving the quality of descriptor learning. Using the initial camera poses derived from GCFDNet, we establish data associations across multiple frames and subsequently propose a combined Bundle Adjustment (BA) optimization framework that integrates hand-crafted features, learned descriptors, and semantic objects. To evaluate the localization performance, we conduct experiments across diverse datasets, including EuRoC, ScanNet, 7 Scenes, TUM RGB-D, and Bonn. The results demonstrate state-of-the-art performance in both static and dynamic scenes, outperforming existing methods. Additionally, we present ablation studies on GCFDNet and the combined BA process to further substantiate the efficacy of our approach.
Rethinking Token-Wise Feature Caching: Accelerating Diffusion Transformers With Dual Feature Caching.
Zou C, Zheng S, Zhang E … +6 more , Guo R, Xu H, Shi Z, He C, Hu X, Zhang L
IEEE Trans Image Process · 2026 · PMID 42241263 · Publisher ↗
Diffusion Transformers (DiT) have become the dominant methods in image and video generation yet still suffer substantial computational costs. As an effective approach for DiT acceleration, feature caching methods are des...
Diffusion Transformers (DiT) have become the dominant methods in image and video generation yet still suffer substantial computational costs. As an effective approach for DiT acceleration, feature caching methods are designed to cache the features of DiT in previous timesteps and reuse them in the next timesteps, allowing us to skip the computation in the next timesteps. Among them, token-wise feature caching has been introduced to perform different caching ratios for different tokens in DiTs, aiming to skip the computation for unimportant tokens while still computing the important ones. In this paper, we propose to carefully check the effectiveness in token-wise feature caching with the following two questions: 1) Is it really necessary to compute the so-called "important" tokens in each step? 2) Are so-called important tokens really important? Surprisingly, this paper gives some counter-intuition answers, demonstrating that consistently computing the selected "important tokens" in all steps is not necessary. The selection of the so-called "important tokens" is often ineffective, and even sometimes shows inferior performance than random selection. Based on these observations, this paper introduces dual feature caching referred to as DuCa, which performs aggressive caching strategy and conservative caching strategy iteratively and selects the tokens for computing randomly. Extensive experimental results demonstrate the effectiveness of our method in DiT, PixArt, FLUX, and OpenSora, demonstrating significant improvements than the previous token-wise feature caching.
Weakly Supervised Temporal Action Localization With Proposal-Level Action Consistency Learning.
Li M, Wang Z, Wang T … +3 more , Wang J, Wang J, Li B
IEEE Trans Image Process · 2026 · PMID 42228666 · Publisher ↗
Existing weakly supervised temporal action localization (WTAL) methods typically follow a decoupled classification-localization pipeline: segment-level classifiers are trained first, and their predictions are then aggreg...
Existing weakly supervised temporal action localization (WTAL) methods typically follow a decoupled classification-localization pipeline: segment-level classifiers are trained first, and their predictions are then aggregated to score proposals at inference. Under this training-inference discrepancy, proposal scoring at inference relies on an additional aggregation step, which can accumulate errors from noisy segment responses and thus undermine score reliability. Moreover, proposal scores are often directly used as confidence without explicit score-quality modeling or quality-aware evaluation, further contributing to pronounced score-quality misalignment and thus widening the classification-localization gap. To address proposal score-quality misalignment, we propose ACL-Net, a framework for proposal score calibration. At its core is a dual-axis Proposal-level Action Consistency Learning (PACL) paradigm, implemented through two complementary modules: (i) a Semantic Consistency Module (SCM) that refines proposal representations by maintaining fused class centers to enforce compact and robust same-class features; within SCM, a cross-modal consistency-driven Classification Enhancement Module (CEM) denoises the fused class centers to mitigate error accumulation under weak supervision; and (ii) a Process Consistency Module (PCM) that derives geometry-aware reference scores from relative temporal relations among overlapping proposals, guiding the model to assess proposal quality in terms of relative process completeness and improve score-quality alignment. By jointly modeling semantic and process consistency to calibrate proposal scores, ACL-Net markedly improves localization accuracy. On THUMOS14 and ActivityNet1.3, it achieves state-of-the-art performance with uniform and substantial gains across multiple established baselines, while markedly lowering the expected calibration error (ECE).
Boosting Few-Shot Continual Learning via Self-Adaptive Evolution.
Gu Z, Xu C, Wang Y … +4 more , Han C, Zhang L, Xia D, Cui Z
IEEE Trans Image Process · 2026 · PMID 42228665 · Publisher ↗
Few-shot continual learning (FSCL) has attracted increasing attention for real-world applications, where models must continuously adapt to new classes with only a few labeled samples while retaining prior knowledge. Thes...
Few-shot continual learning (FSCL) has attracted increasing attention for real-world applications, where models must continuously adapt to new classes with only a few labeled samples while retaining prior knowledge. These abilities are essential in dynamic environments where data availability is often sparse and nonstationary. However, traditional FSCL methods are largely confined to closed data spaces, which limits their generalizability when diverse and evolving distributions are involved. Inspired by the paradigm of human lifelong learning, we propose a new self-adaptive evolution framework for FSCL that enables continuous interaction with and adaptation to external environments. To exploit latent knowledge in large-scale models, we use an adaptive diffusion-based generator that not only implicitly captures the distribution of new few-shot samples but also produces more high-quality samples. To mitigate the inevitable variability in generation quality, we also use a reinforced sample selection module, comprising a generated sample explorer and a selection evaluator, which explicitly guides the retained distributions toward alignment with the large-scale models. Integrated with the continual model, these components are optimized in an iterative self-adaptive evolution framework, ensuring stable knowledge retention while improving adaptability to newly emerging classes. We validate our approach through experiments on three benchmarks, revealing its effectiveness in exploiting external distributions and achieving notable performance improvements.
Two-Timer-KAN: Dual-Exclusive Fourier KANs With Gaussian Fusion for Few-Shot Multimodal Remote Sensing Imagery Classification.
Li J, Wu H, Xu H … +3 more , Song R, Li Y, Du Q
IEEE Trans Image Process · 2026 · PMID 42228664 · Publisher ↗
Multimodal remote sensing imagery classification (MRSIC) aims to synergistically leverage complementary information from heterogeneous data sources, enabling precise land-cover classification. Existing MRSIC approaches p...
Multimodal remote sensing imagery classification (MRSIC) aims to synergistically leverage complementary information from heterogeneous data sources, enabling precise land-cover classification. Existing MRSIC approaches predominantly rely on abundant annotated samples, facing critical performance degradation under data-scarce scenarios that are particularly exacerbated by the inherent complexity of heterogeneous multimodal data. Furthermore, effectively extracting spatial-spectral information of multimodal data and fusing the cross-modal heterogeneous features persists as a significant challenge. To address these obstacles, we propose a pioneering few-shot MRSIC network, Two-timer-KAN, which integrates modality-specific feature extraction for spectral- and spatial-dominant data. Specifically, leveraging the nonlinear power of Kolmogorov-Arnold Networks (KANs), we develop the Dual-Exclusive Fourier KAN (DEF-KAN) encoder, which captures modality-specific global features in the frequency domain, bridging spectral and spatial gaps across various datasets. Following this, a Multivariate-Gaussian-based Cross-KAN (MG-Cross-KAN) is dedicated to enhancing the robustness of cross-modality fusion by capturing modality-shared features in a distribution-based manner. Finally, to further tackle classification ambiguity under limited annotated samples, we present a visual-textual bidirectional alignment strategy, which leverages textual descriptions as supplementary semantical knowledge to clarify class feature centers. Extensive experiments demonstrate that the proposed two-timer-KAN achieves superior performance, outperforming the state-of-the-art methods in both accuracy and robustness.
Interpretable General Image Fusion via Scalable Autoregressive Modeling.
Xin J, Shi B, Liang Z … +5 more , Hao J, Song X, Wang N, Li J, Gao X
IEEE Trans Image Process · 2026 · PMID 42228663 · Publisher ↗
Existing image fusion methods have developed increasingly sophisticated network architectures for exploiting modality-shared and modality-specific features. However, despite these advancements in feature extraction, most...
Existing image fusion methods have developed increasingly sophisticated network architectures for exploiting modality-shared and modality-specific features. However, despite these advancements in feature extraction, most methods ultimately rely on relatively simple implicit or explicit fusion strategies, which can compromise interpretability and limit fusion accuracy. In this paper, we incorporate visual autoregressive modeling to bridge the gap between implicit feature extraction and explicit modality fusion. First, the proposed approach conducts a low-to-high resolution autoregressive objective with modality-specific features, introducing a scalable feature autoregressive mechanism. It aggregates local and global contextual dependencies while enhancing implicit cross-scale interaction. Furthermore, to promote the consistency and complementarity across modalities, we embed an explicit high-order fusion strategy within the progressive modality-specific feature extraction process. This integration facilitates a next-scale synergistic relationship between implicit learning and explicit fusion. Our High-order Feature AutoRegressive Fusion framework (HFARFusion) provides a robust and interpretable solution for general image fusion tasks, effectively balancing fusion performance and transparency through the strengths of autoregressive learning. Extensive experiments demonstrate the outstanding performance of the proposed method in several classical fusion tasks, including infrared-visible, medical, multi-focus, and multi-exposure image fusion. Our code is available at https://github.com/happysbn/HFARFusion.
HP-Gaussian: Head Prior-Guided Gaussian Splatting for Personalized Talking Head Synthesis From Few-Second Video.
Shen S, Li W, Zhang Y … +3 more , Hu W, Lu J, Tan YP
IEEE Trans Image Process · 2026 · PMID 42228662 · Publisher ↗
Gaussian Splatting-based talking head synthesis has made significant progress in recent years, yet existing methods often struggle with generalization beyond specific training identity. In this paper, we propose Head Pri...
Gaussian Splatting-based talking head synthesis has made significant progress in recent years, yet existing methods often struggle with generalization beyond specific training identity. In this paper, we propose Head Prior guided Gaussian Splatting for personalized talking head synthesis (HP-Gaussian) that can generalize to new identities with only few training data. Unlike traditional optimization-based Gaussian Splatting methods, our approach directly predicts Gaussian parameters from multi-modal inputs, including audio and visual cues. This feed-forward design enables multiple identities pre-training, allowing the model to learn shared head priors from large-scale datasets, while supporting flexible speaker-specific adaptation. To further enhance Gaussian feature learning, we introduce a Spatial Gaussian Transformer that captures correlations among neighboring Gaussians, improving parameter estimation accuracy. Additionally, recognizing the critical importance of personalized speaking styles in the synthesis of high-quality talking videos, a two-stage training strategy is implemented. A base model is initially trained across diverse identities to establish the foundational head prior knowledge. Subsequently, we introduce the short-video personalized adaptation phase for more realistic customized talking video generation. Extensive experiments demonstrate that our HP-Gaussian can synthesize high-fidelity and personalized talking videos with remarkably few training examples, setting a new benchmark for efficiency and quality in talking head synthesis. We highly recommend viewing our demonstration video at https://youtu.be/RpjWdvikKhU for intuitive visual comparisons and qualitative results.
EfficientCovNet: Modeling the Pairwise Voxel Dependency for Brain ROI Segmentation.
Sun L, Zhao J, Shao W … +2 more , Zhu Q, Zhang D
IEEE Trans Image Process · 2026 · PMID 42228661 · Publisher ↗
Segmenting the brain magnetic resonance (MR) images to region-of-interest (ROI) is a fundamental step for many medical image analysis tasks. Convolutional neural networks (CNNs) excel in learning the high-level contextua...
Segmenting the brain magnetic resonance (MR) images to region-of-interest (ROI) is a fundamental step for many medical image analysis tasks. Convolutional neural networks (CNNs) excel in learning the high-level contextual features for image segmentation. However, such high-level features are low-order features, which cannot reflect the complex appearance patterns of brain MR images. Intuitively, using the high-order features can enhance the performance of CNNs. Therefore, in this paper, we propose a novel Efficient Covariance Network (EfficientCovNet) that models pairwise voxel dependency features and applies it to the brain ROI segmentation tasks. Our EfficientCovNet consists of two pathways: a pairwise voxel dependency feature learning pathway that uses a novel covariance convolution to efficiently capture the pairwise features from MR images, and a contextual feature learning pathway that extracts high-level contextual features using convolutional operations. The pairwise features and contextual features are then fused together to boost brain ROI segmentation performance. Experimental results on five datasets, i.e., IXI, LONI-LPBA40, OASIS, ADNI, and CC359 datasets, demonstrate that our EfficientCovNet achieves superior performance for brain ROI segmentation in comparison with the state-of-the-art methods.
Semantic-Decoupled and Knowledge-Shared Probabilistic Mapping Network for Multi-Grained Cross-Modal Retrieval.
Li W, Chai Y, Deng LJ … +3 more , Xiong R, Fan X, Tian Y
IEEE Trans Image Process · 2026 · PMID 42228660 · Publisher ↗
Cross-modal retrieval is essential for exploring semantic correlations between multimodal data. However, existing approaches face challenges in resolving semantic ambiguity and transferring knowledge with sparse sample g...
Cross-modal retrieval is essential for exploring semantic correlations between multimodal data. However, existing approaches face challenges in resolving semantic ambiguity and transferring knowledge with sparse sample generalization. To address these challenges, we propose a new Semantic-Decoupled and Knowledge-Shared Probabilistic Mapping Network (SKPMN). Specifically, the Semantic Decoupling and Distinction (SDD) module decomposes complex word-region relationships into relevance-driven representations. The Deep Probability Mapping (DPM) module introduces a paradigm shift by mapping multimodal features into probabilistic distributions, capturing the semantic similarities and the potential uncertainties that define sparse or ambiguous relationships. By combining the Attention Probabilistic Mapping (APM) module, the model can effectively transfer knowledge across similar samples while emphasizing critical distinctions, significantly enhancing generalization to sparse and ambiguous samples. Finally, the multi-grained alignment strategy establishes a novel integration of fine-grained patch-to-word alignment and coarse-grained global alignment. Experimental results show that SKPMN achieves superior retrieval accuracy across major benchmark datasets. Furthermore, we implement a channel resource allocation technique that allocates more transmission resources to semantically significant information. In resource-constrained environments, our approach leverages Joint Source-Channel Coding (JSCC) to enhance the efficiency of visual feature transmission.
Orientation-Guided Homography for Fine-Grained Cross-View Localization.
Zhang Y, Kang X, Duan P … +2 more , Li S, Song Z
IEEE Trans Image Process · 2026 · PMID 42228659 · Publisher ↗
Cross-view localization aims to estimate the precise position and orientation of a ground-view image by aligning it with satellite imagery. However, existing homography-based methods are typically evaluated under limited...
Cross-view localization aims to estimate the precise position and orientation of a ground-view image by aligning it with satellite imagery. However, existing homography-based methods are typically evaluated under limited orientation noise (±45°) and exhibit limited refinement capability, as their local correlation-based refinement relies on a reasonably good initial orientation estimate. To address these limitations, we propose a fine-grained cross-view localization method based on orientation-guided homography (OGH-Net), whose core idea is to predict an initial orientation prior that explicitly guides subsequent homography refinement. Specifically, we first design a hybrid bird's-eye-view (BEV) transformation to generate BEV images with preserved central geometry and expanded coverage. Then, we introduce a lightweight orientation-prior module that provides a coarse yaw estimate across the full ±180° range. Finally, we develop a multiscale iterative homography module that progressively refines the projection matrix through hierarchical iterations across multiple feature resolutions. Under cross-area, unknown-orientation conditions, it reduces mean localization error by 11% and mean orientation error by 27% on VIGOR, and further reduces mean localization error by 27% on KITTI compared with previous state-of-the-art methods. Moreover, OGH-Net runs in real time at up to 107 FPS on a single RTX 3090 GPU, offering a favorable trade-off between accuracy and efficiency. The code and trained models will be released at https://github.com/YC-Zhang2025/OGH-Net.
IQE-CLIP: Instance-Aware Query Embedding for Zero-/Few-Shot Anomaly Detection in Medical Domain.
Huang H, Sun W, Wu Z … +3 more , Lu D, Wu X, Zheng Y
IEEE Trans Image Process · 2026 · PMID 42228658 · Publisher ↗
Recently, the rapid advancements of vision-language models, such as CLIP, have led to significant progress in zero-/few-shot anomaly detection (ZFSAD) tasks. However, most existing CLIP-based ZFSAD methods commonly assum...
Recently, the rapid advancements of vision-language models, such as CLIP, have led to significant progress in zero-/few-shot anomaly detection (ZFSAD) tasks. However, most existing CLIP-based ZFSAD methods commonly assume prior knowledge of categories and rely on carefully crafted prompts tailored to specific scenarios. While such meticulously designed text prompts effectively capture semantic information in the textual space, they fall short of distinguishing normal and anomalous instances within the joint embedding space. Moreover, these ZFSAD methods are predominantly explored in industrial scenarios, with few efforts conducted for medical tasks. To this end, we propose an innovative framework for ZFSAD tasks in the medical domain, denoted as IQE-CLIP. We reveal that query embeddings, which incorporate both textual and instance-aware visual information, are better indicators for abnormalities. Specifically, we first introduce class-based prompting tokens and learnable prompting tokens for better adaptation of CLIP to the medical domain. Then, we design an instance-aware query module (IQM) to extract region-level contextual information from both text prompts and visual features, enabling the generation of query embeddings that are more sensitive to anomalies. Extensive experiments conducted on six medical datasets demonstrate that IQE-CLIP achieves state-of-the-art performance on both zero-shot and few-shot tasks. The source code and data are available at https://github.com/hongh0/IQE-CLIP.
About
- Frequency
- Sun
- Papers found
- 200
- RSS feed
- Subscribe