IEEE Transactions On Image Processing[JOURNAL]
Sun
200 papers
CLASH-CTTA: Class-Wise Shift-Aware Hierarchical Continual Test-Time Adaptation.
Li J, Feng S
IEEE Trans Image Process · 2026 · PMID 42360854 · Publisher ↗
The domain shift between training and test data has emerged as a critical obstacle to the generalization of deep models. Continual Test-Time Adaptation (CTTA), which aims to leverage online test data stream to adapt to c...
The domain shift between training and test data has emerged as a critical obstacle to the generalization of deep models. Continual Test-Time Adaptation (CTTA), which aims to leverage online test data stream to adapt to continuously evolving target distributions, has become a promising and practical solution to address real-world domain shift issues. Existing CTTA methods primarily rely on self-training frameworks based on output consistency and entropy-based loss functions, which often consider either the entire batch or reliable samples within the batch in isolation, failing to combine both of the advantages simultaneously. Inspired by fast and slow learning strategies in continual learning, we propose a fully source-free approach, dubbed CLAss-wise Shift-aware Hierarchical Continual Test-Time Adaptation (CLASH-CTTA). The adaptation process of CLASH-CTTA is carried out with a hierarchical updating strategy, where the test data stream promotes the slow learning of general representations, while the representative samples facilitate fast learning of domain-specific knowledge. The observation reveals that different classes exhibit varying sensitivities to different domains, leading to diverse discriminability across classes under domain shifts. Thus, a Class-wise Shift-aware Representative Set is maintained to provide representatives and mitigate the discrepancy. In addition, using Spearman's rank correlation as a novel perspective to examine the correctness of samples, we further filter representatives and align them to the class prototypes. Extensive experiments on three corruption domain shift datasets and one natural domain shift dataset demonstrate the superiority of our method compared with state-of-the-art methods, including in continual tasks, gradual tasks, and scenarios with diverse batch sizes.
Voxel-based Point Cloud Geometry Compression with Space-to-Channel Context.
Liu B, Ma Y, Luo A … +2 more , Li L, Liu D
IEEE Trans Image Process · 2026 Jun · PMID 42340914 · Publisher ↗
Voxel-based methods are among the most efficient for point cloud geometry compression, particularly with dense point clouds. However, they face limitations due to a restricted receptive field from the upsampling operatio...
Voxel-based methods are among the most efficient for point cloud geometry compression, particularly with dense point clouds. However, they face limitations due to a restricted receptive field from the upsampling operation, especially when handling high-bit-depth point clouds. To overcome this issue, we introduce a stage-wise Space-to-Channel (S2C) context model for both dense point clouds and the low-level part of sparse point clouds. This model utilizes a channel-wise autoregressive strategy to effectively integrate neighborhood information at a coarse resolution. For the high-level part of sparse point clouds, we further propose a level-wise S2C context model that addresses resolution limitations by incorporating Geometry Residual Coding (GRC) for consistent-resolution cross-level prediction. Additionally, we use the spherical coordinate system for its compact representation and enhance our GRC approach with a Residual Probability Approximation (RPA) module, which features a large kernel size. Experimental results show that our S2C context model not only achieves bit savings while maintaining or improving reconstruction quality but also reduces computational complexity compared to state-of-the-art voxel-based compression methods.
RIGI: Rectifying Image-to-3D Generation Inconsistency via Uncertainty-Aware Learning.
Wang J, Zheng Z, Xu W … +1 more , Liu P
IEEE Trans Image Process · 2026 · PMID 42340913 · Publisher ↗
Image-to-3D generation aims to predict a geometrically and perceptually plausible 3D model from a single 2D image. Conventional approaches typically follow a cascaded pipeline: initially generating multi-view projections...
Image-to-3D generation aims to predict a geometrically and perceptually plausible 3D model from a single 2D image. Conventional approaches typically follow a cascaded pipeline: initially generating multi-view projections from the single input image through view synthesis, followed by optimizing 3D geometry and appearance strictly using these projections. However, such deterministic optimization neglects epistemic uncertainty from imperfectly generated data, particularly due to limited observations and inconsistent content. To address this issue, we propose an uncertainty-aware optimization framework that explicitly models and mitigates epistemic uncertainty, leading to more robust and reliable 3D generation. For epistemic uncertainty arising from incomplete viewpoint coverage, we employ a progressive sampling strategy that sinusoidally varies camera elevations and progressively integrates diverse viewpoints into training, enhancing viewpoint coverage and stabilizing optimization. For epistemic uncertainty caused by the deterministic optimization on the noisy and inconsistent generated multi-view frames, we estimate an uncertainty map from the discrepancies between two independently optimized Gaussian models. This map is incorporated into uncertainty-aware regularization, adaptively adjusting loss weights to suppress unreliable supervision. Furthermore, we provide a theoretical analysis of uncertainty-aware optimization by deriving a probabilistic upper bound on the expected generation error, providing insights into its effectiveness. Extensive experiments demonstrate that our method significantly reduces artifacts and inconsistencies, leading to higher-quality 3D generation. More visual results are available at our website https://rigi3d.github.io/.
DA-Cal: Towards Cross-Domain Calibration in Semantic Segmentation.
Li W, Sun R, Li Z … +2 more , Chen Y, Zhang T
IEEE Trans Image Process · 2026 Jun · PMID 42340912 · Publisher ↗
While existing unsupervised domain adaptation (UDA) methods greatly enhance target domain performance in semantic segmentation, they often neglect network calibration quality, resulting in misalignment between prediction...
While existing unsupervised domain adaptation (UDA) methods greatly enhance target domain performance in semantic segmentation, they often neglect network calibration quality, resulting in misalignment between prediction confidence and actual accuracy-a significant risk in safety-critical applications. Our key insight emerges from observing that performance degrades substantially when soft pseudo-labels replace hard pseudo-labels in cross-domain scenarios due to poor calibration, despite the theoretical equivalence of perfectly calibrated soft pseudo-labels to hard pseudo-labels. Based on this finding, we propose DA-Cal, a dedicated cross-domain calibration framework that transforms target domain calibration into soft pseudo-label optimization. DA-Cal introduces a Meta Temperature Network to generate pixel-level calibration parameters and employs bi-level optimization to establish the relationship between soft pseudo-labels and UDA supervision, while utilizing complementary domain-mixing strategies to prevent overfitting and reduce domain discrepancies. Experiments demonstrate that DA-Cal seamlessly integrates with existing self-training frameworks across multiple UDA segmentation benchmarks, significantly improving target domain calibration while delivering performance gains without inference overhead. The code will be released.
Multi-Dimensional Quality Assessment for Single-Image-to-3D Contents: Dataset and Model.
Fu K, Duan H, Zhang Z … +7 more , Liu J, Liu Y, Liu X, Wang J, Min X, Le Callet P, Zhai G
IEEE Trans Image Process · 2026 · PMID 42340911 · Publisher ↗
The rapid advancement of AI generation technologies has led to the widespread use of AI-generated multimedia content, including images, videos, and 3D contents, across various applications. While significant progress has...
The rapid advancement of AI generation technologies has led to the widespread use of AI-generated multimedia content, including images, videos, and 3D contents, across various applications. While significant progress has been made in quality evaluation for 2D content, evaluating the quality of 3D content synthesized from single image remains an underexplored problem. To bridge this gap, we introduce the first comprehensive subjective evaluation database tailored for assessing the quality of 3D content generated from single image. Our database, named AIGC-SI23DCQA, includes three distinct categories of input images, i.e., realistic images, AI-generated images, and computer graphic (CG) images, with 100 images in each category. Using five representative single-image-to-3D algorithms, we produce 1,500 3D contents and collect 94,500 annotations across three quality dimensions, including texture fidelity, shape accuracy, and overall quality. Based on the constructed database, we first benchmark and evaluate the performance of existing quality assessment methods revealing their limitations in addressing this novel task. Thus, we further propose a novel objective quality assessment method, termed I3DQA, for effective single-image-to-3D content quality assessment. Specifically, I3DQA first extracts the reference features from the source image, and the multi-modal features from the generated 3D content, including the projected video, patches, and large-multimodal model (LMM) features. These features are integrated through symmetric transformer blocks, enabling effective quality-related feature fusion and score prediction. Extensive experiments demonstrate the superior performance of our method and validate the effectiveness of its components. This work provides a foundational resource and a robust framework for advancing research in this emerging field, and our database and model are released at https://github.com/ZedFu/SI23DCQA.
Enhancing Underwater Light Field Images via Global Geometry-Aware Diffusion Process.
Lin Y, Zhao Q, Yue Z … +2 more , Hou J, Meng D
IEEE Trans Image Process · 2026 · PMID 42340910 · Publisher ↗
This work studies the challenging problem of acquiring high-quality underwater images via 4-D light field (LF) imaging. To this end, we propose GeoDiff-LF, a novel diffusion-based framework built upon SD-Turbo to enhance...
This work studies the challenging problem of acquiring high-quality underwater images via 4-D light field (LF) imaging. To this end, we propose GeoDiff-LF, a novel diffusion-based framework built upon SD-Turbo to enhance underwater 4-D LF imaging by leveraging its spatial-angular structure. GeoDiff-LF consists of three key adaptations: 1) a modified U-Net architecture with convolutional and attention adapters to model geometric cues, 2) a geometry-guided loss function using tensor decomposition and progressive weighting to regularize global structure, and 3) an optimized sampling strategy with noise prediction to improve efficiency. By integrating diffusion priors and LF geometry, GeoDiff-LF effectively mitigates color distortion in underwater scenes. Extensive experiments demonstrate that our framework outperforms existing methods across both visual fidelity and quantitative performance, advancing the state-of-the-art in enhancing underwater imaging. The code will be publicly available at https://github.com/linlos1234/GeoDiff-LF.
Hierarchical Consistency Learning for Test-Time Adaptation in Camouflage Perception.
Zha M, Li T, Wang G … +5 more , Pei Y, Qiao C, Zhang J, Yang Y, Shen HT
IEEE Trans Image Process · 2026 · PMID 42340909 · Publisher ↗
Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes. Existing methods, constrained by the static train-then-freeze paradigm...
Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes. Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and annotation dependency, limiting their adaptability to scene variations and unseen camouflage patterns. To overcome these, we propose the hierarchical consistency learning (HCL) framework, which integrates test-time adaptation for dynamic representation recalibration. Specifically, we design the hierarchical representation reconstruction (HRR) to alleviate feature entanglement by synergizing spatial reconstruction with dual-stream frequency-domain decomposition, enhancing robustness against appearance homogenization. The pixel and spectrum inference provide structural and contextual priors. We further introduce task affinity guidance (TAG) to propagate knowledge across branches via channel-wise affinity, aligning local discriminative cues and mitigating semantic drift. To ensure semantic invariance, we formulate the prototype consistency calibration (PCC), which aggregates region features into compact prototypes and establishes prototype-feature similarity. This imposes implicit and hierarchical constraints that bridge task and representation gaps. Extensive experiments across four camouflaged and four underwater object benchmarks, under three degradation settings, demonstrate that our method consistently outperforms state-of-the-art approaches, highlighting its robustness and generalization under distribution shifts.
RoLiC: A Robust LiDAR-Camera Fusion Framework for 3D Object Detection.
Wang L, Sun S, Zhao J
IEEE Trans Image Process · 2026 · PMID 42340908 · Publisher ↗
In the 3D object detection task of autonomous driving systems, LiDAR and camera are the most crucial sensors, and current methods primarily focus on fusion strategies for these two complementary modalities. However, in r...
In the 3D object detection task of autonomous driving systems, LiDAR and camera are the most crucial sensors, and current methods primarily focus on fusion strategies for these two complementary modalities. However, in real-world driving scenarios, potential sensor failures pose critical risks, potentially undermining the effectiveness of fusion-based detection approaches. This work presents RoLiC, a robust LiDAR-camera fusion framework designed to handle three challenging deployment scenarios within a unified model: LiDAR failure, camera failure, and simultaneous LiDAR-camera failure. To mitigate cross-modal dependency and recover missing information, RoLiC introduces two cross-modality feature transformers (L2C and C2L) that bidirectionally complete features between modalities when partial data is available. To further enhance feature reliability, we design a Sparse Similarity Loss (SSL) that constrains feature learning within high-probability object regions. Moreover, RoLiC integrates a task-aware two-stage feature knowledge distillation strategy (MS1 and MS2), where MS1 captures cross-modality complementarities, and MS2 distills knowledge between the fused modality and complete modality settings. Extensive experiments on the nuScenes and KITTI benchmarks demonstrate that RoLiC consistently outperforms state-of-the-art methods across all sensor-failure conditions. Code is available at https://github.com/JLIN77/RoLiC.
Foundation Model-Based Zero-Shot Tissue Segmentation of Pathological Images via the Mixture of Local-to-Global Experts.
Ye Y, Yuan J, Tang J … +5 more , Wan P, Sun L, Sheng J, Zhang D, Shao W
IEEE Trans Image Process · 2026 · PMID 42340907 · Publisher ↗
Tissue segmentation in pathological images plays a crucial role for the diagnosis and prognosis of human cancers. However, due to the complexity of tumor micro-environment, it is difficult to annotate all tissue types es...
Tissue segmentation in pathological images plays a crucial role for the diagnosis and prognosis of human cancers. However, due to the complexity of tumor micro-environment, it is difficult to annotate all tissue types especially for the categories with small tissue proportions, which limits the ability of the traditional tissue segmentation models to these tissue types with zero training samples. To address the above issues, we present a novel architecture, ZSPMLG, that relies on pathology vision-language foundation model (i.e., CONCH) to learn pixel-wise classifiers for both seen and unseen tissue types based on their text descriptions. Specifically, we firstly apply large language model (LLM) to generate the descriptions for both seen and unseen tissue categories, followed by feeding them to the CONCH text encoder to acquire their corresponding prototypes that are shared by both vision and semantic space. By considering that the textual descriptions of specific tissue categories can be observed from the pathological images at different scales of magnification, our ZSPMLG consists of Mixture of Local Experts (MoLE) and Mixture of Global Experts (MoGE) modules, where MoLE performs the specialized decoding that can map individual scale patch-level representation to dense pixel-level representation, while MoGE aims at fusing the multi-scale representations together. Finally, a convolutional layer is designed to map the pixel-level representation to the category prototype for tissue segmentation on both seen and unseen categories. We evaluate our method on three datasets and the experimental results demonstrate the superiority of our method on both seen and unseen tissue categories.
Language-Driven Visual Data Generation for Zero-Shot HOI Detection.
Geng P, Zhang S, Yang J
IEEE Trans Image Process · 2026 · PMID 42340906 · Publisher ↗
Zero-shot human-object interaction (HOI) detection aims to recognize both seen and unseen interaction categories while detecting humans and objects in an image. However, due to the absence of training samples for unseen...
Zero-shot human-object interaction (HOI) detection aims to recognize both seen and unseen interaction categories while detecting humans and objects in an image. However, due to the absence of training samples for unseen categories, existing methods often overfit on seen HOIs and struggle to generalize to unseen ones. To address this issue, we introduce a novel Language-Driven Visual Data Generation (LD-VDG) approach that generates pseudo visual features from textual semantics of unseen HOIs. This provides an innovative solution enabling generalization to unseen HOIs without relying on visual samples. Specifically, we first design a text-to-vision (T-V) adapter to align HOI text and visual features, trained on seen HOIs with paired image-text data. For unseen HOIs, we guide the large language model to produce multiple fine-grained textual descriptions based on HOI labels, which are then encoded by the vision-language model and transformed into pseudo visual features via the T-V adapter. After that, these pseudo features together with real features from seen HOIs are jointly used to train a transformer-based HOI detector. In this way, our method enables effective recognition of unseen HOIs by leveraging language-driven visual representations. Experimental results on standard datasets demonstrate that the proposed LD-VDG outperforms previous methods. In particular, it achieves superior performance on unseen categories under various zero-shot settings.
Symmetric Entropy-Constrained Video Coding for Machines.
Sun Y, Liu M, Yao C … +5 more , Tang Q, Jin J, Lin W, Dufaux F, Zhao Y
IEEE Trans Image Process · 2026 · PMID 42340905 · Publisher ↗
As video transmission increasingly serves machine vision systems (MVS) instead of human vision systems (HVS), video coding for machines (VCM) has become a critical research topic. Existing VCM methods often bind codecs t...
As video transmission increasingly serves machine vision systems (MVS) instead of human vision systems (HVS), video coding for machines (VCM) has become a critical research topic. Existing VCM methods often bind codecs to specific downstream models, requiring retraining or supervised data, thus limiting generalization in multi-task scenarios. Recently, unified VCM frameworks have employed visual backbones (VB) and visual foundation models (VFM) to support multiple video understanding tasks with a single codec. They mainly utilize VB/VFM to maintain semantic consistency or suppress non-semantic information, but seldom explore how to directly link video coding with understanding under VB/VFM guidance. Hence, we propose a Symmetric Entropy-Constrained Video Coding framework for Machines (SEC-VCM). It establishes a symmetric alignment between the video codec and VB, allowing the codec to leverage VB's representation capabilities to preserve semantics and discard MVS-irrelevant information. Specifically, a bi-directional entropy-constraint (BiEC) mechanism ensures symmetry between the process of video decoding and VB encoding by suppressing conditional entropy. This helps the codec to explicitly handle semantic information beneficial to MVS while squeezing useless information. Furthermore, a semantic-pixel dual-path fusion (SPDF) module injects pixel-level priors into the final reconstruction. Through semantic-pixel fusion, it suppresses artifacts harmful to MVS and improves machine-oriented reconstruction quality. Experimental results on classical video understanding tasks and MLLM-based tasks show state-of-the-art (SOTA) rate-task performance. It achieves significant bitrate savings over H.266/VVC reference software VTM on video instance segmentation (37.4%), video object segmentation (29.8%), object detection (46.2%), multiple object tracking (44.9%), and MLLM-based video grounding (97.6%). The code is at https://github.com/Ws-Syx/SEC-VCM.
Perceptual Quality Assessment of Low-Light Enhanced Images: A Multi-Annotated Subjective Dataset and a Multimodal Objective Method.
Hu B, Hu Y, Li L … +2 more , Gu K, Gao X
IEEE Trans Image Process · 2026 · PMID 42340904 · Publisher ↗
Low-light Image Enhancement Algorithms (LIEAs) aim to improve the visibility and visual quality of images captured in low-light environments. However, none of the existing LIEAs can comprehensively restore all visual con...
Low-light Image Enhancement Algorithms (LIEAs) aim to improve the visibility and visual quality of images captured in low-light environments. However, none of the existing LIEAs can comprehensively restore all visual contents, which makes it inevitable for the Enhanced Low-light Images (ELIs) to have different degrees of distortion, thereby affecting the visual quality. Currently, there is little research focusing on the quality assessment of these ELIs, partly due to the lack of publicly available datasets. Moreover, existing quality assessment methods primarily focus on a single visual modality and fail to sufficiently exploit the structural information across multiple image attributes, consequently resulting in suboptimal prediction performance. To this end, this paper conducts a systematic study on both subjective and objective quality assessment of ELIs. Firstly, we construct the first Multi-annotated and multi-modal Low-light image Enhancement quality dataset (MLE), which contains 1,000 ELIs, along with subjective studies to obtain multiple attribute annotations, quality scores, and textual descriptions. Based on this, we further propose an Attribute-guided Vision-Language Graph Reasoning Network (AVGR-Net) for ELI quality prediction, which effectively integrates multi-attribute visual and textual information through cross-modal graph reasoning and alignment. Extensive data analysis and experimental results validate both the reliability of the MLE dataset and the superior performance of the AVGR-Net compared to state-of-the-art methods.
Robust Multi-View Clustering via Quadratic Matrix Factorization With Manifold Learning.
Wang Y, Zhang F, Jiang B
IEEE Trans Image Process · 2026 · PMID 42340903 · Publisher ↗
Multi-view clustering has attracted considerable attention due to its efficiency in handling high-dimensional data. Existing approaches based on non-negative matrix factorization can achieve dimensionality reduction and...
Multi-view clustering has attracted considerable attention due to its efficiency in handling high-dimensional data. Existing approaches based on non-negative matrix factorization can achieve dimensionality reduction and yield interpretable representations. However, these methods are often limited by the assumption of linearity and sensitive to noise, making it difficult to effectively capture complex nonlinear structures in data. To address these limitations, this paper proposes a robust multi-view clustering via quadratic matrix factorization with manifold learning. The method performs a quadratic matrix factorization on each view and decouples the linear tangent space and nonlinear normal space components by subspace constraints, which enhances the robustness of the data while fitting its nonlinear structure. Furthermore, we introduce consistency and complementarity regularization terms to effectively integrate multi-view information and derive consensus low-dimensional representations. For the formulated optimization model, we have designed an alternating optimization algorithm and conducted a theoretical analysis of its convergence properties. Experimental results on nine real-world datasets and five synthetic datasets show that the clustering performance and robustness of the method are significantly better than those of existing methods.
LoopExpose: An Unsupervised Framework for Arbitrary-Length Exposure Correction.
Li A, Chen C, Wang Z … +3 more , Huang T, Wu F, Dong W
IEEE Trans Image Process · 2026 Jun · PMID 42340902 · Publisher ↗
Exposure correction is essential for enhancing image quality under challenging lighting conditions. While supervised learning has achieved significant progress in this area, it relies heavily on large-scale labeled datas...
Exposure correction is essential for enhancing image quality under challenging lighting conditions. While supervised learning has achieved significant progress in this area, it relies heavily on large-scale labeled datasets, which are difficult to obtain in practical scenarios. To address this limitation, we propose a pseudo label-based unsupervised method called LoopExpose for arbitrary-length exposure correction. A nested loop optimization strategy is proposed to address the exposure correction problem, where the correction model and pseudo-supervised information are jointly optimized in a two-level framework. Specifically, the upper-level trains a correction model using pseudo-labels generated through multi-exposure fusion at the lower level. A feedback mechanism is introduced where corrected images are fed back into the fusion process to refine the pseudo-labels, creating a self-reinforcing learning loop. Considering the dominant role of luminance calibration in exposure correction, a Luminance Ranking Loss is introduced to leverage the relative luminance ordering across the input sequence as a self-supervised constraint. Extensive experiments on different benchmark datasets demonstrate that LoopExpose achieves superior exposure correction and fusion performance, outperforming existing state-of-the-art unsupervised methods. Code is available at https://github.com/FALALAS/LoopExpose.
An Episode Memory-Guided Dual-Stage Framework for Long-Form Video Temporal Grounding.
Liu T, Bao BK, Lam KM
IEEE Trans Image Process · 2026 · PMID 42340901 · Publisher ↗
Video temporal grounding (VTG) aims to localize video moments that are semantically related to a given natural language query. In spite of recent progress in short-form videos, research on VTG in long-form videos (e.g.,...
Video temporal grounding (VTG) aims to localize video moments that are semantically related to a given natural language query. In spite of recent progress in short-form videos, research on VTG in long-form videos (e.g., hours long) remains highly demanded yet underexplored. Existing methods predominantly adopt sliding window-based or multi-scale anchor-based strategies to generate temporal proposals, which require time-consuming post-processing or are independent of video content, thereby limiting their performance and efficiency. To address this dilemma, in this paper, we propose an episode memory-prompted (EMP) two-stage framework for temporal grounding in long-form videos. Specifically, the first stage generates a set of dynamic episode memories, which explicitly summarize various activities occurring throughout the lengthy video. An unsupervised memory learning paradigm is formulated by imposing discriminability and diversity constraints, eliminating the reliance on additional activity-instance annotations. Then, in the second stage, based on the supplement of frame-level detailed content and the guidance of a language query, the augmented memory prompts function as anchors for efficiently regressing the refined boundaries of the target video moment. Extensive experimental results on two public long-form video data sets, i.e., MAD and Ego4d, validate that the proposed EMP framework saves more than 8.5% trainable parameters and 13.9% FLOPs, while still achieving comparable performance with existing methods.
Revealing Photoshop Inpainting Traces Under JPEG Compressions.
Zhang Y, Zhang L, Qi S … +2 more , Xiao X, Wen W
IEEE Trans Image Process · 2026 · PMID 42335053 · Publisher ↗
Photoshop inpainting has become one of the most challenging targets in image forensics, as its content-aware and patch-based editing mechanisms produce visually coherent manipulations with weak and localized forensic art...
Photoshop inpainting has become one of the most challenging targets in image forensics, as its content-aware and patch-based editing mechanisms produce visually coherent manipulations with weak and localized forensic artifacts. This difficulty is further amplified by JPEG compression, which is routinely introduced during online transmission and tends to suppress the high-frequency tampering traces on which existing forensic detectors largely depend. As a result, current methods face an inherent trade-off: Photoshop-oriented detectors provide strong discriminability under clean conditions but lack robustness to compression, whereas compression-robust forensic methods often fail to capture subtle inpainting artifacts. To overcome this tension, this paper proposes a JPEG-resistant Photoshop inpainting localization method based on multi-frequency representation. The proposed framework employs a set of parameterized frequency-selective filters to extract complementary representations across multiple spectral bands. Each frequency branch is trained independently as a dedicated detector, and a fusion module integrates their outputs to generate a comprehensive localization map that balances discriminability and robustness. A theoretical analysis in the frequency domain is further provided to explain how low-frequency representations remain stable under JPEG-induced attenuation, supporting the design rationale of the proposed framework. In addition, a multi-quality JPEG augmentation strategy is adopted during training to mitigate the mismatch between training and testing compression levels. Extensive experiments on both script-created and hand-created Photoshop inpainting datasets demonstrate that the proposed method consistently outperforms representative forgery localization methods under various JPEG compression strengths. We further evaluate the method on images transmitted through Wechat, Weibo, and Twitter, confirming its effectiveness in practical online social network scenarios. These results demonstrate that the proposed multi-frequency representation strategy offers a principled and effective approach to robust image forensic analysis.
Pixel-Level RGBT Fusion Tracking via Heterogeneous Multi-Expert Distillation and Decoupled Representation Learning.
Lu A, Guo Y, Wang K … +3 more , Li C, Tang J, Luo B
IEEE Trans Image Process · 2026 · PMID 42329939 · Publisher ↗
Pixel-level fusion is widely considered a lightweight yet limited strategy in RGB-Thermal (RGBT) tracking due to its shallow representational capacity. However, its actual limitations and potential remain largely unexplo...
Pixel-level fusion is widely considered a lightweight yet limited strategy in RGB-Thermal (RGBT) tracking due to its shallow representational capacity. However, its actual limitations and potential remain largely unexplored. We systematically analyze fusion location, modality alignment, and tracking performance, revealing that despite lower modality gaps than feature-level fusion, pixel-level fusion lacks task-relevant discrimination, restricting its effectiveness. In this paper, we propose the Task-driven Pixel-level Fusion tracker (TPF), which preserves the efficiency of early fusion while enhancing discriminative capacity. Central to TPF is a lightweight pixel fusion adapter that ensures real-time image fusion with only 14.3KB extra parameters over the baseline at inference. To enhance its limited representational capacity, we propose a task-driven progressive learning framework consisting of two key stages. First, a heterogeneous multi-expert distillation scheme adaptively transfers image fusion knowledge from diverse models under tracking-guided evaluation, mitigating the generalization limitations of single-teacher distillation across varied tracking scenarios. Second, to overcome limited task discrimination caused by sparse, target-focused tracking supervision, we propose a decoupled representation learning strategy that offers dense, complementary guidance to improve target-background separation and fusion quality. A nearest-neighbor dynamic template update further enhances robustness to appearance changes. Extensive experiments on four RGBT tracking benchmarks show that TPF achieves competitive accuracy and speed, outperforming both feature-level and existing pixel-level fusion methods, offering new insights into efficient RGBT tracking.
Bridging Subjectivity in Affective Explanation Captioning via Consensus-Prompted Emotion Reasoning.
Song P, Zhang Z, Chen W … +3 more , Hu J, Yang X, Chang X
IEEE Trans Image Process · 2026 · PMID 42329938 · Publisher ↗
Affective Explanation Captioning (AEC) aims to perform viewer-centered visual emotion analysis by not only identifying the emotions evoked by an image but also explaining their underlying causes. Prior efforts have achie...
Affective Explanation Captioning (AEC) aims to perform viewer-centered visual emotion analysis by not only identifying the emotions evoked by an image but also explaining their underlying causes. Prior efforts have achieved promising results by fine-tuning LLMs on affective data; however, two key challenges remain: 1) the inherent subjectivity of human emotion leads to diverse interpretations of the same image, making it difficult for models to catch dominant emotions; and 2) the affective gap between abstract emotions and concrete visual content hinders models from capturing both semantic and emotional aspects effectively. To tackle these challenges, we propose Consensus-Prompted Emotion Reasoning (CPER), a new framework that explicitly models emotional diversity and enforces emotional-semantic alignment. Inspired by psychological studies, we observe that common emotional patterns often emerge within certain groups, which we refer to as affective consensus. Capturing this consensus across varying levels is helpful for bridging the subjectivity in AEC. Specifically, we introduce a consensus-based bucket prompt, which depicts the consensus level of each emotional perspective, serving as a control signal to adjust the emotion reasoning. To reconcile abstract emotion understanding and concrete visual grounding, we design a dual-space representation, where a CLIP encoder extracts objective semantic evidence and an emotion encoder captures abstract affective cues for AEC. Furthermore, an emotion consistency learning strategy is devised, which explicitly aligns the generated explanation with the input image and the emotion label, ensuring both emotionally and semantically grounded explanations. Extensive experiments on three benchmark datasets, ranging from visual arts (ArtEmis v1.0 and ArtEmis v2.0) and real-world images (Affection), demonstrate the effectiveness of our CPER in terms of emotional diversity and semantic coherence compared to state-of-the-art methods. Our code is publicly available at https://github.com/songpipi/CPER.
LiteMFT: Lightweight Multi-Modal Fine-Tuning for Semantic Segmentation.
Guo C, Zhang Y, Zhang M … +2 more , Liu H, Li W
IEEE Trans Image Process · 2026 · PMID 42313582 · Publisher ↗
Multi-modal image segmentation has recently attracted considerable attention due to its ability to integrate complementary information from diverse sensors, thereby enabling more accurate semantic predictions in complex...
Multi-modal image segmentation has recently attracted considerable attention due to its ability to integrate complementary information from diverse sensors, thereby enabling more accurate semantic predictions in complex or specialized scenarios. However, as data volume and model capacity continue to grow, many existing methods suffer substantial increases in parameters and computational costs, particularly with the widespread adoption of Vision Foundation Models (VFMs). To address these challenges, we introduce a Lightweight Multi-modal Fine-Tuning framework (LiteMFT) designed for efficient and generalizable adaptation of RGB-pretrained VFMs to multi-modal semantic segmentation. By incorporating only a small number of trainable parameters, LiteMFT enables effective extension of existing models to handle multi-modal image fusion tasks. The framework centers around two key components: the Modality Local Competition (MLC) module, which dynamically and efficiently fuses complementary features across modalities, and the Gated Low-Rank Adapter (GLR), which improves the backbone's adaptability to multi-modal data through content-aware low-rank transformation. Extensive experiments on both bi-modal and tri-modal segmentation tasks demonstrate that LiteMFT not only achieves competitive or superior performance but also exhibits strong scalability for additional modalities, underscoring its practicality and broad applicability in multi-modal semantic segmentation.
SSVIF: Self-Supervised Segmentation-Oriented Visible and Infrared Image Fusion.
Zhao Z, Zhang X
IEEE Trans Image Process · 2026 Jun · PMID 42313581 · Publisher ↗
Visible and infrared image fusion (VIF) has gained significant attention in recent years due to its wide application in tasks such as scene segmentation and object detection. VIF methods can be broadly classified into tr...
Visible and infrared image fusion (VIF) has gained significant attention in recent years due to its wide application in tasks such as scene segmentation and object detection. VIF methods can be broadly classified into traditional VIF methods and application-oriented VIF methods. Traditional methods focus solely on improving the quality of fused images, while applicationoriented VIF methods additionally consider the performance of downstream tasks on fused images by introducing task-specific loss terms during training. However, compared to traditional methods, application-oriented VIF methods require datasets labeled for downstream tasks (e.g., semantic segmentation or object detection), making data acquisition labor-intensive and time-consuming. To address this issue, we propose a self-supervised training framework for segmentation-oriented VIF methods (SSVIF). Leveraging the consistency between feature-level fusion-based segmentation and pixel-level fusion-based segmentation, we introduce a novel self-supervised task, i.e., cross-segmentation consistency, that enables the fusion model to learn high-level semantic features without the supervision of segmentation labels. Additionally, we design a two-stage training strategy and a dynamic weight adjustment method for effective joint learning within our self-supervised framework. Extensive experiments on public datasets demonstrate the effectiveness of our proposed SSVIF. Remarkably, although trained only on unlabeled visible-infrared image pairs, our SSVIF outperforms traditional VIF methods and rivals supervised segmentation-oriented ones. Our code will be released upon acceptance.
About
- Frequency
- Sun
- Papers found
- 200
- RSS feed
- Subscribe