MetaWeather: Few-shot Weather-Degraded Image Restoration

KAIST
ECCV 2024

*Indicates Equal Contribution
MetaWeather overview

We propose MetaWeather, a novel few-shot weather-degraded image restoration method that enables restoration under any unseen weather conditions. To the best of our knowledge, this is the first work introducing few-shot learning to the image restoration task for unseen weather conditions without any prior assumption.

Abstract

Real-world weather conditions are intricate and often occur concurrently. However, most existing restoration approaches are limited in their applicability to specific weather conditions in training data and struggle to generalize to unseen weather types, including real-world weather conditions. To address this issue, we introduce MetaWeather, a universal approach that can handle diverse and novel weather conditions with a single unified model. Extending a powerful meta-learning framework, MetaWeather formulates the task of weather-degraded image restoration as a few-shot adaptation problem that predicts the degradation pattern of a query image, and learns to adapt to unseen weather conditions through a novel spatial-channel matching algorithm. Experimental results on the BID Task II.A, SPA-Data, and RealSnow datasets demonstrate that the proposed method can adapt to unseen weather conditions, significantly outperforming the state-of-the-art multi-weather image restoration methods.

Method

MetaWeather architecture

 

MetaWeather architecture

Overall architecture of MetaWeather. MetaWeather consists of a hierarchical encoder-decoder design and matching module. Our matching module matches the degradation pattern between query and support set images, which enables MetaWeather to fully utilize a few-shot support set. The matching results are passed on to the decoder blocks at each level, and the extracted degradation pattern of the query image is then subtracted from the query image, resulting in the clean query image.

 

Attention modules

Attention modules in our matching module.

Results

Qualitative Comparisons

Qualitative comparison

Effect of Degradation Pattern Matching

Effect of Degradation Pattern Matching

Effect of Spatial-Channel Matching

Effect of Spatial-Channel Matchingg

BibTeX

TBA