Face technology: the picture of unclear people is repaired into a high-quality and high-definition image framework (with source code download)

Pay attention to the parallel stars

Never get lost

Institute of computer vision

official account ID｜ComputerVisionGzq

Study Group ｜ Scan the code to get the join mode on the homepage

Address of thesis ：https://arxiv.org/pdf/2201.06374.pdf

Code address ：https://github.com/wzhouxiff/RestoreFormer.git

Computer Vision Institute column

author ：Edison_G

Blind face Restoration is to restore high-quality face images from unknown degradation . Because face images contain rich contextual information , Researchers have proposed a method ,RestoreFormer, It explores the full spatial attention of modeling contextual information , And it goes beyond the existing work of using local operators .

01

summary

Blind face Restoration is to restore high-quality face images from unknown degradation . Because face images contain rich contextual information , Researchers have proposed a method ,RestoreFormer, It explores the full spatial attention of modeling contextual information , And it goes beyond the existing work of using local operators .

Compared with the prior art ,RestoreFormer There are several benefits . First , Compared with the previous Vision Transformers(ViT) The traditional multi head self attention is different ,RestoreFormer A multi head cross attention layer is merged to learn the full space interaction between damaged queries and high-quality key value pairs . secondly ,ResotreFormer The key value pairs in are sampled from a high-quality reconstruction oriented dictionary , Its elements are rich , It has high-quality face features specially for face reconstruction , Thus, it has excellent recovery effect . Third ,RestoreFormer Superior to advanced state-of-the-art methods on one synthetic dataset and three real-world datasets , And generate images with better visual quality .

02

background

Blind face Recovery aims to recover from the complex and diverse degradation that has been suffered （ Sample as follows 、 Fuzzy 、 noise 、 Compress artifacts, etc ） Restore high-quality faces from degraded faces . Because degradation is unknown in the real world , So recovery is a challenging task .Blind face Restoration aims to restore high-quality faces from complex and unknown degradation . Previous work shows that , Additional priors play a crucial role in this task , They can be roughly divided into three types ： The geometric 、 A priori and generative a priori .

Methods based on geometric priors tend to use landmark Heat map or face component heat map gradually restores the face . Because these geometric priors are mainly generated from low-quality faces , Therefore, the damaged face limits the performance of recovery . On the other hand , Reference based works need to have the same identity as the degenerated face , This is not always accessible . Although some researchers have alleviated this limitation by collecting component dictionaries composed of high-quality facial component features as general references , The facial details in these component dictionaries are limited , Because they are extracted with models for offline recognition , And only pay attention to some facial components .

Vision Transformer.Transformer It is a deep neural network originally used in the field of natural language processing . Because of its competitive presentation ability , It began to be applied to computer vision tasks , For example, identification 、 Detection and segmentation . In some papers , Low level visual tasks also benefit . Some researchers use Transformer Advantages in large-scale pre training , Build a complex model , It covers multiple image processing tasks , For example, denoising 、 Rain removal and super resolution . Ethel et al 【Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis】 application transformer High resolution images are generated by predicting a series of codebook indexes of its encoder , Make full use of strong representativeness transformer Capacity within acceptable computing resources . stay 【Mingrui Zhu, Changcheng Liang, Nannan Wang, Xiaoyu Wang, Zhifeng Li, and Xinbo Gao. A sketch-transformer network for face photo-sketch synthesis】 in , use transformer Get the global structure of the face , Help photo-sketch Synthesis .

03

New framework analysis

(a)MHSA It is a kind of self attention with multiple heads transformer, Used in most previous ViT. Its query 、 Keys and values come from degraded information Zd.(b)MHCA It is a multi headed cross attention transformer, For proposed RestoreFormer. It aims to pass Zd As a query , take Zp As a key value pair , Integrate degraded information in space Zd And its corresponding high-quality priors Zp.(c) yes RestoreFormer The whole process of . First deploy the encoder Ed To extract degenerate faces Id It means Zd, And from HQ Dictionaries D Extract its recent high-quality priors Zp. Then use two MHCA Fuse degenerate features Zd And a priori Zp. Last , In fusion, it means Z0f On the application decoder Dd To restore high-quality faces Id.

Comparison of Prior Dictionary.(a)DFDNet The component dictionary proposed in is composed of VGG Generated offline by the network , And use K-means Clustering . They only think about eyes 、 Nose and mouth .(b) Today, researchers put forward HQ Dictionary It is learned through the high-quality face generation network combined with the idea of vector quantization .HQ Dictionary The high-quality priors in are reconstruction oriented , Provide more face details for the restoration of degraded faces . Besides HQ Dictionary A priori in involves all facial regions .

04

Experiment and visualization

 THE END 

Please contact the official account for authorization.

The learning group of computer vision research institute is waiting for you to join ！

We created “ Computer vision society ” Knowledge planet has more than two years , It has also been recognized by many students , Recently, we started the operation of knowledge planet . We Regular meeting Push practical content to share with you , Students on the planet can Ask questions at any time , Be ready to ask for it , We will give timely reply and corresponding reply .

ABOUT

Institute of computer vision

The Institute of computer vision is mainly involved in the field of deep learning , Mainly devoted to face detection 、 Face recognition , Multi target detection 、 Target tracking 、 Image segmentation and other research directions . The Research Institute will continue to share the latest paper algorithm new framework , The difference of our reform this time is , We need to focus on ” Research “. After that, we will share the practice process for the corresponding fields , Let us really experience the real scene of getting rid of the theory , Develop the habit of hands-on programming and brain thinking ！

VX：2311123606

Previous recommendation  

• Yolov7： The latest and fastest real-time detection framework , The most detailed analysis and explanation （ With source code ）

• SSD7 | Embedded friendly target detection network , Product landing

• Accuracy improvement method ： The adaptive Tokens Efficient vision Transformer frame （ Open source ）
  

• ONNX elementary analysis ： How to accelerate the engineering of deep learning algorithm ？

• Explode ！YOLOv6 The fast and accurate target detection framework is open source （ With source code download ）

• FastestDet： Than yolov5 faster ！ stronger ！ Newly designed ultra real time Anchor-free Target detection algorithm （ With source code download ）
  

• At present, the target detection model with the highest accuracy, the fastest efficiency and the smallest storage （ Download the source code ）

• CVPR Small target detection ： Context and attention mechanisms improve small target detection （ Attached thesis download ）
  

• Double-Head： Innovation on the detection head , Improve accuracy （ Attached is the original paper download ）

• Produced by Haikang Research Institute ： Unsupervised target detection with scene adaptive concept learning （ Attached thesis download ）
  

• New technology ： Efficient self supervised visual pre training , There is no need to worry about partial occlusion ！
  

• VS Code Supports configuring remote synchronization

• Improved shadow suppression for illumination robust face recognition
  

• Text driven for creating and editing images （ With source code ）
  

• Based on hierarchical self - supervised learning, vision Transformer Scale to gigapixel images
  

• Top frame ： Industrial testing , A semi supervised method based on difference and commonness is applied to image surface defect detection
  

• CLCNet： Rethink integrated modeling with classified confidence networks （ With source code download ）