Proj CDeepFuzz Paper Reading: Aries: Efficient Testing of Deep Neural Networks via Labeling-Free Accuracy Estimation

Abstract

背景：

1. the de facto standard to assess the quality of DNNs in the industry is to check their performance (accuracy) on a collected set of labeled test data
2. test selection can save labor and then be used to assess the model

前提： the model should have similar prediction accuracy on the data which have similar distances to the decision boundary

本文：Aries
Github: https://github.com/wellido/Aries
Task: estimate the performance of DNNs on new unlabeled data using only the information obtained from the original test data

实验：
数据集：CIFAR-10, Tiny-ImageNet
对象：CIFAR10-ResNet20, CIFAR10-VGG16, TinyImageNet-ResNet101, TinyImageNet-DenseNet, 13 types of data transformation methods.
Competitors: Cross Entropy-based Sampling, Practical Accuracy Estimation
效果：

1. Aries accuracy与真实accuracy的效果只差了0.03%-2.60%
2. outperforms other labeling-free methods in 50/52 cases
3. outperforms other selection-labeling based methods in 96/128 cases

3. Methodology

Finding 1: A DNN has similar accuracy on the data sets that have similar distances to the decision boundary.
Finding 2: There is a linear relationship between the % of highly confident data (LV R = 1) and the accuracy of the whole set. Therefore, given some labeled data, if we know 1) the accuracy of the DNN in each Bucket, and 2) the percentage of highly confident data, it is promising to estimate the accuracy of the new unlabeled data.

本栏目推荐文章
• LocPatcH An efficient long-read hybrid error correction algorithm based on local pHMM
• UNetFormer: A UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery
• Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
• Improving Computer Vision Accuracy using Convolutions
• TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation
• 2023CVPR_Efficient Frequency Domain-based Transformers for High-Quality Image Deblurring
• Checkerboard Context Model for Efficient Learned Image Compression
• ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding
• 【CVPR2023】Efficient and Explicit Modelling of Image Hierarchies for Image Restoration
• 2023ACMMM_Mutual Information-driven Triple Interaction Network for Efficient Image Dehazing

Labeling-Free Estimation CDeepFuzz Efficient Accuracylabeling-free estimation cdeepfuzz efficient backpropagation sparseprop cdeepfuzz efficient labeling-free accuracy tesseract accuracy improve how convolutions improving computer accuracy estimation curve precision-recall precision accuracy precision accuracy recall estimation geolayout geometry driven