librarytesting/muffin repository overview

⁠Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing

This is the implementation docker images of our ICSE'22 paper: Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing.

⁠Description

Deep learning (DL) techniques are shown to be effective in many challenging tasks, and are hence widely-adopted in practice. However, previous work has shown that DL libraries, the basis of building and executing DL models, contain bugs and can cause severe consequences. Unfortunately, existing approaches still cannot comprehensively excise DL libraries. They utilize existing trained models and only detect bugs in model inference phase. In this work we propose Muffin to address these issues. To this end, Muffin applies a specifically-designed model fuzzing approach, which allows it to generate diverse DL models to explore the target library, instead of relying only on existing trained models. Muffin makes differential testing feasible in the model training phase by tailoring a set of metrics to measure the inconsistency between different DL libraries. In this way, Muffin can best excise the library code to detect more bugs. Experiments on three widely-used DL libraries show that Muffin can detect 39 new bugs.

You can access this repository using the following command:

git clone https://github.com/library-testing/Muffin.git

⁠Libraries

We use three widely-used DL libraries (i.e., TensorFlow, Theano, and CNTK) as the backend low-level libraries as our testing targets, and Keras as the frontend high-level library. To sufficiently illustrate the effectiveness of Muffin, we utilize a total of 15 release versions of the three backend libraries, and construct five experimental environments for differential testing as follow:

In order to facilitate other researchers to reproduce Muffin, we provide docker images for each experiments (i.e., E1 ~ E5), named librarytesting/muffin with tags from E1 to E5 respectively.

⁠Details

More details please refer to https://github.com/library-testing/Muffin⁠ .