Picture for Manish Ahuja

Manish Ahuja

Metamorphic Testing of a Deep Learning based Forecaster

Add code
Jul 13, 2019
Anurag Dwarakanath, Manish Ahuja, Sanjay Podder, Silja Vinu, Arijit Naskar, Koushik MV

Figure 1 for Metamorphic Testing of a Deep Learning based Forecaster
Figure 2 for Metamorphic Testing of a Deep Learning based Forecaster
Figure 3 for Metamorphic Testing of a Deep Learning based Forecaster
Figure 4 for Metamorphic Testing of a Deep Learning based Forecaster

In this paper, we present the Metamorphic Testing of an in-use deep learning based forecasting application. The application looks at the past data of system characteristics (e.g. `memory allocation') to predict outages in the future. We focus on two statistical / machine learning based components - a) detection of co-relation between system characteristics and b) estimating the future value of a system characteristic using an LSTM (a deep learning architecture). In total, 19 Metamorphic Relations have been developed and we provide proofs & algorithms where applicable. We evaluated our method through two settings. In the first, we executed the relations on the actual application and uncovered 8 issues not known before. Second, we generated hypothetical bugs, through Mutation Testing, on a reference implementation of the LSTM based forecaster and found that 65.9% of the bugs were caught through the relations.

* Paper published at the 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing (MET) 
Viaarxiv icon
Github IconAccess Paper or Ask Questions
Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon

Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing

Add code
Aug 16, 2018
Anurag Dwarakanath, Manish Ahuja, Samarth Sikand, Raghotham M. Rao, R. P. Jagadeesh Chandra Bose, Neville Dubash, Sanjay Podder

Figure 1 for Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing
Figure 2 for Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing
Figure 3 for Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing
Figure 4 for Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing

We have recently witnessed tremendous success of Machine Learning (ML) in practical applications. Computer vision, speech recognition and language translation have all seen a near human level performance. We expect, in the near future, most business applications will have some form of ML. However, testing such applications is extremely challenging and would be very expensive if we follow today's methodologies. In this work, we present an articulation of the challenges in testing ML based applications. We then present our solution approach, based on the concept of Metamorphic Testing, which aims to identify implementation bugs in ML based image classifiers. We have developed metamorphic relations for an application based on Support Vector Machine and a Deep Learning based application. Empirical validation showed that our approach was able to catch 71% of the implementation bugs in the ML applications.

* Published at 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2018) 
Viaarxiv icon
Github IconAccess Paper or Ask Questions
Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon