[Colloquium] Huiying Li Dissertation Defense/Mar 6, 2023

[Megan Woodward]

This is an announcement of Huiying Li's Dissertation Defense.
===============================================
Candidate: Huiying Li

Date: Monday, March 06, 2023

Time: 10:30 am CST

Location: JCL 298

Title: Revealing and Mitigating Vulnerabilities of Deep Neural Networks in the Wild

Abstract: Although Deep Neural Networks (DNNs) are widely used in applications such as facial or iris recognition and language translation, there is growing concern about their feasibility in safety-critical or security-critical contexts. Researchers have found that DNNs can be manipulated by poison attacks like backdoor attacks, and are vulnerable to evasion attacks like adversarial attacks. Attackers can compromise DNN models by injecting backdoors during the training phase or by adding imperceptible perturbations to model inputs via adversarial attacks. To ensure secure and reliable deep learning systems, it is crucial to identify and mitigate these vulnerabilities.
Despite active efforts within the adversarial machine learning community to identify vulnerabilities in deep neural networks (DNNs), there remains a significant gap between current research and the practical deployment of these systems in the real world. According to recent studies, model practitioners often do not anticipate potential attacks on their models in the near future. This is largely due to the fact that previous research on machine learning security has oversimplified threat models, which do not accurately reflect real-world scenarios.
For example, existing backdoor attack methods assume that users train their own models from scratch, which is not commonly done in practice. Instead, users often customize pre- trained “Teacher” models provided by companies such as Google, using transfer learning techniques. Additionally, current backdoor attack research assumes that models are static and do not change over time. However, in reality, most production machine learning models are continuously updated to address changes in the targeted data distribution. Finally, while black-box adversarial attacks have been proven to be a significant threat to DNN systems in the wild, there are currently no effective scalable defenses against them. Existing work either assumes that the defender can sacrifice normal model performance significantly, or that the attacker cannot send attack queries with multiple sybil accounts, both of which conflict with the reality of the situation.
In this dissertation, I seek to reveal and mitigate DNN vulnerabilities in practical settings by designing and measuring attacks and defenses against DNNs under realistic threat models. Particularly, my dissertation consists of three components that target the aforementioned challenges.
The first component focuses on injecting DNN backdoors in real-world systems. As training a production model from scratch is resource-intensive, entities often use existing massive, centrally trained models (VGG16 model pre-trained on VGG-Face dataset of 2.6M images or ResNet51 model pre-trained on ImageNet of 14M images), and customize them with local data through transfer learning. In practice, the transfer learning process breaks all backdoors embedded in the “Teacher” models. To enable backdoor attack in this scenario, I propose a latent backdoor attack that embeds incomplete backdoors into a “Teacher” model, which are automatically completed through transfer learning and inherited by multiple “Student” models. I also present an effective defense against latent backdoor attacks during transfer learning.
The second component examines the impact of backdoor attacks on time-varying models, where model weights are regularly updated using fine-tuning to handle changes in data distribution over time. While previous studies have focused on injecting backdoor attacks and assumed that they would remain permanently in place, real-world models need to be updated to handle natural data drifts. To understand how backdoors behave after they are injected on time-varying models, I conduct a comprehensive study and find that they are gradually forgotten once the poisoning stops. I propose “backdoor survivability”, a new metric to quantify how long a backdoor can survive on time-varying models and explore the factors that affect backdoor survivability. I also propose a smart training strategy that can reduce backdoor survivability significantly with negligible overhead. Finally, I discuss the need for new backdoor defenses that target time-varying models specifically.
The third component addresses the problem of building a scalable and robust defense system against black-box adversarial attacks on DNNs. Query-based black-box adversarial attacks are real-world threats as they require only inference access to the target model and are cheap and easy to execute. To defend against such attacks, I propose a defense system called Blacklight, which is designed to efficiently detect and reject attack queries. The key insight behind Blacklight is that these attacks perform iterative optimization over the network to compute adversarial examples, resulting in image queries that are highly similar in the input space. By detecting the occurrence of highly similar queries, one can effectively identify attack queries. The key challenge in building such a defense system is scalability. In particular, the system needs to efficiently handle millions of queries per day in industry production systems. To overcome this challenge, Blacklight uses probabilistic fingerprinting to detect highly similar images, achieving a constant runtime empirically. By rejecting all detected queries, Blacklight can prevent any attack from succeeding, even when attackers persist in submitting queries after account bans or query rejections.
Finally, I summarize my work on revealing and mitigating real-world DNN attacks under practical constraints and discuss my insights in this area. I hope my work can bridge the gap between the exploration of DNN attacks and defenses and their application in real-world systems and inspire further research on DNN vulnerabilities under real-world scenarios.

Advisors: Ben Zhao and Heather Zheng

Committee Members: Ben Zhao, Heather Zheng, and Rana Hanocka




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.uchicago.edu/pipermail/colloquium/attachments/20230220/cebdf83a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: thesis-abs-intro-huiyingli.pdf
Type: application/pdf
Size: 796897 bytes
Desc: thesis-abs-intro-huiyingli.pdf
URL: <http://mailman.cs.uchicago.edu/pipermail/colloquium/attachments/20230220/cebdf83a/attachment-0001.pdf>