Our research seeks to empower individuals and organizations to control how their data is used. We use techniques from cryptography, programming languages, machine learning, operating systems, and other areas to both understand and improve the security of computing as practiced today, and as envisioned in the future.

Everyone is welcome at our research group meetings (most Fridays at 11am, but join the slack group for announcements). To get announcements, join our Slack Group (any @virginia.edu email address can join themsleves, or email me to request an invitation).


Adversarial Machine Learning

Secure Multi-Party Computation
Obliv-C · MightBeEvil

Recent Posts

When Relaxations Go Bad: "Differentially-Private" Machine Learning

We have posted a paper by Bargav Jayaraman and myself on When Relaxations Go Bad: “Differentially-Private” Machine Learning (code available at https://github.com/bargavj/EvaluatingDPML).

Differential privacy is becoming a standard notion for performing privacy-preserving machine learning over sensitive data. It provides formal guarantees, in terms of the privacy budget, ε, on how much information about individual training records is leaked by the model.

While the privacy budget is directly correlated to the privacy leakage, the calibration of the privacy budget is not well understood. As a result, many existing works on privacy-preserving machine learning select large values of ϵ in order to get acceptable utility of the model, with little understanding of the concrete impact of such choices on meaningful privacy. Moreover, in scenarios where iterative learning procedures are used which require privacy guarantees for each iteration, relaxed definitions of differential privacy are often used which further tradeoff privacy for better utility.

We evaluated the impacts of these choices on privacy in experiments with logistic regression and neural network models, quantifying the privacy leakage in terms of advantage of the adversary performing inference attacks and by analyzing the number of members at risk for exposure.

Accuracy Loss as Privacy Decreases
(CIFAR-100, neural network model)

Privacy Leakage
(Yeom et al.’s Membership Inference Attack)

Our main findings are that current mechanisms for differential privacy for machine learning rarely offer acceptable utility-privacy tradeoffs: settings that provide limited accuracy loss provide little effective privacy, and settings that provide strong privacy result in useless models.

The table below shows the number of individuals, out of 10,000 members in the training set, exposed by a membership inference attack, given tolerance for false positives of 1% or 5% (and assuming a priori prevalence of 50% members). The key observations is that all the relaxtions provide lower utility (more accuracy loss) than naïve composition for comparable privacy leakage, as measured by the number of actual members exposed in a test dataset. Further, none of the methods provide both acceptable utility and meaningful privacy — at a high level, either nothing is learned from the training data, or some sensitive data is exposed. (See the paper for more details and results.)

 Naïve Composition Advanced Composition Zero Concentrated Rényi
Epsilon Loss 1% 5% Loss 1% 5% Loss 1% 5% Loss 1% 5%
0.1 0.95 0 0 0.95 0 0 0.94 0 0 0.93 0 0
1 0.94 0 0 0.94 0 0 0.92 0 6 0.91 0 94
10 0.94 0 0 0.87 0 1 0.81 0 20 0.80 0 109
100 0.93 0 0 0.61 1 32 0.49 30 281 0.48 11 202
1000 0.59 0 11 0.06 13 359 0.00 28 416 0.07 22 383
0.00 155 2667 No privacy noise added.

Bargav Jayaraman talked about this work at the DC-Area Anonymity, Privacy, and Security Seminar (25 February 2019) at the University of Maryland:

Paper: When Relaxations Go Bad: “Differentially-Private” Machine Learning
Code: https://github.com/bargavj/EvaluatingDPML)

Deep Fools

New Electronics has an article that includes my Deep Learning and Security Workshop talk: Deep fools, 21 January 2019.

A better version of the image Mainuddin Jonas produced that they use (which they screenshot from the talk video) is below:

Markets, Mechanisms, Machines

My course for Spring 2019 is Markets, Mechanisms, Machines, cross-listed as cs4501/econ4559 and co-taught with Denis Nekipelov. The course will explore interesting connections between economics and computer science.

My qualifications for being listed as instructor for a 4000-level Economics course are limited to taking an introductory microeconomics course my first year as an undergraduate.

Its good to finally get a chance to redeem myself for giving up on Economics 28 years ago!

ICLR 2019: Cost-Sensitive Robustness against Adversarial Examples

Xiao Zhang and my paper on Cost-Sensitive Robustness against Adversarial Examples has been accepted to ICLR 2019.

Several recent works have developed methods for training classifiers that are certifiably robust against norm-bounded adversarial perturbations. However, these methods assume that all the adversarial transformations provide equal value for adversaries, which is seldom the case in real-world applications. We advocate for cost-sensitive robustness as the criteria for measuring the classifier’s performance for specific tasks. We encode the potential harm of different adversarial transformations in a cost matrix, and propose a general objective function to adapt the robust training method of Wong & Kolter (2018) to optimize for cost-sensitive robustness. Our experiments on simple MNIST and CIFAR10 models and a variety of cost matrices show that the proposed approach can produce models with substantially reduced cost-sensitive robust error, while maintaining classification accuracy.

This shows the results of cost-sensitive robustness training to protect the odd classes. By incorporating a cost matrix in the loss function for robustness training, we can produce a model where selected transitions are more robust to adversarial transformation.

Xiao will present the paper at ICLR in New Orleans in May 2019.

A Pragmatic Introduction to Secure Multi-Party Computation

A Pragmatic Introduction to Secure Multi-Party Computation, co-authored with Vladimir Kolesnikov and Mike Rosulek, is now published by Now Publishers in their Foundations and Trends in Privacy and Security series.

You can download the book for free (we retain the copyright and are allowed to post an open version) from securecomputation.org, or buy an PDF version from the published for $260 (there is also a printed $99 version).

Secure multi-party computation (MPC) has evolved from a theoretical curiosity in the 1980s to a tool for building real systems today. Over the past decade, MPC has been one of the most active research areas in both theoretical and applied cryptography. This book introduces several important MPC protocols, and surveys methods for improving the efficiency of privacy-preserving applications built using MPC. Besides giving a broad overview of the field and the insights of the main constructions, we overview the most currently active areas of MPC research and aim to give readers insights into what problems are practically solvable using MPC today and how different threat models and assumptions impact the practicality of different approaches.