Our research seeks to empower individuals and organizations to
control how their data is used. We use techniques from cryptography,
programming languages, machine learning, operating systems, and other
areas to both understand and improve the security of computing as
practiced today, and as envisioned in the future.
Everyone is welcome at our research group meetings
(most Fridays at 11am, but join the slack group for announcements). To
get announcements, join our Slack Group (any
@virginia.edu email address can join themsleves, or email me
to request an invitation).
We have posted a paper by Bargav Jayaraman and myself on When Relaxations Go Bad: “Differentially-Private” Machine Learning (code available at https://github.com/bargavj/EvaluatingDPML).
Differential privacy is becoming a standard notion for performing
privacy-preserving machine learning over sensitive data. It provides
formal guarantees, in terms of the privacy budget, ε, on how
much information about individual training records is leaked by the
While the privacy budget is directly correlated to the privacy
leakage, the calibration of the privacy budget is not well
understood. As a result, many existing works on privacy-preserving
machine learning select large values of ϵ in order to get acceptable
utility of the model, with little understanding of the concrete impact
of such choices on meaningful privacy. Moreover, in scenarios where
iterative learning procedures are used which require privacy
guarantees for each iteration, relaxed definitions of differential
privacy are often used which further tradeoff privacy for better
We evaluated the impacts of these choices on privacy in experiments
with logistic regression and neural network models, quantifying the
privacy leakage in terms of advantage of the adversary performing
inference attacks and by analyzing the number of members at risk for
Accuracy Loss as Privacy Decreases
(CIFAR-100, neural network model)
(Yeom et al.’s Membership Inference Attack)
Our main findings are that current mechanisms for differential privacy
for machine learning rarely offer acceptable utility-privacy
tradeoffs: settings that provide limited accuracy loss provide little
effective privacy, and settings that provide strong privacy result in
The table below shows the number of individuals, out of 10,000 members
in the training set, exposed by a membership inference attack, given
tolerance for false positives of 1% or 5% (and assuming a priori
prevalence of 50% members). The key observations is that all the
relaxtions provide lower utility (more accuracy loss) than naïve
composition for comparable privacy leakage, as measured by the number
of actual members exposed in a test dataset. Further, none of the
methods provide both acceptable utility and meaningful privacy —
at a high level, either nothing is learned from the training data, or
some sensitive data is exposed. (See the
paper for more details and
||No privacy noise added.
Bargav Jayaraman talked about this work at the DC-Area Anonymity, Privacy, and Security Seminar (25 February 2019) at the University of Maryland:
Paper: When Relaxations Go Bad: “Differentially-Private” Machine Learning
New Electronics has an article that includes my Deep Learning and Security Workshop talk: Deep fools, 21 January 2019.
A better version of the image Mainuddin Jonas produced that they use
(which they screenshot from the talk video) is below:
My course for Spring 2019 is Markets, Mechanisms,
Machines, cross-listed as cs4501/econ4559
and co-taught with Denis
Nekipelov. The course will explore
interesting connections between economics and computer science.
My qualifications for being listed as instructor for a 4000-level
Economics course are limited to taking an introductory microeconomics
course my first year as an undergraduate.
Its good to finally get a chance to redeem myself for giving up on
Economics 28 years ago!
Xiao Zhang and my paper on Cost-Sensitive Robustness against Adversarial Examples has been accepted to ICLR 2019.
Several recent works have developed methods for training classifiers
that are certifiably robust against norm-bounded adversarial
perturbations. However, these methods assume that all the adversarial
transformations provide equal value for adversaries, which is seldom
the case in real-world applications. We advocate for cost-sensitive
robustness as the criteria for measuring the classifier’s performance
for specific tasks. We encode the potential harm of different
adversarial transformations in a cost matrix, and propose a general
objective function to adapt the robust training method of Wong &
Kolter (2018) to optimize for cost-sensitive robustness. Our
experiments on simple MNIST and CIFAR10 models and a variety of cost
matrices show that the proposed approach can produce models with
substantially reduced cost-sensitive robust error, while maintaining
This shows the results of cost-sensitive robustness training to protect the odd classes. By incorporating a cost matrix in the loss function for robustness training, we can produce a model where selected transitions are more robust to adversarial transformation.
Xiao will present the paper at ICLR in New Orleans in May 2019.
A Pragmatic Introduction to Secure Multi-Party Computation,
co-authored with Vladimir Kolesnikov and Mike Rosulek, is now
published by Now Publishers in their
Foundations and Trends in Privacy and Security series.
You can download the book for free (we retain the copyright and are
allowed to post an open version) from
securecomputation.org, or buy an PDF
version from the published for $260 (there is also a printed $99
Secure multi-party computation (MPC) has evolved from a theoretical
curiosity in the 1980s to a tool for building real systems today. Over
the past decade, MPC has been one of the most active research areas in
both theoretical and applied cryptography. This book introduces
several important MPC protocols, and surveys methods for improving the
efficiency of privacy-preserving applications built using MPC. Besides
giving a broad overview of the field and the insights of the main
constructions, we overview the most currently active areas of MPC
research and aim to give readers insights into what problems are
practically solvable using MPC today and how different threat models
and assumptions impact the practicality of different approaches.