125 lines
8.6 KiB
Markdown
125 lines
8.6 KiB
Markdown
# Revisiting First Impressions: Apple, Parameters and Fuzzy Threshold PSI
|
||
|
||
Last week, Apple published more additional information regarding the parameterization of their new Fuzzy Threshold
|
||
PSI system in the form of a Security Threat Model@@^.
|
||
|
||
<p class="sidenote"><a href="https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf">Security Threat Model Review of Apple’s Child Safety Features</a></p>
|
||
|
||
Contained in the document are various answers to questions that the privacy community had been asking since the initial
|
||
announcement. It also contained information which answered several of my own questions, and in turn invalidated
|
||
a few of the assumptions I had made in a previous article@@^.
|
||
|
||
<p class="sidenote"><a href="/obfuscated_apples.html">Obfuscated Apples</a></p>
|
||
|
||
In particular, Apple have now stated the following:
|
||
|
||
* they claim the false acceptance rate of NeuralHash is 3 in 100M, but are assuming it is 1 in 1M. They have conducted
|
||
tests on both a dataset of 100M photos and on a dataset of 500K pornographic photos.
|
||
* the threshold $t$ they are choosing for the system is **30** with a future option to lower. They claim this is based
|
||
on taking the assumed false positive rate of NeuralHash and applying it to a assumed dataset the size of the largest iCloud photo library to obtain a probability of false reporting of 1 in a trillion.
|
||
|
||
One might ask why if the false acceptance rate of NeuralHash is so low then why take such precautions when estimating
|
||
$t$?
|
||
|
||
I will give Apple the benefit of the doubt here under the assumption that they really are attempting to only catch
|
||
prolific offenders.
|
||
|
||
Even still, I believe the most recent information by Apple still leaves several unanswered questions, and raises several
|
||
more.
|
||
|
||
## On NeuralHash
|
||
|
||
To put it as straightforward as possible, 100.5M photos isn't that large of a sample to compare a perceptual hashing
|
||
algorithm against, and the performance is directly related to the size of the comparison database (which we don't know).
|
||
|
||
Back in 2017 WhatsApp estimated that they were seeing 4.5 billion photos being uploaded to the platform per day@@^, while
|
||
we don't have figures for iCloud we can imagine, given Apples significant customer base, that it is on a similar order
|
||
of magnitude.
|
||
|
||
<p class="sidenote"><a href=">https://blog.whatsapp.com/connecting-one-billion-users-every-day">Connecting One Billion Users Every Day - Whatsapp Blog</a></p>
|
||
|
||
The types of the photos being compared also matter. We know nothing about the 100.5M photos that Apple tested against,
|
||
and only that a small 500K sample was pornographic in nature. While NeuralHash seems to have been designed as a generic
|
||
image comparison algorithm, that doesn't mean that it acts on all images uniformly.
|
||
|
||
## On the Thresholds
|
||
|
||
> Since this initial threshold contains a drastic safety margin reflecting a worst-case assumption about real-world performance, we may change the threshold after continued empirical evaluation of NeuralHash false positive rates – but the match threshold will never be lower than what is required to produce a one-in-one-trillion false positive rate for any given account - Security Threat Model Review of Apple’s Child Safety Features
|
||
|
||
Apples initial value of $t = {30}$ was chosen to include a **drastic safety margin**, but the threat model gives them
|
||
the explicit ability to change it in the future, but they promise the floor is still 1 in a trillion for "any given
|
||
account".
|
||
|
||
We still know very little about how $s$ will be chosen. We can assume it will be in the same magnitude as $t$ and that
|
||
as such the number of synthetics for each user will be relatively low compared to the total size of their image base.
|
||
|
||
Also given that $t$ is fixed across all accounts, we can be relatively sure that $s$ will also be fixed across all accounts,
|
||
with only the probability of choosing a synthetic match being varied on some unknown function.
|
||
|
||
Note that, if the probability of synthetic matches is too high, then the detection algorithm@@^ fails with high probability.
|
||
Requiring more matches, and an extended detection procedure.
|
||
|
||
<p class="sidenote">As an aside, if you are interested in playing with the Detectable Hash Function yourself [I wrote a toy version of it](https://git.openprivacy.ca/sarah/fuzzyhash)</p>
|
||
|
||
## Threat Model Expansions
|
||
|
||
The new threat model includes new jurisdictional protection for the database that were not present in the original
|
||
description - namely that the **intersection** of to ostensibly independent databases managed by different agencies
|
||
in different national jurisdictions will be used instead of a single database@@^ <span class="sidenote">(such as the one
|
||
run by NCMEC)</span>.
|
||
|
||
Additionally, Apple have now stated they will publish a "Knowledge Base" containing root hashes of the encrypted
|
||
database such that it can be confirmed that every device is comparing images to the same database. It is worth
|
||
noting that this claim is only as good as security researchers having access to proprietary Apple code.
|
||
|
||
That such a significant changes were made to the threat model a week after the initial publication is perhaps the
|
||
best testament to the idea, as Matthew Green put it:
|
||
|
||
> "But this illustrates something important: in building this system, the *only limiting principle* is how much heat Apple can tolerate before it changes its policies." - [Matthew Green](https://twitter.com/matthew_d_green/status/1426312939015901185)
|
||
|
||
|
||
## Revisiting First Impressions
|
||
|
||
I think the most important question I can ask of myself right now is that if Apple had put out all these documents on
|
||
day one, would they have been enough to quell the voice inside my head?
|
||
|
||
Assuming that Apple also verified the false acceptance rate of NeuralHash in a way more verifiable than :we tested
|
||
it on some images, it's all good, trust us!" then I think many of my technical objections to this system would have been
|
||
answered.
|
||
|
||
Not all of them though. I still, for example, think that the obfuscation in this system is fundamentally flawed from a practical perspective. And, I still think that the threat model as applied to malicious clients undermines the rest of the system@@^
|
||
|
||
<p class="sidenote">See: [A Closer Look at Fuzzy Threshold PSI](/a_closer_look_at_fuzzy_threshold_psi.html) for more details.</p>
|
||
|
||
## It's About the Principles
|
||
|
||
And, of course, none of that quells my moral objections to such a system.
|
||
|
||
You can wrap that surveillance in any number of layers of cryptography to try and make it palatable, the end result is the same.
|
||
|
||
Everyone on Apple's platform is treated as a potential criminal, subject to continual algorithmic surveillance without warrant or cause.
|
||
|
||
If Apple are successful in introducing this, how long do you think it will be before the same is expected of other providers? Before walled-garden prohibit apps that don't do it? Before it is enshrined in law?@@^ <span class="sidenote"><a href="https://twitter.com/SarahJamieLewis/status/1423403656733290496">Tweet</a></span>
|
||
|
||
How long do you think it will be before the database is expanded to include "terrorist" content"? "harmful-but-legal" content"? state-specific censorship?
|
||
|
||
This is not a slippery slope argument. For decades, we have seen governments and corporations push for ever more surveillance.
|
||
It is obvious how this system will be abused. It is obvious that Apple will not be in control of how it will be
|
||
abused for very long.
|
||
|
||
Accepting client side scanning onto personal devices **is** a rubicon moment, it signals a sea-change in how corporations
|
||
relate to their customers. Your personal device is no long "yours" in theory, nor in practice. It can, and will, be used
|
||
against you.
|
||
|
||
It is also abundantly clear that this is going to happen. While Apple has come under pressure, it has responded
|
||
by painting critics as "confused" (which, if there is any truth in that claim is due to their own lack of technical
|
||
specifications).
|
||
|
||
The media have likewise mostly followed Apples PR lead. While I am thankful that we have answers to some
|
||
questions that were asked, and that we seem to have caused Apple to "clarify"@@^ <span class="sidenote">(or, less subtly, change)</span> their own threat model, we have not seen the outpouring of objection that would have been necessary to
|
||
shut this down before it spread further.
|
||
|
||
The future of privacy on consumer devices is now forever changed. The impact might not be felt today or tomorrow, but in
|
||
the coming months please watch for the politicians (and sadly, the cryptographers) who argue that what can be done for
|
||
CSAM can be done for the next harm, and the next harm. Watch the EU and the UK, among others, declare such scanning mandatory,
|
||
and watch as your devices cease to work for you. |