pseudorandom/posts/ftpsi-parameters.md

# Revisiting First Impressions: Apple, Parameters and Fuzzy Threshold PSI

Last week, Apple published more additional information regarding the parameterization of their new Fuzzy Threshold
PSI system in the form of a Security Threat Model@@^.

<p class="sidenote"><a href="https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf">Security Threat Model Review of Apple’s Child Safety Features</a></p>

Contained in the document are various answers to questions that the privacy community had been asking since the initial
announcement. It also contained information which answered several of my own questions, and in turn invalidated
a few of the assumptions I had made in a previous article@@^.

<p class="sidenote"><a href="/obfuscated_apples.html">Obfuscated Apples</a></p>

In particular, Apple have now stated the following:

* they claim the false acceptance rate of NeuralHash is 3 in 100M, but are assuming it is 1 in 1M. They have conducted
tests on both a dataset of 100M photos and on a dataset of 500K pornographic photos. 
* the threshold $t$ they are choosing for the system is **30** with a future option to lower. They claim this is based
on taking the assumed false positive rate of NeuralHash and applying it to a assumed dataset the size of the largest iCloud photo library to obtain a probability of false reporting of 1 in a trillion.
  
One might ask why if the false acceptance rate of NeuralHash is so low then why take such precautions when estimating
$t$?

I will give Apple the benefit of the doubt here under the assumption that they really are attempting to only catch
prolific offenders.

Even still, I believe the most recent information by Apple still leaves several unanswered questions, and raises several
more.

## On NeuralHash 

To put it as straightforward as possible, 100.5M photos isn't that large of a sample to compare a perceptual hashing
algorithm against, and the performance is directly related to the size of the comparison database (which we don't know).

Back in 2017 WhatsApp estimated that they were seeing 4.5 billion photos being uploaded to the platform per day@@^, while
we don't have figures for iCloud we can imagine, given Apples significant customer base, that it is on a similar order
of magnitude.

<p class="sidenote"><a href=">https://blog.whatsapp.com/connecting-one-billion-users-every-day">Connecting One Billion Users Every Day - Whatsapp Blog</a></p>

The types of the photos being compared also matter. We know nothing about the 100.5M photos that Apple tested against,
and only that a small 500K sample was pornographic in nature. While NeuralHash seems to have been designed as a generic
image comparison algorithm, that doesn't mean that it acts on all images uniformly.

## On the Thresholds

> Since this initial threshold contains a drastic safety margin reflecting a worst-case assumption about real-world performance, we may change the threshold after continued empirical evaluation of NeuralHash false positive rates – but the match threshold will never be lower than what is required to produce a one-in-one-trillion false positive rate for any given account - Security Threat Model Review of Apple’s Child Safety Features

Apples initial value of $t = {30}$ was chosen to include a **drastic safety margin**, but the threat model gives them
the explicit ability to change it in the future, but they promise the floor is still 1 in a trillion for "any given
account".

We still know very little about how $s$ will be chosen. We can assume it will be in the same magnitude as $t$ and that
as such the number of synthetics for each user will be relatively low compared to the total size of their image base.

Also given that $t$ is fixed across all accounts, we can be relatively sure that $s$ will also be fixed across all accounts,
with only the probability of choosing a synthetic match being varied on some unknown function.

Note that, if the probability of synthetic matches is too high, then the detection algorithm@@^ fails with high probability.
Requiring more matches, and an extended detection procedure. 

<p class="sidenote">As an aside, if you are interested in playing with the Detectable Hash Function yourself [I wrote a toy version of it](https://git.openprivacy.ca/sarah/fuzzyhash)</p>

## Threat Model Expansions

The new threat model includes new jurisdictional protection for the database that were not present in the original
description - namely that the **intersection** of to ostensibly independent databases managed by different agencies
in different national jurisdictions will be used instead of a single database@@^ <span class="sidenote">(such as the one
run by NCMEC)</span>.

Additionally, Apple have now stated they will publish a "Knowledge Base" containing root hashes of the encrypted
database such that it can be confirmed that every device is comparing images to the same database. It is worth
noting that this claim is only as good as security researchers having access to proprietary Apple code.

That such a significant changes were made to the threat model a week after the initial publication is perhaps the
best testament to the idea, as Matthew Green put it:

> "But this illustrates something important: in building this system, the *only limiting principle* is how much heat Apple can tolerate before it changes its policies." -  [Matthew Green](https://twitter.com/matthew_d_green/status/1426312939015901185) 


## Revisiting First Impressions

I think the most important question I can ask of myself right now is that if Apple had put out all these documents on
day one, would they have been enough to quell the voice inside my head?

Assuming that Apple also verified the false acceptance rate of NeuralHash in a way more verifiable than :we tested
it on some images, it's all good, trust us!" then I think many of my technical objections to this system would have been
answered.

Not all of them though. I still, for example, think that the obfuscation in this system is fundamentally flawed from a practical perspective. And, I still think that the threat model as applied to malicious clients undermines the rest of the system@@^

<p class="sidenote">See: [A Closer Look at Fuzzy Threshold PSI](/a_closer_look_at_fuzzy_threshold_psi.html) for more details.</p>

## It's About the Principles

And, of course, none of that quells my moral objections to such a system.

You can wrap that surveillance in any number of layers of cryptography to try and make it palatable, the end result is the same.

Everyone on Apple's platform is treated as a potential criminal, subject to continual algorithmic surveillance without warrant or cause.

If Apple are successful in introducing this, how long do you think it will be before the same is expected of other providers? Before walled-garden prohibit apps that don't do it? Before it is enshrined in law?@@^ <span class="sidenote"><a href="https://twitter.com/SarahJamieLewis/status/1423403656733290496">Tweet</a></span>

How long do you think it will be before the database is expanded to include "terrorist" content"? "harmful-but-legal" content"? state-specific censorship?

This is not a slippery slope argument. For decades, we have seen governments and corporations push for ever more surveillance.
It is obvious how this system will be abused. It is obvious that Apple will not be in control of how it will be 
abused for very long.

Accepting client side scanning onto personal devices **is** a rubicon moment, it signals a sea-change in how corporations
relate to their customers. Your personal device is no long "yours" in theory, nor in practice. It can, and will, be used
against you.

It is also abundantly clear that this is going to happen. While Apple has come under pressure, it has responded
by painting critics as "confused" (which, if there is any truth in that claim is due to their own lack of technical
specifications).

The media have likewise mostly followed Apples PR lead. While I am thankful that we have answers to some
questions that were asked, and that we seem to have caused Apple to "clarify"@@^ <span class="sidenote">(or, less subtly, change)</span> their own threat model, we have not seen the outpouring of objection that would have been necessary to
shut this down before it spread further.

The future of privacy on consumer devices is now forever changed. The impact might not be felt today or tomorrow, but in
the coming months please watch for the politicians (and sadly, the cryptographers) who argue that what can be done for 
CSAM can be done for the next harm, and the next harm. Watch the EU and the UK, among others, declare such scanning mandatory,
and watch as your devices cease to work for you.