Browse Source

Fix push

trunk
Sarah Jamie Lewis 2 months ago
parent
commit
bf95da4be1
  1. 8
      header.html
  2. 9
      index.html
  3. 2
      index.md
  4. 131
      obfuscated_apples.html
  5. BIN
      obfuscated_apples.png
  6. 239
      posts/obfuscated_apples.md
  7. 3
      ssb
  8. 39
      styles.css

8
header.html

@ -5,6 +5,14 @@
<title>$TITLE | pseudorandom</title>
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:site" content="@sarahjamielewis" />
<meta name="twitter:creator" content="@sarahjamielewis" />
<meta property="og:url" content="https://pseudorandom.resistant.tech/$LINK.html" />
<meta property="og:description" content="$DESCRIPTION" />
<meta property="og:title" content="$TITLE" />
<meta name="twitter:image" content="https://pseudorandom.resistant.tech/$LINK.png">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" type="text/css" href="styles.css">

9
index.html

@ -5,6 +5,14 @@
<title>Welcome! | pseudorandom</title>
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:site" content="@sarahjamielewis" />
<meta name="twitter:creator" content="@sarahjamielewis" />
<meta property="og:url" content="https://pseudorandom.resistant.tech/index.html" />
<meta property="og:description" content="A site by Sarah Jamie Lewis" />
<meta property="og:title" content="Welcome!" />
<meta name="twitter:image" content="https://pseudorandom.resistant.tech/index.png">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" type="text/css" href="styles.css">
@ -30,6 +38,7 @@
<article>
<h1 id="welcome">Welcome!</h1>
<p>A site by Sarah Jamie Lewis</p>
</article>
<hr/>
<h2>

2
index.md

@ -1,2 +1,4 @@
# Welcome!
A site by Sarah Jamie Lewis

131
obfuscated_apples.html

@ -5,6 +5,14 @@
<title>Obfuscated Apples | pseudorandom</title>
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:site" content="@sarahjamielewis" />
<meta name="twitter:creator" content="@sarahjamielewis" />
<meta property="og:url" content="https://pseudorandom.resistant.tech/obfuscated_apples.html" />
<meta property="og:description" content="Generating noise in a way which is indistinguishable from real signal is a ridiculously hard problem. Obfuscation does not hide signal, it only adds noise. " />
<meta property="og:title" content="Obfuscated Apples" />
<meta name="twitter:image" content="https://pseudorandom.resistant.tech/obfuscated_apples.png">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" type="text/css" href="styles.css">
@ -30,88 +38,103 @@
<article>
<h1 id="obfuscated-apples">Obfuscated Apples</h1>
Generating noise in a way which is indistinguishable from real signal is a ridiculously hard problem. Obfuscation does not hide signal, it only adds noise.
<p>Generating noise in a way which is indistinguishable from real signal is a ridiculously hard problem. Obfuscation does not hide signal, it only adds noise.</p>
<p class="sidenote">
if you take anything away from this article please let it be this fact.
</p>
<p>Sadly, most people operate under the assumption that adding noise to a system is all that it takes to make the signal unrecoverable. This logic is very clearly in operation in Apple’s new proposal of on-device scanning <a class="sidenote" href="https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf">technical summary</a> which,among other things, proposes generating <em>synthetic</em> matches to hide the true number of <em>real</em> matches in the system.</p>
I want to take this opportunity to break down how this kind of obfuscation can be defeated even when not considering the fact that it is Apple themselves who are charged with generating and maintaining the safety parameters of the system.
<p>Sadly, most people operate under the assumption that adding noise to a system is all that it takes to make the signal unrecoverable. This logic is very clearly in operation in Apple’s new proposal for on-device scanning <a class="sidenote" href="https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf">technical summary</a> which, among other things, proposes generating <em>synthetic</em> matches to hide the true number of <em>real</em> matches in the system.</p>
I want to take this opportunity to break down how this kind of obfuscation can be defeated even when not considering the fact that it is <strong>Apple themselves who are charged with generating and maintaining the safety parameters of the system</strong>.
<p class="sidenote">
i.e. even if we treat the people who design and build this system as honest adversaries.
</p>
<h2 id="sketching-a-basic-scheme">Sketching a Basic Scheme</h2>
<p>For the sake of clarity I will omit the technical details of the private set intersection protocol, and the threshold scheme, and we will operate under the assumption that both are cryptographically secure. We will also, for now, assume that the database of images to compare is incorruptible.</p>
<p>At the heart of system is a (mostly) black box that contains a perceptual hash function that analyzes an images and spits out a hash, this hash is then compared against a database of known hashes and if a match is found the system reports <code>true</code> and otherwise reports <code>false</code>.</p>
<p>Throughout this article I will use <strong>match</strong> when talking about both true and false positives.</p>
<p>Without question, the server learns a match occurred on the phone during the PSI protocol.</p>
<p>For the sake of clarity I will omit the technical details of the private set intersection protocol, and the threshold scheme, and we will operate under the assumption that both are cryptographically secure. We will also assume that the database of images to compare is <em>incorruptible</em> <span class="sidenote">This is clearly not the case.</span>.</p>
<p>At the heart of system is a (mostly) black box that contains a <strong>perceptual</strong> hash function that analyzes an image and spits out a hash, this hash is then compared against a database of known hashes and if a match is found the system reports <code>true</code> and otherwise reports <code>false</code>. <span class="sidenote">As we will see later on, perceptual hashes are <strong>not</strong> cryptographic hashes.<span></p>
<p>Throughout this article I will use the term <strong>match</strong> when talking about both true and false positives, though I will mostly assume any matches are false positives.</p>
<p>According to documentation provided by Apple, the server learns any matches occurred on the phone during the PSI protocol.</p>
<blockquote class="sidenote">
“The output of PSI protocol on the server reveals whether there is a match or not” - Apple Technical Summary
</blockquote>
<p>When a certain threshold of real matches is reached, the server gains the ability to decrypt all real matches, a human reviews them, and a determination is made.</p>
As presented the system above has one major flaw
<p class="sidenote">
(besides the gross nature of co-opting a personal device as a surveillance system)
</p>
<p>: the server learns how many matches the device has reported prior to being able to decrypt those matches. This is obviously very important metadata in the context of the system and as such needs to be protected.</p>
<p>To do so Apple rely on the invocation of “Synthetic Vouchers” with the following property:</p>
<ul>
<li>“The probability that a device uploads a synthetic voucher instead of a real voucher for an image is calibrated to ensure the total number of synthetics is of the same order of magnitude as the threshold”</li>
</ul>
<h2 id="additional-metadata">Additional Metadata</h2>
<p>When a certain threshold of matches are reached, the server gains the ability to decrypt all data associated , a human reviews that data, and a determination is made.</p>
<p>As presented the system above has one major flaw <span class="sidenote">(besides the gross nature of co-opting a personal device as a surveillance system)</span>: the server learns how many matches the device has reported prior to being able to decrypt those matches.</p>
<p>This is obviously very important metadata in the context of the system and as such needs to be protected - if it is not then Apple, or someone who can compel Apple to release the data, can identify potential targets based on this metadata. <span class="sidenote">As we shall soon discuss targeting people in way would be highly irrational if your goal was to actually hunt people doing harm, but people are not rational actors.</span></p>
<p>To protect this data Apple relies on the invocation of so-called “Synthetic Vouchers” with the following property:</p>
<blockquote>
“The probability that a device uploads a synthetic voucher instead of a real voucher for an image is calibrated to ensure the total number of synthetics is of the same order of magnitude as the threshold”
</blockquote>
<h2 id="deriving-probabilities">Deriving Probabilities</h2>
<p>So, that is it right? Problem averted? Let’s not be too hasty…</p>
<p>From what we know so far there are a few interest parameters in this system, a threshold <span class="math inline"><em>t</em></span>, a probability of generating a synthetic match <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> and the probability of a false positive match <span class="math inline"><em>P</em>(<code>falsepositive</code>)</span>.</p>
<p>From what we know so far there are a few interesting parameters in this system that Apple must determine values for.</p>
<p>There is the threshold <span class="math inline"><em>t</em></span> of matches necessary to decrypt the data, there is the probability of a device generating a synthetic match <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> and there is the probability of a false positive match <span class="math inline"><em>P</em>(<code>falsepositive</code>)</span>.</p>
<p>We also know that Apple has constructed these parameters such that the probability of an account being flagged for human review (i.e. when number of matches <span class="math inline"><em>M</em>&gt;<em>t</em></span> is <span class="math inline"><em>P</em>(<em>f</em><em>l</em><em>a</em><em>g</em>) = 1<em>e</em><sup> − 12</sup></span> or one in one trillion.</p>
<blockquote class="sidenote">
“The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account.” - Apple Technical Summary
</blockquote>
<p>From these parameters we can start to derive some additional metadata that exists within the system:</p>
<h3 id="matches-over-threshold-without-decryption">Matches over Threshold without Decryption</h3>
<p>One of the most obvious sources of distinguishing information is built explicitly into the design. The threshold scheme as proposed requires <span class="math inline"><em>t</em></span> real matches in order to decrypt the inner envelopes containing the matches images.</p>
<p>Taking the number of observed matches, <span class="math inline"><em>o</em></span>, the system is able to derive a value <span class="math inline"><em>t</em>′ = <em>o</em>/<em>t</em></span> or, the number of observed matches divided by the threshold.</p>
<p>Every time a new match is observed and <span class="math inline"><em>t</em></span> is over the threshold the system learns additional information regarding the makeup of previous matches.</p>
<p>For example, if we set the <span class="math inline"><em>t</em> = 10</span>, then after the system observes <span class="math inline">10</span> matches (made up of a number of an unknown number of real or synthetic matches) then every additional match until decryption can be used to derive information about the previous matches. If the next match to arrive does not allow decryption then the server can derive that there are at least 2 synthetic matches in the bucket. This confidence increases with every observation that does not result in decryption.</p>
<p>Further we know that synthetic matches happen with probability <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> and as such the <strong>rate</strong> at which observed matches in combination with the above allows us to define a probability <span class="math inline"><em>P</em>(<code>match</code>)</span> given that the observation did not result in decryption, and the known probability of a device generating a synthetic value <span class="math inline"><em>P</em>(<code>synthetic</code>)</span>.</p>
<p><br /><span class="math display">$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{match}|\texttt{real}) \times P(\texttt{observation})}{P(\texttt{synthetic})}$$</span><br /></p>
<p>We know that all matches will result in an observation and so…</p>
<p><br /><span class="math display">$$P(\texttt{match}| \texttt{observation}) = \frac{1 \times P(\texttt{observation})}{P(\texttt{synthetic})}$$</span><br /></p>
<p>Or more simply:</p>
<p><br /><span class="math display">$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{observation})}{P(\texttt{synthetic})}$$</span><br /></p>
<p>Finally, we can state the the probability of an observation (say over the period of a day) is the probability of a real match on any given day plus the probability of a synthetic match on any given day:</p>
<p><br /><span class="math display">$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{synthetic})+P(\texttt{match})}{P(\texttt{synthetic})}$$</span><br /></p>
<p>Given that the probability of a synthetic match is defined by Apple, the only unknown in the system is the probability of a match.</p>
While this may start as an unknown
<p class="sidenote">
setting aside the fact that Apple already has enough data to derive this themselves
</p>
<p>, Apple quickly generates a large amount of data relating to when new observations are made. Since people are different in the ways these take and store photos and live in different parts of the world, the exact probability of them triggering a check is dependent on them.</p>
<p>It is also obvious at this point that if <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> is generated <em>naively</em> <span class="sidenote">(<em>naively</em> as in not account specific)</span> then <em>any</em> additional information known about a particular account e.g. the number of photos they are likely to store over a period of time is enough for <strong>anyone</strong> who knows <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> to derive <span class="math inline"><em>P</em>(<code>match</code>)</span>. <span class="sidenote">We will come back to this point shortly</span></p>
<p>The secrecy then is highly dependent on both Apple never deriving <span class="math inline"><em>P</em>(<code>observation</code>)</span> for themselves <em>and</em> on Apple generating a distinct <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> for each account. Or rather, the privacy of one of the most sensitive aspects of this system requires Apple both collecting no information on accounts, and also on Apple knowing enough about accounts to derive the parameters necessary to keep the information private.</p>
<hr/>
<p>It is actually worse than that through. Remember that Apple has stated that the actual probability of observing <span class="math inline"><em>M</em>&gt;<em>t</em></span> false positive matches is 1 in a trillion.</p>
<p>We can work backwards from that number to derive <span class="math inline"><em>P</em>(<code>falsepositive</code>)</span>:</p>
<p><br /><span class="math display">$$P(\texttt{flag}) = \sum_{\substack{x = t}}^T {T \choose x} \cdot P(\texttt{falsepositive})^x \cdot P(\texttt{falsepositive})^{100 - x} \approx 1\mathrm{e}^{-12}$$</span><br /></p>
<p>In order to finalize this we only need to make educated guesses about 2 parameters: the threshold value <span class="math inline"><em>t</em></span> and the total number of photos checked per year <span class="math inline"><em>T</em></span>. Apple throws out the number <span class="math inline"><em>t</em> = 10</span> in their technical summary, which seems like a good place to start.</p>
<p>Assuming an average account generates 3-4 pictures a day to be checked then <span class="math inline"><em>T</em> ≈ 1278</span> over a year. Plugging in those numbers, and we get <span class="math inline"><em>P</em>(<code>falsepositive</code>) ≈ 0.00035</span> or 1 in 2858. <span class="sidenote">yikes</span>.</p>
<p>We can actually work backwards from that number to derive <span class="math inline"><em>P</em>(<code>falsepositive</code>)</span>:</p>
<p><br /><span class="math display">$$P(\texttt{flag}) = \sum_{\substack{x = t}}^T {T \choose x} \cdot P(\texttt{falsepositive})^x \cdot P(\texttt{falsepositive})^{T - x} \approx 1\mathrm{e}^{-12}$$</span><br /></p>
<p>In order to finalize this we only need to make educated guesses about 2 parameters: the threshold value, <span class="math inline"><em>t</em></span>, and the total number of photos checked per year, <span class="math inline"><em>T</em></span>. Apple throws out the number <span class="math inline"><em>t</em> = 10</span> in their technical summary, which seems like a good place to start.</p>
<p>Assuming an average account generates 3-4 pictures a day to be checked then <span class="math inline"><em>T</em> ≈ 1278</span> over a year. Plugging in those numbers, and we get <span class="math inline"><em>P</em>(<code>falsepositive</code>) ≈ 0.00035</span> or <strong>1 in 2858</strong>.</p>
<p>Does that number have any relation to reality? There is evidence to suggest <span class="sidenote"><a href="https://arxiv.org/abs/2106.09820">Adversarial Detection Avoidance Attacks: Evaluating the robustness of perceptual hashing-based client-side scanning.</a> Shubham Jain, Ana-Maria Cretu, Yves-Alexandre de Montjoye</span> that the false acceptance rate for common perceptual hashing algorithms is between 0.001-0.01 for a database size of 500K.</p>
<p>That makes our guesstimate of 0.00035 an order of magnitude smaller than the most generous empirical estimate. We will be generous and assume Apple broke some new ground with NeuralHash and 0.00035 represents a major improvement in perceptual hashing false acceptance rates.</p>
<p>Given that we can go back and calculate the probability of observing, <span class="math inline"><em>P</em>(<code>match</code>)</span>, a match each day…</p>
<p><br /><span class="math display">$$P(\texttt{match}) = 1 - (( 1 - 0.00035)^3.5) \approx 0.001225 \approx \frac{1}{816}$$</span><br /></p>
<p><br /><span class="math display">$$P(\texttt{match}) = 1 - (( 1 - {0.00035})^{3.5}) \approx {0.001225} \approx \frac{1}{{816}}$$</span><br /></p>
<p>Or, a match once on average every 816 days for a person that only stores 3-4 photos per day.</p>
<p>Not everybody is every person though, if we applied the same <span class="math inline"><em>P</em>(<code>falsepositive</code>)</span> to a new parent who takes upwards of 50 photos per day, then their <span class="math inline"><em>P</em>(<code>match</code>)</span> is:</p>
<p><br /><span class="math display">$$P(\texttt{match}) = 1 - (( 1 - 0.00035)^50) \approx 0.01735 \approx \frac{1}{57}$$</span><br /></p>
<p><br /><span class="math display">$$P(\texttt{match}) = 1 - (( 1 - {0.00035})^{50}) \approx {0.01735} \approx \frac{1}{{57}}$$</span><br /></p>
<p>Or, a match on average every 57 days.</p>
<p>At this point I feel compelled to point out that these are <strong>average</strong> match probabilities. For the prolific photo taking parent who takes 18250 photos a year, the probability that they actually exceed the threshold in false matches is 6% <span class="sidenote">assuming <span class="math inline"><em>t</em></span> is 10</span>.</p>
<p>It is also worth mentioning that even though we ballparked <span class="math inline"><em>t</em></span> and <span class="math inline"><em>T</em></span> there are explicit constraints on what their values can be. If Apple generates a single <span class="math inline"><em>t</em></span> for all accounts, then <span class="math inline"><em>T</em></span> needs to be an approximation on the average number of photos an account stores per year. If Apple generates a different <span class="math inline"><em>t</em></span> value for every account, then it has enough information already to derive <span class="math inline"><em>P</em>(<code>observation</code>)</span> and break its own obfuscation.</p>
<hr/>
<p>Using what we now know we can revisit the server side operations show how the observer can calculate the probability of a real match given the probability of any q observation and the probability of a synthetic match.</p>
<p>Using what we now know we can assess the server side operations show how the observer can calculate the probability of a real match given the probability of any q observation and the probability of a synthetic match.</p>
<h3 id="the-probability-of-synthetic-matches">The Probability of Synthetic Matches</h3>
<p>Before we go any further, we should address one particular complication. Apple appears to suggest that images are probabilistically replaced with synthetic matches:</p>
<blockquote>
“the device occasionally produces synthetic vouchers for images as opposed to ones corresponding to their image”
</blockquote>
<p>Given that, the rate of real matches v.s. synthetic matches isn’t independent. <strong>Actual matches might be replaced by synthetic matches</strong>.</p>
<p><br /><span class="math display">$$P(\texttt{observation}) = (P(\texttt{match}) \cdot (1 - P(\texttt{synthetic}))) + ((1-P(\texttt{match})) \cdot P(\texttt{synthetic})) \vphantom{+ \frac{1}{1}}$$</span><br /></p>
<p>Or, to put it another way, the probability of a match being reported as a match is dependent on the probability it isn’t reported as a synthetic. Either way, <span class="math inline"><em>P</em>(<code>observation</code>|<code>match</code>) = 1</span></p>
<p>Further we can actually make a guess at the value of <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> under the assumption that it is calculated globally. Since Apple have stated that <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> is dependent on <span class="math inline"><em>t</em></span> and is designed such that it generates synthetic matches in the same order or magnitude as <span class="math inline"><em>t</em></span> we can derive <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> such that each device generates <span class="math inline"><em>t</em></span> synthetic matches a year on average.</p>
<p>Using our numbers from earlier we can place <span class="math inline"><em>P</em>(<code>synthetic</code>) ≈ 0.01</span> which would mean that over the course of a year, an average account storing 3-4 messages a day would have ~70% chance of generating 10 or more synthetic vouchers.</p>
<p>The exact value doesn’t really matter for our purposes. Any order of magnitude greater than <span class="math inline">0.01</span> results in too many synthetic matches, and any order of magnitude smaller results in too few.</p>
<h3 id="calculating-synthetic-probabilities">Calculating Synthetic Probabilities</h3>
<p>Given what we know about the probabilities in this system we can now piece together a server-side attribution attack to break the privacy provided by synthetic matches.</p>
<p><br /><span class="math display">$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{observation}|\texttt{match}) \times P(\texttt{match})}{P(\texttt{observation})}$$</span><br /></p>
<p>We know that all matches will result in an observation and so…</p>
<p><br /><span class="math display">$$P(\texttt{match}| \texttt{observation}) = \frac{1 \times P(\texttt{match})}{P(\texttt{observation})}$$</span><br /></p>
<p>Or more simply:</p>
<p><br /><span class="math display">$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{match})}{P(\texttt{observation})}$$</span><br /></p>
<p>Given that the probability of a synthetic match is defined by Apple, the only unknown in the system is the probability of a match.</p>
<p>We can now plug in our values from earlier. Remember, <strong>Apple has stated</strong> that the actual probability of observing <span class="math inline"><em>t</em></span> false positive matches, <span class="math inline"><em>P</em>(<code>flag</code>)</span>, is <strong>1 in a trillion</strong> and as such we have been able to derive approximate probabilities for false positives.</p>
<p>For an “average” account that stores 3-4 photos per day we know that <span class="math inline"><em>P</em>(<code>match</code>) =  ≈ 0.001225</span>, allowing Apple, who defines P() to calculate:</p>
<p><br /><span class="math display">$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{synthetic}) + 0.001225}{P(\texttt{synthetic})}$$</span><br /></p>
<p>For our prolific “parent” account that stores 50 photos per day we know that <span class="math inline"><em>P</em>(<code>observation</code>) ≈ 0.01735</span>, allowing Apple, who defines P() to calculate:</p>
<p><br /><span class="math display">$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{synthetic}) + 0.01735}{P(\texttt{synthetic})}$$</span><br /></p>
This clarifies our observation from earlier, if Apple define a global <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> then different accounts will naturally have different server-side distributions of observations, and these can be used to tighten the estimates of true matches.
<p><br /><span class="math display">$$P(\texttt{match}| \texttt{observation}) = \frac{(0.001225 \cdot 0.99)}{(0.001225 \cdot 0.99) + (0.998775 \cdot 0.01))} \approx 0.11 $$</span><br /></p>
<p>Given that we are aiming for ~10 synthetics over the course of a year, an 11% probability of any observation being a real match seems about the right level of indistinguishably.</p>
<p>But, what about our prolific “parent” account that stores 50 photos per day? We know that <span class="math inline"><em>P</em>(<code>match</code>) ≈ 0.01735</span>, allowing Apple, who defines <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> to calculate:</p>
<p><br /><span class="math display">$$P(\texttt{match}| \texttt{observation}) = \frac{(0.01735 \cdot 0.99)}{(0.01735 \cdot 0.99) + (0.98265 \cdot 0.01))} \approx 0.63 $$</span><br /></p>
<p>That is a 63% probability that any reported match is a real match and not a synthetic one!</p>
If Apple define a global <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> then different accounts will naturally have different server-side distributions of observations, and these can be used to tighten the estimates of true matches.
<p class="sidenote">
And, again, if Apple can define <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> on a per-account basis then they have <strong>more</strong> information to use when tightening these estimates
</p>
<p>The secrecy of this metadata is then is <strong>paradoxically dependent</strong> on both Apple never deriving <span class="math inline"><em>P</em>(<code>observation</code>)</span> for themselves <em>and</em> on Apple generating a distinct <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> for each account. Or rather, the privacy of one of the most sensitive aspects of this system requires Apple both collecting no information on accounts, and also on Apple knowing enough about accounts to derive the parameters necessary to keep the information private.</p>
<hr/>
<p>It is actually much worse than that though.</p>
<p>While priors may start as an unknown <span class="sidenote">(setting aside the fact that Apple already has enough data to derive this themselves)</span>, Apple quickly generates a large amount of data relating to when new observations are made. Since people are different in the ways these take and store photos and live in different parts of the world, the exact probability of them triggering a check is dependent on them.</p>
<p>There are also additional discriminating events in the system itself.</p>
<h3 id="matches-over-threshold-without-decryption">Matches over Threshold without Decryption</h3>
<p>One of the most obvious sources of discriminating information is built explicitly into the design. The threshold scheme as proposed requires <span class="math inline"><em>t</em></span> real matches in order to decrypt the inner envelopes containing the matches images.</p>
<p>As such. every time a new match is observed and <span class="math inline"><em>t</em></span> is over the threshold the system learns additional information regarding the makeup of previous matches.</p>
<p>To illustrate, if we set the <span class="math inline"><em>t</em> = 10</span>, then after the system observes <span class="math inline">10</span> matches (made up of a number of an unknown number of real or synthetic matches) then every additional match until decryption can be used to derive information about the previous matches. If the next match to arrive does not allow decryption then the server can derive that there are at least 2 synthetic matches in the bucket. This confidence increases with every observation that does not result in decryption.</p>
<p>Combining that with our bayesian estimates fom earlier we can see how an adversarial observer could update their estimates of <span class="math inline"><em>P</em>(<code>synthetic</code>)</span> based on the total number of observations, and the fact that no decryption has taken place.</p>
<hr/>
<h3 id="what-does-this-all-mean">What does this all mean?</h3>
<p>In this analysis we have deliberately left out other information that Apple, or someone who can compel Apple, may use to tighten these estimates e.g. derived from public social media feeds.</p>
<p>In reality, an adversary will have far greater access to auxiliary data about the environment and target sets than simply raw probabilities.</p>
<p>In that kind of environment, no amount of server-defined obfuscation is enough to protect the metadata that the server holds.</p>
<p>In this case that metadata is a rather controversial number i.e. <strong>the number of possible matches to illegal material detected on the device</strong>.</p>
<p>That is interesting metadata to countless entities including the law enforcement and intelligence agencies of multiple jurisdictions and states.</p>
<p>Even if we strictly limit the type of material that Apple is searching for, the high likelihood of false positive events combined with the ease at which Apple can likely distinguish true matching events from synthetic events <span class="sidenote"> (as worked through above)</span> should concern any potential subject of the system.</p>
<p><strong>Innocence is no defense against judgements made using derived metadata</strong>.</p>
</article>
<hr/>
<h2>

BIN
obfuscated_apples.png

After

Width: 800  |  Height: 418  |  Size: 2.2 KiB

239
posts/obfuscated_apples.md

@ -1,170 +1,229 @@
# Obfuscated Apples
Generating noise in a way which is indistinguishable from real signal is a ridiculously hard problem. Obfuscation does
not hide signal, it only adds noise. <p class="sidenote">if you take anything away from this article please let
Generating noise in a way which is indistinguishable from real signal is a ridiculously hard problem. Obfuscation does not hide signal, it only adds noise.
<p class="sidenote">if you take anything away from this article please let
it be this fact.</p>
Sadly, most people operate under the assumption that adding noise to a system is all that it takes to make the
signal unrecoverable. This logic is very clearly in operation in Apple's new proposal of on-device scanning
<a class="sidenote" href="https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf">technical summary</a> which,among other things, proposes generating *synthetic* matches to hide the true number of *real* matches in the system.
signal unrecoverable. This logic is very clearly in operation in Apple's new proposal for on-device scanning
<a class="sidenote" href="https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf">technical summary</a> which, among other things, proposes generating *synthetic* matches to hide the true number of *real* matches in the system.
I want to take this opportunity to break down how this kind of obfuscation can be defeated even when not considering
the fact that it is Apple themselves who are charged with generating and maintaining the safety parameters of the system.
the fact that it is **Apple themselves who are charged with generating and maintaining the safety parameters of the system**.
<p class="sidenote">i.e. even if we treat the people who design and build this system as honest adversaries.</p>
## Sketching a Basic Scheme
For the sake of clarity I will omit the technical details of the private set intersection protocol, and the threshold
scheme, and we will operate under the assumption that both are cryptographically secure. We will also, for now,
assume that the database of images to compare is incorruptible.
scheme, and we will operate under the assumption that both are cryptographically secure. We will also
assume that the database of images to compare is *incorruptible* <span class="sidenote">This is clearly not the
case.</span>.
At the heart of system is a (mostly) black box that contains a perceptual hash function that analyzes an images and
At the heart of system is a (mostly) black box that contains a **perceptual** hash function that analyzes an image and
spits out a hash, this hash is then compared against a database of known hashes and if a match is found the system
reports `true` and otherwise reports `false`.
reports `true` and otherwise reports `false`. <span class="sidenote">As we will see later on, perceptual hashes
are **not** cryptographic hashes.<span>
Throughout this article I will use **match** when talking about both true and false positives.
Throughout this article I will use the term **match** when talking about both true and false positives, though I will
mostly assume any matches are false positives.
Without question, the server learns a match occurred on the phone during the PSI protocol.
According to documentation provided by Apple, the server learns any matches occurred on the phone during the PSI protocol.
<blockquote class="sidenote">"The output of PSI protocol on the server reveals whether there is a match or not" - Apple Technical Summary</blockquote>
When a certain threshold of real matches is reached, the server gains the ability to decrypt all real matches, a human
reviews them, and a determination is made.
When a certain threshold of matches are reached, the server gains the ability to decrypt all data associated , a human
reviews that data, and a determination is made.
As presented the system above has one major flaw <span class="sidenote">(besides the gross nature of co-opting a personal device
as a surveillance system)</span>: the server learns how many matches the device has reported prior to being able to
decrypt those matches.
As presented the system above has one major flaw <p class="sidenote">(besides the gross nature of co-opting a personal device
as a surveillance system)</p>: the server learns how many matches the device has reported prior to being able to
decrypt those matches. This is obviously very important metadata in the context of the system and as such needs to be
protected.
This is obviously very important metadata in the context of the system and as such needs to be
protected - if it is not then Apple, or someone who can compel Apple to release the data, can identify potential
targets based on this metadata. <span class="sidenote">As we shall soon discuss targeting people in way would be highly
irrational if your goal was to actually hunt people doing harm, but people are not rational actors.</span>
To do so Apple rely on the invocation of "Synthetic Vouchers" with the following property:
To protect this data Apple relies on the invocation of so-called "Synthetic Vouchers" with the following property:
* "The probability that a device uploads a synthetic voucher instead of a real voucher for an image is calibrated to ensure the total number of synthetics is of the same order of magnitude as the threshold"
<blockquote>"The probability that a device uploads a synthetic voucher instead of a real voucher for an image is calibrated to ensure the total number of synthetics is of the same order of magnitude as the threshold"</blockquote>
## Additional Metadata
## Deriving Probabilities
So, that is it right? Problem averted? Let's not be too hasty...
From what we know so far there are a few interest parameters in this system, a threshold $t$, a probability of generating
a synthetic match $P(\texttt{synthetic})$ and the probability of a false positive match $P(\texttt{falsepositive})$.
From what we know so far there are a few interesting parameters in this system that Apple must determine
values for.
There is the threshold $t$ of matches necessary to decrypt the data, there is the probability of a device generating
a synthetic match $P(\texttt{synthetic})$ and there is the probability of a false positive match $P(\texttt{falsepositive})$.
We also know that Apple has constructed these parameters such that the probability of an account being flagged for human review (i.e. when number of matches $M > t$ is $P(flag) = 1\mathrm{e}^{-12}$ or one in one trillion.
<blockquote class="sidenote">"The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account." - Apple Technical Summary</blockquote>
From these parameters we can start to derive some additional metadata that exists within the system:
We can actually work backwards from that number to derive $P(\texttt{falsepositive})$:
### Matches over Threshold without Decryption
$$P(\texttt{flag}) = \sum_{\substack{x = t}}^T {T \choose x} \cdot P(\texttt{falsepositive})^x \cdot P(\texttt{falsepositive})^{T - x} \approx 1\mathrm{e}^{-12}$$
One of the most obvious sources of distinguishing information is built explicitly into the design. The threshold
scheme as proposed requires $t$ real matches in order to decrypt the inner envelopes containing the matches images.
In order to finalize this we only need to make educated guesses about 2 parameters: the threshold value, $t$, and the total
number of photos checked per year, $T$. Apple throws out the number $t = 10$ in their technical summary, which seems
like a good place to start.
Taking the number of observed matches, $o$, the system is able to derive a value $t' = o / t$ or, the number of
observed matches divided by the threshold.
Assuming an average account generates 3-4 pictures a day to be checked then $T \approx {1278}$ over a year. Plugging in those
numbers, and we get $P(\texttt{falsepositive}) \approx 0.00035$ or **1 in 2858**.
Every time a new match is observed and $t'$ is over the threshold the system learns additional information regarding
the makeup of previous matches.
Does that number have any relation to reality? There is evidence to suggest <span class="sidenote"><a href="https://arxiv.org/abs/2106.09820">Adversarial Detection Avoidance Attacks: Evaluating the robustness of perceptual hashing-based client-side scanning.</a> Shubham Jain, Ana-Maria Cretu, Yves-Alexandre de Montjoye</span> that the false acceptance rate for
common perceptual hashing algorithms is between 0.001-0.01 for a database size of 500K.
For example, if we set the $t = 10$, then after the system observes $10$ matches (made up of a number of an unknown number of real
or synthetic matches) then every additional match until decryption can be used to derive information about the previous
matches. If the next match to arrive does not allow decryption then the server can derive that there are at least 2 synthetic
matches in the bucket. This confidence increases with every observation that does not result in decryption.
That makes our guesstimate of 0.00035 an order of magnitude smaller than the most generous empirical estimate. We will
be generous and assume Apple broke some new ground with NeuralHash and 0.00035 represents a major improvement in perceptual hashing false acceptance rates.
Further we know that synthetic matches happen with probability $P(\texttt{synthetic})$ and as such the **rate** at which
observed matches in combination with the above allows us to define a probability $P(\texttt{match})$ given that
the observation did not result in decryption, and the known probability of a device generating a synthetic value $P(\texttt{synthetic})$.
Given that we can go back and calculate the probability of observing, $P(\texttt{match})$, a match each day...
$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{match}|\texttt{real}) \times P(\texttt{observation})}{P(\texttt{synthetic})}$$
$$P(\texttt{match}) = 1 - (( 1 - {0.00035})^{3.5}) \approx {0.001225} \approx \frac{1}{{816}}$$
We know that all matches will result in an observation and so...
Or, a match once on average every 816 days for a person that only stores 3-4 photos per day.
$$P(\texttt{match}| \texttt{observation}) = \frac{1 \times P(\texttt{observation})}{P(\texttt{synthetic})}$$
Not everybody is every person though, if we applied the same $P(\texttt{falsepositive})$ to a new parent who takes upwards
of 50 photos per day, then their $P(\texttt{match})$ is:
Or more simply:
$$P(\texttt{match}) = 1 - (( 1 - {0.00035})^{50}) \approx {0.01735} \approx \frac{1}{{57}}$$
$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{observation})}{P(\texttt{synthetic})}$$
Or, a match on average every 57 days.
Finally, we can state the the probability of an observation (say over the period of a day) is the probability of a
real match on any given day plus the probability of a synthetic match on any given day:
At this point I feel compelled to point out that these are **average** match probabilities. For the prolific photo
taking parent who takes 18250 photos a year, the probability that they actually exceed the threshold in false matches
is 6% <span class="sidenote">assuming $t$ is 10</span>.
$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{synthetic})+P(\texttt{match})}{P(\texttt{synthetic})}$$
It is also worth mentioning that even though we ballparked $t$ and $T$ there are explicit constraints on what their
values can be. If Apple generates a single $t$ for all accounts, then $T$ needs to be an approximation on the average
number of photos an account stores per year. If Apple generates a different $t$ value for every account, then it has
enough information already to derive $P(\texttt{observation})$ and break its own obfuscation.
Given that the probability of a synthetic match is defined by Apple, the only unknown in the system is the probability
of a match.
<hr/>
While this may start as an unknown <p class="sidenote">setting aside the fact that Apple already has enough data to derive this themselves</p>, Apple quickly generates a large amount of data relating to when new observations
are made. Since people are different in the ways these take and store photos and live in different parts of the world,
the exact probability of them triggering a check is dependent on them.
Using what we now know we can assess the server side operations show how the observer can calculate the probability of a real
match given the probability of any q observation and the probability of a synthetic match.
It is also obvious at this point that if $P(\texttt{synthetic})$ is generated *naively* <span class="sidenote">(*naively* as in
not account specific)</span> then *any* additional information known about a particular account e.g. the number of photos
they are likely to store over a period of time is enough for **anyone** who knows $P(\texttt{synthetic})$ to derive
$P(\texttt{match})$. <span class="sidenote">We will come back to this point shortly</span>
### The Probability of Synthetic Matches
The secrecy then is highly dependent on both Apple never deriving $P(\texttt{observation})$ for themselves *and* on Apple
generating a distinct $P(\texttt{synthetic})$ for each account. Or rather, the privacy of one of the most sensitive aspects
of this system requires Apple both collecting no information on accounts, and also on Apple knowing enough about accounts
to derive the parameters necessary to keep the information private.
Before we go any further, we should address one particular complication. Apple appears to suggest that
images are probabilistically replaced with synthetic matches:
<hr/>
<blockquote>"the device occasionally produces synthetic vouchers for images as opposed to ones corresponding to their image"</blockquote>
It is actually worse than that through. Remember that Apple has stated that the actual probability of observing $M > t$
false positive matches is 1 in a trillion.
Given that, the rate of real matches v.s. synthetic matches isn't independent. **Actual matches might be replaced
by synthetic matches**.
We can work backwards from that number to derive $P(\texttt{falsepositive})$:
$$P(\texttt{observation}) = (P(\texttt{match}) \cdot (1 - P(\texttt{synthetic}))) + ((1-P(\texttt{match})) \cdot P(\texttt{synthetic})) \vphantom{+ \frac{1}{1}}$$
$$P(\texttt{flag}) = \sum_{\substack{x = t}}^T {T \choose x} \cdot P(\texttt{falsepositive})^x \cdot P(\texttt{falsepositive})^{100 - x} \approx 1\mathrm{e}^{-12}$$
Or, to put it another way, the probability of a match being reported as a match is dependent on the probability
it isn't reported as a synthetic. Either way, $P(\texttt{observation}|\texttt{match}) = 1$
In order to finalize this we only need to make educated guesses about 2 parameters: the threshold value $t$ and the total
number of photos checked per year $T$. Apple throws out the number $t = 10$ in their technical summary, which seems
like a good place to start.
Further we can actually make a guess at the value of $P(\texttt{synthetic})$ under the assumption that it is calculated
globally. Since Apple have stated that $P(\texttt{synthetic})$ is dependent on $t$ and is designed such that it generates
synthetic matches in the same order or magnitude as $t$ we can derive $P(\texttt{synthetic})$ such that each device
generates $t$ synthetic matches a year on average.
Assuming an average account generates 3-4 pictures a day to be checked then $T \approx 1278$ over a year. Plugging in those
numbers, and we get $P(\texttt{falsepositive}) \approx 0.00035$ or 1 in 2858. <span class="sidenote">yikes</span>.
Using our numbers from earlier we can place $P(\texttt{synthetic}) \approx {0.01}$ which would mean that over the course
of a year, an average account storing 3-4 messages a day would have ~70% chance of generating 10 or more synthetic
vouchers.
Given that we can go back and calculate the probability of observing, $P(\texttt{match})$, a match each day...
The exact value doesn't really matter for our purposes. Any order of magnitude greater than ${0.01}$ results in too many synthetic
matches, and any order of magnitude smaller results in too few.
$$P(\texttt{match}) = 1 - (( 1 - 0.00035)^3.5) \approx 0.001225 \approx \frac{1}{816}$$
### Calculating Synthetic Probabilities
Or, a match once on average every 816 days for a person that only stores 3-4 photos per day.
Given what we know about the probabilities in this system we can now piece together a server-side attribution attack
to break the privacy provided by synthetic matches.
Not everybody is every person though, if we applied the same $P(\texttt{falsepositive})$ to a new parent who takes upwards
of 50 photos per day, then their $P(\texttt{match})$ is:
$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{observation}|\texttt{match}) \times P(\texttt{match})}{P(\texttt{observation})}$$
$$P(\texttt{match}) = 1 - (( 1 - 0.00035)^50) \approx 0.01735 \approx \frac{1}{57}$$
We know that all matches will result in an observation and so...
Or, a match on average every 57 days.
$$P(\texttt{match}| \texttt{observation}) = \frac{1 \times P(\texttt{match})}{P(\texttt{observation})}$$
At this point I feel compelled to point out that these are **average** match probabilities. For the prolific photo
taking parent who takes 18250 photos a year, the probability that they actually exceed the threshold in false matches
is 6% <span class="sidenote">assuming $t$ is 10</span>.
Or more simply:
It is also worth mentioning that even though we ballparked $t$ and $T$ there are explicit constraints on what their
values can be. If Apple generates a single $t$ for all accounts, then $T$ needs to be an approximation on the average
number of photos an account stores per year. If Apple generates a different $t$ value for every account, then it has
enough information already to derive $P(\texttt{observation})$ and break its own obfuscation.
$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{match})}{P(\texttt{observation})}$$
<hr/>
Given that the probability of a synthetic match is defined by Apple, the only unknown in the system is the probability
of a match.
Using what we now know we can revisit the server side operations show how the observer can calculate the probability of a real
match given the probability of any q observation and the probability of a synthetic match.
We can now plug in our values from earlier. Remember, **Apple has stated** that the actual probability of observing $t$
false positive matches, $P(\texttt{flag})$, is **1 in a trillion** and as such we have been able to derive approximate
probabilities for false positives.
For an "average" account that stores 3-4 photos per day we know that $P(\texttt{match}) = \approx 0.001225$, allowing
Apple, who defines P(\texttt{synthetic}) to calculate:
$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{synthetic}) + 0.001225}{P(\texttt{synthetic})}$$
$$P(\texttt{match}| \texttt{observation}) = \frac{(0.001225 \cdot 0.99)}{(0.001225 \cdot 0.99) + (0.998775 \cdot 0.01))} \approx 0.11 $$
For our prolific "parent" account that stores 50 photos per day we know that $P(\texttt{observation}) \approx 0.01735$, allowing
Apple, who defines P(\texttt{synthetic}) to calculate:
Given that we are aiming for ~10 synthetics over the course of a year, an 11% probability of any observation being a
real match seems about the right level of indistinguishably.
But, what about our prolific "parent" account that stores 50 photos per day? We know that $P(\texttt{match}) \approx 0.01735$, allowing Apple, who defines $P(\texttt{synthetic})$ to calculate:
$$P(\texttt{match}| \texttt{observation}) = \frac{(0.01735 \cdot 0.99)}{(0.01735 \cdot 0.99) + (0.98265 \cdot 0.01))} \approx 0.63 $$
$$P(\texttt{match}| \texttt{observation}) = \frac{P(\texttt{synthetic}) + 0.01735}{P(\texttt{synthetic})}$$
That is a 63% probability that any reported match is a real match and not a synthetic one!
This clarifies our observation from earlier, if Apple define a global $P(\texttt{synthetic})$ then different accounts will
If Apple define a global $P(\texttt{synthetic})$ then different accounts will
naturally have different server-side distributions of observations, and these can be used to tighten the estimates of
true matches. <p class="sidenote">And, again, if Apple can define $P(\texttt{synthetic})$ on a per-account basis then
they have **more** information to use when tightening these estimates</p>
The secrecy of this metadata is then is **paradoxically dependent** on both Apple never deriving $P(\texttt{observation})$ for themselves *and* on Apple generating a distinct $P(\texttt{synthetic})$ for each account. Or rather, the privacy of one of the most sensitive aspects of this system requires Apple both collecting no information on accounts, and also on Apple knowing enough about accounts to derive the parameters necessary to keep the information private.
<hr/>
It is actually much worse than that though.
While priors may start as an unknown <span class="sidenote">(setting aside the fact that Apple already has enough data to derive this themselves)</span>, Apple quickly generates a large amount of data relating to when new observations
are made. Since people are different in the ways these take and store photos and live in different parts of the world,
the exact probability of them triggering a check is dependent on them.
There are also additional discriminating events in the system itself.
### Matches over Threshold without Decryption
One of the most obvious sources of discriminating information is built explicitly into the design. The threshold
scheme as proposed requires $t$ real matches in order to decrypt the inner envelopes containing the matches images.
As such. every time a new match is observed and $t'$ is over the threshold the system learns additional information regarding
the makeup of previous matches.
To illustrate, if we set the $t = 10$, then after the system observes $10$ matches (made up of a number of an unknown number of real or synthetic matches) then every additional match until decryption can be used to derive information about the previous
matches. If the next match to arrive does not allow decryption then the server can derive that there are at least 2 synthetic
matches in the bucket. This confidence increases with every observation that does not result in decryption.
Combining that with our bayesian estimates fom earlier we can see how an adversarial observer could update their
estimates of $P(\texttt{synthetic})$ based on the total number of observations, and the fact that no decryption has taken
place.
<hr/>
### What does this all mean?
In this analysis we have deliberately left out other information that Apple, or someone who can compel Apple, may use
to tighten these estimates e.g. derived from public social media feeds.
In reality, an adversary will have far greater access to auxiliary data about the environment and target sets than simply
raw probabilities.
In that kind of environment, no amount of server-defined obfuscation is enough to protect the metadata that the server
holds.
In this case that metadata is a rather controversial number i.e. **the number of possible matches to illegal material detected
on the device**.
That is interesting metadata to countless entities including the law enforcement and intelligence agencies of multiple
jurisdictions and states.
Even if we strictly limit the type of material that Apple is searching for, the high likelihood of false positive events
combined with the ease at which Apple can likely distinguish true matching events from synthetic events <span class="sidenote">
(as worked through above)</span> should concern any potential subject of the system.
**Innocence is no defense against judgements made using derived metadata**.

3
ssb

@ -86,8 +86,9 @@ function make_html_files
file_base=`basename $md_file .md`
output_file="$OUTPUT_DIR/$file_base.html"
post_title=`grep -m 1 "^# .*" $md_file | cut -c 3-`
description=`head -3 $md_file | tail -1`
append_posts_list $posts | cat $md_file - | $MD_RENDERER > $output_file
cat $HEADER_PATH $output_file $FOOTER_PATH | sed -e "s/\$TITLE/$post_title/g" | tee $output_file
cat $HEADER_PATH $output_file $FOOTER_PATH | sed -e "s/\$TITLE/$post_title/g" | sed -e "s/\$LINK/$file_base/g" | sed -e "s/\$DESCRIPTION/$description/g" | tee $output_file
done
}

39
styles.css

@ -13,10 +13,12 @@ body {
padding: 1.5em;
}
article {
margin-left: 0px;
margin-right: auto;
max-width: 60%;
max-width: 60%;
}
.sidenote {
@ -30,6 +32,39 @@ article {
transform: translateY(-100%);
}
@media only screen and (max-width: 796px) {
body {
font-family: Spectral;
background: #111;
color: #eee;
width:95%;
max-width: 100%;
padding:0.5em;
margin: 0px 0px;
}
article {
max-width: 100%;
}
.sidenote {
float:none;
transform: translateY(0%);
color: #999;
font-style: italic;
margin-right: 0;
font-size: 0.7em;
}
.display .katex {
font-size: 0.7em;
overflow: scroll;
}
}
header { font-size: 1.1em; }
footer { border-top: 0.25em solid #eee; margin-top: 1.5em; text-align: right; }
nav { margin-bottom: 1.5em; padding-bottom: 0.5em; }
@ -74,7 +109,7 @@ td {
}
blockquote {
padding: 0 1em;
color: #646464;
color: #ddd;
border-left: 0.25em solid #bfbfbf;
}
pre {

Loading…
Cancel
Save