/

Habeas: fun with secret court orders and web services

I spent some time chasing a shower thought and refreshing my (admittedly basic) experience with the Flask web development framework. In the process, I had quite a bit of fun building something simple but purposeful, and I would be so pleased if it were to inspire someone to be more optimistic about civil society. I’ve uploaded the source to Github here.

Here’s how I built this.

Motivation

The issuing of secret warrants for national security-related surveillance by the Foreign Intelligence Surveillance Court (FISC) in the USA remains controversial. Under the 1978 Foreign Intelligence Surveillance Act (FISA) and later amendments - particularly the 2008 FISA Amendments Act, domestic and foreign intelligence agencies in the United States have significant statutory permission for both warrantless and classified warrant-authorized surveillance of United States citizens. This poses a profound risk to Constitutional rights, particularly under the Fourth Amendment, and abuse of powers by bad actors. Indeed, Edward Snowden’s whistleblower revelations in 2013, regardless of one’s opinions of the man himself or whether constitutional rights are even worth protecting (!), were noteworthy in part because they confirmed that such risks were not merely theoretical, but realized at vast scale. Federal courts, including the Supreme Court, have consistently rejected legal challenges requesting that the government disclose the contents or even existence of secret warrants under FISA.

Let’s assume that Fourth Amendment protections are worth defending and intelligence agencies must be accountable for how they gather evidence to charge or prosecute citizens. (I certainly do.) For this exercise, let’s also acknowledge that there are good-faith reasons to keep details classified for at least some warrants until an individual is charged with a crime. (I’m less convinced of this, but willing to entertain it.) Is there a way to reconcile these?

Note: this also assumes a state apparatus that, if the public demands accountability, is willing to comply. If the public isn’t interested in the rule of law, or intelligence agencies are comfortable ignoring the law without facing consequences, then we have much bigger problems!

This is exactly the kind of problem that digital fingerprinting is intended to solve.

In short:

  • Suppose the FISC authorizes a secret warrant to surveil John Doe.
  • The FISC ought be obligated to publish a timestamped cryptographic hash corresponding to the contents of the warrant order. Other third parties must be able to - and are encouraged to - mirror the history of all published fingerprints.
  • When a suspect is charged with a crime, and evidence was collected under the secret warrant, the government is obligated to produce a warrant that matches a hash in the data store. In fact, publishing the fingerprint acts as “pre-registration” for evidence collection, and if the state cannot produce the court order text that generated the hash, the evidence should be prima facie inadmissable!
  • Alternatively, when a secret warrant is due to be declassified (e.g. Executive Order 13526, sec. 3.3), the government is obligated to produce a warrant that matches a hash in the data store. This includes at the maximum duration of 75 years since the hash’s timestamp.
  • The FISC is also forbidden to retroactively add or remove hashes from the data store; this can be enforced by including the previous hash value in the content to hash for each new warrant issued, similar to how a blockchain may guarantee immutability.
  • Legal recourse must be available to citizens if (a) the government cannot produce the warrant matching a relevant hash in when a suspect is charged or a hash ages into declassification, or (b) the FISC’s data store of all warrant hashes can be shown to have been retroactively changed.

Note that in an ideal world, a solution like this to balance security and accountability for classified court orders requires both a legal/political mechanism and a technological mechanism. I’m not a legal scholar or political scientist, so I can’t offer much about how to engineer the former. However, I thought prototyping the latter would be a fun exercise.

Challenge

A simple and robust system for publishing the existence of a secret warrant at the time of issuance, without revealing details of the warrant text, but allowing it to be validated retroactively.

To prototype this in earnest, the first step is to explicitly write up requirements.

Requirements

  • The system must allow an authorized user to add a record with:
    • A timestamp
    • A cryptographic hash representing (the timestamp, text contents of a warrant, and the previous record’s hash) in addition to salt and other best practices.
  • It must be impossible to recover the contents used to generate the hash from the hash itself.
  • Records are write-only.
  • Records are easy to export, and may be requested by any user with access e.g. through a publicly hosted API. (Good faith protections against DDoS are permissible.)
    • API endpoint to request all records, optionally starting from a provided timestamp
  • The system allows any user with access to upload a text file representing the contents of a warrant, and check whether it matches some record. If so, it will return the record to the requester.
  • The system must be robust and simple to deploy. A web application with a REST API is a good constraint for what this will look like.

This seems to be a straightforward case for cryptographic fingerprinting by hashing documents.

Of course, the devil is in the details.

  • Who is an “authorized user”? A judge or court reporter? Who has permissions to onboard one into the system?
  • Is there a standard for the text of a secret court order? (Not really. Very few examples of FISC surveillance warrants exist in the wild, and they are all heavily redacted.)

The fact that the problem domain is court documents has a neat implication. Ordinarily, when cryptographic hashes are used for authenticating some credentials, hash collision is a concern. That is, if a known document can be found that yields the same hash as a secret document, one could “impersonate” the secret document for verification purposes. While I’ll use a hash algorithm that makes this extremely unlikely, it’s virtually impossible that a secret court order will share a hash with content that also looks like a real court order! Even if a hash collision can be found, the “impostor” is likely to be total gibberish, and in practice couldn’t be used to mislead about the content of a preregistered secret warrant.

Prior art

Some quick online research suggests there has been interest in blockchains for criminal justice and law enforcement. (Some examples: 1 2 3 4 5) But from what I can tell, there isn’t much discussion on using either blockchain or simpler digital fingerprinting to confidentially attest to classified warrants for the public.

Implementation

While this would be a nice, self-contained opportunity to learn another programming language (Rust has looked interesting to me for awhile, and has a popular framework for REST APIs), I decided to start with tried-and-true Python for rapid prototyping. For a simple application like this, Flask was the obvious choice. Even better, I could get away with implementing this as purely a REST API web service - no need to serve web pages as all!

I shamelessly adapted the Flask official tutorial for most of the design. That includes relying on a SQLite database for persistence.

Database schema

I really only need three tables for basic functionality:

CREATE TABLE access_role (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  name TEXT UNIQUE NOT NULL
);


CREATE TABLE user (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  username TEXT UNIQUE NOT NULL,
  password TEXT NOT NULL,
  access_role_id INTEGER,
  FOREIGN KEY (access_role_id) REFERENCES access_role (id)
);


CREATE TABLE warrant (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  hashvalue TEXT NOT NULL,
  created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

In short, the service needs to know which users exist that are authorized to (a) add secret warrants to the database or (b) create other users who can, and the cryptographic fingerprints of the secret warrants. It doesn’t even need to know who uploaded which warrant!

User roles

In principle, only authorized users (e.g. judges, court reporters) should be able to contribute secret warrants. There must be some secure way to add them, though. I implemented this with a very simplistic role-based access control. Just two roles exist:

  • “Standard”, which can upload new warrants.
  • “Admin”, which can create new Standard users. Only one Admin user exists, and is created upon database initialization. The password for that user should be provided in a config file for the app instance.

Warrant document specification

This basically doesn’t exist for FISC orders. I took a detour performing some simple OCR on a few redacted examples of warrant orders, generally released due to FOIA requests. The examples had enough formatting inconsistencies across them that I gave up on trying to implement a standard OCR pipeline. Instead, the prototype assumes that a Standard user has a text file, or its contents as JSON data, to upload.

Fingerprinting

This didn’t require anything too exotic. First, the service concatenates the string representation of the current timestamp, the document’s content, and the previous record’s hash. It then hashes this with SHA3-256 and saves the hex digest in a new database record, along with that timestamp.

API

Again, the service only needs a few key endpoints.

  • auth/get_token: Given a username and password, create a JWT token. This gets passed in the header for any other request that requires authentication (that is, creating a new user or uploading a secret document).
  • auth/new_user: Create a new Standard user, given the desired username and password. The header must have a JWT token corresponding to the Admin user.
  • warrant/new: Submit a new secret document. It can be provided as a JSON request payload or as an uploaded file (max size is configurable on the server). The header must have a JWT token corresponding to a Standard user.
  • warrant/query: Retrieve the fingerprints and timestamps of recorded secret documents. By default, all records are returned, but there is a maximum page size configurable on the server. If there are too many secret documents for one page, the internal ID of the next page’s first record is included in the response. This can be passed as the next_id query parameter to retrieve the next page. In addition, records can be filtered by submission date with the start query parameter. Note that no JWT authentication is necessary - anyone in the public should be able to reach this endpoint.
  • warrant/verify: Test whether a provided document matches a fingerprint in the database. This also requires no authentication.

In addition to automated tests in the project, I also stood up the service locally and tested with Postman, which I’ve found super convenient in the past. I included a Postman collection in the project repo to simplify this for anyone who checks it out.

Where to find it

I committed the project to Github here. Hopefully someone finds it interesting or thought provoking! Ultimately, I had fun tinkering with this, and appreciated that I scoped it to be simple enough to quickly have a complete working prototype.