Information Architecture

This repository handles stage one only: crawling public government glossaries and emitting source-faithful JSON. It does not include downstream transformation code.

The key idea is to split parsing into two stages. They're analogous to the lexer + parser pair in a compiler. Dividing the parsing into two pieces allows each to be simpler.

Stage One: Crawling & Emitting Source-Faithful JSON

• Extract HTML, PDFs, etc. from government sources
• Output JSON that preserves original structure
• Located in this repo (public_law/spiders, public_law/parsers)

The first stage (this repo) crawls and converts original sources to JSON. The actual schema of the JSON mirrors the original content as much as possible. And so, each type of original source will have very different looking JSON. But, being JSON (instead of PDF, HTML, etc.) they're all easily read by the next stage. The second stage can focus on converting the source schema to a particular app's needs.

📌 Design principle: JSON should reflect the structure of the source, not a normalized internal model. This preserves auditability, encourages reuse, and avoids premature flattening.

Stage Two: Transforming JSON into App-Specific Models

• Consumes JSON from Stage One
• Maps content to e.g., Postgres rows, HTML cards, Lucene docs, etc.
• Not included in this repo

The second stage transforms the JSON to an app's internal representation. That code is outside the scope of this repo because many apps by different developers can use the source data. E.g., Public.Law imports the JSON data into a Postgres database and Netlify static pages. That particular code isn't yet open source.

Example: U.S.A. / Oregon Administrative Rules

Example: Canada / Department of Justice Legal Glossaries