Methods & sources

Daylight is a public, observational watchdog for federal .gov infrastructure. We practice on ourselves the transparency we ask of the sites we watch: every source is named here, our bot identifies itself honestly, our own uptime is public on /status, and every correction we make is public on /corrections. Each section below is written twice — a plain lead anyone can read, and an expandable technical version with the exact mechanism.

The one line we never cross

Noting that a certificate, subdomain, or login page exists is fine. We never authenticate past any access wall, guess credentials, probe, port-scan, fuzz, or brute-force. We observe the front door; we never try the handle. This keeps the project legally clean and press-credible — that is the whole strategy.

Public data only. Every source is reachable without authentication.
Observational only. We record that things exist and how they change; we never interact past any access control.
Reproducible & timestamped. Every observation stores its timestamp, source URL, and content hash. Every claim is independently checkable by a stranger.
Neutral presentation. We state what was observed, linked to its source — never an accusation the data does not support.

Data sources

CISA dotgov-dataLedger (live)

The official public list of who owns each federal .gov domain, and the email listed as its security contact. We check it every day and note anything that changed.

the technical version

We `git fetch` the public cisagov/dotgov-data repo and diff current-federal.csv commit-to-commit. Every emitted change links the commit-pinned blob it was read from, so the exact before/after row is re-checkable. Ownership/contact deltas are classified (H1 contact-domain mismatch, H9 contact concentration, watchlist hits); city/state churn is stored but never surfaced as a change.

Certificate Transparency logs (via crt.sh)Lookout (backfill live)

Every time a website gets a security certificate, it's published in a public log. We read those logs to notice when a new subdomain first appears — for example a `previews.` or `photo.` host.

the technical version

Public append-only Certificate Transparency (CT) logs, queried via crt.sh (with backoff + caching). We extract SANs, identify never-before-seen subdomains under watched apexes, and score them against a label list (previews/staging/auth/photo/analytics/infra…). Existence-only: we record that a certificate exists; we never connect to, probe, or authenticate to the host. Each row links back to its crt.sh query.

Federal GitHub organizationsLookout (GitHub signal)

Federal teams build a lot of their sites in the open on GitHub. A brand-new code repository — or the first commit to an empty one — often appears before the website it powers does, so we watch a short list of federal orgs and note when something new shows up.

the technical version

We poll the public GitHub REST API for the repositories of watched federal orgs (config/watchlist.yaml `github_orgs`: GSA, 18F, cisagov, uswds, nationaldesignstudio…), newest-created first, and diff against what we've seen. A new repo or a first commit becomes a Lookout event. We key on GitHub's immutable numeric repo id, so a rename is never mistaken for a new repo; forks never emit; and because a repo missing from one poll can be a transient API/pagination miss, we do not report removals. Existence-only: a public, read-only API — we never touch anything private. The read-only token this uses (required where we run, since a shared cloud egress IP exhausts the unauthenticated per-IP limit) carries no repository permissions.

Live public page source (Playwright)Floodlight (live)

We load a public government page the way your browser would, once, and write down what it loaded on its own — which trackers ran, whether it records your clicks, and whether it links a privacy notice. We never fill in or submit anything.

the technical version

A headless Chromium loads the public URL once (load-only: no auth, no form submission, no clicking, no crawling). We capture every network request (with a bounded POST-body sample) and passive DOM facts, then fingerprint third-party trackers and the reverse-proxy disguise (a first-party endpoint whose payload shape matches an analytics SDK). SSRF-guarded (IP-pinned, RFC1918 blocked); an access-gated page is noted as existing and never entered.

GSA Site Scanning (federal web scan)Site Scanning (breadth net → Floodlight)

The government scans its own public websites every day and publishes the results. We read that daily list to spot when a new tracker or analytics tag shows up on a federal site — and when we and the government's own scan agree, that's a finding that's much harder to wave off. It's a wide, cheap net; Floodlight is the close-up look.

the technical version

We download GSA/TTS's public daily bulk scan of the federal web (~12.6k live sites, one HTTP GET behind a free api.data.gov key) and diff it per URL. When a new, non-benign third-party service or the site's own Google Analytics tag (distinct from the government-wide Digital Analytics Program) appears on a .gov we don't already watch, we QUEUE that apex for a full Floodlight pass rather than trusting the scan — signature scanning is breadth, Floodlight's browser capture is depth. Two limits stated plainly: (1) it detects only KNOWN third parties by domain/tag, so a first-party endpoint on the .gov itself serving an analytics-shaped payload (the reverse-proxy disguise Floodlight exists to catch) is invisible to it by construction — a clean row is never evidence of "no tracking"; (2) a scan can time out, and a timeout is not an absence, so we only read a tracker as newly-appeared when the government's scan completed on both the before and after.

DuckDuckGo Tracker RadarFloodlight (engine live)

An open, public catalog of known trackers and what they do. We use it to recognize the trackers we see on a page.

the technical version

Seeds Floodlight's curated fingerprint set (host + payload-shape signatures) alongside EasyPrivacy and a session-replay vendor list. We keep a high-signal subset — the vendors that actually appear on federal sites plus the common analytics/session-replay tools — not the full multi-thousand-host dataset, so it stays auditable.

Wayback Save Page Now (SPN2)Receipts (live)

An independent public archive (the Internet Archive). We ask it to save a copy of watched pages, so the record of what a page showed on a given day isn't one we control and can't quietly change.

the technical version

On snapshot, we POST the URL to web.archive.org/save/{url} (rate-limited, opt-in) and store the returned archive URL beside our own dated snapshot. When something present before is gone after — a tracker, a privacy clause, a seal, a form field — it becomes a dated `removed` event in the removal ledger with before/after.

Federal Register APIRedtape (gap-finder + human gate live)

The government's own public record of privacy filings. When a site collects personal information, the law generally requires a published privacy notice; we search this record to see whether one exists — and show you the exact searches we ran.

the technical version

An AI research agent (behind a human-approval gate — see below) searches the Federal Register for SORNs matching a collection, returning references OR a documented negative with the exact queries run. Only human-reviewed, published gaps ever appear on /redtape, and only with a non-empty query + source trail so the negative is re-runnable. Copy is constrained to observation ("no published PIA found as of {date}; searches below"), never "illegal." PIAs aren't in the Federal Register — they live on each agency's own privacy pages — so the researcher also web-searches the operating agency's PIA inventory and reads it directly (guarded, .gov-only, redacted) to check the PIA leg, not just the SORN.

How our detections work

Every flag is derived from public data by one of the heuristics below. Each says what it flags and what it deliberately clears — a flag is documented editorial judgment, not a black box.

H1 · Contact-domain mismatch (Ledger)

We flag when the security contact listed for one agency's site is an email at a different, unrelated agency's domain — a sign that control may have quietly moved.

the technical version

For each registry row, we take the registrable apex of the security-contact email and compare it to the row's own apex. A cross-apex contact is a candidate; it escalates to HIGH when the contact apex is itself a watchlisted product .gov (this reproduced the akash@ndstudio.gov → usadf.gov finding structurally, with no name hard-coded).

Deliberately clears: A contact at the domain's own apex, a recognized central mailbox (an allowlist of shared SOC addresses like eop.gov/gsa.gov/cisa.gov), and — when resolvable — a contact at another domain owned by the SAME organization (e.g. vote.gov → @eac.gov, both the Election Assistance Commission).

H9 · Contact concentration (Ledger)

We flag when one small office becomes the listed security contact across three or more unrelated agencies — the pattern of quietly consolidating control.

the technical version

A cross-record pass groups the whole registry by contact apex and flags a foreign, non-allowlisted apex that is the security contact of record for ≥3 distinct owning organizations. Idempotent: a stable concentration is reported once, not every run.

Deliberately clears: Same-apex self-contacts, allowlisted central mailboxes, and any apex serving fewer than three distinct organizations (a normal shared-services contact).

Reverse-proxy disguise (Floodlight, flagship)

Some sites route their tracking through their own web address so ad-blockers don't recognize it. We catch that by the shape of the data being sent, not the brand name — so it still fires if the tracker is renamed.

the technical version

We flag a FIRST-party endpoint whose request shape matches a known analytics SDK: a beacon (POST/xhr/fetch) carrying a PostHog/Amplitude/Segment/Plausible/GA4/Matomo/rrweb body, or an AutoMonitor-style {session_id, events[]} POST to a collect/ingest/metrics/events/track path. Beacon-gated so ordinary content pages never trip it.

Deliberately clears: Plain GET content navigations (even to a path like /decide or /s/…), and any request without a corroborating analytics payload shape.

Session replay & privacy-notice gap (Floodlight)

We note when a page records your actual clicks, scrolls, and keystrokes ("session replay"), and whether it even links a privacy notice.

the technical version

Session replay is detected via vendor fingerprint category or a tightened replay-path signal. A page that collects PII or loads trackers but links no privacy notice — after an in-page probe of canonical /privacy paths to avoid false negatives — is flagged. The bare `/s/` third-party path no longer mints a false 'high'.

Deliberately clears: A page that links (or serves at a canonical path) a genuine privacy notice, and third parties whose only signal is an ambiguous single-letter path.

PII form fields (Floodlight → Redtape)

We record the KINDS of personal information a form asks for — a name, an SSN, a date of birth, a passport number, a photo — without ever filling it in or capturing what you'd type.

the technical version

We classify inputs by type, autocomplete token, and name/id/placeholder pattern into normalized kinds (never raw values). A form collecting SENSITIVE PII (ssn/dob/passport/photo), or ordinary PII with no linked privacy notice, becomes a strong Redtape candidate — the canonical E-Gov Act §208 gap.

Deliberately clears: Generic text/search/submit inputs that name no specific PII category.

New & mimicking subdomains (Lookout)

When a new subdomain like previews.example.gov or photo.example.gov first appears in the public certificate logs, we surface it — especially names that look like staging, internal infrastructure, or an imitation of another agency.

the technical version

We score never-before-seen SANs under watched apexes against a label list (previews/staging/auth/photo/analytics/infra…) and mimic tokens. Existence-only from the public log; we never connect to the host. Each row links its crt.sh query.

Deliberately clears: Ordinary product subdomains without a flagged label; existence in a Certificate Transparency log is never treated as access.

Vendor build-graph (Foundry)

When one contractor builds many agencies' sites, their shared staging addresses show up in the public certificate logs. We join those to the ownership registry to show how many different agencies flow through one vendor — and to spot a site being built that hasn't been officially registered yet.

the technical version

For each vendor apex (detected structurally as one whose build tree stages ≥2 distinct registered target apexes), we extract a project label per CT host (the label left of an env tier previews/staging/prod/int, else the apex-adjacent non-plumbing label), map it to a candidate .gov by stripping decoration affixes, and join to the CISA registry. Output: a build-concentration index (distinct owning agencies per vendor — orthogonal to Ledger's contact-concentration, since each agency's registry contact is legitimately its own) and an unlaunched-project watch (candidate apex confirmed ABSENT from the registry). Existence-only; reuses Lookout's + Ledger's data with no new source.

Deliberately clears: Vendor plumbing with no product label (cdn/analytics/storybook hosts), and short single-word codes marked low-confidence. A project that resolves to a registered apex is not an unlaunched-watch item.

Removal ledger (Receipts)

We keep dated snapshots of watched pages so that if a tracker, a privacy clause, an official seal, or a form is quietly deleted, there's a permanent before-and-after record — plus an independent copy at the Internet Archive.

the technical version

Two snapshots are diffed; a present-then-gone tracker/privacy-clause/seal/form-field emits a dated `removed` change (HIGH), with the raw artifact kept out of the public read path and a Wayback URL stored beside it.

Deliberately clears: Cosmetic churn that doesn't change any tracked dimension; nothing from behind an access gate is ever snapshotted.

Filing-gap findings (Redtape)

When a site appears to collect personal information, we check the government's own public record for the privacy filing the law generally requires — and a human reviews every finding before it's published. We show the exact searches, and we never say anyone broke the law.

the technical version

An AI research agent searches the Federal Register and returns references or a documented negative with the queries run. A hard human-approval gate (enforced in the data layer) means nothing publishes without review AND a non-empty query/source trail. Findings are phrased as observation, dated, and retractable via /corrections.

Deliberately clears: Collections with a covering SORN/PIA on file (found = no gap), and anything a reviewer holds or rejects.

Responsible disclosure

If a scan ever incidentally surfaces an exposed secret, credential, or a real vulnerability, we stop, do not publish, and route it privately to the affected agency and CISA through proper channels. That is not our mission and not ours to weaponize. The public read-path serves only reviewed, redacted data; a redaction pass runs on ingest and anything flagged is withheld pending human review. Home addresses, personal accounts, and family are never surfaced.

How our bot behaves

Requests Daylight makes to public sources carry an honest, self-identifying User-Agent, so anyone can see it is us and reach a contact:

DaylightBot/0.4 (+https://daylight.watch/methods; observational; public-data-only)

We honor robots.txt, cache aggressively, back off exponentially on errors, and never hammer a source. How often we read each source:

Source	Cadence
CISA dotgov-data	Daily; a full git-history backfill runs once.
Certificate Transparency logs (via crt.sh)	Nightly reconcile; certstream (real-time) deferred pending hosting.
Federal GitHub organizations	Daily poll.
Live public page source (Playwright)	Weekly full sweep (Mondays).
GSA Site Scanning (federal web scan)	Daily (GSA refreshes the dump ~daily; one bulk download).
DuckDuckGo Tracker Radar	Bundled dataset; refreshed with releases.
Wayback Save Page Now (SPN2)	With each Receipts snapshot (twice weekly, Mon & Thu).
Federal Register API	On new collection detected + monthly re-sweep (human-gated).

Where AI is used

One module uses an AI model: Redtape, the privacy-filing gap-finder. An automated researcher drafts an assessment by searching the public Federal Register and records the exact queries it ran. Nothing it produces is published automatically. A human reviews every finding behind a gate enforced in the data layer, and only human-approved items with a re-runnable search trail ever appear on /redtape. Everywhere AI-drafted text appears, it is labeled as such, and fact is distinguished from inference. The rest of Daylight — Ledger, Lookout, Floodlight, Receipts — is deterministic code with no model in the loop.

Open & machine-readable

The record is meant to be re-checked and cited, so it’s published for machines as well as people. Every module has an RSS and a JSON feed; a public JSON API lives under /api/v1; every page carries schema.org structured data and a canonical link; a sitemap lists every domain and change permalink; and an llms.txt summarizes the site for AI tools. Our robots.txt deliberately welcomes search engines and AI crawlers — a public accountability record should be findable and quotable. New here? The FAQ & glossary answers the common questions and defines the terms.

Corrections & disputes

A watchdog’s strongest credential is a visible record of its own mistakes. When we retract or amend a finding, it is logged, dated, and public on /corrections in the same feed format as everything else. If you believe a specific observation is wrong, dispute it — resolved disputes are logged as public corrections.

What we watch closely

Everything Daylight records is public regardless, but a curated watchlist decides which identities — organizations, domains, and security contacts — get the loudest, highest-severity flag. It’s open to suggestions; the watchlist page has a submission link.

PII restraint

We surface official public registrant records — which name agency security contacts by design — and public officials acting in an official capacity. We do not enrich, cross-reference, or aggregate personal data about individuals beyond that official capacity.

Contact

Tips, disputes, and watchlist submissions: contact@daylight.watch.

Support this work

I build Daylight on the side. There’s no team or outside funding; I just think it matters that someone keeps a public, timestamped record of who runs the federal web and what quietly changes, so I do. It runs on one small always-on server, a browser image, and an archive push; a few dollars a month. If it’s useful to you and you want to help cover that, there’s a tip jar below. It only goes toward hosting; the findings come from public data, so money doesn’t touch them. No pressure either way.

$3 $10 $25 Customvia Ko-fi ↗

Credit

Built with Claude Code. Research assisted by Claude.