osint-investigation
Public-records OSINT investigation framework — SEC EDGAR filings, USAspending contracts, Senate lobbying, OFAC sanctions, ICIJ offshore leaks, NYC property records (ACRIS), OpenCorporates registries,
Public-records OSINT investigation framework — SEC EDGAR filings, USAspending contracts, Senate lobbying, OFAC sanctions, ICIJ offshore leaks, NYC property records (ACRIS), OpenCorporates registries,
Real data. Real impact.
Emerging
Developers
Per week
Excellent
Skills give you superpowers. Install in 30 seconds.
Investigative framework for public-records OSINT: government contracts, corporate filings, lobbying, sanctions, offshore leaks, property records, court records, web archives, knowledge bases, and global news. Resolve entities across heterogeneous sources, build cross-links with explicit confidence, run statistical timing tests, and produce structured evidence chains.
Python stdlib only. Zero install. Works on Linux, macOS, Windows. Most sources work with no API key (OpenCorporates has an optional free token that raises rate limits).
Adapted from the MIT-licensed ShinMegamiBoson/OpenPlanter project; expanded to cover identity / property / litigation / archives / news sources that the original didn't address.
Use when the user asks for:
Do NOT use this skill for:
web_search / web_extractdomain-intel skillarxiv skillsherlock skill (optional)The agent runs scripts via the
terminal tool. SKILL_DIR is the directory
holding this SKILL.md.
Read the data-source wiki entries to plan the investigation:
ls SKILL_DIR/references/sources/ # Federal financial / regulatory cat SKILL_DIR/references/sources/sec-edgar.md # corporate filings cat SKILL_DIR/references/sources/usaspending.md # federal contracts cat SKILL_DIR/references/sources/senate-ld.md # lobbying cat SKILL_DIR/references/sources/ofac-sdn.md # sanctions cat SKILL_DIR/references/sources/icij-offshore.md # offshore leaks # Identity / property / litigation / archives / news cat SKILL_DIR/references/sources/nyc-acris.md # NYC property records cat SKILL_DIR/references/sources/opencorporates.md # global corporate registry cat SKILL_DIR/references/sources/courtlistener.md # court records (federal + state) cat SKILL_DIR/references/sources/wayback.md # Wayback Machine archives cat SKILL_DIR/references/sources/wikipedia.md # Wikipedia + Wikidata cat SKILL_DIR/references/sources/gdelt.md # global news monitoring
Each entry follows a 9-section template: summary, access, schema, coverage, cross-reference keys, data quality, acquisition, legal, references.
The cross-reference potential section maps join keys between sources — read those first to pick the right pair.
Each source has a stdlib-only fetch script in
SKILL_DIR/scripts/:
Federal financial / regulatory
# SEC EDGAR filings (corporate disclosures) python3 SKILL_DIR/scripts/fetch_sec_edgar.py --cik 0000320193 \ --types 10-K,10-Q --out data/edgar_filings.csv # USAspending federal contracts python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \ --fy 2024 --out data/contracts.csv # Senate LD-1 / LD-2 lobbying disclosures python3 SKILL_DIR/scripts/fetch_senate_ld.py --client "EXAMPLE CORP" \ --year 2024 --out data/lobbying.csv # OFAC SDN sanctions list (full snapshot) python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --out data/ofac_sdn.csv # ICIJ Offshore Leaks — downloads ~70 MB bulk CSV on first use, # then searches it locally. Cached for 30 days under # $HERMES_OSINT_CACHE/icij/ (default: ~/.cache/hermes-osint/icij/). python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \ --out data/icij.csv
Identity / property / litigation / archives / news
# NYC property records (deeds, mortgages, liens) — ACRIS via Socrata python3 SKILL_DIR/scripts/fetch_nyc_acris.py --name "SMITH, JOHN" \ --out data/acris.csv python3 SKILL_DIR/scripts/fetch_nyc_acris.py --address "571 HUDSON" \ --out data/acris_addr.csv # OpenCorporates — 130+ jurisdiction corporate registry # (free token required; set OPENCORPORATES_API_TOKEN or pass --token) python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \ --jurisdiction us_ny --out data/opencorporates.csv # CourtListener — federal + state court opinions, PACER dockets python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Smith v. Example Corp" \ --type opinions --out data/courts.csv # Wayback Machine — historical web captures python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \ --match host --collapse digest --out data/wayback.csv # Wikipedia + Wikidata — narrative bio + structured facts # Set HERMES_OSINT_UA=your-app/1.0 (your@email) to identify yourself python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Bill Gates" \ --out data/wp.csv # GDELT — global news in 100+ languages, ~2015→present python3 SKILL_DIR/scripts/fetch_gdelt.py --query '"Example Corp"' \ --timespan 1y --out data/gdelt.csv
All outputs are normalized CSV with a header row. Re-run scripts idempotently.
When a private individual won't be in a source (e.g. SEC EDGAR for a non-public- company person, USAspending for someone who isn't a federal contractor, Senate LDA for someone who isn't a lobbying client), the script returns 0 rows with a clear warning rather than silently writing an empty CSV. EDGAR specifically flags when the company-name resolver matched an individual Form 3/4/5 filer rather than a corporate registrant.
Rate-limit notes are in each source's wiki entry. Default fetchers sleep politely between paginated requests. API keys raise rate limits for sources that support them (
SEC_USER_AGENT, SENATE_LDA_TOKEN,
OPENCORPORATES_API_TOKEN, COURTLISTENER_TOKEN). All scripts surface
429 responses immediately with the upstream's quota message so the user
knows to slow down or supply a key.
Normalize names and find matches between two CSV files:
# Match lobbying clients (Senate LDA) against contract recipients (USAspending) python3 SKILL_DIR/scripts/entity_resolution.py \ --left data/lobbying.csv --left-name-col client_name \ --right data/contracts.csv --right-name-col recipient_name \ --out data/cross_links.csv
Three matching tiers with explicit confidence:
| Tier | Method | Confidence |
|---|---|---|
| Normalized strings equal after suffix/punctuation strip | high |
| Sorted-token equality (word-bag match) | medium |
| ≥60% token overlap, ≥2 shared tokens, tokens ≥4 chars | low |
Output
cross_links.csv columns: match_type, confidence, left_name, right_name, left_normalized, right_normalized, left_row, right_row.
Test whether two time series cluster suspiciously close together — e.g. lobbying filings near contract awards — using a permutation test:
python3 SKILL_DIR/scripts/timing_analysis.py \ --donations data/lobbying.csv --donation-date-col filing_date \ --donation-amount-col income --donation-donor-col client_name \ --donation-recipient-col registrant_name \ --contracts data/contracts.csv --contract-date-col award_date \ --contract-vendor-col recipient_name \ --cross-links data/cross_links.csv \ --permutations 1000 \ --out data/timing.json
The script's column flags are intentionally generic — the original tool was written for donations vs awards, but it works for any (event, payee) time series joined through cross-links. Null hypothesis: event timing is independent of award dates. One-tailed p-value = fraction of permutations with mean nearest-award distance ≤ observed. Minimum 3 events per (payer, vendor) pair to run the test.
python3 SKILL_DIR/scripts/build_findings.py \ --cross-links data/cross_links.csv \ --timing data/timing.json \ --out data/findings.json
Every finding has
id, title, severity, confidence, summary, evidence[], sources[].
Each evidence item points back to a specific row in a source CSV. The user (or a
follow-up agent) can verify every claim against its source.
This is the load-bearing rule of the skill. Tell the user:
match_type=fuzzy is "probable",
not "confirmed."fuzzy match
between "ACME LLC" and "Acme Holdings Group" is a lead, not a fact.Use the template:
cp SKILL_DIR/templates/source-template.md \ SKILL_DIR/references/sources/<your-source>.md
Fill in all 9 sections. Write a
fetch_<source>.py script in scripts/ that
uses stdlib only and writes a normalized CSV. Update the source list in the
"When to use" section above.
entity_resolution.py does NOT use external fuzzy libraries (no rapidfuzz,
no jellyfish). Token-bag matching is the upper bound here. If you need
Levenshtein, transliteration, or phonetic matching, pip-install separately.timing_analysis.py uses Python's random for permutations. For
reproducibility, pass --seed N.fetch_*.py scripts use urllib.request and respect Retry-After. Heavy
bulk usage may still violate ToS — read each source's legal section first.All Phase-1 sources are public records. Bulk acquisition is permitted under their respective access terms (FOIA, public records law, ICIJ explicit publication, OFAC public data). However:
MIT
mkdir -p ~/.hermes/skills/research/osint-investigation && curl -o ~/.hermes/skills/research/osint-investigation/SKILL.md https://raw.githubusercontent.com/NousResearch/hermes-agent/main/optional-skills/research/osint-investigation/SKILL.md1,500+ AI skills, agents & workflows. Install in 30 seconds. Part of the Torly.ai family.
© 2026 Torly.ai. All rights reserved.