01
What it is
SpiderFoot is an open-source reconnaissance automation framework that runs a target, such as a domain, IP address, email or username, through more than 200 data source modules and correlates the results into a single relationship graph. It answers the breadth question no single manual tool can: what does the full footprint of a target look like across every data source at once.
Manual OSINT work checks sources one at a time. SpiderFoot automates that process by dispatching a single target through dozens or hundreds of modules in parallel, covering DNS records, breach databases, social media presence, port scans, threat intelligence feeds and more, then correlating overlapping findings into a graph showing how entities connect.
Investigators use SpiderFoot to establish a comprehensive baseline footprint of a domain or individual before manual deep-dive work, to identify data sources worth investigating further by hand, and to monitor a target for new exposure over time through scheduled scans.
When to use this guide
- Establishing a comprehensive baseline footprint of a domain, IP or individual before manual investigation
- Correlating findings across many data sources automatically rather than checking each manually
- Discovering data sources and exposures a manual workflow might miss entirely
- Monitoring a target for newly exposed information over repeated scans
- Mapping infrastructure relationships between domains, IPs and organisations at scale
02
How do you run a SpiderFoot OSINT scan?
Install the framework, scope the scan to relevant modules, then interpret the correlation graph it produces.
Free before paid. SpiderFoot itself is free and open source; some of the modules it calls query third-party APIs that require their own free-tier keys for full coverage.
SpiderFoot (self-hosted): Free and open source, MIT licensed. Runs locally or on a server you control, with a web interface for configuring and reviewing scans. The version this guide covers.
SpiderFoot HX: Free trial, then paid. The hosted commercial version, offering additional modules and managed infrastructure; most individual investigators run the free self-hosted edition instead.
VirusTotal: Free API tier. One of many third-party data sources SpiderFoot can query given a free API key, adding domain and file reputation data to a scan.
Shodan: Free tier with limited queries. Another commonly integrated data source; adding a Shodan API key to SpiderFoot enriches infrastructure scans with exposed-service data.
Before you begin
Stop at the login
The self-hosted SpiderFoot edition requires no login to install and run locally; it exposes a web interface on localhost by default. Several of its 200-plus modules require a free API key from the respective third-party service (VirusTotal, Shodan, and others) to return results; SpiderFoot runs without them but with reduced module coverage.
Legal considerations
SpiderFoot only queries data sources that are themselves either fully public or accessed through an API you hold valid credentials for; it does not exploit or bypass access controls. Running an aggressive scan against infrastructure you do not own or have permission to assess can still trigger abuse detection on the target's side and, in some configurations, cross into activity that requires authorisation; scope scans to open-source data modules when investigating a third party.
The method
01
Install and launch SpiderFoot locally
Goal · Get the framework running before configuring a scan
Install via pip or Docker following the project's documentation, then launch the web interface, which runs on localhost by default. No account or registration is required for the self-hosted edition.
02
Add API keys for the data sources you need
Goal · Unlock the modules that require third-party authentication
In the settings panel, add free-tier API keys for services relevant to your investigation, such as VirusTotal and Shodan. SpiderFoot will run with whatever keys are present and skip modules for services without one.
03
Define the target and scan scope
Goal · Set exactly what SpiderFoot investigates and how aggressively
Enter the target (domain, IP, email or username) and choose a scan mode. The "Passive" mode uses only data sources that never directly contact the target, appropriate for most OSINT investigations; "All" mode includes active techniques such as port scanning.
04
Run the scan and monitor progress
Goal · Let the framework dispatch the target across its module set
A full passive scan against a domain typically completes within minutes to an hour depending on how many modules are enabled. The interface shows live progress and a running count of data elements discovered.
05
Review the correlation graph
Goal · Interpret how discovered entities relate to each other
SpiderFoot's graph view shows relationships between the target and everything it discovered, such as related domains, email addresses and IP ranges. Use this view to identify which findings warrant manual follow-up rather than treating every node as equally significant.
06
Export and verify high-value findings manually
Goal · Confirm automated findings before relying on them
Export results to CSV or JSON for record-keeping, then manually verify any high-value finding, such as a breach exposure or a newly discovered subdomain, using the primary source rather than citing SpiderFoot's output alone.
03
What false positives affect SpiderFoot scans?
Automated aggregation across 200-plus modules produces a specific set of recurring errors.
Module noise from indirect associations: Several SpiderFoot modules flag loosely related entities, such as a shared hosting provider or a co-located IP, as connections. A shared data centre is not evidence of a meaningful relationship between two targets.
Verifying check: Check the specific module that generated a correlation before treating it as significant; infrastructure-adjacency modules produce far more noise than identity-specific modules.
Stale data from cached third-party sources: Some data source modules return cached results that can be months old, particularly for breach databases and WHOIS records.
Verifying check: Note the data source and, where the module exposes it, the record date; re-verify time-sensitive findings such as active infrastructure directly rather than relying solely on SpiderFoot's cached result.
Passive mode still misses actively hidden infrastructure: A target using Cloudflare or a similar proxy service will not have its true origin IP or hosting exposed by a passive scan, and SpiderFoot cannot bypass that concealment on its own.
Verifying check: Treat an absence of infrastructure findings as inconclusive when a target is known to use a reverse proxy, rather than assuming no exposure exists.
Free-tier API rate limits truncate results silently: Modules relying on free-tier third-party APIs can hit a rate limit mid-scan, returning a partial result set without a prominent warning.
Verifying check: Check each module's log output within the scan for rate-limit or error messages before treating a low result count from a specific module as a genuine absence of data.
Chain of custody: SpiderFoot results reflect a point-in-time aggregation across many third-party sources that themselves change independently. Export and document scans promptly rather than re-running them and assuming identical results.
Export the full scan result set to CSV or JSON immediately after completion.
Record the scan configuration used, including which modules and API keys were active.
Note the scan start and end timestamps.
Save the correlation graph as an image or exportable format for the case file.
Manually re-verify and independently archive any finding relied upon for a conclusion, rather than citing the aggregated tool output alone.
04
Go deeper
Reference cards, structured tutorials and tools for practitioners who want to build on this guide.
CARD · DIG-001
WHOIS investigation: domain name to attribution chain.
ViewDNS, SecurityTrails and crt.sh. Registrant email pivots that map infrastructure and hosting networks.
CARD · OPS-001
OPSEC for investigators: eight-step baseline.
Signal, Mullvad and ProtonMail. Device separation and session documentation to chain-of-custody standard.
Evidentiary standard
Signal & Shadow operates to the LST-001 evidentiary standard. All claims are graded against the LST-001 v1.0.3 confidence tiers (Confirmed, Corroborated, Reported, Alleged) per the canonical voice and structural specification.
About Signal & Shadow
Signal & Shadow is an independent forensic investigation and methodology practice publishing tutorials, reference cards, and forensic dossiers for working practitioners. Founded by Derek Bowler.
