In a world where data is currency and everyone’s got something to hide, Gospider doesn’t ask questions—it just digs. Born from the brains of @thebl4ckturtle and @j3ssiejjj, this Go-powered beast doesn’t care if your target is a pristine marketing site or a dark, cluttered corner of the web. It gets in, gets what it needs, and gets out—fast.

Why Gospider Is More Than Just a Web Crawler

Gospider hunts through HTML, follows links like a bloodhound, and scoops up everything from page metadata to JavaScript-ridden secrets. It doesn’t just crawl; it investigates.

Getting Started

Installation is dead simple. Grab the binary or build it yourself. Then it’s game on.

https://github.com/jaeles-project/gospider

gospider -q -s "https://targetsite.com"

Quiet mode. Like a digital ninja.

gospider -s "https://targetsite.com" -o ./loot -c 15 -d 2

Fifteen bots crawling two layers deep. Quick and dirty.

Hit List Mode

gospider -S targets.txt -t 25 -c 10 -d 3 -o ./campaign

25 targets, each getting hit with 10 bots. Maximum chaos, minimum effort.

Headers, Cookies & Camouflage

gospider -s "https://targetsite.com" --user-agent "Mozilla/5.0 (iPhone; CPU iPhone OS 15_0)" --cookie "authToken=deadbeef;" -H "X-Fake-Header: ghost"

Crawl like you’re logged in and undercover.

gospider -s "https://targetsite.com" --burp ./burp_req.txt

Load full raw requests from Burp Suite exports.

Pull From the Shadows

gospider -s "https://targetsite.com" --other-source --include-subs

Dig through Archive.org, CommonCrawl, AlienVault, and scoop subdomains while you’re at it.

Stealth Mode

gospider -s "https://targetsite.com" -k 3 -K 2

Add delay and randomness to your requests to avoid rate-limiting or detection.

Noise Control

gospider -s "https://targetsite.com" --blacklist ".(pdf|woff|png)"

Ignore junk files. Focus on actionable URLs.

Only the Good Stuff

gospider -s "https://targetsite.com" --length --filter-length "4200,8000"

Hunt based on file size. Find that juicy mid-weight content.

OSINT Flex

gospider -s "https://targetsite.com" --json -o ./intel

Pipe results into your automation scripts or analysis tools. JSON is your friend.

Final Words

Gospider isn’t just a crawler. It’s a recon artist with zero patience for fluff. It walks in, scans the place, grabs what matters, and ghosts.

For the digital detectives, the red teamers, and the ones who thrive in the static—this one’s for you.

Crawl smart, crawl safe, and don’t be a dick.

 

Get WordPress help, plugins, themes and tips at Macho.com