Open Directories

I stumbled upon the OpenDirectories subreddit while looking for media archives,
and i think it's really cool how these people are trawling through the scrapyards
of the internet and finding these gems of data.


Open Directories are directories that are open. These are folders (& other data
storage) on servers that have been, whether intentionally or not, left open to the
internet, and they can contain anything, from radio recordings to software archives
and television shows.
It's really cool to have a look through the archives and see what is there, and what
you can take (usually all of it).


there's a small guide to scraping open directories on the subreddit linked above,
but i want to go into what i think works.

my most important recommendation is that you throttle your download speed, so you
don't waste the server owners' bandwidth. like, c'mon, that just isn't sustainable

I use duckduckgo as my search engine, so i get a mixed bag of results when searching,
but go with whatever search engine you want.

I use linux, and so use the wget utility to download from a directory.


it's honestly real fun to explore the web for these, it really feels like the l33t counterpart
to surfing neocities and people's personal websites, plus they technically count as part of the
dark web, so stick that on ur resume