I'm trying to find the "hidden" pages behind a webpage, that appears to be blank, but resolves to a blank page. I have a suspicion the site is being used, but not the main index page. Does anyone know of a way to "crawl" the site? A true site crawl only reveals links to other pages within the domain name. I don't think there are links on this site...only standalone HTML pages from the URL. Anyone know how to find these pages?

Anyone? I really want to accomplish this!

This is one link i found

http://monkey.org/~provos/crawl/

looking for more...

i do know frontpage has a feature where you can download an entire website..but umm, i'm not sure if thats kinda what you mean...it downloads every page from the root.

wget on linux :P