(OT) Script for getting list of all files/directories on website?

buttroast has asked for the wisdom of the Perl Monks concerning the following question:

Hello,

I was busy searching this morning to see if there were any Perl scripts out there for pulling a list of directories and/or files from a REMOTE website with directory listing turned off.

The script would probably have to cycle through some type of dictionary to do this. I was wondering if someone was aware of an existing script or utility that could accomplish this.

I don't necessarily want to use the script/utility, I just want to know if it is possible. I am working for a client that has a lot of files on their web server that I don't think should be there...i.e. source code for DLLs, etc. They don't seem too worried because they have the directory listing turned off and they think it would be nearly impossible for someone to guess the filenames. Since they aren't linked to any other files, they aren't worried about a spider from locating the files.

The reason the files are there, is that they use a DTS package to refresh their external www website with data and files from their internal system without excluding any files such as source code.

Does anyone have any input on this topic?

Thanks buttroast

2006-06-21 Retitled by holli, as per Monastery guidelines
Original title: 'Script for getting list of all files/directories on website?'

Comment on (OT) Script for getting list of all files/directories on website?

Replies are listed 'Best First'.
Re: (OT) Script for getting list of all files/directories on website? by Asim (Hermit) on Jun 20, 2006 at 16:27 UTC
First, I recommend a search on Google for the phrase "security through obscurity", which is exactly the problem you're encountering. A couple of well-written papers, with examples, by security analysts might sway more minds than any program you'll write. And yes, it's possible -- script kiddies have been doing similar for over a decade and a half, now. All it involves, at its most basic, is "best guesses" at directory names, like "home" and "etc", coupled, perhaps, with a full-blown dict attack. I don't know of any mainstream utility that would do it, but I cannot imagine it would be hard to build, just tedious to deploy and run. If you're serious about this, I'd suggest looking into "SQL Injection" attacks, which, as I recall, use similar techniques on some levels. Does that help? ----Asim, known to some as Woodrow.	[reply]
Re^2: (OT) Script for getting list of all files/directories on website? by buttroast (Scribe) on Jun 21, 2006 at 14:28 UTC
Asim, Yes, this helps me. I was having trouble coming up with a good search phrase for google. I've heard of "Security by Obscurity" before but it didn't come to mind when I was searching on this topic. Thanks again. Thanks buttroast	[reply]