Sounds Like Weird
05 November 2013
So I had a phone call that was supposed to happen via Skype today. Only it didn’t. At the time, our household was under a DoS attack.
Inadvertent, in all likelihood.
However, after 192,482 request for pages in a 48-hour period, I’m gonna add you to iptables no matter what your intent was.
It turns out that the culprit is that a handful of pages kept making the URL longer and longer. For reasons that I have not yet figured out (and probably won’t sleep until I do), apache did its best to serve up the pages even though those directories don’t exist.
So I have log files for things like (the URL is fake, but the patterns in the post are real):
http://foo.com/woof/a/a/a/a/b/a/a/b/b/b/b/b/cat.html
I found one of the offending pages:
http://foo.com/woof/a/b/cat.html
I can click a link on that page, and it’ll go to:
http://foo.com/woof/a/b/b/kitty.html
…and click a link that looks the same, and it’ll go to:
http://foo.com/woof/a/b/b/b/kitty.html
…then…
http://foo.com/woof/a/b/b/b/b/kitty.html
http://foo.com/woof/a/b/b/b/b/b/kitty.html
http://foo.com/woof/a/b/b/b/b/b/b/kitty.html
http://foo.com/woof/a/b/b/b/b/b/b/b/kitty.html
Except that directory structure’s not on disk. There’s no symlinks up or down the directory tree (do not do that!) that would cause this structure.
It’s html (old hmtl), no PHP, no server-side includes, no JavaScript (or CSS), no .htaccess, no rewrite rules. We’re talking stuff that’s pushing 15-20 years old, here.
It should work.
Yet, obviously, there’s a problem.
So, when someone tries to recursively wget the site’s document tree and uses a high enough number of levels (at least 13 in this case), suddenly 192,482 files get delivered and the requests will never terminate because some pages go (apparently) infinitely deep.
FWIW, we turned symlinks off anyway, and that didn’t prevent it from happening. It’s completely not obvious to me what the source of the issue actually is.
Hell, I was beginning to suspect mod_speling and that’s not even enabled.
It turns out that it was an Apache directive I’ve seen so often in examples that I’d overlooked it, even though I never enable it myself.
MultiViews.
Specifically, in directory a, there was no directory b, but there was a b.html. So it would serve that instead, and the apparent directory would get longer and longer and longer, leading wget to think there was another directory level to fetch.
So all that was needed was to turn off MultiViews and restart apache. None of us could remember exactly when that changed, but I think t thhere was a server rebuild in there somewhen.
Also, to the person who’s requested 60,000 copies of the same file that’s so old Rick doesn’t even remember what it is? Dude.