How to use:
First, we need to get all the relevant to what we're looking for. Let's suppose we want to get all the pages from Danbooru. We use this command line:
dan-pages danbooru.donmai.us
To get the first 10 pages of the top level we use:
dan-pages danbooru.donmai.us _null_ 10
The first argument is the server (other examples: nekobooru.net, konachan.com). The second argument is the tag. _null_ signals the program not to use any tags. The third argument is the number of pages. The program can automatically detect when there are no more pages.
Examples:
Download all files tagged with tohno_akiha:
dan-pages danbooru.donmai.us tohno_akiha
Download all files tagged with tohno_akiha and highres:
dan-pages danbooru.donmai.us tohno_akiha+highres

Once this is done, run dan-extract to extract the URLs from the pages. You can pass it the number of pages you want to extract from, but it can detect them anyway.
The program will generate urls.txt, which contains all the URLs that could be found (ideal to use with wget), and urls####.txt, which contain the same list split every 1000 files (some graphical download managers, e.g. FlashGet, become rather sluggish when adding too many files at once).


Enjoy.

Not a bad little piece of code if you don't mind me saying.

I'm a comp sci student and I'm studying C and C++ programming right now, could anyone maybe tell me a little more about the code itself in this thing?

The code is actually very simple.

All it does is prepare a command line string based on the user's input or the arguments passed and pass that to wget through system(). I know it's not the best method and about the bad reputation system() has, but anything more sophisticated would have been overkill and probably not as portable.

Once all the pages are downloaded, the code looks line by line in the resulting HTMLs for a certain pattern I noticed a while ago. Namely, "\"file_url\"". Immediately after this string, follows a colon, and a quote-enclosed URL, which is the direct link to the file itself (not the thumbnail). After that, it's simply a matter of finding the next quote. This string has the peculiarity of using "\\/" instead of decent slashes. My guess is that it's some limitation of the language the string is in (my knowledge of web is very limited, but I'm guessing JavaScript; although I really don't know). Removing the backslashes from the string is trivial.

That's it.

Uh.. Welcome to /c/.

Thanks for the tool... But I don't think I'll be using anything posted on 4chan.
I'd just rather write my own in Perl. Which seems more appropriate for the task, anyway.

So this can do a complete site rip?

Yes, that's right.

Also, bump.

Aren't there already programs out there that do that? and I thought wget essentially grabbed all content from pages. I made something like this a year ago, but I haven't had time lately to work on it. It works on all chan sites, plus the ones op mentioned, several wallpaper sites (including 4scrape) and numerous other sites I haven't event tested. It's sadly still in a beta, but has an executable available, which I only released to friends to help test it. (Yes, I have a Miku skin for it)