>>8052084
What? I can't hear you.

Thanks, man.

Post in this thread if you are a strong beautiful little girl who don't need no Eksopl.

Is this a dedicated machine?

>>8052107
Dedicated as in hosting 24/7, yes. Dedicated as in only for the archive, no. It's the same machine the FTP is hosted on.

>>8052094
Uh, we still kinda need Eksopl for the software used for this and for the rest of the archive. Even if he's not hosting it himself, without him there would still be no archive.

>>8052119
I meant as in its a machine separate from the one you usually use.

>>8052127
Ambiguous phrasings...'use' as in 'use for hosting'? Yes for FTP, no for usual web pages (only just set up a web server for this, normally use a machine elsewhere). 'Use' as in browse /jp/ and play games on it? No, I use my laptop for that.

>>8052136
Meant the latter.

Have fun with it, I'd like you know how you merge the current DBs with the ones Eksopl will supply sometime soon.

>>8052144
Depends on how it's exported. It's possible to export in such a way that you can just import everything into your existing database with a simple command.

>>8052166
Guess we'll see how that works then, I'd like to know more when it does.

Merging two databases together shouldn't be a huge project, just check Google for that.

However, doc_id seems to be the primary key, so those may need to be incremented in your current database based on Eksopl's database numbers.

>>8052197
Wouldn't the primary key being the post number make the most sense?

The general idea that I have in mind (as autistic as it sounds) is to have maybe three or five archives running at a single time as a failsafe in case one or mail fail.

Going to mess around with the modules on the server a bit later and give it a whirl.

>>8052223
Create an archive of the archives.

>>8052221
I thought so too. I don't know what doc_id is used for, the first thing that came to my mind was searching but... dunno. Ask Eksi, I don't want to bother wading through the damn perl.

Bless you, VDZ. Why don't we have tripfag circlejerks about you?

>>8052221

General best practice as far as I'm aware of is to having the primary key in databases associate with nothing concrete. Weird exceptions and overlapping crop up when people least expect them.

>>8052248
because the circlejerks center around shitposters. Good trips need only silent reverence.

So, it was true.

Thanks man, keep up the good work.

Fuuka updater is working, but /b/ackwash isn't. Hmm...

>>8052221
doc_id was introduced in the last commit. Has to do with the new search backend.

>>8052253
Even when you're guaranteed that the items will be unique? Post number isn't something that will revert, and it provides a primary key that's much more meaningful than, say, an ID automatically incremented by the database. And since post numbers are supposed to be unique and are a potential element to do searches on, you'd want to make the post number field a unique index anyway.

>>8052289
Oh, then it's probably just fine. It can probably be simply rebuilt for the merged db.

>>8052292

You can't guarantee it; moot could reset the board or something. It's not something anyone would count on happening, but it's defensive programming. Make no assumptions about the integrity of the data you're using.

>>8052292
It's called a rollback. boards.4chan.org crashes and has data errors. Board is restored from a backup. Posts made after the last backup will have the same post numbers as the ones after the backup has been restored.

>>8052284
Strike that, updater doesn't work either.

>>8052329
How would easymodo handle duplicate post numbers, anyway, though? Would it actually display two posts with the same post number in a thread?

Nice one OP. Now add /tv/, /a/, /m/, and if you still have room, /pol/

>>8052376

Yeah, no.

>>8052144
>>8052127
>>8052107
Somebody is jealoooooous....

U JELLY VOILE?

>>8052380
Jealous of what?

>>8052379
I need those boards archived. It's not like /jp/ is more worth of archive than them anyway. Even 2010's /jp/ shitposts were more archive worthy than they are today

>>8052376
Shitty boards don't deserve an archive, except for /m/.

Easymodo is gone for good then?

>>8052386
Your needs won't magically come up with extra diskspace and resources that archiving all those boards would require.

>>8052390
What's your problem with /tv/? I swear any time a board doesn't act exactly like /jp/ does they are complete trash. Yeah whatever. /tv/ is ok

>>8052074
VDZ, please never leave us like all the other decent tripfags did..
/jp/ is nothing but a bunch of /b/ level retards now.
if you stay maybe we can get back the good old days.

>>8052386

None of those are worth archiving, though.

If anything, I'd just like to see someone pick /tg/ back up. Sup/tg/ only seems to archive shitty quest threads these days.

>>8052386
Why not make your own archive then if you badly need them for those boards?

>>8052376
no-ip has /pol/. /a/ is a waste of bandwidth as long as it's limited.

>>8052398
Stop bringing this shit up in every thread.

Thanks.

Just add /m/, its the only board with a decent userbase these days.

>>8052406
No, no-ip has /new/. Installgentoo is the one with /pol/.

>>8052418
http://archive.no-ip.org/pol/

What's this then

>>8052426
Amusing.

>>8052426
http://archive.installgentoo.net/cgi-board.pl/pol/

>>8052448
That some redundant redundancy.

>>8052448
So there's two archives of the same board. I fail to see why VDZ should strain his server with /pol/.

Archive /soc/ and /cgl/.

>>8052477
Die

I do not intend to archive any other boards than /jp/. If you want other boards to be archived, get someone else to archive them.

>>8052477
http://archive.no-ip.org/soc/
No-IP has /soc/ for some reason. No /cgl/ as far as I can see though.

>>8052481
How about /m/? I think that /jp/ gets along well with /m/.

I'd say /po/ but no-ip has it.

>>8052426
I had no clue he was stupid enough to add yet another board to the archive.

>>8052481
>>8052514
I was just kidding. I have no desire to see those boards archived at all.

I'd love to see /m/ get archived (unless it's already archived elsewhere, and I'm just not aware of it).

Who will maintain the code now?

vdz owns.

>>8052595
Eksopl maybe. He's just not going to run the site any longer.

just do /m/, we are best friends besides /m/ hates our shitposting/shitposter shit. they are part of the older /jp/.

>>8052654
Maybe he should move the code to github.

>>8052595
http://code.google.com/p/fuuka/updates/list
It's still maintained.

>>8052672
If you want /m/, then do it yourself. VDZ is only bothering with /jp/

>>8052712
Are you the same Mayhem that works on 4chan X?

>>8052732
Yeah.

>>8052740
Thank you for your work.

>>8052748

>>8052756
Also, please tell the other coders I thank them for their work as well. You all do very good work, and 4chan is better for your efforts.

>>8052756
Thanks for your hard work.

>>8052795
>>8052768
>Also, please tell the other coders I thank them for their work as well.
Done.

>>8052074
>maybe my internet will be cut off for excess bandwith usage
Where is this being hosted from...

Wait so will this have all the older threads?

>>8052964
Depends on if eksi will release them and if vdz and merge them with what he's got now.

the iqdb links are broken. they add the archive/ part into the thumbnail link which breaks it.

>>8053007
Because they're configured to be stored in a directory different from where the IQDB section is configured to look.

I think

>>8053007
Fixed now. Thanks.

Very cool.
Do you expect this to be able to host this archive for a decent amount of time? If you have to stop hosting it, will you consider uploading a dump (assuming you'll merge it with easymodo's dump, when that is released).

Nice, thanks for all the effort, VDZ.
Hopefully your mind won't break as badly as in Eksopl's case. May him rest in peace.

>>8053207
If all goes well, yes. We'll see soon enough. As for uploading a dump, of course. Somebody would need to take over and for that they'd need the database.

Thanks for hosting!

Symlink http://jparchive.dyndns.org/index.html to http://jparchive.dyndns.org/archive/cgi-board.pl/jp/index.html at least. I trust you can do that. If the archiving software lets you, please make http://jparchive.dyndns.org/archive/cgi-board.pl/jp/ the root directory!

VDZ, can you make a promise to us to not normal-fag-ize and/or suddenly get all testy about the archive like a certain Eksopl did?

If you have problems with your ISP or bandwidth issues, then just say it. Please don't pull the plug on a whim without warning, so we can have proper time to continue the archives. I appreciate all Eksopl's done, although the occasional "lol it's dead / nevermind it isn't" episodes were not cool.

http://archive.foolz.us/jp/

Hey guys what's going on in here?

>>8053313
You can never have too many archives.

>>8053313
> /a/ is there as well
Looks like Woxxy went through with it after all.

>>8053259
>I trust you can do that.
You think too highly of me. This is the first time I've hosted a webserver, I'm even failing at something this basic...

>>8053283
Sorry, I'm already a normalfag. But I won't pull the plug for no reason.

Kyou was so superior to that filthy cockroach. She should have been the main girl.

..thanks for the archive op.

>>8053331
Rome wasn't built in a day. Early mistakes are just stepping stones.

>>8053331
Please just dont turn into a shitposter later on, you can stay a normalfag but dont ever turn into what happened to zunbar. We need good tripfags like you.

>>8053313
Quoting seems to be broke when you use the search.

>>8053331
Go to your web directory. Find it with "ls" (show files) and "cd" (change directory). You can use tabs to complete filenames. If you mess up, type "cd ~" (return home). Once you're there, type this into the shell: "ln -s (wherever the archive page is, find it with the same process) index.html"

So if your web directory is called "web" and the archive page is in "~/web/archive/jp/index.html" or something like that, you could go into your web folder and type "ln -s archive/jp/index/html index.html". This would make it so people visiting the site would see that page instead of the normal homepage.

This is all assuming you have shell access though. Disregard this if it does not.

>>8053313
Oh but he won't add /tv/ and /m/?

Fuck this

>>8053408
/tv/ is on gentoomen.

/m/ is too boring.

>>8053413
/m/ is the only decent board on 4chan.

Why the fuck would he add /a/.

>>8053413
Fuck off, a lot of people here browse /m/ too.

>>8053422
Our boards have always shared the same userbase. Get over it.

>>8053430
No, only the shitposters do. /jp/ and /m/ and way closer.

>>8053422
/a/ was the first board to be archived anyway.

Also, to the webmaster of archive.foolz or whatever, you fucked up the linking somehow. I appreciate your effort but get on that.

>>8053439
Also, can you add /m/? It's a slow moving board.

>>8053439
There is no point in archiving /a/, no reason at all.
/a/ is not even gonna use it or know about it and the real /jp/ gives no fucks about /a/.

>>8053425
That explains why it's so terrible.

>>8053452
Go back to /a/.

>>8053451
They use their archive pretty frequently, actually.

>>8053451
>/a/ is not even gonna use it or know about it

/a/ uses it's archive all the time. It's one of the only boards that does aside from /jp/.

/jp/ is honestly delusional when it goes to their perception of /a/.

>>8053460
>comes*

>>8053456
>>8053460
We dont care. All /a/ does to us is shitpost.

>>8053476
I don't like /a/ but /jp/ shitposts itself into oblivion. Stop being delusional

>>8053416
/po/ is good.

>>8053484
This whole site shit posts you faggot.

Someone needs to set up a mirroring system between archives, so ghost posts from one board can be mirrored to another.

The archivers should all have an "other boards" link that has a page where it shows what sites archive what boards, automatically updated.

Don't archive /a/, let them figure it out themselves.

Besides, if vdz's worried about too much bandwidth coming from archiving /jp/, a pretty sluggish board overall, he shouldn't even think about archiving 3rd largest board on 4chan.

>>8053790
/a/'s the third biggest? Huh, would have thought that it'd have been /g/.

Wonder what /r9k/ is

>>8053530
That would mean someone has to host a separate MySQL server for the ghost posts and make it public. The source code would require rewriting, and the site itself would become much slower for having to make sql queries over the internet before the page can even load.

Basically, it's doable but not worth the effort and result. When ths situation has calmed down and we have some more-or-less permanent archives going on, we should just agree upon one of them as the main ghost board. Also a backup in case the main one is down.

>>8053813

/a/, /b/, and /v/ have all been the biggest draws to 4chan, it's no wonder the boards themselves have famously awful userbases and are practically incontinent in holding back their shitposting and meme spam from the rest of the site.

>and the site itself would become much slower for having to make sql queries over the internet before the page can even load.

Or the server the post originates at could just forward the post onward at the time it's created.
>>8053960

>>8053960
Not a good idea of hosting a public SQL server for the posts as it would reveal private information (such as poster's IPs). A better solution would be to just have some API (be it json or xml or s-exp) which allows accessing the ghost posts programmatically, so they can be easily archived (so as to not force people to archive your archive and copy your posts).
There would still be a problem in the case of conflicting ghostpost numbers (post being identified as: *chan-board/postid{,ghostid}, where postid is what follows after "No.", and ghostid is a number starting from 1 (seen on ghostposts). In this case, you might have to name the posts as *chan-board/postid{,ghostid}{,ghostsource} where ghostsource is the source archiver from where the post was retrieved. ghostid is not included for original 4chan posts and ghostsource is not included when ghostsource corresponds to the archive.

>>8053971
> (so as to not force people to archive your archive and copy your posts).
By that I meant by parsing the output HTML thus straining your resources (bandwidth, memory, cpu time).
Having an API would reduce strain when archives are mirroring each other's content.

>>8053971
True enough, but it would still mean there needs to be a main ghost server where the posts come from, unless you really mean to make a script that checks a whole network of archives. I would still expect that to slow things down, maybe even worse depending on their strain at the time, unless it's ajax or something that modifies the page after it has loaded.

>>8053994

Why would there be a "main server" for posts to originate from? That would mean everyone is viewing the board from one site and using up the bandwidth of one archive. Just have them broadcast posts to each other when they're received.

Regardless, none of this is going to happen because there aren't this many people running archives.

>>8053994
The idea would be that each ghost board would provide an API for downloading sets of ghost posts from their own board (also, optionally other archived posts, but this option may be disabled if needed).
The archives would then perform the usual archiving of the board, but also ocassionaly download compact, compressed ghostpost updates via the API from other archives (which you'd have to list in some configuration file). Posts from other archives would be suffixed by the ghostboard of origin.
I think this is symmetric and simple enough - the most popular ghostboard would just look like easymodo looked, but if someone makes a post on another board, the post would get automatically added to the ghostboard and tagged accordingly (with ,ghostboard). On less popular archives, I'd be common to see the tag of the more popular archives(s), instead of untagged posts (unlike the popular one).
This would probably be not too hard to implement in fuuka, but my Perl isn't too good, so someone else would probably have to do it.

>>8054002
>Just have them broadcast posts to each other when they're received.
Oh of course. Silly me. Yes, if they're periodically checked and then stored locally for each it'd be fine. That would work.

I don't imagine it'd be even very difficult to implement, but I can't into perl and it'd be silly to have a single script running in some other language.

>>8054004
RSS could work

http://easymodo.net/archive.html
archive dump is up.

Just wanted to thank you, VDZ. It's funny, when I noticed easymodo was down my first impulse was to check the archive to see whether there was already a thread about it... that shit is vital and I'm really glad a replacement archive is up.

Could someone re-upload the jp archive files? I'm interested in getting a copy but I don't want to waste Eksy-kun's bandwidth.