The deal with the thread pruning

[Citizen Bleys]

Alright; so you're probably wondering what the thread deletion has to do with today's downtime. It's a bit of a lengthy (and somewhat technical) explanation, so I thought it best to leave it out of the announcement in order to keep it brief.

All I tried to do this morning was upgrade the boards (completed in about two minutes) and rebuild the search index in order to improve speed. (The search index is used every time you open a thread, not just during searches). The first step to rebuilding the search index is to delete the old, obsolete index. Thus, once the rebuild is started, it MUST be allowed to procede to full completion; Monkeying around in threads while the rebuild is going on can lead to data corruption (I'm posting this during the rebuild, but I know precisely how far along the rebuild is coming: It's handling threads 11900-11925, and this is thread 20352, so I'm safe).

Rebuilding the index should take five minutes. Dunno if you noticed, but it's taking a bit longer.

Our database is now so huge that the rebuild keeps timing out. There are simply too many old, dead threads cluttering up the works. This has to be done. I've not consulted Scottie or any of the other staff about this--it has to be done, even if everybody doesn't see it.

vBulletin will allow me to prune threads by date of last post--hence, I'll be deleting anything in which the last post was more than 90 days ago. However, it runs through ALL forums, including the archive forums. I don't know a workaround (if you do, please post a reply in this thread ;p).

I'll do a full backup of the database, but for security reasons*, I cannot release the backup to the general public for download

*you shouldn't have to think about that very hard about that one. Everybody's password is stored in the database.

What I'd like to know from you, is
a) How often would most of you be able to handle a pruning operation? Once every month? Once every three? It's got to be done sometime.
b) do you know of any alternatives--Maybe an app that will save the threads? It should be simple enough to write--Just do an SQL dump of the table and then write a php script to parse it...But it's got to be a server-side app, and I have to have access to the source code, so that I can be sure that ALL it's taking from the database is the threads to be saved. We (EoFF staff) are not going to allow even the slightest chance that somebody's password will be taken out of the database.

[Griever]

I think once every two months would be sufficient.
As for the passwords, i think everyone'll just have to change them more often. Just regular changes, some people might accidently let them slip during a conversation or something. I did it once, but i was stupid back then.

[Super Christ]

Hmm - I have no luck finding any such utility or hack. Perhaps if, just before the pruning, an administrator adds one post to the end of each thread in at least the General Archive forum, and then after the pruning, deletes them. Just a thought. As far as threads in other forums, while there's a thread or two I'll miss, I won't miss them enough to really worry about it. Ahh well, that was just a thought.

[Shlup]

Thank you, Bleys. =^_^=

[Nait]

Of course it is sad.

Oh, well. C'est la vie.

[Squally Leonharty]

Hmm, I think there's a way to save the archived threads. I'm not too sure, though.

I believe there's an option to prune threads PER forum, so why don't you do that? It's more work, of course, but at least you're able to save the Archive forum.

I think it's going to take you about 30 - 45 minutes to do all of the forums. Just that in one day. It's not that bad. It has to happen someday, right? Well, yeah... I got an idea how to backup the Archive forum, but I don't think it's possible to write a script for it, since the mysqldump doesn't allow partial dumps of tables (meaning certain threadids and posts). It only dumps the whole tables. I'll look in backup.php and try to get the code from there and edit it.

Edit:

Bleh, I skimmed through the whole code and it looks like it is possible after all. But it's gonna be very hard, and I can't really bother with that. I'm not THAT skilled. Oh well. I guess you'll have to prune per forum.

[Citizen Bleys]

And Squally says the obvious, and it hits me in the head like a hammerblow after a night of hard drinking. Of sake.

Yeah, we can prune by forum. *sheepish look*

[Squally Leonharty]

*makes fun of Bleys*

Just kidding. At least you know it now. Have fun with pruning.

[The Man]

Is there any way to save the old threads that get pruned into some external archive so that people who really want to view them can do so? It'd be a shame to lose them all for good.

But yeah, if the index is that messed up, it's probably a necessity.

Peace
The Man

[Squally Leonharty]

Well, that's possible, The Man. Bleys will skip the Archive forum, so if there are any threads that needs to be saved, an admin needs to move them to the Archive forum. It's that simple.

But yeah, it all depends on the administrators.

[The Man]

Periodically, the staff decide that there are more threads that need to get archived, and if those threads are deleted, there won't be a chance to do that any longer. *shrug*

Well, dunno. One thing I really liked about the boards is that you could go back a year and still have an historical reference point. I was just wondering if there would be any way to save that for posterity... but y'know.

Peace
The Man

[Rainecloud]

If you really love some threads, and don't want them to die, why not save them to your hard drive? Who knows, there may even be a way to integrate them back into the board after you do this. I'm not sure whether that will work though, as I know nothing about VB.

If it has to be done to keep EoFF working, then go right ahead, Bleys.

[Dr Unne]

PLEASE Bleys, make a huge board-wide announcement before you prune, and give at least one or two week's notice, OK? We lost some good threads last time the MB was pruned, possibly because people didn't know about the pruning before you did it. I sure didn't know about it. I'm not sure if I was around at the time though. A good idea might be to make an announcement and possibly a thread for people to post links to threads they want moved to the Archives. That'd give everyone a chance to save the threads they want on the MB, rather than downloading the HTML to their hard drives or whatever.

I think once every 6 months is about as often as I could handle a pruning. Not sure if we need to do it more often than that.

If you can prune by forum, that'd be good. Some forums you wouldn't even need to touch, maybe. FF1-4 forum for example. Certainly not the Archives. I hope you make sure the Archives don't get pruned. Prune the heck out of General Chat though, that forum is huge and spam-filled.

[The Man]

I agree completely with Unne. In fact I wouldn't touch any of the gaming fora. The fact is, in the gaming fora people often link back to past threads that have been dead for a long time, and were those fora to be pruneed, we would no longer be able to do that. There is a forum-wide announcement, but I think there should be a few more weeks before the pruning happens, and I agree with the whole "submit-links-to-your-favourite-threads-for-archiving" dealy.

Also, if the Open Source forum gets touched, I will KILL YOU. Well, not really. But still.

Edit: Maybe the Daily Grind threads should be saved as well. And maybe Bleys should make a backup of the files before the actual pruning happens, and put them online... somewhere.

Peace
The Man

[Nait]

And the Creative Fora! Save the Artist Galleries and Writer's Corners!