Archive of UserLand's first discussion group, started October 5, 1998.

A by-product of archiving in XML

Author:Dave Winer
Posted:8/15/1999; 8:03:42 AM
Topic:A by-product of archiving in XML
Msg #:9540
Prev/Next:9539 / 9541

I've been working on a more efficient archiving system for our discussion groups, it logs every change to every message by writing an XML file to calendar-structured folder hierarchy.

I wanted this for two reasons -- first, every night we're copying the whole discss.root database to back it up. This is quite wasteful, esp as we approach 10,000 messages, on any given day, most of them don't change, but we copy them anyway.

The second reason, with every new message we're pushing Frontier's object database into uncharted territory. I must say I am very pleased with the reliability of the odb, on the path to 10K messages there hasn't been a single database glitch.

Dear Murphy

If you're listening, I know you're crashing the bits right now, as I write this.

Anyway

Because the track record has been so perfect, I am bothered because I don't want to fall into complacency. At some time there will be a problem with the database, and to be sure the message base survives such a problem, I wanted a second-source for recovery in case the first one (rebuilding discuss.root) doesn't work. After revisiting MORE with its binary file format, I want a text-based format to work with for the future.

It was easy to hook into the archive flow and produce a folder of static XMLizations of all the messages in the odb. This is not our archive, it's a bit fancier than this, but it is a by-product of the archiving.

http://linux2.userland.com/users/admin/static/mainDiscussMessages/

Every time we do an XMLization it stimulates some ideas, if you are interested in this stuff, have a look, and post ideas here, if they're bugging you.

Caveat

I can already see that the system wheezes with a folder with so many files in it, so we possibly will add a hiearchy here.

There are responses to this message:

This page was archived on 6/13/2001; 4:51:55 PM.

© Copyright 1998-2001 UserLand Software, Inc.