Archive of UserLand's first discussion group, started October 5, 1998.
Progress report on My.UserLand
Author: Dave Winer Posted: 9/29/1999; 5:24:36 AM Topic: Msg #: 11589 (In response to 0) Prev/Next: 11588 / 11590 Zeroing in
Almost everything is working now. But since one of the goals of this transition was transparency, the changes might not be apparent.
As I finish the transition, which is taking longer than I hoped it would, I'm finding that I can explain what's going on. Perhaps that has something to do with things working!
A more logical split in functionality
This is the second rewrite of the My.UserLand functionality.
The first time thru, all the functionality was on a single server. It was both the user interface and the aggregation engine. Having it all on a single server made the programming pretty easy.
The first rewrite offloaded the scanning of the XML files. It would drop the resulting files in a folder and then send a message to the main server saying "start churning". Again, most of the functionality was on a single server. It helped performance, somewhat, to get the tcp.httpClient calls running on a different server, but it didn't make it easy to distribute functionality.
Aggregator and affiliates
This rewrite makes the split along different lines. Now there is an aggregator and an affiliate. The interface is implemented in XML-RPC, and it's going to be open and documented, so that affiliates can be written in any programming environment that can make XML-RPC calls and field them. You need to be both a client and server to implement an affiliate.
The aggregator does all the scanning and all the churning. The affiliate receives an XML-RPC message when the hourly churn is finished, containing the new stories.
The affiliate puts them in a local database, maintains a browsable channel list, and allows users to choose which channels they want to receive stories from. It's like a client-server split, but the client in this case is also a server, so that's why I call it an affiliate.
There are lots of different kinds of affiliates possible. Mail agents, search engines, even simply including headlines on your home page from sites you want to associate with. I'm thinking of putting hourly headlines on Scripting News from sites like Tomalak, Salon, Wired, Red Herring, MacWEEK, XML.COM, Perl, whatever. I like the association with these brands, and they get the benefit of gaining space on my home page, and getting the flow that generates.
affiliate.root
We will do a source release of the affiliate software for Frontier, so all Frontier-managed sites can have this capability. The timeframe is still undecided. As we've been doing this work we've been doing burn-ins at every level, sometimes lasting as long as ten days. It's absolutely essential to be certain that the servers can handle the load, and that we catch bugs and design inefficiencies before depending on the software. We've learned in the past that if we go too fast in releasing software the end result is that we move more slowly. For example, I did a release of the first version back in April, and then promptly broke it because we learned more about the problem and recoded the software to reflect that learning. We want not just Frontier people to do affiliates, we will open this up, but we will go slowly.
It's a time-burner
Imagine debugging software that depends on changes made in other people's websites!
Imagine debugging software that runs once an hour. If you have a theory about why something is going wrong, look at the clock. If it's close to the top of the hour, hurry up. If it's just after the hour, you'll have to wait. You can pass days and days working on this stuff, because time is an important part of the equation.
It's groupware. Luckily the environment has been pretty solid. There have been a couple of Frontier mysteries, but for the most part it's been behaving like a Turing Machine. Garbage in, garbage out.
My partner is Murphy!
Where we're at
My.UserLand is now completely running off the aggregator.
There are glitches for sure, channels that are not updating, but so far the problems all seem to be related to bad URLs in the database on the aggregator. There are a small number of these, probably less than ten. We'll work with the channel providers to get these glitches out.
New channels are not showing up in the channel list on the affiliate. This is the next problem I'm going to work on. Probably will have it working in the next hour or so. I have all day Thursday blocked out for working out glitches. On Friday I'm going on a day trip to Sebastapol to visit with O'Reilly people. We're partnering with them on the next revs of this stuff.
Next week I hope to be working on some of the new ideas re taxonomy and categorization. Let's reserve next week for burning-in on the aggregator-affiliate connection, and if all is well, we'll start distributing the affiliate software the following week, the week of October 11.
I expect that we'll document the XML-RPC interface before then so that people working in other environments can be ready to hook up to the aggregator when we open it publicly, so it's not just a Frontier-Frontier thing.
Questions are welcome
I will answer them to the extent that I can.
Thanks!
This page was archived on 6/13/2001; 4:52:52 PM.
© Copyright 1998-2001 UserLand Software, Inc.