Archive of UserLand's first discussion group, started October 5, 1998.

Response to Flutterby

Author:Dave Winer
Posted:10/23/1999; 5:35:09 AM
Topic:Response to Flutterby
Msg #:12280
Prev/Next:12279 / 12281

Quote from Flutterby:

From Dan Lyke on Flutterby, 10/12/99: "Dave Winer has discovered why reinventing protocols is a bad idea, he's running into problems with his content distribution system built on top of XML-RPC because his protocols don't take into account a lot of the issues that SMTP and NNTP deal with effectively. This wouldn't be noteworty except that this sort of arrogance pops up fairly frequently, developers slap together something without thinking through the ramifications and it ends up biting them. I'd have thought that Dave's been in the business long enough to know this, but he seems to be remarkably short-sighted."

He was referring to this post.

Comments

So, I gave this a few hours to settle in, and I wondered if Flutterby had come to the correct (albeit harsh) conclusion. I don't agree that I have discovered that reinventing protocols is a bad idea! I took a deep look at that statement, to see if it was true or not. It's not true.

Could I have used SMTP or NNTP to distribute stories? Yes, I could have, possibly, done that. However, if I had used either of those two protocols, I would have had the same problem on my server. (I am using HTTP, btw.) The problem had nothing to do with the protocol, it has to do with bugs in the encodings of the XML files the aggregator reads. What protocol goes over the wire is not related to the problem I was backing-off from.

Dan would have no way of knowing this, of course, and probably would have no way of knowing that XML-RPC is serving us very well in building out our network, in other areas that don't get as much attention. For example, it's excellent for doing prefs-distribution. And for updating Frontier users. And for managing the connection between our search engine and our content management system.

Re the affiliate-aggregator connection, what was needed was a partnership, and some time. Just setting up the server and connecting it to random affiliates was not a good development plan. I've been working one-one with Jon Udell to connect his server and mine, using XML-RPC, and we've worked out a bunch of glitches, that mostly have to do with differences between our XML parsers. Encoding has proven to be a big problem, and since the material is coming to us in XML (thru RSS and ScriptingNews formats) and people are writing buggy XML (the encoding attribute on the header is often wrong!). The shit was hitting the fan deep in the bowels of whatever scripting environment was implementing the affiliate. When the affiliate is Frontier, no problemmo. When it's Zope -- big problemmo!

However, I am pleased to report, that as of yesterday, due to some hard thinking by Jon and Peter Wiggins at O'Reilly, we have found a good solution, we're entity-encoding all 8-bit characters, so that no matter what, the content will make it thru the parsers, and we can deal with the problems in scripts. The problem was that the scripts were never getting control because the environment was rejecting the text. To understand this problem, you have to think about the nature of the aggregator. We are creating a new XML-based feed from 400+ other XML feeds. NNTP or SMTP never had to deal with the issues this situation creates.

Anyway, how does this turn into a problem with how Frontier handles asynchronous messages? Well, our queue would back up, with dozens of errored-out messages in the queue. We would age them and take them out of the queue after 10 errors. But the problem, when I took the step-back, was that almost none of them were going thru without error, when the affiliate was not Frontier. I didn't know why.

So the reason for my step-back was that I wanted to take a deeper look, learn why it wasn't working, and then come up with a workaround. Unfortunately, I had moved on, I was working on the release of Frontier 6.1, and wasn't able to get back to this project until yesterday.

Arrogance?

I've seen people do this with the Web Hosting protocol we published a week ago. One critic said it's just like FTP. Look deeper. It does something FTP can't do. Think of it as a puzzle to test your understanding of the web. Maybe I didn't spell it out (I'm working so hard my body hurts), but don't jump to the conclusion that I don't know what I'm doing. Maybe, consider the possibility, that I understand something that you don't. FTP was invented before the web. It doesn't know about the web. Our purpose is to make writing for the web easy. We need a way to store pages on static sites that doesn't require the user to remember a pair of addresses that are equivalent. We insist that software do that for the user. The server that WHP defines is a simple bit of software that knows how to store data (like FTP) but also returns the HTTP url for the stored information. And it also provides support for a content-based search engine, and there's room for growth for other post-FTP-level storage services. It would be a really good idea for people to look deeper at this stuff now. Like XML-RPC last summer, a lot of people are sniffing. A year from now you'll be angry with me because I went forward without waiting for you. But it won't be my fault if you don't look and think. We have not deployed the WHP server on anything but a test basis now. It's easy to change what we're doing. Later it will not be easy.

How to deal with harsh criticism

Software is not really something to get personal over. I decided to respond to Flutterby with politeness, because he comes highly recommended by someone who I have deep respect for, Wes Felter.

I read Wes's Hack The Planet weblog every day, and Wes lists Flutterby at the top of his home page as "one degree of separation" from his site, along with other sites I respect, including Tomalak and my own Scripting News. Wes is one of the few people in the weblog world who points to my site with respect. Most of the pointers are in some way dismissive, and in many cases people don't point to my site.

Oh well. I'd like to be part of this ring of weblogs, so I offer my respect to Dan because he has the respect of someone who I respect. There only two steps between us. Let's try to work together. If Dan has some input that's helpful to my work, I want to acknowledge that. It's good for me to question the design of the network we're building, so for that I am thankful to Flutterby.

Peace!



There are responses to this message:

This page was archived on 6/13/2001; 4:53:10 PM.

© Copyright 1998-2001 UserLand Software, Inc.