on uploadChangedFiles () { «Changes «3/28/03; 4:46:16 PM by JES «If system.temp.radio.misc.flFullUpstreamScanNextPass is true, scan all folders. «3/20/03; 6:27:15 PM by JES «Track when a folder last contained a file which was upstreamed. If no files in a given folder have been upstreamed for a minute, only scan the folder once a minute. If no files have upstreamed for an hour, only scan the folder once every 5 minutes. «Scans fan out over the 5 minute or 1 minute period to minimize instantaneous CPU usage. «1/22/03; 3:21:33 PM by JES «If the serial number is expired, log an error to the Events page if upstreaming to UserLand's server fails as a result. «12/21/02; 4:13:17 PM by JES «Added support for serial number renewals. «6/23/02; 11:17:34 PM by JES «If a file's modification date is in the future, set adrfile^.whenLastUpstream to the file's mod-date, instead of to now. Prevents files with future mod-dates from being repeatedly upstreamed. «2/1/02; 1:59:54 PM by JES «Upload a maximum of 25 files per pass. «1/31/02; 3:05:23 AM by JES «Skip busy files until the next time through the loop. (A busy file is most likely opened with write permissions by another application.) «1/29/02; 3:29:30 AM by JES «Break upstreaming passes up into 128KB blocks. (The last file that breaks the 128KB limit is included with the pass.) «1/7/02; 11:41:41 AM by JES «Don't upstream files whose name begins with '.' or is contained in a folder whose name begins with '.'. «12/12/01; 8:38:29 PM by DW «Don't upstream if the file is contained within a folder whose name begins with a #. «12/11/01; 9:38:47 AM by DW --Tweaks and tuneups after rewrite. «Trick the upstreamer into doing "index" first. «The goal is to make it more likely that when you click on Home on the DWHP that it will already be updated. If index is moved to the front of the list, that becomes more likely. Why should it wait for the other two files? It shouldn't. «Tell the thread not to go to sleep if directory.opml needs to be written «See comment at head of radio.thread.script. «12/10/01; 9:20:24 AM by DW -- Total rewrite. «The main loop now fills a table with lists of files that need to be upstreamed. «Each list is actually a table (for performance). All files in a given list are going to the same place. «Then in a separate pass iterate over the lists, send all the files in a given list in one shot. «This required a rearchitect of the upstream drivers. (Better now than later.) «12/9/01; 7:43:57 AM by DW «In the main loop, instead of checking against the modified file attribute, check against the upstream.whenLastUploaded attribute. «12/8/01; 5:24:00 PM by DW «If there was a change to a #upstream.xml, mark all the files under its control as needing upstreaming. «12/7/01; 9:24:29 AM by DW «Optimization -- after upstreaming, be sure the file isn't in recentlyWrittenWwwFiles, avoid upstreaming it twice. «12/6/01; 10:34:54 AM by DW «Re-coded for performance. «bundle //old code «on dofolder (folder) «local (f, lastfileurl = "") «fileloop (f in folder) «thread.sleepTicks (0) //PBS 11/16/01: more background-friendly //PBS 11/20/01: interferes with debugging «if file.isVisible (f) //skip invisible files -- icon files on the Mac, in particular «local (fname = file.filefrompath (f)) «if not (fname beginswith "#") «if file.isfolder (f) «lastfileurl = dofolder (f) «else «local (extension = string.nthfield (fname, ".", string.countfields (fname, "."))) «radio.file.getFileAttributes (f, @adrfile) «if file.modified (f) > adrfile^.upstream.whenLastUploaded «upstreamfile (adrfile) «adrfile^.upstream.whenLastUploaded = now //only try once «lastfileurl = adrfile^.upstream.url «local (folderurl = "") «bundle //set the folder's url «if lastfileurl != "" «local (s) «bundle //set s «local (ix) «s = lastfileurl «if s endswith "/" «s = string.delete (s, sizeof (s), 1) «for ix = sizeof (s) downto 1 «if s [ix] == "/" «s = string.mid (s, 1, ix) «break «local (adrfolder) «radio.file.getFileAttributes (folder, @adrfolder) «adrfolder^.upstream.url = s «folderurl = s «return (folderurl) «local (now = clock.now ()) «dofolder (user.radio.prefs.upstream.folder) «12/4/01; 2:41:59 PM by JES «If there's an error uploading a file, record it in flError, in the upstream sub-table of the filetable, instead of at the top-level of the filetable. If there's no error, set adrfile^.upstream.flError to false. «12/1/01; 10:55:59 PM by JES «Fixed a bug where directory.opml would only be updated when files in the same folder as the upstream spec were modified. «11/17/01; 3:11:23 AM by JES «When making log entries, the text of the link is the upstreamed filename, instead of the on-disk filename. «11/16/01; 8:24:21 PM by PBS «To make this more background-friendly, call thread.sleepTicks (0) in the fileloop. «11/13/01; 2:24:05 AM by JES «Set an item in temp.radio.recentlyChangedFolders.[adrfile^.baseUpstreamFolder], when a file is upstreamed. This triggers the re-writing of the directory.opml file for the upstream folder. «Note: This item is *not* set for files named directory.opml, which live in the baseUpstreamFolder. Don't change that, or you'll get an infinite loop of directory.opml's upstreaming into the cloud. «9/28/01; 2:25:56 PM by JES «Call callbacks when a file is upstreamed. First call callbacks at radio.upstream.callbacks.upstream, and then call user callbacks at user.radio.callbacks.upstream. «9/5/01; 12:08:59 AM by JES «Lock a semaphore on the file path while processing. This enables us to stay out of the way of weblog Theme application, and possibly others in the future. «7/24/01; 1:20:23 AM by JES «Don't attempt to upstream invisible files. This avoids upstreaming icon files on MacOS. «6/19/01; 10:03:02 AM by DW «Rewrite. Out with the two passes. Implement #upstream.xml. local (flDebug = false); local (ticks = clock.ticks ()); if flDebug { //debugging code if not defined (scratchpad.folders) { new (outlineType, @scratchpad.folders)}; if not window.isOpen (@scratchpad.folders) { edit (@scratchpad.folders)}; target.set (@scratchpad.folders); op.insert (clock.now (), down)}; on logadd (htmltext) { if user.radio.prefs.upstream.logging { radio.log.add ("Upstream", htmltext, startticks)}}; on runCallbacks (filelist, adrcallbacktable) { local (nomad); try { for nomad in adrcallbacktable { while typeOf (nomad^) == addressType { nomad = nomad^}; try {nomad^ (filelist)}}}}; //site of an outage -- 12/6/01; 7:49:55 PM by DW local (maxBytesPerPass = 128 * 1024); //128KB local (maxFilesPerPass = 25); local (now = clock.now ()); local (streams); new (tabletype, @streams); bundle { //pass 1 -- fill streams with files that changed, sorted by stream local (f, pc = file.getPathChar (), ctbytes = 0, ctfiles = 0); on visitFile (f) { local (whenModified = file.modified (f)); local (flupstream = true); try { if whenModified <= user.radio.settings.files.[f].upstream.whenLastUploaded { flupstream = false}}; if flupstream { ctbytes = ctbytes + file.size (f); local (adrfile); radio.file.getFileAttributes (f, @adrfile); bundle { //optimization «12/9/01; 7:43:57 AM by DW «If it's a new file make sure it has an entry in the cache, makes future scans faster. Inside the "try" above, it is looking for the cache element to exist for the file. Consider the case of an invisible file (doesn't happen often) or a #file (more often) -- they should be in the cache too, with a "whenLastUploaded" field set, so we can dispose of them in future scans without having to see if they're visible or have a name that begins with a #. if now < whenModified { adrfile^.upstream.whenLastUploaded = whenModified} else { adrfile^.upstream.whenLastUploaded = now}}; if file.isvisible (f) { if not file.isBusy (f) { //1/31/02 by JES: skip this file until it's not busy (opened with write permissions) if not (f contains (pc + ".")) { //01/07/02 by JES: files/folders that start with '.' are invisible if not radio.upstream.inPoundFolder (f) { //12/12/01 by DW local (name = file.filefrompath (f)); if name beginswith "#" { if string.lower (name) == radio.data.fileNames.upstreamFileName { radio.upstream.folderNeedsUpstream (file.folderfrompath (f))}} else { local (adrspec); if radio.upstream.getUpstreamSpec (adrfile, @adrspec) { local (adrstream = @streams.[nameof (adrspec^)]); if not defined (adrstream^) { new (tabletype, adrstream); adrstream^.adrspec = adrspec; adrstream^.files = {}}; adrstream^.files = adrstream^.files + {adrfile}; if ctbytes >= maxBytesPerPass { break}; if ++ctfiles >= maxFilesPerPass { break}}}}}}}}; return (flupstream)}; on visitOneFolder (folder) { bundle { //check for upstream spec of type="none" if file.exists (folder + radio.data.fileNames.upstreamFileName) { local (s = file.readWholeFile (folder + radio.data.fileNames.upstreamFileName)); try { //this should never fail due to a faulty upstream spec local (xstruct); xml.compile (s, @xstruct); local (adrupstream = xml.getAddress (@xstruct, "upstream")); if adrupstream^.["/atts"].type == "none" { return (false)}}}}; local (folderChanged = false); local (adrfolder); radio.file.getFileAttributes (folder, @adrfolder); if not defined (adrfolder^.upstream.whenLastScanned) { adrfolder^.upstream.whenLastScanned = date.yesterday (clock.now ())}; if adrfolder^.upstream.whenLastScanned > now { //reality check adrfolder^.upstream.whenLastScanned = date.yesterday (now)}; local (secsSinceLastScan = number (now - adrfolder^.upstream.whenLastScanned) ); local (secsSinceLastUpload = number (now - adrfolder^.upstream.whenLastUploaded) ); local (flScanThisFolder = system.temp.radio.misc.flFullUpstreamScanNextPass); if not flScanThisFolder { //set flScanThisFolder bundle { //more reality checks if secsSinceLastUpload < 0 { //workaround for date calculation from date(0) adrfolder^.upstream.whenLastUploaded = date.yesterday (now); secsSinceLastUpload = number (now - adrfolder^.upstream.whenLastUploaded)}; if secsSinceLastScan < 0 { //workaround for date calculation from date(0) adrfolder^.upstream.whenLastScanned = date.yesterday (now); secsSinceLastScan = infinity}}; if folder == user.radio.prefs.upstream.folder { //always scan files in the top level folder flScanThisFolder = true} else { //based on how long it's been since the last scan if secsSinceLastUpload >= 60 { if secsSinceLastUpload >= 3600 { //1 hour or more -- only scan once every 5 minutes if secsSinceLastScan >= 300 { flScanThisFolder = true}} else { //between one minute and one hour -- scan once each minute if secsSinceLastScan >= 60 { flScanThisFolder = true}}} else { //less than one minute since last upload -- scan it flScanThisFolder = true}}}; if flScanThisFolder { if flDebug { //debugging code op.insert (folder + " (" + secsSinceLastScan + ", " + secsSinceLastUpload + ")", right); op.go (left, 1)}; bundle { //set whenLastScanned for this folder «JES 3/20/03: Why do we subtract a random number of seconds? Explanation below... «Consider the case where a full scan takes less than ten seconds: «If we always throttled by the same amount of time, we'd end up with a big CPU hit at the top of each minute, and an even bigger one every five minutes, since all throttled folders would be synchronized. «So each time a folder is scanned, we schedule it for re-scanning a few seconds earlier, so that over time the re-scans will spread out through the whole minute or 5-minute block. if secsSinceLastScan <= 60 { adrfolder^.upstream.whenLastScanned = now + (secsSinceLastScan % 60) - random (1, 5)} else { adrfolder^.upstream.whenLastScanned = now + (secsSinceLastScan % 300) - random (1, 60)}}; local (ct = 0); fileloop (f in folder) { ct++; if file.isFolder (f) { if visitOneFolder (f) { folderChanged = true}} else { if visitFile (f) { folderChanged = true}}; if (ct % 5) == 0 { //every 5th file, yield some CPU thread.sleepTicks (6)}}}; if folderChanged { adrfolder^.upstream.whenLastUploaded = now}; if adrfolder^.upstream.whenLastUploaded > now { //reality check adrfolder^.upstream.whenLastUploaded = now}; return (folderChanged)}; visitOneFolder (user.radio.prefs.upstream.folder); system.temp.radio.misc.flFullUpstreamScanNextPass = false}; «scratchpad.streams = streams «streams = scratchpad.streams //for debugging bundle { //pass 2 -- iterate over streams table, routing the changed files to their proper destinations local (startticks); local (adrstream, adrspec, type, adrdriver); for adrstream in @streams { startticks = clock.ticks (); adrspec = adrstream^.adrspec; bundle { //expiration check -- if expired, don't upstream to UserLand if defined (user.radio.settings.whenSNExpires) { local (whenexpires = date (user.radio.settings.whenSNExpires)); if whenexpires < now { case string.lower (adrspec^.server) { "rcs.salon.com"; "radio.xmlstoragesystem.com" { logadd ("Can't upstream to " + adrspec^.server + " because your serial number is expired."); continue}}}}}; type = adrspec^.type; if not radio.upstream.findDriver (type, @adrdriver) { if string.lower (type) != "none" { logadd ("Can't upstream because there is no driver for type \"" + type + "\".")}; continue}; bundle { //trick the upstreamer into doing "index" first. local (adrfile, ix = 1); for adrfile in adrstream^.files { if string.lower (file.filefrompath (nameof (adrfile^))) beginswith "index" { if ix != 1 { local (adrtmp = adrstream^.files [1]); adrstream^.files [1] = adrfile; adrstream^.files [ix] = adrtmp}; break}; ix++}}; try { local (response); bundle { //set relativePath for each file local (adrfile); for adrfile in adrstream^.files { local (s = string.delete (nameof (adrfile^), 1, sizeof (adrfile^.baseUpstreamFolder))); s = string.replaceAll (s, file.getpathchar (), "/"); adrfile^.relativePath = s}}; adrdriver^.upstreamMultipleFiles (adrstream^.files, adrspec, @response); bundle { //set an item in system.temp.radio.recentlyChangedFolders to trigger rebuild of directory.opml local (adrfile); for adrfile in adrstream^.files { if string.lower (file.filefrompath (nameof (adrfile^))) != string.lower (radio.data.upstream.directoryFileName) { system.temp.radio.recentlyChangedFolders.[adrfile^.baseUpstreamFolder] = clock.now (); system.temp.radio.misc.flThreadSleeps = false}}}; //12/11/01; 10:23:43 AM by DW bundle { //recording info in file tables, log the upstream local (url, ct = 1, adrfile, htmltext = ""); for url in response.urllist { adrfile = adrstream^.files [ct++]; if url == "" { //error on the server processing this file adrfile^.upstream.flError = true; htmltext = htmltext + "" + string.nthfield (url, "/", string.countfields (url, "/")) + ", "} else { //no error on this file adrfile^.upstream.ctUploads++; adrfile^.upstream.url = url; adrfile^.upstream.flError = false; htmltext = htmltext + "" + string.nthfield (url, "/", string.countfields (url, "/")) + ", "}}; htmltext = string.mid (htmltext, 1, sizeof (htmltext) - 2) + "."; //pop off extra comma and space, add period if response.flerror { htmltext = htmltext + " The server reported an error: " + response.message}; if sizeof (response.urllist) == 1 { htmltext = "1 file: " + htmltext} else { htmltext = sizeof (response.urllist) + " files: " + htmltext}; logadd (htmltext)}; bundle { //set the parent folder urls local (adrfile); for adrfile in adrstream^.files { local (s, adrfolder); radio.file.getFileAttributes (file.folderfrompath (nameof (adrfile^)), @adrfolder); bundle { //set s local (ix); s = adrfile^.upstream.url; if s endswith "/" { s = string.delete (s, sizeof (s), 1)}; for ix = sizeof (s) downto 1 { if s [ix] == "/" { s = string.mid (s, 1, ix); break}}}; adrfolder^.upstream.url = s}}; bundle { //run callbacks local (filepathlist = {}, adrfile); for adrfile in adrstream^.files { filepathlist = filepathlist + {nameof (adrfile^)}}; if defined (radio.upstream.callbacks.upstream) { runCallbacks (filepathlist, @radio.upstream.callbacks.upstream)}; if defined (user.radio.callbacks.upstream) { runCallbacks (filepathlist, @user.radio.callbacks.upstream)}}} else { logadd ("Can't upstream because \"" + tryerror + "\"")}}}; «bundle //pre 12/10/01 code «local (startticks = clock.ticks (), ctupstreams = 0, htmltext = "") «on upstreamfile (f) «local (adrfile, adrspec) «radio.file.getFileAttributes (f, @adrfile) «adrfile^.upstream.whenLastUploaded = clock.now () //mark as upstreamed, don't retry errant uploads until they change again «if not radio.upstream.getUpstreamSpec (adrfile, @adrspec) «return (false) «bundle //upstream the file, according to the spec «local (adrdriver) «if not radio.upstream.findDriver (adrspec^.type, @adrdriver) «return (false) «bundle //set the relative path «local (s = string.delete (f, 1, sizeof (adrfile^.baseUpstreamFolder))) «s = string.replaceAll (s, file.getpathchar (), "/") «adrfile^.relativePath = s «try «adrdriver^.upstream (adrfile, adrspec) «adrfile^.upstream.flError = false «adrfile^.upstream.ctUploads++ «bundle //set an item in system.temp.radio.recentlyChangedFolders to trigger rebuild of directory.opml «if string.lower (file.folderFromPath (f)) == string.lower (adrfile^.baseUpstreamFolder) «if string.lower (file.filefrompath (f)) != string.lower (radio.data.upstream.directoryFileName) «system.temp.radio.recentlyChangedFolders.[adrfile^.baseUpstreamFolder] = clock.now () «if defined (radio.upstream.callbacks.upstream) «runCallbacks (adrfile, @radio.upstream.callbacks.upstream) «if defined (user.radio.callbacks.upstream) «runCallbacks (adrfile, @user.radio.callbacks.upstream) «local (upstreamFname = string.nthField (adrfile^.upstream.url, "/", string.countFields (adrfile^.upstream.url, "/"))) «htmltext = htmltext + "" + upstreamFname + ", " «ctupstreams++ «else «adrfile^.upstream.flError = true «htmltext = htmltext + file.filefrompath (f) + "(Error = " + tryError + "), " «bundle //set the parent folder url «local (s, adrfolder) «radio.file.getFileAttributes (file.folderfrompath (f), @adrfolder) «bundle //set s «local (ix) «s = adrfile^.upstream.url «if s endswith "/" «s = string.delete (s, sizeof (s), 1) «for ix = sizeof (s) downto 1 «if s [ix] == "/" «s = string.mid (s, 1, ix) «break «adrfolder^.upstream.url = s «bundle //make sure file isn't in recentlyWrittenWwwFiles, don't upstream twice «try {delete (@system.temp.radio.recentlyWrittenWwwFiles.[f])} «return (true) «local (f) «fileloop (f in user.radio.prefs.upstream.folder, infinity) «local (flupstream = true) «try «if file.modified (f) <= user.radio.settings.files.[f].upstream.whenLastUploaded «flupstream = false «if flupstream «bundle //optimization «If it's a new file make sure it has an entry in the cache, makes future scans faster. Inside the "try" above, it is looking for the cache element to exist for the file. Consider the case of an invisible file (doesn't happen often) or a #file (more often) -- they should be in the cache too, with a "whenLastUploaded" field set, so we can dispose of them in future scans without having to see if they're visible or have a name that begins with a #. «local (adrfile) «radio.file.getFileAttributes (f, @adrfile) «adrfile^.upstream.whenLastUploaded = clock.now () «if file.isvisible (f) «local (name = file.filefrompath (f)) «if name beginswith "#" «if string.lower (name) == radio.data.fileNames.upstreamFileName «radio.upstream.folderNeedsUpstream (file.folderfrompath (f)) «else «upstreamfile (f) «if user.radio.prefs.upstream.logging «if ctupstreams > 0 «if ctupstreams == 1 «htmltext = ctupstreams + " file: " + htmltext «else «htmltext = ctupstreams + " files: " + htmltext «htmltext = string.delete (htmltext, sizeof (htmltext) - 1, 2) + "." «radio.log.add ("Upstream", htmltext, startticks, nil) «return (true) if flDebug { //debugging code op.setLineText (op.getLineText () + " (" + (clock.ticks () - ticks) + " ticks, " + op.countSubs (1) + " folders)"); if op.countSubs (1) == 1 { op.collapse ()}}}