Andy writes:

You were good enough to publish some questions I had about implementing good RSS behaviour in my Aggregator. I got some very useful feedback and have published an updated version of my Aggregator at:

http://www.seeita.com/RSSA/RSSA.html

I thought you might like an update on what I found.

I had already implemented ETAG and Last-Modified request headers. In .net it's just a case of using HttpWebResponse.Headers.Get to pick up ETAG and Last-Modified headers and then use HttpWebRequest.IfModifiedSince and HttpWebRequest.Headers.Add("If-None-Match", LastETAGValue) to pass the values back to the next request (provided the date supplied was valid). Then just monitor for a 'Not modified' response from the server.

Handling ttl (Time To Live) tags in feeds is unfortunately more subjective. It appears the intent of this tag has drifted since its definition. Originally it told file sharing programs the maximum time they should cache data. Since then, however, it's been used by some feeds to specify the minimum polling interval they expect from Aggregators. So, what I've done is implement ttl as a minimum polling interval subject to a blanket minimum of 60 minutes (unless people click a manual 'update now' button). That seems to safely split out the two different kinds of use.

I have a similar problem with the Syndication tags: updatePeriod and updateFrequency. They describe the feed's update frequency which does not strictly relate to polling frequency. Again, I have implemented a blanket minimum poll interval of 60 minutes. I also assume that 24 hours is a suitable minimum polling interval for any feed updated less frequently than once a day (otherwise it could take up to a week to spot a change in a feed that is updated weekly). I then take values in between as minimum poll intervals.

I found out more about feed compression. The .net framework doesn't handle it automatically. However, there is a free .net library that supports gzip and deflate compression available at:

http://www.icsharpcode.net/OpenSource/SharpZipLib/Default.aspx

There is a good article explaining how to use it at:

http://feralboy.com/log/archives/000420/

Note, however, that the code as supplied uses Microsoft encoding defaults i.e. UTF-8 encoding. That's true for many pages but not all (if your Aggregator drops or misrepresents UK pound signs you now know why). Note that I did not have to code for the server 406 response. Unless told otherwise, servers assume they can send an uncompressed response - which is what they do if they do not support either gzip or deflate. So it's simply a matter of requesting compression and then checking HttpWebResponse.ContentEncoding to see what you get back. I was disappointed, though, to find out how few feeds supported compression.

I have decided not to implement the skipDays tag. It is rarely used and, I suspect, getting rarer. It is IMHO fatally flawed. The RSS spec doesn't say how different time zones should interpret 'Sunday', for example. If both feed supplier and aggregator assume GMT then, during the Winter, a UK feed that specifies skipping on Saturday and Sunday will not be polled after 1pm Friday from some locations and up to 11am Monday from others. If either feed suppliers or Aggregators assume local times then the skipping could be 'out' by up to 23 hours!

I'm still thinking about the skipHours tag. The RSS spec says times are GMT. Even if that is correctly interpreted by both feed suppliers and Aggregators it is inappropriate for Aggregators to interpret the tag literally otherwise a feed made available during New York business hours would never be read by a Singaporean using an Aggregator at work during normal hours. Right now, I'm thinking it is difficult to develop a watertight implementation for limited benefit given: - Using header tags makes the interaction a very small one - Very few feeds implement skipHours according to Syndic8.

The Syndication spec says it "supercedes (sic) the RSS 0.91 skipDay and skipHour elements". I'm guessing there are other people out there as wary as I am about them.

I got an excellent suggestion to suspend polling when a PC is inactive. This has two major benefits: - When people are away from their desks, the Aggregator does not waste bandwidth looking for changes - When people return, the Aggregator is reactivated and they get to see the latest news.

To start with it looked tricky to implement an inactivity routine because it used to require 'hooking' keyboard and mouse events - and that looks like spyware activity. However, since W2K we have had a new Win32 routine called GetLastInputInfo. The description at:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winui/winui/windowsuserinterface/userinput/keyboardinput/keyboardinputreference/keyboardinputfunctions/getlastinputinfo.asp

Talks about keyboard activity but it also tracks mouse movements and clicks. This routine makes it real easy to build in an inactivity monitor - which I've done. I have had to allow people to switch it off, though, since some people leave their PCs switched on overnight to pick up Podcasts and BitTorrents.

I hope that's helpful. Certainly I've found the feedback to date to be helpful. I remain open to any suggestions to improve or refine my attempts to be well-behaved.

Andy

Andy writes:

I am a developer of an Aggregator - the CITA RSS Aggregator available from www.SeeITA.com/RSSA/RSSA.html. I developed it for a specific target market and I do not expect its use to grow to the extent that it will materially affect any RSS servers. However, I want to be responsible so I am trying to take suggestions from this forum seriously. I have already implemented ETag and Last-Modified header processing and now I am considering Randy Charles Morin's HowTo document to improve my Aggregator's behaviour.

It gives rise to several issues:

1) How should I understand the word 'Hint' in this context? Is the information an instruction that the Aggregator should obey, or is it a suggestion the Aggregator should convey to its user and allow them to decide how they act on it?

2) The skipHours/skipDays tags gives me several concerns:

a) A straw poll of 29 sites that I monitor reveals that none of them implement skipHours or skipDays. Syndic8 says that 1.98% of feeds use skipHours and 0.18% use skipDays. That's a pretty small minority to code for.

b) How many RSS publishers properly understand that the hours are in GMT? Most Americans I have met (I'm a Brit) think London is GMT - but that's only for half the year. Alternatively, how many will simply use their local time zone by mistake?

c) There is an obvious issue when people are in different time zones. For example, a reader in Singapore has no working hours overlapping with working hours in, say, New York. Simple implementation of the skipHours tag could mean that an Aggregator would never poll some feeds for some people.

d) Similarly, not polling on, say, Sundays is subject to interpretation. Is it Sunday for the reader or the publisher that the Aggregator should exclude from polling? I guess it has to be subject to the reader's time zone, but does that implement the publisher's expectation?

3) ttl is more widely implemented - Syndic8 says 7.74%. Use of the Syndication module will push this up a bit. However, I find that most sites specify hourly caching or less. My Aggregator's minimum poll interval is one hour, so implementing ttl would increase polling in many cases! That having been said, I plan to implement the higher of ttl and hourly polling as a minimum polling interval - subject to question 1 above.

4) I'm obviously interested in the Accept-Encoding tag because it looks like everyone wins from correct implementation of that one. However:

a) I use the .net HttpWebRequest class to read RSS feeds but I can't find any authoritative statement about whether this tag is implemented automatically in the .net framework, or not. I can't see why it wouldn't be (and the referenced w3.org document suggests that, by default, the server can send a compressed response), but we are talking Microsoft here.

b) As I understand it, the reader tries Accept-Encoding with, say, "gzip,deflate" if it gets a 406 (Not acceptable) status code back it has to try again without the Accept-Encoding. But doesn't this mean a double hit on servers that don't support compression? OK, it could remember the initial response but suppose the server is upgraded to support compression later on?

c) I can't find any tutorial on how to handle compressed responses from a server. If .net doesn't handle them automatically, can anyone point me in the right direction?

Sorry if my naivety is showing. Obviously, I don't expect you to be able to answer my questions but maybe one of your readers will be able to help.

Andy Henderson Constructive IT Advice Andy@SeeITA.com

Also, the system load on the machine I'm running to handle many many thousands of MySQL based queries often just at the top of an hour was more than it needed to be: it wasn't stressing the machine, but I have too many demands on my systems to add that load and have good performance right now.

So I'll throttle off during Epiphany and before and we'll see what materializes after looking at revised statistics.

A main point here is that aggregators need to be well behaved. Publishers can tweak all we want, but lightly implemented or buggy aggregators that don't observe Web server and feed directives and responses are (in my opinion) most of the problem. That's not to say that those of us running servers can't tweak our settings to improve matters, too!

Tristan also pointed to this great summary of RSS issues at NetCraft from October. It covers quite a lot of the current ground.

Jamie's post is dissected by Dan Sandler.

His conclusions, brief already, rewritten to be even terser by yours truly: Crappy aggregators abandon ship. Opportunistic programmers abandon ship. Abandoned ships are abandoned. How long has RSS been the Marie Celeste?

But as two Dans discuss--see earlier posts on this blog linking to their comments--my approach is probably restricting multiple users behind NAT and anonymous proxies. More rethink needed to strike a balance between appropriate throttling and losing subscribers.

I can see how to implement this using PHP, but there is a load factor. I would want every page load with an RSS icon on it to have a unique session ID--unique in a MySQL database table--rather than a redirect.

If I just had rss.xml -> rss_GUID.xml calculated through my throttling script, then aggregators that don't honor redirects during subscription would still request the old file time and again. Actually, that's probably okay. That would mean less-sophisticated aggregators would be restricted across IP + proxy + http_via, while more sophisticated (and thus better behaved aggregators) would be throttled just by their GUID.

There are some issues about what happens to differentiate the same aggregator that keeps getting new GUIDs that it doesn't "recall" over time. More thinking on this is necessary to avoid restricting users.

Dan offers a presentation he made on switching RSS from direct polling to peer-to-peer distribution.

I've seen a lot of discussion lately about RSS and aggregation bandwidth usage and behavior. This blog will collect details. I also welcome email that I can post. And comments are open. Let's talk about regular sucking schedules.