You have probably come to this page because you have seen the use-agent string shown above in your server log. The purpose of this document is to explain about the HTTP requests using this user-agent string.
This user-agent is used by me (Tomas Schild, personal site - ODP editor profile) as a contribution to quality control of the Open Directory, of which I am an editor. This supplements the work of the official Open Directory robot, Robozilla, which does checks on all sites listed in the ODP from time to time, and a large number of diverse quality control approaches by other ODP editors.
URLs which are requested by this user-agent are:
The main function of this user-agent is to produce reports for the ODP editor community on entries that it might be a good idea to check manually, and to research patterns of 'soft' error messages.
This user-agent identifies itself as MS Internet Explorer 6.0. The reason for this is that there are brain-dead browser-detection scripts in use that do not behave intelligently when the user-agent is not detected to be one of a few popular browsers.
Considerate behaviour towards servers is a design requirement of the program using this user-agent string. HTTP requests from the same domain are spaced at least a second. Repeat retrieval attempts (usually minutes to hours after the first attempts) are usually only performed the URL did not return a HTTP header with a '200' result code.
The software is being tweaked frequently, thus an URL in my test data set might sometimes in testing been requested every few minutes. This should not be a cause for concern.
2006-04-04 (changed word 'links' to 'entries' in agent identification because some sites apparently misbehave when 'links' is part of user-agent string) and
2006-09-11 (apparently the substring 'test' also causes error codes being returned from some sites so I replaced it with 't_st'; also some updates re purposes and behaviour)