Edit: /usr/share/doc/lynx/docs/CRAWL.announce (5978B)
The TRAVERSAL code from old versions of Lynx has been upgraded by David
Mathog (mathog@seqaxp.bio.caltech.edu) so that it works again, can be
implemented via a command line switch (-traversal) instead of via a
compilation symbol for creating a separate Lynx executable as in those
previous versions, and can be used in conjunction with a -crawl switch
to make Lynx a front end for a Web Crawler.
Usage:
lynx [-traversal] [-realm] [-crawl] ["startpage"]
Added switches are:
-traversal Follow all http links derived from startpage that are
on the same server as startpage. If startpage isn't
specified then the traversal begins with the default
startfile or WWW_HOME.
-realm Further restrict http links to ones in the same realm
(having a matching base URI) as the startpage (e.g.,
http://host/~user/ will restrict the traversal to that
user's public html tree).
-crawl With [-traversal] outputs each unique hypertext page
as an lnk###########.dat file in the format specified
below. With [-dump] outputs only the startpage, in
the same format, to stdout.
Note on startpage:
If a startpage is specified and contains any uppercase
characters, on VMS it should be enclosed in double-quotes.
The code that extracts the access and host fields from
startpage for comparsions with links to ensure they are
not on another server, and the comparisons with already
traversed links, are case sensitive, and the startpage
will go to all lowercase on VMS if no double-quotes are
supplied, such that it might be treated as a new link if
encounted with uppercase letters.
Files created and/or used with the -traversal switch, based on definitions
in userdefs.h:
TRAVERSE_FILE (traverse.dat):
Contains a list of all URLs that were traversed. Note
that if a URL appears in this file it will not be
traversed again (important if runs are started and
stopped). Placing an entry in this file BEFORE the
run will block traversal of that URL. Unlike reject.dat
a final * has no effect (see below). Note that Lynx
internal client-side image MAP URLs will be included in
this file (e.g., LYNXIMGMAP:http://server/foo.html#map1),
in addition to the "real" (external) http URLs.
TRAVERSE_FOUND_FILE (traverse2.dat):
Contains a list of all URLs that were traversed, in the
order encountered or re-encountered (but not re-travered)
during a traversal run, and the TITLEs of the documents
(separated from the URLs by TABs) A URL and TITLE may be
present in this list many times. To simplify the list,
on VMS use: sort/nodups traverse2.dat;1 ;2
Note that the URLs and TITLEs of the Lynx internal
client-side image MAP pseudo-documents will not be
included in this file, though "traversed", but only the
http URLs and TITLEs derived from the MAP's AREA tag
HREFs that were traversed.
TRAVERSE_REJECT_FILE (reject.dat):
Contains a list of URLs that have been rejected from the
traversal. Once a URL has been entered in this list, it
will not be traversed. URLs that end in a * will cause
rejection of all URLs that match up to the character before
the *. So for instance, to reject all htbin references on a
site put this line in the reject.dat file BEFORE starting
the run: http://www.site.wherever:8000/htbin*
TRAVERSE_ERRORS (traverse.errors):
A list of links that could not be accessed or had an
unknown status returned by the http server. If the
owner of the document containing the link is know via
a LINK REV="made" HREF="mailto:foo" in it and the
MAIL_SYSTEM_ERROR_LOGGING was set true in userdefs.h
or lynx.cfg (not recommended!!!), a message about the
problem will be mailed to the owner as well.
Files created during traversals if the -crawl switch is included with the
-traversal switch:
lnk########.dat Numbered output files containing the contents of traversed
hypertext documents in text format. All hypertext links
within the document have been stripped, and the URL and
TITLE of the document are recorded as the first two lines,
e.g., for the seqaxp.bio.caltech.edu home page the first
two lines will be:
THE_URL:http://seqaxp.bio.caltech.edu:8000/
THE_TITLE:SAF Web server home page
The VMSIndex software is being adapted to use this
information to extract the corresponding URL and TITLE
for use in indexing the lnk########.dat files, e.g.:
$ build_index -
/url=(text="THE_URL:") -
/topic=(text="THE_TITLE:",EXCLUDE) -
/output=INDEX_NAME -
lnk*.dat
A clever person should be able to figure out a way to
index the lnk########.dat files on Unix as well.
If you want the hypertext links in the document to be
numbered, include the -number_links switch. By default,
this will cause the list of References (URLs for the
numbered links) to be appended as well. If you want
numbered links but not the References list, include the
-nolist switch as well.
Note that any client-side image MAP pseudo documents
that were "traversed" will not have lnk########.dat
output files created for them, but output files will
be created for "real" documents that were traversed
based on the HREFs of the MAP's AREA tags.
This functionality is still under development. Feedback and suggestions
are welcome.