import Str, Farm;

 

export var Crawler =

[.

// Pages visited so far (and removed from queue)

// and pages waiting in the queue.

enqueued = [. .],

 

// Will contain the farm after the start method is

// called.

farm = nil,

 

// Method that should be overridden.

Visit = meth(s, page) PrintLn(page.URL) end,

ShouldVisit = meth(s, url) true end,

 

Enqueue = meth(s, url)

// First remove everything following #

var pos = Str_IndexOf("#", url);

if pos != -1 then

url = Select(url, 0, pos)

end;

lock s do

var present = s.enqueued[url] ? false;

if !present and s.ShouldVisit(url) then

s.enqueued[url] := true;

s.farm.Perform(s.ProcessPage(s, url))

end

end

end,

 

ProcessPage = fun(s, url)

try

var page = GetURL(url); // fetch the page

s.Visit(page);

 

// Process all the links from this page.

every a in Elem(page, "a") do

s.Enqueue(a.href) ? nil

end

catch E

on true do PrintLn(url, " err: ", E.msg)

end;

end,

 

Start = meth(s, noworkers)

s.farm = Farm_NewFarm(noworkers);

end,

 

Abort = meth(s) s.Stop() end

.];

 

First we need to keep track of all pages visited so far with an associative array (aka a WebL object) where the fields are the visited URLs, and the value is either true or false (line 7). Note that an alternative implementation could use a set instead of an object without any performance penalty.

Lines 14 and 15 define the two methods that need to be overridden to customize the crawler. The Visit method is called each time a new page is visited, and the ShouldVisit method indicates whether a specific URL should be crawled or not.

The Enqueue method (lines 17-30) adds a URL to the queue of pages to be fetched. The first task is to strip off any references from the URL (lines 19-22). Line 24 then checks if we visited the page already. Note the use of the ? service combinator to catch the exception should the URL not be in the visited array. If the URL is not present, and we should visit this page (line 25), we remember that we have seen the page (line 26), and then pass the job of retrieving the page to a farm object (line 27).

Eventually when a worker on the farm reaches a new job, the ProcessPage function is invoked (lines 32-44). After the page is fetched (line 34), we call the method Visit to let the crawler process the page (line 35). Lines 38-40 take care of enqueing all the anchors found on the page.

A custom crawler

Now we look at how we can create a custom crawler using the generic crawler above. To override the Visit and ShouldVisit methods, we use the Clone builtin applied to the generic crawler and our own object that contains the modifications to the generic crawler we would like to make (lines 3-16).