import Str, Files;

 

var Scores = fun(page)

var txt = Text(page);

var letters = Size(Str_Search(txt, "[A-Za-z]"));

var words = Size(Str_Search(txt, "[0-9a-zA-Z']+"));

var syllables = Size(Str_Search(txt, "[aeiouy]+"));

 

var exceptions = Pat(page, "[ ]([A-Z0-9][.])+");

Replace(exceptions, NewPiece(" X ", "text/plain"));

var sentences = Size(Pat(page, "[.]"));

 

[.

Sentences = sentences,

Words = words,

Syllables = syllables,

 

ARI = 4.71 * (letters / words) +

0.5 * (words /sentences) - 21.43,

Kincaid = 11.8 * (syllables / words) +

0.39 * (words / sentences) - 15.59,

CLF = 5.89 * (letters / words) -

0.3 * (sentences / (words / 100)) - 15.8,

Flesch = 206.835 - 84.6 * (syllables / words)

- 1.015 * (words / sentences)

.]

end;

 

var ScorePageList = fun(L)

var res = [];

var count = 1;

every s in L do

try

PrintLn(count, " scoring ", s);

count = count + 1;

var page = GetURL(s);

var sc = Scores(page);

sc.URL := s;

sc.Title := Text(Elem(page, "title")[0]);

res = res + [sc];

catch e // just report errors

on true do PrintLn(e.msg)

end;

end;

res

end;

 

var FollowLink = fun(page, anchortext)

var dest = (Elem(page, "a") contain

Pat(page, anchortext))[0];

GetURL(dest.href)

end;

 

var GetStories = fun()

var res = [];

var P = GetURL("http://www.news.com");

var H = FollowLink(P, "(?i)all the headlines");

var A = (Elem(H, "a") directlyafter Pat(H, "&#149"))

!inside Elem(H, "strong");

every a in A do

res = res + [a.href];

end;

PrintLn(Size(res), " articles found.");

res

end;

 

var pages = GetStories();

var res = ScorePageList(pages);

 

res = Sort(res,

fun(a, b)

var diff = a.Kincaid - b.Kincaid;

if diff > 0.0 then 1

elsif diff == 0.0 then 0

else -1

end

end);

 

var s = "";

every x in res do

PrintLn(x.Kincaid, " ", x.Title);

s = s + x.Kincaid + " " + x.Title + "\r\n";

end;

Files_SaveToFile("kincaid.txt", s);

 

Lines 3-24 implement the core of the scoring function. After extracting the text of the page (line 4), we proceed to calculate the number of letters (line 5), words (line 6), and syllables (line 7) on the page, using a few simple regular expressions. Lines 9-11 take care of removing any initials that might appear in the page. This is necessary as the number of periods in the page is used as a measure of how many sentences are present, and initials containing periods will skew that count. Finally lines 13-26 calculate a few common reading scores, and return a "score" object with the results to the caller.

Lines 29-46 calculate reading scores for a list of URLs. Lines 36-37 fetch the individual pages and calculate the score. In lines 38-39 we extend the score object with fields to identify the URL and title of the page. The ScorePageList function returns a list of these score objects.

The purpose of the GetStories function (lines 54-65) is to retrieve a list of URLs representing newspaper articles. After fetching the root page from news.com (line 56), we follow the link called "All the headlines" to a page that contains all the stories of the day (line 57). Lines 58-59 perform the extraction of the story URLs. We locate all the anchors appearing after a bullet symbol (identified by the "&#149"; character entity) that are not written in the strong font. Lines 60-62 construct a list object of all the URLs found.

The main program starts at line 67. First we fetch the stories, and then score them. Lines 70-77 take care of sorting the stories according to score.

Finally, lines 79-84 take care of printing the result and writing it to a file.