B659 Project Recap

Crawled NNDB to retrieve 11,800+ names. Example of the page that was crawled: http://www.nndb.com/lists/493/000063304/.

After compiling the list, each name was cleaned of punctuation, non-English letters and converted to lower case.
The list below displays examples of the "cleaning" process.

We randomly selected 100 names from the list of 11,800+ names.
The list below displays the beginning and ending names in our study set.

Creating the Retrieve Set

For each name we performed a query using the Google Web API like the following:
http://www.google.com/search?hl=en&q=abraham+benrubi
We retrieved the top ten URLs for each person in our study set.
This set of 1,000 pages is considered the seed set of pages that built our vicinity graph of 53,532 pages.

Creating the NNDB Collection

After creating our retrieve set, the next task we accomplished was to built our NNDB Collection.
We crawled NNDB to get the home page for each person in our study set.
Below is an example to the home page of one of our people:
http://www.nndb.com/people/147/000024075/. We parsed this page for all NNDB outlinks and used the Google Web API to gather all NNDB inlinks. The pages inlinks, home page, and outlinks for a particular person were grouped together to be used as the comparison information against our retrieve set.

Retrieve Set vs. NNDB Collection Comparision

Extracting Information from the Crawled Pages

For both the retrieve set and the NNDB collection, we parsed the pages by stripping HTML code and performing the same modifications that were done to "clean" the list of NNDB names. We parsed the pages to collect term and name occurrence information. For each web page we stored information just like the following:

Similarly, we counted the number of NNDB name occurrences for each page:

We kept track of each page that was associated with a person so that we could total the term and name occurrence information. From this information we were able to calculate the TDIDF using both terms and name occurrences for the group of 100 people in our study set.

Example of the TFIDF name occurrence information stored for one of the people in the study set:

Similar information was captured for each person in the study set using terms:

Using the information from the retrieve set, we calculated the cosine similarity between two people (for both name occurrences and terms) in the study set and compared this with the information from the NNDB collection.

Additionally, we used the name occurrences to determine the similarity between name occurrences in the retrieve set and those found in the NNDB collection.