Week 9

String Methods and Data Mining

String method demo

>>> s = 'I never, never did anything...'
>>> s.upper()
'I NEVER, NEVER DID ANYTHING...'
>>> import math
>>> math.sqrt(4)
2
>>> s.lower()
'i never, never did anything...'
>>> s.find('never')
2
>>> s.find('Never')
-1
>>> s.index('never')
2
>>> s.index('Never')
Traceback (most recent call last):
  File "", line 1, in 
ValueError: substring not found
>>> s.find('never', 3)
9
>>> s.index('never', 3)
9
>>> word = s[0: s.index('never')]
>>> word
'I '
>>> word.strip()
'I'
>>> s[0 : s.index('never')].strip()
'I'
>>> '123'.isdigit()
True
>>> ' 123 '.isdigit()
False
>>> ' 123 '.strip().isdigit()
True
>>>

Method call syntax

• General method call syntax:

  object_expression . method_name ( expression ,..., expression )

  • note the dot between object_expression and method_name
  • this is really a special kind of function call in which the function experssion is an attribute reference, in which
    • the expression evaluates to an object, known as the object of the call
    • the attribute name refers to an method belonging to the object

Method call semantics

• Recall that an object is an instance of a class
  • all instances of the same class have the same methods
• A Python method is a special kind of function that is associated with an object and has access to the object when it runs
  • the object of the call is effectively an additional argument of the call
  • in other respects it is called and returns just like any other function
• Message passing metaphor
  • method calls are often thought of as "sending a message" to the object of the call
    • the message consists of the method name and the call arguments
  • the value returned by the method is "sent back" to the caller

Optional arguments

• We have seen that the second argument of the find and index methods is optional and defaults to 0 if it is missing

  • FYI: they actually have an optional third argument, indicating the largest index that can be returned, but we don't need it since its default value (the length of the string) is all we need

• FYI: many functions and methods takes some number of optional argument, or even an unlimited number of arguments

  • it is even possible to define your own functions and methods with

    variable numbers of arguments

  • there is even related mechanism for "keyword" arguments, but we do not need these features in this course

Some string methods

• Python strings are all objects: instances of the built-in class str
• The string methods we will be using are among the more common ones
  • s.find(s1, [begin_index]) returns the index of the beginning of the first occurrence of the substring s1 in s that is greater than or equal to begin_index, or -1 if s1 is no such substring
  • s.index(s1, [begin_index]) is just like find, except that it raises and error rather than returning -1 if there is no matching substring
  • s.isdigit() returns a boolean value indicating if s contains only digit characters (0 through 9)
  • s.lower() converts all uppercase characters in s to lowercase
  • s.upper() converts all lowercase characters in s to uppercase
  • s.strip() removes all whitespace characters from the beginning and end of s

Since strings are immutable, wording such as in the last three points above means that a copy of the string s is returned with just the indicated changes.

Other string methods and functions

• There are a large number of additional string methods
  • see the standard Python documentation or call help(str)
• Python also has a module named string that contains a number of functions, most of which duplicate the behavior of the string methods
  • these functions were in Python before it was "object oriented"
  • your text refers to a few of these functions, but the preferred style now is to use string methods instead
    • these string functions are "deprecated," meaning they will probably disappear in a future version of Python
    • don't import the string module if you just use the string methods

Which of the following is a method call to find the index of 'a' in s?

1. find(s, 'a')
2. string.find(s, 'a')
3. s.find('a')
4. a.find(s)
5. A or B

Answer: C

What does the following print?

s = 'AIR RAID SIRENS'
s.lower()
print s

1. 'AIR RAID SIRENS'
2. 'air raid sirens'
3. None of the above, there is an error

Answer: A

What does '[812]'[1:4].isdigit() return?

1. True
2. False
3. '812`
4. None of the above, there is an error

Answer: A

Data mining background

• We briefly introduce
  • Uniform Resource Locators (URLs)
  • HTML
• You will do data mining of your choice in the next assignment

Uniform Resource Locators (URLs)

• Uniform Resource Locators (URL s) identify resources on the Internet
  • they can be used for much more than identifying resources provided by Internet servers
  • examples include resources for file transfer, email, and newsgroups, as well as web access
• FYI: URLs are an instance of the more general notion of a Uniform Resource Identifier (URI)
  • URIs may indicate anything!
  • they are location independent (may not be identified with any server)
  • URIs are the building blocks of the Resource Description Framework (RDF), which is poised to become the foundation of the semantic web
  • the semantic web aims to make Internet content understandable to computers, as the existing hypertext web has made it understandable by humans

HTML

• A markup language provides a way of adding notations to text for some purpose
• The Hyper-Text Markup Language (HTML) is used to add markups to web documents
• HTML is derived from the more general eXtensible Markup Language (XML)
• XML has become the basis for many data exchange formats
• XML markups use tags delimited by matching angle brackets, <...>
• HTML's most important use is to support hyper-text, the linking of words or phrases of one document to another document
  • sample syntax:: < a href=" url "> linked text < /a>
• HTML also provides a variety of formatting tags to indicate document appearance and some content identification markups
  • examples: < b > to begin bold font, < p > to begin a paragraph

Data mining

• Data mining refers to the process of extracting useful information from vast amounts of data
• The web is an incredible supply of data to mine
• Web data mining often uses specialized software for parsing and navigating data in XML format
• Some useful data mining can be accomplished using the simple string manipulation techniques we have learned
  • this approach is more fragile than using XML tools: more likely to stop working if the format of the web page is changed a bit
  • most HTML data mining is fragile even if XML-based because HTML was intended for data display, not automated searching
• RDF is intended to solve this problem

Accessing the web with Python

• The standard Python library module urllib supports accessing web servers in a variety of ways
  • urllib.FancyURLopener({}).open(url) returns a connection to a web server for the given url
  • data can be read from the connection as from a file
• The http.py module file simplifies web access
  • the http.get(url) function returns in a string a server's response for the given URL
  • it handles the most common kind of redirection with a loop

Getting the world population, Part 1

• Visit http://www.census.gov/main/www/popclock.html

• Next we run the View > Source command (in IE and Firefox browsers)

• Then we cut the URL from the browser URL window into the population_test.py program that follows, run it, and compare the printed text with the browser's source (they should be the same)

  # population_test.py, by chaynes@indiana.edu
  
  import http
  
  population_url = 'http://www.census.gov/main/www/popclock.html'
  
  def main():
      text = http.get(population_url)
      print text ##
      ## insert datamining code here
  
  main()
      
  
  

Getting the world population, Part 2

• How can we locate the temperature?

• One possible answer: between id="wclocknum"> and the following <, as in population.py

  # population.py, by chaynes@indiana.edu
  
  import http
  
  population_url = 'http://www.census.gov/main/www/popclock.html'
  
  def main():
      text = http.get(population_url)
      start_text = 'id="wclocknum">'
      start = text.index(start_text) + len(start_text)
      stop = text.index('<', start)
      print text[start : stop]
  
  main()
      
  
  

Ebert's recommended movies

• View Roger Ebert's recommended movie list at http://rogerebert.suntimes.com/apps/pbcs.dll/section?category=REVIEWS05

• How can we locate the movie list in the source text?

• First, we note that the list is between the strings -- main content -- and !Middle1

  • by narrowing our search to this region, we avoid picking up stuff from other parts of the page

• How can we locate each successive movie title?

  • one possible answer: each movie name is between b> and < characters following href="/apps/pbcs.dll

• See ebert_movies.py

  # ebert_movies.py, by chaynes@indiana.edu
  '''
  Get the text of Roger Ebert's recommendations page and print
  the recommended movie titles.
  '''
  
  import http
  
  ebert_recommends_url = 'http://rogerebert.suntimes.com/apps/pbcs.dll/'
  ebert_recommends_url += 'section?category=REVIEWS05'
  
  # Search between the first occurrence of '-- main content--' and the next
  # '!Middle1' for movie names. Names start after the first b> after 
  # 'href=/apps/pbcs.dll' and end with the following '<'.
  def main():
      text = http.get(ebert_recommends_url)
      begin = text.index('-- main content --')
      endRecs = text.index('!Middle1', begin)
      while True:
          i = text.find('href="/apps/pbcs.dll', begin)
          if i == -1 or i > endRecs:
              break
          begin = text.find('b>', i) + 2
          end = text.find('<', begin)
          print text[begin : end]
      
  main()
      
  
  

Some limitations of this data mining methodology

• Browser source and program connection text might not be the same in exceptional circumstances such as a forwarding technique not known to http.get
• Avoid web sites that require you to login
• Avoid web sites that use frames
  • the text will be brief with several mentions of frame tags
• Avoid web sites that display the information you want via client-side programs
  • for example, Javascript code or a Java applet
  • client-side programs are ok if they are not generating your information
  • example, Ebert's web uses Javascript, but not to generate movie names information generated by server-side programs is fine
• Fragility, as mentioned earlier
• Some strange errors
  • example: IOError: [Errno 2] No such file or directory: '\cgi-bin\findweather\getForecast?query=47408'
  • choose another site

Exercise: similar

def similar(s1, s2):
    """
    Return a boolean value indicating if strings s1 and s2 are equal,
    ignoring case (case insensitive) and leading and trailing whitespace.

    >>> similar('  Ni  ', 'ni')
    True
    >>> similar('eggs', ' EGGS')
    True
    >>> similar('eggs', 'egg')
    False
    >>>
    """
    #...

Solution: similar

def similar(s1, s2):
    """
    Return a boolean value indicating if strings s1 and s2 are equal,
    ignoring case (case insensitive) and leading and trailing whitespace.

    >>> similar('  Ni  ', 'ni')
    True
    >>> similar('eggs', ' EGGS')
    True
    >>> similar('eggs', 'egg')
    False
    >>>
    """
    return s1.lower().strip() == s2.lower().strip()

Exercise: inches1

def inches1(length):
    """
    Return the number of inches (an int) in length, a string that should be
    in the format <inches>", where <inches> is a non-negative integer, Return None
    if length is not a string in the proper format.

    >>> inches1('3"')
    3
    >>> inches1('3') # None returned by this and following tests
    >>> inches1('+3"')
    >>>
    """
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.

Solution: inches1

def inches1(length):
    """
    Return the number of inches (an int) in length, a string that should be
    in the format <inches>", where <inches> is a non-negative integer, Return None
    if length is not a string in the proper format.

    >>> inches1('3"')
    3
    >>> inches1('3') # None returned by this and following tests
    >>> inches1('+3"')
    >>>
    """
    if length == '' or length[len(length) -1] != '"':
        return None # no double quote
    else:
        # remove the double quote character
        length = length[0 : len(length) - 1]
        if not length.isdigit():
            return None # <inches> is not an integer
    return int(length)

Exercise: inches2

def inches2(length):
    """
    Return the number of inches (an int) in length, a string in the format
    <feet>'<inches>", where both <feet> and <inches> are non-negative integers,
    Return None if length is not in the proper format.

    >>> inches2('2\'3"')
    27
    >>> inches2("bodus") # None returned by this and following tests
    >>> inches2('11"')
    >>>
    """
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.

Solution: inches2

def inches2(length):
    """
    Return the number of inches (an int) in length, a string in the format
    <feet>'<inches>", where both <feet> and <inches> are non-negative integers,
    Return None if length is not in the proper format.

    >>> inches2('2\'3"')
    27
    >>> inches2("bodus") # None returned by this and following tests
    >>> inches2('11"')
    >>>
    """
    index = length.find("'")
    if index == -1:
        return None # Missing ' mark
    else:
        feet_string = length[0 : index]
        inches_string = length[index + 1 : len(length)]
        if feet_string.isdigit():
            feet = int(feet_string)
        else:
            return None # <feet> is not an integer
    if inches_string == '':
        return None # no <inchex>"
    elif inches_string[len(inches_string) - 1] != '"':
        return None # no double quote at end
    else:
        # remove the double quote character
        inches_string = inches_string[0 : len(inches_string) - 1]
        if inches_string.isdigit():
            inches = int(inches_string)
        else:
            return None # <inches> is not an integer
    return feet * 12 + inches

Exercise: inches

def inches1(length):
    """
    Return the number of inches (an int) in length, a string that should be
    in the format <inches>", where <inches> is a non-negative integer, Return None
    if length is not a string in the proper format.

    >>> inches1('3"')
    3
    >>> inches1('3') # None returned by this and following tests
    >>> inches1('+3"')
    >>>
    """
    #.
    #.
    #.
    #.
    #.
    #.
    #.
    #.

Solution: inches

def inches1(length):
    """
    Return the number of inches (an int) in length, a string that should be
    in the format <inches>", where <inches> is a non-negative integer, Return None
    if length is not a string in the proper format.

    >>> inches1('3"')
    3
    >>> inches1('3') # None returned by this and following tests
    >>> inches1('+3"')
    >>>
    """
    if length == '' or length[len(length) -1] != '"':
        return None # no double quote
    else:
        # remove the double quote character
        length = length[0 : len(length) - 1]
        if not length.isdigit():
            return None # <inches> is not an integer
    return int(length)

What is the value of 'a'.strip == 'a' ?

1. True
2. False
3. there is a syntax error
4. there is a runtime error

Answer: B

What does the test print?

def mystery(s, n):
    """
    >>> mystery('Chamilot', 2)
    #...
    >>>
    """
    while 0 < n:
        s = s[1 : len(s) - 2]
        n = n - 1
    print s

1. Chamilot
2. hamilo
3. am
4. nothing (but a newline)
5. none of the above, but no error
6. none of the above, there is an error

Answer: C