6.13.1.4. URL input

• Currently, only the following protocols are supported: HTTP, (versions 0.9 and 1.0), Gopher (but not Gopher-+), FTP, and local files.
• The caching feature of urlretrieve() has been disabled until I find the time to hack proper processing of Expiration time headers.
• There should be a function to query whether a particular URL is in the cache.
• For backward compatibility, if a URL appears to point to a local file but the file can't be opened, the URL is re-interpreted using the FTP protocol. This can sometimes cause confusing error messages.
• The urlopen() and urlretrieve() functions can cause arbitrarily long delays while waiting for a network connection to be set up. This means that it is difficult to build an interactive web client using these functions without using threads.
• The data returned by urlopen() or urlretrieve() is the raw data returned by the server. This may be binary data (e.g. an image), plain text or (for example) HTML. The HTTP protocol provides type information in the reply header, which can be inspected by looking at the Content-type header. For the Gopher protocol, type information is encoded in the URL; there is currently no easy way to extract it. If the returned data is HTML, you can use the module htmllib to parse it.
• Although the urllib module contains (undocumented) routines to parse and unparse URL strings, the recommended interface for URL manipulation is in module urlparse.

     1: #line 292 "source_drivers.ipk"
     2: # gets input from a URL
     3: from interscript.drivers.sources.base import file_source
     4: from interscript.drivers.sources.base import eof
     5: import string
     6: class url_source(file_source):
     7:   def __init__(self,filename,encoding='utf8',**kwds):
     8:     apply(file_source.__init__,(self,encoding), kwds)
     9:     self.name = filename
    10:     self.file = urllib.urlopen(filename)
    11:     self.closed = 0
    12: 
    13: