MINI MINI MANI MO
ó
y]Rc @ s´ d d l Z d d l Z d d l Z d d l Z d d l Z d Z y d d l m Z Wn# e k
r{ Z
d d l m Z n Xd e f d „ ƒ YZ d e j
f d „ ƒ YZ d e f d „ ƒ YZ d d d
„ ƒ YZ d e j f d „ ƒ YZ d d
l m Z m Z m Z m Z m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l Z d e j f d „ ƒ YZ d e j! f d „ ƒ YZ! d a" d „ Z# d „ Z$ d „ Z% d S( iÿÿÿÿN( t StringIOt
RangeErrorc B s e Z d Z RS( s6 Error raised when an unsatisfiable range is requested.( t __name__t
__module__t __doc__( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR " s t HTTPRangeHandlerc B s e Z d Z d „ Z d „ Z RS( s» Handler that enables HTTP Range headers.
This was extremely simple. The Range header is a HTTP feature to
begin with so all this class does is tell urllib2 that the
"206 Partial Content" response from the HTTP server is what we
expected.
Example:
import urllib2
import byterange
range_handler = range.HTTPRangeHandler()
opener = urllib2.build_opener(range_handler)
# install it
urllib2.install_opener(opener)
# create Request and set Range header
req = urllib2.Request('http://www.python.org/')
req.header['Range'] = 'bytes=30-50'
f = urllib2.urlopen(req)
c C s1 t j | | | j ƒ ƒ } | | _ | | _ | S( N( t urllibt
addinfourlt get_full_urlt codet msg( t selft reqt fpR R
t hdrst r( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt http_error_206> s c C s t d d ƒ ‚ d S( Ni s Requested Range Not Satisfiable( R ( R R R
R R
R ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt http_error_416E s ( R R R R R ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR & s t HTTPSRangeHandlerc B s e Z d Z d „ Z d „ Z RS( s! Range Header support for HTTPS. c C s | j | | | | | ƒ S( N( R ( R R R
R R
R ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt https_error_206L s c C s | j | | | | | ƒ d S( N( t https_error_416( R R R
R R
R ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR O s ( R R R R R ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR I s t RangeableFileObjectc B sh e Z d Z d „ Z d „ Z d „ Z d d „ Z d d „ Z d d „ Z d „ Z d
„ Z
d „ Z RS( s" File object wrapper to enable raw range handling.
This was implemented primarilary for handling range
specifications for file:// urls. This object effectively makes
a file object look like it consists only of a range of bytes in
the stream.
Examples:
# expose 10 bytes, starting at byte position 20, from
# /etc/aliases.
>>> fo = RangeableFileObject(file('/etc/passwd', 'r'), (20,30))
# seek seeks within the range (to position 23 in this case)
>>> fo.seek(3)
# tell tells where your at _within the range_ (position 3 in
# this case)
>>> fo.tell()
# read EOFs if an attempt is made to read past the last
# byte in the range. the following will return only 7 bytes.
>>> fo.read(30)
c C s> | | _ t | ƒ \ | _ | _ d | _ | j | j ƒ d S( s Create a RangeableFileObject.
fo -- a file like object. only the read() method need be
supported but supporting an optimized seek() is
preferable.
rangetup -- a (firstbyte,lastbyte) tuple specifying the range
to work over.
The file object provided is assumed to be at byte offset 0.
i N( t fot range_tuple_normalizet firstbytet lastbytet realpost _do_seek( R R t rangetup( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt __init__g s c C s/ t | j | ƒ r" t | j | ƒ St | ‚ d S( s« This effectively allows us to wrap at the instance level.
Any attribute not found in _this_ object will be searched for
in self.fo. This includes methods.N( t hasattrR t getattrt AttributeError( R t name( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt __getattr__u s c C s | j | j S( s5 Return the position within the range.
This is different from fo.seek in that position 0 is the
first byte position of the range tuple. For example, if
this object was created with a range tuple of (500,899),
tell() will return 0 when at byte position 500 of the file.
( R R ( R ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt tell} s i c C s¡ | d k s t ‚ | d k r. | j | } n7 | d k rJ | j | } n | d k re t d ƒ ‚ n | j r‰ | | j k r‰ | j } n | j | | j ƒ d S( se Seek within the byte range.
Positioning is identical to that described under tell().
i i i s$ seek from end of file not supported.N( i i i ( t AssertionErrorR R t IOErrorR R ( R t offsett whencet
realoffset( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt seek† s iÿÿÿÿc C s: | j | ƒ } | j j | ƒ } | j t | ƒ 7_ | S( s` Read within the range.
This method will limit the size read based on the range.
( t _calc_read_sizeR t readR t len( R t sizet rslt( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR+ ™ s c C s: | j | ƒ } | j j | ƒ } | j t | ƒ 7_ | S( sf Read lines within the range.
This method will limit the size read based on the range.
( R* R t readlineR R, ( R R- R. ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR/ ¢ s c C sX | j rT | d k rA | j | | j k rQ | j | j } qQ qT | j | j } n | S( sS Handles calculating the amount of data to read based on
the range.
iÿÿÿÿ( R R ( R R- ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR* « s c C s^ | d k s t ‚ t | j d ƒ s4 | j | ƒ n | j j | j | ƒ | j | 7_ d S( s Seek based on whether wrapped object supports seek().
offset is relative to the current position (self.realpos).
i R) N( R$ R R t _poor_mans_seekR) R ( R R& ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR · s
c C s€ d } d } xm | | k r{ | | | k r8 | | } n | j j | ƒ } t | ƒ | k rn t d d ƒ ‚ n | | 7} q Wd S( s¯ Seek by calling the wrapped file objects read() method.
This is used for file like objects that do not have native
seek support. The wrapped objects read() method is called
to manually seek to the desired position.
offset -- read this number of bytes from the wrapped
file object.
raise RangeError if we encounter EOF before reaching the
specified offset.
i i i s Requested Range Not SatisfiableN( R R+ R, R ( R R& t post bufsizet buf( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR0 Â s
( R R R R R" R# R) R+ R/ R* R R0 ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR R s t FileRangeHandlerc B s e Z d Z d „ Z RS( s~ FileHandler subclass that adds Range support.
This class handles Range headers exactly like an HTTP
server would.
c
C sÕ d d l } d d l } | j ƒ } | j ƒ } t j | ƒ } t j | ƒ } | t j } t j
| t j ƒ } | j | ƒ d }
| rÕ t j
| ƒ \ } } | sÀ t j | ƒ | j ƒ k rÕ t j d ƒ ‚ qÕ n t | d ƒ } | j j d d ƒ }
t |
ƒ }
|
d k st ‚ |
r–|
\ } } | d k r>| } n | d k sb| | k sb| | k rtt d d ƒ ‚ n | | } t | | | f ƒ } n | j t d |
p«d
| | f ƒ ƒ } t j | | d | ƒ S(
Niÿÿÿÿi s file not on local hostt rbt Ranget i s Requested Range Not Satisfiables6 Content-Type: %s
Content-Length: %d
Last-modified: %s
s
text/plains file:( ( t mimetypest mimetoolst get_hostt get_selectorR t url2pathnamet ost statt ST_SIZEt rfc822t
formatdatet ST_MTIMEt
guess_typet splitportt sockett
gethostbynamet get_namest urllib2t URLErrort opent headerst gett Nonet range_header_to_tupleR$ R R t MessageR R ( R R R8 R9 t hostt filet localfilet statsR- t modifiedt mtypet portR t branget fbt lbRK ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt open_local_fileÛ s:
! $
( R R R RZ ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR4 Ö s ( RD t splitusert splitpasswdt splitattrt unquotet addclosehookR t FTPRangeHandlerc B s e Z d „ Z d „ Z RS( c C s¸ | j ƒ } | s t d ‚ n t | ƒ \ } } | d k rH t j } n t | ƒ } t | ƒ \ } } | r t | ƒ \ } } n d } t | ƒ } t | pŸ d ƒ } t | p± d ƒ } y t
j | ƒ } Wn% t
j k
rñ } t
j | ƒ ‚ n Xt | j ƒ ƒ \ } } | j d ƒ } t t | ƒ } | d | d } }
| r[| d r[| d } n y$| j | | | | | ƒ } |
r…d pˆd } xM | D]E }
t |
ƒ \ }
} |
j ƒ d
k r’| d k r’| j ƒ } q’q’Wd } t | j j d d ƒ ƒ } | d k st ‚ | r8| \ } } | d k r8| } q8n | j |
| | ƒ \ } } | rî| \ } } | d k rÌ| d k s‰| d k r›t d d ƒ ‚ n | } | | } | d k rët d d ƒ ‚ qëqî| | } t | d | f ƒ } n d } t j | j ƒ ƒ d } | r$| d | 7} n | d k rM| d k rM| d | 7} n t | ƒ } t! j" | ƒ } t# | | | j ƒ ƒ SWn2 t j$ k
r³} t d | f t% j& ƒ d ‚ n Xd S( Ns ftp errors
no host givenR7 t /iÿÿÿÿi i t It Dt typet at At it dR6 i s@ Requested Range Not Satisfiable due to unobtainable file length.s Requested Range Not Satisfiables Content-Type: %s
s Content-Length: %d
i ( s ftp errors
no host given( Re Rf Rg Rb Rh Rc ( (' R: R% RD RM t ftplibt FTP_PORTt intR[ R\ R^ RE RF t errorRH RI R] R; t splitt mapt connect_ftpt lowert upperRN RK RL R$ t retrfileR R R8 RC R R R9 RO R t
all_errorst syst exc_info( R R RP RV t usert passwdR
t patht attrst dirsRQ t fwRd t attrt valuet restt range_tupRX RY R
t retrlenRK RU t sf( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt ftp_open
s|
c C s t | | | | | ƒ } | S( N( t
ftpwrapper( R Rv Rw RP RV Rz R{ ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyRo X s ( R R R‚ Ro ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR` s NRƒ c B s e Z d d „ Z RS( c
C s | j ƒ | d k r% d } d } n d | } d } y | j j | ƒ Wn. t j k
ry | j ƒ | j j | ƒ n Xd } | r§| r§y | j j | ƒ Wn2 t j k
rÕ } t d | f t
j ƒ d ‚ n X| j j | ƒ y# d | } | j j | | ƒ } Wq§t j k
r£} t
| ƒ d
d k rk| j | | ƒ \ } } t | | d f ƒ } | | f St
| ƒ d
d
k r¤t d | f t
j ƒ d ‚ q¤q§Xn | së| j j d ƒ | rÐd | } n d } | j j | ƒ } n d | _ t | d j d ƒ | j ƒ | d f S( NRh Rc s TYPE Ai s TYPE i s ftp errori s RETR i t 501R7 t 550s LIST t LISTR5 ( Rh Rc ( t endtransfert ftpt voidcmdRi Rs t initRM t nlstt
error_permR% Rt Ru t ntransfercmdt strRr R t busyR_ t makefile(
R RQ Rd R~ t cmdt isdirt connt reasonR
R€ ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyRr a sJ
&
N( R R RM Rr ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyRƒ \ s c C sš | d k r d St d k r: d d l } | j d ƒ a n t j | ƒ } | r– t | j d d ƒ ƒ } | r’ | d r’ | d | d d f } n | Sd S( sØ Get a (firstbyte,lastbyte) tuple from a Range header value.
Range headers have the form "bytes=<firstbyte>-<lastbyte>". This
function pulls the firstbyte and lastbyte values and returns
a (firstbyte,lastbyte) tuple. If lastbyte is not specified in
the header value, it is returned as an empty string in the
tuple.
Return None if range_header is None
Return () if range_header does not conform to the range spec
pattern.
iÿÿÿÿNs ^bytes=(\d{1,})-(\d*)i i i ( ( RM t _rangeret ret compilet matchR t group( t range_headerR– R˜ t tup( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyRN ‘ s c C sS | d k r d St | ƒ } | rO | d rG | d | d d f } n d | Sd S( s• Convert a range tuple to a Range header value.
Return a string of the form "bytes=<firstbyte>-<lastbyte>" or None
if no range is needed.
i i s bytes=%s-%sN( RM R ( R ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt range_tuple_to_header¬ s
c C sÛ | d k r d S| d } | d k r/ d } n t | ƒ } y | d } Wn t k
rb d } n1 X| d k rx d } n | d k r“ t | ƒ } n | | f d k r© d S| | k rÑ t d d | | f ƒ ‚ n | | f S( s7 Normalize a (first_byte,last_byte) range tuple.
Return a tuple whose first element is guaranteed to be an int
and whose second element will be '' (meaning: the last byte) or
an int. Finally, return None if the normalized tuple == (0,'')
as that is equivalent to retrieving the entire file.
i R7 i i s Invalid byte range: %s-%sN( NR7 ( i R7 ( RM Rk t
IndexErrorR ( R RX RY ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR ¸ s&
( (& R= R> R RH R@ RM t DEBUGt cStringIOR t ImportErrorR
R% R t BaseHandlerR R R t FileHandlerR4 RD R[ R\ R] R^ R_ R Ri RE Rt R8 R9 t
FTPHandlerR` Rƒ R• RN Rœ R ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt <module> s4 # „+4 S4
OHA YOOOO