[Python-talk] How would one do something like this?

[Lloyd Kvam]

I think Kent pointed this out some time ago.

http://www.dabeaz.com/generators-uk/index.html


The suggested style of coding is generally easy to read and modify
because it so closely mimics the Unix pipe approach.  The use of
iterators also tends to be fairly efficient - especially as compared to
writing C style code in Python.

On Wed, 2008-12-10 at 15:00 -0500, bruce.labitt at autoliv.com wrote:
> Exactly what I need.  (I only have 8G... :) )
> 
> > itertools has an islice function.
> > 
> > from itertools import islice
> > inf = open('tmp.txt')
> > start = 4099998
> > length = 1000
> > chunk = islice(inf, start, start+length, 1)
> > for line in chunk:
> >     process(line)
> > 
> 
> Very slick!  This greatly reduced the time to read in the file!
> Ahem, 
> once 
> the indexing (start point) was fixed that is.  I just tried it on a
> 2M 
> line file to check it out.
> A few minutes compared to >30 minutes.
> 
-- 
Lloyd Kvam
Venix Corp
DLSLUG/GNHLUG library
http://dlslug.org/library.html
http://www.librarything.com/catalog/dlslug
http://www.librarything.com/rsshtml/recent/dlslug
http://www.librarything.com/rss/recent/dlslug