[Python-talk] Was Generator Question: Now list comprehension

[Bruce Labitt]

Bruce Labitt wrote:
> Lloyd Kvam wrote:
>> On Fri, 2009-08-14 at 17:19 -0400, bruce.labitt at autoliv.com wrote:
>>  
>>> kent3737 at gmail.com wrote on 08/13/2009 12:42:04 PM:
>>>
>>>    
>>>> On Thu, Aug 13, 2009 at 11:58 AM, <bruce.labitt at autoliv.com> wrote:
>>>>      
>>>>> Will the group of generators cascade the
>>>>> same?
>>>>>         
>>>> Yes, all that is changing is the source of the first generator (from a
>>>> generator function instead of a generator expression).
>>>>
>>>>      
>>>>> Say we use the previous example.  Something like this, or would I 
>>>>> need         
>>> to
>>>    
>>>>> do something different?
>>>>>
>>>>> seeds = [ <some sequence that defines the arrays to be created> ]
>>>>> score = 0
>>>>> while score < threshold:
>>>>>   #arrays = (make_array(seed) for seed in seeds)
>>>>>    arrays = make_array(seed)    # instead of line above
>>>>>    fft_results = (compute_fft(array) for array in arrays)
>>>>>    powers = (compute_power(result) for result in fft_results)
>>>>>    score, seeds = optimize(powers)
>>>>>         
>>>> Right.
>>>>
>>>>      
>>>>> This is really powerful stuff...
>>>>>         
>>>> Yes :-)
>>>>
>>>> Kent
>>>>       
>>> Question about the last stage, what I would call the consumer of the 
>>> generator data.
>>>
>>> I have written make_array, compute_fft, and compute_power.  Instead 
>>> of using the routine optimize (which I have not written yet), I want 
>>> to test what I have.
>>>
>>> I am trying to write a function called max_power.
>>>
>>> The output of the function is the element by element column maximum 
>>> of the arrays in powers.
>>>
>>> Consider the following as a matrix
>>>
>>>     
>>
>> matrix = [    # added commas
>>  
>>> [ 0, 1, 2, 3, 4],
>>> [ 5, 6, 7, 8, 9],
>>> [10, 0, 1, 9, 7],
>>>     
>> ]    # 2 dim matrix as list of lists
>>
>>  
>>>  10 6 7 9 9  <== column max
>>>     
>>
>> # the transpose function is zip( *)
>> # find the max of each col
>> # collect the maxs with a list comprehension
>> In [6]: maxs = [max(col) for col in zip(*matrix)]
>>
>> In [7]: maxs
>> Out[7]: [10, 6, 7, 9, 9]
>>
>>  
>>> There is a numpy function called maximum
>>> It returns an array c whose ith element is the maximum of (ai, bi)
>>>
>>> c = maximum( a, b ), ci = max(ai, bi) for i in c
>>>     
>>
>> maxes = max(col for col
>>  
>>> The output of max_power should be one array that is the column 
>>> maximum of the matrix
>>>
>>> [ powers row 0 ] first iteration of generator
>>> [ powers row 1 ] 2nd
>>>     .
>>>     .
>>>     .
>>> [ powers row last ] last
>>>
>>>
>>> So I want max_power to consume all of the data from the previous 
>>> stages and only return
>>> a single row which is the column by column maximum.
>>>
>>> Now there really isn't a real matrix here, just a series of arrays 
>>> (or rows) that are output
>>> by the previous generator.
>>>     
>>
>> list( generator output) will collect the outputs into a list
>> I presume     list( powers)
>> will create the matrix as a list of list
>>
>> maxs = [ max(col) for col in zip( *list( powers)) ]
>>
>> I hope that's close to what you're looking for.
>>
>>   
> Lloyd, it could be, but I was hoping to stay within the numpy 
> paradigm.  This is because there are optimized array operations in 
> numpy and scipy.  The operations are vectorized.  This is 
> significantly faster than a for loop.
>
> The arrays I am dealing with are ~10M long.  I've got to do complex 
> math on all the elements and process a minimum of 100 arrays.  
> Eventually I may need to do 4096 arrays.  I am trying to exploit the 
> numpy module as much as possible.  It keeps me from having to write C 
> :).  If I write code in C, 1) It takes much longer, 2) It can be 
> faster because I can exploit multiple cores, 3) It makes my head hurt...
>
> If you are familiar with Matlab, the issue is similar.  Vectorized 
> operations can be 100x faster than for loops.
>
> Now all that being said, I will try it out because it is a good thing 
> to know.  :)  Some day I may need to do something like this, where 
> numpy is not available or not a good choice.
>
>
List comprehensions!  Ooh, thanks for pointing that out.  Sorry I was 
too dense to understand what you were saying.  After replying, I cracked 
open the "book".  In the same chapter as lambda, apply, map, filter, 
reduce, 'list comprehensions and matrices', and generators!  Chapter 17, 
Advanced Function Topics, in "Learning Python", by Mark Lutz.  Quite the 
meaty chapter.

It would appear my needs have brought me to this chapter...  So much to 
read...  There is a section on the speed of these methods, too!  Too bad 
there is no comparison with numpy.  Hmm, maybe that is a little side 
project...

Thanks!