[Python-talk] Generator Question

[Bruce Labitt]

Lloyd Kvam wrote:
> On Fri, 2009-08-14 at 17:19 -0400, bruce.labitt at autoliv.com wrote:
>   
>> kent3737 at gmail.com wrote on 08/13/2009 12:42:04 PM:
>>
>>     
>>> On Thu, Aug 13, 2009 at 11:58 AM, <bruce.labitt at autoliv.com> wrote:
>>>       
>>>> Will the group of generators cascade the
>>>> same?
>>>>         
>>> Yes, all that is changing is the source of the first generator (from a
>>> generator function instead of a generator expression).
>>>
>>>       
>>>> Say we use the previous example.  Something like this, or would I need 
>>>>         
>> to
>>     
>>>> do something different?
>>>>
>>>> seeds = [ <some sequence that defines the arrays to be created> ]
>>>> score = 0
>>>> while score < threshold:
>>>>   #arrays = (make_array(seed) for seed in seeds)
>>>>    arrays = make_array(seed)    # instead of line above
>>>>    fft_results = (compute_fft(array) for array in arrays)
>>>>    powers = (compute_power(result) for result in fft_results)
>>>>    score, seeds = optimize(powers)
>>>>         
>>> Right.
>>>
>>>       
>>>> This is really powerful stuff...
>>>>         
>>> Yes :-)
>>>
>>> Kent
>>>       
>> Question about the last stage, what I would call the consumer of the 
>> generator data.
>>
>> I have written make_array, compute_fft, and compute_power.  Instead of 
>> using the 
>> routine optimize (which I have not written yet), I want to test what I 
>> have.
>>
>> I am trying to write a function called max_power.
>>
>> The output of the function is the element by element column maximum of the 
>> arrays in powers.
>>
>> Consider the following as a matrix
>>
>>     
>
> matrix = [	# added commas
>   
>> [ 0, 1, 2, 3, 4],
>> [ 5, 6, 7, 8, 9],
>> [10, 0, 1, 9, 7],
>>     
> ]	# 2 dim matrix as list of lists
>
>   
>>  10 6 7 9 9  <== column max
>>     
>
> # the transpose function is zip( *)
> # find the max of each col
> # collect the maxs with a list comprehension
> In [6]: maxs = [max(col) for col in zip(*matrix)]
>
> In [7]: maxs
> Out[7]: [10, 6, 7, 9, 9]
>
>   
>> There is a numpy function called maximum
>> It returns an array c whose ith element is the maximum of (ai, bi)
>>
>> c = maximum( a, b ), ci = max(ai, bi) for i in c
>>     
>
> maxes = max(col for col
>   
>> The output of max_power should be one array that is the column maximum of 
>> the matrix
>>
>> [ powers row 0 ] first iteration of generator
>> [ powers row 1 ] 2nd
>>     .
>>     .
>>     .
>> [ powers row last ] last
>>
>>
>> So I want max_power to consume all of the data from the previous stages 
>> and only return
>> a single row which is the column by column maximum.
>>
>> Now there really isn't a real matrix here, just a series of arrays (or 
>> rows) that are output
>> by the previous generator.
>>     
>
> list( generator output) will collect the outputs into a list
> I presume 
> 	list( powers)
> will create the matrix as a list of list
>
> maxs = [ max(col) for col in zip( *list( powers)) ]
>
> I hope that's close to what you're looking for.
>
>   
Lloyd, it could be, but I was hoping to stay within the numpy paradigm.  
This is because there are optimized array operations in numpy and 
scipy.  The operations are vectorized.  This is significantly faster 
than a for loop.

The arrays I am dealing with are ~10M long.  I've got to do complex math 
on all the elements and process a minimum of 100 arrays.  Eventually I 
may need to do 4096 arrays.  I am trying to exploit the numpy module as 
much as possible.  It keeps me from having to write C :).  If I write 
code in C, 1) It takes much longer, 2) It can be faster because I can 
exploit multiple cores, 3) It makes my head hurt...

If you are familiar with Matlab, the issue is similar.  Vectorized 
operations can be 100x faster than for loops.

Now all that being said, I will try it out because it is a good thing to 
know.  :)  Some day I may need to do something like this, where numpy is 
not available or not a good choice.

This list has been very helpful for the dissemination of Python 
knowledge.  As a python noob of only one year I'm still amazed at how 
easy it has been to get stuff done in python.  

Thanks for everyone's help!

Bruce