[Python-talk] Profile results... Better pack?

[bruce.labitt at autoliv.com]

Lloyd Kvam <python at venix.com> wrote on 10/19/2009 10:02:18 AM:

> On Thu, 2009-10-15 at 11:50 -0400, bruce.labitt at autoliv.com wrote:
> > Hmm, ~14.7 sec per FFT cycle - not so good.
> > 
> > Client side: (python/numpy)
> > 
> > 2.75 sec creating random data, slower than I thought
> > 7.00 sec packing data
> > 1.7 sec  sending data = very good > 700 Mibps
> 
> This is just idle curiosity, but did you post the solution to the
> network throughput issues?  I think you had been stuck at ~ 200 Mibps.
> 
> -- 
> Lloyd Kvam
> Venix Corp
> DLSLUG/GNHLUG library
> http://dlslug.org/library.html
> http://www.librarything.com/catalog/dlslug
> http://www.librarything.com/rsshtml/recent/dlslug
> http://www.librarything.com/rss/recent/dlslug
> 

Err, I was off to other things...  Turns out, that most of the issue was 
bad metrology on my part.  Once I started the timer just AFTER the 
transaction started, but before it completed, I was able to get an 
estimate of speed.  In other words, if one sets the timer too soon, then 
one gets an artificially low estimate.  <blush>

For general networking setup I used some of the hints at 
http://fasterdata.es.net/TCP-tuning/tcp-wan-perf.pdf

For my code I used numpy to speed things up.  On my laptop the routine 
executes pack in 6.9sec, numpy method in 0.4sec !  (I did that measurement 
just now to see if it was fast on another platform.)

Pack was designed to be general purpose.  It can handle heterogeneous 
data.  This is a complete example showing what I ended up with.

#!/usr/bin/env python
from numpy import random
import time
from struct import pack
 
lenfft = 10084200 
sigma = 0.01
stim = random.normal(0., sigma, lenfft )        # takes ~3.4 sec on my 
workstation
cmd = 4
 
fmt = '!h'+str(stim.size)+'d'
print fmt

startTime = time.time()
data1 = pack( fmt, cmd, *stim )
elapsed = time.time() -startTime

print "Time to pack data original %.6f sec " % elapsed

startTime2 = time.time()
str1 = pack( '!h', cmd )
str2 = stim.byteswap().tostring(order='C') # <== byteswap to convert to 
network order
                                                         # tostring 
converts double to string
data2 = str1+str2                                        # append str2 to 
str1.  done!
elapsed2 = time.time() -startTime2

print "Time to pack data new method %.6f sec " % elapsed2


-Bruce


******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an <br>electronic signature and/or legally binding agreement in the absence of an <br>express statement or Autoliv policy and/or procedure to the contrary.<br>This E-mail and any attachments hereto are Autoliv property and may contain legally <br>privileged, confidential and/or proprietary information.<br>The recipient of this E-mail is prohibited from distributing, copying, forwarding or in any way <br>disseminating any material contained within this E-mail without prior written <br>permission from the author. If you receive this E-mail in error, please <br>immediately notify the author and delete this E-mail.  Autoliv disclaims all <br>responsibility and liability for the consequences of any person who fails to <br>abide by the terms herein. <br>
******************************