[Python-talk] Sockets questions...

Wed Aug 26 09:49:59 EDT 2009

It would appear the best way for me to accelerate my Power Spectral 
Density program is to connect to an FFT server.  To do this requires the 
use of sockets.  Some issues to resolve are the data payload size.

Let me pass this by everyone to see if my reasoning (thought process) 
makes sense...  Please pipe up if this appears to be rubbish.  I have 
spent a bit of time think about this, well OK, I've walked around looking 
dazed, but I digress.

The FFTs that need to be calculated are ~10M double precision complex 
samples.  This is 160MB.  At a 800Mb/s link (measured link rate) this 
would take 1.6 sec to transfer from the client to the server (best case). 

The client would run in python, which would make for 'easy coding'.  Save 
for the sockets, things are mostly done.  Everyone can see the code at 
PySIG on Thursday, if they want.

The server would be written in C/C++ to take advantage of FFTW.  If anyone 
thinks I'm a noob at Python, wait til you see my C code...  Nonetheless, I 
need to use the FFTW library to get the performance required.  (200ms vs. 
15 sec)*80.

On the client side ( really the master program ): Since the data array is 
complex, and struct.pack does not seem to know about the type double 
complex, the array would need to be split into a real and complex part. 
Each part would need to be sent.  Each part is larger than the maximum 
packet size allowed by UDP (which is a puny 16 bits including the 20 byte 
header).  To avoid real crazyness, I guess the packet should be say 32K. 
Then lots of packets will be sent until the array is built up on the other 
side.  Make sense? 

For this app it would appear I'd use sendto( ) commands. 

Does python sendto( ) return the number of bytes sent? For UDP, what does 
one do if (numbytes sent) < numbytes ? The documentation doesn't make 
that clear to me. Maybe I just don't know where to look.

For the server side - I have to write an unpack, or embed python into my C 
code, to convert the data back to real and imaginary arrays.  Anyone have 
a link to a double (IEEE-754) unpack routine in C? 

Would it be easier to embed python to do the sockets, message handling, 
then launch FFTW? 

I'd love to try it all in python on the server side, but I only have 
'regular' python on the server.  I don't have scipy or numpy for it.  I 
don't have the skills to build it for that platform - (non-Intel/AMD). 
Where it gets ugly is building ATLAS and LAPACK.  The ATLAS/LAPACK build 
is a messy thing, somewhat iterative - and most definately hacky.  I was 
never able to get through it on the Cell Processor.  It was hard enough to 
do on an Intel CPU.

Sorry, that was rambling, I'm trying to sort through things at this point. 
 Comments?

Bruce Labitt
Autoliv Electronics
1011B Pawtucket Blvd, PO Box 1858
Lowell, MA  01853

Email: bruce.labitt at autoliv.com. 
Tel:  (978) 674-6526
Fax: (978) 674-6581 

******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an electronic signature and/or legally binding agreement in the absence of an express statement or Autoliv policy and/or procedure to the contrary. This E-mail and any attachments hereto are Autoliv property and may contain legally privileged, confidential and/or proprietary information. The recipient of this E-mail is prohibited from distributing, copying, forwarding or in any way disseminating any material contained within this E-mail without prior written permission from the author. If you receive this E-mail in error, please immediately notify the author and delete this E-mail. Autoliv disclaims all responsibility and liability for the consequences of any person who fails to abide by the terms herein. 
******************************