[Python-talk] Sockets questions...

[Larry Keber]

bruce.labitt at autoliv.com wrote:
> It would appear the best way for me to accelerate my Power Spectral 
> Density program is to connect to an FFT server.  To do this requires the 
> use of sockets.  Some issues to resolve are the data payload size.
>
> Let me pass this by everyone to see if my reasoning (thought process) 
> makes sense...  Please pipe up if this appears to be rubbish.  I have 
> spent a bit of time think about this, well OK, I've walked around looking 
> dazed, but I digress.
>
> The FFTs that need to be calculated are ~10M double precision complex 
> samples.  This is 160MB.  At a 800Mb/s link (measured link rate) this 
> would take 1.6 sec to transfer from the client to the server (best case). 
>
> The client would run in python, which would make for 'easy coding'.  Save 
> for the sockets, things are mostly done.  Everyone can see the code at 
> PySIG on Thursday, if they want.
>
> The server would be written in C/C++ to take advantage of FFTW.  If anyone 
> thinks I'm a noob at Python, wait til you see my C code...  Nonetheless, I 
> need to use the FFTW library to get the performance required.  (200ms vs. 
> 15 sec)*80.
>
>   

Maybe you could use netcat to handle the socket stuff on the server 
side? (I don't know what the environment is there, but if python is 
available, I'm sure netcat is).
http://en.wikipedia.org/wiki/Netcat

Have it wait for a connection, then run your program (with netcat 
linking the socket input to your C program's stdin) similar to:

nc -l -p 1234 -e /bin/bash


substituting your program for /bin/bash.


> On the client side ( really the master program ): Since the data array is 
> complex, and struct.pack does not seem to know about the type double 
> complex, the array would need to be split into a real and complex part. 
> Each part would need to be sent.  Each part is larger than the maximum 
> packet size allowed by UDP (which is a puny 16 bits including the 20 byte 
> header).  To avoid real crazyness, I guess the packet should be say 32K. 
> Then lots of packets will be sent until the array is built up on the other 
> side.  Make sense? 
>
> For this app it would appear I'd use sendto( ) commands. 
>
> Does python sendto( ) return the number of bytes sent?  For UDP, what does 
> one do if (numbytes sent) < numbytes ?  The documentation doesn't make 
> that clear to me.  Maybe I just don't know where to look.
>
>   

I think for something like this you could just use TCP and 
socket.sendall() ?

> For the server side - I have to write an unpack, or embed python into my C 
> code, to convert the data back to real and imaginary arrays.  Anyone have 
> a link to a double (IEEE-754) unpack routine in C? 
>
> Would it be easier to embed python to do the sockets, message handling, 
> then launch FFTW? 
>   

Or write the server side in python to simply receive the data, massage 
it appropriately, then call your C program to process it.

Too busy right now to think more, good luck :-).

LArry
> I'd love to try it all in python on the server side, but I only have 
> 'regular' python on the server.  I don't have scipy or numpy for it.  I 
> don't have the skills to build it for that platform - (non-Intel/AMD). 
> Where it gets ugly is building ATLAS and LAPACK.  The ATLAS/LAPACK build 
> is a messy thing, somewhat iterative - and most definately hacky.  I was 
> never able to get through it on the Cell Processor.  It was hard enough to 
> do on an Intel CPU.
>
> Sorry, that was rambling, I'm trying to sort through things at this point. 
>  Comments?
>
>
>
> Bruce Labitt
> Autoliv Electronics
> 1011B Pawtucket Blvd, PO Box 1858
> Lowell, MA  01853
>
> Email: bruce.labitt at autoliv.com. 
> Tel:  (978) 674-6526
> Fax: (978) 674-6581 
>
> ******************************
> Neither the footer nor anything else in this E-mail is intended to or constitutes an <br>electronic signature and/or legally binding agreement in the absence of an <br>express statement or Autoliv policy and/or procedure to the contrary.<br>This E-mail and any attachments hereto are Autoliv property and may contain legally <br>privileged, confidential and/or proprietary information.<br>The recipient of this E-mail is prohibited from distributing, copying, forwarding or in any way <br>disseminating any material contained within this E-mail without prior written <br>permission from the author. If you receive this E-mail in error, please <br>immediately notify the author and delete this E-mail.  Autoliv disclaims all <br>responsibility and liability for the consequences of any person who fails to <br>abide by the terms herein. <br>
> ******************************
> _______________________________________________
> Python-talk mailing list
> Python-talk at dlslug.org
> http://dlslug.org/mailman/listinfo/python-talk
>