[Python-talk] A few more socket related questions

Fri Aug 28 13:40:43 EDT 2009

Lloyd Kvam <lkvam at venix.com> wrote on 08/28/2009 12:46:32 PM:

> On Fri, 2009-08-28 at 12:05 -0400, bruce.labitt at autoliv.com wrote:
> > Thanks for peoples' observations at PySIG last night.  I heard folks 
> > muttering why use UDP.  This got me thinking about my decision.  Maybe 
I 
> > chose UDP for the wrong reasons.  So allow me a few questions... 
Forgive 
> > me if the questions are malformed, I really don't understand this 
stuff 
> > yet.
> > 
> > Using TCP, How would one send a large array >> MTU ?  Does it still 
need 
> > to be broken up into chunks? 
> 
> Not by you.
> You deal with data that makes sense to you.  The data will be broken
> into packets by the network stack.
> 

Is this true for UDP also? 

> > And the chunk size would be...? 
> 
> What ever you devise in terms of handling data on both sides.  Your
> chunk size is constrained by your application, memory, and those kinds
> of issues.

As far as I'm concerned, I want to send the whole array.  Ship it as it is 
generated.  Unfortunately, the array is large, so I have to be concerned 
with some of the mechanics.  The array is 10084200 double precision 
complex numbers - ~ 160MB.  Since there is no format code in struct.pack 
for double precision complex, the array will have to be broken into real 
and imaginary parts. 

> 
> > (In UDP a chunk is limited to a number slightly less than 2**16)
> > 
> > When would I consider using sendall?
> > 
> I think you would normally use sendall.
> 
> > When it comes down to it, what is the real advantage of tcp? 
> 
> TCP allows you to deal with streams of data - much like files.   UDP
> allows you to deal with packets of data - ideal for quick messages over
> the network.
> 
> > The socket.send documentation still says to check the return value for 
the 
> > number of bytes sent...  Is it just that tcp keeps trying to send? or 
that 
> > it 'guarantees' delivery.  When does it give up?
> 
> Sorry I could not make the meeting.  We have
> http://www.librarything.com/catalog.php?
> view=dlslug&searchall=1&deepsearch=foundations&shelf=list
> Foundations of Python Network Programming in the library
> 
> It covers these issues quite nicely with useful example code.
> 

That is good to know.  I was wondering if the book would be useful.

> > 
> > As I understand it, in tcp the client has to create and destroy a 
> > socket(s) for each transaction.
> > 
> > How does the client know if all the data is sent?
> > How does the server know it got everything?
> 
> Have the server send a confirmation.  How were you doing this with UDP?

Well, sheepish grin, it is not written yet, but... In the code so far, 
upon receipt of all the bytes, a confirmation will be sent, if not, a 
resend command is sent...  Hence, I was wondering whether tcp was worth 
the effort if there was a lot of book keeping to do.

> You can use the same approach, but the chunk size can be something that
> makes sense for you as opposed to being dictated by the packet size.
> 

Having to deal with a chunksize = arraysize is a great benefit to me.  One 
less bit of code to work out the details...  Too bad I just finished doing 
something like that...

> > 
> > Can the server send data to the client without a request?  (Delayed 
> > output?)
> 
> Yes.  Data will get buffered until a request finally pulls it out of the
> socket.  You can saturate the buffers and create a deadlock.  The book
> includes a cute deadlock example (pp 61-62) where buffers get saturated.

That is the kind of thing I'm worried about.  After all, the point of this 
whole exercise is to 'quickly' access and use an accelerator engine on the 
network.  If the time saved is minimal compared to doing it on my PC, 
there isn't any point to all this thrashing.

Sounds like I need to get a hold of a copy of the book.  I'll contact you 
offlist for that.

> 
> > As seen by the example below:
> > ----------------------------------------------------------------
> > client sends data for FFT.
> >                                                         Server starts 
FFT.
> >                                                         Server sends 
I'm 
> > busy
> > client receives I'm busy
> >                                                         Server sends 
> > results
> > client receives results
> 
> This approach makes more sense to me than the following one.  Your
> client knows that the data was sent successfully and that the FFT is
> underway.  You can have timeouts for both the data transmission and the
> FFT processing in the client.

I was, and am still a little unclear on how to construct the 
communications.  Would something like this be reasonable?

For every client command, 
1. client requests a socket to server.  thru magic this happens
2. client sends command to server.
3. the server must respond with an acknowlege message.  The acknowledge 
message can contain 'BUSY','ACK', or 'ACK'+data.  'ACK'+data shall be sent 
if the data is available.
4. if the command requires it, auxilliary data will be sent by the server 
to the client
5. on receipt of the acknowledgement (or ack + data) the client closes the 
connection

-Bruce

******************************
Neither the footer nor anything else in this E-mail is intended to or constitutes an electronic signature and/or legally binding agreement in the absence of an express statement or Autoliv policy and/or procedure to the contrary. This E-mail and any attachments hereto are Autoliv property and may contain legally privileged, confidential and/or proprietary information. The recipient of this E-mail is prohibited from distributing, copying, forwarding or in any way disseminating any material contained within this E-mail without prior written permission from the author. If you receive this E-mail in error, please immediately notify the author and delete this E-mail. Autoliv disclaims all responsibility and liability for the consequences of any person who fails to abide by the terms herein. 
******************************