[Python-talk] A few more socket related questions
Lloyd Kvam
lkvam at venix.com
Fri Aug 28 14:56:08 EDT 2009
On Fri, 2009-08-28 at 13:40 -0400, bruce.labitt at autoliv.com wrote:
> Lloyd Kvam <lkvam at venix.com> wrote on 08/28/2009 12:46:32 PM:
>
> > On Fri, 2009-08-28 at 12:05 -0400, bruce.labitt at autoliv.com wrote:
> > > Thanks for peoples' observations at PySIG last night. I heard folks
> > > muttering why use UDP. This got me thinking about my decision. Maybe
> I
> > > chose UDP for the wrong reasons. So allow me a few questions...
> Forgive
> > > me if the questions are malformed, I really don't understand this
> stuff
> > > yet.
> > >
> > > Using TCP, How would one send a large array >> MTU ? Does it still
> need
> > > to be broken up into chunks?
> >
> > Not by you.
> > You deal with data that makes sense to you. The data will be broken
> > into packets by the network stack.
> >
>
> Is this true for UDP also?
I'm inclined to say no, but I've never put it to the test. With UDP
there is no guarantee that any packet got delivered. So even if you
could send a stream of packets, you need some scheme for verifying
delivery. So if your data is too large for a packet, you need to handle
the packet management issues. TCP has packet management "for free".
>
> > > And the chunk size would be...?
> >
> > What ever you devise in terms of handling data on both sides. Your
> > chunk size is constrained by your application, memory, and those kinds
> > of issues.
>
> As far as I'm concerned, I want to send the whole array. Ship it as it is
> generated. Unfortunately, the array is large, so I have to be concerned
> with some of the mechanics. The array is 10084200 double precision
> complex numbers - ~ 160MB. Since there is no format code in struct.pack
> for double precision complex, the array will have to be broken into real
> and imaginary parts.
>
You could devise a simple protocol where you:
* send the count of numbers.
* send the numbers (160MB or whatever matches the count). You can
send as a huge block or in smaller blocks as you pull from the
generator. (Though I've never actually sent a block of more
than a few MB, I would expect it to work.)
* recv delivery confirmation
* recv results
That allows for testing with small chunks of data. TCP will deliver the
160MB or die trying. You catch the exception or timeout waiting for
confirmation. You can also timeout on the recv results processing.
> >
> > > (In UDP a chunk is limited to a number slightly less than 2**16)
> > >
> > > When would I consider using sendall?
> > >
> > I think you would normally use sendall.
> >
> > > When it comes down to it, what is the real advantage of tcp?
> >
> > TCP allows you to deal with streams of data - much like files. UDP
> > allows you to deal with packets of data - ideal for quick messages over
> > the network.
> >
> > > The socket.send documentation still says to check the return value for
> the
> > > number of bytes sent... Is it just that tcp keeps trying to send? or
> that
> > > it 'guarantees' delivery. When does it give up?
> >
> > Sorry I could not make the meeting. We have
> > http://www.librarything.com/catalog.php?
> > view=dlslug&searchall=1&deepsearch=foundations&shelf=list
> > Foundations of Python Network Programming in the library
> >
> > It covers these issues quite nicely with useful example code.
> >
>
> That is good to know. I was wondering if the book would be useful.
>
> > >
> > > As I understand it, in tcp the client has to create and destroy a
> > > socket(s) for each transaction.
> > >
> > > How does the client know if all the data is sent?
> > > How does the server know it got everything?
> >
> > Have the server send a confirmation. How were you doing this with UDP?
>
> Well, sheepish grin, it is not written yet, but... In the code so far,
> upon receipt of all the bytes, a confirmation will be sent, if not, a
> resend command is sent... Hence, I was wondering whether tcp was worth
> the effort if there was a lot of book keeping to do.
>
> > You can use the same approach, but the chunk size can be something that
> > makes sense for you as opposed to being dictated by the packet size.
> >
>
> Having to deal with a chunksize = arraysize is a great benefit to me. One
> less bit of code to work out the details... Too bad I just finished doing
> something like that...
>
> > >
> > > Can the server send data to the client without a request? (Delayed
> > > output?)
> >
> > Yes. Data will get buffered until a request finally pulls it out of the
> > socket. You can saturate the buffers and create a deadlock. The book
> > includes a cute deadlock example (pp 61-62) where buffers get saturated.
>
> That is the kind of thing I'm worried about. After all, the point of this
> whole exercise is to 'quickly' access and use an accelerator engine on the
> network. If the time saved is minimal compared to doing it on my PC,
> there isn't any point to all this thrashing.
>
> Sounds like I need to get a hold of a copy of the book. I'll contact you
> offlist for that.
>
> >
> > > As seen by the example below:
> > > ----------------------------------------------------------------
> > > client sends data for FFT.
> > > Server starts
> FFT.
> > > Server sends
> I'm
> > > busy
> > > client receives I'm busy
> > > Server sends
> > > results
> > > client receives results
> >
> > This approach makes more sense to me than the following one. Your
> > client knows that the data was sent successfully and that the FFT is
> > underway. You can have timeouts for both the data transmission and the
> > FFT processing in the client.
>
> I was, and am still a little unclear on how to construct the
> communications. Would something like this be reasonable?
Yes. Keep responses short. Use a finite state machine unless your
protocol is trivial
>
> For every client command,
> 1. client requests a socket to server. thru magic this happens
> 2. client sends command to server.
So far, the only command appears to be
process nnnnnnnnnnnn numbers
<stream of nnnnnn numbers>
> 3. the server must respond with an acknowlege message. The acknowledge
> message can contain 'BUSY','ACK', or 'ACK'+data. 'ACK'+data shall be sent
> if the data is available.
The server could simply send('ACK') for the process command and start
processing the number stream.
Then send
result <length>
<result stream>
> 4. if the command requires it, auxilliary data will be sent by the server
> to the client
You should include a length specifier in your protocol (you may have
been doing that implicitly) since you're constructing a protocol. The
simpler case is too simply assume you have the whole transmission when
recv returns no data. Essentially that is the netcat approach.
> 5. on receipt of the acknowledgement (or ack + data) the client closes the
> connection
Right, close the connection after getting your results.
>
You could also consider using Larry's netcat (nc) suggestion.
>
> -Bruce
>
>
> ******************************
> Neither the footer nor anything else in this E-mail is intended to or constitutes an <br>electronic signature and/or legally binding agreement in the absence of an <br>express statement or Autoliv policy and/or procedure to the contrary.<br>This E-mail and any attachments hereto are Autoliv property and may contain legally <br>privileged, confidential and/or proprietary information.<br>The recipient of this E-mail is prohibited from distributing, copying, forwarding or in any way <br>disseminating any material contained within this E-mail without prior written <br>permission from the author. If you receive this E-mail in error, please <br>immediately notify the author and delete this E-mail. Autoliv disclaims all <br>responsibility and liability for the consequences of any person who fails to <br>abide by the terms herein. <br>
> ******************************
> _______________________________________________
> Python-talk mailing list
> Python-talk at dlslug.org
> http://dlslug.org/mailman/listinfo/python-talk
--
Lloyd Kvam
Venix Corp.
1 Court Street, Suite 378
Lebanon, NH 03766-1358
voice: 603-653-8139
fax: 320-210-3409
More information about the Python-talk
mailing list