Line Delimited Network IO - "Recommended Approach"

debio Joined: 05/24/08 Posts: 17

Posted: 12/29/08 22:19:43 Modified: 12/29/08 22:27:18 I'm trying to write an app that needs to read lines of text from network sockets in a non-blocking manner, and I'm not sure how I should read them. In Java, I would just have a thread for each socket that would make a blocking call (to the equivalent of LineInput?.readln) to get a line. This worked because in Java, closing a socket caused all threads making use of the socket to return failure, so I could stop my "listener threads" fairly simply. I'm not sure if this approach would work in D, although even if it didn't, I could make it work by setting all "listener threads" to be daemon threads. While this would waste resources, the program would function normally. In C and C++, I would use normal network functions and store incoming characters to a buffer, searching for line endings. This approach was more that a little inefficient and unsafe, but it worked. It could be made to work in D, but it seems like there should be a better way.

Author	Message
larsivi Joined: 03/27/04 Posts: 614	Posted: 12/29/08 23:29:41 For the non-blocking IO (whether sockets or files) you should check out tango.io.selector.* - the Lines can then be read from the socket as you normally would when you know there is input.
debio Joined: 05/24/08 Posts: 17	Posted: 12/29/08 23:44:25 So if I attack a LineInput? to a SocketConduit? and then put the LineInput? into a Selector, the Selector doesn't fire an event until there's a line to be read in the LineInput?? Tango is amazing. Thanks.
larsivi Joined: 03/27/04 Posts: 614	Posted: 12/30/08 01:14:41 Hehe, try it - I haven't used it myself :)
debio Joined: 05/24/08 Posts: 17	Posted: 12/30/08 02:45:45 -- Modified: 12/31/08 22:12:26 by debio -- Modified 3 Times Ergh, that doesn't seem to work. Selectors will only work with conduits because they require a conduit to implement ISelectable, which LineInput? does not. Because ISelectable requires that the OS be able to get a handle to the conduit, there's no way to LineInput? to implement that interface. If I just attach a LineInput? to the conduit and put the conduit into a Selector, the Selector will flag it as readable if there's any text, not necessarily a complete line of text. Am I missing something? Is there another way?
debio Joined: 05/24/08 Posts: 17	Posted: 12/31/08 22:13:14 -- Modified: 12/31/08 23:06:26 by debio Should I just go back to the C/C++ way? Does Tango have a buffer class that would work well here? If I do the C/C++ way, should I do everything manually with an array (I think there would be a bunch of unavoidable heap allocations this way) or use tango.text.Text (it either moves things within the buffer or does the same heap allocations)?
larsivi Joined: 03/27/04 Posts: 614	Posted: 01/01/09 00:55:34 I'm sorry that most of those familiar with this API seems to be on xmas vacation - however, Tango do have buffering available. If you use the latest trunk, you will find buffer streams that you can wrap around the socket stream and let that be wrapped by the line stream again. If you use earlier Tango aka 0.99.7, you could use tango.io.GrowBuffer? (which should implement the necessary interfaces to have it properly working with the appropriate Line iteration interface). I would love it if your efforts (and if you are successful) could be condensed into an example or tutorial for publication here - it is a somewhat complex area in terms of understanding, although it should make your code much more efficient and probably easier to handle later on.
debio Joined: 05/24/08 Posts: 17	Posted: 02/04/09 00:44:53 -- Modified: 02/05/09 02:02:21 by debio -- Modified 3 Times I'm finally getting around to working at this again. Long story. I'm thinking I could just put all the SocketConduit?'s into a Selector and copy data from a Socket to a GrowBuffer? when data is available. From there, I would just have a LineInput? attached to the GrowBuffer? that I would try to read lines from. When there are no lines in the buffer, this will fail. I'll need to go back through some documentation, but is GrowBuffer? circular? I remember Tango having a circular buffer, but I don't remember where. Couldn't I just make a BufferStream? that wraps around the GrowBuffer? and use BufferStream?.copy(SocketConduit?) when a SocketConduit? has data? EDIT: GAH, I'm an idiot. I just reread your post that this is pretty much exactly what you suggested. I'll get back to coding then...
debio Joined: 05/24/08 Posts: 17	Posted: 02/08/09 19:51:30 -- Modified: 02/08/09 22:45:53 by debio -- Modified 3 Times Okay, so far I've hit one large snag. I'm using SocketConduits?, which are effectively endless sources of data, but the StreamIterator? tree assumes that the visible end of the stream is the end of the stream; no more data will arrive. This results in this function from StreamIterator: private bool consume () { if (input.next (&scan)) return true; auto tmp = input.slice (buffer.readable); slice = (cast(T*) tmp.ptr) [0 .. tmp.length/T.sizeof]; return false; } I think my problem is line 7, which looks like it uses the remainder of the buffered input if no token can be found. I'm currently trying to figure out how to avoid that (in a graceful way). EDIT: Yes, this is the problem. If I chain the buffer to the socket and chain the LineIterator? to the Buffer, any calls to try and find a line that isn't there cause the socket to block while it tries to read more data, which I'm trying to avoid. On the other hand, if I don't chain the buffer to the socket, the remainder of the buffer is returned instead of a line when a line cannot be found. It looks like the Iterator isn't designed to return failure until it hits an EOF, so I'm not sure what I can do aside from reimplementing the whole thing. Which I may end up doing. I should clarify that I can't see anything wrong with the conceptual design of the current system, which deals with streams that go on forever for which data can be retreived and streams that reach a definite end. The problem is that I'm dealing with streams that go on forever but for which data is not immediately available.
debio Joined: 05/24/08 Posts: 17	Posted: 02/08/09 22:15:05 Well, I have it working, but I don't think I should write up my method... I unceremoniously copied all the code from StreamIterator? and added the scan() method from LineIterator?, and then altered the consume() method and the next() method to make it all work. When no line ending is found, I set slice.length to 0 and return false when client code requests data. Hopefully I'll find a more elegant way of doing this, but I can't see a way to do it while staying within the StreamIterator? tree. If you would still like me to write up what I did, I will gladly do so.