| View previous topic :: View next topic |
| Author |
Message |
teqdruid
Joined: 11 May 2004
Posts: 390
Location: UMD
|
 Posted: Fri Dec 02, 2005 10:01 pm Post subject: GrowableBuffers don't grow! |
 |
|
So for larger responses my XML-RPC server library fails. Why? I use a growable buffer with an initial capacity of 1024 bytes. Once my XML response goes past that, the UMango.StringEncoder8 gets in an infinite loop trying to add the information to the buffer. This particular buffer cannot be flushed, so it must grow to accomodate more info. The IBuffer.write method, however, does not make a call to the grow method, thus infinite loop.
I'm not sure if IBuffer.write is the proper place to place the call to grow, since it doesn't know exactly how much to grow by, but here's a patch anyway. I also improved the grow() documentation.
| Code: | Index: mango/io/Buffer.d
===================================================================
--- mango/io/Buffer.d (revision 571)
+++ mango/io/Buffer.d (working copy)
@@ -286,8 +286,8 @@
/***********************************************************************
- Overridable method to grow the buffer size when it becomes
- full. Default is to not grow at all.
+ Overridable method to grow the buffer to the specified size
+ when it becomes full. Default is to not grow at all.
***********************************************************************/
@@ -479,6 +479,9 @@
{
limit += count;
assert (limit <= capacity);
+ if (limit == capacity) {
+ grow(capacity + cast(uint)(capacity*1.5));
+ }
}
return count;
}
@@ -651,9 +654,10 @@
/***********************************************************************
- Overridable method to grow the buffer size when it becomes
- full. Default is to not grow at all.
-
+ Overridable method to grow the buffer to the specified size
+ when it becomes full. GrowableBuffer.grow(size) will not grow
+ the buffer by a factor larger than 2 for each call.
+
***********************************************************************/
protected override bool grow (uint size)
Index: mango/io/model/IBuffer.d
===================================================================
--- mango/io/model/IBuffer.d (revision 571)
+++ mango/io/model/IBuffer.d (working copy)
@@ -294,8 +294,8 @@
/***********************************************************************
- Overridable method to grow the buffer size when it becomes
- full. Default is to not grow at all.
+ Overridable method to grow the buffer to the specified size
+ when it becomes full. Default is to not grow at all.
***********************************************************************/
|
~John |
|
| Back to top |
|
 |
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Fri Dec 02, 2005 10:34 pm Post subject: |
|
|
Hmmm. GrowBuffer was really intended to grow via append() only, But I certainly see the issue. I'm doing quite a bit of work in this area right now, so can hopefully address it. Can you describe the scenario a bit further please?
I'm guessing you have some generated content, in wchar or dchar form, which you need to pump out into a Utf8-encoded response buffer before setting "content-length" to the resultant Utf8 content?
- Kris |
|
| Back to top |
|
|
teqdruid
Joined: 11 May 2004
Posts: 390
Location: UMD
|
| Posted: Sat Dec 03, 2005 1:53 pm Post subject: |
|
|
| Code: | + if (limit == capacity) {
+ grow(capacity + cast(uint)(capacity*1.5));
+ } |
The first thing I though of when I woke up this morning is how stupid that middle line of code is. If should be:
| Code: | + if (limit == capacity) {
+ grow(cast(uint)(capacity*1.5));
+ } |
Wow. |
|
| Back to top |
|
|
teqdruid
Joined: 11 May 2004
Posts: 390
Location: UMD
|
| Posted: Sat Dec 03, 2005 2:16 pm Post subject: |
|
|
| kris wrote: | Hmmm. GrowBuffer was really intended to grow via append() only, But I certainly see the issue. I'm doing quite a bit of work in this area right now, so can hopefully address it. Can you describe the scenario a bit further please?
I'm guessing you have some generated content, in wchar or dchar form, which you need to pump out into a Utf8-encoded response buffer before setting "content-length" to the resultant Utf8 content?
- Kris |
That's precisly it. The XmlRpcServer has to serialize the response object to XML before it sends it so that it knows the length of the data it's sending for the HTTP-Content-Length header. One way to do this would be to create some sort of MemoryConduit that acted like a GrowableBuffer and serialize everything to that. Then I'll end up with the entire thing in memory before I send it. Since this is bascially duplicating the behavior of GrowableBuffer, however, I was hoping I wouldn't have to do it.
Having to send this content-length before hand is a real PITA and eats a lot more memory than I'd like. Oh well 
Plus, GrowableBuffer only growing when data is added in a certain way and not others seems pretty inconsistent, especially since it's not overtly obvious which adding method is being used when I serialize via some sort of writer- it actually took me awhile to figure out what was going on. Either GrowableBuffer should act consistently all the time, or it shouldn't exist because it's not viable in the Mango IO framework.
~John |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Sat Dec 03, 2005 3:06 pm Post subject: |
|
|
OK; thanks. Next release will have a much cleaner way to do this.
Fair comments on the Buffer and the grow() method ... though there really is only one supported means of adding to a buffer, via append(). The write/read Buffer methods were intended as low-level access to the existing internals.
But the real problem is that the Utf converter hangs when it runs out of space
I have the fixes, but there's some dependencies ... |
|
| Back to top |
|
|
teqdruid
Joined: 11 May 2004
Posts: 390
Location: UMD
|
| Posted: Sat Dec 03, 2005 8:39 pm Post subject: |
|
|
| kris wrote: | | But the real problem is that the Utf converter hangs when it runs out of space |
I agree, but (from my perspective at least) that's a much more complex problem to solve.
| Quote: |
I have the fixes, but there's some dependencies ... |
Dependencies? How such? |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Sat Dec 03, 2005 10:14 pm Post subject: |
|
|
| teqdruid wrote: | | Dependencies? How such? |
Just meant I'd have to check in a bunch of stuff that's not ready yet. Hopefully I'll have something good-to-go tomorrow PM  |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Mon Dec 05, 2005 10:36 pm Post subject: |
|
|
There's been a flurry of check-ins recently. It's now at a reasonably clean stage to have a go at the following, if you wish.
Recap: need to gather a bunch of wchar content, convert it to some encoding, and get the resultant byte length. There's three different ways to accomplish this:
1) Use a plain GrowBuffer to write all the content. Then use
| Code: | | char[] result = Unicode.toUtf8 (cast(wchar[]) buffer.toString()); |
... to convert it all at once. This is the most efficient of all ~ will probably take less than 10 microseconds to convert the output.
2) Attach an AbstractEncoder to your Writer, as you've been doing, but use the UnicodeImporter from io.BufferCodec instead. All the ICU codecs stll operate as before, but they currently cannot expand the buffer. This will be slower than #1, since each write is converted at that time.
3) Attach a filter to the conduit. You're presumably using SocketConduit for output, so you could potentially attach a UnicodeFilter to it. Conduit filters are great for this kind of thing, where a conversion should be applied to the entire stream as it goes by. Other filter examples include Endian swapping, and Zip compression.
Option 1 and 2 are available now. Option 3 has been available for a long time, but the filters have just not been written. I checked in an EndianFilter as an example, and will try to get a UnicodeFilter done later this week.
For now, if it were me, I'd choose option 1. Note that GrowBuffer is now in its own module. I had to move it as part of the cleanup effort (which I believe is now finished  ) |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Wed Dec 07, 2005 7:14 pm Post subject: |
|
|
The new checks and balances for binary/text conflicts with Readers/Writers is checked in. It works great for files, but I need to make a change to the SocketConduit ctors to make it work properly there also.
The difficulty is figuring out whether to default a SocketConduit to text or binary. Many existing uses are for HTML or XML streams. These should be text based. Others are for binary streams between peers. These should clearly not be text based
What should the default be? |
|
| Back to top |
|
|
teqdruid
Joined: 11 May 2004
Posts: 390
Location: UMD
|
| Posted: Wed Dec 07, 2005 7:53 pm Post subject: |
|
|
| kris wrote: | The new checks and balances for binary/text conflicts with Readers/Writers is checked in. It works great for files, but I need to make a change to the SocketConduit ctors to make it work properly there also.
The difficulty is figuring out whether to default a SocketConduit to text or binary. Many existing uses are for HTML or XML streams. These should be text based. Others are for binary streams between peers. These should clearly not be text based
What should the default be? |
Defaults? We don't need no stinking defaults!
The only reason I can think of for having a default is to not break existing code. Assuming the last statement to be true, go with whatever the current code uses- probably text- and deprecate the ctor. If you know of any current code that uses a binary SocketConduit, it might be better to not have a default and break the code- better it not compile than silently fail.
I haven't had the chance to look at anything for a few days and probably won't have the chance to again for a week or two, so please excuse my delays (in advance) for not getting back to you on anything that would actually require me looking at code.
~John |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Wed Dec 07, 2005 9:16 pm Post subject: |
|
|
| That's good, since you are probably one of those SocketConduit users. I would also like to deprecate the no-arg ctor, and add an explicit one |
|
| Back to top |
|
|
teqdruid
Joined: 11 May 2004
Posts: 390
Location: UMD
|
| Posted: Wed Dec 07, 2005 10:27 pm Post subject: |
|
|
| kris wrote: | | That's good, since you are probably one of those SocketConduit users. I would also like to deprecate the no-arg ctor, and add an explicit one |
Actually, I don't think I've ever used SocketConduit. The only network communitcation I recall doing through Mango is HTTP, so I've just made extensive use of the HTTP library.
~John |
|
| Back to top |
|
|
kris
Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific
|
| Posted: Wed Dec 07, 2005 11:33 pm Post subject: |
|
|
| Bleah ... had to add three-way logic to Buffer, for supporting Text, Binary and Mixed. The compiler has bugs related to Typedef and overload matching, so that curtailed activity here somewhat. Still, the changes are in SVN if you're interested in kicking the tyres. |
|
| Back to top |
|
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
