FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Bug in mango.io.utf?

 
Post new topic   Reply to topic     Forum Index -> Mango
View previous topic :: View next topic  
Author Message
uwe



Joined: 05 Apr 2005
Posts: 34
Location: Stuttgart, Germany

PostPosted: Fri May 20, 2005 7:03 am    Post subject: Bug in mango.io.utf? Reply with quote

Hi Kris,

during my work on the UTF converters i discovered a bug in your converter (basically because i copied it and modified then :) ). In the UTF-16 to UTF-8 converter you have to replace 0x8000 by 0x0800 (that value is the first to be encoded in 3 bytes).

Ciao
uwe
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Fri May 20, 2005 10:07 am    Post subject: Re: Bug in mango.io.utf? Reply with quote

uwe wrote:
Hi Kris,

during my work on the UTF converters i discovered a bug in your converter (basically because i copied it and modified then Smile ). In the UTF-16 to UTF-8 converter you have to replace 0x8000 by 0x0800 (that value is the first to be encoded in 3 bytes).

Ciao
uwe

Thanks, uwe; I'll check it out.

I noticed a couple in your versions also, though you may have fixed 'em by now. From memory:

~ in String.toUtf8, the realloc does not do a complete job. The base pointer was not updated, I recall.

~ in the newer conversion module, there's a similar conflict going on with the starting address of the local pointer; sometimes it is zero-based and sometimes it is offset by the 'index' argument passed in (second arg). The inconsistency will break in a boundary condition.

Sorry this is short on detail ~ I don't have the code in front of me.

If you'd give your code a non-viral license, then I'd use it in Mango (it is kind of annoying when people take open-source code and produce a modified version with a more restrictive license -- it destroys the notion of rolling changes back into the original code, which surely negates the spirit of open source). In fact, what you appear to be doing with your String class and ICU is strikingly similar to what mango.icu provides.

What do you think about combining efforts instead? I mean, Mango is about to adopt a Container package, and perhaps your design is a better approach than, say, MinTL?
Back to top
View user's profile Send private message
teqdruid



Joined: 11 May 2004
Posts: 390
Location: UMD

PostPosted: Fri May 20, 2005 11:48 pm    Post subject: OT: Containers Reply with quote

Quote:
Mango is about to adopt a Container package


Wha??? I didn't hear a word about this... Given I've been a bit busy lately, but when is this happening? This is great news!

I've been waiting for Ben to finish his interface-type stuff for MinTL, but based on how Mango is currently structured, I'm betting it's built with a good class hierarchy.

So? When's it happening? If not soon, can I (we) have a sneak peak? Wink

Alright... where's the "anxious" emoticon thingy?
Back to top
View user's profile Send private message Send e-mail AIM Address
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Fri May 20, 2005 11:58 pm    Post subject: Reply with quote

Why didn't you say so before? Wink

It's really just been a case of priority, and I've been waiting for Ben too! I want to keep the options open though, and perhaps Indigo has something nice that we can adopt, or merge with?
Back to top
View user's profile Send private message
teqdruid



Joined: 11 May 2004
Posts: 390
Location: UMD

PostPosted: Sat May 21, 2005 12:11 am    Post subject: Reply with quote

I don't really care where it comes from, but I need a good Containers API with a good class hierarchy. If it's in Mango, all the easier for me, plus it'll probably have the same great hierarchy that the rest of Mango has.

I have't looked at either Indigo, or DTL (which I doubt even compiles at this point) but the little I've seen of MinTL doesn't make me think that it would be a good candidate for inclusion in Mango- unless Ben is making some pretty radical changes.

It seemed more like a C-style API to me. Almost no inheritance, and really boils down to a collection of structs and associated functions. I'm not saying this is a bad thing- I'm betting Ben gets a lot of speed out of it, but it just didn't seem in the Mango style... I'll reserve final judgement until I see some of his latest stuff, however.

I'm hoping it won't take too long. I just finished finals today, so I've now got some free time (famous last words) to get back to work on my project.

I'll take a look at Indigo soon.
Back to top
View user's profile Send private message Send e-mail AIM Address
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Sat May 21, 2005 1:45 am    Post subject: Reply with quote

teqdruid wrote:
I'm hoping it won't take too long. I just finished finals today, so I've now got some free time (famous last words) to get back to work on my project.

Congratulations dude!

And welcome back, too.
Back to top
View user's profile Send private message
uwe



Joined: 05 Apr 2005
Posts: 34
Location: Stuttgart, Germany

PostPosted: Sun May 22, 2005 11:53 pm    Post subject: Reply with quote

Sorry i took so long for my answer, i simply forgot to check back :(

I have already discovered the first bug you spoke about. But i am not so sure about the second... The conversion module has passed all unit tests after some debugging, but i'll look into it again, thanks a lot.

I am not sure what you mean with "viral license". Do you mean you can't take something from my code back into your project because you would have to change your license?? Well, no prob, i was not thinking much about licensing issues (all that stuff sometimes annoys me), just taking the "popular" one i know. Can be changed, you can simply take my code, distribute it under your license and write somewhere "uwe helped me" or similar :)

Sure you can use Indigo in Mango. I was dissatisfied about the String class either (that's why i did not write it to perfection), mostly because of the dumb ICU porting that just produces bloat, and was already done by you. I did not want to be dependant on ICU _and_ Mango, that's all. If a string class weren't so damn important for almost any program, i would have abandoned it.

Ciao
uwe
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Mon May 23, 2005 1:05 am    Post subject: Reply with quote

uwe wrote:
I am not sure what you mean with "viral license".

Oh, that's a general term for a GPL-type license. As you noted, adoption of the code tends to require adoption of the license. Thanks for clarifying!

uwe wrote:
I did not want to be dependant on ICU _and_ Mango, that's all. If a string class weren't so damn important for almost any program, i would have abandoned it.

Yeah; I know what you mean, though it would be beneficial to users if the two string classes were equivalent at the interface level. BTW, mango.icu is completely standalone when you compile it with
Code:
-version=Isolated
as is the mango.log package.

On a different note ~ if you're going to continue expanding the Indigo container facilities, are you making them thread-safe? If so, I'd be interested in them. Also, you may find mango.cache.HashMap useful for Indigo, since it's a port of Doug Lea's non-blocking thread-safe hashmap. Prof Lea writes very high quality code, and is a prolific writer of invaluable thread-oriented utilities. Ben Hinkle's Locks lib is a straight port of Doug's Java library ~ effectively the one that now ships with Java 1.5
Back to top
View user's profile Send private message
uwe



Joined: 05 Apr 2005
Posts: 34
Location: Stuttgart, Germany

PostPosted: Mon May 23, 2005 4:02 am    Post subject: Reply with quote

As i said, if you need something of the Indigo code (or the utf conversion package), just take it. No need to include the license as well. Something like "other authors: uwe" would be nice. I'm always happy if i can help.

I have already looked at your hashmap, and will perhaps use it as an implementation draft. I do not plan to make Indigo thread-safe in the near future, as the Qt containers are not, too (ok, they are somehow thread-safe, but you cannot access the same container from two threads, just copy it thread-safely). But i will look over Doug Lea's code, thanks for the hint.

By the way, did you need thread-safe containers often in your projects? I never needed one yet, and i program for quite a long time (ok, did not write so much threaded code either). Just once i had a very large and complex data structure that had to be thread-safe, but that was really a more complicated issue than solvable with a thread-safe vector/list/whatever.

Ciao
uwe
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Mon May 23, 2005 11:32 am    Post subject: Reply with quote

uwe wrote:
As i said, if you need something of the Indigo code (or the utf conversion package), just take it. No need to include the license as well. Something like "other authors: uwe" would be nice. I'm always happy if i can help.

I have already looked at your hashmap, and will perhaps use it as an implementation draft. I do not plan to make Indigo thread-safe in the near future, as the Qt containers are not, too (ok, they are somehow thread-safe, but you cannot access the same container from two threads, just copy it thread-safely). But i will look over Doug Lea's code, thanks for the hint.

By the way, did you need thread-safe containers often in your projects? I never needed one yet, and i program for quite a long time (ok, did not write so much threaded code either). Just once i had a very large and complex data structure that had to be thread-safe, but that was really a more complicated issue than solvable with a thread-safe vector/list/whatever.

The mango.http.server and mango.cluster packages are quite heavily mulit-threaded, and that's where the need for an efficient thread-aware container came me up (I like to avoid synch if possible; I'm sure you can sympathize). Those packages make heavy use of thread-locals to avoid shared resources for the most part.

That aside; the industry has been heading towards multi-core devices for a while now, and I suspect it'll continue that way for at least the next decade or two. I'm sure you're aware of commercial designs like Cell, and Niagara (the latter has 32 hardware-supported threads). I mentioned a long time ago that it'd be nice for D to investigate some of the Occam constructs for parrallel programming ~ things like concurrent loops and so on; Threads are good for now though Smile

BTW; some desktop-style packages are making the transition by building a producer/consumer aspect into the software ~ that way they can pipeline heavy processing across two threads, and take advantage of extra hardware without the added complexity of a fully threaded design. Makes a lot of sense.

Debugging multi-threaded software is still the nightmare it always was; but it's made a lot easier by having pre-tested thread-aware containers, and by the existence of libraries such as Doug Lea's utilities.
Back to top
View user's profile Send private message
uwe



Joined: 05 Apr 2005
Posts: 34
Location: Stuttgart, Germany

PostPosted: Mon May 23, 2005 1:55 pm    Post subject: Reply with quote

I looked at Doug Lea's stuff, especially at the containers. But there were no concurrent lists/arrays, just copy-on-write versions, and versions wrapped in synchronizations (which can be easily realized in D by synchronized{} statements where necessary). Thus the Vector and List/LinkedList can stay as they are, String anyways. As the Map is a linked list too, it cannot be made thread-safe as well.

But i will try to do the trick with the Hash, if it does not impact on performance and simplicity too much. As you said, Ben Hinkle ported the stuff already. Much as i would be pleased to see Indigo in Mango, the aim for Indigo is to provide simple but high-performance containers for people that want to write productivity apps in D without understanding the difficult design of yet another template library.

Ciao
uwe
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Mon May 23, 2005 2:24 pm    Post subject: Reply with quote

uwe wrote:
But i will try to do the trick with the Hash, if it does not impact on performance and simplicity too much. As you said, Ben Hinkle ported the stuff already. Much as i would be pleased to see Indigo in Mango, the aim for Indigo is to provide simple but high-performance containers for people that want to write productivity apps in D without understanding the difficult design of yet another template library.

Just to be clear, Ben has a seperate library called Locks (outside of MinTL). It is class based, since it's a straight port of Lea's synchronization lib.

And I certainly agree regarding the difficulties inherent in templated designs. They're not for the faint-of-heart, and not easy to pick up. Plus, they can be truly hideous to debug once deployed (just like enormous macros; the debugger gets very confused). It's unlikely anyone will see much of that in Mango, either.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Mango All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group