FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Ares updates
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9  Next
 
Post new topic   Reply to topic     Forum Index -> Ares
View previous topic :: View next topic  
Author Message
sean



Joined: 24 Jun 2004
Posts: 609
Location: Bay Area, CA

PostPosted: Mon Jan 23, 2006 6:59 am    Post subject: Reply with quote

Versio 0.10 is online with these changes:

* Merged in DMD .144 changes.
* Renamed std.gc to std.memory.
* Redefined GC library interaction. Some code that was in dmdgc now lives in dmdrt within the new file, memory.d.
* Defined new standard interface to the GC, in std.memory. All calls now go through an instance of the GC struct: gc. This dummy class is present to allow for future interface changes that may allow dynamic swapping of GCs.
* Removed an unnecessary 'static' from std.exception.

Please let me know if you don't like the GC interface or if it could be improved, as I'm still not quite happy with the design, but it's the best I've come up with so far given the design contraints. Also, I am planning to add a 'comments' section to each module header that outlines some of the thought that went into the design that's been chosen, what to look at, etc. I'd planned to get the comments into this release until Walter went and released DMD .144 and I decided I'd just get an update out the door instead Smile The next issue I intend to address is IO, and I'm talking with Kris about using a subset of Mango for that purpose. So stay tuned. Afterwords, std.regexp needs some attention as it was imported nearly as-is from Phobos and I haven't taken a very close look at the inteface. This should be the last of the existing modules that has been completely ignored to date.

By the way... the GC changes for this release vastly simplify the integration of a new GC into a D program. Now, just implement all the extern (C) functions mentioned in std.memory and relink ares.lib, dmdrt.lib and yourGC.lib into a file named phobos.lib and you're set (currently, dmdrt.lib, ares.lib, and dmdgc.lib are what gets linked into phobos.lib, so you're basically just replacing dmdgc.lib). I'll document all this in the next release, but don't have time to at the moment and wanted to get the release up today.
Back to top
View user's profile Send private message
sean



Joined: 24 Jun 2004
Posts: 609
Location: Bay Area, CA

PostPosted: Mon Jan 23, 2006 12:56 pm    Post subject: Reply with quote

I just posted a minor update to 0.10 -- it fixes gc.add and gc.remove in std.memory.
Back to top
View user's profile Send private message
sean



Joined: 24 Jun 2004
Posts: 609
Location: Bay Area, CA

PostPosted: Tue Jan 24, 2006 1:49 pm    Post subject: Reply with quote

Looks like the last few DMD updates fixed a bunch of bugs that were preventing std.atomic from working properly, so I've made all relevant changes to this module in SVN.
Back to top
View user's profile Send private message
sean



Joined: 24 Jun 2004
Posts: 609
Location: Bay Area, CA

PostPosted: Wed Jan 25, 2006 1:08 pm    Post subject: Reply with quote

Version 0.11 is online with these changes:

* std.atomic - Uncommented all unit tests and removed bug workaround code, as the problems have been addressed. Also added module comments.
* Added std.memory and std.stdarg to documentation.
* Added TypeInfo declarations to object.d, as they are now required by DMD.
* Added documentation for std.memory.
* std.memory - Added the sizeOf function.
* Added the object file for std.memory to the build (it was accidentally omitted).

This is largely a bugfix release to address some problems I found in 0.10. Expect some new features in 0.12.
Back to top
View user's profile Send private message
sean



Joined: 24 Jun 2004
Posts: 609
Location: Bay Area, CA

PostPosted: Wed Feb 01, 2006 5:22 pm    Post subject: Reply with quote

Version 0.12 is online with these changes:

* Merged in DMD .145 changes.
* Fixed DMD runtime to use new GC functionality where appropriate.
* Within GC implementation, renamed capacityOf to sizeOf to match function name to actual behavior.
* std.memory - Fixed documentation to include throw specs.
* Fixed compiler flags for std.stard and std.c.stdarg.
* Added module-level documentation for std.intrinsic and std.stdarg.
* Added the ThreadGroup class to std.thread.
* Changed the Ares makefile to be a bit more generic. Now, the only explicitly handled files are those that need explicit handling.
* Added comments to all files that have been modified from their original form as shipped with DMD. While adding these comments to the TypeInfo modules, the standard Digital Mars copyright header was added if missing as well. This was done for consistency, as some TypeInfo modules contained this header while others did not (so I assumed its presence was intended in all such modules).
* Ares now has IO functionality! With Kris' permission, a minimal working subset of Mango has beed added to Ares in the new std.io package area. The primary goal for this release was to get the basics in and working, so expect more integration work or more features to be added in a future release.
* Ares now does math! The first of a collection of mathematics modules have been added to the new std.math package area, following a model suggested by Don Clugston. Please note that the mathematic functions mentioned in the documentation for std.intrinsic in Phobos currently do not work as a result of compiler and library issues in DMD. This will be addressed in the next Ares release.

Much thanks to all those involved in contributing code and suggestions. As always, feedback is very much appreciated. I'm starting to feel like Ares is almost usable now, and am pretty excited about how things might shape up in the coming months Smile
Back to top
View user's profile Send private message
kylefurlong



Joined: 16 Nov 2005
Posts: 29

PostPosted: Wed Feb 01, 2006 6:54 pm    Post subject: Reply with quote

Very exciting Sean! Keep on truckin! Very Happy
Back to top
View user's profile Send private message
sean



Joined: 24 Jun 2004
Posts: 609
Location: Bay Area, CA

PostPosted: Thu Feb 16, 2006 5:56 pm    Post subject: Reply with quote

Versio 0.13 is online with these changes:

* Removed std.regexp as it is somewhat redundant with DMD 147 regexp support.
* Added regexp support code as required by DMD 147.
* Moved all Phobos std modules needed by DMD runtime code into new package dmdrt.util.
* Modified dmdrt and dmdrt.typeinfo modules to use util functions where appropriate (some modules previously contained inline code as a hack).
* Refactored code in std.thread to eliminate code duplication.
* Fixed inline asm in std.math.core as per an update in DMD 146.
* Added std.math.ieee.fcis function. Please note that there may be offset issues on non-Windows platforms.
* Fixed Ares makefile macro errors and cleaned up file format a bit more.
* Altered dmdgc and dmdrt makefiles to match Ares "standard" makefile format.
* Added trace.obj to dmdrt build. Note that this object is not necessary for a successful DMD runtime.
* Added std.c.config for internal data size configuration. Note that this file is for internal use and should never be imported publicly.
* Changed std.c files to use aliases defined in std.c.config where appropriate.
* Beautified std.c declarations format for readability and fixed a few declaration errors.

You'll also notice a new branch in the SVN. This was created before any DMD 147 changes were made in case something happens with the new regex support.
Back to top
View user's profile Send private message
sean



Joined: 24 Jun 2004
Posts: 609
Location: Bay Area, CA

PostPosted: Sat Feb 25, 2006 4:49 pm    Post subject: Reply with quote

Version 0.14 is online with these changes:

* Made substantial changes to std.c.posix modules for Linux and Darwin compatibility.
* Changed Linux makefiles to match Win32 makefiles.
* Added std.c.posix.sys.mman.
* Trimmed out some now redundant declarations from sys.linux.c.linux.
* Added atomicIncrement and atomicDecrement for use with value and pointer types.
* Made miscellanious improvements in std.thread to support Posix threads.
* Added Windows and Linux constants to std.c.errno for error reporting.

Ares should now build using DMD on Linux. Please let me know if there are any problems.
Back to top
View user's profile Send private message
sean



Joined: 24 Jun 2004
Posts: 609
Location: Bay Area, CA

PostPosted: Sun Feb 26, 2006 6:33 pm    Post subject: Reply with quote

I should have waited one more day Wink

Version 0.15 is online with these changes:

* Merged in DMD 148 changes.
* Renamed some TypeInfo files and split a few into per-class modules.
* Removed util.regexp and util.outbuffer from the build, as they are no longer needed. std.regexp will reappear in the next release.

Along with std.regexp, std.bitarray and a subset of std.string may appear as well. And as usual, all feedback is very welcome Smile
Back to top
View user's profile Send private message
sean



Joined: 24 Jun 2004
Posts: 609
Location: Bay Area, CA

PostPosted: Mon Feb 27, 2006 2:17 pm    Post subject: Reply with quote

kris wrote:
Hi Sean,

Can you add 'daemon' support to the Thread class and runtime support? This would cause the runtime to exit at end-of-main only if all threads are daemons; any non-daemon threads would keep the process alive (via a sleep(forever) call, or something).

For purposes of compatability, it might be useful to assume all threads are daemons by default, and a setDaemon(false) would need to be requested first?

What do you think?

I decided to hold off on building anything explicit into Ares to support this largely because I couldn't think of a clean way to do it. However, this can be modeled very efficiently with what's currently available:
Code:
import std.thread;

ThreadGroup daemons = new ThreadGroup();

void main()
{
    on_scope_exit daemons.joinAll();

    void daemonFn() { Thread.sleep( 30000 ); }
    daemons.create( &daemonFn );
}
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Tue Mar 07, 2006 1:58 am    Post subject: Reply with quote

sean wrote:
Version 0.14 is online with these changes:

* Made substantial changes to std.c.posix modules for Linux and Darwin compatibility.
* Changed Linux makefiles to match Win32 makefiles.
* Added std.c.posix.sys.mman.
* Trimmed out some now redundant declarations from sys.linux.c.linux.
* Added atomicIncrement and atomicDecrement for use with value and pointer types.
* Made miscellanious improvements in std.thread to support Posix threads.
* Added Windows and Linux constants to std.c.errno for error reporting.

Ares should now build using DMD on Linux. Please let me know if there are any problems.

Hi Sean;

I've tried using recent versions of Ares (0.14) with multiple threads on Win32, and it dies on a GC collection. Kind of odd, since it's just fine with the main thread + one more created one ~ but when I create an additional one, the GC fails right in the same spot every time. I removed all additional logic around the created threads, so they just spin doing nothing in particular.

Any ideas?
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Tue Mar 07, 2006 3:01 am    Post subject: Reply with quote

Here's an example:
Code:

import gc;
import std.thread;

extern (Windows) void Sleep (int millisecs);

void foo()
{
        static int count;
        auto me = ++count;

        while (true)
              {
              printf ("thread ?d\n", me);
              Sleep (1000);               
              }
}


void main ()
{
           for (auto i=10; i--;)
               {
               auto thread = new Thread (&foo);
               thread.start();
               }

           while (true)
                 {
                 printf ("collecting ... ");

                 _gc.fullCollect();

                 printf ("done\n");
                 Sleep (3000);
                 }
}


The above should do one of several things: (1) appear to run, but only some of the threads are actually live ~ there should be thread id's from 1 to 10 inclusive; (2) appear to run, but the "done" message never shows up because the GC is blocked; (3) stall completely during collection, where all threads are blocked; (4) throw an access-violation

Try compiling with variations of -g, -O, -release.

The issue appears to be a deadlock scenario between the GC and something else ~ you can sometimes see this when catching just the right conditions with MSVC
Back to top
View user's profile Send private message
sean



Joined: 24 Jun 2004
Posts: 609
Location: Bay Area, CA

PostPosted: Tue Mar 07, 2006 11:02 am    Post subject: Reply with quote

Gah. I haven't seen a deadlock yet, but that may be because I'm getting an access violation first Confused I'll give it a look today.
Back to top
View user's profile Send private message
sean



Joined: 24 Jun 2004
Posts: 609
Location: Bay Area, CA

PostPosted: Tue Mar 07, 2006 2:37 pm    Post subject: Reply with quote

I found the problem. There's a race condition in the GC/thread code. Basically, since new threads add themselves to the global thread list and because the lock on this list is not held during the full collection run (only for the duration of suspendAllThreads, scanAllThreads, and resumeAllThreads), it's possible for a starting thread to add itself to this global list after suspendAllThreads has been called and before scanAllThreads is called. Adding a new thread to the global list may allocate memory, which will block because the GC thread is busy, so when the GC thread calls scanAllThreads it will deadlock with the new thread that's waiting to allocate.

I've given it some thought and the only reasonable alternative I can think of is to copy Phobos and add newly created threads to the global list from within Thread.start instead of letting them add themselves in threadMain. The only alternatives I can think of would be to either have the thread list and the GC share a common mutex (which seems unreasonable) or to require that the thread list not allocate any memory, which would restrict it to a fixed size. Can you think of another approach that might be preferable?
Back to top
View user's profile Send private message
kris



Joined: 27 Mar 2004
Posts: 1494
Location: South Pacific

PostPosted: Tue Mar 07, 2006 5:59 pm    Post subject: Reply with quote

Not sure on that one, although having a fixed size thread-list would be OK assuming it were easily configurable somehow ...

Talking about Phobos, it exhibits a similar deadlock ~ just not with that example (it's a lot more complex, unfortunately). However, when Phobos deadlocks, the threads are in what seem to be identical scenarios vis-a-vis the GC: one thread appears to be attempting a bigAlloc() whilst another is trying to GC ~ the killer is that *all* threads end up with a 'disable' count of more than zero: a value of 1 for the two contending threads and a value of 2 for all others. This would seem to indicate that Thread.pauseAll() was actually invoked twice (for those with a value of 2).

Thus the thread-state exposed by the debugger, both in terms of the disable state and the relevant stack traces, look almost identical for Phobos in the complex case as it does with Ares in the simplistic case of the above example.

What does this all mean? Well, it tends to indicate the Phobos design has a hole in it (because it certainly does deadlock in the same manner: that's the reason I tried Ares for this issue). Thus, adopting the Phobos strategy may not be a good idea. In other words, I was vaguely hoping, once you get this example working, that the real application will work on Ares where it ultimately fails on Phobos ... the threading models may be sufficiently different to warrant that (I'm trying to get Valgrind to help catch the Phobos issue for me, but not having much luck).

If you can send me a zip file with support for 25 threads, I'll let you know if it is deadlock free Smile
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Ares All times are GMT - 6 Hours
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9  Next
Page 7 of 9

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group