Download Reference Manual
The Developer's Library for D
About Wiki Forums Source Search Contact

Ticket #1997 (closed enhancement: fixed)

Opened 8 years ago

Last modified 8 years ago

Integrate CDGC (D Concurrent Garbage Collector)

Reported by: llucax Assigned to: community
Priority: major Milestone: 1.0
Component: Runtime Version: 0.99.9 Kai
Keywords: gc, cdgc Cc: llucax

Description

I would like to know if there is any interest in integrating CDGC to Tango.

See http://llucax.com.ar/blog/blog/tag/dgc for details about the collector.

Attachments

0002-bob-Add-an-option-to-select-the-GC-implementation.patch (3.0 kB) - added by llucax on 10/21/10 00:19:30.
Add an option to bob to select the GC implementation to include in the runtime
0001-Add-the-D-Concurrent-Garbage-Collector-CDGC.patch (126.9 kB) - added by llucax on 10/21/10 00:52:50.
Add the D Concurrent Garbage Collector (CDGC)
diff (3.7 kB) - added by llucax on 01/27/11 01:02:56.
Differences between the updated patch and the old one
0001-Add-PointerMap-support-if-the-compiler-supports-it.patch (69.3 kB) - added by llucax on 01/27/11 01:04:42.
Re-upload, I think nothing is changed in this patch, but just in case
0002-Add-the-D-Concurrent-Garbage-Collector-CDGC.patch (127.9 kB) - added by llucax on 01/27/11 01:05:52.
Updated patch with gc_monitor() implemented and gc_term() collection bug fixed

Change History

10/08/10 08:11:32 changed by tomni

Wow thats kinda interesting...let's give it a go... :-)

10/08/10 08:39:58 changed by Marenz

absolutely! Waiting for that for quite some time :)

--Marenz

10/10/10 19:32:34 changed by llucax

While I get some time to work on this, you might try the collector using this simple howto:

http://llucax.com.ar/blog/blog/post/-4d1fefb4

10/11/10 18:53:00 changed by mwarning

I'm all for it. But dmd would need to integrate the patch.

10/12/10 02:43:40 changed by llucax

I'll try to make the precise scanning a compile-time option in the meanwhile, so it can be integrated. I'll have to take a look at the issues with weakref too.

10/20/10 16:01:44 changed by llucax

Attached a preliminary patch to enable the runtime to pass pointer information to the GC (if available). This is mainly for testing. Testing using different platforms and compilers is appreciated.

10/21/10 00:19:30 changed by llucax

  • attachment 0002-bob-Add-an-option-to-select-the-GC-implementation.patch added.

Add an option to bob to select the GC implementation to include in the runtime

10/21/10 00:22:49 changed by llucax

I've updated the previous patch (it had a stupid error in the LDC runtime which triggered a compile error).

Also, I've attached a patch to add an option to bob to select which GC implementation to include in the runtime. I think it will be best to keep using the basic implementation for a while, but allowing users to easily choose the CDGC implementation when it's merged. If bob patch is included, I guess the binaries should be updated too.

Next step is to finally integrate the CDGC.

10/21/10 00:52:50 changed by llucax

  • attachment 0001-Add-the-D-Concurrent-Garbage-Collector-CDGC.patch added.

Add the D Concurrent Garbage Collector (CDGC)

10/21/10 00:56:00 changed by llucax

There is the first try, I did not testing at all, but since Marenz is so eager to try it out, it seemed like a good idea to publish it sooner than later ;)

Please let me know of failure or success stories, as a comment here or contacting me directly to llucax at gmail.

And BTW, if you want to see more details about each patch, take a look at the raw versions, as they have the commit message which is often fairly descriptive.

10/21/10 01:00:03 changed by llucax

I forgot to mention that you can try building tango with CDGC as the collector by using bob. The steps should be something like:

1. Apply the 3 patches in order (using patch -p1 < the-patch in the tango top-level dir):

  1. 0001-Add-PointerMap?-support-if-the-compiler-supports-it.patch
  2. 0002-bob-Add-an-option-to-select-the-GC-implementation.patch
  3. 0001-Add-the-D-Concurrent-Garbage-Collector-CDGC.patch

2. Rebuild bob (for example: ldc -inline -O -release build/src/bob.d) 3. Use the rebuilt bob to build tango, adding the option -g=cdgc 4. Tell me what went wrong ;)

Testing in other platforms than Linux/dmd is specially appreciated, as never been tested with anything else.

10/21/10 01:05:30 changed by llucax

Damn! Trac has messed up my numbered lists :S

  1. Apply the 3 patches in order (using patch -p1 < the-patch in the tango top-level dir):
    1. 0001-Add-PointerMap??-support-if-the-compiler-supports-it.patch
    2. 0002-bob-Add-an-option-to-select-the-GC-implementation.patch
    3. 0001-Add-the-D-Concurrent-Garbage-Collector-CDGC.patch
  2. Rebuild bob (for example: ldc -inline -O -release build/src/bob.d)
  3. Use the rebuilt bob to build tango, adding the option -g=cdgc
  4. Tell me what went wrong ;)

10/22/10 16:41:32 changed by mwarning

Applied 0002-bob-Add-an-option-to-select-the-GC-implementation.patch in [5574]. I will upload binaries when the other patches are applied.

11/06/10 00:24:38 changed by llucax

I was only able to test this patches with DMD (1.062 unpatched, and 1.063 patched to provide pointermap info). When compiling Tango with LDC(1) I hit an LLVM assertion(2), and I can compile Tango with GDC(3) but I get a SIGSEGV in collect2 when trying to compile a hello-world(4).

(1) LDC based on DMD v1.056 and llvm 2.6 (Fri Feb 26 11:01:51 2010) (Debian testing package) (3) GDC 1:1.060-4.3.5-2) 4.3.5 (Debian testing package also)

(2)

ldc -c -I./tango/core -I./tango/core/rt/compiler/ldc -I. -I./tango/core/vendor \
    -O3 -release -gc -oftango-text-Regex-O3--release--gc.o ./tango/text/Regex.d 
object.Exception: Process was killed with signal 6
ldc: /home/arthur/llvm/llvm-2.6/lib/VMCore/Instructions.cpp:921: \
    void llvm::StoreInst::AssertOK(): Assertion `getOperand(0)->getType() == \
    cast<PointerType>(getOperand(1)->getType())->getElementType() && \
    "Ptr must be a pointer to Val type!"' failed.
0   ldc             0x0000000000db269f
1   ldc             0x0000000000db2ead
2   libpthread.so.0 0x00007fb0369adf60
3   libc.so.6       0x00007fb0358ab165 gsignal + 53
4   libc.so.6       0x00007fb0358adf70 abort + 384
5   libc.so.6       0x00007fb0358a42b1 __assert_fail + 241
6   ldc             0x0000000000d24911 llvm::StoreInst::AssertOK() + 145
7   ldc             0x000000000065f160 DtoStore(llvm::Value*, llvm::Value*) + 112
8   ldc             0x0000000000641960 DtoCatAssignElement(Type*, DValue*, Expression*) + 496
9   ldc             0x0000000000603c0f CatAssignExp::toElem(IRState*) + 239
10  ldc             0x00000000005eb9fc ExpStatement::toIR(IRState*) + 108
11  ldc             0x00000000005ee8f3 IfStatement::toIR(IRState*) + 675
12  ldc             0x00000000005ebb79 CompoundStatement::toIR(IRState*) + 89
13  ldc             0x00000000005eceb7 ForStatement::toIR(IRState*) + 983
14  ldc             0x00000000005ebb79 CompoundStatement::toIR(IRState*) + 89
15  ldc             0x00000000005eceb7 ForStatement::toIR(IRState*) + 983
16  ldc             0x00000000005ebb79 CompoundStatement::toIR(IRState*) + 89

(4)

$ LANG=C gdmd -defaultlib=tango -debuglib=tango -release -O hello.d -L-L. -L-lpthread -L-ldl
WARNING: no atomic operations on this architecture
WARNING: this is *slow* you probably want to change this!
collect2: ld terminated with signal 11 [Segmentation fault]
./libtango.a(tango-core-rt-compiler-gdc-rt-lifetime-O3--frelease--g.o): \
    In function `_d_arraycatnT':
/home/luca/tesis/tango/./tango/core/rt/compiler/gdc/rt/lifetime.d:990: \
    undefined reference to `_D3gcc8builtins13__va_list_tag6__initZ'
/home/luca/tesis/tango/./tango/core/rt/compiler/gdc/rt/lifetime.d:990: \
    undefined reference to `_D3gcc8builtins13__va_list_tag6__initZ'
/home/luca/tesis/tango/./tango/core/rt/compiler/gdc/rt/lifetime.d:990: \
    undefined reference to `_D3gcc8builtins13__va_list_tag6__initZ'

I also get the atomics warning when compiling Tango (several times), and a couple of some other errors (but the compilation is not interrupted, using bob):

gdc -c -I./tango/core -I. -O3 -frelease -g \
    -otango-net-device-Berkeley-O3--frelease--g.o ./tango/net/device/Berkeley.d 
./tango/net/device/Berkeley.d:2223: Error: function std.intrinsic.bts (uint*,uint) \
    does not match parameter types (ulong*,ulong)
./tango/net/device/Berkeley.d:2223: Error: cannot implicitly convert expression \
    (cast(ulong*)&(this.first()[cast(ulong)this.fdelt(s)])) of type ulong* to uint*
./tango/net/device/Berkeley.d:2261: Error: function std.intrinsic.btr (uint*,uint) \
    does not match parameter types (ulong*,ulong)
./tango/net/device/Berkeley.d:2261: Error: cannot implicitly convert expression \
    (cast(ulong*)&(this.first()[cast(ulong)this.fdelt(s)])) of type ulong* to uint*
gdc -c -I./tango/core -I. -O3 -frelease -g -otango-net-device-Socket-O3--frelease--g.o \
    ./tango/net/device/Socket.d 

gdc -c -I./tango/core -I. -O3 -frelease -g -otango-core-BitArray-O3--frelease--g.o \
    ./tango/core/BitArray.d 
./tango/core/BitArray.d:590: Error: function std.intrinsic.bt (uint*,uint) does not \
    match parameter types (ulong*,ulong)
./tango/core/BitArray.d:590: Error: cannot implicitly convert expression \
    (cast(ulong*)this.ptr) of type ulong* to uint*
./tango/core/BitArray.d:930: Error: function std.intrinsic.bts (uint*,uint) does not \
    match parameter types (ulong*,ulong)
./tango/core/BitArray.d:930: Error: cannot implicitly convert expression \
    (cast(ulong*)this.ptr) of type ulong* to uint*
./tango/core/BitArray.d:932: Error: function std.intrinsic.btr (uint*,uint) does not \
    match parameter types (ulong*,ulong)
./tango/core/BitArray.d:932: Error: cannot implicitly convert expression \
    (cast(ulong*)this.ptr) of type ulong* to uint*

PS: Long lines were manually broken to avoid making the bug report completely unreadable (broken lines end with "\").

(follow-up: ↓ 14 ) 11/06/10 01:05:28 changed by ibuclaw

Your experience with GDC is known, it's a problem with the version in Debian. All will be fixed again (in Debian unstable, at least) when I push the next round of package updates.

As for testing on GDC; using the trunk. I've been getting no problems whatsoever running applications in 32bit. I will be sure to test out 64bit runtime tomorrow morning.

Regards

(in reply to: ↑ 13 ) 11/06/10 04:52:37 changed by llucax

Replying to ibuclaw:

Your experience with GDC is known, it's a problem with the version in Debian. All will be fixed again (in Debian unstable, at least) when I push the next round of package updates. As for testing on GDC; using the trunk. I've been getting no problems whatsoever running applications in 32bit. I will be sure to test out 64bit runtime tomorrow morning.

Thanks a lot for the testing! I'll wait for the new Debian packages 8-)

(follow-up: ↓ 16 ) 11/15/10 04:21:29 changed by winterwar

Seems to work for me, but I had to add a stub for the missing gc_monitor:

+alias extern(D) void delegate() ddel;
+alias extern(D) void delegate(int, int) dint;
+
+void gc_monitor(ddel begin, dint end )
+{
+    /+ unsupported +/
+}
+

(in reply to: ↑ 15 ) 12/01/10 00:08:44 changed by llucax

Replying to winterwar:

Seems to work for me, but I had to add a stub for the missing gc_monitor: {{{ +alias extern(D) void delegate() ddel; +alias extern(D) void delegate(int, int) dint; + +void gc_monitor(ddel begin, dint end ) +{ + /+ unsupported +/ +} + }}}

I think gc_monitor() could be implemented the same way it's implemented in the basic collector. I'll try to give it a shot and to weakrefs too (making all gc_weak*() functions block until the collection is done) when I have the time.

Thanks for the testing!

01/24/11 20:49:08 changed by mwarning

Ok, we have green light. But I need some free time to make sure we can easily undo the change in case it becomes unmaintained, buggy and a major burden. I don't mean any offense, but to have a back-out strategy seems to be a good idea.

01/27/11 01:01:39 changed by llucax

Well, I came back to post an updated patch and bump into the last comment. I'm glad the patch has green light. I'll upload a new patch with the problem reported by winterwar (gc_monitor) fixed (and by "fixed" I mean implemented :).

Seems perfectly reasonable to have a back-out strategy. I still think CDGC should not be the default GC at first. Mostly because it still has a "major" bug, the problem with weak references. I want to fix that (by making weak references block until the concurrent mark phase is done, not ideal but not worse that it is with the basic GC either), but it can be fixed once the patch is folded in too (it might even be good to encourage other people to fix it, or improve the GC in other aspects).

Anyway, I'm very glad this work doesn't get lost in this bug tracker :)

01/27/11 01:02:56 changed by llucax

  • attachment diff added.

Differences between the updated patch and the old one

01/27/11 01:04:42 changed by llucax

  • attachment 0001-Add-PointerMap-support-if-the-compiler-supports-it.patch added.

Re-upload, I think nothing is changed in this patch, but just in case

01/27/11 01:05:52 changed by llucax

  • attachment 0002-Add-the-D-Concurrent-Garbage-Collector-CDGC.patch added.

Updated patch with gc_monitor() implemented and gc_term() collection bug fixed

(follow-up: ↓ 20 ) 01/27/11 09:41:22 changed by larsivi

A question that popped up yesterday; what is the state of the GC for non-linux platforms.

(in reply to: ↑ 19 ; follow-up: ↓ 21 ) 01/27/11 14:36:19 changed by llucax

Replying to larsivi:

A question that popped up yesterday; what is the state of the GC for non-linux platforms.

Untested. It should work without forking (i.e. no concurrent marking :( ).

(in reply to: ↑ 20 ) 01/27/11 14:37:25 changed by llucax

Replying to llucax:

Replying to larsivi:

A question that popped up yesterday; what is the state of the GC for non-linux platforms.

Untested. It should work without forking (i.e. no concurrent marking :( ).

I mean for windows, for other POSIX platforms it should work concurrently as in Linux, but is untested too.

(follow-up: ↓ 23 ) 01/27/11 19:41:04 changed by kris

Let's get this integrated. Tango supports alternate GC implementations at link-time, so that's perhaps the best way to expose this initially? We could make it the default-GC once people get some further experience with it?

Really nice work, llucax :)

(in reply to: ↑ 22 ) 01/28/11 14:53:37 changed by llucax

Replying to kris:

Let's get this integrated. Tango supports alternate GC implementations at link-time, so that's perhaps the best way to expose this initially? We could make it the default-GC once people get some further experience with it?

I agree, I already did a patch to add an option to bob to select what collector to use (already integrated), so people can easily try CDGC when compiling Tango.

Really nice work, llucax :)

Thanks ^_^

01/28/11 16:40:07 changed by llucax

Please, remember to rebuild bob after the patch is applied, so people can try the new GC without having to recompile bob themselves.

01/28/11 22:05:01 changed by mwarning

(In [5609]) refs #1997 :: Apply patch to support for pointer maps if compiler supports it; thanks&cake for llucax

01/28/11 22:08:36 changed by mwarning

(In [5610]) refs #1997 :: Apply patch for CDGC; thanks&crumpets for llucax

01/28/11 22:10:51 changed by mwarning

  • status changed from new to closed.
  • resolution set to fixed.

(In [5611]) fixes #1997 :: Missing files for CDGC

01/28/11 22:21:07 changed by mwarning

One minor problem appeared. dmdfe 1.064 (or before) has a bug that is fixed in 1.065. ldc is also affected since it used dmdfe 1.064.

tango/tango/core/rt/gc/cdgc/gc.d(2621): Error: cannot implicitly convert expression (begin) of type void delegate() to void delegate()
tango/tango/core/rt/gc/cdgc/gc.d(2622): Error: cannot implicitly convert expression (end) of type void delegate(int, int) to void delegate(int freed, int pagebytes)

I've added a workaround with casts.

01/28/11 22:43:03 changed by llucax

I had a similar error when I didn't defined the private alias for the delegates. Doing that fixed the error for me in 1.062 at least.

Thanks for committing this!

01/29/11 14:33:44 changed by mwarning

(In [5614]) refs #1997 :: new bob binary for win32