[Gc] adding profiling callbacks

[Lucas Meijer]

Hi,

Unity is a game development tool that for some platforms uses Boehm to
collect garbage generated by our users .NET code.

Our builtin profiler can profile all the different game subsystems we have,
and I've added support to it for also profiling the boehm collector.

I've added profiling callbacks to our version of boehm, and would like to
ask this list if there's any interest in taking these patches. (the
friendly folks at the mono project did pretty much the same for their fork
of boehm).

Here's the patches we have now. I'd be more than happy to adapt/fix any
concerns they might have.

Bye, Lucas

[Paul Bone]

Post by Lucas Meijer
Hi,
Unity is a game development tool that for some platforms uses Boehm to
collect garbage generated by our users .NET code.
Our builtin profiler can profile all the different game subsystems we have,
and I've added support to it for also profiling the boehm collector.
I've added profiling callbacks to our version of boehm, and would like to
ask this list if there's any interest in taking these patches. (the
friendly folks at the mono project did pretty much the same for their fork
of boehm).

I'd like to add my support for this patch or a similar patch. We (the
Mercury project) have also customised the collector in order to collect
profiling data. It would be great if this kind of thing was supported
upstream.

[Bruce Hoult]

Paul,

Would Lucas's patch fill your needs? If not, what have you done differently?

Post by Paul Bone

Post by Lucas Meijer
Hi,
Unity is a game development tool that for some platforms uses Boehm to
collect garbage generated by our users .NET code.
Our builtin profiler can profile all the different game subsystems we

have,

Post by Lucas Meijer
and I've added support to it for also profiling the boehm collector.
I've added profiling callbacks to our version of boehm, and would like to
ask this list if there's any interest in taking these patches. (the
friendly folks at the mono project did pretty much the same for their

fork

Post by Lucas Meijer
of boehm).

I'd like to add my support for this patch or a similar patch. We (the
Mercury project) have also customised the collector in order to collect
profiling data. It would be great if this kind of thing was supported
upstream.
--
Paul Bone
_______________________________________________
bdwgc mailing list
https://lists.opendylan.org/mailman/listinfo/bdwgc
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

[Paul Bone]

Post by Bruce Hoult
Paul,
Would Lucas's patch fill your needs? If not, what have you done differently?

It looks like it. But I'm also curious about some of the events. Lucas can
you tell me/document what: GC_EVENT_START, GC_EVENT_RECLAIM_START,
GC_EVENT_RECLAIM_END and GC_EVENT_END mean. Reclaim is probably obvious
with more knowledge about how the collector works but I'm afraid I don't
have/don't remember that.

The other thing we use with Mercury is the heap tracing code that Peter Wang
wrote about on this list a few months ago.

[Lucas Meijer]

Hi all,

@Paul: GC_EVENT_RECLAIM_START gets sent when marking is finished, and
sweeping starts.
(implementation not actually in this patch. should the maintainers decide
"yeah we'd take this functionality", I'll produce a complete patch with all
these events implemented).

The ones I assume would generate most discussion are these:

+ GC_EVENT_SUSPENDED_THREAD,
+ GC_EVENT_UNSUSPENDED_THREAD,

which would be implemented in each OS' pthreads_stop_world.c like file,
and which casts whatever is the platform specific thread identifier into a
void* into the callback signature (as the data argument).

(FYI: we use these thread suspension callbacks to be able to visualize in
our profiler which thread is doing the actual GC, but also which threads
were stopped to allow to the gc to run (and therefor also visualize which
threads were not stopped :) ).

If general consensus is "happy", I'll go back and produce mergable patches.
(excuse my newbness around this project: is the preferred method of
contribution a patch file sent to this list, or a github pull request sent
here: https://github.com/ivmai/bdwgc ?)

Bye, Lucas

Post by Bruce Hoult

Post by Bruce Hoult
Paul,
Would Lucas's patch fill your needs? If not, what have you done

differently?
It looks like it. But I'm also curious about some of the events. Lucas can
you tell me/document what: GC_EVENT_START, GC_EVENT_RECLAIM_START,
GC_EVENT_RECLAIM_END and GC_EVENT_END mean. Reclaim is probably obvious
with more knowledge about how the collector works but I'm afraid I don't
have/don't remember that.
The other thing we use with Mercury is the heap tracing code that Peter Wang
wrote about on this list a few months ago.
--
Paul Bone

[Bruce Hoult]

Post by Lucas Meijer
@Paul: GC_EVENT_RECLAIM_START gets sent when marking is finished, and
sweeping starts.

There really isn't a discrete sweeping phase, as such.

When you call GC_malloc() (or friends) and the free list for that object
size is empty then one 4KB block is swept and the unmarked objects in it
added to the free list.

So the sweeping is spread somewhat evenly throughout the execution of your
program.

It would of course make sense to be able to instrument those individual
block sweepings. But it's a lot of small things spread out through each GC
cycle, not one big thing just after marking.

[Lucas Meijer]

Ok, in that case I think it's best if I remove them. Any further comments
from anyone?

Bye, Lucas

Post by Bruce Hoult

Post by Lucas Meijer
@Paul: GC_EVENT_RECLAIM_START gets sent when marking is finished, and
sweeping starts.

There really isn't a discrete sweeping phase, as such.
When you call GC_malloc() (or friends) and the free list for that object
size is empty then one 4KB block is swept and the unmarked objects in it
added to the free list.
So the sweeping is spread somewhat evenly throughout the execution of your
program.
It would of course make sense to be able to instrument those individual
block sweepings. But it's a lot of small things spread out through each GC
cycle, not one big thing just after marking.

[Paul Bone]

Post by Lucas Meijer
Ok, in that case I think it's best if I remove them. Any further comments
from anyone?

Or just discourage people from using them when it might distort their
profile and/or when distortion would be a problem.

[Paul Bone]

Post by Lucas Meijer
Hi all,
@Paul: GC_EVENT_RECLAIM_START gets sent when marking is finished, and
sweeping starts.
(implementation not actually in this patch. should the maintainers decide
"yeah we'd take this functionality", I'll produce a complete patch with all
these events implemented).

Okay cool.

Post by Lucas Meijer
+ GC_EVENT_SUSPENDED_THREAD,
+ GC_EVENT_UNSUSPENDED_THREAD,
which would be implemented in each OS' pthreads_stop_world.c like file,
and which casts whatever is the platform specific thread identifier into a
void* into the callback signature (as the data argument).

In Mercury I implemented something similar, but I determined the identity of
the thread (I wanted to find the thread's logging buffer so I could lock
that it was being suspended) by simply using thread local storage. A thread
identifier will also work / could be useful.

Post by Lucas Meijer
(FYI: we use these thread suspension callbacks to be able to visualize in
our profiler which thread is doing the actual GC, but also which threads
were stopped to allow to the gc to run (and therefor also visualize which
threads were not stopped :) ).

Same, In particular I wanted to calculate the amount of real time spent
running the application vs marking. Also I wanted to measure how long it
took for threads to be suspended and resumed, if it happens often then this
latency could add up.

[Ivan Maidanski]

Hi Lucas,

1. The preferred way is to checkout master branch of bdwgc and submit pull request duplicating details (and link to pull request) to this ML.

2. If I missed something regarding events (suspend/resume/etc.) to review, let me know (submit pull request).

Thank you

Post by Lucas Meijer
Hi all,
@Paul: GC_EVENT_RECLAIM_START gets sent when marking is finished, and sweeping starts.
(implementation not actually in this patch.  should the maintainers decide "yeah we'd take this functionality", I'll produce a complete patch with all these events implemented).
+    GC_EVENT_SUSPENDED_THREAD,
+    GC_EVENT_UNSUSPENDED_THREAD,
which would be implemented in each OS'  pthreads_stop_world.c like file, and which casts whatever is the platform specific thread identifier into a void* into the callback signature (as the data argument).
(FYI: we use these thread suspension callbacks to be able to visualize in our profiler which thread is doing the actual GC, but also which threads were stopped to allow to the gc to run  (and therefor also visualize which threads were not stopped :) ).
If general consensus is "happy", I'll go back and produce mergable patches.  (excuse my newbness around this project: is the preferred method of contribution a patch file sent to this list,  or a github pull request sent here: https://github.com/ivmai/bdwgc  ?)
Bye, Lucas  

Post by Bruce Hoult
Paul,
Would Lucas's patch fill your needs? If not, what have you done differently?

It looks like it.  But I'm also curious about some of the events.  Lucas can
you tell me/document what: GC_EVENT_START, GC_EVENT_RECLAIM_START,
GC_EVENT_RECLAIM_END and GC_EVENT_END mean.  Reclaim is probably obvious
with more knowledge about how the collector works but I'm afraid I don't
have/don't remember that.
The other thing we use with Mercury is the heap tracing code that Peter Wang
wrote about on this list a few months ago.
--
Paul Bone

_______________________________________________
bdwgc mailing list
https://lists.opendylan.org/mailman/listinfo/bdwgc

[Ivan Maidanski]

Hi Lucas,

With some modifications I've integrated profiling callbacks into master (see https://github.com/ivmai/bdwgc/compare/daa6cd9...97f8e10). The style is close to that of Mono GC.

Regards,
Ivan

Hi,
Unity is a game development tool that for some platforms uses Boehm to collect garbage generated by our users .NET code.
Our builtin profiler can profile all the different game subsystems we have, and I've added support to it for also profiling the boehm collector.
I've added profiling callbacks to our version of boehm, and would like to ask this list if there's any interest in taking these patches. (the friendly folks at the mono project did pretty much the same for their fork of boehm).
Here's the patches we have now. I'd be more than happy to adapt/fix any concerns they might have.
Bye, Lucas
_______________________________________________
bdwgc mailing list
https://lists.opendylan.org/mailman/listinfo/bdwgc