[Gc] Difference between version 6.1 and latest which is 6.4

[Emmanuel Stapf [ES]]

I'm upgrading my application so that it uses the more recent version 6.4 but when
I use it, I'm getting very strange memory corruption. Any help for debugging is
appreciated.

Here are some info about my configuration of the GC:
- The platform is Windows XP Pro.
- ALL_INTERIOR_POINTERS is not defined (I'm simply hand editing NT_MAKEFILE to get
rid of its definition)
- I'm calling GC_register_displacement (8)
- I'm simply calling GC_malloc, GC_malloc_atomic, GC_realloc and
GC_register_finalizer

I've enabled the assertions to see if I was getting something before the actual
crash, but nothing.

To summarize, with 6.1 it works great, with 6.4 it does not.

Thanks,
Manu

------------------------------------------------------------------------
Eiffel Software
805-685-1006
http://www.eiffel.com
Customer support: http://support.eiffel.com
Product information: mailto:***@eiffel.com
User group: http://groups.eiffel.com/join
------------------------------------------------------------------------

[Hans Boehm]

I can't immediately think of a reason for that. If it's easy to do,
you might try 6.5 as well, though I'm not sure why it would help.
I also don't recall other similar issues being reported.

Under Windows, the root scanning code was changed to scan only from
MEM_IMAGE mappings. This should be enough to catch dynamic library data,
or at least so we believe. If you were relying the collector tracing from
other kinds of memory mappings, that might explain the problem.
If that is indeed the issue, it would be good to understand it better.

If all else fails, there are some hints for tracking down premature
deallocation issues at

http://www.hpl.hp.com/personal/Hans_Boehm/gc/debugging.html

Hans

Post by Emmanuel Stapf [ES]
I'm upgrading my application so that it uses the more recent version 6.4 but when
I use it, I'm getting very strange memory corruption. Any help for debugging is
appreciated.
- The platform is Windows XP Pro.
- ALL_INTERIOR_POINTERS is not defined (I'm simply hand editing NT_MAKEFILE to get
rid of its definition)
- I'm calling GC_register_displacement (8)
- I'm simply calling GC_malloc, GC_malloc_atomic, GC_realloc and
GC_register_finalizer
I've enabled the assertions to see if I was getting something before the actual
crash, but nothing.
To summarize, with 6.1 it works great, with 6.4 it does not.
Thanks,
Manu
------------------------------------------------------------------------
Eiffel Software
805-685-1006
http://www.eiffel.com
Customer support: http://support.eiffel.com
User group: http://groups.eiffel.com/join
------------------------------------------------------------------------
_______________________________________________
Gc mailing list
http://www.hpl.hp.com/hosted/linux/mail-archives/gc/

[Emmanuel Stapf [ES]]

Post by Hans Boehm
I can't immediately think of a reason for that. If it's easy
to do, you might try 6.5 as well, though I'm not sure why it
would help.
I also don't recall other similar issues being reported.

Updated to 6.5 and got the same problem. By the way the default source download
link will download 6.4 and not 6.5.

Post by Hans Boehm
Under Windows, the root scanning code was changed to scan
only from MEM_IMAGE mappings. This should be enough to catch
dynamic library data, or at least so we believe. If you were
relying the collector tracing from other kinds of memory
mappings, that might explain the problem.
If that is indeed the issue, it would be good to understand it better.

How do I revert to the previous scanning in 6.5 so that we can figure out if this
is the problem or not?

Post by Hans Boehm
If all else fails, there are some hints for tracking down
premature deallocation issues at
http://www.hpl.hp.com/personal/Hans_Boehm/gc/debugging.html

I've done some debugging, but I'm not sure to understand the results better. After
adding GC_DEBUG and GC_ASSERTIONS, it crashes much earlier. The crash occurring
after the second call to `GC_finish_finalization'. Doing `GC_find_header' on my
allocated block shows that the block was last reclaimed at the second cycle. The
strange thing is that just after allocating it, it reports it was reclaimed at the
first cycle.

At the point of the crash, my code is trying to access some memory which should be
alive, but it has been zeroed. My block is referenced in my C code by a local
variable, the C generated output is optimized and I don't clearly see where the
reference is stored. The fact the code is optimized should not matter as it works
just fine with 6.1 (same code, but linked with gc.lib from 6.1 rather than from
6.5)

Although I tried GC_dump but nothing gets printed. Is it not working on windows
from the VS debugger?

I've tried enabling ALL_INTERIOR_POINTERS and it fails much later, but still
fails. By the way, although I never noticed it before, I've been using
`GC_register_displacement (8)', but I've noticed that the implementation of
`GC_is_valid_displacement' will fail if the block is larger than a certain size?
Does it mean that I cannot use `GC_register_displacement' at all if I cannot
guarantee that my blocks will be smaller than a certain size?

Thanks,
Manu

PS: I'm using VC++ 6.0 sp5 in case it matters.

[Emmanuel Stapf [ES]]

In addition, I've down testing on the various version of the GC. And from 6.1 to
6.3 it works for me. So the breaking change occurred between version 6.3 and 6.4.
Then I found in `dyn_load.c' the part about MEM_IMAGE you were mentioning in your
first email, and that was it. Reverting to the 6.3 version made it work on both
6.4 and 6.5.

Any hints on why it would fail for me? Could it be because I'm using a mixture of
malloc and gc_malloc? The use of `malloc' is beyond my control.

Thanks,
Manu

[Hans Boehm]

Do the malloc'ed objects contain pointers to the garbage-collected heap?
If so, that might explain it. Older versions of the collector also
tried not to trace from the system malloc heap, but they often did so
by accident. The restriction to MEM_IMAGE may be more effective at
excluding the malloc heap.

Tracing from the malloc heap is problematic, since it's hard on some
other platforms, and the GC ends up tracing from deallocated memory
in the malloc heap as well as active memory.

Is it possible to use one of the "API hook" packages to redirect the
malloc calls to GC_malloc_uncollectable? I haven't pursued this, but
someone else suggested this approach in a different context on this list,
IIRC.

Hans

Post by Emmanuel Stapf [ES]
In addition, I've down testing on the various version of the GC. And from 6.1 to
6.3 it works for me. So the breaking change occurred between version 6.3 and 6.4.
Then I found in `dyn_load.c' the part about MEM_IMAGE you were mentioning in your
first email, and that was it. Reverting to the 6.3 version made it work on both
6.4 and 6.5.
Any hints on why it would fail for me? Could it be because I'm using a mixture of
malloc and gc_malloc? The use of `malloc' is beyond my control.
Thanks,
Manu

[Emmanuel Stapf [ES]]

Post by Hans Boehm
Do the malloc'ed objects contain pointers to the
garbage-collected heap?

As far as I know they don't. But I'll double check this morning.

Regards,
Manu

[Emmanuel Stapf [ES]]

So I was using `malloc' for some C structures that actually had some references to
objects I wanted to maintain alive. So my bad. I removed those occurences to use
`GC_malloc' instead.

It did go further, but it still fails. It is collecting a piece of memory that is
directly referenced in the per thread data of a thread. Could it be that the gc
does not know about those per thread data location?

Thanks,
Manu

-----Original Message-----
Sent: Tuesday, June 21, 2005 7:56 AM
To: 'Hans Boehm'
Subject: RE: [Gc] Difference between version 6.3 and latest
which is 6.4

Post by Hans Boehm
Do the malloc'ed objects contain pointers to the garbage-collected
heap?

As far as I know they don't. But I'll double check this morning.
Regards,
Manu
_______________________________________________
Gc mailing list
http://www.hpl.hp.com/hosted/linux/mail-archives/gc/

[Ben Hutchings]

Post by Emmanuel Stapf [ES]
So I was using `malloc' for some C structures that actually had some references to
objects I wanted to maintain alive. So my bad. I removed those occurences to use
`GC_malloc' instead.
It did go further, but it still fails. It is collecting a piece of memory that is
directly referenced in the per thread data of a thread. Could it be that the gc
does not know about those per thread data location?

Correct. The GC generally doesn't see thread-local storage. See the
thread "Registering TLS as roots" in the archive at
<http://www.hpl.hp.com/hosted/linux/mail-archives/gc/2004-October/subject.html>.

Ben.

[Emmanuel Stapf [ES]]

Post by Emmanuel Stapf [ES]
It did go further, but it still fails. It is collecting a
piece of memory that is directly referenced in the per thread
data of a thread. Could it be that the gc does not know about
those per thread data location?

To be clearer, the per thread data was allocated with GC_malloc and the field in
it too. During one of the collection, the field is somehow reset to 0 where it
should not.

Regards,
Manu

[Boehm, Hans]

The collector generally has trouble finding thread-specific data.

Does Windows have a way to enumerate it? In general it seems to
be very hard to find it, since it's very dependent on the specific
threads implementation. (I wouldn't be opposed to platform-specific
patches to trace from thread-specific data, provided they do no
harm if you are running the "wrong" version of the threads
package, etc.)

The standard work-around is to also store a copy of the pointer in
some static data structure, or to allocate it with
GC_malloc_uncollectable.

Hans

-----Original Message-----
Sent: Tuesday, June 21, 2005 11:36 AM
Subject: RE: [Gc] Difference between version 6.3 and latest
which is 6.4
So I was using `malloc' for some C structures that actually
had some references to objects I wanted to maintain alive. So
my bad. I removed those occurences to use `GC_malloc' instead.
It did go further, but it still fails. It is collecting a
piece of memory that is directly referenced in the per thread
data of a thread. Could it be that the gc does not know about
those per thread data location?
Thanks,
Manu

-----Original Message-----

Stapf [ES]

Sent: Tuesday, June 21, 2005 7:56 AM
To: 'Hans Boehm'
Subject: RE: [Gc] Difference between version 6.3 and latest
which is 6.4

Post by Hans Boehm
Do the malloc'ed objects contain pointers to the garbage-collected
heap?

As far as I know they don't. But I'll double check this morning.
Regards,
Manu
_______________________________________________
Gc mailing list
http://www.hpl.hp.com/hosted/linux/mail-archives/gc/

[Ben Hutchings]

Post by Boehm, Hans
The collector generally has trouble finding thread-specific data.
Does Windows have a way to enumerate it? In general it seems to
be very hard to find it, since it's very dependent on the specific
threads implementation.

<snip>

It's part of the TEB structure defined in <winternl.h> and that can be
found through fs:[18h]. We know how to read each thread's saved
registers so it shouldn't be too hard to iterate over all TLS.

Ben.

[Emmanuel Stapf [ES]]

Post by Boehm, Hans
Does Windows have a way to enumerate it? In general it seems

I don't think so, but I'm not an expert here.

Post by Boehm, Hans
The standard work-around is to also store a copy of the
pointer in some static data structure, or to allocate it with
GC_malloc_uncollectable.

I've done that and it now seems to work fine. I have a question. In my code I use
my own wrapper of `malloc', `realloc', `calloc' and `free' for the memory
management of data which does not contain references to other allocated memory
areas. Would it be better to wrap my `malloc' using `GC_malloc_uncollectable'
rather than `GC_malloc' so that it behaves the same as the normal `malloc'. That
is to say, I do not want the GC to collect it, because it would be an error in the
way my program is written. Is this more expensive than `GC_malloc', in other words
would it make the gc slower by having too many roots to scan?

Regards,
Manu

[Boehm, Hans]

I don't think that there is a big difference between
GC_malloc_uncollectable and GC_malloc performance, though I
haven't timed it. GC_malloc_uncollectable memory is
effectively treated as part of the root set, unlike the
system malloc. If you're sure it contains no pointers,
I would probably use GC_malloc_atomic_uncollectable,
though the system malloc is not a bad choice either.

Hans

-----Original Message-----
Sent: Tuesday, June 21, 2005 2:24 PM
To: Boehm, Hans
Subject: RE: [Gc] Difference between version 6.3 and latest
which is 6.4

Post by Boehm, Hans
Does Windows have a way to enumerate it? In general it seems

I don't think so, but I'm not an expert here.

Post by Boehm, Hans
The standard work-around is to also store a copy of the
pointer in some static data structure, or to allocate it with
GC_malloc_uncollectable.

I've done that and it now seems to work fine. I have a
question. In my code I use my own wrapper of `malloc',
`realloc', `calloc' and `free' for the memory management of
data which does not contain references to other allocated
memory areas. Would it be better to wrap my `malloc' using
`GC_malloc_uncollectable' rather than `GC_malloc' so that it
behaves the same as the normal `malloc'. That is to say, I do
not want the GC to collect it, because it would be an error
in the way my program is written. Is this more expensive than
`GC_malloc', in other words would it make the gc slower by
having too many roots to scan?
Regards,
Manu

[m***@rydia.net]

If you're sure it contains no pointers, I would probably use
GC_malloc_atomic_uncollectable, though the system malloc is
not a bad choice either.

Isn't GC_malloc_atomic_uncollectable only available in special
builds of libgc?

-mental

[Hans Boehm]

It's a configurable option. But I think it's on by default in my source
distributions. I didn't check other distributions.

Hans

Post by m***@rydia.net

If you're sure it contains no pointers, I would probably use
GC_malloc_atomic_uncollectable, though the system malloc is
not a bad choice either.

Isn't GC_malloc_atomic_uncollectable only available in special
builds of libgc?
-mental

[m***@rydia.net]

Post by Hans Boehm
It's a configurable option. But I think it's on by default in my
source distributions. I didn't check other distributions.

Is there a penalty for having it enabled?

-mental

[Boehm, Hans]

The penalty isn't very significant. I think it's a small amount
of extra code in the dynamic library, and the static data associated
with one more object "kind".

I can't see a good reason to configure it out unless you are dealing
with a very memory-constrained system.

Hans

-----Original Message-----
Sent: Wednesday, June 22, 2005 12:06 PM
To: Boehm, Hans
Subject: RE: [Gc] Difference between version 6.3 and latest
which is 6.4

Post by Hans Boehm
It's a configurable option. But I think it's on by default in my
source distributions. I didn't check other distributions.

Is there a penalty for having it enabled?
-mental