This is the mail archive of the
gdb@sourceware.cygnus.com
mailing list for the GDB project.
GDB's bcache improved
- To: gdb at sourceware dot cygnus dot com, Srikanth Adayapalam <srikanth at cup dot hp dot com>
- Subject: GDB's bcache improved
- From: Jim Blandy <jimb at cygnus dot com>
- Date: Sat, 20 Nov 1999 12:03:01 -0500 (EST)
I spent some time looking at the bcache. The bcache overhead is now
substantially reduced: in a simple test, GDB's overall memory
consumption is down by 17%. The hash table has a better hash function
now, and is much more effective.
If the HP folks are willing, I'd like to see how this new GDB behaves
on their compiler; I've included a specific request below.
The bcache is indeed an inefficient data structure. I extended the
bcache statistics to record the the space overhead of the bcache ---
the amount of memory used over the amount occupied by its actual
contents.
I ran GDB on itself, set a breakpoint in main, ran the inferior, and
did `maint print statistics'. The results (edited for brevity, but
I've included the full results at the end):
Byte cache statistics for '/umbra/jimb/i686/build/comp-tools/gdb/gdb':
Space used for caching: 1594850
Space overhead of caching: 124%
Average hash table population: 8%
Byte cache statistics for '/lib/libdl.so.2':
Space used for caching: 535053
Space overhead of caching: 2213%
Average hash table population: 0%
Byte cache statistics for '/lib/libm.so.6':
Space used for caching: 571754
Space overhead of caching: 1326%
Average hash table population: 1%
Byte cache statistics for '/lib/libc.so.6':
Space used for caching: 1030356
Space overhead of caching: 336%
Average hash table population: 3%
Byte cache statistics for '/lib/ld-linux.so.2':
Space used for caching: 554422
Space overhead of caching: 1300%
Average hash table population: 1%
That's overhead above and beyond the actual size of the data. It's
bad enough that I thought I must be collecting the data incorrectly.
But I checked, and didn't see anything wrong.
I changed the bcache to use a single hash table, instead of a separate
hash table for each string length. This meant I had to store the
length with each string in the table, increasing the per-string
overhead.
*FISHY MOVE ALERT*: This also made the statistics-collecting code
irrelevant, so I rewrote it. The statistics GDB collects now are
slightly different. However, the 17% figure quoted above was obtained
using 'ps', so it's not brought into question by this change.
Anyway, here are the corresponding results from the new GDB. I've
added a "net memory savings" figure, comparing the memory consumption
using the bcache with what you would get with no bcache at all:
Byte cache statistics for '/umbra/jimb/i686/build/comp-tools/gdb/gdb':
Total memory used by bcache, including overhead: 940303
Percentage memory overhead: 32%
Net memory savings: 37%
Hash table population: 100%
Byte cache statistics for '/lib/libdl.so.2':
Total memory used by bcache, including overhead: 31173
Percentage memory overhead: 34%
Net memory savings: 17%
Hash table population: 63%
Byte cache statistics for '/lib/libm.so.6':
Total memory used by bcache, including overhead: 54854
Percentage memory overhead: 36%
Net memory savings: 77%
Hash table population: 84%
Byte cache statistics for '/lib/libc.so.6':
Total memory used by bcache, including overhead: 317584
Percentage memory overhead: 34%
Net memory savings: 73%
Hash table population: 100%
Byte cache statistics for '/lib/ld-linux.so.2':
Total memory used by bcache, including overhead: 53458
Percentage memory overhead: 35%
Net memory savings: 46%
Hash table population: 82%
The overhead is down to something more normal --- around a third of
the size of the actual content. I suspect that most of this comes
from small objects, so there are further opportunities for improvement
here.
Also, objects are spread out much better across the hash table
entries. For the biggest objfile, GDB itself, we have:
Hash table population: 100%
Median hash chain length: 28
Average hash chain length: 28
Maximum hash chain length: 50
For overall memory savings, I compared a GDB from Thursday with a
GDB using the new bcache, on i386 Linux. In both cases, the inferior
program was the new GDB, and the commands were:
$ ${GDB executable name} -nw gdb
(gdb) break main
(gdb) run
(gdb) ^Z
$ ps uww ${pid of running GDB}
For Thursday's GDB:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
jimb 22044 4.1 11.5 18028 14732 pts/0 T 11:42 0:01 /umbra/jimb/i686/bin/gdb -nw gdb
For the new GDB:
jimb 22048 15.8 9.1 14948 11644 pts/0 T 11:43 0:01 ./gdb -nw gdb
Comparing the virtual memory sizes (VSZ), the new GDB is 17% smaller.
So, getting back to the original problem:
I recognize that HP removes duplicate debug information from their
executables, so in theory the bcache is irrelevant to them. But
rather than excising the bcache on their platform, I'd like the bcache
overhead to be so small that HP wouldn't have noticed it in their
attempts to reduce GDB's memory consumption.
Srikanth, I'd be very interested to see how the new GDB behaves on
your C compiler. If you have time, please try the following on the
next GDB snapshot:
$ ${new GDB} -nw ${your C compiler}
(gdb) break main (or wherever is appropriate for C++ nowadays)
(gdb) run
(gdb) maint print statistics
Anyway, here are the full results from Thursday's GDB:
Stock GDB. The only changes made were to bcache.c to collect and
report more information.
The thing to notice is that the space _overhead_ is uniformly pretty
terrible:
91: Space overhead of caching: 124%
107: Space overhead of caching: (not applicable)
123: Space overhead of caching: (not applicable)
139: Space overhead of caching: 2213%
155: Space overhead of caching: 1326%
171: Space overhead of caching: 336%
187: Space overhead of caching: 1300%
No file specified:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
jimb 22002 0.0 1.1 4796 1524 pts/0 T 11:02 0:00 /umbra/jimb/i686
After `file gdb':
jimb 22002 0.6 5.2 9952 6708 pts/0 T 11:02 0:00 /umbra/jimb/i686
After `break main' and `run':
jimb 22002 0.9 10.4 16628 13336 pts/0 T 11:02 0:01 /umbra/jimb/i686
If you were using GDBtk, you'd have found your bug by now.
To disable command-line taunts, run gdb with "-w".
GNU gdb 4.18.1
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
Setting up the environment for debugging gdb.
Breakpoint 1 at 0x80a205b: file /ecliptic/jimb/main/comp-tools/devo/gdb/utils.c, line 644.
Breakpoint 2 at 0x809fdd7: file /ecliptic/jimb/main/comp-tools/devo/gdb/top.c, line 2931.
Breakpoint 3 at 0x804eaa8: file /ecliptic/jimb/main/comp-tools/devo/gdb/main.c, line 821.
Breakpoint 3, main (argc=1, argv=0xbffff544)
at /ecliptic/jimb/main/comp-tools/devo/gdb/main.c:821
821 args.argc = argc;
Statistics for '/umbra/jimb/i686/build/comp-tools/gdb/gdb':
Number of "stab" symbols read: 205494
Number of "minimal" symbols read: 10345
Number of "partial" symbols read: 31736
Number of "types" defined: 4509
Space used by a.out string tables: 1355634
Total memory used for psymbol obstack: 1551190
Total memory used for psymbol cache: 1703164
Total memory used for symbol obstack: 1584408
Total memory used for type obstack: 523004
Statistics for '/usr/lib/libncurses.so.4':
Number of "minimal" symbols read: 648
Total memory used for psymbol obstack: 4072
Total memory used for psymbol cache: 4072
Total memory used for symbol obstack: 35680
Total memory used for type obstack: 4072
Statistics for '/usr/X11R6/lib/libX11.so.6':
Number of "minimal" symbols read: 2425
Total memory used for psymbol obstack: 4072
Total memory used for psymbol cache: 4072
Total memory used for symbol obstack: 140372
Total memory used for type obstack: 4072
Statistics for '/lib/libdl.so.2':
Number of "stab" symbols read: 1701
Number of "minimal" symbols read: 61
Number of "partial" symbols read: 832
Space used by a.out string tables: 36806
Total memory used for psymbol obstack: 57266
Total memory used for psymbol cache: 580228
Total memory used for symbol obstack: 4072
Total memory used for type obstack: 4072
Statistics for '/lib/libm.so.6':
Number of "stab" symbols read: 21013
Number of "minimal" symbols read: 1264
Number of "partial" symbols read: 5712
Space used by a.out string tables: 41957
Total memory used for psymbol obstack: 107209
Total memory used for psymbol cache: 621144
Total memory used for symbol obstack: 61928
Total memory used for type obstack: 4072
Statistics for '/lib/libc.so.6':
Number of "stab" symbols read: 137855
Number of "minimal" symbols read: 3972
Number of "partial" symbols read: 26684
Space used by a.out string tables: 520856
Total memory used for psymbol obstack: 814140
Total memory used for psymbol cache: 1116600
Total memory used for symbol obstack: 216424
Total memory used for type obstack: 4072
Statistics for '/lib/ld-linux.so.2':
Number of "stab" symbols read: 13541
Number of "minimal" symbols read: 303
Number of "partial" symbols read: 2245
Number of "types" defined: 626
Space used by a.out string tables: 83852
Total memory used for psymbol obstack: 124672
Total memory used for psymbol cache: 608732
Total memory used for symbol obstack: 126212
Total memory used for type obstack: 65152
Byte cache statistics for '/umbra/jimb/i686/build/comp-tools/gdb/gdb':
Cached 'partial symbol cache' statistics:
Cache hits: 34627
Cache misses: 28845
Cache hit ratio: 54%
Space used for caching: 1594850
Size of cache content: 709422
Space overhead of caching: 124%
Space saved by cache hits: 797901
Number of bcache overflows: 0
Number of index buckets used: 47
Number of hash table buckets used: 16851
Number of chained items: 28845
Average hash table population: 8%
Average chain length: 1
Maximum chain length 16 at 32:1689
Byte cache statistics for '/usr/lib/libncurses.so.4':
Cached 'partial symbol cache' statistics:
Cache hits: 0
Cache misses: 0
Cache hit ratio: (not applicable)
Space used for caching: 0
Size of cache content: 0
Space overhead of caching: (not applicable)
Space saved by cache hits: 0
Number of bcache overflows: 0
Number of index buckets used: 0
Number of hash table buckets used: 0
Number of chained items: 0
Average hash table population: (not applicable)
Average chain length: (not applicable)
Maximum chain length 0 at -1073746536:136374592
Byte cache statistics for '/usr/X11R6/lib/libX11.so.6':
Cached 'partial symbol cache' statistics:
Cache hits: 0
Cache misses: 0
Cache hit ratio: (not applicable)
Space used for caching: 0
Size of cache content: 0
Space overhead of caching: (not applicable)
Space saved by cache hits: 0
Number of bcache overflows: 0
Number of index buckets used: 0
Number of hash table buckets used: 0
Number of chained items: 0
Average hash table population: (not applicable)
Average chain length: (not applicable)
Maximum chain length 0 at -1073746536:136374592
Byte cache statistics for '/lib/libdl.so.2':
Cached 'partial symbol cache' statistics:
Cache hits: 658
Cache misses: 1006
Cache hit ratio: 39%
Space used for caching: 535053
Size of cache content: 23125
Space overhead of caching: 2213%
Space saved by cache hits: 14479
Number of bcache overflows: 0
Number of index buckets used: 31
Number of hash table buckets used: 899
Number of chained items: 1006
Average hash table population: 0%
Average chain length: 1
Maximum chain length 5 at 32:741
Byte cache statistics for '/lib/libm.so.6':
Cached 'partial symbol cache' statistics:
Cache hits: 9577
Cache misses: 1847
Cache hit ratio: 83%
Space used for caching: 571754
Size of cache content: 40078
Space overhead of caching: 1326%
Space saved by cache hits: 207189
Number of bcache overflows: 0
Number of index buckets used: 32
Number of hash table buckets used: 1714
Number of chained items: 1847
Average hash table population: 1%
Average chain length: 1
Maximum chain length 4 at 32:279
Byte cache statistics for '/lib/libc.so.6':
Cached 'partial symbol cache' statistics:
Cache hits: 43145
Cache misses: 10223
Cache hit ratio: 80%
Space used for caching: 1030356
Size of cache content: 235800
Space overhead of caching: 336%
Space saved by cache hits: 949008
Number of bcache overflows: 0
Number of index buckets used: 46
Number of hash table buckets used: 7496
Number of chained items: 10223
Average hash table population: 3%
Average chain length: 1
Maximum chain length 12 at 32:2869
Byte cache statistics for '/lib/ld-linux.so.2':
Cached 'partial symbol cache' statistics:
Cache hits: 2755
Cache misses: 1735
Cache hit ratio: 61%
Space used for caching: 554422
Size of cache content: 39578
Space overhead of caching: 1300%
Space saved by cache hits: 60150
Number of bcache overflows: 0
Number of index buckets used: 31
Number of hash table buckets used: 1576
Number of chained items: 1735
Average hash table population: 1%
Average chain length: 1
Maximum chain length 4 at 32:408
Here are the results from the new GDB:
No file specified:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
jimb 21983 0.0 1.1 4796 1524 pts/0 T 10:59 0:00 ./gdb -nw
After `file gdb':
jimb 21983 0.9 4.6 9240 5996 pts/0 T 10:59 0:00 ./gdb -nw
After `break main' and `run':
jimb 21983 1.4 8.0 13544 10244 pts/0 T 10:59 0:01 ./gdb -nw
This is (- 1.0 (/ 13544.0 16628)) 0.19 better than before.
Nine out of ten dentists agree that using GDBtk helps prevent tooth decay.
To disable command-line taunts, run gdb with "-w".
GNU gdb 4.18.1
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
Setting up the environment for debugging gdb.
Breakpoint 1 at 0x80a22bb: file /ecliptic/jimb/main/comp-tools/devo/gdb/utils.c, line 644.
Breakpoint 2 at 0x80a0037: file /ecliptic/jimb/main/comp-tools/devo/gdb/top.c, line 2931.
Breakpoint 3 at 0x804ead8: file /ecliptic/jimb/main/comp-tools/devo/gdb/main.c, line 821.
Breakpoint 3, main (argc=1, argv=0xbffff544)
at /ecliptic/jimb/main/comp-tools/devo/gdb/main.c:821
821 args.argc = argc;
Statistics for '/umbra/jimb/i686/build/comp-tools/gdb/gdb':
Number of "stab" symbols read: 205491
Number of "minimal" symbols read: 10347
Number of "partial" symbols read: 31744
Number of "types" defined: 4508
Space used by a.out string tables: 1356324
Total memory used for psymbol obstack: 1551880
Total memory used for psymbol cache: 969136
Total memory used for symbol obstack: 1584480
Total memory used for type obstack: 523004
Statistics for '/usr/lib/libncurses.so.4':
Number of "minimal" symbols read: 648
Total memory used for psymbol obstack: 4072
Total memory used for psymbol cache: 4072
Total memory used for symbol obstack: 35680
Total memory used for type obstack: 4072
Statistics for '/usr/X11R6/lib/libX11.so.6':
Number of "minimal" symbols read: 2425
Total memory used for psymbol obstack: 4072
Total memory used for psymbol cache: 4072
Total memory used for symbol obstack: 140372
Total memory used for type obstack: 4072
Statistics for '/lib/libdl.so.2':
Number of "stab" symbols read: 1701
Number of "minimal" symbols read: 61
Number of "partial" symbols read: 832
Space used by a.out string tables: 36806
Total memory used for psymbol obstack: 57266
Total memory used for psymbol cache: 32576
Total memory used for symbol obstack: 4072
Total memory used for type obstack: 4072
Statistics for '/lib/libm.so.6':
Number of "stab" symbols read: 21013
Number of "minimal" symbols read: 1264
Number of "partial" symbols read: 5712
Space used by a.out string tables: 41957
Total memory used for psymbol obstack: 107209
Total memory used for psymbol cache: 57008
Total memory used for symbol obstack: 61928
Total memory used for type obstack: 4072
Statistics for '/lib/libc.so.6':
Number of "stab" symbols read: 137855
Number of "minimal" symbols read: 3972
Number of "partial" symbols read: 26684
Space used by a.out string tables: 520856
Total memory used for psymbol obstack: 814140
Total memory used for psymbol cache: 329832
Total memory used for symbol obstack: 216424
Total memory used for type obstack: 4072
Statistics for '/lib/ld-linux.so.2':
Number of "stab" symbols read: 13541
Number of "minimal" symbols read: 303
Number of "partial" symbols read: 2245
Number of "types" defined: 626
Space used by a.out string tables: 83852
Total memory used for psymbol obstack: 124672
Total memory used for psymbol cache: 57008
Total memory used for symbol obstack: 126212
Total memory used for type obstack: 65152
Byte cache statistics for '/umbra/jimb/i686/build/comp-tools/gdb/gdb':
Cached 'partial symbol cache' statistics:
Total object count: 63488
Unique object count: 28849
Percentage of duplicates, by count: 54%
Total object size: 1507660
Unique object size: 709511
Percentage of duplicates, by size: 52%
Total memory used by bcache, including overhead: 940303
Percentage memory overhead: 32%
Net memory savings: 37%
Hash table population: 100%
Median hash chain length: 28
Average hash chain length: 28
Maximum hash chain length: 50
Byte cache statistics for '/usr/lib/libncurses.so.4':
Cached 'partial symbol cache' statistics:
Total object count: 0
Unique object count: 0
Percentage of duplicates, by count: (not applicable)
Total object size: 0
Unique object size: 0
Percentage of duplicates, by size: (not applicable)
Total memory used by bcache, including overhead: 0
Percentage memory overhead: (not applicable)
Net memory savings: (not applicable)
Hash table population: 0%
Median hash chain length: 0
Average hash chain length: (not applicable)
Maximum hash chain length: 0
Byte cache statistics for '/usr/X11R6/lib/libX11.so.6':
Cached 'partial symbol cache' statistics:
Total object count: 0
Unique object count: 0
Percentage of duplicates, by count: (not applicable)
Total object size: 0
Unique object size: 0
Percentage of duplicates, by size: (not applicable)
Total memory used by bcache, including overhead: 0
Percentage memory overhead: (not applicable)
Net memory savings: (not applicable)
Hash table population: 0%
Median hash chain length: 0
Average hash chain length: (not applicable)
Maximum hash chain length: 0
Byte cache statistics for '/lib/libdl.so.2':
Cached 'partial symbol cache' statistics:
Total object count: 1664
Unique object count: 1006
Percentage of duplicates, by count: 39%
Total object size: 37604
Unique object size: 23125
Percentage of duplicates, by size: 38%
Total memory used by bcache, including overhead: 31173
Percentage memory overhead: 34%
Net memory savings: 17%
Hash table population: 63%
Median hash chain length: 1
Average hash chain length: 1
Maximum hash chain length: 5
Byte cache statistics for '/lib/libm.so.6':
Cached 'partial symbol cache' statistics:
Total object count: 11424
Unique object count: 1847
Percentage of duplicates, by count: 83%
Total object size: 247267
Unique object size: 40078
Percentage of duplicates, by size: 83%
Total memory used by bcache, including overhead: 54854
Percentage memory overhead: 36%
Net memory savings: 77%
Hash table population: 84%
Median hash chain length: 2
Average hash chain length: 2
Maximum hash chain length: 6
Byte cache statistics for '/lib/libc.so.6':
Cached 'partial symbol cache' statistics:
Total object count: 53368
Unique object count: 10223
Percentage of duplicates, by count: 80%
Total object size: 1184808
Unique object size: 235800
Percentage of duplicates, by size: 80%
Total memory used by bcache, including overhead: 317584
Percentage memory overhead: 34%
Net memory savings: 73%
Hash table population: 100%
Median hash chain length: 10
Average hash chain length: 10
Maximum hash chain length: 20
Byte cache statistics for '/lib/ld-linux.so.2':
Cached 'partial symbol cache' statistics:
Total object count: 4490
Unique object count: 1735
Percentage of duplicates, by count: 61%
Total object size: 99728
Unique object size: 39578
Percentage of duplicates, by size: 60%
Total memory used by bcache, including overhead: 53458
Percentage memory overhead: 35%
Net memory savings: 46%
Hash table population: 82%
Median hash chain length: 2
Average hash chain length: 2
Maximum hash chain length: 6