I have a windows service to receive emails that uses OpenPop. However memory usage goes up to 8G about 3 days after a restart. Operation staff gives me a dump file, So I use windbg to analysis it.
When I run !address -summary I got:
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 468 7fd`942f6000 ( 7.991 TB) 99.88%
Heap 645 1`c3f95000 ( 7.062 GB) 72.92% 0.09%
<unknown> 1347 0`9305f000 ( 2.297 GB) 23.72% 0.03%
Image 1985 0`0d28d000 ( 210.551 MB) 2.12% 0.00%
Stack 366 0`077c0000 ( 119.750 MB) 1.21% 0.00%
Other 11 0`001c4000 ( 1.766 MB) 0.02% 0.00%
TEB 122 0`000f4000 ( 976.000 kB) 0.01% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 1725 2`5b3b9000 ( 9.426 GB) 97.33% 0.12%
MEM_IMAGE 2692 0`0f249000 ( 242.285 MB) 2.44% 0.00%
MEM_MAPPED 60 0`016f8000 ( 22.969 MB) 0.23% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 468 7fd`942f6000 ( 7.991 TB) 99.88%
MEM_COMMIT 3373 2`10e49000 ( 8.264 GB) 85.33% 0.10%
MEM_RESERVE 1104 0`5aeb1000 ( 1.421 GB) 14.67% 0.02%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 1528 2`0092c000 ( 8.009 GB) 82.70% 0.10%
PAGE_EXECUTE_READ 308 0`0b666000 ( 182.398 MB) 1.84% 0.00%
PAGE_READONLY 924 0`0323a000 ( 50.227 MB) 0.51% 0.00%
PAGE_WRITECOPY 321 0`012f3000 ( 18.949 MB) 0.19% 0.00%
PAGE_EXECUTE_READWRITE 112 0`005d2000 ( 5.820 MB) 0.06% 0.00%
PAGE_READWRITE|PAGE_GUARD 122 0`0023c000 ( 2.234 MB) 0.02% 0.00%
PAGE_EXECUTE_WRITECOPY 57 0`00178000 ( 1.469 MB) 0.01% 0.00%
PAGE_EXECUTE 1 0`00004000 ( 16.000 kB) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 5`2d890000 7f9`6d640000 ( 7.974 TB)
Heap 0`25d40000 0`00fd0000 ( 15.813 MB)
<unknown> 0`32e89000 0`0eb07000 ( 235.027 MB)
Image 7fe`f466a000 0`01338000 ( 19.219 MB)
Stack 0`19c60000 0`000fc000 (1008.000 kB)
Other 0`00de0000 0`00181000 ( 1.504 MB)
TEB 7ff`ffdb8000 0`00002000 ( 8.000 kB)
PEB 7ff`fffdf000 0`00001000 ( 4.000 kB)
and !eeheap -gc I got:
Number of GC Heaps: 1
generation 0 starts at 0x000000051da29040
generation 1 starts at 0x000000051d891000
generation 2 starts at 0x0000000001281000
ephemeral segment allocation context: none
segment begin allocated size
0000000001280000 0000000001281000 000000000b1eae80 0x9f69e80(167157376)
0000000031990000 0000000031991000 0000000032e68ee0 0x14d7ee0(21855968)
000000007fff0000 000000007fff1000 0000000081a99330 0x1aa8330(27951920)
000000008fff0000 000000008fff1000 0000000099d1bb00 0x9d2ab00(164801280)
000000009fff0000 000000009fff1000 00000000a77989c0 0x77a79c0(125467072)
000000051d890000 000000051d891000 000000052352fe40 0x5c9ee40(97119808)
Large object heap starts at 0x0000000011281000
segment begin allocated size
0000000011280000 0000000011281000 0000000019236c28 0x7fb5c28(133913640)
0000000048fa0000 0000000048fa1000 0000000050a87908 0x7ae6908(128870664)
0000000050fa0000 0000000050fa1000 000000005387c418 0x28db418(42841112)
00000000afff0000 00000000afff1000 00000000b3bf0840 0x3bff840(62912576)
Total Size: Size: 0x39fd2518 (972891416) bytes.
------------------------------
GC Heap Size: Size: 0x39fd2518 (972891416) bytes.
From these two commands, there is some unmanaged memory leak.
How can I find out what is in the unmanaged memory and by what method?
There is no evidence yet, that your situation is a native memory leak. Yes, you have 8 GB in native heaps. But you also have 900 MB in managed heaps which could hold the native objects alive.
I suggest using sos and starting with !dumpheap -stat if you really want to go with WinDbg. Otherwise use a .NET memory profiler. Many people have a JetBrains dotMemory license because they use R# Ultimate. It's much easier to use than WinDbg and has better support for comparing snapshots over time.
As far as I can see there is no breakdown into memory usage of individual .net components. To do this you would need to load the SOS debugger extension. The article Hunting .NET memory leaks with Windbg by André Snede Kock provides some more details.
If you are new to memory debugging I would probably also recommend a commercial tool, like dotMemory or ANTS, since they tend to be easier to use.
This should give you a list of .Net objects, how many instances are alive, and how much memory they use. I usually try to look for unexpected things, typically if there are more objects alive than I would expect, and then try to find out why. This helps with detecting problems in managed code, and I would say this is the most likely problem.
Another possibility is that there is a un-managed memory leak in some libary you use, but then you would need to contact the vendor of that library to get it fixed.
Related
In address -summary, I found inside MEM_COMMIT to 4.423 GB, but I can only get in eeheap 405618688 bytes, I passed! Heap -s can get the memory is very small.Memory.
Now I don't know, how can I investigate the remaining memory of MEM COMMIT .
!address -summary MEM_COMMIT 4.423 GB
0:000> !address -summary
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 579 7df9`74541000 ( 125.974 TB) 98.42%
<unknown> 912 206`7c9dc000 ( 2.025 TB) 99.99% 1.58%
Image 972 0`06a01000 ( 106.004 MB) 0.00% 0.00%
Heap 81 0`05a27000 ( 90.152 MB) 0.00% 0.00%
Stack 114 0`02a80000 ( 42.500 MB) 0.00% 0.00%
Other 9 0`001de000 ( 1.867 MB) 0.00% 0.00%
TEB 38 0`0004c000 ( 304.000 kB) 0.00% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_MAPPED 220 200`062b0000 ( 2.000 TB) 98.74% 1.56%
MEM_PRIVATE 935 6`7edfe000 ( 25.982 GB) 1.25% 0.02%
MEM_IMAGE 972 0`06a01000 ( 106.004 MB) 0.00% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 579 7df9`74541000 ( 125.974 TB) 98.42%
MEM_RESERVE 296 205`709f6000 ( 2.021 TB) 99.79% 1.58%
MEM_COMMIT 1831 1`1b0b9000 ( 4.423 GB) 0.21% 0.00%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 862 1`0e1cc000 ( 4.221 GB) 0.20% 0.00%
PAGE_READONLY 579 0`0659a000 ( 101.602 MB) 0.00% 0.00%
PAGE_EXECUTE_READ 159 0`04f2b000 ( 79.168 MB) 0.00% 0.00%
PAGE_NOACCESS 80 0`01550000 ( 21.312 MB) 0.00% 0.00%
PAGE_EXECUTE_READWRITE 19 0`002ba000 ( 2.727 MB) 0.00% 0.00%
PAGE_WRITECOPY 94 0`001aa000 ( 1.664 MB) 0.00% 0.00%
PAGE_READWRITE | PAGE_GUARD 38 0`00074000 ( 464.000 kB) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 21e`6c05c000 7bd5`f48f4000 ( 123.836 TB)
<unknown> 7dfd`d8ac0000 1f7`68898000 ( 1.966 TB)
Image 7ff9`71121000 0`009ba000 ( 9.727 MB)
Heap 218`83c71000 0`00eff000 ( 14.996 MB)
Stack 22`cad80000 0`0017c000 ( 1.484 MB)
Other 218`f1b50000 0`00181000 ( 1.504 MB)
TEB 22`ca410000 0`00002000 ( 8.000 kB)
PEB 22`ca40f000 0`00001000 ( 4.000 kB)
!eeheap -gc 405618688 bytes.
0:000> !eeheap -gc
Number of GC Heaps: 12
------------------------------
Heap 0 (00000218F3182A40)
generation 0 starts at 0x00000218F34C9DB0
generation 1 starts at 0x00000218F3477478
generation 2 starts at 0x00000218F3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
00000218F3460000 00000218F3461000 00000218F3A0BDC8 00000218F3F02000 0x5aadc8(5942728) 0xaa1000(11145216)
Large object heap starts at 0x0000021BF3461000
segment begin allocated committed allocated size committed size
0000021BF3460000 0000021BF3461000 0000021BF449E8D0 0000021BF44BF000 0x103d8d0(17029328) 0x105e000(17162240)
Pinned object heap starts at 0x0000021CB3461000
0000021CB3460000 0000021CB3461000 0000021CB3464FF0 0000021CB3472000 0x3ff0(16368) 0x11000(69632)
Allocated Heap Size: Size: 0x15ec688 (22988424) bytes.
Committed Heap Size: Size: 0x1aff000 (28307456) bytes.
------------------------------
Heap 1 (00000218F31AE470)
generation 0 starts at 0x00000219334C05F8
generation 1 starts at 0x0000021933471E28
generation 2 starts at 0x0000021933461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021933460000 0000021933461000 0000021933A20610 0000021933EE2000 0x5bf610(6026768) 0xa81000(11014144)
Large object heap starts at 0x0000021C03461000
segment begin allocated committed allocated size committed size
0000021C03460000 0000021C03461000 0000021C0451F4E8 0000021C04520000 0x10be4e8(17556712) 0x10bf000(17559552)
Pinned object heap starts at 0x0000021CC3461000
0000021CC3460000 0000021CC3461000 0000021CC3462030 0000021CC3472000 0x1030(4144) 0x11000(69632)
Allocated Heap Size: Size: 0x167eb28 (23587624) bytes.
Committed Heap Size: Size: 0x1b40000 (28573696) bytes.
------------------------------
Heap 2 (00000218F31DD200)
generation 0 starts at 0x00000219734967E0
generation 1 starts at 0x000002197346DFB0
generation 2 starts at 0x0000021973461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021973460000 0000021973461000 00000219734987F8 0000021973E52000 0x377f8(227320) 0x9f1000(10424320)
Large object heap starts at 0x0000021C13461000
segment begin allocated committed allocated size committed size
0000021C13460000 0000021C13461000 0000021C14B631F8 0000021C14B83000 0x17021f8(24125944) 0x1722000(24256512)
Pinned object heap starts at 0x0000021CD3461000
0000021CD3460000 0000021CD3461000 0000021CD3461018 0000021CD3462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1739a08 (24353288) bytes.
Committed Heap Size: Size: 0x2113000 (34680832) bytes.
------------------------------
Heap 3 (00000218F3208F70)
generation 0 starts at 0x00000219B34B7828
generation 1 starts at 0x00000219B3489AB8
generation 2 starts at 0x00000219B3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
00000219B3460000 00000219B3461000 00000219B3A17840 00000219B3EC2000 0x5b6840(5990464) 0xa61000(10883072)
Large object heap starts at 0x0000021C23461000
segment begin allocated committed allocated size committed size
0000021C23460000 0000021C23461000 0000021C24B43768 0000021C24B83000 0x16e2768(23996264) 0x1722000(24256512)
Pinned object heap starts at 0x0000021CE3461000
0000021CE3460000 0000021CE3461000 0000021CE3462030 0000021CE3472000 0x1030(4144) 0x11000(69632)
Allocated Heap Size: Size: 0x1c99fd8 (29990872) bytes.
Committed Heap Size: Size: 0x2183000 (35139584) bytes.
------------------------------
Heap 4 (00000218F3234CE0)
generation 0 starts at 0x00000219F34DC428
generation 1 starts at 0x00000219F34730F0
generation 2 starts at 0x00000219F3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
00000219F3460000 00000219F3461000 00000219F3BAA440 00000219F3EE2000 0x749440(7640128) 0xa81000(11014144)
Large object heap starts at 0x0000021C33461000
segment begin allocated committed allocated size committed size
0000021C33460000 0000021C33461000 0000021C3410D580 0000021C3412E000 0xcac580(13288832) 0xccd000(13422592)
Pinned object heap starts at 0x0000021CF3461000
0000021CF3460000 0000021CF3461000 0000021CF3465030 0000021CF3472000 0x4030(16432) 0x11000(69632)
Allocated Heap Size: Size: 0x13f99f0 (20945392) bytes.
Committed Heap Size: Size: 0x174e000 (24436736) bytes.
------------------------------
Heap 5 (0000021D7EDD1EF0)
generation 0 starts at 0x0000021A334F9AF8
generation 1 starts at 0x0000021A33485C40
generation 2 starts at 0x0000021A33461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021A33460000 0000021A33461000 0000021A33A59B10 0000021A33F02000 0x5f8b10(6261520) 0xaa1000(11145216)
Large object heap starts at 0x0000021C43461000
segment begin allocated committed allocated size committed size
0000021C43460000 0000021C43461000 0000021C444BCE08 0000021C4451C000 0x105be08(17153544) 0x10bb000(17543168)
Pinned object heap starts at 0x0000021D03461000
0000021D03460000 0000021D03461000 0000021D03461018 0000021D03462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1654930 (23415088) bytes.
Committed Heap Size: Size: 0x1b5c000 (28688384) bytes.
------------------------------
Heap 6 (0000021D7EDFD920)
generation 0 starts at 0x0000021A73495A38
generation 1 starts at 0x0000021A73469D68
generation 2 starts at 0x0000021A73461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021A73460000 0000021A73461000 0000021A73A15A50 0000021A73E92000 0x5b4a50(5982800) 0xa31000(10686464)
Large object heap starts at 0x0000021C53461000
segment begin allocated committed allocated size committed size
0000021C53460000 0000021C53461000 0000021C549CAFA8 0000021C549CB000 0x1569fa8(22454184) 0x156a000(22454272)
Pinned object heap starts at 0x0000021D13461000
0000021D13460000 0000021D13461000 0000021D13461018 0000021D13462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1b1ea10 (28437008) bytes.
Committed Heap Size: Size: 0x1f9b000 (33140736) bytes.
------------------------------
Heap 7 (0000021D7EE29690)
generation 0 starts at 0x0000021AB35B3700
generation 1 starts at 0x0000021AB3580548
generation 2 starts at 0x0000021AB3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021AB3460000 0000021AB3461000 0000021AB39D3718 0000021AB3FC2000 0x572718(5711640) 0xb61000(11931648)
Large object heap starts at 0x0000021C63461000
segment begin allocated committed allocated size committed size
0000021C63460000 0000021C63461000 0000021C649CCBF8 0000021C649ED000 0x156bbf8(22461432) 0x158c000(22593536)
Pinned object heap starts at 0x0000021D23461000
0000021D23460000 0000021D23461000 0000021D23461018 0000021D23462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1ade328 (28173096) bytes.
Committed Heap Size: Size: 0x20ed000 (34525184) bytes.
------------------------------
Heap 8 (0000021D7EE55400)
generation 0 starts at 0x0000021AF3496810
generation 1 starts at 0x0000021AF3472A28
generation 2 starts at 0x0000021AF3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021AF3460000 0000021AF3461000 0000021AF39F6828 0000021AF3E92000 0x595828(5855272) 0xa31000(10686464)
Large object heap starts at 0x0000021C73461000
segment begin allocated committed allocated size committed size
0000021C73460000 0000021C73461000 0000021C7492E5F8 0000021C7494F000 0x14cd5f8(21812728) 0x14ee000(21946368)
Pinned object heap starts at 0x0000021D33461000
0000021D33460000 0000021D33461000 0000021D33461830 0000021D33462000 0x830(2096) 0x1000(4096)
Allocated Heap Size: Size: 0x1a63650 (27670096) bytes.
Committed Heap Size: Size: 0x1f1f000 (32632832) bytes.
------------------------------
Heap 9 (0000021D7EE81170)
generation 0 starts at 0x0000021B334DE9E0
generation 1 starts at 0x0000021B334A73C8
generation 2 starts at 0x0000021B33461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021B33460000 0000021B33461000 0000021B33A409F8 0000021B33E82000 0x5df9f8(6158840) 0xa21000(10620928)
Large object heap starts at 0x0000021C83461000
segment begin allocated committed allocated size committed size
0000021C83460000 0000021C83461000 0000021C849E9DF8 0000021C84A0A000 0x1588df8(22580728) 0x15a9000(22712320)
Pinned object heap starts at 0x0000021D43461000
0000021D43460000 0000021D43461000 0000021D43461018 0000021D43462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1b68808 (28739592) bytes.
Committed Heap Size: Size: 0x1fca000 (33333248) bytes.
------------------------------
Heap 10 (0000021D7EEADD80)
generation 0 starts at 0x0000021B73544EC8
generation 1 starts at 0x0000021B73504C18
generation 2 starts at 0x0000021B73461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021B73460000 0000021B73461000 0000021B73AEAEE0 0000021B73F42000 0x689ee0(6856416) 0xae1000(11407360)
Large object heap starts at 0x0000021C93461000
segment begin allocated committed allocated size committed size
0000021C93460000 0000021C93461000 0000021C955DEF78 0000021C955DF000 0x217df78(35118968) 0x217e000(35119104)
Pinned object heap starts at 0x0000021D53461000
0000021D53460000 0000021D53461000 0000021D53468450 0000021D53472000 0x7450(29776) 0x11000(69632)
Allocated Heap Size: Size: 0x280f2a8 (42005160) bytes.
Committed Heap Size: Size: 0x2c5f000 (46526464) bytes.
------------------------------
Heap 11 (0000021D7EED9980)
generation 0 starts at 0x0000021BB34C6550
generation 1 starts at 0x0000021BB34784A8
generation 2 starts at 0x0000021BB3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021BB3460000 0000021BB3461000 0000021BB3906568 0000021BB3EA2000 0x4a5568(4871528) 0xa41000(10752000)
Large object heap starts at 0x0000021CA3461000
segment begin allocated committed allocated size committed size
0000021CA3460000 0000021CA3461000 0000021CA55A4DB8 0000021CA55A5000 0x2143db8(34880952) 0x2144000(34881536)
Pinned object heap starts at 0x0000021D63461000
0000021D63460000 0000021D63461000 0000021D6346CF88 0000021D63472000 0xbf88(49032) 0x11000(69632)
Allocated Heap Size: Size: 0x25f52a8 (39801512) bytes.
Committed Heap Size: Size: 0x2b85000 (45633536) bytes.
------------------------------
GC Allocated Heap Size: Size: 0x14459f90 (340107152) bytes.
GC Committed Heap Size: Size: 0x182d4000 (405618688) bytes.
!heap -s
0:000> !heap -s
************************************************************************************************************************
NT HEAP STATS BELOW
************************************************************************************************************************
LFH Key : 0xa893144b6b979747
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
00000218f1710000 00000002 65320 14712 64928 5575 163 16 2 c LFH
External fragmentation 38 % (163 free blocks)
00000218f1660000 00008000 64 4 64 2 1 1 0 0
00000218f16f0000 00001002 3516 2388 3124 200 58 3 0 0 LFH
00000218f3360000 00001002 60 8 60 2 1 1 0 0
00000218f3450000 00041002 60 8 60 5 1 1 0 0
0000021d7fa40000 00001002 3516 1460 3124 89 25 3 0 0 LFH
0000021d7fc00000 00001002 60 8 60 5 1 1 0 0
0000021d7fbc0000 00001002 452 32 60 5 2 1 0 0 LFH
-------------------------------------------------------------------------------------
github：https://github.com/hueifeng/MemoryLeak
Unfortunately I can't compile that code given in the Github repository. But what I see in the source code totally makes sense to me.
The code creates 400 objects with a link to an image and adds them to a list:
for (int i = 0; i < 400; i++)
{
list.Add(new TestDto { ImageUrl = "https://gimg2.baidu.com/..." });
}
I guess that the Excel exporter will load those images. The image in the link is 2560×1440 pixels with 24 bits color depth, so that is an uncompressed size of 2560×1440×3 = 11 MB. 400 of them is 4.42 GB.
As per this answer, Bitmaps will be allocated via VirtualAlloc() and that's what you see.
IMHO it's not a memory leak. It's exactly what you have implemented.
In this case, the dump analysis strategies of Dmitry Vostokov might be interesting. What is he doing: he's converting the memory into a bitmap and he claims that he can diagnose quite a few problems just from the look at the resulting picture. You can see examples of that on the cover on one of his books. In your case I'd expect to see at least 400 repeating patterns of the image. Maybe you want to try dump2picture just for fun.
One of our PRD small win-service suddenly skyrocketed to 80+ GB memory for a few hours and then came down and stabilized at 6+ GB memory, it is usually use only under 200 MB. I have grab a memory dump at the time of 6+ GB, and restarted the server, everything go back to normal. I am investigating why it it skyrocketed to that high and why it's stabled at 6+ GB.
I have found that it is an unmanaged memory leak issue, because the CLR heap is small:
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x000000910083ff20
generation 1 starts at 0x00000091007caba0
generation 2 starts at 0x0000009100001000
ephemeral segment allocation context: none
segment begin allocated size
0000009100000000 0000009100001000 00000091029691f0 0x29681f0(43418096)
Large object heap starts at 0x0000009110001000
segment begin allocated size
0000009110000000 0000009110001000 000000911013dcb8 0x13ccb8(1297592)
Total Size: Size: 0x2aa4ea8 (44715688) bytes.
------------------------------
***GC Heap Size: Size: 0x2aa4ea8 (44715688) bytes.***
But the Heap Summary is high, More suspiciously, the lock count is high:
0:000> !heap -s
************************************************************************************************************************
NT HEAP STATS BELOW
************************************************************************************************************************
LFH Key : 0x45d65d36d8f8b642
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
**000000917aa90000 00000002 6718396 6682116 6718196 71860 5580 419 2 93 LFH**
000000917a850000 00008000 64 4 64 2 1 1 0 0
000000917acd0000 00001002 1280 88 1080 15 7 2 0 0 LFH
000000917ac90000 00001002 1280 108 1080 24 8 2 0 0 LFH
000000917b160000 00001002 1280 124 1080 7 10 2 0 0 LFH
000000917b310000 00041002 60 8 60 5 1 1 0 0
000000917bd40000 00041002 260 36 60 4 3 1 0 0 LFH
0000009118080000 00001002 1084 1024 1084 1021 2 1 0 0
-------------------------------------------------------------------------------------
0:000> !heap -stat -h 000000917aa90000
heap # 000000917aa90000
group-by: TOTSIZE max-display: 20
size #blocks total ( %) (percent of total busy bytes)
2d0b0 373e - 9b845aa0 (39.37)
10d1 41962 - 44eed902 (17.45)
ddc8 373e - 2fdbae70 (12.12)
7ab8 373e - 1a7b4090 (6.70)
7168 373e - 1878cf30 (6.20)
29d1 6e6b - 1209485b (4.57)
35d1 372d - b995cbd (2.94)
31d1 372d - abca8bd (2.72)
12d1 6e5a - 81c6b7a (2.05)
21d1 3746 - 74d2626 (1.85)
fd1 6e6c - 6d27a2c (1.73)
1cd1 372d - 635f7bd (1.57)
40 22b52 - 8ad480 (0.14)
100 6ef7 - 6ef700 (0.11)
200 374f - 6e9e00 (0.11)
c3500 5 - 3d0900 (0.06)
80 6edb - 376d80 (0.05)
d8 374e - 2ea9d0 (0.05)
249f18 1 - 249f18 (0.04)
90 3792 - 1f4220 (0.03)
Then I did !heap -flt s 2d0b0
I can't do !heap -p -a to see the stack because I don't have gflags enabled in PRD.
Instead, I am trying to do a dc 00000092b7088bb0 L200 to guess base on the string value. Nothing meaningful in the first chuck (2d0b0 373e - 9b845aa0 (39.37) But in the second chuck ( 10d1 41962 - 44eed902 (17.45)), I found something like
00000092`b70b0250 00000000 00000000 44440000 4e4f4d2d ..........DD-MON
00000092`b70b0260 0052522d 00000000 00000000 00000000 -RR.............
00000092`b70b0270 00000000 00000000 00000000 00000000 ................
00000092`b70b0280 00000000 00000000 00000000 00000000 ................
00000092`b70b0290 00000000 48480000 2e494d2e 46585353 ......HH.MI.SSXF
00000092`b70b02a0 4d412046 00000000 00000000 00000000 F AM............
00000092`b70b02b0 00000000 00000000 00000000 00000000 ................
00000092`b70b02c0 00000000 44440000 4e4f4d2d 2052522d ......DD-MON-RR
00000092`b70b02d0 4d2e4848 53532e49 20464658 00004d41 HH.MI.SSXFF AM..
00000092`b70b02e0 00000000 00000000 00000000 00000000 ................
00000092`b70b02f0 00000000 00000000 00000000 00000000 ................
00000092`b70b0300 00000000 00000000 00000000 00000000 ................
00000092`b70b0310 00000000 48480000 2e494d2e 46585353 ......HH.MI.SSXF
00000092`b70b0320 4d412046 525a5420 00000000 00000000 F AM TZR........
00000092`b70b0330 00000000 00000000 00000000 00000000 ................
00000092`b70b0340 00000000 00000000 00000000 00000000 ................
00000092`b70b0350 00000000 00000000 44440000 4e4f4d2d ..........DD-MON
00000092`b70b0360 2052522d 4d2e4848 53532e49 20464658 -RR HH.MI.SSXFF
00000092`b70b0370 54204d41 0000525a 00000000 00000000 AM TZR..........
It looks like Oracle Database Query Date Time Format string. And We do use Oracle, and I am hard to believe that we are leaking oracle connections, because this small service have run for years and this is the first time this happened.
That's how far I can go now.
Sorry for the long question and thanks for spend time reading
To summarize my question:
what is the Lock Count Column mean in !heap -s , that looks like an
indicator of an issue where I can keep digging, please point me a
direction or blog I can read.
Are those Raw data string ring another bell on you instead of Oracle? Have you ever have similar issue?
We do use a lot .net remoting. I know it's old and I never deep dived into it. Does it could be a factor of this issue?
I can't do anything in PRD, and I can't reproduce this issue in DEV/TEST, is there any other direction I can take?
I have a .NET application that uses many IDisposable types. In one of environment, application is showing high memory usage. I collected a dump file and see following.
0:000> !address -summary
Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 73 7ff`0e99f000 ( 6.112 Tb) 99.95%
Heap 35 0`d06ce000 ( 4.017 Gb) 86.34% 0.04%
<unknown> 56 0`1aebe000 ( 430.742 Mb) 11.15% 0.01%
Image 247 0`05600000 ( 86.000 Mb) 2.23% 0.00%
Stack 27 0`00900000 ( 9.000 Mb) 0.23% 0.00%
Other 9 0`001b2000 ( 1.695 Mb) 0.04% 0.00%
TEB 9 0`00012000 ( 72.000 kb) 0.00% 0.00%
PEB 1 0`00001000 ( 4.000 kb) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 116 0`ea4a6000 ( 3.661 Gb) 97.06% 0.04%
MEM_IMAGE 248 0`05601000 ( 86.004 Mb) 2.23% 0.00%
MEM_MAPPED 20 0`01baa000 ( 27.664 Mb) 0.72% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 73 7ff`0e99f000 ( 6.112 Tb) 99.95%
MEM_COMMIT 337 0`d605a000 ( 3.344 Gb) 88.66% 0.04%
MEM_RESERVE 47 0`1b5f7000 ( 437.965 Mb) 11.34% 0.01%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 120 0`d0427000 ( 4.014 Gb) 86.27% 0.04%
PAGE_EXECUTE_READ 32 0`03de6000 ( 61.898 Mb) 1.60% 0.00%
PAGE_READONLY 102 0`00fe7000 ( 15.902 Mb) 0.41% 0.00%
PAGE_WRITECOPY 44 0`00cf9000 ( 12.973 Mb) 0.34% 0.00%
PAGE_EXECUTE_WRITECOPY 10 0`0012f000 ( 1.184 Mb) 0.03% 0.00%
PAGE_READWRITE|PAGE_GUARD 9 0`00022000 ( 136.000 kb) 0.00% 0.00%
PAGE_EXECUTE_READWRITE 19 0`0001b000 ( 108.000 kb) 0.00% 0.00%
PAGE_EXECUTE 1 0`00001000 ( 4.000 kb) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 1`00061000 7fd`834df000 ( 7.990 Tb)
Heap 0`1b050000 0`10001000 ( 256.004 Mb)
<unknown> 0`025f2000 0`0ffee000 ( 255.930 Mb)
Image 7fe`e1aba000 0`0102d000 ( 16.176 Mb)
Stack 0`1ac30000 0`000fc000 (1008.000 kb)
Other 0`00830000 0`00181000 ( 1.504 Mb)
TEB 7ff`ffef8000 0`00002000 ( 8.000 kb)
PEB 7ff`fffda000 0`00001000 ( 4.000 kb)
The memory usage seem to coming out from the native side of things. The managed memory consumption doesn't seem to be significant.
0:000> !EEHeap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00000000025e1030
generation 1 starts at 0x00000000025e1018
generation 2 starts at 0x00000000025e1000
ephemeral segment allocation context: none
segment begin allocated size
00000000025e0000 00000000025e1000 00000000025effe8 0xefe8(61416)
Large object heap starts at 0x00000000125e1000
segment begin allocated size
00000000125e0000 00000000125e1000 00000000125f1940 0x10940(67904)
Total Size: Size: 0x1f928 (129320) bytes.
------------------------------
GC Heap Size: Size: 0x1f928 (129320) bytes.
What can I do to determine what objects are responsible for this much memory?
I'm analyzing a memory dump of a .NET process using WinDbg and I've noticed it reports the size of all System.Int32 variables on the heap as 24 bytes. Here's an example of a relevant DumpObj call on one of the variables:
0:000> !DumpObj /d 00000061c81c0e80
Name: System.Int32
MethodTable: 00007fff433f37c8
EEClass: 00007fff42e30130
Size: 24(0x18) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007fff433f37c8 4000456 8 System.Int32 1 instance 141 m_value
As far as I know, the size System.Int32 is supposed to be 4 bytes. What is the source of this discrepancy?
There's an overhead to any object on the heap. On 32-bit MS.NET runtime, this is 8 bytes, and on 64-bit, 16 bytes (disclaimer: this isn't strictly contractual and may change in the future or in a conforming implementation of a .NET runtime).
Since your int is boxed, it will have the 16-byte overhead. So you might expect 20 bytes to be used total. Well, except that on 64-bit systems, objects (and structures) are padded to 8-byte boundaries, so you actually get 24-bytes per int.
In contrast, when you use a struct with 16 integers, you'll only use 16 + 4 * 16 = 80 bytes of memory, for a total of 5 bytes per integer.
And again, most of this is an implementation detail, so not something you can rely on; it's perfectly possible for a valid .NET runtime to store a single int in 1 MiB of memory if it desires to do so, and it could also store it in some compact representation or with interning, as long as it conforms to all the contractual behaviour of the type. It's also quite simplified even compared to actual MS runtime implementations - for example, if your object gets large enough, it will need more overhead.
it is not size of int32 do dd or dq address and see your int32 stuck in the second dword or qword
there is an implicit overhead of 12 bytes or 24 bytes per object for x86 / x64 respectively
0:004> .shell -ci "!DumpObj /d 01c72360" grep -i size
Size: 12(0xc) bytes
.shell: Process exited
0:004> dd 01c72360 l4
01c72360 5890c770 000001b5 80000000 5890afb0
0:004> .shell -ci "!DumpObj /d 01c72360" grep -i method
MethodTable: 5890c770
.shell: Process exited
0:004> .shell -ci "!DumpObj /d 01c72360" grep -i value
MT Field Offset Type VT Attr Value Name
5890c770 400044f 4 System.Int32 1 instance 437 m_value
.shell: Process exited
0:004> ? 1b5
Evaluate expression: 437 = 000001b5
leaving int32 apart lets dissect a widestring "stream" in x86
actualsizereqdfor(L"stream\0") = 7 * sizeof(wchar_t) == 7 * 2 == 0n14;
sizeof(method table ) == 0n04;
sizeof(sizeof(L"stream)) == 0n04;
sizeof(padding ?? terminator ?? whatever ?? ) == 0n04;
so total size == 0n26
result of dumpobj
0:004> !DumpObj /d 01c73ad0
Name: System.String
MethodTable: 5890afb0
Size: 26(0x1a) bytes
String: stream
Fields:
MT Field Offset Type VT Attr Value Name
5890c770 40000aa 4 System.Int32 1 instance 6 m_stringLength
5890b9a8 40000ab 8 System.Char 1 instance 73 m_firstChar
5890afb0 40000ac c System.String 0 shared static Empty
raw display
0:004> db 01c73ad0 l1a
01c73ad0 b0 af 90 58 06 00 00 00-73 00 74 00 72 00 65 00 ...X....s.t.r.e.
01c73ae0 61 00 6d 00 00 00 00 00-00 00 a.m.......
I've written a loop that loops from 0x00000000 to 0x7FFFFFFF, but I think this is wrong. First of it takes for ages even to reach 1% when im looping thru this interval:
for (uint adr = 0x00000000; adr <= 0x7FFFFFFF; adr++)
{
...
}
I mentioned it before, but is 0x00000000 correct start value and what about stop value? My test app that I'm trying to read is taking up 388kB, is 0x7FFFFFFF the correct value to use?
EDIT:
The testapplication im trying to read is written by myself in c++ and contains only a int, with the value of 10
Im trying to read the testapps memory with c#
My test app [...] is taking up 388kB, is 0x7FFFFFFF the correct value to use?
Those 2 numbers are totally unrelated.
And whatever you're trying to do inside the loop is probably wrong.
0x7fffffff hex is 2,147,483,647 decimal - looping more than 2 billion times might take a while ...
Scanning through memory one byte at a time can take a while.
I suggest first stepping by 4 rather than by 1, as x86 CPUs are way better at aligned memory accesses. Look at everything through the lens of four bytes at a time, and it'll doubtless run way faster than just four times faster.
Second, I suggest trying to find some mechanism to discover what the process memory map looks like. I'm not sure what the procedure is under windows; process explorer may be a good first start. Under Linux, it'd just be reading from /proc/self/maps:
$ cat /proc/self/maps
00400000-0040b000 r-xp 00000000 08:03 701781 /bin/cat
0060a000-0060b000 r--p 0000a000 08:03 701781 /bin/cat
0060b000-0060c000 rw-p 0000b000 08:03 701781 /bin/cat
006a9000-006ca000 rw-p 00000000 00:00 0 [heap]
7f46fb0ca000-7f46fb36e000 r--p 00000000 08:03 489906 /usr/lib/locale/locale-archive
7f46fb36e000-7f46fb4e8000 r-xp 00000000 08:03 228784 /lib/libc-2.12.1.so
7f46fb4e8000-7f46fb6e7000 ---p 0017a000 08:03 228784 /lib/libc-2.12.1.so
7f46fb6e7000-7f46fb6eb000 r--p 00179000 08:03 228784 /lib/libc-2.12.1.so
7f46fb6eb000-7f46fb6ec000 rw-p 0017d000 08:03 228784 /lib/libc-2.12.1.so
7f46fb6ec000-7f46fb6f1000 rw-p 00000000 00:00 0
7f46fb6f1000-7f46fb711000 r-xp 00000000 08:03 228792 /lib/ld-2.12.1.so
7f46fb8f1000-7f46fb8f4000 rw-p 00000000 00:00 0
7f46fb90f000-7f46fb911000 rw-p 00000000 00:00 0
7f46fb911000-7f46fb912000 r--p 00020000 08:03 228792 /lib/ld-2.12.1.so
7f46fb912000-7f46fb913000 rw-p 00021000 08:03 228792 /lib/ld-2.12.1.so
7f46fb913000-7f46fb914000 rw-p 00000000 00:00 0
7fffe4f02000-7fffe4f23000 rw-p 00000000 00:00 0 [stack]
7fffe4f5d000-7fffe4f5e000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
You can see in my little output that the clear majority of address space is in fact empty; as it will probably be for your application too, since two gigabytes is a huge amount of memory for simple tasks. :) Yay.
Restricting your loop to just what is actually mapped for your process would avoid inevitable segmentation violation errors when you try to access memory that isn't mapped.