In address -summary, I found inside MEM_COMMIT to 4.423 GB, but I can only get in eeheap 405618688 bytes, I passed! Heap -s can get the memory is very small.Memory.
Now I don't know, how can I investigate the remaining memory of MEM COMMIT .
!address -summary MEM_COMMIT 4.423 GB
0:000> !address -summary
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 579 7df9`74541000 ( 125.974 TB) 98.42%
<unknown> 912 206`7c9dc000 ( 2.025 TB) 99.99% 1.58%
Image 972 0`06a01000 ( 106.004 MB) 0.00% 0.00%
Heap 81 0`05a27000 ( 90.152 MB) 0.00% 0.00%
Stack 114 0`02a80000 ( 42.500 MB) 0.00% 0.00%
Other 9 0`001de000 ( 1.867 MB) 0.00% 0.00%
TEB 38 0`0004c000 ( 304.000 kB) 0.00% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_MAPPED 220 200`062b0000 ( 2.000 TB) 98.74% 1.56%
MEM_PRIVATE 935 6`7edfe000 ( 25.982 GB) 1.25% 0.02%
MEM_IMAGE 972 0`06a01000 ( 106.004 MB) 0.00% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 579 7df9`74541000 ( 125.974 TB) 98.42%
MEM_RESERVE 296 205`709f6000 ( 2.021 TB) 99.79% 1.58%
MEM_COMMIT 1831 1`1b0b9000 ( 4.423 GB) 0.21% 0.00%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 862 1`0e1cc000 ( 4.221 GB) 0.20% 0.00%
PAGE_READONLY 579 0`0659a000 ( 101.602 MB) 0.00% 0.00%
PAGE_EXECUTE_READ 159 0`04f2b000 ( 79.168 MB) 0.00% 0.00%
PAGE_NOACCESS 80 0`01550000 ( 21.312 MB) 0.00% 0.00%
PAGE_EXECUTE_READWRITE 19 0`002ba000 ( 2.727 MB) 0.00% 0.00%
PAGE_WRITECOPY 94 0`001aa000 ( 1.664 MB) 0.00% 0.00%
PAGE_READWRITE | PAGE_GUARD 38 0`00074000 ( 464.000 kB) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 21e`6c05c000 7bd5`f48f4000 ( 123.836 TB)
<unknown> 7dfd`d8ac0000 1f7`68898000 ( 1.966 TB)
Image 7ff9`71121000 0`009ba000 ( 9.727 MB)
Heap 218`83c71000 0`00eff000 ( 14.996 MB)
Stack 22`cad80000 0`0017c000 ( 1.484 MB)
Other 218`f1b50000 0`00181000 ( 1.504 MB)
TEB 22`ca410000 0`00002000 ( 8.000 kB)
PEB 22`ca40f000 0`00001000 ( 4.000 kB)
!eeheap -gc 405618688 bytes.
0:000> !eeheap -gc
Number of GC Heaps: 12
------------------------------
Heap 0 (00000218F3182A40)
generation 0 starts at 0x00000218F34C9DB0
generation 1 starts at 0x00000218F3477478
generation 2 starts at 0x00000218F3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
00000218F3460000 00000218F3461000 00000218F3A0BDC8 00000218F3F02000 0x5aadc8(5942728) 0xaa1000(11145216)
Large object heap starts at 0x0000021BF3461000
segment begin allocated committed allocated size committed size
0000021BF3460000 0000021BF3461000 0000021BF449E8D0 0000021BF44BF000 0x103d8d0(17029328) 0x105e000(17162240)
Pinned object heap starts at 0x0000021CB3461000
0000021CB3460000 0000021CB3461000 0000021CB3464FF0 0000021CB3472000 0x3ff0(16368) 0x11000(69632)
Allocated Heap Size: Size: 0x15ec688 (22988424) bytes.
Committed Heap Size: Size: 0x1aff000 (28307456) bytes.
------------------------------
Heap 1 (00000218F31AE470)
generation 0 starts at 0x00000219334C05F8
generation 1 starts at 0x0000021933471E28
generation 2 starts at 0x0000021933461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021933460000 0000021933461000 0000021933A20610 0000021933EE2000 0x5bf610(6026768) 0xa81000(11014144)
Large object heap starts at 0x0000021C03461000
segment begin allocated committed allocated size committed size
0000021C03460000 0000021C03461000 0000021C0451F4E8 0000021C04520000 0x10be4e8(17556712) 0x10bf000(17559552)
Pinned object heap starts at 0x0000021CC3461000
0000021CC3460000 0000021CC3461000 0000021CC3462030 0000021CC3472000 0x1030(4144) 0x11000(69632)
Allocated Heap Size: Size: 0x167eb28 (23587624) bytes.
Committed Heap Size: Size: 0x1b40000 (28573696) bytes.
------------------------------
Heap 2 (00000218F31DD200)
generation 0 starts at 0x00000219734967E0
generation 1 starts at 0x000002197346DFB0
generation 2 starts at 0x0000021973461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021973460000 0000021973461000 00000219734987F8 0000021973E52000 0x377f8(227320) 0x9f1000(10424320)
Large object heap starts at 0x0000021C13461000
segment begin allocated committed allocated size committed size
0000021C13460000 0000021C13461000 0000021C14B631F8 0000021C14B83000 0x17021f8(24125944) 0x1722000(24256512)
Pinned object heap starts at 0x0000021CD3461000
0000021CD3460000 0000021CD3461000 0000021CD3461018 0000021CD3462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1739a08 (24353288) bytes.
Committed Heap Size: Size: 0x2113000 (34680832) bytes.
------------------------------
Heap 3 (00000218F3208F70)
generation 0 starts at 0x00000219B34B7828
generation 1 starts at 0x00000219B3489AB8
generation 2 starts at 0x00000219B3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
00000219B3460000 00000219B3461000 00000219B3A17840 00000219B3EC2000 0x5b6840(5990464) 0xa61000(10883072)
Large object heap starts at 0x0000021C23461000
segment begin allocated committed allocated size committed size
0000021C23460000 0000021C23461000 0000021C24B43768 0000021C24B83000 0x16e2768(23996264) 0x1722000(24256512)
Pinned object heap starts at 0x0000021CE3461000
0000021CE3460000 0000021CE3461000 0000021CE3462030 0000021CE3472000 0x1030(4144) 0x11000(69632)
Allocated Heap Size: Size: 0x1c99fd8 (29990872) bytes.
Committed Heap Size: Size: 0x2183000 (35139584) bytes.
------------------------------
Heap 4 (00000218F3234CE0)
generation 0 starts at 0x00000219F34DC428
generation 1 starts at 0x00000219F34730F0
generation 2 starts at 0x00000219F3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
00000219F3460000 00000219F3461000 00000219F3BAA440 00000219F3EE2000 0x749440(7640128) 0xa81000(11014144)
Large object heap starts at 0x0000021C33461000
segment begin allocated committed allocated size committed size
0000021C33460000 0000021C33461000 0000021C3410D580 0000021C3412E000 0xcac580(13288832) 0xccd000(13422592)
Pinned object heap starts at 0x0000021CF3461000
0000021CF3460000 0000021CF3461000 0000021CF3465030 0000021CF3472000 0x4030(16432) 0x11000(69632)
Allocated Heap Size: Size: 0x13f99f0 (20945392) bytes.
Committed Heap Size: Size: 0x174e000 (24436736) bytes.
------------------------------
Heap 5 (0000021D7EDD1EF0)
generation 0 starts at 0x0000021A334F9AF8
generation 1 starts at 0x0000021A33485C40
generation 2 starts at 0x0000021A33461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021A33460000 0000021A33461000 0000021A33A59B10 0000021A33F02000 0x5f8b10(6261520) 0xaa1000(11145216)
Large object heap starts at 0x0000021C43461000
segment begin allocated committed allocated size committed size
0000021C43460000 0000021C43461000 0000021C444BCE08 0000021C4451C000 0x105be08(17153544) 0x10bb000(17543168)
Pinned object heap starts at 0x0000021D03461000
0000021D03460000 0000021D03461000 0000021D03461018 0000021D03462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1654930 (23415088) bytes.
Committed Heap Size: Size: 0x1b5c000 (28688384) bytes.
------------------------------
Heap 6 (0000021D7EDFD920)
generation 0 starts at 0x0000021A73495A38
generation 1 starts at 0x0000021A73469D68
generation 2 starts at 0x0000021A73461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021A73460000 0000021A73461000 0000021A73A15A50 0000021A73E92000 0x5b4a50(5982800) 0xa31000(10686464)
Large object heap starts at 0x0000021C53461000
segment begin allocated committed allocated size committed size
0000021C53460000 0000021C53461000 0000021C549CAFA8 0000021C549CB000 0x1569fa8(22454184) 0x156a000(22454272)
Pinned object heap starts at 0x0000021D13461000
0000021D13460000 0000021D13461000 0000021D13461018 0000021D13462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1b1ea10 (28437008) bytes.
Committed Heap Size: Size: 0x1f9b000 (33140736) bytes.
------------------------------
Heap 7 (0000021D7EE29690)
generation 0 starts at 0x0000021AB35B3700
generation 1 starts at 0x0000021AB3580548
generation 2 starts at 0x0000021AB3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021AB3460000 0000021AB3461000 0000021AB39D3718 0000021AB3FC2000 0x572718(5711640) 0xb61000(11931648)
Large object heap starts at 0x0000021C63461000
segment begin allocated committed allocated size committed size
0000021C63460000 0000021C63461000 0000021C649CCBF8 0000021C649ED000 0x156bbf8(22461432) 0x158c000(22593536)
Pinned object heap starts at 0x0000021D23461000
0000021D23460000 0000021D23461000 0000021D23461018 0000021D23462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1ade328 (28173096) bytes.
Committed Heap Size: Size: 0x20ed000 (34525184) bytes.
------------------------------
Heap 8 (0000021D7EE55400)
generation 0 starts at 0x0000021AF3496810
generation 1 starts at 0x0000021AF3472A28
generation 2 starts at 0x0000021AF3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021AF3460000 0000021AF3461000 0000021AF39F6828 0000021AF3E92000 0x595828(5855272) 0xa31000(10686464)
Large object heap starts at 0x0000021C73461000
segment begin allocated committed allocated size committed size
0000021C73460000 0000021C73461000 0000021C7492E5F8 0000021C7494F000 0x14cd5f8(21812728) 0x14ee000(21946368)
Pinned object heap starts at 0x0000021D33461000
0000021D33460000 0000021D33461000 0000021D33461830 0000021D33462000 0x830(2096) 0x1000(4096)
Allocated Heap Size: Size: 0x1a63650 (27670096) bytes.
Committed Heap Size: Size: 0x1f1f000 (32632832) bytes.
------------------------------
Heap 9 (0000021D7EE81170)
generation 0 starts at 0x0000021B334DE9E0
generation 1 starts at 0x0000021B334A73C8
generation 2 starts at 0x0000021B33461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021B33460000 0000021B33461000 0000021B33A409F8 0000021B33E82000 0x5df9f8(6158840) 0xa21000(10620928)
Large object heap starts at 0x0000021C83461000
segment begin allocated committed allocated size committed size
0000021C83460000 0000021C83461000 0000021C849E9DF8 0000021C84A0A000 0x1588df8(22580728) 0x15a9000(22712320)
Pinned object heap starts at 0x0000021D43461000
0000021D43460000 0000021D43461000 0000021D43461018 0000021D43462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1b68808 (28739592) bytes.
Committed Heap Size: Size: 0x1fca000 (33333248) bytes.
------------------------------
Heap 10 (0000021D7EEADD80)
generation 0 starts at 0x0000021B73544EC8
generation 1 starts at 0x0000021B73504C18
generation 2 starts at 0x0000021B73461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021B73460000 0000021B73461000 0000021B73AEAEE0 0000021B73F42000 0x689ee0(6856416) 0xae1000(11407360)
Large object heap starts at 0x0000021C93461000
segment begin allocated committed allocated size committed size
0000021C93460000 0000021C93461000 0000021C955DEF78 0000021C955DF000 0x217df78(35118968) 0x217e000(35119104)
Pinned object heap starts at 0x0000021D53461000
0000021D53460000 0000021D53461000 0000021D53468450 0000021D53472000 0x7450(29776) 0x11000(69632)
Allocated Heap Size: Size: 0x280f2a8 (42005160) bytes.
Committed Heap Size: Size: 0x2c5f000 (46526464) bytes.
------------------------------
Heap 11 (0000021D7EED9980)
generation 0 starts at 0x0000021BB34C6550
generation 1 starts at 0x0000021BB34784A8
generation 2 starts at 0x0000021BB3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021BB3460000 0000021BB3461000 0000021BB3906568 0000021BB3EA2000 0x4a5568(4871528) 0xa41000(10752000)
Large object heap starts at 0x0000021CA3461000
segment begin allocated committed allocated size committed size
0000021CA3460000 0000021CA3461000 0000021CA55A4DB8 0000021CA55A5000 0x2143db8(34880952) 0x2144000(34881536)
Pinned object heap starts at 0x0000021D63461000
0000021D63460000 0000021D63461000 0000021D6346CF88 0000021D63472000 0xbf88(49032) 0x11000(69632)
Allocated Heap Size: Size: 0x25f52a8 (39801512) bytes.
Committed Heap Size: Size: 0x2b85000 (45633536) bytes.
------------------------------
GC Allocated Heap Size: Size: 0x14459f90 (340107152) bytes.
GC Committed Heap Size: Size: 0x182d4000 (405618688) bytes.
!heap -s
0:000> !heap -s
************************************************************************************************************************
NT HEAP STATS BELOW
************************************************************************************************************************
LFH Key : 0xa893144b6b979747
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
00000218f1710000 00000002 65320 14712 64928 5575 163 16 2 c LFH
External fragmentation 38 % (163 free blocks)
00000218f1660000 00008000 64 4 64 2 1 1 0 0
00000218f16f0000 00001002 3516 2388 3124 200 58 3 0 0 LFH
00000218f3360000 00001002 60 8 60 2 1 1 0 0
00000218f3450000 00041002 60 8 60 5 1 1 0 0
0000021d7fa40000 00001002 3516 1460 3124 89 25 3 0 0 LFH
0000021d7fc00000 00001002 60 8 60 5 1 1 0 0
0000021d7fbc0000 00001002 452 32 60 5 2 1 0 0 LFH
-------------------------------------------------------------------------------------
github：https://github.com/hueifeng/MemoryLeak
Unfortunately I can't compile that code given in the Github repository. But what I see in the source code totally makes sense to me.
The code creates 400 objects with a link to an image and adds them to a list:
for (int i = 0; i < 400; i++)
{
list.Add(new TestDto { ImageUrl = "https://gimg2.baidu.com/..." });
}
I guess that the Excel exporter will load those images. The image in the link is 2560×1440 pixels with 24 bits color depth, so that is an uncompressed size of 2560×1440×3 = 11 MB. 400 of them is 4.42 GB.
As per this answer, Bitmaps will be allocated via VirtualAlloc() and that's what you see.
IMHO it's not a memory leak. It's exactly what you have implemented.
In this case, the dump analysis strategies of Dmitry Vostokov might be interesting. What is he doing: he's converting the memory into a bitmap and he claims that he can diagnose quite a few problems just from the look at the resulting picture. You can see examples of that on the cover on one of his books. In your case I'd expect to see at least 400 repeating patterns of the image. Maybe you want to try dump2picture just for fun.
Related
I was practicing in app memory profiling. I have an ASP.NET Core 7 MVC app running on Ubuntu. I notice that some memory wasn't released, so I've tried to use dotnet-counters monitor -p <pid> to see what's going on here and found some strange values like GC Committed Byte is huge.
I don't understand what that means. How can I interpret result to understand way for further investigation?
I couldn't find any information in the official documentation that would help me explain the meaning of this counter and how to interpret it.
dotnet-counters output example:
[System.Runtime]
% Time in GC since last GC (%) 0
Allocation Rate (B / 1 sec) 8,168
CPU Usage (%) 0.015
Exception Count (Count / 1 sec) 0
GC Committed Bytes (MB) 1,269.08
GC Fragmentation (%) 54.622
GC Heap Size (MB) 19.337
Gen 0 GC Count (Count / 1 sec) 0
Gen 0 Size (B) 0
Gen 1 GC Count (Count / 1 sec) 0
Gen 1 Size (B) 554,856
Gen 2 GC Count (Count / 1 sec) 0
Gen 2 Size (B) 27,092,392
IL Bytes Jitted (B) 2,410,959
LOH Size (B) 8,032,824
Monitor Lock Contention Count (Count / 1 sec) 0
Number of Active Timers 5
Number of Assemblies Loaded 158
Number of Methods Jitted 29,767
POH (Pinned Object Heap) Size (B) 3,765,072
ThreadPool Completed Work Item Count (Count / 1 sec) 0
ThreadPool Queue Length 0
ThreadPool Thread Count 4
Time spent in JIT (ms / 1 sec) 0
Working Set (MB) 1,552.4
I have a windows service to receive emails that uses OpenPop. However memory usage goes up to 8G about 3 days after a restart. Operation staff gives me a dump file, So I use windbg to analysis it.
When I run !address -summary I got:
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 468 7fd`942f6000 ( 7.991 TB) 99.88%
Heap 645 1`c3f95000 ( 7.062 GB) 72.92% 0.09%
<unknown> 1347 0`9305f000 ( 2.297 GB) 23.72% 0.03%
Image 1985 0`0d28d000 ( 210.551 MB) 2.12% 0.00%
Stack 366 0`077c0000 ( 119.750 MB) 1.21% 0.00%
Other 11 0`001c4000 ( 1.766 MB) 0.02% 0.00%
TEB 122 0`000f4000 ( 976.000 kB) 0.01% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 1725 2`5b3b9000 ( 9.426 GB) 97.33% 0.12%
MEM_IMAGE 2692 0`0f249000 ( 242.285 MB) 2.44% 0.00%
MEM_MAPPED 60 0`016f8000 ( 22.969 MB) 0.23% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 468 7fd`942f6000 ( 7.991 TB) 99.88%
MEM_COMMIT 3373 2`10e49000 ( 8.264 GB) 85.33% 0.10%
MEM_RESERVE 1104 0`5aeb1000 ( 1.421 GB) 14.67% 0.02%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 1528 2`0092c000 ( 8.009 GB) 82.70% 0.10%
PAGE_EXECUTE_READ 308 0`0b666000 ( 182.398 MB) 1.84% 0.00%
PAGE_READONLY 924 0`0323a000 ( 50.227 MB) 0.51% 0.00%
PAGE_WRITECOPY 321 0`012f3000 ( 18.949 MB) 0.19% 0.00%
PAGE_EXECUTE_READWRITE 112 0`005d2000 ( 5.820 MB) 0.06% 0.00%
PAGE_READWRITE|PAGE_GUARD 122 0`0023c000 ( 2.234 MB) 0.02% 0.00%
PAGE_EXECUTE_WRITECOPY 57 0`00178000 ( 1.469 MB) 0.01% 0.00%
PAGE_EXECUTE 1 0`00004000 ( 16.000 kB) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 5`2d890000 7f9`6d640000 ( 7.974 TB)
Heap 0`25d40000 0`00fd0000 ( 15.813 MB)
<unknown> 0`32e89000 0`0eb07000 ( 235.027 MB)
Image 7fe`f466a000 0`01338000 ( 19.219 MB)
Stack 0`19c60000 0`000fc000 (1008.000 kB)
Other 0`00de0000 0`00181000 ( 1.504 MB)
TEB 7ff`ffdb8000 0`00002000 ( 8.000 kB)
PEB 7ff`fffdf000 0`00001000 ( 4.000 kB)
and !eeheap -gc I got:
Number of GC Heaps: 1
generation 0 starts at 0x000000051da29040
generation 1 starts at 0x000000051d891000
generation 2 starts at 0x0000000001281000
ephemeral segment allocation context: none
segment begin allocated size
0000000001280000 0000000001281000 000000000b1eae80 0x9f69e80(167157376)
0000000031990000 0000000031991000 0000000032e68ee0 0x14d7ee0(21855968)
000000007fff0000 000000007fff1000 0000000081a99330 0x1aa8330(27951920)
000000008fff0000 000000008fff1000 0000000099d1bb00 0x9d2ab00(164801280)
000000009fff0000 000000009fff1000 00000000a77989c0 0x77a79c0(125467072)
000000051d890000 000000051d891000 000000052352fe40 0x5c9ee40(97119808)
Large object heap starts at 0x0000000011281000
segment begin allocated size
0000000011280000 0000000011281000 0000000019236c28 0x7fb5c28(133913640)
0000000048fa0000 0000000048fa1000 0000000050a87908 0x7ae6908(128870664)
0000000050fa0000 0000000050fa1000 000000005387c418 0x28db418(42841112)
00000000afff0000 00000000afff1000 00000000b3bf0840 0x3bff840(62912576)
Total Size: Size: 0x39fd2518 (972891416) bytes.
------------------------------
GC Heap Size: Size: 0x39fd2518 (972891416) bytes.
From these two commands, there is some unmanaged memory leak.
How can I find out what is in the unmanaged memory and by what method?
There is no evidence yet, that your situation is a native memory leak. Yes, you have 8 GB in native heaps. But you also have 900 MB in managed heaps which could hold the native objects alive.
I suggest using sos and starting with !dumpheap -stat if you really want to go with WinDbg. Otherwise use a .NET memory profiler. Many people have a JetBrains dotMemory license because they use R# Ultimate. It's much easier to use than WinDbg and has better support for comparing snapshots over time.
As far as I can see there is no breakdown into memory usage of individual .net components. To do this you would need to load the SOS debugger extension. The article Hunting .NET memory leaks with Windbg by André Snede Kock provides some more details.
If you are new to memory debugging I would probably also recommend a commercial tool, like dotMemory or ANTS, since they tend to be easier to use.
This should give you a list of .Net objects, how many instances are alive, and how much memory they use. I usually try to look for unexpected things, typically if there are more objects alive than I would expect, and then try to find out why. This helps with detecting problems in managed code, and I would say this is the most likely problem.
Another possibility is that there is a un-managed memory leak in some libary you use, but then you would need to contact the vendor of that library to get it fixed.
One of our PRD small win-service suddenly skyrocketed to 80+ GB memory for a few hours and then came down and stabilized at 6+ GB memory, it is usually use only under 200 MB. I have grab a memory dump at the time of 6+ GB, and restarted the server, everything go back to normal. I am investigating why it it skyrocketed to that high and why it's stabled at 6+ GB.
I have found that it is an unmanaged memory leak issue, because the CLR heap is small:
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x000000910083ff20
generation 1 starts at 0x00000091007caba0
generation 2 starts at 0x0000009100001000
ephemeral segment allocation context: none
segment begin allocated size
0000009100000000 0000009100001000 00000091029691f0 0x29681f0(43418096)
Large object heap starts at 0x0000009110001000
segment begin allocated size
0000009110000000 0000009110001000 000000911013dcb8 0x13ccb8(1297592)
Total Size: Size: 0x2aa4ea8 (44715688) bytes.
------------------------------
***GC Heap Size: Size: 0x2aa4ea8 (44715688) bytes.***
But the Heap Summary is high, More suspiciously, the lock count is high:
0:000> !heap -s
************************************************************************************************************************
NT HEAP STATS BELOW
************************************************************************************************************************
LFH Key : 0x45d65d36d8f8b642
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
**000000917aa90000 00000002 6718396 6682116 6718196 71860 5580 419 2 93 LFH**
000000917a850000 00008000 64 4 64 2 1 1 0 0
000000917acd0000 00001002 1280 88 1080 15 7 2 0 0 LFH
000000917ac90000 00001002 1280 108 1080 24 8 2 0 0 LFH
000000917b160000 00001002 1280 124 1080 7 10 2 0 0 LFH
000000917b310000 00041002 60 8 60 5 1 1 0 0
000000917bd40000 00041002 260 36 60 4 3 1 0 0 LFH
0000009118080000 00001002 1084 1024 1084 1021 2 1 0 0
-------------------------------------------------------------------------------------
0:000> !heap -stat -h 000000917aa90000
heap # 000000917aa90000
group-by: TOTSIZE max-display: 20
size #blocks total ( %) (percent of total busy bytes)
2d0b0 373e - 9b845aa0 (39.37)
10d1 41962 - 44eed902 (17.45)
ddc8 373e - 2fdbae70 (12.12)
7ab8 373e - 1a7b4090 (6.70)
7168 373e - 1878cf30 (6.20)
29d1 6e6b - 1209485b (4.57)
35d1 372d - b995cbd (2.94)
31d1 372d - abca8bd (2.72)
12d1 6e5a - 81c6b7a (2.05)
21d1 3746 - 74d2626 (1.85)
fd1 6e6c - 6d27a2c (1.73)
1cd1 372d - 635f7bd (1.57)
40 22b52 - 8ad480 (0.14)
100 6ef7 - 6ef700 (0.11)
200 374f - 6e9e00 (0.11)
c3500 5 - 3d0900 (0.06)
80 6edb - 376d80 (0.05)
d8 374e - 2ea9d0 (0.05)
249f18 1 - 249f18 (0.04)
90 3792 - 1f4220 (0.03)
Then I did !heap -flt s 2d0b0
I can't do !heap -p -a to see the stack because I don't have gflags enabled in PRD.
Instead, I am trying to do a dc 00000092b7088bb0 L200 to guess base on the string value. Nothing meaningful in the first chuck (2d0b0 373e - 9b845aa0 (39.37) But in the second chuck ( 10d1 41962 - 44eed902 (17.45)), I found something like
00000092`b70b0250 00000000 00000000 44440000 4e4f4d2d ..........DD-MON
00000092`b70b0260 0052522d 00000000 00000000 00000000 -RR.............
00000092`b70b0270 00000000 00000000 00000000 00000000 ................
00000092`b70b0280 00000000 00000000 00000000 00000000 ................
00000092`b70b0290 00000000 48480000 2e494d2e 46585353 ......HH.MI.SSXF
00000092`b70b02a0 4d412046 00000000 00000000 00000000 F AM............
00000092`b70b02b0 00000000 00000000 00000000 00000000 ................
00000092`b70b02c0 00000000 44440000 4e4f4d2d 2052522d ......DD-MON-RR
00000092`b70b02d0 4d2e4848 53532e49 20464658 00004d41 HH.MI.SSXFF AM..
00000092`b70b02e0 00000000 00000000 00000000 00000000 ................
00000092`b70b02f0 00000000 00000000 00000000 00000000 ................
00000092`b70b0300 00000000 00000000 00000000 00000000 ................
00000092`b70b0310 00000000 48480000 2e494d2e 46585353 ......HH.MI.SSXF
00000092`b70b0320 4d412046 525a5420 00000000 00000000 F AM TZR........
00000092`b70b0330 00000000 00000000 00000000 00000000 ................
00000092`b70b0340 00000000 00000000 00000000 00000000 ................
00000092`b70b0350 00000000 00000000 44440000 4e4f4d2d ..........DD-MON
00000092`b70b0360 2052522d 4d2e4848 53532e49 20464658 -RR HH.MI.SSXFF
00000092`b70b0370 54204d41 0000525a 00000000 00000000 AM TZR..........
It looks like Oracle Database Query Date Time Format string. And We do use Oracle, and I am hard to believe that we are leaking oracle connections, because this small service have run for years and this is the first time this happened.
That's how far I can go now.
Sorry for the long question and thanks for spend time reading
To summarize my question:
what is the Lock Count Column mean in !heap -s , that looks like an
indicator of an issue where I can keep digging, please point me a
direction or blog I can read.
Are those Raw data string ring another bell on you instead of Oracle? Have you ever have similar issue?
We do use a lot .net remoting. I know it's old and I never deep dived into it. Does it could be a factor of this issue?
I can't do anything in PRD, and I can't reproduce this issue in DEV/TEST, is there any other direction I can take?
I have a .NET application that uses many IDisposable types. In one of environment, application is showing high memory usage. I collected a dump file and see following.
0:000> !address -summary
Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 73 7ff`0e99f000 ( 6.112 Tb) 99.95%
Heap 35 0`d06ce000 ( 4.017 Gb) 86.34% 0.04%
<unknown> 56 0`1aebe000 ( 430.742 Mb) 11.15% 0.01%
Image 247 0`05600000 ( 86.000 Mb) 2.23% 0.00%
Stack 27 0`00900000 ( 9.000 Mb) 0.23% 0.00%
Other 9 0`001b2000 ( 1.695 Mb) 0.04% 0.00%
TEB 9 0`00012000 ( 72.000 kb) 0.00% 0.00%
PEB 1 0`00001000 ( 4.000 kb) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 116 0`ea4a6000 ( 3.661 Gb) 97.06% 0.04%
MEM_IMAGE 248 0`05601000 ( 86.004 Mb) 2.23% 0.00%
MEM_MAPPED 20 0`01baa000 ( 27.664 Mb) 0.72% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 73 7ff`0e99f000 ( 6.112 Tb) 99.95%
MEM_COMMIT 337 0`d605a000 ( 3.344 Gb) 88.66% 0.04%
MEM_RESERVE 47 0`1b5f7000 ( 437.965 Mb) 11.34% 0.01%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 120 0`d0427000 ( 4.014 Gb) 86.27% 0.04%
PAGE_EXECUTE_READ 32 0`03de6000 ( 61.898 Mb) 1.60% 0.00%
PAGE_READONLY 102 0`00fe7000 ( 15.902 Mb) 0.41% 0.00%
PAGE_WRITECOPY 44 0`00cf9000 ( 12.973 Mb) 0.34% 0.00%
PAGE_EXECUTE_WRITECOPY 10 0`0012f000 ( 1.184 Mb) 0.03% 0.00%
PAGE_READWRITE|PAGE_GUARD 9 0`00022000 ( 136.000 kb) 0.00% 0.00%
PAGE_EXECUTE_READWRITE 19 0`0001b000 ( 108.000 kb) 0.00% 0.00%
PAGE_EXECUTE 1 0`00001000 ( 4.000 kb) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 1`00061000 7fd`834df000 ( 7.990 Tb)
Heap 0`1b050000 0`10001000 ( 256.004 Mb)
<unknown> 0`025f2000 0`0ffee000 ( 255.930 Mb)
Image 7fe`e1aba000 0`0102d000 ( 16.176 Mb)
Stack 0`1ac30000 0`000fc000 (1008.000 kb)
Other 0`00830000 0`00181000 ( 1.504 Mb)
TEB 7ff`ffef8000 0`00002000 ( 8.000 kb)
PEB 7ff`fffda000 0`00001000 ( 4.000 kb)
The memory usage seem to coming out from the native side of things. The managed memory consumption doesn't seem to be significant.
0:000> !EEHeap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00000000025e1030
generation 1 starts at 0x00000000025e1018
generation 2 starts at 0x00000000025e1000
ephemeral segment allocation context: none
segment begin allocated size
00000000025e0000 00000000025e1000 00000000025effe8 0xefe8(61416)
Large object heap starts at 0x00000000125e1000
segment begin allocated size
00000000125e0000 00000000125e1000 00000000125f1940 0x10940(67904)
Total Size: Size: 0x1f928 (129320) bytes.
------------------------------
GC Heap Size: Size: 0x1f928 (129320) bytes.
What can I do to determine what objects are responsible for this much memory?
When I Read this article，I don't comprehend the statement which I bold && italic it as follow:
Conditions for a garbage collection
Garbage collection occurs when one of the following conditions is true:
The system has low physical memory.
The memory that is used by allocated objects on the managed heap surpasses an acceptable threshold. This threshold is continuously adjusted as the process runs.
The GC.Collect method is called. In almost all cases, you do
not have to call this method, because the garbage collector runs
continuously. This method is primarily used for unique situations
and testing.
But there is another limit(ephemeral segment value) for GC's Ephemeral generations.
|--------------------------------|32-bit |64-bit
|Workstation GC | 16 MB |256 MB
|Server GC |64 MB |4 GB
|Server GC with > 4 logical CPUs |32 MB |2 GB
|Server GC with > 8 logical CPUs |16 MB |1 GB
=======================================================================
Question
I want to know:
is there a relation between the GC's ephemeral segment value Value
&& managed heap threshold value?
The managed heap threshold Value is ?(max & min)
Is there a detail for the threshold adjust algorithm?
=======================================================================
Reference
GC on msdn