I was practicing in app memory profiling. I have an ASP.NET Core 7 MVC app running on Ubuntu. I notice that some memory wasn't released, so I've tried to use dotnet-counters monitor -p <pid> to see what's going on here and found some strange values like GC Committed Byte is huge.
I don't understand what that means. How can I interpret result to understand way for further investigation?
I couldn't find any information in the official documentation that would help me explain the meaning of this counter and how to interpret it.
dotnet-counters output example:
[System.Runtime]
% Time in GC since last GC (%) 0
Allocation Rate (B / 1 sec) 8,168
CPU Usage (%) 0.015
Exception Count (Count / 1 sec) 0
GC Committed Bytes (MB) 1,269.08
GC Fragmentation (%) 54.622
GC Heap Size (MB) 19.337
Gen 0 GC Count (Count / 1 sec) 0
Gen 0 Size (B) 0
Gen 1 GC Count (Count / 1 sec) 0
Gen 1 Size (B) 554,856
Gen 2 GC Count (Count / 1 sec) 0
Gen 2 Size (B) 27,092,392
IL Bytes Jitted (B) 2,410,959
LOH Size (B) 8,032,824
Monitor Lock Contention Count (Count / 1 sec) 0
Number of Active Timers 5
Number of Assemblies Loaded 158
Number of Methods Jitted 29,767
POH (Pinned Object Heap) Size (B) 3,765,072
ThreadPool Completed Work Item Count (Count / 1 sec) 0
ThreadPool Queue Length 0
ThreadPool Thread Count 4
Time spent in JIT (ms / 1 sec) 0
Working Set (MB) 1,552.4
Related
In address -summary, I found inside MEM_COMMIT to 4.423 GB, but I can only get in eeheap 405618688 bytes, I passed! Heap -s can get the memory is very small.Memory.
Now I don't know, how can I investigate the remaining memory of MEM COMMIT .
!address -summary MEM_COMMIT 4.423 GB
0:000> !address -summary
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 579 7df9`74541000 ( 125.974 TB) 98.42%
<unknown> 912 206`7c9dc000 ( 2.025 TB) 99.99% 1.58%
Image 972 0`06a01000 ( 106.004 MB) 0.00% 0.00%
Heap 81 0`05a27000 ( 90.152 MB) 0.00% 0.00%
Stack 114 0`02a80000 ( 42.500 MB) 0.00% 0.00%
Other 9 0`001de000 ( 1.867 MB) 0.00% 0.00%
TEB 38 0`0004c000 ( 304.000 kB) 0.00% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_MAPPED 220 200`062b0000 ( 2.000 TB) 98.74% 1.56%
MEM_PRIVATE 935 6`7edfe000 ( 25.982 GB) 1.25% 0.02%
MEM_IMAGE 972 0`06a01000 ( 106.004 MB) 0.00% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 579 7df9`74541000 ( 125.974 TB) 98.42%
MEM_RESERVE 296 205`709f6000 ( 2.021 TB) 99.79% 1.58%
MEM_COMMIT 1831 1`1b0b9000 ( 4.423 GB) 0.21% 0.00%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 862 1`0e1cc000 ( 4.221 GB) 0.20% 0.00%
PAGE_READONLY 579 0`0659a000 ( 101.602 MB) 0.00% 0.00%
PAGE_EXECUTE_READ 159 0`04f2b000 ( 79.168 MB) 0.00% 0.00%
PAGE_NOACCESS 80 0`01550000 ( 21.312 MB) 0.00% 0.00%
PAGE_EXECUTE_READWRITE 19 0`002ba000 ( 2.727 MB) 0.00% 0.00%
PAGE_WRITECOPY 94 0`001aa000 ( 1.664 MB) 0.00% 0.00%
PAGE_READWRITE | PAGE_GUARD 38 0`00074000 ( 464.000 kB) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 21e`6c05c000 7bd5`f48f4000 ( 123.836 TB)
<unknown> 7dfd`d8ac0000 1f7`68898000 ( 1.966 TB)
Image 7ff9`71121000 0`009ba000 ( 9.727 MB)
Heap 218`83c71000 0`00eff000 ( 14.996 MB)
Stack 22`cad80000 0`0017c000 ( 1.484 MB)
Other 218`f1b50000 0`00181000 ( 1.504 MB)
TEB 22`ca410000 0`00002000 ( 8.000 kB)
PEB 22`ca40f000 0`00001000 ( 4.000 kB)
!eeheap -gc 405618688 bytes.
0:000> !eeheap -gc
Number of GC Heaps: 12
------------------------------
Heap 0 (00000218F3182A40)
generation 0 starts at 0x00000218F34C9DB0
generation 1 starts at 0x00000218F3477478
generation 2 starts at 0x00000218F3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
00000218F3460000 00000218F3461000 00000218F3A0BDC8 00000218F3F02000 0x5aadc8(5942728) 0xaa1000(11145216)
Large object heap starts at 0x0000021BF3461000
segment begin allocated committed allocated size committed size
0000021BF3460000 0000021BF3461000 0000021BF449E8D0 0000021BF44BF000 0x103d8d0(17029328) 0x105e000(17162240)
Pinned object heap starts at 0x0000021CB3461000
0000021CB3460000 0000021CB3461000 0000021CB3464FF0 0000021CB3472000 0x3ff0(16368) 0x11000(69632)
Allocated Heap Size: Size: 0x15ec688 (22988424) bytes.
Committed Heap Size: Size: 0x1aff000 (28307456) bytes.
------------------------------
Heap 1 (00000218F31AE470)
generation 0 starts at 0x00000219334C05F8
generation 1 starts at 0x0000021933471E28
generation 2 starts at 0x0000021933461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021933460000 0000021933461000 0000021933A20610 0000021933EE2000 0x5bf610(6026768) 0xa81000(11014144)
Large object heap starts at 0x0000021C03461000
segment begin allocated committed allocated size committed size
0000021C03460000 0000021C03461000 0000021C0451F4E8 0000021C04520000 0x10be4e8(17556712) 0x10bf000(17559552)
Pinned object heap starts at 0x0000021CC3461000
0000021CC3460000 0000021CC3461000 0000021CC3462030 0000021CC3472000 0x1030(4144) 0x11000(69632)
Allocated Heap Size: Size: 0x167eb28 (23587624) bytes.
Committed Heap Size: Size: 0x1b40000 (28573696) bytes.
------------------------------
Heap 2 (00000218F31DD200)
generation 0 starts at 0x00000219734967E0
generation 1 starts at 0x000002197346DFB0
generation 2 starts at 0x0000021973461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021973460000 0000021973461000 00000219734987F8 0000021973E52000 0x377f8(227320) 0x9f1000(10424320)
Large object heap starts at 0x0000021C13461000
segment begin allocated committed allocated size committed size
0000021C13460000 0000021C13461000 0000021C14B631F8 0000021C14B83000 0x17021f8(24125944) 0x1722000(24256512)
Pinned object heap starts at 0x0000021CD3461000
0000021CD3460000 0000021CD3461000 0000021CD3461018 0000021CD3462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1739a08 (24353288) bytes.
Committed Heap Size: Size: 0x2113000 (34680832) bytes.
------------------------------
Heap 3 (00000218F3208F70)
generation 0 starts at 0x00000219B34B7828
generation 1 starts at 0x00000219B3489AB8
generation 2 starts at 0x00000219B3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
00000219B3460000 00000219B3461000 00000219B3A17840 00000219B3EC2000 0x5b6840(5990464) 0xa61000(10883072)
Large object heap starts at 0x0000021C23461000
segment begin allocated committed allocated size committed size
0000021C23460000 0000021C23461000 0000021C24B43768 0000021C24B83000 0x16e2768(23996264) 0x1722000(24256512)
Pinned object heap starts at 0x0000021CE3461000
0000021CE3460000 0000021CE3461000 0000021CE3462030 0000021CE3472000 0x1030(4144) 0x11000(69632)
Allocated Heap Size: Size: 0x1c99fd8 (29990872) bytes.
Committed Heap Size: Size: 0x2183000 (35139584) bytes.
------------------------------
Heap 4 (00000218F3234CE0)
generation 0 starts at 0x00000219F34DC428
generation 1 starts at 0x00000219F34730F0
generation 2 starts at 0x00000219F3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
00000219F3460000 00000219F3461000 00000219F3BAA440 00000219F3EE2000 0x749440(7640128) 0xa81000(11014144)
Large object heap starts at 0x0000021C33461000
segment begin allocated committed allocated size committed size
0000021C33460000 0000021C33461000 0000021C3410D580 0000021C3412E000 0xcac580(13288832) 0xccd000(13422592)
Pinned object heap starts at 0x0000021CF3461000
0000021CF3460000 0000021CF3461000 0000021CF3465030 0000021CF3472000 0x4030(16432) 0x11000(69632)
Allocated Heap Size: Size: 0x13f99f0 (20945392) bytes.
Committed Heap Size: Size: 0x174e000 (24436736) bytes.
------------------------------
Heap 5 (0000021D7EDD1EF0)
generation 0 starts at 0x0000021A334F9AF8
generation 1 starts at 0x0000021A33485C40
generation 2 starts at 0x0000021A33461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021A33460000 0000021A33461000 0000021A33A59B10 0000021A33F02000 0x5f8b10(6261520) 0xaa1000(11145216)
Large object heap starts at 0x0000021C43461000
segment begin allocated committed allocated size committed size
0000021C43460000 0000021C43461000 0000021C444BCE08 0000021C4451C000 0x105be08(17153544) 0x10bb000(17543168)
Pinned object heap starts at 0x0000021D03461000
0000021D03460000 0000021D03461000 0000021D03461018 0000021D03462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1654930 (23415088) bytes.
Committed Heap Size: Size: 0x1b5c000 (28688384) bytes.
------------------------------
Heap 6 (0000021D7EDFD920)
generation 0 starts at 0x0000021A73495A38
generation 1 starts at 0x0000021A73469D68
generation 2 starts at 0x0000021A73461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021A73460000 0000021A73461000 0000021A73A15A50 0000021A73E92000 0x5b4a50(5982800) 0xa31000(10686464)
Large object heap starts at 0x0000021C53461000
segment begin allocated committed allocated size committed size
0000021C53460000 0000021C53461000 0000021C549CAFA8 0000021C549CB000 0x1569fa8(22454184) 0x156a000(22454272)
Pinned object heap starts at 0x0000021D13461000
0000021D13460000 0000021D13461000 0000021D13461018 0000021D13462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1b1ea10 (28437008) bytes.
Committed Heap Size: Size: 0x1f9b000 (33140736) bytes.
------------------------------
Heap 7 (0000021D7EE29690)
generation 0 starts at 0x0000021AB35B3700
generation 1 starts at 0x0000021AB3580548
generation 2 starts at 0x0000021AB3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021AB3460000 0000021AB3461000 0000021AB39D3718 0000021AB3FC2000 0x572718(5711640) 0xb61000(11931648)
Large object heap starts at 0x0000021C63461000
segment begin allocated committed allocated size committed size
0000021C63460000 0000021C63461000 0000021C649CCBF8 0000021C649ED000 0x156bbf8(22461432) 0x158c000(22593536)
Pinned object heap starts at 0x0000021D23461000
0000021D23460000 0000021D23461000 0000021D23461018 0000021D23462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1ade328 (28173096) bytes.
Committed Heap Size: Size: 0x20ed000 (34525184) bytes.
------------------------------
Heap 8 (0000021D7EE55400)
generation 0 starts at 0x0000021AF3496810
generation 1 starts at 0x0000021AF3472A28
generation 2 starts at 0x0000021AF3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021AF3460000 0000021AF3461000 0000021AF39F6828 0000021AF3E92000 0x595828(5855272) 0xa31000(10686464)
Large object heap starts at 0x0000021C73461000
segment begin allocated committed allocated size committed size
0000021C73460000 0000021C73461000 0000021C7492E5F8 0000021C7494F000 0x14cd5f8(21812728) 0x14ee000(21946368)
Pinned object heap starts at 0x0000021D33461000
0000021D33460000 0000021D33461000 0000021D33461830 0000021D33462000 0x830(2096) 0x1000(4096)
Allocated Heap Size: Size: 0x1a63650 (27670096) bytes.
Committed Heap Size: Size: 0x1f1f000 (32632832) bytes.
------------------------------
Heap 9 (0000021D7EE81170)
generation 0 starts at 0x0000021B334DE9E0
generation 1 starts at 0x0000021B334A73C8
generation 2 starts at 0x0000021B33461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021B33460000 0000021B33461000 0000021B33A409F8 0000021B33E82000 0x5df9f8(6158840) 0xa21000(10620928)
Large object heap starts at 0x0000021C83461000
segment begin allocated committed allocated size committed size
0000021C83460000 0000021C83461000 0000021C849E9DF8 0000021C84A0A000 0x1588df8(22580728) 0x15a9000(22712320)
Pinned object heap starts at 0x0000021D43461000
0000021D43460000 0000021D43461000 0000021D43461018 0000021D43462000 0x18(24) 0x1000(4096)
Allocated Heap Size: Size: 0x1b68808 (28739592) bytes.
Committed Heap Size: Size: 0x1fca000 (33333248) bytes.
------------------------------
Heap 10 (0000021D7EEADD80)
generation 0 starts at 0x0000021B73544EC8
generation 1 starts at 0x0000021B73504C18
generation 2 starts at 0x0000021B73461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021B73460000 0000021B73461000 0000021B73AEAEE0 0000021B73F42000 0x689ee0(6856416) 0xae1000(11407360)
Large object heap starts at 0x0000021C93461000
segment begin allocated committed allocated size committed size
0000021C93460000 0000021C93461000 0000021C955DEF78 0000021C955DF000 0x217df78(35118968) 0x217e000(35119104)
Pinned object heap starts at 0x0000021D53461000
0000021D53460000 0000021D53461000 0000021D53468450 0000021D53472000 0x7450(29776) 0x11000(69632)
Allocated Heap Size: Size: 0x280f2a8 (42005160) bytes.
Committed Heap Size: Size: 0x2c5f000 (46526464) bytes.
------------------------------
Heap 11 (0000021D7EED9980)
generation 0 starts at 0x0000021BB34C6550
generation 1 starts at 0x0000021BB34784A8
generation 2 starts at 0x0000021BB3461000
ephemeral segment allocation context: none
segment begin allocated committed allocated size committed size
0000021BB3460000 0000021BB3461000 0000021BB3906568 0000021BB3EA2000 0x4a5568(4871528) 0xa41000(10752000)
Large object heap starts at 0x0000021CA3461000
segment begin allocated committed allocated size committed size
0000021CA3460000 0000021CA3461000 0000021CA55A4DB8 0000021CA55A5000 0x2143db8(34880952) 0x2144000(34881536)
Pinned object heap starts at 0x0000021D63461000
0000021D63460000 0000021D63461000 0000021D6346CF88 0000021D63472000 0xbf88(49032) 0x11000(69632)
Allocated Heap Size: Size: 0x25f52a8 (39801512) bytes.
Committed Heap Size: Size: 0x2b85000 (45633536) bytes.
------------------------------
GC Allocated Heap Size: Size: 0x14459f90 (340107152) bytes.
GC Committed Heap Size: Size: 0x182d4000 (405618688) bytes.
!heap -s
0:000> !heap -s
************************************************************************************************************************
NT HEAP STATS BELOW
************************************************************************************************************************
LFH Key : 0xa893144b6b979747
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
00000218f1710000 00000002 65320 14712 64928 5575 163 16 2 c LFH
External fragmentation 38 % (163 free blocks)
00000218f1660000 00008000 64 4 64 2 1 1 0 0
00000218f16f0000 00001002 3516 2388 3124 200 58 3 0 0 LFH
00000218f3360000 00001002 60 8 60 2 1 1 0 0
00000218f3450000 00041002 60 8 60 5 1 1 0 0
0000021d7fa40000 00001002 3516 1460 3124 89 25 3 0 0 LFH
0000021d7fc00000 00001002 60 8 60 5 1 1 0 0
0000021d7fbc0000 00001002 452 32 60 5 2 1 0 0 LFH
-------------------------------------------------------------------------------------
github：https://github.com/hueifeng/MemoryLeak
Unfortunately I can't compile that code given in the Github repository. But what I see in the source code totally makes sense to me.
The code creates 400 objects with a link to an image and adds them to a list:
for (int i = 0; i < 400; i++)
{
list.Add(new TestDto { ImageUrl = "https://gimg2.baidu.com/..." });
}
I guess that the Excel exporter will load those images. The image in the link is 2560×1440 pixels with 24 bits color depth, so that is an uncompressed size of 2560×1440×3 = 11 MB. 400 of them is 4.42 GB.
As per this answer, Bitmaps will be allocated via VirtualAlloc() and that's what you see.
IMHO it's not a memory leak. It's exactly what you have implemented.
In this case, the dump analysis strategies of Dmitry Vostokov might be interesting. What is he doing: he's converting the memory into a bitmap and he claims that he can diagnose quite a few problems just from the look at the resulting picture. You can see examples of that on the cover on one of his books. In your case I'd expect to see at least 400 repeating patterns of the image. Maybe you want to try dump2picture just for fun.
My apologies if this has been asked/answered before but I'm honestly not even sure how to word this as a question properly. I have the following bit pattern:
0110110110110110110110110110110110110110110110110110110110110110
I'm trying to perform a shift that'll preserve my underlying pattern; my first instinct was to use right rotation ((x >> count) | (x << (-count & 63))) but the asymmetry in my bit pattern results in:
0011011011011011011011011011011011011011011011011011011011011011 <--- wrong
The problem is that the most significant (far left) bit ends up being 0 instead of the desired 1:
1011011011011011011011011011011011011011011011011011011011011011 <--- right
Is there a colloquial name for this function I'm looking for? If not, how could I go about implementing this idea?
Additional Information:
While the question is language agnostic I'm currently trying to solve this using C#.
The bit patterns I'm using are entirely predictable and always have the same structure; the pattern starts with a single zero followed by n - 1 ones (where n is an odd number) and then repeats infinitely.
I'd like to accomplish this without conditional operations since they'd defeat the purpose of using bitwise manipulation in the first place but maybe I have no choice...
You've got a number structured like this:
B16 B15 B14 B13 B12 B11 B10 B09 B08 B07 B06 B05 B04 B03 B02 B01 B00
? 0 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0
The ? needs to appear in the MSB (B15, or B63, or whatever) after the shift. Where does it come from? Well, the closest copy is found n places to the right:
B13 0 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0
^--------------/
If your word has width w, this is 1 << (w-n)
*
So you can do:
var selector = 1 << (w-n);
var rotated = (val >> 1) | ((val & selector) << (n-1));
But you may want a multiple shift. Then we need to build a wider mask:
? 0 1 1 0 1 1 0 1 1 0 1 1 0 1 1 0
* * * * *
Here I've chosen to pretend n = 6, it just needs to be a multiple of the basic n, and larger than shift. Now:
var selector = ((1UL << shift) - 1) << (w - n);
var rotated = (val >> shift) | ((val & selector) << (n - shift));
Working demonstration using your pattern: http://rextester.com/UWYSW47054
It's easy to see that the output has period 3, as required:
1:B6DB6DB6DB6DB6DB
2:DB6DB6DB6DB6DB6D
3:6DB6DB6DB6DB6DB6
4:B6DB6DB6DB6DB6DB
5:DB6DB6DB6DB6DB6D
6:6DB6DB6DB6DB6DB6
7:B6DB6DB6DB6DB6DB
8:DB6DB6DB6DB6DB6D
9:6DB6DB6DB6DB6DB6
10:B6DB6DB6DB6DB6DB
11:DB6DB6DB6DB6DB6D
Instead of storing a lot of repetitions of a pattern, just store one recurrence and apply modulo operations on the indexes
byte[] pattern = new byte[] { 0, 1, 1 };
// Get a "bit" at index "i", shifted right by "shift"
byte bit = pattern[(i - shift + 1000000 * byte.Length) % byte.Length];
The + 1000000 * byte.Length must be greater than the greatest expected shift and ensures that we get a posistive sum.
This allows you to store patterns of virtually any length.
An optimization would be to store a mirrored version of the pattern. You could then shift left instead of right. This would simplify the index calculation
byte bit = pattern[(i + shift) % byte.Length];
Branchless Answer after a poke by #BenVoigt:
Get the last bit b by doing (n & 1);
Return n >> 1 | b << ((sizeof(n) - 1).
Original Answer:
Get the last bit b by doing (n & 1);
If b is 1, right shift the number by 1 bit and bitwise-OR it with 1 << (sizeof(n) - 1);
If b is 0, just right shift the number by 1 bit.
The problem was changed a bit through the comments.
For all reasonable n, the following problem can be solved efficiently after minimal pre-computation:
Given an offset k, get 64 bits starting at that position in the stream of bits that follows the pattern of (zero, n-1 ones) repeating.
Clearly the pattern repeats with a period of n, so only n different ulongs have to be produced for every given value of n. That could either be done explicitly, constructing all of them in pre-processing (they could be constructed in any obvious way, it doesn't really matter since that only happens once), or left more implicitly by storing only two ulongs per value for n (this works under the assumption that n < 64, see below) and then extracting a range from them with some shifting/ORing. Either way, use offset % n to compute which pattern to retrieve (since the offset is increasing in a predictable manner, no actual modulo operation is required[1]).
Even with the first method, memory consumption will be reasonable since this optimization is only an optimization for low n: in particular for n > 64 there will be fewer than 1 zero per word on average, so the "old fashioned way" of visiting every multiple of n and resetting that bit starts to skip work while the above trick would still visit every word and would not be able anymore to reset multiple bits at once.
[1]: if there are multiple n's in play at the same time, a possible strategy is keeping an array offsets where offsets[n] = offset % n, which could be updated according to: (not tested)
int next = offsets[n] + _64modn[n]; // 64 % n precomputed
offsets[n] = next - (((n - next - 1) >> 31) & n);
The idea being that n is subtracted whenever next >= n. Only one subtraction is needed since the offset and thing added to the offset are already reduced modulo n.
This offset-increment can be done with System.Numerics.Vectors, which is very feature-poor compared to actual hardware but is just about able to do this. It can't do the shift (yes, it's weird) but it can implement a comparison in a branchless way.
Doing one pass per value of n is easier, but touches lots of memory in a cache unfriendly manner. Doing lots of different n at the same time may not be great either. I guess you'd just have to bechmark that..
Also you could consider hard-coding it for some low numbers, something like offset % 3 is fairly efficient (unlike offset % variable). This does take manual loop-unrolling which is a bit annoying, but it's actually simpler, just big in terms of lines of code.
I need help with how this works. Microsoft's site doesn't have an answer so I was hoping there is a way to deal with this.
What would
stopwatch.ElapsedMilliseconds % 120000 == 0
mean?
It is in an if statement but I need to understand what is meant by that. Does it mean every 120 seconds (2 minutes), it will equal 0 and restart?
1000 milliseconds = 1 second and therefore 120 000 milliseconds = 2 minutes
% (Mod operator), means it returns the remainder of stopwatch.ElapsedMilliseconds divided by 120000.
== 0 means you're checking for a remainder of zero.
So therefor your statement is true when elapsed time is exactly divisible by 2 minutes (i.e. exactly 2 mins, 4 mins, 6 mins etc.), otherwise it will return false
The Modulus Operator (%) operator computes the remainder after dividing its first operand by its second. All numeric types have predefined remainder operators.
stopwatch.ElapsedMilliseconds % 120000 == 0 means, ElapsedMilliseconds is multiplications of 120000 (which is 2 minutes). So, your if condition hits for every 2 mins.
stopwatch.ElapsedMilliseconds % 120000 == 0
this statement is checking condition on milisecond elapsed in stopwatch and is it completely divisible by 120000, if it is then ok else there might be some other condition also
When I Read this article，I don't comprehend the statement which I bold && italic it as follow:
Conditions for a garbage collection
Garbage collection occurs when one of the following conditions is true:
The system has low physical memory.
The memory that is used by allocated objects on the managed heap surpasses an acceptable threshold. This threshold is continuously adjusted as the process runs.
The GC.Collect method is called. In almost all cases, you do
not have to call this method, because the garbage collector runs
continuously. This method is primarily used for unique situations
and testing.
But there is another limit(ephemeral segment value) for GC's Ephemeral generations.
|--------------------------------|32-bit |64-bit
|Workstation GC | 16 MB |256 MB
|Server GC |64 MB |4 GB
|Server GC with > 4 logical CPUs |32 MB |2 GB
|Server GC with > 8 logical CPUs |16 MB |1 GB
=======================================================================
Question
I want to know:
is there a relation between the GC's ephemeral segment value Value
&& managed heap threshold value?
The managed heap threshold Value is ?(max & min)
Is there a detail for the threshold adjust algorithm?
=======================================================================
Reference
GC on msdn
We have an application written in .Net/C# 3.5 Compact Framework on an instrument.
We are using Windows Compact Embedded 7 operating system.
This application in a large sense reads some data from the device and stores it in files on disk or SD Card and displays
the data in graphical/numerical fashion.
To read the data we are calling the driver by passing a byte array in the following manner.
[System.Runtime.InteropServices.DllImport("coredll.dll")]
private static unsafe extern bool ReadFile(
IntPtr hFile,
byte* lpBuffer,
uint nNumberOfBytesToRead,
uint* lpNumberOfBytesRead,
uint lpOverlapped);
public int Read(byte[] buffer, int index, int count)
{
uint temp;
unsafe
{
fixed (byte* pByte = &(buffer[index]))
{
if (!ReadFile(handle, pByte, (uint)count, &temp, 0))
{
return -1;
}
}
}
return (int)temp;
}
And our read method which is running in a separate thread looks like this.
private void Reader()
{
try
{
while(true)
{
byte[] data = new byte[1024*10];
Read(data,0,data.Length);
// Do something with data
}
}
catch(Exception ex)
{
}
}
Our requirement is that the application is supposed to run for months continuously.
I have enabled the performance monitor on the instrumnet and when I run the app, stop it and see the statistics file generated by the perf monitor, i see we have millions of pinned objects which may eventually end up in memory leak.
Here is the stat file.
counter total last datum n mean min max
Total Program Run Time (ms) 60240358 - - - - -
App Domains Created 1 - - - - -
App Domains Unloaded 2 - - - - -
Assemblies Loaded 16 - - - - -
Classes Loaded 2466 - - - - -
Methods Loaded 9685 - - - - -
Closed Types Loaded 598 - - - - -
Closed Types Loaded per Definition 598 1 61 9 1 97
Open Types Loaded 16 - - - - -
Closed Methods Loaded 184 - - - - -
Closed Methods Loaded per Definition 184 3 55 3 1 13
Open Methods Loaded 1 - - - - -
Threads in Thread Pool - 9 19 6 1 9
Pending Timers - 0 1986882 0 0 4
Scheduled Timers 662294 - - - - -
Timers Delayed by Thread Pool Limit 0 - - - - -
Work Items Queued 662294 - - - - -
Uncontested Monitor.Enter Calls 30435505 - - - - -
Contested Monitor.Enter Calls 72 - - - - -
Peak Bytes Allocated (native + managed) 10869964 - - - - -
Managed Objects Allocated 386020616 - - - - -
Managed Bytes Allocated 25202701076 16 386022691 65 8 1048588
Managed String Objects Allocated 39325752 - - - - -
Bytes of String Objects Allocated 3041772660 - - - - -
Garbage Collections (GC) 24644 - - - - -
Bytes Collected By GC 25218571932 936288 24644 1023314 49132 1528716
Managed Bytes In Use After GC - 6162292 24644 5988047 235620 6162292
Total Bytes In Use After GC - 10729796 24644 10045775 2113544 10729796
GC Compactions 24640 - - - - -
Code Pitchings 1 - - - - -
Calls to GC.Collect 0 - - - - -
GC Latency Time (ms) 532689 23 24644 21 4 103
**Pinned Objects 846589** - - - - -
Objects Moved by Compactor 15728750 - - - - -
Objects Not Moved by Compactor 261474730 - - - - -
Objects Finalized 14917931 - - - - -
Objects on Finalizer Queue - 0 14944139 303 0 2546
Boxed Value Types 17838485 - - - - -
Process Heap - 5968 101947181 156314 336 209656
Short Term Heap - 0 4050880 51 0 79952
JIT Heap - 0 27236 895917 0 1861844
App Domain Heap - 1536 29669 960504 1536 1297080
GC Heap - 0 208 4088477 0 7512064
Native Bytes Jitted 3495028 136 8431 414 84 111248
Methods Jitted 8431 - - - - -
Bytes Pitched 1738556 164 4269 407 72 111248
Methods Pitched 4269 - - - - -
Method Pitch Latency Time (ms) 78 78 1 78 78 78
Exceptions Thrown 171 - - - - -
Platform Invoke Calls 9528151 - - - - -
COM Calls Using a vtable 0 - - - - -
COM Calls Using IDispatch 0 - - - - -
Complex Marshaling 1929605 - - - - -
Runtime Callable Wrappers 0 - - - - -
Socket Bytes Sent 784 - - - - -
Socket Bytes Received 1934 - - - - -
Controls Created 255 - - - - -
Brushes Created 3022728 - - - - -
Pens Created 1888616 - - - - -
Bitmaps Created 166286 - - - - -
Regions Created 19 - - - - -
Fonts Created 31 - - - - -
Graphics Created (FromImage) 0 - - - - -
Graphics Created (CreateGraphics) 23 - - - - -
After reading on internet I found/understood that in the Read method we will have a pinned object but it should be unpinned again when we go out of the Read method.
We don't have any other code apart from the above code which is interacting to unmanaged environment.
I am looking for advices to fix this problem of pinned objects, is there any other way/tool which can help us in finding what are those pinned objects exactly are.
Is there any other better way to do this kind of managed/unmanaged communication ?
I think that the counter is referred to the number of object the system finds in the pinned state during a GC. If the read call can take a long time this may happen quite often. On the other side I think you can easily reduce the number of memory allocation performed by your app by simply moving the allocation of the data buffer outside the while loop. If it's size is fixed (as it seems from the code) or, at least, it's easy to determine a maximum size, re-allocating it at each iteration doesn't make sense. You mark the old one for gc and take new memory. You'll reach max memory threshold and gc will start. I don't know how long your app has been running, but allocating 386 million objecs looks like quite a big number. Avoid useless re-allocation (like the one in the code you posted) and using StringBuffers instead of strings, for example, can reduce the number of memory allocations, prevent the GC from running too often and, in general, improve your code efficence.