Develop Reference

WeakReferences are not freed in embedded OS

I've got a strange behavior here:
I get a massive memory leak in production running a WPF application that runs on a DLOG-Terminal (Windows Embedded Standard SP1) that behaves perfectly fine if I run it localy on a normal desktop (Win7 prof.)
After many unsucessful attempts to find any problem I put one of those directly beside my monitor, installed the ANTs MemoryProfiler and did one hour test run simulating user operations on both the terminal and my development PC.
Result is, that due to some strange reasons the embedded system piles up a huge amount of WeakReference and EffectiveValueEntry[] Objects.
Here are are some pictures:
Development (PC):
And the terminal:
Just look at the class list...
Has anyone seen something like this before and are there known solutions to this?
Where can I get help?
(PS the terminals where installed with images prepared for .net4)
PPS: for the close-voter: I think the question is clear: how can I fix this.
You could argue if this is a IT/OS problem vs. a programming problem but I think if I post this in Server Fault it will get a off-topic close in no time...
UPDATE:
I was able to find a big portion of the problem - but it feels a bit like C++:
I use a ViewModel-like Items class for a WPF-List that provides (among others) a ICommand (RelayCommand-pattern). The Items where created on the fly in the getter of a ViewModel-Property for the view and it seems that the application/GC did never free those unused commands - or the subscribtions to their CanExecuteChanged - the memory profiler shows those as "held by a weak reference". I changed my code to reuse those item-viewmodels and Dispose/set to null every used properties in their Dispose and use this too as clean up - as I said: feels like "delete" in those old C++ days.
On top of this I use a forced GC.Collect every 30mins (yeah I know - you never should - but I got no other solution till now).
With this setup the applications runs for 6+ hours without problems so far but it don't feel right.
I cannot understand why those WeakReferences are not claimed as they are on my desktop machine...
Any thoughts on this? Please!
UPDATE:
I am still not able to pin down this problem but I see a strange behavior:
If I use PC-Anywhere to observe the operation of my software on one of the terminals the problem goes away!
Even after running 8hr. straight the software runs as it should - it will even free memory (I put a little memorycounter-display in the main-screen - let's say I connect to the terminal and see that memory is low - after waiting a few minutes the memory is reclaimed)
So I think Devin (one Answer below) has a lead in the right direction - something in the Remote-Control software unblocks the finalizer-thread or whatever is blocking the GC - be it the simulated keyboard/mouse or whatever.
Any thoughts on this?

We had a (somewhat) similar issue running my app on a tablet. The memory would be reclaimed when run on a desktop, but not when run on a tablet or some other device that used a PC Input panel. The problem is that the finalization queue was getting stuck. The COM object finalizer was waiting to run something on the main thread, which didn't have a message loop.
The solution was to find an adequate time to invoke Application.DoEvents(). We had a method that would be called intermittently and we invoke it with every 10th call. I don't know if this is the same issue you are having, but maybe it can shed some light.
EDIT: I do need to make it clear, in general, calling DoEvents() is a bad idea. It works in that case because there isn't any UI on that thread or anything else happening that those events can interfere with.

From the screenshots it is interesting to see that the LOH grows at the same time the used space does not grow much. The free space is growing a lot at the LOH which indicates memory fragmentation due to pinned objects. This looks like a stuck finalizer thread which does prevent the cleanup of managed objects. You should get a memory dump and check in which method the finalizer thread was stuck. You can do this quite easy with Windbg.

What do I do When Dispose Method Fails?

Well, here I am again with another frustrating question.
I need to end my main process and restart it, but I can't just end the application gracefully...
I am using a C# application in conjunction with proprietary (not to me) data capture hardware, so right there it's already complicated.
There is a scenario when my software is happily running, collecting its data as it should, when the hardware I'm interfacing with suddenly loses power and the connection to my application. My application eventually figures this out and I just need to dispose of my old connection, and make a new one to connect to my hardware again. Wrong...
Of course, the .Dispose() method on my object (the interface object with the hardware), that terminates the connection does nothing, and actually just locks in place forever when I try to run it. Apparently there is some kind of communication that never times out on the dispose method that requires the device to be connected when the disconnection happens. I didn't write the method, so I don't really know.
Finally, here's my question. The only way to get my application up and running again is to close it and reopen it. Of course, I can't actually close it nicely because I can't run the Dispose method. I am forced to end it's process via task-manager. Yes, the process, not just the application. If I just close it, the process will stay alive forever, I have no choice.
I need to find a way to automate this process assassination so my users can actually use it, but what can I do? I know process termination is taboo, but what options do I have?
I'd love to use Application.Restart(), but that doesn't work at all, my form doesn't even close, it just locks. Is there a way to axe the process just before telling itself to launch again? Maybe I can do this with a batch file or something? Application.Exit at least takes the form off the screen.
As of now, I'm killing it from Task Manager, or my users are killing it by popping the breaker on the PC. Considerably more harsh than anything software-wise.

Have you considered isolating the problematic component in another process? I know it sounds complicated, but if you create another "application" which solely exists as a conduit to your device, you can make your main application just start a new one of those if the old one becomes unresponsive. It can nuke the old one, start a new one, and be "clean" again.
It does mean all kinds of inter-process communication of course, but the general idea of isolating something flaky is often a useful one.

Assuming that it is Dispose() that is the problem and that there is a proper IDisposable pattern implementation where the finalizer calls Dispose() I think that a solution might be to call GC.SupressFinalize(objWithFailingDispose) to prevent Dispose from being called at all.
It is ugly, but I might work.

This question should probably be titled "What do I do when I'm dealing with a Dispose() method I can't change, and has been written without considering a very real and very troubling real-world scenario?" And my suggestion would be to write a better one!
The simplest approach would be to create a wrapper for the object that will be disposed, and then calling GC.SuppressFinalize(internalConnectionObject); if you've detected that the connection has dropped. That way, if it's not responsive, it won't get stuck, but if it's there, it will be disposed properly. Isolation is your friend when you have troublesome components.

What could cause a Windows Service to hang when a Console App doing the exact same thing using the exact same base libraries doesn't?

I hate asking questions like this - they're so undefined... and undefinable, but here goes.
Background:
I've got a DLL that is the guts of an application that is a timed process. My timer receives a configuration for the interval at which it runs and a delegate that should be run when the interval elapses. I've got another DLL that contains the process that I inject.
I created two applications, one Windows Service and one Console Application. Each of the applications read their own configuration file and load the same libraries pushing the configured timer interval and delegate into my timed process class.
Problem:
Yesterday and for the last n weeks, everything was working fine in our production environment using the Windows Service. Today, the Windows Service will run for a period of around 20-30 minutes and hangs (with a timer interval of 30 secods), but the console application runs without issue and has for the past 4 hours. Detailed logging doesn't indicate any failure. It's as if the Windows Service just...dies quietly - without stopping.
Given that my Windows Service and Console Applications are doing the exact same thing, I can only think that there is something that is causing the Windows Service process to hang - but I have no idea what could be causing that. I've checked the configuration files, and they're both identical - I even copied and pasted the contents of one into the other just to be sure. No dice.
Can anyone make suggestions as to what might cause a Windows Service to hang, when a counterpart Console Application using the same base libraries doesn't; or can anyone point me in the direction of tools that would allow me to diagnose what could be causing this issue?
Thanks for everyone's help - still digging.

You need to figure out what changed on the production server. At first, the IT guys responsible will swear that nothing changed but you have to be persistent. i've seen this happen to often i've lost count. Software doesn't spoil. Period. The change must have been to the environment.
Difference in execution: You have two apps running the same code. The most likely difference (and culprit) is that the service is running with a different set of security credentials than your console app and might fall victim to security vagaries. Check on that first. Which Windows account is running the service? What is its role and scope? Is there any 3rd party security software running on the server and perhaps Killing errant apps? Do you have to register your service with a 3rd party security service? Is your .Net assembly properly signed? Are your .Net assemblies properly registered and configured on the server? Last but not least, don't forget that a debugger user, which you most likely are, gets away with a lot more stuff than many other account types.
Another thought: Since timing seems to be part of the issues, check the scheduled tasks on the machine. Perhaps there's a process that is set to go off every 30 minutes that is interfering with your own.

You can debug a Windows service by running it interactively within Visual Studio. This may help you to isolate the problem by setting (perhaps conditional) breakpoints.
Alternatively, you can use the Visual Studio "Attach to process" dialog window to find the service process and attach to it with the "Debug CLR" option enabled. Again this allows you to set breakpoints as needed.
Are you using any assertions? If an assertion fires without being re-directed to write to a log file, your service will hang. If the code throws an unhandled exception, perhaps because of a memory leak, then your service process will crash. If you set the Service Control Manager (SCM) to restart your process in the event of a crash, you should be able to see that the service has been restarted. As you have identical code running in both environments, these two situations don't seem likely. But remember that your service is being hosted by the SCM, which means a very different environment to the one in which your console app is running.
I often use a "heartbeat", where each active thread in the service sends a regular (say every 30 seconds) message to a local MSMQ. This enables manual or automated monitoring, and should give you some clues when these heartbeat messages stop arriving.
Annother possibility is some sort of permissions problem, because the service is probably running with a different local/domain user to the console.
After the hang, can you use the SCM to stop the service? If you can't, then there is probably some sort of thread deadlock problem. After the service appears to hang, you can go to a command-line and type sc queryex servicename. This should give you the current STATE of the service.

I would probably put in some file logging just to see how far the program is getting. It may give you a better idea of what is looping/hanging/deadlocked/crashing.

You can try these techniques
Logging start logging the flow of the code in the service. Have this parameter based so you dont have a deluge after you are done. You should log all function names, parameters, timestamps.
Attach Debugger Locally or Remotely attach a debugger with the code to the running service, set appropriate breakpoints (can be based on the data gathered from logging)
PerfMon Run this utility and gather information about the machine that the service is running on for any additional clues (high CPU spikes, IO spikes, excessive paging, etc)

Microsoft provides a good resource on debugging a Windows Service. That essentially sounds like what you'd have to do given that your question is so generic. With that said, has any changes been made to the system over the last few days that could aversely affect the service? Have you made any updates to the code that change the way the service might possibly work?
Again, I think you're going to have to do some serious debugging to find your problem.

What type of timer are you using in the windows service? I've seen numberous people on SO have problems with timers and windows services. Here is a good tutorial just to make sure you are setting it up correctly and using the right type of timer. Hope that helps.

Another potential problem in reference to psasik's answer is if your application is relying on something only available when being run in User Mode.
Running in service mode runs in (is it desktop0?) which can cause some issues in my experience if you are trying to determine states of something that can only be seen in user mode.

Smells like a threading issue to me. Is there any threading or async work being done at all? One crucial question is "does the service hang on the same line of code or same method every time?" Use your logging to find out the last thing that happens before a hang, and if so, post the problem code.
One other tool you may consider is a good profiler. If it is .NET code, I believe RedGate ANTS can monitor it and give you a good picture of any threadlock scenarios.

What can make a .NET app freeze the computer?

I know this is probably the canonical "It depends..." question but I'd appreciate any pointers as to where to start looking.
I have a client/server app talking over ethernet. In one computer I run the server and a client and on another just the client. One runs Vista and one runs XP. After an uptime of about 3 weeks the entire computer freezes and nothing works, not mouse, not keyboard, nothing -just power off. Every ten seconds the server sends a ping message to see if the clients are alive, other than that just a few small messages go back and forth every day.
I'm trying to find out if it's me causing it or something else. I've started a session and after a few days I thought I'd check for strange increases in memory use but beyond that I have very few ideas.

Some thoughts to consider:
You know the computer doesn't respond, but that doesn't mean it's hung. Does it respond to a ping?
Maybe the disk activity light is on all the time?
You say "no keyboard" - do you mean no caps lock or num lock lights?
Although the .NET application may be the only one you're running at the time, that does not imply it is the cause of the problem. Some background job could be doing it.
For example, I notice that Retrospect backup, when it is creating a snapshot, freezes the entire system for 10-15 minutes. I mean, no caps lock, the clock in the task bar doesn't update, no CTRL-ALT-DEL, can't type into an "Answer" text box in SO, nothing. It had nothing to do with what I was doing at the time, which was answering a question on SO.
After it came back, SO asked if I was a human. My feelings were hurt. ;-)

You could attach a kernel debugger to the OS. That way you should be able to inspect the state of the OS and your process even if the OS is completely unresponsive. (Unfortunately, it's a lot harder than just hitting "break" in VS. I suggest reading John Robbin's "Debugging Applications for .NET and Windows" before trying that.)
You could also try to create memory dumps of your application in regular intervals. You might have to do a little scripting for that, though. (usually, you'd create a dump with a keystroke, using a tool like userdump or adplus, but if the OS is not responding to keystrokes, that won't work.) That way, you know what state your process is in during or shortly before a hang.
This page: http://blogs.msdn.com/debuggingtoolbox/default.aspx is a good starting point for scripting WinDbg. (If you don't know what to do with a memory dump, I'd again suggest John Robbin's excellent book on debugging!)
Other than that, I can only think of standard debugging tricks: does the problem occur on every PC? Does it happen if there are no client requests? Does it happen sooner if there are more client requests? Does it happen sooner if there is less available physical memory? Try removing parts of your application (maybe on a separate server for testing) and see if the problem still occurs, and so on. Try running it in a VM so you can see if it uses the CPU, harddisk, or network during those "hangs".

This isn't going to be the answer, but I'd advise starting by checking your OS event logs and running a perfmon to keep track of memory, cpu usage etc.

Which computer freezes, the server or client? And what OSes are they running respectively?
As Daniel L noted, tight polling loops can really kill the CPU. If you can, change your code to use event handlers, it's a much more robust solution.
Finally, are you certain there's not a hardware problem on the freezing computer?

.net remoting stops every 100 seconds

We have very strange problem, one of our applications is continually querying server by using .net remoting, and every 100 seconds the application stops querying for a short duration and then resumes the operation. The problem is on a client and not on the server because applications actually queries several servers in the same time and stops receiving data from all of them in the same time.

100 Seconds is a give away number as it's the default timeout for a webrequest in .Net.
I've seen in the past that the PSI (Project Server Interface within Microsoft Project) didn't override the timeout and so the default of 100 seconds was applied and would terminate anything talking to it for longer than that time.
Do you have access to all of the code and are you sure you have set timeouts where applicable so that any defaults are not being applied unbeknownst to you?

I've never seen that behavior before and unfortunately it's a vague enough scenario I think you're going to have a hard time finding someone on this board who's encountered the problem. It's likely specific to your application.
I think there are a few investigations you can do to help you narrow down the problem.
Determine whether it's the client or server that is actually stalling. If you have problems determining this, try installing a packet filter and monitor the traffic to see who sent the last data. You likely won't be able to read the binary data but at least you will get a sense of who is lagging behind.
Once you figure out whether it's the client or server causing the lag, attempt to debug into the application and get a breakpoint where the hang occurs. This should give you enough details to help track down the problem. Or at least ask a more defined question on SO.

How is the application coded to implement the continuous querying? Is it in a continuous loop? or a loop with a Thread.Sleep? or is it on a timer ?,
It would first be useful to determine if your system is executing this "trigger" in your code when you expect it to, or if it is, and the remoting server is not responding... so, ...
if you cannot reproduce this issue in a development environment where you can debug it, then, if you can, I suggest you add code to this Loop to write out to a log file (or some other persistence mechanism) each time it "should" be examining whatever conditions it uses to decide whether to query the remoting server or not, and then review those logs when the problem reoccurs...
If you can do the same in your remoting server, to record when the server receives a remoting request, this would help as well...
... and oh yes, just a thought, (I don;t know how you have coded this... ) but if you are using a separate thread in client to issue the remoting request, and the channel is being registered, and unregistered on that separate thread, make sure you are deconflicting the requests, cause you can't register the same port twice on the same machine at the same time...
(although this should probably have raised an exception in your client if this was the issue)