createWindowEx failed exception is thrown by my server which is using overbyteICS dll in .net C# windowsforms.
I have a server which handles large number of clients throughout the day. But when the total connections(i.e Connection and disconnections altogether) count reaches to 10000 the above error appears and the server stops accepting user connections and also hangs the machine.
I agree with Roger, but let's confirm it first - When this error occurs, run SPY++ from MicrosoftVisualStudio\Tools in the start Menu and look through the window tree. Expand the branches and look for duplicates of some windows. Surely there will be many of them, but you are interested in hundreds and thousands of copies. If you hit that, then it's what Roger said... ...and there's almost no solution other that periodically restarting the connection-server process (or whole machine, just in case) just to be sure it doesnt hang (of course, server restart will irritate the users almost as much..), or fixing/patching/reimplementing the connection-server process to be more resource-friendly..
Note that while opening a hidden window per single connection is a very wasteful approach, it still shuold not hang the machine. It simply should drop the connections that it cannot handle. Here, it seems it has no limits implemented at all, which is a bug.
edit: on pre-NT (i.e. win9x) the limit is hardcoded. On NT class systems, you can try to tweak the pool:
http://weblogs.asp.net/israelio/archive/2007/02/07/max-num-of-open-windows-under-xp-2003-vista-resolved.aspx
but still, I'd consider that as a last restort, as problem will return when number of connection rises again. First, try to ping the server developers to fix that permanently..
You diagnosed it well. Yes, a CreateWindowEx() failure and 10,000 belong together. 10,000 is the default user32 object quota for a process. In other words, a single process isn't allowed to create more than 10,000 windows. This is a counter-measure against apps that leak window handles, a very common bug. The total number of windows that can be created in a session is a limited resource, having one process consume them all would cause outright failure, you couldn't shut down Windows anymore.
Clearly it is not a leak in your case. You can find temporary relief by changing a registry setting, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows\USERProcessHandleQuota. Reboot to make it effective.
Increasing from 10,000 to the maximum of 18,000 should be okayish if the machine doesn't otherwise run processes that require a lot of windows. Something you can see with Taskmgr.exe, Processes tab. Choose View + Select Columns and tick USER objects. Also tick GDI objects and Handles while you are at it, other resources that have a quota.
Long term, this behavior does not scale well. You'll need to find the code that creates a window handle for every web request and fix it.
Related
I've written a C#/WPF/Windows 10 application for a client that, when started, runs beautifully - it connects to another app over the network, begins data acquisition, as the data comes in it processes it and displays charts and tables to the user. All is well.
And then Something Bad Happens.
At some point, for no reason I've been able to determine, there's a spike in CPU usage and then the program becomes totally GC-bound. The client sent me this screenshot:
The client is running the app in a completely different environment than what I have (he has several hundred thousand dollars of actual data acquisition hardware, I have a simulator) and is several thousand miles away.
When the problem happens, the client is willing to let me take remote control over WebEx and run the debugger, but since the problem is so infrequent I only get called in once the problem has happened - and by then it's too late. Often times Visual Studio is so overwhelmed by the memory situation that it gives up when asked to do a memory snapshot or even a stack trace (the client has full source so can do full debugging).
Initially I thought this was a memory leak, so for key classes I have code that keeps WeakReferences for each object that is created, and a regular process that scans those lists every 2 minutes, reports back how many objects of each are in existence, and writes that data to a log file. The log file usually starts getting pretty wonky when the problem starts happening, so the log file data I have doesn't show anything abnormal.
I have also run the ANTS Performance Profiler and Memory Profiler on the application - on my computer. Neither one show anything unexpected happening.
QUESTION
Since this only happens on the client machine, and only happens infrequently, is there something I can put in my code to (1) detect the GC becoming unhappy, and (2) log what it is that is making the GC unhappy?
Here is the thing. I'm connecting via COM to some devices at KNX/EIB. But sometimes - and I want to be ready for worst-case anyways - my application crashes leaving all objects and libraries exposed somewhere, somehow. I noticed when I restart the app I have trouble to get a connection again. I get an error for a connection procedure that is actually working well normally. Sometimes this connect procedure is working sometimes it is not, randomly. That is bad! After some time (several minutes) it seems to work again after a series of complete fails. But I think I see a pattern now. It doesn't work after a crash with no clean disconnect. My guess is there are objects that hold a connection to the device that us why I can't get a new connection. This is why I ask this question.
Question:
How do I unload those unused objects to kill undead connections?
How do I make Windows to check for unused libraries to be unloaded?
I just want to tell Windows, "I messed up badly and I need to continue my work. Please clean up my mess for me, so I can start fresh! Do I deserve a 2nd chance?"
Edit:
The scenario is the app has crashed and closed. I have no references to anything anymore. No finally clause or anything. The app can only be started again. What can I do to clean up the mess that has been made before, programmatically?
Edit 2:
Hans gave me the hint of killing the responsible server. So for now I solve that with calling taskkill on startup (at least as long I'm in dev). And it works!
C:\Windows\System32\taskkill.exe /F /IM Falcon.exe
This is the failure mode of an out-of-process COM server. If the client program crashes to the desktop without releasing the interface pointers then the server is completely unaware that the client isn't around anymore. And tends to get balky when you try to reconnect, many servers just permit one client.
By far the most common way that programmers induce this failure mode is by using a debugger. They'll click the Red Button or use the Stop Debugging command. Bam, no cleanup of course.
COM garbage-collects unused servers automatically. But that isn't particularly fast, takes an easy 10 minutes before it decides it needs to step in. And doesn't always work for every server, Office programs notoriously don't get cleaned-up for example.
Not much you can do about this when your app keels over in regular usage. Otherwise the kind of problem that killed middle-ware. Still, having such a mishap in a C# program is pretty unusual, the CLR releases interface pointers at program termination even when the app crashed with an exception. You'd have to have the very nasty kind of mishaps to bypass this, critical exceptions like ExecutionEngineException or the one this site is named after.
Don't focus too much on the Stop Debugging induced failures, it is normal and using Task Manager to kill the server is expected and required. Otherwise just be sure to get the nasty bugs out of your code and you won't have a problem. If you need more help then be sure to contact the owner of the server, be sure to have a small repro project available that demonstrates the issue.
I'm not sure if the question title is the best one but it was the best one I could come up with...
I have this .NET (C#) applications which starts up with Windows and remains opened until the computer is turned off. The app stays on the tray and I open it by clicking the tray icon and close it the same way.
The app is not slow at first, it works normally, no problems there. But after long periods of inactivity of itself, it gets very slow when showing it again for the first time in a long period. Know what I mean.
For instance, I could not use/open (click the tray icon) for a few days and between those days I opened and closed and used lots of other apps, heavy apps too and I probably hibernated and resumed the computer a few times and when I needed to open my app again, it was slow. After a few minutes of using it, it goes back to normal and works fine.
I believe this has something to with memory management and the system probably frees up most of my app's memory so other programs can use it more efficiently. And maybe .NET memory management as something to do with it...
Whatever the reason, is there anything I can do to optimize my app regarding that issue?
This is almost certainly due to memory being paged out to disk.
When you cease using your application and other applications or tasks start exerting memory pressure, pages of memory from your app can be written out to disk. When you try to use your application again, all this data must then be read in causing the stalls that you see.
Unfortunately, there is no good solution - if you run as administrator or have the SeLockMemoryPrivilege, you can lock parts of your application into physical memory. You can try "touching" pages periodically to keep them in physical memory. Unfortunately, both these options will cause your application to interact badly with other applications on the system - your memory is getting paged out because the physical memory is needed for something else. You can attempt to lower your overall memory footprint, but you will still run into this issue in some cases. There are options for tweaking your working set size, but the OS is free to ignore those, and will in many cases.
You can use the .NET memory profiler to get a good idea of what your application is doing with memory over time. You can check if anything is building up where you don't expect it to (collections, lists, etc.) and causing your memory footprint to grow.
Have you tried double-clicking in Process Explorer on your running app and observe it for leaks? Is it CPU utilization, UI leak, memory leak, recurrent I/O?
If the issue is indeed that the program is being paged out due to an extended period of inactivity, have you tried setting a "keep-alive" timer, that fires every few minutes, that basically keeps the application in a state where the paging system doesn't ever see it as idle?
I reckon that this behavior is normal in Windows. Windows will serve the application that is need it most, ie, the application that is currently in used. Thus, if an application is minimized or not active for pro-long duration, like the case specified above, it will be the best candidate for Windows to page the application memory and make available memory to active application. Windows will restore the application memory when it get active from paging memory (hard disk), thus this explain why it is slow.
This might be not the best answer, but if I were you, I will experiment by increasing the base priority of the application. This can be done by using Thread object in .NET framework.
Thanks.
This isn't a root cause answer or solution, but if Michael is right and there's nothing you can really do about it, then you might just want to consider visually informing the user that your application is active with a progress bar or nifty progress circle.
It won't speed up your program, but it may ease the minds of your users just a little bit.
I've been experiencing a high degree of flicker and UI lag in a small application I've developed to test a component that I've written for one of our applications. Because the flicker and lag was taking place during idle time (when there should--seriously--be nothing going on), I decided to do some investigating. I noticed a few threads in the Threads window that I wasn't aware of (not entirely unexpected), but what caught my eye was one of the threads was set to Highest priority. This thread exists at the time Main() is called, even before any of my code executes. I've discovered that this thread appears to be present in every .NET application I write, even console applications.
Being the daring soul that I am, I decided to freeze the thread and see what happened. The flickering did indeed stop, but I experienced some oddness when it came to doing database interaction (I'm using SQL CE 3.5 SP1). My thought was that this might be the thread that the database is actually running on, but considering it's started at the time the application loads (before any references to the DB) and is present in other, non-database applications, I'm inclined to believe this isn't the case.
Because this thread (like a few others) shows up with no data in the Location column and no Call Stack listed if I switch to it in the debugger while paused, I tried matching the StartAddress property through GetCurrentProcess().Threads for the corresponding thread, but it falls outside all of the currently loaded modules address ranges.
Does anyone have any idea what this thread is, or how I might find out?
Edit
After doing some digging, it looks like the StartAddress is in kernel32.dll (based upon nearby memory contents). This leads me to think that this is just the standard system function used to start the thread, according to this page, which basically puts me back at square one as far as determining where this thread actually comes from. This is further confirmed by the fact that ALL of the threads in this list have the same value for StartAddress, leading me to ask exactly what the purpose is...?
Edit 2
Process Explorer let me to an actually meaningful start address. It looks like it's mscorwks.dll!CreateApplicationContext+0xbbef. This dll is in %WINDOWS%\Microsoft.NET\Framework\v2.0.50, so it looks like it's clearly a runtime assembly. I'm still not sure why
it's Highest priority
it appears to be causing hiccups in my application
You could try using Sysinternals. Process Explorer let's you dig in pretty deep. Right click on the Process to access Properties. Then "Threads" tab. In there, you can see the thread's stack and module.
EDIT:
After asking around some, it seems that your "Highest" priority thread is the Finalizer thread that runs due to a garbage collection. I still don't have a good reason as to why it would constantly keep running. Maybe you have some funky object lifetime behavior going on in your process?
I'm not sure what this is, but if you turn on unmanaged debugging, and set up Visual Studio with the Windows symbol server, you might get some more clues.
Might be the Garbage Collector thread. I noticed it too when I was once investigating a finalizer-related bug. Perhaps your system memory is low and the GC is trying to collect all the time? This was the case in the previously mentioned bug too. I couldn't reproduce it on my machine, but a co-worker of mine had a machine with less RAM where it would reappear like clockwork.
I know this is probably the canonical "It depends..." question but I'd appreciate any pointers as to where to start looking.
I have a client/server app talking over ethernet. In one computer I run the server and a client and on another just the client. One runs Vista and one runs XP. After an uptime of about 3 weeks the entire computer freezes and nothing works, not mouse, not keyboard, nothing -just power off. Every ten seconds the server sends a ping message to see if the clients are alive, other than that just a few small messages go back and forth every day.
I'm trying to find out if it's me causing it or something else. I've started a session and after a few days I thought I'd check for strange increases in memory use but beyond that I have very few ideas.
Some thoughts to consider:
You know the computer doesn't respond, but that doesn't mean it's hung. Does it respond to a ping?
Maybe the disk activity light is on all the time?
You say "no keyboard" - do you mean no caps lock or num lock lights?
Although the .NET application may be the only one you're running at the time, that does not imply it is the cause of the problem. Some background job could be doing it.
For example, I notice that Retrospect backup, when it is creating a snapshot, freezes the entire system for 10-15 minutes. I mean, no caps lock, the clock in the task bar doesn't update, no CTRL-ALT-DEL, can't type into an "Answer" text box in SO, nothing. It had nothing to do with what I was doing at the time, which was answering a question on SO.
After it came back, SO asked if I was a human. My feelings were hurt. ;-)
You could attach a kernel debugger to the OS. That way you should be able to inspect the state of the OS and your process even if the OS is completely unresponsive. (Unfortunately, it's a lot harder than just hitting "break" in VS. I suggest reading John Robbin's "Debugging Applications for .NET and Windows" before trying that.)
You could also try to create memory dumps of your application in regular intervals. You might have to do a little scripting for that, though. (usually, you'd create a dump with a keystroke, using a tool like userdump or adplus, but if the OS is not responding to keystrokes, that won't work.) That way, you know what state your process is in during or shortly before a hang.
This page: http://blogs.msdn.com/debuggingtoolbox/default.aspx is a good starting point for scripting WinDbg. (If you don't know what to do with a memory dump, I'd again suggest John Robbin's excellent book on debugging!)
Other than that, I can only think of standard debugging tricks: does the problem occur on every PC? Does it happen if there are no client requests? Does it happen sooner if there are more client requests? Does it happen sooner if there is less available physical memory? Try removing parts of your application (maybe on a separate server for testing) and see if the problem still occurs, and so on. Try running it in a VM so you can see if it uses the CPU, harddisk, or network during those "hangs".
This isn't going to be the answer, but I'd advise starting by checking your OS event logs and running a perfmon to keep track of memory, cpu usage etc.
Which computer freezes, the server or client? And what OSes are they running respectively?
As Daniel L noted, tight polling loops can really kill the CPU. If you can, change your code to use event handlers, it's a much more robust solution.
Finally, are you certain there's not a hardware problem on the freezing computer?