I have set up a local WCF service, self hosted in a console application using NetNamedPipeBinding for client access.
To make calls to the service I reference a library.dll where I have the following method:
public static string GetLevel(Point p)
{
ChannelFactory<IService> pipeFactory = new ChannelFactory<IService>(new NetNamedPipeBinding(), new EndpointAddress("net.pipe://localhost/PTS_Service"));
IService pipeProxy = pipeFactory.CreateChannel();
string result = pipeProxy.GetLevel(p);
((IClientChannel)pipeProxy).Close();
pipeFactory.Close();
}
The GetLevel() command returns a string from a list stored in the service, based on the Z coordinate of Point(X,Y,Z) p.
This works and gives speeds of 8ms total if the method is called from the above console app.
However when the same method from the library.dll is called from another app.exe or plugin.dll (loaded by an external program) the times increase drastically. I've stop watched the above 5 lines of code:
consoleHost.exe : 0 - 3 - 6 - 7 - 8
app.exe : 89 - 155 - 248 - 259 - 271
plugin.dll : 439 - 723 - 1210 - 1229 - 1245
Shouldn't the times be the same, not dependent on who makes the call to library.dll?
EDIT
Since I've cancelled out all methods to just retrieving a string from a running service, I believe the problem lies in the first creation run of the channelFactory, all subsequent calls in the same app/plugin run are equal in time.
I understand the first call is slower, but as I see this is around 30ms in a new app and around 900ms in my plugins, I believe there is another thing causing this.
I have found a question with similar delays:
First WCF connection made in new AppDomain is very slow to which the solution was to set LoaderOptimizationAttribute to MultiDomain. Could it be possible everytime the plugin runs it has to JIT-compile instead of use native code?
I tried adding this code above main in consoleHost.exe but see no gain in the plugin run time. Could this be because of the external program in between and is there a way around this? Say could my plugin create a new Appdomain whenever it wants to access the service and call from within this new Appdomain the above method from my library.dll or does this make no sense?
EDIT2
I recorded the time spent in JIT compiling with a profiling program as suggested in the comments, this gives 700ms for JIT compiling and total execution time of 800ms for the plugin.
I used ngen to precompile the library.dll to create a native image .ni.dll. I see in process explorer that this image is loaded by the external program, though there is no time gain in the plugin? As I understand there shouldn't be a reason the plugin would still JIT compile or am I doing something wrong?
I also noticed when debugging in VS that the console & app only do some loading of assemblies, the plugin loads and unloads everytime it creates or modifies a plugin instance. I believe this is the way plugins work and should not explain the difference in first execution time?
The communication should not depend on a caller, but the way the calls are done.
The most time expensive operation is creating a Channel.
Should the Proxy once created, then every next call will be done with a an average similar speed. (Of cause if the callers are using the service from the same place: same Machine in the Network. In your case should be the same, while in your case you use the localhost)
Some performance increase can be also archived by service configuration (SingleInstance should be faster than PerCall).
Another point to pay attention is to exam the possible locks in your service method. It can happen than some service clients are waiting for a call, while the service is busy.
if the service call is not an async one, try to make it async and use it.
After some further investigating: the external program prevented the sharing of loaded assemblies/JIT compilations through a setting in a .ini file when the process is started. Fortunately this could be disabled so sharing also becomes possible in the plugin.
After altering this setting (1 line in the .ini to No instead of Yes!) the time reduced to 30ms and every next call 3ms or less.
Related
I have a service which can automatically update itself. It does so by downloading and running the installer/updater, which is another executable. That executable stops the service with the ServiceController class, makes sure it is stopped using WaitForStatus(ServiceControllerStatus.Stopped), and then copies the relevant files. Those files include the service's main assembly and its dependencies.
Sometimes, the installation works as expected, but sometimes, I get an IOException telling me that it cannot access one of the service's assemblies because it is being used by another process (presumably the service which hasn't completely shut down). To remedy this I tried adding a fairly large delay of 1000ms after the WaitForStatus call, before starting to copy the files, but the IOException still gets thrown (or not) at random, i.e. sometimes the update is successful and sometimes it isn't.
I then tried adding a call to Environment.Exit() at the end of the ServiceBase.OnStop implementation of my service, and the update seems to work all the time now. However, I can tell this is not good practice since when I try stopping my service from the SCM, it stops, but gives the error Service process closed unexpectedly.
So what is the best way to do what I am trying to do? I could increase the delay, but it seems to me that 1000ms should be ample time for the service to properly shut down and release its lock on its assemblies. Perhaps I am doing something else incorrectly.
I'll write about what I did to solve this problem. I made it so that when a file isn't able to be copied because of that exception, the process enters a loop whereby it waits 1000ms and tries to copy that file again. It does so 5 times, and if it isn't able to copy the file after 5 times, the installer fails. In practice, from the log information I am receiving, it can take up to 3 seconds from a service process to properly shut down. I think this is the best solution for my problem.
I am trying to monitor the amount of Remote procedure call (RPC) interface count to stop my program before the infamous 256 interface limit.
Right now i am trying to use the following code to get all assembly but i cannot figure out which property gives the actual count or that actually tell me that the interface is taking one of the crucial 256 limits.
foreach (var assemblyName in System.Reflection.Assembly.GetExecutingAssembly().GetReferencedAssemblies())
{
// get the assembly
var assembly = System.Reflection.Assembly.Load(assemblyName.ToString());
// get all types in the assembly
var types = assembly.DefinedTypes.ToList();
}
the types seems good (not sure) but i think i see all classes and interface but there is no property telling me one of them is taking a spot in the interface limit.
Can i have the information i am looking for somehow from the list of assemblies or is there another way to get it ?
Here is the RPC call limit document from Microsoft
I am aware that this error might happen because there is not enough ram available but as far as i know there is around 10-11 gb free on the computers when the issues occur and they all have access to 32 gb of page file.
After more tests
I have finally manage to use the ComVisible property to figure out which DLL are applicable and it worked. Now that i know which DLL can be problematic i am still stuck at where can i look to find how many RPC interface are currently loaded into my program.
Taking some step back made me think that this kind of information should be somewhere in the current process and not within the DLL reference itself. As the count must be the count of the currently instantiated distinct interface. There is so little information about this kind of problem that it is very very difficult to figure it out.
On the bright side note i figured out how to replicate easily and it's not a specific DLL that throw the error. I made dummy ones with a single class with a string value and when called at the good time it throws the error.
It is probably replicable in a simple test project by creating 260 small unmanaged dll with a different class in each called Class1 in dll1.dll and Class2 in dll2.dll and so on. Then add reference to each 260 dll and instantiate the class in each of them. It will crash on the instantiation of Class256 in dll256.dll. Change the order however you want and it will crash anytime on the 256th instantiation.
I use multiple manufacturer DLL that has between 10 and 100 classes each that are managed and unmanaged and i can only call a few of them before it completely crash the application. So far .NET dll (even those that are not mine) does not seems to count toward this limit. Do not take this as "it actually does not count", I am not sure. It just seems to not affect the count in my primary tests.
New things came up
We absolutely needed to bypass this limit and still know one has a project of that scale and can figure this out. Out of hope we decided to temporary create separate EXE and pass bunch of commands unsafely in a local text file, make the EXE run and read the text file to know what to do. That EXE has a couple DLL reference that the main project has so it has it's own RPC count limit of 256 that doesn't count toward the main project limit.
Then the EXE run what it need and save the result in the another text file that we wait for it to be created and read it. This is ugly and required months of coding but it work~ish. Main problem is that anti-virus and windows defender blocks all these EXE especially those that access the web and download files so we had to manually contact each of our clients and manually green list all these new EXE files and boy that is long and difficult.
We are still looking for a solution were we can up the limit on windows registry or count the amount of RPC instantiated so we can force the user to shutdown and restart the application as he used more. Why are we still looking for a solution ? Well it's easy, now it completely killed our agile points for each bug. By charts our usual 2 pts went to 8 or 13 pts.
I'm trying to determine the cause of a very long (imho) initial start up of an ASP.NET application.
The application uses various third party libraries, and lots of references that I'm sure could be consolidated, however, I'm trying to identify (and apportion blame) the dlls and how much they contribute to the extended startup process.
So far, the start up times vary from 2-5 minutes depending on usage of other things on the box. This is unacceptable in my opinion based on the complexity of the site, and I need to reduce this to something in the region of 30 seconds maximum.
To be clear on the scope of the performance I'm looking for, it's the time from first request to the initial Application_Start method being hit.
So where would I start with getting information on which DLL's are loaded, and how long they take to load so I can try to put a cost/benefit together on which we need to tackle/consolidate.
From an ability perspective, I've been using JetBrains dotTrace for a while, and I'm clear on how benchmark the application once we're in the application, but it appears this is outside of the application code, and therefore outside of what I currently know.
What I'm looking for is methodologies on how to get visibility of what is happening before the first entry point into my code.
Note: I know that I can call the default page on recycle/upgrade to do an initial load, but I'd rather solve the actual problem rather than papering over it.
Note2: the hardware is more than sufficiently scaled and separated in terms of functionality, therefore I'm fairly sure that this isn't the issue.
Separate answer on profiling/debugging start up code:
w3wp is just a process that runs .Net code. So you can use all profiling and debugging tools you would use for normal .Net application.
One tricky point is that w3wp process starts automatically on first request to an application and if your tools do not support attaching to process whenever it start it makes problematic to investigate startup code of your application.
Trick to solve it is to add another application to the same Application Pool. This way you can trigger w3wp creation by navigating to another application, than attach/configure your tools against already running process. When you finally trigger your original application tools will see loading happening in existing w3wp process.
With 2-5 minutes delay you may not even need profiler - simply attach Visual Studio debugger the way suggested above and randomly trigger "break all" several times during load of your site. There is a good chance that slowest portion of the code will be on the stack of one of many threads. Also watch out for debug output - may give you some clues what is going on.
You may also use WinDbg to capture stacks of all threads in similar way (could be more light way than VS).
Your DLL references are loaded as needed, not all at once.
Do external references slow down my ASP.NET application? (VS: Add Reference dialog)
If startup is taking 2-5 minutes, I would look at what happens in Application_Start, and at what the DLLs do once loaded. Are they trying to connect to a remote service that is very slow? Is the machine far too small for what it's doing (e.g. running a DB with large amounts of data plus the web server on an AWS micro instance or similar)?
Since the load time is probably not the IIS worker process resolving references, I would turn to traditional application profilers (e.g. Jetbrains, Antz, dotTrace) to see where the time is being spent as the DLLs initialize, and in your Application_Start method.
Entertainment options check along with profiling:
profile everything, add time tracing to everything and log the information
if you have many ASPX views that need to be compiled on startup (I think it is default for release configuration) than it will take some time
references to Web services or other XML serialization related code will need to compile serialization assemblies if none are present yet
access to remote services (including local SQL) may require the services start up too
aggressive caching in application/remote services may require per-population of caches
Production:
What is the goal for start up time? Figure out it first, otherwise you will not be able to reach it.
What is price you are willing to pay to decrease start up time. Adding 1-10 more servers may be cheaper than spending months of development/test time and delaying the product.
Consider multiple servers, rolling restarts with warm up calls, web gardens
If caching of DB objects or in general is an issue consider existing distributed in-memory caches...
Despite a large number of dlls I'm almost sure that for a reasonable application it cannot be a cause of problem. Most of the time it is static objects initialization is causing slow startup.
In C# static variables are initialized when a type is first time accessed. I would recommend to use a sql profiler and see what are the queries that are performed during the start time of the application and from there see what are the objects that are expensive to initialized.
I'm running Win 7 Pro 64-bit. I wrote a service in C# using the .NET 4 framework. It installs properly and starts to run. I know that it runs because it writes some output to a log file. However, after a few seconds it dies. When I use Visual Studio 2010 Pro to run this same code not as a service it never dies. So, my obvious question is regarding the appropriate approach for debugging this since I can't figure out why it should die as a service but not die as a non-service. I've put writes to the log file in several places in the code but it seems to die in a different place every time. The application has 3 threads. Any suggestions are welcomed.
If you're running your code directly from within the Service's Start method, this behavior can easily occur. The problem is that the service's Start method is expected to start the service and immediately return. If it sits there executing code, Windows will kill the service on you.
The correct way to handle this is to have the service's Start() method run your code in a dedicated thread. It shouldn't really need anything except the thread creation and an immediate return. If this is the problem, just setup a foreground thread and put your logic there, and it will likely work correctly.
Use System.Diagnostics.Debugger.Launch to run it as a service and debug. If it doesn't crash in that scenario add additional logging and make sure to add a top level catch to write out any error. If that still doesn't do it create a crashdump file and examine with with SOS and windbg.
Calling a WCF published orchestration from a C# program usually is sub-second response time. However, on some occasions, it can take 20-50- seconds between the call in the C# program and the first trace message from the orchestration. The C# that runs calls the WCF runs under HIS/HIP (Host Integration Services/CICS Host-Initiated Processing).
Almost everytime I restart the HIS/HIP service, we have a very slow response time, and thus a timeout in CICS. I'm also afraid it might happen during the day if things "go cold" - in other words maybe things are being cached. Even JIT first-times compiles shouldn't take 20-50 seconds should they? The other thing that seem strange is that the slow response time seems to be the load of the orchestration, which is running under the BizTalk service, not the HIP/Service which I cycled.
The fear is that when we go live, the first user in the morning (or after a "cold-spell" will get the timeout). The second time they try it after the time-out, it is always fast.
I've done a few tests by restarting each of the following:
1) BizTalk services
2) IIS
3) HIS/HIP Transaction Integrator (HIP Service)
Restarting any one of them tends to cause about a 20 second delay.
Restarting all 3 is like the kiss of death - about a 60 second delay before first trace appears from orchestration.
The HIP program always gives its first trace quickly, even when the HIP service is restarted. Not sure why restarting HIP slows down the starting of the orchestration.
Thanks,
Neal Walters
I have seen this kind of behavior with the MQSeries adapter as well. After a period of inactivity the COM+ components which enable communication with MQSeries will shut down due to inactivity.
What we had was a 10 minute timer which would force some sort of a keep-alive message. I don't know if you have a non-destructive call which can be sent, or if you can build one into the system just for this purpose.
I have the same problem with a BizTalk flow that needs to work in 2 seconds, but when it was unused for some time the reload of the dll into cache generated a timeout.
We found a solution in MS's Orchestration Engine Configuration documentation, where they explain how to avoid unloading of the dlls:
Using the options SecondsIdleBeforeShutdown and SecondsEmptyBeforeShutdown from AppDomainSpecs and assigning to the desired dlls in the ExactAssignmentRules or PatternAssignmentRules sections, you can have your dlls permanently loaded, and maybe you can avoid the caller application.
Take into account that if you restart the BizTalk host, the dll will be loaded again.