I'm doing some experiments with Microsoft Dynamics CRM. You interact with it through web services and I have added a Web Reference to my project. The web service interface is very rich, and the generated "Reference.cs" is some 90k loc.
I'm using the web reference in a console application. I often change something, recompile and run. Compilation is fast, but newing up the web service reference is very slow, taking some 15-20 seconds:
CrmService service = new CrmService();
Profiling reveals that all time is spent in the SoapHttpClientProtocol constructor.
The culprit is apparently the fact that the XML serialization code (not included in the 90k loc mentioned above) is generated at run time, before being JIT'ed. This happens during the constructor call. The wait is rather frustrating when playing around and trying things out.
I've tried various combinations of sgen.exe, ngen and XGenPlus (which takes several hours and generates 500MB of additional code) but to no avail. I've considered implementing a Windows service that have few CrmService instances ready to dish out when needed but that seems excessive.
Any ideas?
The following is ripped from this thread on the VMWare forums:
Hi folks,
We've found that sgen.exe does work. It'just that there is a couple of additional steps beyond pre-generating the serializer dll's that we missed in this thread. Here is the detailed instruction
PROBLEM
When using the VIM 2.0 SDK from .NET requires long time to instantiate the VimService class. (The VimService class is the proxy class generated by running 'wsdl.exe vim.wsdl vimService.wsdl')
In other words, the following line of code:
_service = new VimService();
Could take about 50 seconds to execute.
CAUSE
Apparently, the .NET XmlSerializer uses the System.Xml.Serialization.* attributes annotating the proxy classes to generate serialization code in run time. When the proxy classes are many and large, as is the code in VimService.cs, the generation of the serialization code can take a long time.
SOLUTION
This is a known problem with how the Microsoft .NET serializer works.
Here are some references that MSDN provides about solving this problem:
http://msdn2.microsoft.com/en-us/library/bk3w6240.aspx
http://msdn2.microsoft.com/en-us/library/system.xml.serialization.xmlserializerassemblyattribute.aspx
Unfortunately, none of the above references describe the complete solution to the problem. Instead they focus on how to pre-generate the XML serialization code.
The complete fix involves the following steps:
Create an assembly (a DLL) with the pre-generated XML serializer code
Remove all references to System.Xml.Serialization.* attributes from the proxy code (i.e. from the VimService.cs file)
Annotate the main proxy class with the XmlSerializerAssemblyAttribute to point it to where the XML serializer assembly is.
Skipping step 2 leads to only 20% improvement in the instantiation time for the VimService class. Skipping either step 1 or 3 leads to incorrect code. With all three steps 98% improvement is achieved.
Here are step-by-step instructions:
Before you begin, makes sure you are using .NET verison 2.0 tools. This solution will not work with version 1.1 of .NET because the sgen tool and the XmlSerializationAssemblyAttribute are only available in version 2.0 of .NET
Generate the VimService.cs file from the WSDL, using wsdl.exe:
wsdl.exe vim.wsdl vimService.wsdl
This will output the VimService.cs file in the current directory
Compile VimService.cs into a library
csc /t:library /out:VimService.dll VimService.cs
Use the sgen tool to pre-generate and compile the XML serializers:
sgen /p VimService.dll
This will output the VimService.XmlSerializers.dll in the current directory
Go back to the VimService.cs file and remove all System.Xml.Serialization.* attributes. Because the code code is large, the best way to achieve that is by using some regular expression substitution tool. Be careful as you do this because not all attributes appear on a line by themselves. Some are in-lined as part of a method declaration.
If you find this step difficult, here is a simplified way of doing it:
Assuming you are writing C#, do a global replace on the following string:
[System.Xml.Serialization.XmlIncludeAttribute
and replace it with:
// [System.Xml.Serialization.XmlIncludeAttribute
This will get rid of the Xml.Serialization attributes that are the biggest culprits for the slowdown by commenting them out. If you are using some other .NET language, just modify the replaced string to be prefix-commented according to the syntax of that language. This simplified approach will get you most of the speedup that you can get. Removing the rest of the Xml.Serialization attributes only achieves an extra 0.2 sec speedup.
Add the following attribute to the VimService class in VimService.cs:
[System.Xml.Serialization.XmlSerializerAssemblyAttribute(AssemblyName = "VimService.XmlSerializers")]
You should end up with something like this:
// ... Some code here ...
[System.Xml.Serialization.XmlSerializerAssemblyAttribute(AssemblyName = "VimService.XmlSerializers")]
public partial class VimService : System.Web.Services.Protocols.SoapHttpClientProtocol {
// ... More code here
Regenerate VimSerice.dll library by
csc /t:library /out:VimService.dll VimService.cs
Now, from your application, you can add a reference to VimSerice.dll library.
Run your application and verify that VimService object instanciation time is reduced.
ADDITIONAL NOTES
The sgen tool is a bit of a black box and its behavior varies depending on what you have in your Machine.config file. For example, by default it is supposed to ouptut optimized non-debug code, but that is not always the case. To get some visibility into the tool, use the /k flag in step 3, which will cause it to keep all its temporary generated files, including the source files and command line option files it generated.
Even after the above fix the time it takes to instantiate the VimService class for the first time is not instantaneous (1.5 sec). Based on empirical observation, it appears that the majority of the remaining time is due to processing the SoapDocumentMethodAttribute attributes. At this point it is unclear how this time can be reduced. The pre-generated XmlSerializer assembly does not account for the SOAP-related attributes, so these attributes need to remain in the code. The good news is that only the first instantiation of the VimService class for that app takes long. So if the extra 1.5 seconds are a problem, one could try to do a dummy instantiation of this class at the beginning of the application as a means to improve user experience of login time.
You might wish to look into the Sgen.exe tool that comes with .NET. There's also a handy little thing in Visual Studio's C# project properties "Build" page, at the very bottom, called "Build serialization assembly" that automatically runs Sgen for you.
I believe that this is not an SGEN issue. I have looked at the constructor code, and I see that it is doing a lot of reflection (based on the XmlIncludeAttribute on the class). It reflects on all of them, and can take a really long time.
There is a pre-generated XmlSerializer assembly that comes with CRM. Check to see whether you have SdkTypeProxy.XmlSerializers.dll and SdkProxy.XmlSerializers.dll in the GAC.
If you don't then that means that when you create the CrmService, .net will generate the XmlSerializer assembly which can take some time.
Hope this helps
I came across this thread when trying to find out why my initial SoapHttpClientProtocol calls were taking so long.
I found that setting the Proxy to null/Empty stopped the Proxy AutoDetect from occurring - This was taking up to 7 seconds on the initial call:
this.Proxy = GlobalProxySelection.GetEmptyWebProxy();
I have used above detailed answer as guide, and went a few steps forward, making a script to automate process. Script is made out of two files :
generateproxy.bat :
REM if your path for wsdl, csc or sgen is missing, please add it here (it varies from machine to machine)
set PATH=%PATH%;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools;C:\Program Files (x86)\MSBuild\14.0\Bin
wsdl http://localhost:57237/VIM_WS.asmx?wsdl REM create source code out of WSDL
PowerShell.exe -ExecutionPolicy Bypass -Command "& '%~dpn0.ps1'" REM proces source code (remove annotations, add other annotation, put class into namespace)
csc /t:library /out:references\VIM_Service.dll VIM_WS.cs REM compile source into dll
sgen /p references\VIM_Service.dll /force REM generate serializtion dll
generateproxy.ps1
(Get-Content VIM.cs) |
ForEach-Object {
$_ -replace "(?<attr>\[global::System.Xml.Serialization.[^\]]*\])", "/*${attr}*/" `
-replace "public partial class VIM", "[System.Xml.Serialization.XmlSerializerAssemblyAttribute(AssemblyName = ""VIM_Service.XmlSerializers"")] `npublic partial class VIM" `
-replace "using System;", "namespace Classes.WS_VIM { `n`nusing System;"
} |
Set-Content VIM.cs
Add-Content VIM.cs "`n}"
I have added those two files to client project, and in the pre-build event I have added lines
cd..\..
generateproxy
So, before every build, proxy classes are regenerated, and developer has (almost) no need to think about it. While building, WS must be up and running, and its URL must be in bat file. As a result of prebuild, two dll files will regenerate in client project's subfolder references.
After first execution of scripts, you should add reference to new dll.
Related
I am trying to modify a class that I have obtained from an open source library. While I have referenced the library in my project, the original class contained references to parts of the library that are marked internal. This is causing build errors in my code.
I read this answer: How do you "override" an Internal Class in C#?. Note that it indicates that one is out of luck when wanting to access an internal class when one does not have access to the source code. This is, however, not the case for me.
I can fix this, but I want to know if there is a good way to do it. It appears to me that I have two options. I could modify the source code for the internal class so that it is not internal (bad), or I could import the class itself directly into my project (worse). If I import the class, there are further reference issues, and I will probably have to import the entire library to fix the dependencies.
Note that I am aware that I could add my change to the open source library and then build it as new library to reference in my code. I do not want to do this at this time because I would like to be able to step into my method in the debugger. Once I have debugged my method and assured its utility, I will make it available as part of my version of the open source project.
I am new to c# and this is my first time working on a large open source project so I would like some advise on how to deal with this issue.
Update:
Here is the link to the source code of the library that I would like to modify: https://github.com/accord-net/framework/blob/development/Sources/Accord.Math/Optimization/Unconstrained/Least%20Squares/LevenbergMarquardt.cs
To modify this code, I opened the source in visual studio and attempted to build the version as it currently exists. On line 304 the following code appears:
int[] block = Vector.Range(s * blockSize, s * blockSize + B);.
When I have this file in visual studio, it gives me an error saying that Vector is inaccessible due to its protection level. When I look at the definition of that code in the intellisense window, it shows that Vector is marked as internal.
If I attempt to bring the source code of Vector into my project, there are even more issued of the same type because Vector uses other internal classes.
Library version:
Are you sure that you're using the latest version of the library ?
By looking at the source of Vector.Range I can see that it is a public member.
Nuget packages of Accord.NET seems up to date, so check out your current version.
Additionally, I would recommend you to depend on the NuGet package instead of a manually installed assembly (you would see update notifications in Nuget Package Manager as a bonus).
Extending the library:
You have a couple of options here,
discuss with the authors about the possibility of opening some internal types, explain your needs and they might even suggest an alternative path without making such changes on their side
fork the library and make your changes, eventually PR
(personally I'd try step 1 first)
Debugging:
When referencing the Nuget package, you can create a PDB out assembly references using Telerik JustDecompile (freeware). Doing so relieves you to drag your own build and so on.
I am attempting to test some some basic WCF service code via LinqPad. When I attach to the service process, all of the serialized parameters are set to null. I believe the problem is that it is unable to serialization due to the classes being in the wrong namespace. However, I am unable to figure out how to place them in the correct namespace without having them be compiled outside of Linqpad. This works, but prevents the LinqPad project from being as self-contained.
Things I've tried:
Creating a self-contained visual studio project. This worked once I moved everything into a namespace matching that of my existing code. However, I wish to run the test code in LinqPad.
Have my Linqpad project reference a dll containing the service class. This works, but makes the lingpad project less self-contained.
ContractNamespaceAttribute.ClrNamespace looks like it might resolve the problem, but it can only be applied to assemblies and modules.
Is there a way to get this working in Linqpad as a self-contained project (e.g., by somehow telling LinqPad to put my classes in a specific namespace)?
I hava a quite complex solution, containing 10 projects aside from Test projects.
It is a network of distributed applications & services that communicate using remoting; therefore having the proper referenced assemblies (& versions) is crucial. That's why I want the whole thing to be compiled and schrink-wrapped in ONE build.
One of the applications is a demo/analysis-tool that runs a subprocess of another - much bigger - application based on the user's input and displays the results; That way engineers have a tool to help tweak their settings for "the big computation". Obviously that subprocess is contained in another assembly, and a big part of te results presented to the engineers is generated by
#if ENABLE_TRACE_MATCHING
Trace.WriteLine("Some engineering output");
#endif
My problem is that Conditional Compilation Symbols in the project settings are limited to that project's assembly, and do not propagate over referenced assemblies.
How can I configure my build in such a way that all projects will be built without ENABLE_TRACE_MATCHING being defined, except for the one debug/analysis-app project where all referenced projects/assemblies must be compiled with ENABLE_TRACE_MATCHING being defined
I also cannot replace #if ENABLE_TRACE_MATCHING by #if DEBUG, since that would enable a whole lot of different output our engineers wouldn't know how to handle.
Thanks in advance.
PS: If you think my code smells, then I agree. Additionally: It's mostly not my code ;)
You need to learn more about Microsoft Build, which is an out-of-the-box Microsoft .NET tool present in any framework's installation.
Using MSBuild you can define these "symbols" (properties) and a batch of commands (targets).
That's you can create a MSBuild script that imports default Visual Studio targets from all projects in your solution, and declare in the script these properties ("symbols").
In fact, the property to set such symbols already exists: "DefineConstants".
So, since you have it, you can have that MSBuild script that provides that property value, re-declaring it there, so, ALL MSBuild targets will be knowing about these symbols.
EDIT:
Check this other question too:
msbuild, defining Conditional Compilation Symbols
I started looking into IronPython to develop scripts against my .NET assemblies for certain tasks and ad-hoc processes. I like the ease of development with IronPython and the small footprint compared to handling this with a .NET console application project. However, I immediately ran into a roadblock with settings from app.config file. The assemblies I am planning to use require settings from app.config file such as database connection string, application settings, etc. I saw this question on SO How to use IronPython with App.Config. However, from what I gather and assume, none of the suggested solutions worked or were acceptable. Modifying ipy.exe.config file has a potential. However, I would like to keep this as simple as possible and minimize the dependencies. So that anyone can grab the IronPython script and run it without having to modify ipy.exe.config file.
So I decided to try the following: I create a new application domain in the script and have AppDomainSetup.ConfigurationFile point to the app.config file. Then I could call AppDomain.DoCallBack and pass a delegate that has my logic. So below is the script that has my attempt. Note that I am just learning IronPython/Python so please keep that in mind.
import clr
import sys
sys.path.append(r"C:\MyApp");
clr.AddReference("System")
clr.AddReference("Foo.dll")
from System import *
from System.IO import Path
from Foo import *
def CallbackAction():
print "Callback action"
baseDirectory = r"C:\MyApp"
domainInfo = AppDomainSetup()
domainInfo.ApplicationBase = baseDirectory
domainInfo.ConfigurationFile = Path.Combine(baseDirectory,"MyApp.exe.config")
appDomain = AppDomain.CreateDomain("Test AppDomain",AppDomain.CurrentDomain.Evidence,domainInfo)
appDomain.DoCallBack(CallbackAction) #THIS LINE CAUSES SERIALIZATION EXCEPTION
Console.WriteLine("Hit any key to exit...")
Console.ReadKey()
In the above code, "c:\MyApp" folder contains everything; exe, dlls, and the app.config file. Hopefully the second appDomain will use MyApp.exe.config. CallbackAction method is intended to contain the code that will use the api from the .NET assemblies to do some work. CallbackAction will be invoked via appDomain.DoCallBack. Well, this is the part I am having a problem. When appDoming.DoCallBack is executed, I get a System.Runtime.Serialization.SerializationException:
Cannot serialize delegates over
unmanaged function pointers, dynamic
methods or methods outside the
delegate creator's assembly.
I can't make a complete sense out of this. I assume that something is being attempted to be serialized across appDomains and that operation is failing. I can create a CrossAppDomainDelegate and execute it just fine.
test = CrossAppDomainDelegate(lambda: CallbackAction())
test()
So does anyone have any ideas or recommendations? Basically, I need to have the assemblies that I want to code against in IronPython to have access to the app.config file.
Thanks for your time and recommendations in advance.
btw I have IronPyhton 2.0.1 installed and am using VS2008 Pro.
This is probably not the answer you are looking for, but.... Since this is for your own testing purposes, I'd recommend installing Ironpython 2.7.1 and see if the problem continues. There have been many improvements to Ironpython since 2.0.1.
I am working on a project which generates an assembly. I just noticed that an additional assembly *.XmlSerializers.dll is being generated. Why this file is auto generated and what it is used for?
In .NET implementation, the XmlSerializer generates a temporary assembly for serializing/deserializing your classes (for performance reasons). It can either be generated on the fly (but it takes time on every execution), or it can be pregenerated during compilation and saved in this assembly you are asking about.
You can change this behaviour in project options (tab Compile -> Advanced Compile Options -> Generate serialization assemblies, Auto or On, respectively). The corresponding element in the project file is GenerateSerializationAssemblies, for example, <GenerateSerializationAssemblies>Auto</GenerateSerializationAssemblies>.
FYI. The exact steps to stop the XmlSerializers.dll from being auto-generated are:
In VS, right-click your project file and select "Properties"
Click the "Build" tab
Change the "Generate serialization assembly" dropdown from "Auto" to "Off"
Rebuild and it will be gone
I think this is the JIT (Just in time) compilation of XML serialisers for performance reasons.
You get the same thing with RegEx instances using the RegexOptions.Compiled option turned on.
I'm no .NET CLR expert, sorry for lack of precise technical detail.
*.XmlSerializers.dll are generated using the Sgen.exe [XML Serializer Generator Tool]
See Sgen.exe on MSDN
Typically the Sgen.exe is used in Post Build events of Projects. See if your project has a post build event which generates the *.XmlSerializers.dll
The project only generates the project.XMLSerialisers.dll for web applications. For other applications you have to run sgen separately.