What is the difference between Assembly Definitions and namespaces in Unity?
Is writing your own namespaces mean you manually add AsmDefs in code?
Is code compile time faster in one or the other, or all the same?
Apples and oranges.
Namespaces allow you to partition code to avoid name collision. You can have multiple per assembly.
Assembly Definitions (ASMDEFs) is a purely Unity creation and simply a means to speed up compilation time. They are best thought of as the equivalent to a project file in Visual Studio in the respect of speeding compilation time. Rather than recompiling all your code, only the ASMDEF whose files were modified are recompiled whilst any follow-on ASMDEFs or non-ASMDEF code are simply “re-linked” (if applicable).
NOTE: Don’t make the mistake of thinking ASMDEFs only benefit the Unity build operation (the thing that takes 10 minutes+). They don’t. They are mainly for decreasing the delays experienced in the Editor where you are waiting to hit the Play button.
Since the output of an ASMDEF is an assembly, naturally whatever you can do in c# code also applies such as confining the code to one or more namespaces.
With regards to compilation times, it doesn’t take much to push Unity over the edge whereby delays of minutes or more are not unheard of. This always surprises new Unity users even those experienced with VS. Unity takes considerable more time to compile the same amount of code compared to VS however there is a subtle explanation as to why. Once you understand the mechanics you realise that comparing compilation times is also apples and oranges.
The reason is that, by default, ASMDEFs are disabled for user code and when Unity decides to compile your code, it does so for all platforms including x86, x64, Editor etc irrespective of your OS bitness or whether you intend to click that Play button. This also applies to any 3rd party code you have.
For this reason, even if you don’t intend to use ASMDEFs in your own code, create a child folder for all the Standard Unity Assets and 3rd party code and create a corresponding ASMDEF for it. Don’t forget to limit the ASMDEF to just the platforms you need for the maximum benefit in speedy compilation.
Related
I'm writing a UWP program to detect colors from LEDs, this program runs on a Raspberry Pi 3 with Windows 10 IoT with attached display.
What the program does is take a reference image with the LED turned off, then take a image from the LED turned on.
Both images are converted to the same pixelformat and then are cropped to a smaller size, in which only the LED is shown (both the reference and the lighted LED).
Then those picture parts are converted to grayscale wich is followed by creating a difference picture of the two, so that only pixels that changed from the reference to the lighted LED are shown.
To do so I use the NuGet-Package portable.AForge.imaging. The code is shown below.
LEDBildNeu = LEDBild.Clone(PixelFormat.Format24bppRgb);
ReferenzbildNeu = Referenzbild.Clone(PixelFormat.Format24bppRgb);
Crop cropping = new Crop(new System.Drawing.Rectangle(Convert.ToInt32(x), Convert.ToInt32(y), 100, 100));
CroppedLED = cropping.Apply(LEDBildNeu);
CroppedReferenz = cropping.Apply(ReferenzbildNeu);
Grayscale grayscale = new Grayscale(0.2125, 0.7154, 0.0721);
GrayscaleReferenz = grayscale.Apply(CroppedReferenz);
GrayscaleLED = grayscale.Apply(CroppedLED);
Difference difference = new Difference(GrayscaleReferenz);
Differenzbild = difference.Apply(GrayscaleLED);
This code works fine as long as im in debug mode, all of the functions are working.
However when i change to release mode, i get this error while building:
1>C:\Users\morsch.nuget\packages\microsoft.net.native.compiler\1.7.2\tools\Microsoft.NetNative.targets(697,5): warning : MCG : warning MCG0007: Unresolved P/Invoke method 'ntdll.dll!memcpy' for method 'System.Byte* AForge.SystemTools.memcpy(System.Byte*, System.Byte*, System.Int32)'. Calling this method would throw exception at runtime. Please make sure the P/Invoke either points to a Windows API allowed in UWP applications, or a native DLL that is part of the package. If for some reason your P/Invoke does not satisify those requirements, please use [DllImport(ExactSpelling=true) to indicate that you understand the implications of using non-UWP APIs.
1>C:\Users\morsch.nuget\packages\microsoft.net.native.compiler\1.7.2\tools\Microsoft.NetNative.targets(697,5): warning : MCG : warning MCG0007: Unresolved P/Invoke method 'ntdll.dll!memset' for method 'System.Byte* AForge.SystemTools.memset(System.Byte*, System.Int32, System.Int32)'. Calling this method would throw exception at runtime. Please make sure the P/Invoke either points to a Windows API allowed in UWP applications, or a native DLL that is part of the package. If for some reason your P/Invoke does not satisify those requirements, please use [DllImport(ExactSpelling=true) to indicate that you understand the implications of using non-UWP APIs.
When I run the code in release mode and get to the part where the difference picture is created, I get the exception
System.TypeLoadException: 'Unresolved P/Invoke method 'memcpy!ntdll.dll' from this method. Please look for this method in build warnings for more details.'
According to this 'memset' and 'memcpy' are not supported by UWP. My questions now are:
Why does the program run in debug mode without any problems even when those two entry points are not supported, but as soon as i turn to release mode i get the exceptions?
Is there a workaround for the problem?
I already tried to use
[DllImport("ntdll.dll", EntryPoint = "memset")]
and
[DllImport("ntdll.dll", EntryPoint = "memcpy")]
But either I did it wrong or it just don't work that way.
I know I could just program a workaround in which I check the pixels manually and create a new image, but I wanted to solve that problem if possible.
Finding the correct combination of directives can be a very frustrating and time consuming process. Here is additional information that I received from Microsoft via email, hope this helps:
Helpful links:
https://devblogs.microsoft.com/dotnet/net-native-deep-dive-dynamic-features-in-static-code/
https://learn.microsoft.com/en-us/dotnet/framework/net-native/runtime-directives-rd-xml-configuration-file-reference
https://learn.microsoft.com/en-us/dotnet/framework/net-native/runtime-directive-policy-settings
The analysis we do to get your application ready to be ahead of time compiled is quite extensive. We need to generate code for various generic types, reflection callable wrappers, serialization information, marshalling stubs etc etc. In come cases (as you could imagine) we end up generating more than is strictly necessary due to run away combinatorics. It’s completely possible that some fiddling with our heuristics can get you application to a place where it compiles without any loss of functionality.
Practically speaking, there’s two ways to manipulate the behavior of the compiler. One is through some of our compiler flags available through dropping elements into your csproj. The other is making edits to your applications Properties\Default.rd.xml file.
Compiler flags
There are a wide range of flags available but here’s a couple that may help out:
<ShortcutGenericAnalysis>true</ShortcutGenericAnalysis> - Can help stop runaway analysis of generic types and reduce overall generation requirements.
<UseDotNetNativeSharedAssemblyFrameworkPackage>false</UseDotNetNativeSharedAssemblyFrameworkPackage> - Eliminates one of the linking boundaries the compiler has to fight with. I actually suspect turning this off will make things worse not better but whole program optimizers are hard to reason about but rebuilds are cheap enough to try.
Runtime Directives
There’s lots of reading above but the tl;dr is that this file is read by the compiler and can contain lots of hints about what we want it to do or ignore etc. The overall syntax of the file is also included in the reading above but I don’t think we’re very clear about the one special directive that’s include by default:
<Assembly Name="*Application*" Dynamic="Required All" />
This directive says: “Please save/generate enough information so that all user types can be inspected and created via reflection.” Where ‘user types’ means any type in an assembly that isn’t signed with the .NET key token. So, basically everything that isn’t explicitly .NET Framework. This in leads to lots of bloat but also makes it so most folks don’t ever have to think about these things. In cases where we don’t have enough information, you’ll get runtime exceptions like MissingMetadataException or TypeLoadException or NullReferenceException. Each instance will require a bit of code inspection and fiddling with directives to get patched up. This can be an annoying a fragile process. All that said, the analysis engine is quite sophisticated and you’ll get lots and lots of things ‘for free’ without the special directive or any hassle. It’s entirely possible that your app runs great with just a little bit of tweaking.
Okay, the goal now is to remove this directive but still have a working application. There’s two approaches that have tradeoffs, so I’ll describe both and let you decide if either methodology suits you. Roughly here’s what the two workflows look like:
Start from nothing.
a. Remove the special Application directive
b. Build the app
c. If the build fails, email us, else…
d. Test the app and see if you hit any runtime errors
e. If you do you’ll need to look at the error location and see if adding some directives can help then head back to (b).
f. If you find no errors, you’re done! Hooray!
Start from everything
a. Remove the special Application directive
b. Get a list of the full set of dlls for your project, for example by inspecting here: obj[architecture]\Release\ilc\in
c. For each dll, add a Dynamic directive. They’ll look like: <Assembly Name="ASSEMBLYNAMEWITHOUTEXTENTION" Dynamic="Required All"/>
d. Comment out some subset of these libraries
e. Build the app
f. If the build fails again in RHBIND go to (d)
g. Test the app and see if you hit any runtime errors
h. If you do you’ll need to look at the error location and see if adding some directives can help then head back to (e)
i. If you find no errors, you’re done! Hooray!
I found a solution which worked:
Instead of downloading the portable.AForge package with NuGet i downloaded the portable.AForge from GitHub.
Find the .cs-file called SystemTools.cs (located in AForge/Sources/Core/).
Open it with any .cs editing porgram, now search for all code like
#if !MONO
...
#else
and remove it.
This clears the use of memcpu() or memset() from ntdll.dll.
Save the SystemTools.cs, create the library and add the AForge-Package manually to the application.
After the change it worked without any problems.
While working on tAPI (a Terraria (if you don't know the game, just look it up, it's awesome) modding API), we hit a problem with cross-mod code references. Basically, you can call methods from the other mod (as long as you tell the mod builder to add references to that mod obviously), but the moment you try to reload the mods, you can't really be sure stuff will work: if you reference any class from the other mod, it will always use the FIRST version the game loaded, not the latest. That means you can't call anything static or instantiate the classes from the other mod, which is a real problem when you're working on some kind of an API mod.
The mods themselves are standard C# code, with some extra JSON data we parse.
I tried doing the same thing outside of the tAPI source code, here is my take at it (long code, told pastebin.com to never expire the paste though): http://pastebin.com/hjY57xJh
It has exactly the same problem though. The second time the static method is called, it should print "456", not "123" like it does.
So, my question is: can I do anything to actually force the game to use the latest assembly and not the first one?
We thought about using AppDomains, but they're a pain to implement and as I understand it, they work by serializing the data going through, which might make the game plainly unplayable due to the amount of calls it has to do to the mods.
EDIT: I don't really mind the fact, that the assembly still uses up memory after "reloading" mods. I just want the new assembly after the "reload" to take priority over the old one.
I totally forgot about this question, but: in the end we kinda ignored the problem. tAPI mods now either extend ModBase or APIModBase. If you're using the latter, the game will never really try to even reload the mod (assembly) - it will just stay the same after the first load, until you restart the game. It's a little annoying for modders, but there isn't much that can be done about this fact. At least all the content mods that only depend on API mods can be freely reloaded without causing any problems.
I am embedding IronPython into my game engine, where you can attach scripts to objects. I don't want scripts to be able to just access the CLR whenever they want, because then they could pretty much do anything.
Having random scripts, especially if downloaded from the internet, being able to open internet connections, access the users HDD, or modify the internal game state is a very bad thing.
Normally people would just suggest, "Use a seperate AppDomain". However, unless I am severely mistaken, cross-AppDomains are slow. Very slow. Too slow for a game engine. So I am looking at alternatives.
I thought about compiling a custom version of IronPython that stops you from being able import clr or any namespace, thus limiting it to the standard library.
The option I would rather go with goes along the following lines:
__builtins__.__import__ = None #Stops imports working
reload = None #Stops reloading working (specifically stops them reloading builtins
#giving back an unbroken __import___!
I read this in another stack overflow post.
Assume that instead of setting __ builtins_._ import__ to none, I instead set it to a custom function that lets you load the standard API.
The question is, using the method outlined above, would there be any way for a script to be able to be able to get access to the clr module, the .net BCL, or anything else that could potentially do bad things? Or should I go with modifying the source? A third option?
The only way to guarantee it is to use an AppDomain. I don't know what the performance hit is; it depends on your use case, so you should measure it first to make sure that it actually is too slow.
If you only need a best-effort system, and if the scripts don't need to import anything, ever, and you supply all of the objects they need from the host, then your scheme should be acceptable. You can also avoid shipping the Python standard library, which will save some space.
You'll want to check the rest of the builtins for anything that might talk to the outside world; open, file, input, raw_input, and execfile come to mind, but there may be others. exec might be an issue as well, and as it's a keyword it might be trickier to turn off if there are openings there. Never underestimate the ability of a determined attacker!
I have embedded Iron Python in apps before and shared similar security concerns. What I did to help mitigate the risk was to create special objects just for the scripting run-time that were essentially wrappers around my core objects that only exposed "safe" functionality.
Another benefit from creating objects just for scripting is that you can optimize them for scripting with helper functions that make your scripts more terse and tidy.
Appdomain or not, there is nothing stopping somebody from loading an external .py module in their script.... Its a price you pay for the flexibility.
We have a program that is used in one specific industry and has strings that are specific to that industry. We now have the situation where it can be used in another industry and we want to customise the strings for that industry without duplicating our code base.
The problem space appears very similar to localisation. Are we going to have a separate resource assembly for each industry? If so when would we choose which assembly to use, could we do this at install time or would it need to be at compile time?. How do we keep the separate resource assemblies synchronised, so that the same keys to messages appear in each one?
What is the best way to do this?
Let me re-phrase it: you have an industrial application which could be used in various industries and the only things that are different are resources (that is strings, layout, maybe images and sounds). The other code stays the same.
In such case your problem is not just similar it is actually identical to Localization. And as such you can use Satellite Assemblies.
Now, it is up to you if you want to package such created applications separately or distribute one application with both problem spaces.
The first seem more realistic scenario to me - you would need to decide on which .resx file to include at compile time (i.e. during project preparation you would overwrite existing resources with problem-space resources and then proceed with compilation, that should give you different flavors of your application; I would also modify their names in such case).
The latter would require you to manually instantiate ResourceManager at runtime to read from valid satellite assembly - it could be based on some configuration file. It means more work (you would need to actually modify your code) and you will end up distributing both flavors of your application at once, that is you won't have control over how your customers will use it. From the business perspective it could be a little dangerous.
EDIT (Note to self: read whole question carefully)
Somehow I managed to miss install time vs. compile time. I believe compile time is the answer because of the same reason I gave in config-driven switch section: you would package the resources and you won't have any control on how customers use it. Some clever guy would figure it out, that is for sure.
I would recommend having a properties file with key value pairs. Where you currently have industry specific strings, replace them with calls to the properties file. Obviously you would cache these strings in some container. I don't know the C# container - Java would use java.util.Properties.
aerospace.props:
INDUSTRY_NAME=aerospace
INDUSTRY_START_YEAR=1903
manufacturing.props:
INDUSTRY_NAME=manufacturing
INDUSTRY_START_YEAR=1600
So Jeff Atwood rightly complained about Visual Studio not performing background compilation see: http://www.codinghorror.com/blog/2007/05/c-and-the-compilation-tax.html
The solution from most sources seems to be Reshaper which will incrementally perform background compilation as you write. This leads to their great realtime re-factoring tips and error detection.
But what I don't understand is with R# continually compiling my code, why does it take so long when executing a compilation via VS (i.e. Ctrl + Shift + B or similar). What I mean by this is, if R# has already compiled my code then why would I need a recompilation?
My assumption is of course that R# is not overriding the assemblies in my bin directories but instead holding the compilation results in memory. In which case, is it possible to tell R# to simply override my assemblies when compilation is successful?
I don't know about "rightly complained" - that's an opinion I happen to disagree with:)
However, the VB.NET (and probably Resharper c#) background compilers do not actually compile full assemblies - they cannot! If you think about it, the natural state of your code while you are working is not compilable! Almost every keystroke puts your code in an invalid state. Think of this line:
var x = new Something();
As you type this, from the key "v" to the key ")", your code is "wrong". Or what if you are referencing a method you haven't defined yet? And if this code is in an assembly that another assembly requires, how would you compile that second assembly at all, background or not?
The background compilers get around this by compiling small chunks of your code into multiple transient "assemblies" that are actually just metadata holders - really, they don't care about the actual effects of the code as much as the symbols defined, used, etc. When you finally hit build, the actual full assemblies still need to be built.
So no, I don't believe it's possible because they're not built to do actual full compilation - they are built to check your code and interpret symbols on the fly.
Reshaper which will incrementally perform background compilation as you write
It doesn't, it just parses the source code. The exact same thing Visual Studio already does if you don't have Resharper, that's how it implements IntelliSense, its own refactoring features and commands like GoTo Definition and Find All References. Visual Studio also parses in the background, updating its data while you type. Resharper just implements more bells and whistles with that parsing data.
Going from parsing the code to actually generating the assembly is a pretty major step. The internal format of an assembly is too convoluted to allow this to happen in the background without affecting the responsiveness of the machine.
And the C# compiler is still a large chunk of unmanaged C++ code that is independent from the IDE. An inevitable consequence of having to have the compiler first. It is however a stated goal for the next version of C# to provide compile-on-demand services. Getting true background compilation is a possibility.
I don't really have an answer but I just wanted to say that I have been using Eclipse and Java for 4 months now and I love the automatic compilation. I have a very large java code base and compilation happens constantly as I save code changes. When I hit Run everything is ready to go! It's just awesome. It also deploys to the local web server instance (Tomcat in my case) automatically as I make code changes. All this is setup by default in Eclipse.
I hope Microsoft does something similar with .net in the near future.