We have a C# .NET application which binds to an unmanaged C dll.
The C dll uses CreateFileMapping() and MapViewOfFileEx() to share memory with other processes on the same machine.
MapViewOfFileEx() uses the last parameter (lpBaseAddress) and set it to 0x08000000 in our case. This is fine for all c/c++ applications if you call the initializing function in the dll early enough.
In c#, however, bigger application allocate a lot of memory before we even get the chance to call the dll's init function in the autogenerated static method Main(). (Small c# console apps work fine).
Is there a way to like preallocate the area 0x08000000 to 0x08400000 at startup of the .NET application before the automatic allocation for classes and stuff kicks in? Then it should be no problem to map the shared memory to 0x08000000 any time we like.
Restrictions:
The obvious solutions cannot be applied here.
It is not possible to use a dynamically assigned virtual address by calling MapViewOfFile as this would require heavy changes in the C dll which currently works with absolute pointers in all processes.
It is also not possible to negotiate a common base address between all processes, as they don't startup at the same time.
Related
Okay, I messed something up. I've written in C++ a DLL which I call from the managed code (C# .NET). The library works like diamonds and is blazingly fast.
My DLL uses its internal state i.e. allocates heaps of memory and uses myriad of variables which are not cleared off between the calls from .NET. Instead they stay there and C# code is aware of that (there is preprocessing and building data structures), actually this is required for performance.
So what is the problem?
I want to add multi-threading, effectively by allowing each .NET thread access his own DLL. Without storing any data between the calls it would be easy achievable with just one DLL.
But in my case, do I have to copy the *.DLL the number of times equal to the number of threads and write pInvoke wrapper for each file separately?? :O I mean [DllImport(...)] for each out of like 40 functions?
No way, there must be something more clever. Help?
Simply put you need to stop sharing variables between threads.
Your global variables are the problem. Instead you need each different thread to have its own copy of the state that persists between calls. Typically you would put this state into a structure of some sort, perhaps a struct. Then an initial call to the DLL would return a new instance of this structure. You then pass that structure back into the DLL every time you call a function that requires access to the persistent state. When you are done, call back into the DLL to deallocate the structure. You don't need to declare the structure in the managed code. You can just treat it as an opaque pointer. Use IntPtr.
Of course, perhaps you'd just be better off with a C++/CLI assembly.
I wish to understand some unexpected behaviour in the running of COM DLLs, where is appears static C++ data is being shared across multiple processes. The environment is a little complex and my understanding of the various COM threading models rather weak, which I am hoping someone can help with.
The environment
An IIS Server on a 64 bit OS running multiple C# web services, with each service in its own 32-bit application pool, and therefore process
The pools have the "Enable 32-Bit Applications"=True setting
each 32 bit C# service calls a different in-process 32 bit COM DLL (so service A calls COM DLL 1, service C calls COM DLL 2. The COM DLLs are written in C++ using Qt 4.8 ActiveQt
The COM DLLs depend on a number of 32 bit C++ DLLs, which are shared i.e. both COM DLLs 1 and 2 depend on Utilities.dll
As far as I can tell, there is no ThreadingModel set for the COM DLLs, so I am expecting the system will fall back on the main STA.
I am aware this is frowned upon, but I do not have enough knowledge to change it currently.
Utilities.dll contains some static C++ data
The COM DLLs were registered using "regsvr32" and do not appear to be listed in "Component Services", though my knowledge of the latter is minimal.
The observed issue is that the static data in Utilities.dll appears to end up being shared between the different IIS processes, with undesirable consequences. I had expected that as the COM were in the main STA they would be accessed as if it were not thread safe, that each process would get its own copy of the DLL static data, but this appears not to be the case.
Can someone explain how the static data ends up shared between processes?
How can I avoid this? (apart from refactoring the code to remove all static data, which is not really viable currently)
If you are seeing data shared between COM objects, it means they are hosted in the same process. Yes, it's possible to share data between processes, but not accidentally. Since your app pools are different processes, it must be that these COM objects are hosted out of process, and it's just stubs that are loaded into the app pool.
If you have control of the Utilities.dll (and it sounds like you do) I would try adding some debugging information to find out what process id is hosting the COM objects. I expect you'll find that it doesn't match the app pool id, and you'll be able to use that id to find out what's going on.
Ideally it shouldn't matter where well-designed COM objects live, that's supposed to be something of an implementation detail. Is it possible to do away with the shared data structures?
The reason I ask is that I have an application that (among other things) calls a MATLAB .NET component whenever data is written into a specific file. The component reads the file and creates an image out of the data contained within. This works fine.
However when I am using the underlying application to additionally process a "significant" amount of data and display the processed data in table, the call to MATLAB throws an out of memory exception, but only when I am processing this large amount of data.
Is this not an indication that the MATLAB process called will rely on the available memory of the application? I guess I just don't understand how the MATLAB memory works when being called from a .NET standpoint.
(I should also note that I call clear all before every call to the MATLAB function in an attempt to "start from scratch", but it fails regardless)
COM components built by Matlab Builder NE are in-process COM servers. This means that they are DLLs which are loaded into your application memory space. This means that the MCR, which is a kind of Matlab-Virtual-Machine is in your memory space.
I believe that .NET components should behave exactly the same.
It is entirely possible, and from what you've described, even likely that the MATLAB component is making use of unmanaged memory (memory that is not managed by the .NET garbage collector.) There is very little you can do about this, apart from ensuring that you only feed it expected data in expected quantities. You may also wish to create a support ticket with MATLAB, if you believe you are using it properly.
Never used MATLAB from C#, but as much I see it uses COM components to interact with the CLR world. You load the MATLAB unmanaged DLL's into your process memory heap. And considering that for CLR process on 32 bit machines you have approximately 1.2 GB memory space, so you go out of that available space.
Some interesting description of how the load of the unmanged COM component into the managed memory done, you can find here: Memory Management Of Unmanaged Component By CLR
there's a WinForms-application written in C# using .NET Framework 3.5. This application uses a C++ Dll which is imported using the following declaration:
[DllImport(DllName)]
public static unsafe extern int LoadDBData(String dsn, String userid, String password);
This method imports data from a given ODBC-DSN using a SQL Server database. The call crashes when there is too much data in database. The provider of this extern dll said this happens because the dll is unable to grab more heap size and my application should provide more heap memory.
How could I solve this problem? As far as I know the only possibility to exclude a component from automatic garbage collection is the unsafe keyword which I already used.
Any idea would be appreciated.
Thanks in advance
Martin
This seems like a problem with the vendor's library, rather than your code.
Managed and unmanaged memory should be considered to be completely separate. Managed memory is typically memory allocated on a garbage-collected
heap, while unmanaged memory is anything else: the ANSI C memory pool
allocated through malloc(3), custom memory pools, and
garbage-allocated heaps outside the control of the CLI implementation...
Note that the above quote is from the Mono documentation, but I believe (if I'm not mistaken) the same is true for .NET in general. If the data is being loaded in the DLL's internal data structures, then it should allocate its own memory. If you're providing a buffer which will get filled up with the data, then it will only get filled up with as much data as you've allocated for the buffer (and pinned before marshalling). So where is the data being loaded?
You can't increase the heap size in .NET.
You could create an EXE in c/c++ that your .NET app calls using Process.Start.
Your c/c++ EXE would just call the DLL function and return the result (or if you have more than one function it could take a command line parameter). If you don't want a separate EXE you could try using RunDll32 instead.
I doubt this is specific to .NET, managed memory, garbage collection etc. It's a native DLL so it uses regular, unmanaged memory. Of course, the .NET runtime will also use it's share of memory but a native application using the DLL would do the same.
If you're running in a 32 bit process, the total heap size for .NET and unmanaged code can be limited to 1.5 GB. It's difficult to tell without additional information, but you might have hit that limit.
So one option would be to ask your vendor, whether they have a 64 bit version of the library and switch to a 64 process. In a 64 bit process, memory is almost unlimited (according to today's standard).
i want to write a C# lib, or a reference service,
so that if a application reference my lib, and when the application runs,
the function in my function can be run without any invoked?
for example, i got a lib, keep reading the memory usage of the platform,
when the memory was up to 80%, i pop up a message to the user, it is time to close the application.
so..when i create a another application, and want my new application has the mem check feature, i just need to reference it then after my new application run, the mem check will be fired together..
so..how can i do these?
Create a static constructor for your class - this constructor will be run once, and only once, the first time the host application references anything related to your class.
You could create a thread to run some background checking process, hook into an event of the Application object, or whatever you need.
Note however, that you'll need to do more than just add a project reference to your assembly - an assembly is only loaded into an App Domain when something in that assembly is referenced by existing code.
The term you're looking for is "Win32 application hook." There's a decent introduction to managed hook injection here: http://msdn.microsoft.com/en-us/magazine/cc188966.aspx
I'm not sure you can monitor system memory consumption with an application hook, though. And unfortunately, you cannot write a global hook in managed code because global hooks require a well-defined entry point (a DLL export). This is really something best suited for C++.
It is possible to create traditional DLL exports in an assembly (via ILASM or the now defunct Delphi.NET) but not really recommended. (search for reverse p/invoke)
Another approach is to create a C++/CLI intermediate dll to call your managed code. But to be perfectly honest, I reckon you are simply better off just creating a native DLL with C++ or Delphi.
Edit:
Ok, firstly a disclaimer, I work for Quest Software (the company that makes this tool that I am about to plug). That said...
OS Monitoring is actually not as straight forward as you might think, things like memory consumption, process monitoring etc is...well, pernickety. You may find that somthing like Spotlight on Windows (Freeware for first 10 licences) would suit your purpose ?