.NET 4.5, 64bit Win8
I have two MSMQ running under WCF and hosted under Windows Activation Service.
Execute MSMQ: Responsible for calling an unmanaged function and obtaining a pointer IntPtr to 4GB array in addition to other limited size variables. This array is not that important and shouldn't be blocking the queue until it is passed to the DB and having this queue free is a priority. Passing the pointer to the Save MSMQ (Hypothesis).
Save MSMQ: Should save the array to the DB across the network then deallocate it from unmanaged memory.
The problem here would be:
Making sure the array is deallocated on poison messages and other queue errors. This case is the easy bit.
Passing an IntPtr cross app domains which might not be possible, as I recall, from the old good days, that pointer values are relative (and the word offsetting comes to mind, or maybe I am mixing processes and AppDomains) to app domains and for security reasons .NET might not allow this? Is this possible or am I dreaming, also is this a cross domain issue or does my problem have a different title?
N.B. I have used NetPipes communication before, but this has a queue in the middle, so NetPipes would not be a good solution.
Related
I want to use 2 C# apps to communicate to each other by memory. For the host side I used:
int* ptr = &level;
And for the client side I want to use:
ReadProcessMemory((int)_handle, <returned result of int* ptr>, _buffer, 2, ref bytesRead);
But the ReadProcessMemory doesn't work. For example: level set to 3 but ReadProcessMemory returns 0. What the hell out of this? (NOTE: "level" field is not cleared from memory)
I tried int* ptr lots of times because lots of websites tell me to do that but that doesn't work so well with ReadProcessMemory.
I set level = 3 but the result of ReadProcessMemory of level = 0
What you ask, in the way you ask is pretty much dangerous as the process is entirely managed by the CLR. Depending what you want to share and how, you could consider sockets or pipes.
Alternatively, you could use interop, but it requires a certain expertise and tinkering in my opinion.
The cleanest way for two C# applications to communicate via memory is by use memory mapped files. Messing with the memory in a managed process can get subtle issues. Memory mapped files are a way to share information.
Keep in mind that each memory mapped file is loaded at different memory addresses, therefore you need to structure it without the use of absolute pointers.
Edit:
Direct raw memory access requires knowing the exact physical address to access as the virtual addresses allocated in the target process are different and possibly overlapping with those of the source process. C# applications are hosted by the Common Language Runtime which is in control of everything, including memory allocation. In particular, a standard c# application di not manage a bit of its own memory: as the runtime moves the objects as part of the normal application lifetime, such addresses change over time.
If you are in control of the target application, you can pin the object via the GC class to forbid movements, then you have to take the address of the object, and pass it to the other process. The other process must then open the target process for reading, mapping the memory segments, calculate the location of the memory to read by translating the virtual address.
What you ask requires cooperating processes and a lot of low level knowledge, and in the end, you also might never be able to read updated memory changes, as the CLR might not write back the values to memory (have a look to volatile for this).
It is clearing exciting to write such software, but when you are in control of both the applications, there are cleaner and much more reliable ways to achieve your goal.
As a side note, this technique is used by trainers, hacker tools and viruses, therfore antivirus softwares will raise red flags when they see such behavior.
I've been working on an internal developer tool on and off for a few weeks now, but I'm running into an ugly stumbling block I haven't managed to find a good solution for. I'm hoping someone can offer some ideas or guidance on the best ways to use the existing frameworks in .NET.
Background: the purpose of this tool is to load multiple different types of log files (Windows Event Log, IIS, SQL trace, etc.) to the same database table so they can be sorted and examined together. My personal goal is to make the entire thing streamlined so that we only make a single pass and do not cache the entire log either in memory or to disk. This is important when log files reach hundreds of MB or into the GB range. Fast performance is good, but slow and unobtrusive (allowing you to work on something else in the meantime) is better than running faster but monopolizing the system in the process, so I've focused on minimizing RAM and disk usage.
I've iterated through a few different designs so far trying to boil it down to something simple. I want the core of the log parser--the part that has to interact with any outside library or file to actually read the data--to be as simple as possible and conform to a standard interface, so that adding support for a new format is as easy as possible. Currently, the parse method returns an IEnumerable<Item> where Item is a custom struct, and I use yield return to minimize the amount of buffering.
However, we quickly run into some ugly constraints: the libraries provided (generally by Microsoft) to process these file formats. The biggest and ugliest problem: one of these libraries only works in 64-bit. Another one (Microsoft.SqlServer.Management.Trace TraceFile for SSMS logs) only works in 32-bit. As we all know, you can't mix and match 32- and 64-bit code. Since the entire point of this exercise is to have one utility that can handle any format, we need to have a separate child process (which in this case is handling the 32-bit-only portion).
The end result is that I need the 64-bit main process to start up a 32-bit child, provide it with the information needed to parse the log file, and stream the data back in some way that doesn't require buffering the entire contents to memory or disk. At first I tried using stdout, but that fell apart with any significant amount of data. I've tried using WCF, but it's really not designed to handle the "service" being a child of the "client", and it's difficult to get them synchronized backwards from how they want to work, plus I don't know if I can actually make them stream data correctly. I don't want to use a mechanism that opens up unsecured network ports or that could accidentally crosstalk if someone runs more than one instance (I want that scenario to work normally--each 64-bit main process would spawn and run its own child). Ideally, I want the core of the parser running in the 32-bit child to look the same as the core of a parser running in the 64-bit parent, but I don't know if it's even possible to continue using yield return, even with some wrapper in place to help manage the IPC. Is there any existing framework in .NET that makes this relatively easy?
WCF does have a P2P mode however if all your processes are local machine you are better off with IPC such as named pipes due to the latter running in Kernel Mode and does not have the messaging overhead of the former.
Failing that you could try COM which should not have a problem talking between 32 and 64 bit processes. - Tell me more
In case anyone stumbles across this, I'll post the solution that we eventually settled on. The key was to redefine the inter-process WCF service interface to be different from the intra-process IEnumerable interface. Instead of attempting to yield return across process boundaries, we stuck a proxy layer in between that uses an enumerator, so we can call a "give me an item" method over and over again. It's likely this has more performance overhead than a true streaming solution, since there's a method call for every item, but it does seem to get the job done, and it doesn't leak or consume memory.
We did follow Micky's suggestion of using named pipes, but still within WCF. We're also using named semaphores to coordinate the two processes, so we don't attempt to make service calls until the "child service" has finished starting up.
I have a service that's using ASP WebApi. Each http request translates to a thread that needs to do some data manipulation (possibly changing the data). The API layer is written in C# and the data manipulation is written in C++. The C# layer calls the native library and supplies a pointer to some managed buffer.
Couple of questions:
How can I make sure there are no races? is std::mutex in the native library enough in this case? (do managed threads map to native threads? will they share the same std::mutex?)
How can I make sure that the GC doesn't release the pointer to the managed buffer while the native library is manipulating it?
Do you need a shared buffer? If the buffer is only ever used on one thread, you save yourself a lot of trouble. Managed threads do not map to native threads 1:1, but I'm not sure if that has any effect on your scenario.
You need to fix the buffer, and keep it fixed the whole time the native code has a pointer to it - releasing is the least of your worries, the .NET memory is moved around all the time. This is done using the fixed block.
Fixing managed memory:
byte[] theBuffer = new byte[256];
fixed (byte* ptr = &theBuffer[0])
{
// The pointer is now fixed - the GC is prohibited from moving the memory
TheNativeFunction(ptr);
}
// Unfixed again
However, note that prohibiting the GC from moving memory around can cause you quite a bit of trouble - it can prevent heap compaction altogether in a high-throughput server, for example.
If you don't need to work with the memory in the managed environment, you can simply allocate unmanaged memory for the task, such as by using Marshal.AllocHGlobal.
.NET 4.5, Win 8, 64bit
I have an IntPtr to 4 GB unmanaged array. I want to pass it async to a WCF NetPipes Windows Service, which is supposed to read it, push it to DB then deallocate it.
I am aware that the caller and the callee are on two different appdomains, so this might not be possible.
Is this possible and how?
Ofcourse, I can copy the data to a managed array then pass the managed array to the service, but this is not good performance-wise.
N.B. I asked a question of how to pass an IntPtr on a queue Passing an unmanaged pointer between two AppDomains (By indirect call) , the answer to this question and the other question will help me taking a design decision.
Short version:
Could passing the handle from GlobalAlloc(GMEM_MOVEABLE, Size) to Marshal.PtrToStructure() and Marshal.FreeHGlobal() cause memory corruption?
Long version:
I'm using Windows Global memory allocation to pass a data structure back and forth between a Delphi and C# application (the fact that it's Delphi isn't really significant to this question, because it's just Win32 API calls).
On the Delphi side, I pass in a record, it allocates the space, locks the memory, copies the structure into memory, and the unlocks the memory:
function MarshalRec(SourceRec: TInteropItemRec): THandle;
var
Size: integer;
Buffer: Pointer;
begin
Size := sizeof(SourceRec);
result := GlobalAlloc(GMEM_MOVEABLE and GMEM_ZEROINIT, Size);
Buffer := GlobalLock(result);
try
CopyMemory(Buffer, #SourceRec, Size);
finally
GlobalUnlock(result);
end;
end;
On the C# side, it gets that THandle (which is basically an unsigned int) into an IntPtr and uses Marshal.PtrToStructure to copy the data into the C# structure:
public void FromMemory(IntPtr Source)
{
Marshal.PtrToStructure(Source, this);
Marshal.FreeHGlobal(Source);
}
The problem I'm running into, is very rarely (as in 4 times over 6 months for me), the whole application goes down (This application has encountered an error and must close). If I try to pause execution in Visual Studio, I get "A fatal error has occurred and debugging needs to be terminated. For more details, please see the Microsoft Help and Support web site. HRESULT=0x80131c08."
Anyway, we managed to get a couple logs of it happening, and in both cases, it showed a recent call to that "MarshalRec" function above, a few other function calls, and then some processing of Windows messages on the event loop on the Delphi thread (yeah, it has its own thread and event loop to deal with a time sensitive device driver).
So my suspicion is falling on the GMEM_MOVEABLE flag to GlobalAlloc. I couldn't find anything in the Marshal class that does the GlobalLock and GlobalUnlock stuff, so I had assumed it was handled internally by PtrToStructure().
Does PtrToStructure deal properly with a handle, or does it need an actual pointer obtained from GlobalLock()? Is it possible that in very rare cases, Windows happens to move the memory I allocated, which means I need to call GlobalLock() to get an actual pointer to pass in? That FreeHGlobal is actually freeing something it shouldn't have, which brings down the whole app the next time that resource is accessed?
And if so, should changing the GMEM_MOVABLE to GMEM_FIXED prevent this from happening again? Or do I need to DllImport GlobalLock() and GlobalUnlock()?
I'm tempted to just blindly make those changes, but given the non-reproducibility of this issue, there's no way to tell if it's fixed until it happens again. So I'm looking for feedback on whether this code could lead to the symptoms I'm seeing, or if I need to start coming up with other theories.
Well, you are explicitly violating the contract for GlobalAlloc(). Your hope that Marshal.PtrToStructure() will call GlobalLock is unfounded, it has no way to tell whether the passed IntPtr is a handle or a pointer.
GlobalAlloc is a fairly hopelessly outdated legacy function from the Windows 3.x era. Yes, it is quite likely to return the address as the handle value with GlobalLock() being a no-op. But it is certainly not documented to do this. CoTaskMemAlloc() is the better mouse trap.