Develop Reference

C#: keeping variables in Visual Studio's memory between program compiles

I have a computationally intense program that starts by filling a very large array using data from a binary file. The program then goes on to manipulate this data, etc. Loading my variable from the bin file is the most time consuming step.
Is there a way to keep the variable in Visual Studio's memory so I can just edit/recompile the meat of the program, without repeating this loading process each time?
I'm coming from Matlab, where I could load a variable into the workspace and then use it as much as I wanted from any script until I closed the Matlab environment.

I don't think that's possible. You will have to load it to memory everytime you compile. Matlab is very different from C#. As you said it runs scripts under a closed environment.
C# compiles a standalone program that communicates with Visual Studio (debug version), but doesn't belong to VS.

I don't think you can do it with Visual Studio, but you might consider using an in memory cache like redis to store your data. It's a bit of a pain to set up on Windows, so I recommend getting it via Chocolatey. Once it's running StackExchange.Redis is the go to provider.

loading huge data into memory is not time consuming, processing it is time consuming.
loading huge files mostly depends on read speed of disk drive. if you have SSD that should not take more than few seconds.
anyway, you can always store your data in form of json or somewhat contrived way, store binary form of your data (deserialized) into disk.
you will load your baked data from disk, rather than getting raw data and processing it again.

QuickFix/N logging performance

Has anyone tried creating their own logger in c# as per:
https://github.com/connamara/quickfixn/blob/master/QuickFIXn/FileLog.cs
I am testing this and I don't seem to be able to increase logging speed - there appears to be a finite amount of time taken in getting to:
public void OnIncoming(string msg){}
Or I am doing something wrong.
In my version, I purely timestamp the message and add to list in memory and write to file later on a background thread. However, the time between log entries appears same as if I wrote straight to disk. So, I am wondering if there is much hidden processing prior to this call.
I have disabled "Crack" - so it is not doing that.

I seems your processing time is not taken by logging, but in your program. Doing an add to a list should be faster than writing it to disk. However if your program does much processing you will only notice a small difference in the overall performance. You can add a profiler to find where the time is taken.

Finding file locks in legacy c# code

I have an interesting problem - I've inherited a large code base (brown field).
The application runs on a schedule and takes a large amount of data files (text) in, processes them, and then exports a report and cleans up.
There is a bug that has been discovered whereby when trying to clean up afterwards, some files are left in a locked state, even though all file activity has long gone out of scope. This stops the application from being able to delete them during clean up.
There are literally hundreds of IO and stream objects etc being used in this application, and I'm wanting to find out where to start looking to save reviewing every instance of their use.
What are some good tools for investigating File locks in c# managed code, and how do you use them to do so?

This happens normally when you forgot to dispose the parent object that owns a file handle. E.g. you forgot to call Close/Dispose to a FileStream. Then the finalizer will clean up the file handles when they are no longer referenced during the next full GC.
You can check with Windbg if you have SafeFileHandles in the finalization queue ready for finalization. A profiler which can track such things is e.g. YourKit which can when you enable probes also search for files closed in the finalizer and gives you the creation call stack which gives you the ability to search in your code for the offending line.
Check out the process Inspection tab of YourKit to find the probe check.

You can monitor file access (read/write) using ProcMon from SysInternals.
Its not specific to c# but a general tool that can be used for many other things. Note you can export the results to csv and investigate it later.
You can use one of the following guides:
Detailed Windows I/O: Process Monitor - How to do simple file monitoring.
Using Process Monitor to Monitor File Access - More detailed guide explaining how to export the results into a csv you can investigate later.
Edit:
I didn't found anything for this purpose, so if I was you I would inherit from the steam used, and wrap it with logging logic.
This logging stream object, for example named LogStream will write log before each method entrance, call the base.function() and write another log when done.
This way you can monitor the file access as you wish. For example, logging each stream instance an Id using Guid.NewGuid(), logging Thread Id using System.Threading.Thread.CurrentThread.ManagedThreadId etc.
This way you can identify the instances and slowly investigate the calls.
A point to start is to check whether there is equal number both stream open and close, an exception might avoided one of the Dispose() calls.

put in and get out of clipboard in a loop without delays

I'm using the following code to copy text to Clipboard.
System.Windows.Forms.SendKeys.SendWait("^c");
Then I use
Clipboard.GetText()
to get the text from Clipboard. It works fine, but it looks like it's delaying when I work with clipboard in a loop and I get content that should be overwritten with next copied text. If I put the Thread.sleep, it works fine. How could I fast copy and get the right content from Clipboard in a loop without delay?

This appears to be a documented issue. MSDN acknowledges "timing issues" but doesn't include a way to completely get around them, although there does appear to be a "newer" method that you need to tell your program to use by default. Here's a portion of the documentation:
The SendKeys class has been updated for the .NET Framework 3.0. The SendKeys class is susceptible to timing issues, which some developers have had to work around. The updated implementation is still susceptible to timing issues, but is slightly faster and may require changes to the workarounds. The SendKeys class tries to use the previous implementation first, and if that fails, uses the new implementation. As a result, the SendKeys class may behave differently on different operating systems. Additionally, when the SendKeys class uses the new implementation, the SendWait method will not wait for messages to be processed when they are sent to another process.
If your application relies on consistent behavior regardless of the operating system, you can force the SendKeys class to use the new implementation by adding the following application setting to your app.config file.
<appSettings>
<add key="SendKeys" value="SendInput"/>
</appSettings>
I found a similar (old) issue on another bulletin board, but unfortunately their fix was the same as yours - to delay for a fraction of a second before accessing the clipboard. I couldn't find any other workarounds for the issue. Considering there's a Send and a SendWait, it doesn't seem too much to expect the latter to actually wait after the send! :)

You definitely can NOT update the clipboard in a loop and expect the data to be available (and accessible to your app) immediately. The application that you're sending the keystroke to is running in its own process, and windows is multi-processing, multi-threading, etc.. So you're looking for the clipboard to be updated, before the other app has gotten a chance to copy it.
Furthermore, since there can be other programs running on the system, monitoring the clipboard for updates (clipboard viewers), you are going to be colliding with those programs when you attempt to get the data from the clipboard.
I don't know why you're trying to do what you're doing, but you should be aware that it's not going to work all the time. You may be able to get it to work in some cases, but not all cases. Unless this is an educational exercise for your own use, you should abandon this approach.
And please read this quote on the subject:
"Programs should not transfer data into our out of the clipboard without an explicit instruction from the user.”
— Charles Petzold, Programming Windows 3.1, Microsoft Press, 1992

C# How to dump all variables & current values during runtime

Are there any in-built or 3rd party libraries that allow you to simply dump all variables in memory during run time? What I would like is to be able to view variables & current values similarly to viewing them by hitting a break point and hovering over variables, but without actually having to halt the program execution (i.e. just get a snapshot). Would be good if it could dump them to a file which can then be opened later in a program to get a nice GUI interface to view them, but simple text file dump would be good enough.

I can't think of an easy way to do this in a generic fashion. What could work is programmatically creating a dump file of your running process. You could either do this with P/Invoke to the dbghelp.dll routines or spawn a cdb.exe process to create the dump file. Once you have the file, you could open it up in a debugger for later analysis using SOS.dll with cdb.exe/windbg.exe, or even write a debugger script to dump the data you want (mostly) automatically.

I believe some sort of logging framework would help you to do that...
Check out:
http://www.dotnetlogging.com/
At my workplace we use log4net which works pretty well for us.
So how come you're wanting to dump out all the variables for later analysis? Have you considered writing your code test first so that you can reduce your reliance on the debugger and have a suite of automated test checking the values for you?

In the past I've used the YourKit .Net profiler in order to profile .Net applications.
While I've only ever used it to connect to running applications personally the Snapshot documentation does state that they have a Profiler API that can be used to programmatically dump snapshots for later review.
Code wise this looks to be as simple as the following:
Controller c = new Controller();
String snapshotPath = c.CaptureSnapshot();
I believe you can then load the snapshot files into the YourKit GUI at a later date to review them.
I would not be surprised if some of the other popular profilers like JetBrains dotTrace Performance and RedGates ANTS Performance Profiler have similar programmatic APIs but I couldn't quickly find obvious documentation on their websites (and I didn't want to watch their webinars to find out if this feature existed!)

For this you can use WMemoryProfiler to
Get all objects in all appdomains as an object array
Create a memory dump of your own process
Serialize specific objects to disc
To make this happen you need Windbg of course but the Api of WMemoryProfiler is fully managed and you can basically self debug your process. The library takes care of the usual debugger oddities since it does wrap Windbg in a nice accessible library.
The code below does get all instances of System.Threading.Thread objects into an object array. This way you can write a visualizer for your own application objects at runtime. The other overload does simply give you all objects in all AppDomains.
using (var debugger = new MdbEng())
{
var dummy = new Thread(() => {});
dummy.Name = "Dummy Thread";
// Get all thread objects in all AppDomains
var threads = debugger.GetObjects("System.Threading.Thread", true);
foreach (Thread t in threads)
{
Console.WriteLine("Managed thread {0} has Name {1}", t.ManagedThreadId, t.Name);
}
GC.KeepAlive(dummy);
}
Since it is a wrapper around Windbg you can also create a memory dump on the fly and later load a memory dump from your process to extract object data for visualization from the dump. Commerical Memory Profilers (e.g. MemoryProfiler from Scitech) employ this technique since years but it is quite slow when you have a huge memory dump since they are using also Windbg as dump analyzer.

You can try Intellitrace tool provided with ultimate version of visual studio. It is what you describe - it records what is happening in your app and allows you to debug it without executing your program with hovering over variables and all other debug windows to help you.

You can use PostSharp . I found it very useful to record debug times because of the environment application was deployed. And instrumented/recorded many things.
But obviously you'll need to specify all variables you need to record.
Check more details here.