Has anyone tried creating their own logger in c# as per:
https://github.com/connamara/quickfixn/blob/master/QuickFIXn/FileLog.cs
I am testing this and I don't seem to be able to increase logging speed - there appears to be a finite amount of time taken in getting to:
public void OnIncoming(string msg){}
Or I am doing something wrong.
In my version, I purely timestamp the message and add to list in memory and write to file later on a background thread. However, the time between log entries appears same as if I wrote straight to disk. So, I am wondering if there is much hidden processing prior to this call.
I have disabled "Crack" - so it is not doing that.
I seems your processing time is not taken by logging, but in your program. Doing an add to a list should be faster than writing it to disk. However if your program does much processing you will only notice a small difference in the overall performance. You can add a profiler to find where the time is taken.
Related
A little information about what I'm trying to do:
I'm trying to make a simple program in c# that analyses the usage of the computer, how many hours a day does a user use his computer, within those hours how much time does he spend using any application and stuff like that.
My question is:
I am trying to do a quick log of the time of shutdown and i need a way for it to be a long term solution without the possibility of corruption.
What is a safe and reliable method of logging that kind of information?
I thought of logging it in an interval of lets say 5 minutes, but what if i am going to make a log at the exact same time the power goes out. There's a chance of corruption and i need a very reliable method to keep this information intact
Have a look at SystemEvents.SessionEnding Event :https://learn.microsoft.com/en-us/dotnet/api/microsoft.win32.systemevents.sessionending?view=netframework-4.7.2
When the event is raised you can capture the shutdown/logoff time in a log entry to a text file to keep things simple and quick.
I have not tested it but I believe that there is a window of a few seconds, In case you want to test out an API call to capture the shutdown timings.
I'm new to profiling. I'm trying to profile a C# application which connects to an SQLite database and retrieve data. The database contains 146856400 rows and the select query retrieves 428800 rows after execution.
On the first execution the main thread takes 246686 ms
On second execution of the same code the main thread takes only 4296 ms
After restarting the system
On the first execution the main thread takes 244533 ms
On the second execution of the same code the main thread takes only 4053 ms
Questions:
1) Why is there a big difference between the first execution timing and the second execution timing
2) After restarting the system why I'm not getting the same results.
Pls help
You experience the difference between cold and warm execution of your query. Cold means the first time and warm all subsequent invocations of your db query.
The first time everything is "cold"
OS file system cache is empty.
SQLLite cache is empty.
ORM dynamic query compilation is not done and cached yet.
ORM Mapper cache is empty.
Garbage Collector needs to tune your working set
....
When you execute your query a second time all these first time initializations (caching) are done and you are measuring the effects of different cache levels as long as there is enough memory available to cache a substantial amount of your requested data.
A performance difference between 4 minutes and 4s is impressive. Both numbers are valid. Measuring something is easy. Telling someone else what exactly you have measured and how the performance can be improved by changing this or that is much harder.
The performance game goes often like this:
Customer: It is slow
Dev: I cannot repro your issue.
Customer: Here is my scenario ....
Dev: I still cannot repro it. Can you give me data set you use and the exact steps you did perform?
Customer: Sure. Here is the data and the test steps.
Dev: Ahh I see. I can make it 10 times faster.
Customer: That is great. Can I have the fix?
Dev: Sure here it is.
Customer: **Very Angry** It has become faster yes. But I cannot read my old data!
Dev: Ups. We need to migrate all your old data to the new much more efficient format.
We need to develop a a conversion tool which will take 3 weeks and your site will
have 3 days downtime while the conversion tool is running.
Or
We keep the old inefficient data format. But then we can make it only 9 times faster.
Customer: I want to access my data faster without data conversion!
Dev: Here is the fix which is 10% slower with no schema changes.
Customer: Finally. The fix does not break anything but it has not become faster?
Dev: I have measured your use case. It is only slow for the first time.
All later data retrievals are 9 times faster than before.
Customer: Did I mention that in my use case I read always different data?
Dev: No you did not.
Customer: Fix it!
Dev: That is not really possible without a major rewrite of large portions of our software.
Customer: The data I want to access is stored in a list. I want to process it sequentially.
Dev: In that case we can preload the data in the background while you are working the current data set. You will only experience a delay for the first data set on each working day.
Customer: Can I have the fix?
Dev: Sure here it is.
Customer: Perfect. It works!
Performance is hard to grasp since most of the time you deal with perceived performance which is subjective. Bringing it down to quantitative measurements is a good start but you need to tune your metrics to reflect actual customer use cases or you will likely optimize at the wrong places like above. A complete understanding of customer requirements and use cases is a must. On the other hand you need to understand your complete system (profile it as hell) to be able to tell the difference between cold and warm query execution and where you can tune the whole thing. These caches become useless if you query for different data all of the time (not likely). Perhaps you need a different index to speed up queries or you buy a SSD or you keep all of the data in memory and do all subsequent queries in memory....
I have a c# application that generates data every 1 second (stock tick data) which can be discarded after each itteration.
I would like to pass this data to a Coldfusion (10) application and I have considered having the c# application writing the data to a file every second and then having the Coldfusion application reading that data, but this is most likely going to cause issues with the potential for both applications trying to read or write to the file at the same time ?
I was wondering if using Memory Mapped Files would be a better approach ? If so, how could I access the memory mapped file from Coldfusion ?
Any advice would be greatly appreciated. Thanks.
We have produced a number of stock applications that include tick by tick tracking of watchlists, charting etc. I think the idea of a file is probably not a great idea unless you are talking about a single stock with regular intervals. In my experience a change every "second" is probably way understating the case. Some stokes (AAPL or GOOG are good examples) have hundreds of "ticks" per second during peak times.
So if you are NOT taking every tick but really are "updating the file" every 1 second then your idea has some merit in that you could use a file watching gateway to fire events for you and "see" that the file is updated.
But keep in mind that you are in effect introducing something "in the middle". A file now stands between your Java or CF applications and the quote engine. That's going to introduce latency no matter what you choose to do (file handles getting and releasing etc). And the locks of one process may interfere with the other.
When you are dealing with facebook updates miliseconds don't really matter much - in spite of all the teenage girls who probably disagree with me :) With stock quotes however, half of the task is shaving off miliseconds to get your processes as close to real time as possible.
Our choice is usually to choose sockets instead of something in the middle bridging the data. The quote engine then keeps it's watchlist and updates it's arrays like normal but also sends any updates down stream to the socket engine which pushes it to something taht can handle it (a chart application, watchlist, socketgateway for webpage etc).
Hope this helps - it's not a clear answer but more of a clarification to the hurdles you face.
I really don't want to introduce any delays in my high frequency trading software and at the same time I need to store thousands of lines of logs every second. 1 ms delay would be huge, I only agree to have 0.01-0.05 ms delay.
*Now*I just allocate 500 Mb in memory at start-up, store logs there and when application finish I put this log on disk.
However now I realized that I want more logs and I want them during application execution. So I now want to store logs during application execution (probably once per minute or once per 10 minute). How slow StreamWriter.WriteLine is? Would it be slower than just "adding to preallocated collection"?
Should I use StreamWriter.WriteLine directly (is it syncrhonous or asynchronous, is AutoFlush option affects perfomance?). I also can use BlockingCollection to add items to log and then use dedicated thread to process this blocking collection and to store logs on disk in another thread.
Don't
Reinvent a wheel
Do
Use a logging framework
Properly configure loggers and levels for each logger
Use sync logging for memory (it's simple and fast, but has problems with event persistence onto drive) and async for IO (it is difficult to get right, slow, harder to test) loggers
If you hadn't done so, check out log4net and NLog, this will be a good place to start.
Probably you could store your logs in circular buffer and spawn a new thread of execution which will just send data from that buffer in shared memory to disk.
Use log4net as Andre Calil suggests. It logs to SQL, disks and whatnot and is extremely customizable. It can seem a bit complicated at first, but it is worth the effort.
What you need is probably the RollingFileAppender. log4net is in nuget, but you should read the documentation at the log4net site. Start by looking at the appender config.
I am working on a program that reads in from a serial port, then parses, formats, and displays the information appropriately. This program, however, needs to be run for upwards of 12+ hours - constantly handling a stream of incoming data. I am finding that as I let my app run for a while, the memory usage increases at a linear rate - not good for a 12 hour usage.
I have implemented a logger that writes the raw incoming binary data to a file - is there a way I can utilize this idea to clear my memory cache at regular intervals? I.e. how can I, every so often, write to the log file in such a way that the data doesn't need to be stored in memory?
Also - are there other aspects of a Windows Form Application that would contribute to this? E.g. I print the formatted strings to a textbox, which ends up displaying the entire string. Because this is running for so long, it easily displays hundreds of thousands of lines of text. Should I be writing this to a file and clearing the text? Or something else?
Obviously, if the string grows over time, your app's memory usage will also grow over time. Also, WinForms textboxes can have trouble dealing with very large strings. How large does the string get?
Unless you really want to display the entire string onscreen, you should definitely clear it periodically (depending on your users' expectations); this will save memory and probably improve performance.
Normally, memory management in .NET is completely automatic. You should be careful about extrapolating a short observations (minutes) to a 12 hour period. And please note that TaskManager is not a very good tool to measure memory usage.
Writing the incoming data should not increase memory usage significantly. But there are a few thing you should avoid doing, and concatenating to a string over and over is one of them. Your TextBox is probably costing a lot more than you seem to think. Using a ListBox would be more efficient. And easier.
I have several serial applications which run either as an application or as a windows service. These are required to be up 24/7-365. The best mechanism I have found to avoid this same problem is two-fold.
1) Write the information out to a log file. For a service, this is the only way of getting the info out. The log file does not increase your memory usage.
2) For the application, write the information out to a log file as well as put it into a listbox. I generally limit the listbox to the last 500 or 1000 entries. With the newer .net controls, the listboxes are virtualized which helps but you also don't run into other memory issues such as the textbox concatenation.
You can take a system down with a textbox by constantly appending the string over a number of hours as it is not intended for that kind of abuse out of the box.