Develop Reference

How to read and write more then 25000 records/lines into text file at a time?

I am connecting my application with stock market live data provider using web socket. So when market is live and socket is open then it's giving me nearly 45000 lines in a minute. at a time I am deserializing it line by line
and then write that line into text file and also reading text file and removing first line of text file. So handling another process with socket becomes slow. So please can you help me that how should I perform that process very fast like nearly 25000 lines in a minute.
string filePath = #"D:\Aggregate_Minute_AAPL.txt";
var records = (from line in File.ReadLines(filePath).AsParallel()
select line);
List<string> str = records.ToList();
str.ForEach(x =>
{
string result = x;
result = result.TrimStart('[').TrimEnd(']');
var jsonString = Newtonsoft.Json.JsonConvert.DeserializeObject<List<LiveAMData>>(x);
foreach (var item in jsonString)
{
string value = "";
string dirPath = #"D:\COMB1\MinuteAggregates";
string[] fileNames = null;
fileNames = System.IO.Directory.GetFiles(dirPath, item.sym+"_*.txt", System.IO.SearchOption.AllDirectories);
if(fileNames.Length > 0)
{
string _fileName = fileNames[0];
var lineList = System.IO.File.ReadAllLines(_fileName).ToList();
lineList.RemoveAt(0);
var _item = lineList[lineList.Count - 1];
if (!_item.Contains(item.sym))
{
lineList.RemoveAt(lineList.Count - 1);
}
System.IO.File.WriteAllLines((_fileName), lineList.ToArray());
value = $"{item.sym},{item.s},{item.o},{item.h},{item.c},{item.l},{item.v}{Environment.NewLine}";
using (System.IO.StreamWriter sw = System.IO.File.AppendText(_fileName))
{
sw.Write(value);
}
}
}
});
How to make process fast, if application perform this then it takes nearly 3000 to 4000 symbols. and if there is no any process then it executes 25000 lines per minute. So how to increase line execution time/process with all this code ?

First you need to cleanup you code to gain more visibility, i did a quick refactor and this is what i got
const string FilePath = #"D:\Aggregate_Minute_AAPL.txt";
class SomeClass
{
public string Sym { get; set; }
public string Other { get; set; }
}
private void Something() {
File
.ReadLines(FilePath)
.AsParallel()
.Select(x => x.TrimStart('[').TrimEnd(']'))
.Select(JsonConvert.DeserializeObject<List<SomeClass>>)
.ForAll(WriteRecord);
}
private const string DirPath = #"D:\COMB1\MinuteAggregates";
private const string Separator = #",";
private void WriteRecord(List<SomeClass> data)
{
foreach (var item in data)
{
var fileNames = Directory
.GetFiles(DirPath, item.Sym+"_*.txt", SearchOption.AllDirectories);
foreach (var fileName in fileNames)
{
var fileLines = File.ReadAllLines(fileName)
.Skip(1).ToList();
var lastLine = fileLines.Last();
if (!lastLine.Contains(item.Sym))
{
fileLines.RemoveAt(fileLines.Count - 1);
}
fileLines.Add(
new StringBuilder()
.Append(item.Sym)
.Append(Separator)
.Append(item.Other)
.Append(Environment.NewLine)
.ToString()
);
File.WriteAllLines(fileName, fileLines);
}
}
}
From here should be more easy to play with List.AsParallel to check how and with what parameters the code is faster.
Also:
You are opening the write file twice
The removes are also somewhat expensive, in the index 0 is more (however, if there are few elements this could not make much difference
if(fileNames.Length > 0) is useless, use a for, if the list is empty, then he for will simply skip
You can try StringBuilder instead string interpolation
I hope this hints can help you to improve your time! and that i have not forgetting something.
Edit
We have nearly 10,000 files in our directory. So when process is
running then it's passing an error that The Process can not access the
file because it is being used by another process
Well, is there a possibility that in your process lines there is duplicated file names?
If that is the case, you could try a simple approach, a retry after some milliseconds, something like
private const int SleepMillis = 5;
private const int MaxRetries = 3;
public void WriteFile(string fileName, string[] fileLines, int retries = 0)
{
try
{
File.WriteAllLines(fileName, fileLines);
}
catch(Exception e) //Catch the special type if you can
{
if (retries >= MaxRetries)
{
Console.WriteLine("Too many tries with no success");
throw; // rethrow exception
}
Thread.Sleep(SleepMillis);
WriteFile(fileName, fileLines, ++retries); // try again
}
}
I tried to keep it simple, but there are some annotations:
- If you can make your methods async, it could be an improvement by changing the sleep for a Task.Delay, but you need to know and understand well how async works
- If the collision happens a lot, then you should try another approach, something like a concurrent map with semaphores
Second edit
In real scenario I am connecting to websocket and receiving 70,000 to
1 lac records on every minute and after that I am bifurcating those
records with live streaming data and storing in it's own file. And
that becomes slower when I am applying our concept with 11,000 files
It is a hard problem, from what i understand, you're talking about 1166 records per second, at this size the little details can become big bottlenecks.
At that phase i think it is better to think about other solutions, it could be so much I/O for the disk, could be many threads, or too few, network...
You should start by profiling the app to check where the app is spending more time to focus in that area, how much resources is using? how much resources do you have? how is the memory, processor, garbage collector, network? do you have an SSD?
You need a clear view of what is slowing you down so you can attack that directly, it will depend on a lot of things, it will be hard to help with that part :(.
There are tons of tools for profile c# apps, and many ways to attack this problem (spread the charge in several servers, use something like redis to save data really quick, some event store so you can use events....

Convert from console to string for return [duplicate]

What I really want to do is this
static string Main(string[] args)
but that doesn't work, your only options are void and int. So, What are some different ways to return the string that I need to return to the calling application?
Background
I need to write a console app that is specifically designed to be called from another application
Process.Start("MyCode.exe -Option 12aaa1234");
How can this calling program receive a string returned from that executable?
Research
From what I can tell, at this point in time my only option is to have the calling application attach a listening stream to the Standard Output stream of the process before starting it, and send the "return" using Console.Out.Write from inside my executable. Is this in fact the ONLY way to do this, or is there something different/better I can use?

Is this in fact the ONLY way to do this, or is there something different/better I can use?
This isn't the only way to do this, but it is the most common.
The other options would involve some form of interprocess communication, which is likely going to be significantly more development effort for a single string.
Note that, if the calling application is a .NET application, and you have control over both applications, it might make more sense to just write a class library instead of a console application. This would allow you to keep the code completely separate, but have the executable "call into" your library to get the string data.

Idea 1:
Using MyCode.exe, create an encrypted text file, which is saved in a specified path, which can then be decrypted in the current app and read.
In the app: "MyCode.exe", add this code:
public void ReturnToOther()
{
string ToReturn = "MyString";
System.IO.File.WriteAllText("Path", Encrypt(ToReturn));
}
public String Encrypt(string ToEncrypt)
{
string Encrypted = null
char[] Array = ToEncrypt.ToCharArray();
for (int i = 0; i < Array.Length; i++)
{
Encrypted += Convert.ToString(Convert.ToChar(Convert.ToInt32(Array[i]) + 15));
}
return Encrypted;
}
In the app you are making now:
public void GetString()
{
string STR = Decrypt(System.IO.File.ReadAllText("Path"));
Console.WriteLine("The string is: {0}", STR);
}
// If you want to keep this running before the file exists, use this:
/*
public void GetString()
{
for(int i = 0; i > -1; ++i)
{
if(System.IO.File.Exists("Path"))
{
string STR = Decrypt(System.IO.File.ReadAllText("Path"));
Console.WriteLine("The string is: {0}", STR);
break;
}
else
{
//Do something if you want
}
}
} */
public String Decrypt(string ToDecrypt)
{
string Decrypted = null
char[] Array = ToDecrypt.ToCharArray();
for (int i = 0; i < Array.Length; i++)
{
Decrypted += Convert.ToString(Convert.ToChar(Convert.ToInt32(Array[i]) - 15));
}
return Decrypted;
}
Idea 2:
Use TCP to upload the string to a port, e.g. LocalHost (127.0.0.1), and then receive the string on the app you are developing, using a TCP Listener
An article on TCP - http://msdn.microsoft.com/en-us/library/system.net.sockets.tcpclient.aspx
Hope this helps :)
EDIT:
Have a look at Sockets too: http://msdn.microsoft.com/en-us/library/system.net.sockets.socket.aspx

Store a short string

I have many prompts to be used in the code, each prompt is a string such as "welcome to our company" or "goodby" etc.
Now I want to manage these prompts. There are two ways, one is to store each string in a file then load it in the code.
private string LoadPrompt(string inid, string PromptsPath,string promptFile)
{
StringBuilder filetopen = new StringBuilder();
StringBuilder content = new StringBuilder();
filetopen.Append(PromptsPath + inid + "_" + promptFile);
try
{
if (File.Exists(filetopen.ToString()))
{
using (StreamReader reader = File.OpenText(filetopen.ToString()))
{
string line;
while ((line = reader.ReadLine()) != null)
{
content.Append(line);
}
}
}
}
catch (Exception ex)
{
AppLogEx(ex, "Utilities:LoadPrompt");
content.Clear();
content.Append("ERROR");
}
return content.ToString();
}
The other one is put prompts in app.config. Which way is better and fast to load them?

Set up the strings as Resources.
The advantage to using resources over a flat file or app.config is that you can then localize your strings into various languages, and there is support and tooling available to make this process much easier. If you provide multiple languages, your program can automatically select the appropriate language/culture based on the local system without you needing to do any extra work.

About the code, you can do this simpler
instead
using (StreamReader reader = File.OpenText(filetopen.ToString()))
{
string line;
while ((line = reader.ReadLine()) != null)
{
content.Append(line);
}
}
you can write
File.ReadAllLines(filetopen.ToString())
.ToList() - added only for Foreach, I think you've your own Foreach extension for IEnumerable
.ForEach(x => content.AppendLine(x));
about prompts, config or resources

Well, if you do it through app.config, it's essentially an XML file. So you'll be able to find the right prompt fairly easily. Since it's a setting file, it would make sense to store in it any settings that you might change down the road. You can do the same thing with a regular text file, but then you'll need to somehow mark different prompts... in which case, might as well use XML.

Search String takes a long time the first time only?

No shortage of search for string performance questions out there yet I still can not make heads or tails out of what the best approach is.
Long story short, I have committed to moving from 4NT to PowerShell. In leaving the 4NT I am going to miss the console super quick string searching utility that came with it called FFIND. I have decided to use my rudimentary C# programming skills to try an create my own utility to use in PowerShell that is just as quick.
So far search results on a string search in 100's of directories across a few 1000 files, some of which are quite large, are FFIND 2.4 seconds and my utility 4.4 seconds..... after I have ran mine at least once????
The first time I run them FFIND does it near the same time but mine takes over a minute? What is this? Loading of libraries? File indexing? Am I doing something wrong in my code? I do not mind waiting a little longer but the difference is extreme enough that if there is a better language or approach I would rather start down that path now before I get too invested.
Do I need to pick another language to write a string search that will be lighting fast
I have the need to use this utility to search through 1000 of files for strings in web code, C# code, and another propitiatory language that uses text files. I also need to be able to use this utility to find strings in very large log files, MB size.
class Program
{
public static int linecounter;
public static int filecounter;
static void Main(string[] args)
{
//
//INIT
//
filecounter = 0;
linecounter = 0;
string word;
// Read properties from application settings.
string filelocation = Properties.Settings.Default.FavOne;
// Set Args from console.
word = args[0];
//
//Recursive search for sub folders and files
//
string startDIR;
string filename;
startDIR = Environment.CurrentDirectory;
//startDIR = "c:\\SearchStringTestDIR\\";
filename = args[1];
DirSearch(startDIR, word, filename);
Console.WriteLine(filecounter + " " + "Files found");
Console.WriteLine(linecounter + " " + "Lines found");
Console.ReadKey();
}
static void DirSearch(string dir, string word, string filename)
{
string fileline;
string ColorOne = Properties.Settings.Default.ColorOne;
string ColorTwo = Properties.Settings.Default.ColorTwo;
ConsoleColor valuecolorone = (ConsoleColor)Enum.Parse(typeof(ConsoleColor), ColorOne);
ConsoleColor valuecolortwo = (ConsoleColor)Enum.Parse(typeof(ConsoleColor), ColorTwo);
try
{
foreach (string f in Directory.GetFiles(dir, filename))
{
StreamReader file = new StreamReader(f);
bool t = true;
int counter = 1;
while ((fileline = file.ReadLine()) != null)
{
if (fileline.Contains(word))
{
if (t)
{
t = false;
filecounter++;
Console.ForegroundColor = valuecolorone;
Console.WriteLine(" ");
Console.WriteLine(f);
Console.ForegroundColor = valuecolortwo;
}
linecounter++;
Console.WriteLine(counter.ToString() + ". " + fileline);
}
counter++;
}
file.Close();
file = null;
}
foreach (string d in Directory.GetDirectories(dir))
{
//Console.WriteLine(d);
DirSearch(d,word,filename);
}
}
catch (System.Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
}

If you want to speed up your code run a performance analysis and see what is taking the most time. I can almost guaruntee the longest step here will be
fileline.Contains(word)
This function is called on every line of the file, on every file. Naively searching for a word in a string can taken len(string) * len(word) comparisons.
You could code your own Contains method, that uses a faster string comparison algorithm. Google for "fast string exact matching". You could try using a regex and seeing if that gives you a performance enhancement. But I think the simplest optimization you can try is :
Don't read every line. Make a large string of all the content of the file.
StreamReader streamReader = new StreamReader(filePath, Encoding.UTF8);
string text = streamReader.ReadToEnd();
Run contains on this.
If you need all the matches in a file, then you need to use something like Regex.Matches(string,string).
After you have used regex to get all the matches for a single file, you can iterate over this match collection (if there are any matches). For each match, you can recover the line of the original file by writing a function that reads forward and backward from the match object index attribute, to where you find the '\n' character. Then output that string between those two newlines, to get your line.
This will be much faster, I guarantee it.
If you want to go even further, some things I've noticed are :
Remove the try catch statement from outside the loop. Only use it exactly where you need it. I would not use it at all.
Also make sure your system is running, ngen. Most setups usually have this, but sometimes ngen is not running. You can see the process in process explorer. Ngen generates a native image of the C# managed bytecode so the code does not have to be interpreted each time, but can be run natively. This speeds up C# a lot.
EDIT
Other points:
Why is there a difference between first and subsequent run times? Seems like caching. The OS could have cached the requests for the directories, for the files, for running and loading programs. Usually one sees speedups after a first run. Ngen could also be playing a part here, too, in generating the native image after compilation on the first run, then storing that in the native image cache.
In general, I find C# performance too variable for my liking. If the optimizations suggested are not satisfactory and you want more consistent performance results, try another language -- one that is not 'managed'. C is probably the best for your needs.

Is string builder the right choice?

So from what I understand of a string vs StringBuilder is that string builder will actually modify the instance of itself while string will just make a new one. So if I understand this correctly then by using the string method for a constantly changing variable I could basically be eventually using all the memory until the computer needs to dump it to make room.
What I am doing is using an event handler to monitor serial communication. I will take in the data and parse it out plus display it in a text box. The event handler uses string to accomplish this currently. In concern for better programming and not using up all the memory when I don't need to I am trying to clean up my code.
I started to code with string builder and begun to get the build error that StringBuilder does not contain a .contains method.
Basically I am curious if i should leave it alone? Should I approach this differently? and do I have the right understanding in that string will inevitably run me out of memory?
private void DataReceivedHandler(object sender, SerialDataReceivedEventArgs e)
{
SerialPort sp = (SerialPort)sender;
string indata = sp.ReadExisting();
rx.AppendText(Environment.NewLine + indata);
string dataCheck = indata.ToUpper();
if (dataCheck.Contains("CONNECT") || indata.Contains("CONNECTED"))
{
cState.Text = "Connected";
connectLink();
}
if (dataCheck.Contains("NO CARRIER"))
{
cState.Text = "Disconnected";
disconnect();
}
dataCheck = null;
}

You are incorrect; using String will not (in general) cause you to run out of memory.
If you're doing lots of concatenation, using string is less efficient, since it needs to build a new string and throw away the old string every time you concatenate.
In such scenarios, you should use a StringBuilder to build the string, then call ToString() whenever you want to display it.
Your code does not contain any concatenation, so using StringBuilder wouldn't do any good.