Develop Reference

How can I read Persian line in csv file c#

I want to read a simple CSV file with comma separated with this code:
var reader = new StreamReader(File.OpenRead(#"d:\34.csv"));
List<string> listA = new List<string>();
List<string> listB = new List<string>();
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var values = line.Split(',');
listA.Add(values[0]);
listB.Add(values[1]);
}
MessageBox.Show("READ IT!!!");
But when I read the file an debug that code,attention can not read Persian or Arabic character! How can I solve that? I think my file is not valid encoding?

if your CSV file contains just one line the ReadToEnd could be acceptable, but if you have a log file composed by more than one line then it is better to read line by line using ReadLine of the StreamReader object
link for true answer and more information
using (StreamReader sr = new StreamReader("c:/temp/34.csv"))
{
string currentLine;
// currentLine will be null when the StreamReader reaches the end of file
while((currentLine = sr.ReadLine()) != null)
{
// Search, case insensitive, if the currentLine contains the searched keyword
if(currentLine.IndexOf("I/RPTGEN", StringComparison.CurrentCultureIgnoreCase) >= 0)
{
Console.WriteLine(currentLine);
}
}
}
More information

You can create a class composed of get and set for each line of the CSV . You can then instantiate an object list to retrieve the CSV lines.
Try this way :
class Program
{
static void Main(string[] args)
{
var reader = new StreamReader(File.OpenRead(#"YourCSV"),Encoding.Unicode);
List<Customer> customer = new List<Customer>();
while (!reader.EndOfStream)
{
Customer c = new Customer
{
m_line1 = null,
m_line2 = null,
};
var line = reader.ReadLine();
var tokens = line.Split(',');
c.m_line1 = tokens[0];
c.m_line2 = tokens[1];
customer.Add(c);
}
foreach(var s in customer)
{
Console.Writline(s);
Console.Readline();
}
}
}
class Customer
{
private string line1;
public string m_line1
{
get
{
return line1;
}
set
{
line1= value;
}
}
private string line2;
public string m_line2
{
get
{
return line2;
}
set
{
line2= value;
}
}

You will have to pass the character encoding to the StreamReader constructor. There is no such thing as plain text. Reading text requires knowing its encoding.
The line
using (StreamReader sr = new StreamReader("c:/temp/34.csv"))
should be
using (StreamReader sr = new StreamReader("c:/temp/34.csv"), myencoding)
what myencoding is is something only you can know. With what encoding was the file saved? That's the encoding you need there. If the file was generated on Windows, and educated guess of the most likely encoding would be it is UTF-16LE. That encoding is available as Encoding.Unicode - which is a bad name, it should have been Encoding.UTF16LE, but that's the name the .NET framework uses.
Other possible encodings that are supported by StreamReader are listed on https://msdn.microsoft.com/en-us/library/System.Text.Encoding_properties(v=vs.110).aspx
If you don't know with what encoding the file was saved, some encodings leave hints in the form of a Byte order mark sometimes abbreviated to BOM. A byte order mark are the first few bytes of a text document that tell you its encoding. You can find more information on the byte order mark, and some of its values on http://en.wikipedia.org/wiki/Byte_order_mark
Relying on the BOM is generally a bad idea, because
it's not a full-proof solution: some encodings don't use a BOM, or make the BOM optional
Even if you successfully determine the encoding, that doesn't mean that StreamReader knows how to handle that encoding (though this is unlikely, but possible)
the BOM might not be a BOM at all, but be part of the actual text (also unlikely but possible)
In some cases it is impossible to know the encoding of a file, notably if the file comes from a file upload on the web, or if someone just mailed you the file, and they don't know how they encoded it. This can be a good reason not to allow "plain text" uploads (which is reasonable because, it can do with a little repetition, there is no such thing as plain text).
tl;dr: The most likely thing to work is one of
using (StreamReader sr = new StreamReader(File.OpenRead(#"c:/temp/34.csv"),Encoding.Unicode) {
...
}
or
using (StreamReader sr = new StreamReader(File.OpenRead(#"c:/temp/34.csv"),Encoding.UTF8)
or
using (StreamReader sr = new StreamReader(File.OpenRead(#"c:/temp/34.csv"),Encoding.UTF32)

Removing formatting (\t\r) from a Stream Reader

I'm importing an .sql file that I created with SQLExplorer into my .NET program using StreamReader (which will eventually be passed through OdbcConnection, fyi.) The problem is that when I use the ReadToEnd() method it not only imports the SQL itself, but it imports all of the formatting. So the string is littered with \r and \t and the like.
I've been looking at both using split or possibly regex to break the string down and remove the unwanted bits and pieces. But before throwing a bunch of effort into that I wondered if there was perhaps something I'm missing in the StreamReader class? Is there a way to tell it to just ignore the formatting characters?
Here's the code I have right now:
public static Object SQLQueryFileCall(String SQLQueryFileName){
string SQLQuery = "";
string directory = System.IO.Path.GetDirectoryName( System.Reflection.Assembly.GetExecutingAssembly().Location);
SQLQueryFileName = directory + "\\" + SQLQueryFileName;
//read in the file and pass to ODBC, return a Object[] of whatever comes back...
try{
using (StreamReader myStreamReader = new StreamReader(SQLQueryFileName)) {
SQLQuery = myStreamReader.ReadToEnd();
Console.WriteLine(SQLQuery);
}
}
catch (Exception e){
//Find the error
Console.WriteLine("File could not be read:");
string error = e.Message;
MessageBox.Show(error);
return null;
}
}
Feel free to offer any advice you might have on the code, seeing as I'm pretty new.
But yeah, mostly I'm just hoping there's a method in the StreamReader class that I'm just not understanding. I've gone to Microsoft's online documentation, and I feel I've given it a good look, but then again, I'm new and perhaps the concept skipped over my head?
Any help?
NOTE: There are multiple \t that are in the middle of some of the lines, and they do need to be removed. Hence using trim would...be tricky at least.

Well, myStreamReader.ReadToEnd(); will get you everything. The easiest way to get rid of most unneeded whitespace is to read it line-by-line and then simply .Trim() every line.
using (StreamReader myStreamReader = new StreamReader(SQLQueryFileName))
{
string line;
List<string> lines = new List<string();
while ((line = myStreamReader.ReadLine()) != null)
lines.Add(line.Trim());
SQLQuery = string.Join("\n", lines);
Console.WriteLine(SQLQuery);
}

SQL, by definition, shouldn't have a problem with whitespaces like tabs and newlines throughout code. Verify that your actual SQL is correct first.
Also, blindly stripping whitespace could potentially have an impact on textual data contained within your script; what happens if you have a string literal that contains a tab character?

using (StreamReader myStreamReader = new StreamReader(SQLQueryFileName))
{
while ((line = myStreamReader.ReadLine()) != null)
{
Console.WriteLine(line.Trim().Replace(#"\t"," ")
}
}
If it is a display and not content issue I think you could use WPF and a RichTextBox.

SQLQuery = Regex.Replace(SQLQuery, #"\t|\n|\r", "");
Better to use as you could remove content of SQL insertions:
using (StreamReader reader = new StreamReader(fileName))
{
String line;
List<string> lines = new List<string();
while ((line = myStreamReader.ReadLine()) != null)
SQLQuery += line.Trim() + "\n";
}

This is what helped me in 2020, incase if someone is looking for this
using (StreamReader sr = new StreamReader(stream))
{
return Regex.Replace(sr.ReadToEnd().Trim(), #"\s", "");
}

replacing \n with \r\n in a large text file

I have a large text file with a lot of \n that I need to replace with \r\n. With small text files, I was using the ReadToEnd method to get the file as a string and then use the Replace method and then write the string to a file. With a big file, however, I get an OutOfMemory exception because the string is too big. Any help would be greatly appreciated. Thanks.

private void foo() {
StreamReader reader = new StreamReader(#"D:\InputFile.txt");
StreamWriter writer = new StreamWriter(#"D:\OutputFile.txt");
string currentLine;
while (!reader.EndOfStream) {
currentLine = reader.ReadLine();
writer.Write(currentLine + "\r\n");
}
reader.Close();
writer.Close();
}
This should resolve your problem. Please note, that reader.ReadLine() cuts of the trailing "\n".

DiableNoir's solution is the right idea, but the implementation is buggy and it needs some explanation. Here's an improved one:
using (var reader = new StreamReader(#"D:\InputFile.txt"))
using (var writer = new StreamWriter(#"D:\OutputFile.txt")) // or any other TextWriter
{
while (!reader.EndOfStream) {
var currentLine = reader.ReadLine();
writer.Write(currentLine + "\r\n");
}
}
You use a TextReader for input and a TextWriter for output (this one might direct to a file or to an in-memory string). reader.ReadLine will not return the line ending as part of the line, so you need to write it explicitly (instead of using string.Replace, which will not accomplish anything at all).
Also, exactly because you will never see \n or \r as part of currentLine, this program is safe to run again on the output it has produced (in this case its output will be exactly identical to its input). This would not be the case if currentLine included the line ending, because it would change \n to \r\n the first time, and then make it \r\r\n the second time, etc.

You could use Read and specify how many bytes to read each time. Such as read the file in 10 MB chunks.

Or if you need like a larger buffer you can use StreamReader.ReadBlock();

Best way to write huge string into a file

In C#, I'm reading a moderate size of file (100 KB ~ 1 MB), modifying some parts of the content, and finally writing to a different file. All contents are text. Modification is done as string objects and string operations. My current approach is:
Read each line from the original file by using StreamReader.
Open a StringBuilder for the contents of the new file.
Modify the string object and call AppendLine of the StringBuilder (until the end of the file)
Open a new StreamWriter, and write the StringBuilder to the write stream.
However, I've found that StremWriter.Write truncates 32768 bytes (2^16), but the length of StringBuilder is greater than that. I could write a simple loop to guarantee entire string to a file. But, I'm wondering what would be the most efficient way in C# for doing this task?
To summarize, I'd like to modify only some parts of a text file and write to a different file. But, the text file size could be larger than 32768 bytes.
== Answer == I'm sorry to make confusin to you! It was just I didn't call flush. StremWriter.Write does not have a short (e.g., 2^16) limitation.

StreamWriter.Write
does not
truncate the string and has no limitation.
Internally it uses String.CopyTo which on the other hand uses unsafe code (using fixed) to copy chars so it is the most efficient.

The problem is most likely related to not closing the writer. See http://msdn.microsoft.com/en-us/library/system.io.streamwriter.flush.aspx.
But I would suggest not loading the whole file in memory if that can be avoided.

can you try this :
void Test()
{
using (var inputFile = File.OpenText(#"c:\in.txt"))
{
using (var outputFile = File.CreateText(#"c:\out.txt"))
{
string current;
while ((current = inputFile.ReadLine()) != null)
{
outputFile.WriteLine(Process(current));
}
}
}
}
string Process(string current)
{
return current.ToLower();
}
It avoid to have to full file loaded in memory, by processing line by line and writing it directly

Well, that entirely depends on what you want to modify. If your modifications of one part of the text file are dependent on another part of the text file, you obviously need to have both of those parts in memory. If however, you only need to modify the text file on a line-by-line basis then use something like this :
using (StreamReader sr = new StreamReader(#"test.txt"))
{
using (StreamWriter sw = new StreamWriter(#"modifiedtest.txt"))
{
while (!sr.EndOfStream)
{
string line = sr.ReadLine();
//do some modifications
sw.WriteLine(line);
sw.Flush(); //force line to be written to disk
}
}
}

Instead of of running though the hole dokument i would use a regex to find what you are looking for Sample:
public List<string> GetAllProfiles()
{
List<string> profileNames = new List<string>();
using (StreamReader reader = new StreamReader(_folderLocation + "profiles.pg"))
{
string profiles = reader.ReadToEnd();
var regex = new Regex("\nname=([^\r]{0,})", RegexOptions.IgnoreCase);
var regexMatchs = regex.Matches(profiles);
profileNames.AddRange(from Match regexMatch in regexMatchs select regexMatch.Groups[1].Value);
}
return profileNames;
}

Reading from file not fast enough, how would I speed it up?

This is the way I read file:
public static string readFile(string path)
{
StringBuilder stringFromFile = new StringBuilder();
StreamReader SR;
string S;
SR = File.OpenText(path);
S = SR.ReadLine();
while (S != null)
{
stringFromFile.Append(SR.ReadLine());
}
SR.Close();
return stringFromFile.ToString();
}
The problem is it so long (the .txt file is about 2.5 megs). Took over 5 minutes. Is there a better way?
Solution taken
public static string readFile(string path)
{
return File.ReadAllText(path);
}
Took less than 1 second... :)

S = SR.ReadLine();
while (S != null)
{
stringFromFile.Append(SR.ReadLine());
}
Of note here, S is never set after that initial ReadLine(), so the S != null condition never triggers if you enter the while loop. Try:
S = SR.ReadLine();
while (S != null)
{
stringFromFile.Append(S = SR.ReadLine());
}
or use one of the other comments.
If you need to remove newlines, use string.Replace(Environment.NewLine, "")

Leaving aside the horrible variable names and the lack of a using statement (you won't close the file if there are any exceptions) that should be okay, and certainly shouldn't take 5 minutes to read 2.5 megs.
Where does the file live? Is it on a flaky network share?
By the way, the only difference between what you're doing and using File.ReadAllText is that you're losing line breaks. Is this deliberate? How long does ReadAllText take?

return System.IO.File.ReadAllText(path);

Marcus Griep has it right. IT's taking so long because YOU HAVE AN INFINITE LOOP. copied your code and made his changes and it read a 2.4 M text file in less than a second.
but I think you might miss the first line of the file. Try this.
S = SR.ReadLine();
while (S != null){
stringFromFile.Append(S);
S = SR.ReadLine();
}

Do you need the entire 2.5 Mb in memory at once?
If not, I would try to work with what you need.

Use System.IO.File.RealAllLines instead.
http://msdn.microsoft.com/en-us/library/system.io.file.readalllines.aspx
Alternatively, estimating the character count and passing that to StringBuilder's constructor as the capacity should speed it up.

Try this, should be much faster:
var str = System.IO.File.ReadAllText(path);
return str.Replace(Environment.NewLine, "");

By the way: Next time you're in a similar situation, try pre-allocating memory. This improves runtime drastically, regardless of the exact data structures you use. Most containers (StringBuilder as well) have a constructor that allow you to reserve memory. This way, less time-consuming reallocations are necessary during the read process.
For example, you could write the following if you want to read data from a file into a StringBuilder:
var info = new FileInfo(path);
var sb = new StringBuilder((int)info.Length);
(Cast necessary because System.IO.FileInfo.Length is long.)

ReadAllText was a very good solution for me. I used following code for 3.000.000 row text file and it took 4-5 seconds to read all rows.
string fileContent = System.IO.File.ReadAllText(txtFilePath.Text)
string[] arr = fileContent.Split('\n');

The loop and StringBuilder may be redundant; Try using
ReadToEnd.

To read a text file fastest you can use something like this
public static string ReadFileAndFetchStringInSingleLine(string file)
{
StringBuilder sb;
try
{
sb = new StringBuilder();
using (FileStream fs = File.Open(file, FileMode.Open))
{
using (BufferedStream bs = new BufferedStream(fs))
{
using (StreamReader sr = new StreamReader(bs))
{
string str;
while ((str = sr.ReadLine()) != null)
{
sb.Append(str);
}
}
}
}
return sb.ToString();
}
catch (Exception ex)
{
return "";
}
}
Hope this will help you. and for more info, please visit to the following link-
Fastest Way to Read Text Files