Develop Reference

Splitting of text file not working properly in c#

I have requirement of writing to text file.
If the file size exceeds 700MB, create new file & write to it.
I am currently writing data with “|” delimited from database to file & after that check the file size & splitting into multiple files, but the file splits in middle of the line.
It should write till end of line or start that particular line in new file .
I need to write the column names in the first line in the newly splited file.
I am new to c#, could you please suggest me the solution with the sample code.
Please find below code to splitting the file
private static void ReadWriteToFile(string fileNames)
{
string sourceFileName = fileNames;
string destFileLocation = Path.GetDirectoryName(fileNames);
int index = 0;
long maxFileSize = 700 * 1024 * 1024;
byte[] buffer = new byte[65536];
using (Stream source = File.OpenRead(sourceFileName))
{
while (source.Position < source.Length)
{
index++;
string newFileName = Path.Combine(destFileLocation, Path.GetFileNameWithoutExtension(sourceFileName));
newFileName += index.ToString() + Path.GetExtension(sourceFileName);
using (Stream destination = File.OpenWrite(newFileName))
{
while (destination.Position < maxFileSize)
{
int bytes = source.Read(buffer, 0, (int)Math.Min(maxFileSize, buffer.Length));
destination.Write(buffer, 0, bytes);
if (bytes < Math.Min(maxFileSize, buffer.Length))
{
break;
}
}
}
}
}
}
Thanks in advance.
Could you please let me know if there is any alternative best way to do this

Try this, a rewrite of a line file splitter i wrote in my beginning c# times.
(You only have to add the column header as a string in the beginning of a new file.)
private static void SplitAfterMBytes(int splitAfterMBytes, string filename)
{
// Variable for max. file size.
var maxFileSize = splitAfterMBytes * 1048576;
int fileCount = 0;
long byteCount = 0;
StreamWriter writer = null;
try
{
var inputFile = new FileInfo(filename);
var index = filename.LastIndexOf('.');
//get only the name of the file.
var fileStart = filename.Substring(0, index);
// get the file extension
var fileExtension = inputFile.Extension;
// generate a new file name.
var outputFile = fileStart + '_' + fileCount++ + fileExtension;
// file format is like: QS_201101_321.txt.
writer = new StreamWriter(outputFile);
using (var reader = new StreamReader(filename))
{
for (string str; (str = reader.ReadLine()) != null;)
{
byteCount = byteCount + System.Text.Encoding.Unicode.GetByteCount(str);
if (byteCount >= maxFileSize)
{
// max number of bytes reached
// write into the old file, without Newline,
// so that no extra line is written.
writer.Write(str);
// 1. close the actual file.
writer.Close();
// 2. open a new file with number incresed by 1.
outputFile = fileStart + '_' + fileCount++ + fileExtension;
writer = new StreamWriter(outputFile);
byteCount = 0; //reset the counter.
}
else
{
// Write into the old file.
// Use a Linefeed, because Write works without LF.
// like Java ;)
writer.Write(str);
writer.Write(writer.NewLine);
}
}
}
}
catch (Exception ex)
{
// do something useful, like: Console.WriteLine(ex.Message);
}
finally
{
writer.Dispose();
}
}

Unusual Character addition after writing back decoded file

I am using ZXing.Net library to encode and decode my video file using RS Encoder. It works well by adding and and removing parity after encoding and decoding respectively. But When writing decoded file back it is adding "?" characters in file on different locations which was not part of original file. I am not getting why this problem is occurring when writing file back.
Here is my code
using ZXing.Common.ReedSolomon;
namespace zxingtest
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
string inputFileName = #"D:\JM\bin\baseline_30.264";
string outputFileName = #"D:\JM\bin\baseline_encoded.264";
string Content = File.ReadAllText(inputFileName, ASCIIEncoding.Default);
//File.WriteAllText(outputFileName, Content, ASCIIEncoding.Default);
ReedSolomonEncoder enc = new ReedSolomonEncoder(GenericGF.AZTEC_DATA_12);
ReedSolomonDecoder dec = new ReedSolomonDecoder(GenericGF.AZTEC_DATA_12);
//string s = "1,2,4,6,1,7,4,0,0";
//int[] array = s.Split(',').Select(str => int.Parse(str)).ToArray();
int parity = 10;
List<byte> toBytes = ASCIIEncoding.Default.GetBytes(Content.Substring(0, 500)).ToList();
for (int index = 0; index < parity; index++)
{
toBytes.Add(0);
}
int[] bytesAsInts = Array.ConvertAll(toBytes.ToArray(), c => (int)c);
enc.encode(bytesAsInts, parity);
bytesAsInts[1] = 3;
dec.decode(bytesAsInts, parity);
string st = new string(Array.ConvertAll(bytesAsInts.ToArray(), z => (char)z));
File.WriteAllText(outputFileName, st, ASCIIEncoding.Default);
}
}
}
And here is the Hex file view of H.264 bit stream

The problem is that you're handling a binary format as if it is a Text file with an encoding. But based on what you are doing you only seem to be interested in reading some bytes, process them (encode, decode) and then write the bytes back to a file.
If that is what you need then use the proper reader and writer for your files, in this case the BinaryReader and BinaryWriter. Using your code as a starting point this is my version using the earlier mentioned readers/writers. My inputfile and outputfile are similar for the bytes read and written.
string inputFileName = #"input.264";
string outputFileName = #"output.264";
ReedSolomonEncoder enc = new ReedSolomonEncoder(GenericGF.AZTEC_DATA_12);
ReedSolomonDecoder dec = new ReedSolomonDecoder(GenericGF.AZTEC_DATA_12);
const int parity = 10;
// open a file as stream for reading
using (var input = File.OpenRead(inputFileName))
{
const int max_ints = 256;
int[] bytesAsInts = new int[max_ints];
// use a binary reader
using (var binary = new BinaryReader(input))
{
for (int i = 0; i < max_ints - parity; i++)
{
//read a single byte, store them in the array of ints
bytesAsInts[i] = binary.ReadByte();
}
// parity
for (int i = max_ints - parity; i < max_ints; i++)
{
bytesAsInts[i] = 0;
}
enc.encode(bytesAsInts, parity);
bytesAsInts[1] = 3;
dec.decode(bytesAsInts, parity);
// create a stream for writing
using(var output = File.Create(outputFileName))
{
// write bytes back
using(var writer = new BinaryWriter(output))
{
foreach(var value in bytesAsInts)
{
// we need to write back a byte
// not an int so cast it
writer.Write((byte)value);
}
}
}
}
}

Can't fully correct encoding issue from website [duplicate]

I have a string that I receive from a third party app and I would like to display it correctly in any language using C# on my Windows Surface.
Due to incorrect encoding, a piece of my string looks like this in Spanish:
Acción
whereas it should look like this:
Acción
According to the answer on this question:
How to know string encoding in C#, the encoding I am receiving should be coming on UTF-8 already, but it is read on Encoding.Default (probably ANSI?).
I am trying to transform this string into real UTF-8, but one of the problems is that I can only see a subset of the Encoding class (UTF8 and Unicode properties only), probably because I'm limited to the windows surface API.
I have tried some snippets I've found on the internet, but none of them have proved successful so far for eastern languages (i.e. korean). One example is as follows:
var utf8 = Encoding.UTF8;
byte[] utfBytes = utf8.GetBytes(myString);
myString= utf8.GetString(utfBytes, 0, utfBytes.Length);
I also tried extracting the string into a byte array and then using UTF8.GetString:
byte[] myByteArray = new byte[myString.Length];
for (int ix = 0; ix < myString.Length; ++ix)
{
char ch = myString[ix];
myByteArray[ix] = (byte) ch;
}
myString = Encoding.UTF8.GetString(myByteArray, 0, myString.Length);
Do you guys have any other ideas that I could try?

As you know the string is coming in as Encoding.Default you could simply use:
byte[] bytes = Encoding.Default.GetBytes(myString);
myString = Encoding.UTF8.GetString(bytes);
Another thing you may have to remember: If you are using Console.WriteLine to output some strings, then you should also write Console.OutputEncoding = System.Text.Encoding.UTF8;!!! Or all utf8 strings will be outputed as gbk...

string utf8String = "Acción";
string propEncodeString = string.Empty;
byte[] utf8_Bytes = new byte[utf8String.Length];
for (int i = 0; i < utf8String.Length; ++i)
{
utf8_Bytes[i] = (byte)utf8String[i];
}
propEncodeString = Encoding.UTF8.GetString(utf8_Bytes, 0, utf8_Bytes.Length);
Output should look like
Acción
day’s displays
day's
call DecodeFromUtf8();
private static void DecodeFromUtf8()
{
string utf8_String = "day’s";
byte[] bytes = Encoding.Default.GetBytes(utf8_String);
utf8_String = Encoding.UTF8.GetString(bytes);
}

Your code is reading a sequence of UTF8-encoded bytes, and decoding them using an 8-bit encoding.
You need to fix that code to decode the bytes as UTF8.
Alternatively (not ideal), you could convert the bad string back to the original byte array—by encoding it using the incorrect encoding—then re-decode the bytes as UTF8.

Encoding.Convert(Encoding.Default, Encoding.UTF8, Encoding.Default.GetBytes(mystring));

#anothershrubery answer worked for me. I've made an enhancement using StringEntensions Class so I can easily convert any string at all in my program.
Method:
public static class StringExtensions
{
public static string ToUTF8(this string text)
{
return Encoding.UTF8.GetString(Encoding.Default.GetBytes(text));
}
}
Usage:
string myString = "Acción";
string strConverted = myString.ToUTF8();
Or simply:
string strConverted = "Acción".ToUTF8();

If you want to save any string to mysql database do this:->
Your database field structure i phpmyadmin [ or any other control panel] should set to utf8-gerneral-ci
2) you should change your string [Ex. textbox1.text] to byte, therefor
2-1) define byte[] st2;
2-2) convert your string [textbox1.text] to unicode [ mmultibyte string] by :
byte[] st2 = System.Text.Encoding.UTF8.GetBytes(textBox1.Text);
3) execute this sql command before any query:
string mysql_query2 = "SET NAMES 'utf8'";
cmd.CommandText = mysql_query2;
cmd.ExecuteNonQuery();
3-2) now you should insert this value in to for example name field by :
cmd.CommandText = "INSERT INTO customer (`name`) values (#name)";
4) the main job that many solution didn't attention to it is the below line:
you should use addwithvalue instead of add in command parameter like below:
cmd.Parameters.AddWithValue("#name",ut);
++++++++++++++++++++++++++++++++++
enjoy real data in your database server instead of ????

Use the below code snippet to get bytes from csv file
protected byte[] GetCSVFileContent(string fileName)
{
StringBuilder sb = new StringBuilder();
using (StreamReader sr = new StreamReader(fileName, Encoding.Default, true))
{
String line;
// Read and display lines from the file until the end of
// the file is reached.
while ((line = sr.ReadLine()) != null)
{
sb.AppendLine(line);
}
}
string allines = sb.ToString();
UTF8Encoding utf8 = new UTF8Encoding();
var preamble = utf8.GetPreamble();
var data = utf8.GetBytes(allines);
return data;
}
Call the below and save it as an attachment
Encoding csvEncoding = Encoding.UTF8;
//byte[] csvFile = GetCSVFileContent(FileUpload1.PostedFile.FileName);
byte[] csvFile = GetCSVFileContent("Your_CSV_File_NAme");
string attachment = String.Format("attachment; filename={0}.csv", "uomEncoded");
Response.Clear();
Response.ClearHeaders();
Response.ClearContent();
Response.ContentType = "text/csv";
Response.ContentEncoding = csvEncoding;
Response.AppendHeader("Content-Disposition", attachment);
//Response.BinaryWrite(csvEncoding.GetPreamble());
Response.BinaryWrite(csvFile);
Response.Flush();
Response.End();

How can I transform string to UTF-8 in C#?

I have a string that I receive from a third party app and I would like to display it correctly in any language using C# on my Windows Surface.
Due to incorrect encoding, a piece of my string looks like this in Spanish:
Acción
whereas it should look like this:
Acción
According to the answer on this question:
How to know string encoding in C#, the encoding I am receiving should be coming on UTF-8 already, but it is read on Encoding.Default (probably ANSI?).
I am trying to transform this string into real UTF-8, but one of the problems is that I can only see a subset of the Encoding class (UTF8 and Unicode properties only), probably because I'm limited to the windows surface API.
I have tried some snippets I've found on the internet, but none of them have proved successful so far for eastern languages (i.e. korean). One example is as follows:
var utf8 = Encoding.UTF8;
byte[] utfBytes = utf8.GetBytes(myString);
myString= utf8.GetString(utfBytes, 0, utfBytes.Length);
I also tried extracting the string into a byte array and then using UTF8.GetString:
byte[] myByteArray = new byte[myString.Length];
for (int ix = 0; ix < myString.Length; ++ix)
{
char ch = myString[ix];
myByteArray[ix] = (byte) ch;
}
myString = Encoding.UTF8.GetString(myByteArray, 0, myString.Length);
Do you guys have any other ideas that I could try?

As you know the string is coming in as Encoding.Default you could simply use:
byte[] bytes = Encoding.Default.GetBytes(myString);
myString = Encoding.UTF8.GetString(bytes);
Another thing you may have to remember: If you are using Console.WriteLine to output some strings, then you should also write Console.OutputEncoding = System.Text.Encoding.UTF8;!!! Or all utf8 strings will be outputed as gbk...

string utf8String = "Acción";
string propEncodeString = string.Empty;
byte[] utf8_Bytes = new byte[utf8String.Length];
for (int i = 0; i < utf8String.Length; ++i)
{
utf8_Bytes[i] = (byte)utf8String[i];
}
propEncodeString = Encoding.UTF8.GetString(utf8_Bytes, 0, utf8_Bytes.Length);
Output should look like
Acción
day’s displays
day's
call DecodeFromUtf8();
private static void DecodeFromUtf8()
{
string utf8_String = "day’s";
byte[] bytes = Encoding.Default.GetBytes(utf8_String);
utf8_String = Encoding.UTF8.GetString(bytes);
}

Your code is reading a sequence of UTF8-encoded bytes, and decoding them using an 8-bit encoding.
You need to fix that code to decode the bytes as UTF8.
Alternatively (not ideal), you could convert the bad string back to the original byte array—by encoding it using the incorrect encoding—then re-decode the bytes as UTF8.

Encoding.Convert(Encoding.Default, Encoding.UTF8, Encoding.Default.GetBytes(mystring));

#anothershrubery answer worked for me. I've made an enhancement using StringEntensions Class so I can easily convert any string at all in my program.
Method:
public static class StringExtensions
{
public static string ToUTF8(this string text)
{
return Encoding.UTF8.GetString(Encoding.Default.GetBytes(text));
}
}
Usage:
string myString = "Acción";
string strConverted = myString.ToUTF8();
Or simply:
string strConverted = "Acción".ToUTF8();

If you want to save any string to mysql database do this:->
Your database field structure i phpmyadmin [ or any other control panel] should set to utf8-gerneral-ci
2) you should change your string [Ex. textbox1.text] to byte, therefor
2-1) define byte[] st2;
2-2) convert your string [textbox1.text] to unicode [ mmultibyte string] by :
byte[] st2 = System.Text.Encoding.UTF8.GetBytes(textBox1.Text);
3) execute this sql command before any query:
string mysql_query2 = "SET NAMES 'utf8'";
cmd.CommandText = mysql_query2;
cmd.ExecuteNonQuery();
3-2) now you should insert this value in to for example name field by :
cmd.CommandText = "INSERT INTO customer (`name`) values (#name)";
4) the main job that many solution didn't attention to it is the below line:
you should use addwithvalue instead of add in command parameter like below:
cmd.Parameters.AddWithValue("#name",ut);
++++++++++++++++++++++++++++++++++
enjoy real data in your database server instead of ????

Use the below code snippet to get bytes from csv file
protected byte[] GetCSVFileContent(string fileName)
{
StringBuilder sb = new StringBuilder();
using (StreamReader sr = new StreamReader(fileName, Encoding.Default, true))
{
String line;
// Read and display lines from the file until the end of
// the file is reached.
while ((line = sr.ReadLine()) != null)
{
sb.AppendLine(line);
}
}
string allines = sb.ToString();
UTF8Encoding utf8 = new UTF8Encoding();
var preamble = utf8.GetPreamble();
var data = utf8.GetBytes(allines);
return data;
}
Call the below and save it as an attachment
Encoding csvEncoding = Encoding.UTF8;
//byte[] csvFile = GetCSVFileContent(FileUpload1.PostedFile.FileName);
byte[] csvFile = GetCSVFileContent("Your_CSV_File_NAme");
string attachment = String.Format("attachment; filename={0}.csv", "uomEncoded");
Response.Clear();
Response.ClearHeaders();
Response.ClearContent();
Response.ContentType = "text/csv";
Response.ContentEncoding = csvEncoding;
Response.AppendHeader("Content-Disposition", attachment);
//Response.BinaryWrite(csvEncoding.GetPreamble());
Response.BinaryWrite(csvFile);
Response.Flush();
Response.End();

Editing a text file in place through C#

I have a huge text file, size > 4GB and I want to replace some text in it programmatically. I know the line number at which I have to replace the text but the problem is that I do not want to copy all the text (along with my replaced line) to a second file. I have to do this within the source file. Is there a way to do this in C#?
The text which has to be replaced is exactly the same size as the source text (if this helps).

Since the file is so large you may want to take a look at the .NET 4.0 support for memory mapped files. Basically you'll need to move the file/stream pointer to the location in the file, overwrite that location, then flush the file to disk. You won't need to load the entire file into memory.
For example, without using memory mapped files, the following will overwrite a part of an ascii file. Args are the input file, the zero based start index and the new text.
static void Main(string[] args)
{
string inputFilename = args[0];
int startIndex = int.Parse(args[1]);
string newText = args[2];
using (FileStream fs = new FileStream(inputFilename, FileMode.Open, FileAccess.Write))
{
fs.Position = startIndex;
byte[] newTextBytes = Encoding.ASCII.GetBytes(newText);
fs.Write(newTextBytes, 0, newTextBytes.Length);
}
}

Unless the new text is exactly the same size as the old text, you will have to re-write the file. There is no way around it. You can at least do this without keeping the entire file in memory.

Hello I tested the following -works well.This caters to variable length lines separated by Environment.NewLine. if you have fixed length lines you can straightaway seek to it.For converting bytes to string and vice versa you can use Encoding.
static byte[] ReadNextLine(FileStream fs)
{
byte[] nl = new byte[] {(byte) Environment.NewLine[0],(byte) Environment.NewLine[1] };
List<byte> ll = new List<byte>();
bool lineFound = false;
while (!lineFound)
{
byte b = (byte)fs.ReadByte();
if ((int)b == -1) break;
ll.Add(b);
if (b == nl[0]){
b = (byte)fs.ReadByte();
ll.Add(b);
if (b == nl[1]) lineFound = true;
}
}
return ll.Count ==0?null: ll.ToArray();
}
static void Main(string[] args)
{
using (FileStream fs = new FileStream(#"c:\70-528\junk.txt", FileMode.Open, FileAccess.ReadWrite))
{
int replaceLine=1231;
byte[] b = null;
int lineCount=1;
while (lineCount<replaceLine && (b=ReadNextLine(fs))!=null ) lineCount++;//Skip Lines
long seekPos = fs.Position;
b = ReadNextLine(fs);
fs.Seek(seekPos, 0);
string line=new string(b.Select(x=>(char)x).ToArray());
line = line.Replace("Text1", "Text2");
b=line.ToCharArray().Select(x=>(byte)x).ToArray();
fs.Write(b, 0, b.Length);
}
}

I'm guessing you'll want to use the FileStream class and seek to your positon, and place your updated data.