Develop Reference

How to encode a csv file to UTF-8 using C# and asp.net core?

I have a csv file which contains latin characters (Ascii value > 127). The file gets uploaded with any type of encoding and shows the right data after uploading. But it gets converted automatically to UTF8 after performing operations on the file.
But I am not able to see the same characters when it is converted to UTF8 after performing operations.
I believe if I will upload the files with UTF8 encoding only then I will see the same characters that were present while uploading the file. So I want to encode the file with UTF8 Encoding.
I am getting IForm File from the function. I tried these methods to change the encoding but it does not affect the file in any way.
First method
//'file' is the IForm file
string[] filecontent;
StreamReader sr = new StreamReader(file.FileName);
string data = sr.ReadLine();
filecontent = data.Split(",");
File.WriteAllLines(file.FileName, filecontent, Encoding.UTF8);
Second method
var fileStream2 = File.OpenWrite(file.FileName);
var sw = new StreamWriter(fileStream2, Encoding.UTF8, 1024, false);
sw.Write(fileStream2);
sw.Close();
Is there any other method to do this or is there any other library to encode the csv file with UTF 8 directly?

Open .prn file that includes image with right Encoding using c#

I need to open a .prn file and replace some strings.
In the .prn file I included an image, that has a string like this:
When I open the .prn file, C# is not able to read the string as it is.
Probably, it misses some encoding, but not sure which one.
I tried different encodings, but without success.
Here is the code that opens the file in read mode:
string text = File.ReadAllText(root + #"testImage.prn");
c# reads that string in this way
and i'm not able to print the file with the image included.
Thanks in advance for your help.

Most PRN files contain ISO encoding. So, try using ISO encoding and read the file using System.IO.StreamReader with explicitly specifying the desired encoding.
The following example worked perfectly in my case:
System.Text.Encoding encoding = System.Text.Encoding.GetEncoding("ISO-8859-1");
string text;
using (System.IO.StreamReader sr = new System.IO.StreamReader(path, encoding))
{
text = sr.ReadToEnd();
}

In Java, it worked this way for me: Using Stream and charset ISO-8859-1.
Stream<String> stream = Files.lines(Paths.get(filePath), Charset.forName("ISO-8859-1"));

How do you read a UTF8 Arabic text file in Metro?

I'm using the following code to read the contents of the text file. The file is encoded in some sort of Utf8 format:
String File = "ms-appx:///Arabic/file.txt";
contents = await Windows.Storage.PathIO.ReadTextAsync(File, Windows.Storage.Streams.UnicodeEncoding.Utf8);
But the above gives me the error:
WinRT information: No mapping for the Unicode character exists in the target multi-byte code page.
Any ideas what I'm doing wrong here?
Thanks

I had a similar issue trying to read text files that contained certain characters (’, °, –) in a file that was using "Western European (Windows) - Codepage 1252" encoding.
The solution in my case was to force Visual Studio to save the files using UTF-8 encoding.
Open the file in Visual Studio
File > Advanced Save Options... >
Change the encoding to "Unicode (UTF-8 with signature) - Codepage 65001"
Save the file

Try using Windows.Storage.Streams.DataReader:
StorageFolder folder =
Windows.ApplicationModel.Package.Current.InstalledLocation;
StorageFile file = await folder.GetFileAsync("ms-appx:///Arabic/file.txt");
var stream = (await file.OpenAsync(FileAccessMode.Read));
Windows.Storage.Streams.DataReader mreader =
new Windows.Storage.Streams.DataReader(stream.GetInputStreamAt(0));
byte[] dgram = new byte[file.Size];
await mreader.LoadAsync((uint)dgram.Length);
mreader.ReadBytes(dgram);
Hope it helps.

c#, Excel + csv: how to get the correct encoding?

I've been trying this for quite a while now, but can't figure it out. I'm trying to export data to Excel via a *.csv file. It works great so far, but I have some encoding problems when opening the files in Excel.
(original string on the left, EXCEL result on the right):
Messwert(µm / m) ==> Messwert(µm / m)
Dümme Mässöng ==> Dümme Mässöng
Notepad++ tells me that the file is encoded "ANSI as UTF8"(WTF?)
So here are different ways I tried to get a valid result:
obvious implementation:
tWriter.Write(";Messwert(µm /m)");
more sophisticated one (tried probably a dozen or more encoding combinations:)
tWriter.Write(Encoding.Default.GetString(Encoding.Unicode.GetBytes(";Messwert(µm /m)")));
tWriter.Write(Encoding.ASCII.GetString(Encoding.Unicode.GetBytes(";Messwert(µm /m)")));
and so on
Whole source code for the method creating the data:
MemoryStream tStream = new MemoryStream();
StreamWriter tWriter = new StreamWriter(tStream);
tWriter.Write("\uFEFF");
tWriter.WriteLine(string.Format("{0}", aMeasurement.Name));
tWriter.WriteLine(aMeasurement.Comment);
tWriter.WriteLine();
tWriter.WriteLine("Zeit in Minuten;Messwert(µm / m)");
TimeSpan tSpan;
foreach (IMeasuringPoint tPoint in aMeasurement)
{
tSpan = new TimeSpan(tPoint.Time - aMeasurement[0].Time);
tWriter.WriteLine(string.Format("{0};{1};", (int)tSpan.TotalMinutes, getMPString(tPoint)));
}
tWriter.Flush();
return tStream;
Generated CSV file:
Dümme Mössäng
Testmessung die erste
Zeit in Minuten;Messwert(µm / m)
0;-703;
0;-381;
1;1039;
1;1045;
2;1457;
2;1045;

This worked perfect for me:
private const int WIN_1252_CP = 1252; // Windows ANSI codepage 1252
this._writer = new StreamWriter(fileName, false, Encoding.GetEncoding(WIN_1252_CP));
CSV encoding issues (Microsoft Excel)

try the following:
using (var sw = File.Create(Path.Combine(txtPath.Text, "UTF8.csv")))
{
var preamble = Encoding.UTF8.GetPreamble();
sw.Write(preamble, 0, preamble.Length);
var data = Encoding.UTF8.GetBytes("懘荧,\"Hello\",text");
sw.Write(data, 0, data.Length);
}
It writes the proper UTF8 preamble to the file before writing the UTF8 encoded CSV.

This solution is written up as a fix for a Java application however you should be able to do something similar in C#. You may also want to look at the documentation on the StreamWriter class, in the remarks it refers to the Byte Order Mark (BOM).

"ANSI as UTF8"(WTF?)
NotePad++ is probably correct. The encoding is UTF8 (i.e., correct Unicode header), but only contains ANSI data (i.e., é is not encoded in correct UTF8 way, which would mean two bytes).
Or: it is the other way around. It is ANSI (no file header BOM), but the encoding of the individual characters is, or looks like, UTF8. This would explain the ü and other characters expanding in more than one other character. You can fix this by forcing the file to be read as Unicode.
If it's possible to post (part of) your CSV, we may be able to help fixing it at the source.
Edit
Now that we've seen your code: can you remove the StreamWriter and replace it with a TextWriter? Also, remove the hand-encoding of the BOM, it is not necessary. When you create a TextWriter, you can specify the encoding (don't use ASCII, try UTF8).

Trevor Germain's helped me to save in the correct encoded format
using (var sw = File.Create(Path.Combine(txtPath.Text, "UTF8.csv")))
{
var preamble = Encoding.UTF8.GetPreamble();
sw.Write(preamble, 0, preamble.Length);
var data = Encoding.UTF8.GetBytes("懘荧,\"Hello\",text");
sw.Write(data, 0, data.Length);
}

I'd suggest you open up the text file in a hex editor, and see what it really is. The BOM for UTF-16 is 0xFEFF, which the writing code is apparently writing to the stream - but the rest of the writing doesn't specify an encoding to use - it would use the default encoding of the StreamWriter, which is UTF-8. There appears to be a mix up of encodings.
When you pop open the file in hex view, if you see lots of 0x00 between the characters, you're working with UTF-16, which is Encoding.Unicode in C#. If there are no 0x00 between chars, the encoding is probably UTF-8.
If the latter case, just fix up the BOM to be EF BB BF rather than FE FF, and read normally with UTF-8 encoding.

For my scenario using StreamWriter I found explicitly passing UTF8 encoding to the StreamWriter enabled excel to read the file using the correct encoding.
See this answer for more details:
https://stackoverflow.com/a/22306937/999048

C# Help reading foreign characters using StreamReader

I'm using the code below to read a text file that contains foreign characters, the file is encoded ANSI and looks fine in notepad. The code below doesn't work, when the file values are read and shown in the datagrid the characters appear as squares, could there be another problem elsewhere?
StreamReader reader = new StreamReader(inputFilePath, System.Text.Encoding.ANSI);
using (reader = File.OpenText(inputFilePath))
Thanks
Update 1: I have tried all encodings found under System.Text.Encoding. and all fail to show the file correctly.
Update 2: I've changed the file encoding (resaved the file) to unicode and used System.Text.Encoding.Unicode and it worked just fine. So why did notepad read it correctly? And why didn't System.Text.Encoding.Unicode read the ANSI file?

You may also try the Default encoding, which uses the current system's ANSI codepage.
StreamReader reader = new StreamReader(inputFilePath, Encoding.Default, true)
When you try using the Notepad "Save As" menu with the original file, look at the encoding combo box. It will tell you which encoding notepad guessed is used by the file.
Also, if it is an ANSI file, the detectEncodingFromByteOrderMarks parameter will probably not help much.

I had the same problem and my solution was simple: instead of
Encoding.ASCII
use
Encoding.GetEncoding("iso-8859-1")
The answer was found here.
Edit: more solutions. This maybe more accurate one:
Encoding.GetEncoding(1252);
Also, in some cases this will work for you too if your OS default encoding matches file encoding:
Encoding.Default;

Yes, it could be with the actual encoding of the file, probably unicode. Try UTF-8 as that is the most common form of unicode encoding. Otherwise if the file ASCII then standard ASCII encoding should work.

Using Encoding.Unicode won't accurately decode an ANSI file in the same way that a JPEG decoder won't understand a GIF file.
I'm surprised that Encoding.Default didn't work for the ANSI file if it really was ANSI - if you ever find out exactly which code page Notepad was using, you could use Encoding.GetEncoding(int).
In general, where possible I'd recommend using UTF-8.

Try a different encoding such as Encoding.UTF8. You can also try letting StreamReader find the encoding itself:
StreamReader reader = new StreamReader(inputFilePath, System.Text.Encoding.UTF8, true)
Edit: Just saw your update. Try letting StreamReader do the guessing.

For swedish Å Ä Ö the only solution form the ones above working was:
Encoding.GetEncoding("iso-8859-1")
Hopefully this will save someone time.

File.OpenText() always uses an UTF-8 StreamReader implicitly. Create your own StreamReader
instance instead and specify the desired encoding.
like
using (StreamReader reader = new StreamReader(#"C:\test.txt", Encoding.Default)
{
// ...
}

I solved my problem of reading portuguese characters, changing the source file on notepad++.
C#
var url = System.Web.HttpContext.Current.Server.MapPath(#"~/Content/data.json");
string s = string.Empty;
using (System.IO.StreamReader sr = new System.IO.StreamReader(url, System.Text.Encoding.UTF8,true))
{
s = sr.ReadToEnd();
}

I'm also reading an exported file which contains french and German languages. I used Encoding.GetEncoding("iso-8859-1"), true which worked out without any challenges.

for Arabic, I used Encoding.GetEncoding(1256). it is working good.

I had a similar problem with ProcessStartInfo and the property StandardOutputEncoding. I set it for German language console output to code page 850. This way I could read the output like ausführen instead of ausf�hren.