Develop Reference

c# encoding issues with CsvHelper

var orders = new List<Order>();
....
orders.Add(...)
string csvstring;
using (var ms = new MemoryStream())
using (var wr = new StreamWriter(stream, Encoding.UTF8))
using (var csvWriter = new CsvWriter(wr, CultureInfo.InvariantCulture, false))
{
csvWriter.WriteRecords(orders);
csvstring = Encoding.UTF8.GetString(stream.ToArray());
}
And then
sftp.WriteAllText(fileNameAbsolutePath, csvstring, Encoding.UTF8);
The content of the file created in sftp has "feff" in the begining.
" orders.csv: text/plain; charset=utf-8".
This is the first part of the problem. What I am looking is to convert this UTF8 to IS0-8859-1 as
the charset expected in the end file is IS0-8859-1.
May be I should do something like this ?
byte[] bytesSS = Encoding.Convert(Encoding.UTF8, Encoding.GetEncoding("ISO-8859-1"), Encoding.UTF8.GetBytes(csvstring));
string s1 = Encoding.GetEncoding("ISO-8859-1").GetString(bytesSS, 0, bytesSS.Length);
Tried to google for "<feff>" and I quite didn't get the concept of BOM and a way to fix this.

I have no idea which SFTP class you use as .NET itself doesn't have an SFTP client. I'll assume you use this one simply because it came first in a Google search for sftp WriteAllText.
If you want to create a file with a specific encoding, specify it in the StreamWriter constructor instead of UTF8 :
using (var ms = new MemoryStream())
using (var wr = new StreamWriter(stream, Encoding.GetEncoding("ISO-8859-1")))
using (var csvWriter = new CsvWriter(wr, CultureInfo.InvariantCulture, false))
{
csvWriter.WriteRecords(orders);
}
On the other hand, UTF8 and Latin1 (or any codepage) use the exact same values for characters in the range 0-127. If you want to send only English text, there won't be any difference no matter which encoding you use. If the actual requirement is to create a UTF8 file without a BOM, you can specify it by using the appropriate UTF8Encoding constructor :
var utf8NoBom=new UTF8Encoding(false);
using (var ms = new MemoryStream())
using (var wr = new StreamWriter(stream, utf8NoBom)))
using (var csvWriter = new CsvWriter(wr, CultureInfo.InvariantCulture, false))
{
csvWriter.WriteRecords(orders);
}
All SFTP clients have (or should have) a way to upload data using a stream. This means you can use Stream.CopyTo to copy data from the memory stream to the upload stream. Assuming OpenWrite is available, you can modify the code to:
using (var ms = new MemoryStream())
{
using (var wr = new StreamWriter(stream, Encoding.GetEncoding("ISO-8859-1")))
using (var csvWriter = new CsvWriter(wr, CultureInfo.InvariantCulture, false))
{
csvWriter.WriteRecords(orders);
}
ms.Position=0;
using(var stream=sftp.OpenWrite(somePath))
{
ms.CopyTo(stream);
}
}
When the CsvHelper completes, the MemoryStream's position is at the end of the stream and CopyTo wouldn't copy anything. By using ms.Position you move the position to the start of the stream.

Converting DotNetZip memory stream to string

I am trying to read a file within a zip to check if that file has a certain string in it. But I can seem to get the "file" (memory stream) into a string in order to search it.
When I use the following code "stringOfStream" is always blank, what am I doing wrong? The reader always has a length and read byte returns different numbers.
using (ZipFile zip = ZipFile.Read(currentFile.FullName))
{
ZipEntry e = zip[this.searchFile.Text];
using (MemoryStream reader = new MemoryStream())
{
e.Extract(reader);
var stringReader = new StreamReader(reader);
var stringOfStream = stringReader.ReadToEnd();
}
}
Thanks

I think when you call Extract the position of the stream goes to the end of the file, so you need to reposition again to get the data.
Can you try this please :
using (ZipFile zip = ZipFile.Read(currentFile.FullName))
{
ZipEntry e = zip[this.searchFile.Text];
using (MemoryStream reader = new MemoryStream())
{
e.Extract(reader);
reader.Position = 0;
var stringReader = new StreamReader(reader);
var stringOfStream = stringReader.ReadToEnd();
}
}
Check if it works or not.

Using a generic stream to create a zipped file with SharpCompress

Since System.IO.Compression seems to be out of reach for now if I want to use both dotnet core + net461, I've tried with SharpCompress.
The "read zip" part was easy, but I am having trouble finding out how to write to a zip stream.
The wiki of the project is a bit outdated. This is the only example that I've found that applies to writing to streams. I've tried to follow it and adapt it to my needs, but I am stuck at the exception it throws:
using Microsoft.VisualStudio.TestTools.UnitTesting;
using SharpCompress.Common;
using SharpCompress.Compressors.Deflate;
using SharpCompress.Writers;
using System;
using System.IO;
namespace DbManager.DjdbCore.Tests
{
[TestClass]
public class ZipTests
{
public ZipTests()
{
Directory.SetCurrentDirectory(AppContext.BaseDirectory);
}
[TestMethod]
public void Test()
{
var zip = File.OpenWrite(#"..\..\..\..\..\test-resources\zip_file_test.zip");
var writerOptions = new WriterOptions(CompressionType.Deflate);
var zipWriter = WriterFactory.Open(zip, ArchiveType.Zip, writerOptions);
var memoryStream = new MemoryStream();
var binaryWriter = new BinaryWriter(memoryStream);
binaryWriter.Write("Test string inside binary file - text to fill it up: qoiwjqefñlawijfñlaskdjfioqwjefñalskvndñaskvnqo`wiefowainvñaslkfjnwpowiqjfeopwiqjnfjñlaskdjfñlasdfjiowiqjefñaslkdjfñalskjfpqwoiefjqw");
var deflateStream = new DeflateStream(memoryStream, SharpCompress.Compressors.CompressionMode.Compress);
deflateStream.Write(memoryStream.ToArray(), 0, Convert.ToInt32(memoryStream.Length));
// EXCEPTION: SharpCompress.Compressors.Deflate.ZlibException: 'Cannot Read after Writing.'
// Source code: if (_streamMode != StreamMode.Reader) { throw new ZlibException("Cannot Read after Writing."); }
zipWriter.Write("test_file_inside_zip.bin", deflateStream, DateTime.Now);
zip.Flush();
zipWriter.Dispose();
zip.Dispose();
}
}
}
In case it helps, this is what I used (and it worked, but only in dotnet core) using the library System.IO.Compression:
private void WriteAsZipBinary()
{
//Open the zip file if it exists, else create a new one
var zip = ZipPackage.Open(this.FileFullPath, FileMode.OpenOrCreate, FileAccess.ReadWrite);
var zipStream = ZipManager.GetZipWriteStream(zip, nameOfFileInsideZip);
var memoryStream = new MemoryStream();
var binaryWriter = new BinaryWriter(memoryStream);
// Here is where strings etc are written to the binary file:
WriteStuffInBinaryStream(ref binaryWriter);
//Read all of the bytes from the file to add to the zip file
byte[] bites = new byte[Convert.ToInt32(memoryStream.Length - 1) + 1];
memoryStream.Position = 0;
memoryStream.Read(bites, 0, Convert.ToInt32(memoryStream.Length));
binaryWriter.Dispose();
binaryWriter = null;
memoryStream.Dispose();
memoryStream = null;
zipStream.Position = 0;
zipStream.Write(bites, 0, bites.Length);
zip.Close();
}
public static Stream GetZipWriteStream(Package zip, string renamedFileName)
{
//Replace spaces with an underscore (_)
string uriFileName = renamedFileName.Replace(" ", "_");
//A Uri always starts with a forward slash "/"
string zipUri = string.Concat("/", Path.GetFileName(uriFileName));
Uri partUri = new Uri(zipUri, UriKind.Relative);
string contentType = "Zip"; // System.Net.Mime.MediaTypeNames.Application.Zip;
//The PackagePart contains the information:
// Where to extract the file when it's extracted (partUri)
// The type of content stream (MIME type): (contentType)
// The type of compression: (CompressionOption.Normal)
PackagePart pkgPart = zip.CreatePart(partUri, contentType, CompressionOption.Normal);
//Compress and write the bytes to the zip file
return pkgPart.GetStream();
}

I'll post here the answer on github from #adamhathcock (the owner of the project):
[TestMethod]
public void Test()
{
var writerOptions = new WriterOptions(CompressionType.Deflate);
using(var zip = File.OpenWrite(#"..\..\..\..\..\test-resources\zip_file_test.zip"))
using(var zipWriter = WriterFactory.Open(zip, ArchiveType.Zip, writerOptions))
{
var memoryStream = new MemoryStream();
var binaryWriter = new BinaryWriter(memoryStream);
binaryWriter.Write("Test string inside binary file - text to fill it up: qoiwjqefñlawijfñlaskdjfioqwjefñalskvndñaskvnqo`wiefowainvñaslkfjnwpowiqjfeopwiqjnfjñlaskdjfñlasdfjiowiqjefñaslkdjfñalskjfpqwoiefjqw");
memoryStream.Position = 0;
zipWriter.Write("test_file_inside_zip.bin", memoryStream, DateTime.Now);
}
}
2 things:
You forgot to reset the MemoryStream after writing to it so it can be read.
You don't need to manually use the DeflateStream. You've told the ZipWriter what compression to use. If it worked, you would have double compressed the bytes which would be garbage really.

Looking for preventive from "not responding" C#

Yesterday my teachers gives me a task to make something like database in .txt file which has to contains hexes and a C# application which takes all hexes from this database, also with its offSets. Then i gotta use it,the offset, to take the hex from file on this offset and compare the both haxes, does they are same.
I am using fileSystemWatcher to "spy" chosen directory for new files and with one, two, three or little bit more files it works perfect but if i try to copy very "big" folder the application stops - "not responding".
I have try to find from where the problem comes like i adding and deleting functions and found the "black sheep" -the function which have to take the file's hex which is comply on the given offset.
public string filesHex(string path,int bytesToRead,string offsetLong)
{
byte[] byVal;
try
{
using (Stream fileStream = new FileStream(path, FileMode.Open, FileAccess.Read))
{
BinaryReader brFile = new BinaryReader(fileStream);
offsetLong = offsetLong.Replace("x", string.Empty);
long result = 0;
long.TryParse(offsetLong, System.Globalization.NumberStyles.HexNumber, null, out result);
fileStream.Position = result;
byte[] offsetByte = brFile.ReadBytes(0);
string offsetString = HexStr(offsetByte);
//long offset = System.Convert.ToInt64(offsetString, 16);
byVal = brFile.ReadBytes(bytesToRead);
}
string hex = HexStr(byVal).Substring(2);
return hex;
}

You could create a new Thread and run the filesHex method in it.
You can change your string inside the thread code and get it's value after like this:
public string hex="";
public void filesHex(string path,int bytesToRead,string offsetLong)
{
byte[] byVal;
using (Stream fileStream = new FileStream(path, FileMode.Open, FileAccess.Read))
{
BinaryReader brFile = new BinaryReader(fileStream);
offsetLong = offsetLong.Replace("x", string.Empty);
long result = 0;
long.TryParse(offsetLong, System.Globalization.NumberStyles.HexNumber, null, out result);
fileStream.Position = result;
byte[] offsetByte = brFile.ReadBytes(0);
string offsetString = HexStr(offsetByte);
//long offset = System.Convert.ToInt64(offsetString, 16);
byVal = brFile.ReadBytes(bytesToRead);
}
hex = HexStr(byVal).Substring(2);
}
This would be your call:
Thread thread = new Thread(() => filesHex("a",5,"A"));//example for parameters.
thread.Start();
string hexfinal=hex;//here you can acess the desired string.
Now it would not freeze the main UI thread because you run your method on a sperate thread.
Goodluck.

Why does one method of string compression return an empty string but another one doesn't?

I'm using GZipStream to compress a string, and I've modified two different examples to see what works. The first code snippet, which is a heavily modified version of the example in the documentation, simply returns an empty string.
public static String CompressStringGzip(String uncompressed)
{
String compressedString;
// Convert the uncompressed source string to a stream stored in memory
// and create the MemoryStream that will hold the compressed string
using (MemoryStream inStream = new MemoryStream(Encoding.Unicode.GetBytes(uncompressed)),
outStream = new MemoryStream())
{
using (GZipStream compress = new GZipStream(outStream, CompressionMode.Compress))
{
inStream.CopyTo(compress);
StreamReader reader = new StreamReader(outStream);
compressedString = reader.ReadToEnd();
}
}
return compressedString;
and when I debug it, all I can tell is nothing is read from reader, which is compressedString is empty. However, the second method I wrote, modified from a CodeProject snippet is successful.
public static String CompressStringGzip3(String uncompressed)
{
//Transform string to byte array
String compressedString;
byte[] uncompressedByteArray = Encoding.Unicode.GetBytes(uncompressed);
using (MemoryStream outStream = new MemoryStream())
{
using (GZipStream compress = new GZipStream(outStream, CompressionMode.Compress))
{
compress.Write(uncompressedByteArray, 0, uncompressedByteArray.Length);
compress.Close();
}
byte[] compressedByteArray = outStream.ToArray();
StringBuilder compressedStringBuilder = new StringBuilder(compressedByteArray.Length);
foreach (byte b in compressedByteArray)
compressedStringBuilder.Append((char)b);
compressedString = compressedStringBuilder.ToString();
}
return compressedString;
}
Why is the first code snippet not successful while the other one is? Even though they're slightly different, I don't know why the minor changes in the second snippet allow it to work. The sample string I'm using is SELECT * FROM foods f WHERE f.name = 'chicken';

I ended up using the following code for compression and decompression:
public static String Compress(String decompressed)
{
byte[] data = Encoding.UTF8.GetBytes(decompressed);
using (var input = new MemoryStream(data))
using (var output = new MemoryStream())
{
using (var gzip = new GZipStream(output, CompressionMode.Compress, true))
{
input.CopyTo(gzip);
}
return Convert.ToBase64String(output.ToArray());
}
}
public static String Decompress(String compressed)
{
byte[] data = Convert.FromBase64String(compressed);
using (MemoryStream input = new MemoryStream(data))
using (GZipStream gzip = new GZipStream(input, CompressionMode.Decompress))
using (MemoryStream output = new MemoryStream())
{
gzip.CopyTo(output);
StringBuilder sb = new StringBuilder();
return Encoding.UTF8.GetString(output.ToArray());
}
}
The explanation for a part of the problem comes from this question. Although I fixed the problem by changing the code to what I included in this answer, these lines (in my original code):
foreach (byte b in compressedByteArray)
compressedStringBuilder.Append((char)b);
are problematic, because as dlev aptly phrased it:
You are interpreting each byte as its own character, when in fact that is not the case. Instead, you need the line:
string decoded = Encoding.Unicode.GetString(compressedByteArray);
The basic problem is that you are converting to a byte array based on an encoding, but then ignoring that encoding when you retrieve the bytes.
Therefore, the problem is solved, and the new code I'm using is much more succinct than my original code.

You need to move the code below outside the second using statement:
using (GZipStream compress = new GZipStream(outStream, CompressionMode.Compress))
{
inStream.CopyTo(compress);
outStream.Position = 0;
StreamReader reader = new StreamReader(outStream);
compressedString = reader.ReadToEnd();
}
CopyTo() is not flushing the results to the underlying MemoryStream.
Update
Seems that GZipStream closes and disposes it's underlying stream when it is disposed (not the way I would have designed the class). I've updated the sample above and tested it.