c# - Create and upload csv file to ADLS - c#

I am trying to create a .csv file to upload into ADLS using DataLakeFileClient and MemoryStream. It works fine but the csv file has a garbage value after each letter in the csv. (I can see those as 'NUL's when I download and open the file in notepad++).
Here is my code:
DataLakeFileClient fileClient = new DataLakeFileClient(new
Uri($"https://{accountName}.dfs.core.windows.net/{container}/{directory}/{filename}"), credential);
UnicodeEncoding uniEncoding = new UnicodeEncoding();
using (MemoryStream memStream = new MemoryStream(100))
{
byte[] colString = uniEncoding.GetBytes("a,b,c,d,e,f");
memStream.Write(colString, 0, colString.Length);
memStream.WriteByte(0x0A);
byte[] dataString = uniEncoding.GetBytes(string.Join(",", "val1","val2","val3","val4","val5","val6"));
memStream.Write(dataString, 0, dataString.Length);
memStream.Position = 0;
fileClient.Upload(memStream);
}

It's because your LF (0x0A) is one byte instead of 2.
Add after it another byte:
memStream.WriteByte(0x0A);
memStream.WriteByte(0x0);

Related

C# RijndaelManaged to Python equivalent

I have the following C# code (code is inherited and can't compile it). This is used to decrypt and unzip a saved file.
using System.Security.Cryptography;
using System.Text;
using ICSharpCode.SharpZipLib.Zip;
//Not the real key but same amount of chars
private const string kEncyptionKey = "01234567";
public string DecryptAndDecompressText (string strFileName)
{
// Decryption ///
FileStream fin = null;
try
{
fin = new FileStream(strFileName, FileMode.Open, FileAccess.Read);
}
catch (System.IO.FileNotFoundException)
{
return "";
}
MemoryStream memoryToDecompress = new MemoryStream();
UnicodeEncoding UE = new UnicodeEncoding();
RijndaelManaged RMCrypto = new RijndaelManaged();
// This is the encryption key for our file
byte[] key = UE.GetBytes(kEncyptionKey);
// Decrypt the data to a stream
CryptoStream cs = new CryptoStream( memoryToDecompress,
RMCrypto.CreateDecryptor(key, key),
CryptoStreamMode.Write);
byte [] fileBuffer = new byte[fin.Length];
fin.Read(fileBuffer, 0, fileBuffer.Length);
cs.Write(fileBuffer, 0, fileBuffer.Length);
fin.Close();
// Reset the index of the Memory Stream
memoryToDecompress.Position = 0;
// Let the GC clean this up, we still need the memory stream
//cs.Close();
// Decompress the File
ZipInputStream s;
s = new ZipInputStream(memoryToDecompress);
ZipEntry theEntry;
try
{
theEntry = s.GetNextEntry();
}
catch (System.Exception)
{
// Could not open the file...
return "";
}
}
I'm trying to create a python program to do the same. This is what I've got:
from Crypto.Cipher import AES
KEY = '01234567'.encode('utf-16be')
_f = open('<file>', 'r')
_content = _f.read()
_cipher = AES.new(KEY, AES.MODE_CBC, KEY)
_dcontent = _cipher.decrypt(_content)
with open('extract.zip', 'w') as newfile:
newfile.write(_dcontent)
_f.close()
I'm writing the result to the disk since I expect it to be a zip file (which contains one file). However I can't open the file with Archive Manager.
Any suggestions are welcome!
You have to use the same key. System.Text.UnicodeEncoding is the UTF-16le encoding which also has an equivalent in python:
KEY = '01234567'.encode('utf-16le')
You have to read and write the files in binary mode if you're on Windows:
_f = open('<file>', 'rb')
...
open('extract.zip', 'wb')
You should use the proper zip file library. I am guessing that is something format specific that is failing on your write statement. Using this library should avoid such drawbacks. The open function can take a password as optional in case it is protected.

Decode LZMA2 file with .xz extension using 7zip SDK

I need to decode .xz file created with 7zip. Compression alghorithm is LZMA2. How should I use 7zip SDK for C# to solve this problem?
I already tried with this code, but it didn't work:
var coder = new SevenZip.Compression.LZMA.Decoder();
var input = new MemoryStream(); // filled with byte array of .xz file
var output = new MemoryStream();
// Read the decoder properties
byte[] properties = new byte[5];
zipFile.Read(properties, 0, 5);
// Read in the decompress file size.
byte[] fileLengthBytes = new byte[8];
zipFile.Read(fileLengthBytes, 0, 8);
long fileLength = BitConverter.ToInt64(fileLengthBytes, 0);
coder.SetDecoderProperties(properties); // this throws exception
coder.Code(zipFile, output, zipFile.Length, fileLength, null);
output.Flush();
output.Close();
I marked a line which causes exception.

Convert open xml string to byte[]

so, I am editing a word document, using OpenXML. And for some reasons, I convert it all into a string:
//conversion du byte en memorystream
using (var file = new MemoryStream(text))
using (var reader = new StreamReader(file))
{
WordprocessingDocument wordDoc = WordprocessingDocument.Open(file, true);
using (StreamReader sr = new StreamReader(wordDoc.MainDocumentPart.GetStream()))
{
docText = sr.ReadToEnd();
}
}
And then, I convert it as a byte.
But, a simple convert will not work:
byte[] back2Byte = System.Text.Encoding.ASCII.GetBytes(docText );
Because the string is a open xml string.
Tried this, but always got a corrupted file when I tried to open it with Word:
var repo = new System.IO.MemoryStream(System.Text.Encoding.UTF8.GetBytes(docText));
byte[] buffer = new byte[16 * 1024];
MemoryStream ms = new MemoryStream();
int read;
while ((read = repo.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
byte[] back2Byte = ms.ToArray();
So, this doesn't work either:
byte[] back2Byte = new byte[docText.Length * sizeof(char)];
System.Buffer.BlockCopy(docText.ToCharArray(), 0, back2Byte, 0, back2Byte.Length);
edit : After some checkings, it seems it is write as a openxml document into the database, and so, word cannot read it. There is no error when i open it with notepad
How can I correct this?
So, the real issue is, how can I convert a OpenXML string to a byte that can be open in word?
You cannot do this sort of thing. You are getting the bytes for only one part of an OpenXML document. By definition, all Microsoft Office documents are multi-part OpenXML documents. You could theoretically capture the bytes for all the parts using a technique like you're currently using, but you would also have to capture all the part/relationship information necessary to reconstruct the multi-part document. You'd be better off just reading all the bytes of the file and storing them as-is:
// to read the file as bytes
var fileName = #"C:\path\to\the\file.xlsx";
var fileBytes = File.ReadAllBytes(fileName);
// to recreate the file from the bytes
File.WriteAllBytes(fileName, fileBytes)
If you need a string form of those bytes, try this:
// to convert bytes to a (non-readable) text form
var fileContent = Convert.ToBase64String(fileBytes);
// to convert base-64 back to bytes
var fileBytes = Convert.FromBase64String(fileContent);
Either way, there is absolutely no need to use the OpenXML SDK for your use case.

Zip files and attach them to MailMessage without saving a file

I'm working on a little C# ASP.NET web app that pulls 3 files from my server, creates a zip of those files, and sends the zip file to an e-mail recipient.
The problem I'm having is finding a way to combine those 3 files without creating a zip file on the hard drive of the server. I think I need to use some sort of memorystream or filestream, but I'm in a little beyond my understanding when it comes to merging them into 1 zip file. I've tried SharpZipLib and DotNetZip, but I haven't been able to figure it out.
The reason I don't want the zip saved locally is that there might be a number of users on this app at once, and I don't want to clog up my server machine with those zips. I'm looking for 2 answers, how to zip files without saving the zip as a file, and how to attach that zip to a MailMessage.
Check this example for SharpZipLib:
https://github.com/icsharpcode/SharpZipLib/wiki/Zip-Samples#wiki-anchorMemory
using ICSharpCode.SharpZipLib.Zip;
// Compresses the supplied memory stream, naming it as zipEntryName, into a zip,
// which is returned as a memory stream or a byte array.
//
public MemoryStream CreateToMemoryStream(MemoryStream memStreamIn, string zipEntryName) {
MemoryStream outputMemStream = new MemoryStream();
ZipOutputStream zipStream = new ZipOutputStream(outputMemStream);
zipStream.SetLevel(3); //0-9, 9 being the highest level of compression
ZipEntry newEntry = new ZipEntry(zipEntryName);
newEntry.DateTime = DateTime.Now;
zipStream.PutNextEntry(newEntry);
StreamUtils.Copy(memStreamIn, zipStream, new byte[4096]);
zipStream.CloseEntry();
zipStream.IsStreamOwner = false; // False stops the Close also Closing the underlying stream.
zipStream.Close(); // Must finish the ZipOutputStream before using outputMemStream.
outputMemStream.Position = 0;
return outputMemStream;
// Alternative outputs:
// ToArray is the cleaner and easiest to use correctly with the penalty of duplicating allocated memory.
byte[] byteArrayOut = outputMemStream.ToArray();
// GetBuffer returns a raw buffer raw and so you need to account for the true length yourself.
byte[] byteArrayOut = outputMemStream.GetBuffer();
long len = outputMemStream.Length;
}
Try this:
public static Attachment CreateAttachment(string fileNameAndPath, bool zipIfTooLarge = true, int bytes = 1 << 20)
{
if (!zipIfTooLarge)
{
return new Attachment(fileNameAndPath);
}
var fileInfo = new FileInfo(fileNameAndPath);
// Less than 1Mb just attach as is.
if (fileInfo.Length < bytes)
{
var attachment = new Attachment(fileNameAndPath);
return attachment;
}
byte[] fileBytes = File.ReadAllBytes(fileNameAndPath);
using (var memoryStream = new MemoryStream())
{
string fileName = Path.GetFileName(fileNameAndPath);
using (var zipArchive = new ZipArchive(memoryStream, ZipArchiveMode.Create))
{
ZipArchiveEntry zipArchiveEntry = zipArchive.CreateEntry(fileName, CompressionLevel.Optimal);
using (var streamWriter = new StreamWriter(zipArchiveEntry.Open()))
{
streamWriter.Write(Encoding.Default.GetString(fileBytes));
}
}
var attachmentStream = new MemoryStream(memoryStream.ToArray());
string zipname = $"{Path.GetFileNameWithoutExtension(fileName)}.zip";
var attachment = new Attachment(attachmentStream, zipname, MediaTypeNames.Application.Zip);
return attachment;
}
}

Create new FileStream out of a byte array

I am attempting to create a new FileStream object from a byte array. I'm sure that made no sense at all so I will try to explain in further detail below.
Tasks I am completing:
1) Reading the source file which was previously compressed
2) Decompressing the data using GZipStream
3) copying the decompressed data into a byte array.
What I would like to change:
1) I would like to be able to use File.ReadAllBytes to read the decompressed data.
2) I would then like to create a new filestream object usingg this byte array.
In short, I want to do this entire operating using byte arrays. One of the parameters for GZipStream is a stream of some sort, so I figured I was stuck using a filestream. But, if some method exists where I can create a new instance of a FileStream from a byte array - then I should be fine.
Here is what I have so far:
FolderBrowserDialog fbd = new FolderBrowserDialog(); // Shows a browser dialog
fbd.ShowDialog();
// Path to directory of files to compress and decompress.
string dirpath = fbd.SelectedPath;
DirectoryInfo di = new DirectoryInfo(dirpath);
foreach (FileInfo fi in di.GetFiles())
{
zip.Program.Decompress(fi);
}
// Get the stream of the source file.
using (FileStream inFile = fi.OpenRead())
{
//Create the decompressed file.
string outfile = #"C:\Decompressed.exe";
{
using (GZipStream Decompress = new GZipStream(inFile,
CompressionMode.Decompress))
{
byte[] b = new byte[blen.Length];
Decompress.Read(b,0,b.Length);
File.WriteAllBytes(outfile, b);
}
}
}
Thanks for any help!
Regards,
Evan
It sounds like you need to use a MemoryStream.
Since you don't know how many bytes you'll be reading from the GZipStream, you can't really allocate an array for it. You need to read it all into a byte array and then use a MemoryStream to decompress.
const int BufferSize = 65536;
byte[] compressedBytes = File.ReadAllBytes("compressedFilename");
// create memory stream
using (var mstrm = new MemoryStream(compressedBytes))
{
using(var inStream = new GzipStream(mstrm, CompressionMode.Decompress))
{
using (var outStream = File.Create("outputfilename"))
{
var buffer = new byte[BufferSize];
int bytesRead;
while ((bytesRead = inStream.Read(buffer, 0, BufferSize)) != 0)
{
outStream.Write(buffer, 0, bytesRead);
}
}
}
}
Here is what I ended up doing. I realize that I did not give sufficient information in my question - and I apologize for that - but I do know the size of the file I need to decompress as I am using it earlier in my program. This buffer is referred to as "blen".
string fi = #"C:\Path To Compressed File";
// Get the stream of the source file.
// using (FileStream inFile = fi.OpenRead())
using (MemoryStream infile1 = new MemoryStream(File.ReadAllBytes(fi)))
{
//Create the decompressed file.
string outfile = #"C:\Decompressed.exe";
{
using (GZipStream Decompress = new GZipStream(infile1,
CompressionMode.Decompress))
{
byte[] b = new byte[blen.Length];
Decompress.Read(b,0,b.Length);
File.WriteAllBytes(outfile, b);
}
}
}

Categories

Resources