I want to compress a json string and save it als gzip file. I have the following code below which works but only saving it as a .json file. Using compression the files are smaller. If I compress the json string and change the filename to .gz I cannot open it using 7zip.
string containerEndpoint = string.Format("https://xxx.blob.core.windows.net/datafolder");
BlobContainerClient containerClient = new BlobContainerClient(new Uri(containerEndpoint), null);
byte[] byteArray = Encoding.ASCII.GetBytes(jsonString);
MemoryStream stream = new MemoryStream(byteArray);
string fileName = "test.gz";
string blobPath = string.Format("folder/{0}", fileName);
await containerClient.UploadBlobAsync(blobPath, stream);
I added your suggestions and it worked. My code is now as follows:
byte[] buffer = Encoding.UTF8.GetBytes(json);
using (var stream = new MemoryStream())
{
using (var gZipStream = new GZipStream(stream, CompressionMode.Compress, true))
{
gZipStream.Write(buffer, 0, buffer.Length);
}
var compressedData = new byte[stream.Length];
stream.Read(compressedData, 0, compressedData.Length);
var gZipBuffer = new byte[compressedData.Length + 4];
Buffer.BlockCopy(compressedData, 0, gZipBuffer, 4, compressedData.Length);
Buffer.BlockCopy(BitConverter.GetBytes(buffer.Length), 0, gZipBuffer, 0, 4);
stream.Position = 0;
string blobPath = string.Format("folder/{0}", fileName);
await containerClient.UploadBlobAsync(blobPath, stream);
}
Related
Does anyone know why I'm getting the "Unexpected end of data" error message when un-gzipping the gzip file?
To verify the bytes data is not corrupted, I use the FooTest4.csv to write to file and was able to opened the file successfully.
Both 'FooTest3.csv.gzand 'FooTest2.csv.gz ran into "Unexpected end of data" when un-gzipping.
public static List<byte> CompressFile(List<byte> parmRawBytes)
{
//Initialize variables...
List<byte> returnModifiedBytes = null;
File.WriteAllBytes(#"X:\FooTest4.csv", parmRawBytes.ToArray());
using (var memoryStream = new MemoryStream())
{
using (var gzipStream = new GZipStream(memoryStream, CompressionMode.Compress, false))
{
gzipStream.Write(parmRawBytes.ToArray(), 0, parmRawBytes.ToArray().Length);
gzipStream.Flush();
File.WriteAllBytes(#"X:\FooTest3.csv.gz", memoryStream.ToArray());
returnModifiedBytes = memoryStream.ToArray().ToList();
}
}
File.WriteAllBytes(#"X:\FooTest2.csv.gz", returnModifiedBytes.ToArray());
return returnModifiedBytes;
}
GZipStream needs to be closed so it can write some terminating data to the end of the buffer to complete the gzip encoding.
byte[] inputBytes = ...;
using (var compressedStream = new MemoryStream())
{
using (var compressor = new GZipStream(compressedStream, CompressionMode.Compress))
{
compressor.Write(inputBytes, 0, inputBytes.Length);
}
// get bytes after the gzip stream is closed
File.WriteAllBytes(pathToFile, compressedStream.ToArray());
}
Instead of loading the bytes, compressing and saving them you could do do compression and writing at once. Also I don't know why you're using List<Byte> instead of byte[], maybe this could be it.
void CompressFile(string inputPath, string outputPath)
{
Stream readStream = new FileStream(inputPath, Filemode.Open);
Stream writeStream = new FileStream(outputPath, FileMode.Create);
Stream compressionStream = new GZipStream(writeStream. CompressionMode.Compress);
byte[] data = new byte[readStream.Length];
readStream.Read(data, 0, data.Length);
compressionStream.Write(data, 0, data.Length);
readStream.Close();
writeStream.Close();
}
byte[] CompressFile(string inputPath)
{
byte[] data = File.ReadAllBytes(inputPath);
MemoryStream memStream = new MemoryStream(data);
var gzipStream = new GZipStream(memStream, CompressionMode.Compress);
gzipStream.Write(data, 0, data.Length);
gzipStream.Close();
return gzipStream.ToArray();
}
PS: I wrote the code in the text editor, so there might be errors. Also you say the error is on the "unzippiing", why no show us the unzip code?
I have some PDFs stored in S3 and I'm trying to make a link that will download them. This is the method that I have so far:
public virtual ActionResult DownloadPDF(string filename)
{
string secretKey = this.UnitOfWork.ApplicationSettingRepository.GetOrCreateByName<string>("StorageProvider_AmazonS3_SecretKey");
string accessKey = this.UnitOfWork.ApplicationSettingRepository.GetOrCreateByName<string>("StorageProvider_AmazonS3_AccessKey");
var bucket = this.UnitOfWork.ApplicationSettingRepository.GetOrCreateByName<string>("StorageProvider_AmazonS3_BucketName");
var serviceUrl = this.UnitOfWork.ApplicationSettingRepository.GetOrCreateByName<string>("StorageProvider_AmazonS3_ServiceUrl");
AmazonS3Config config = new AmazonS3Config();
config.ServiceURL = serviceUrl;
var client = Amazon.AWSClientFactory.CreateAmazonS3Client(
accessKey,
secretKey,
config
);
GetObjectRequest request = new GetObjectRequest();
request.BucketName = bucket;
request.Key = "userfiles/MSD IMAGES/ProductDocumentation/" + filename;
GetObjectResponse response = client.GetObject(request);
response.WriteResponseStreamToFile("\\Downloads\\" + filename);
}
I got that method from this documentation here but I want to download the download to show in the browser, and go to the standard downloads folder. I created this method previously for downloading csv files:
var csv = new StringBuilder();
csv.AppendLine("col1,col2,col3");
var bytes = Encoding.UTF8.GetBytes(csv.ToString());
var response = new FileContentResult(bytes, "text/csv");
response.FileDownloadName = fileName;
return response;
I tried doing the same thing with the S3 file:
GetObjectResponse response = client.GetObject(request);
// response.WriteResponseStreamToFile("\\Downloads\\" + filename);
var bytes = Encoding.UTF8.GetBytes(response.ToString());
var download = new FileContentResult(bytes, "application/pdf");
download.FileDownloadName = filename;
return download;
it downloads a file, but the pdf doesn't work (it fails to load). How do I download the file that I got from S3?
Here's the working solution I came up with:
GetObjectResponse response = client.GetObject(request);
using (Stream responseStream = response.ResponseStream)
{
var bytes = ReadStream(responseStream);
var download = new FileContentResult(bytes, "application/pdf");
download.FileDownloadName = filename;
return download;
}
public static byte[] ReadStream(Stream responseStream)
{
byte[] buffer = new byte[16 * 1024];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
return ms.ToArray();
}
}
you would need to do something like the following
int byteCount
byte[] buffer = new byte[2048]; // read in chunks of 2KB
Stream responseStream= Response.GetResponseStream();
Response.ContentType = "application/pdf";
Response.AddHeader("Content-Disposition", "attachment;filename=" + filename);
while ((byteCount = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
Response.OutputStream.Write(buffer, 0, byteCount);
}
Response.Flush();
Response.Close();
Response.End();
I have an XML data which I would like to compress it using GZipStream and upload it to webservice. I would like to create the gzip file in memory instead of of creating it in local disk. I have tried the following:
public string class1(string url, string xml)
{
byte[] data = Encoding.ASCII.GetBytes(xml);
MemoryStream memory = new MemoryStream();
GZipStream gzip = new GZipStream(memory, CompressionMode.Compress, true);
gzip.Write(data, 0, data.Length);
byte[] zip=memory.ToArray();
HttpWebRequest wReq = (HttpWebRequest)WebRequest.Create(url);
wReq.Method = "POST";
wReq.ContentType = "application/zip";
var reqStream = wReq.GetRequestStream();
reqStream.Write(zip,0,zip.Length);
reqStream.Close();
var wRes = wReq.GetResponse();
var resStream = wRes.GetResponseStream();
var resgzip = new GZipStream(resStream, CompressionMode.Decompress);
var reader = new StreamReader(resgzip);
var textResponse = reader.ReadToEnd();
reader.Close();
resStream.Close();
wRes.Close();
return textResponse;
}
After writing data to webservice the server unzips the file and processess it. While the server decompresses the data an exception is thrown in server "Premature end of file". Please help me in this.
Add below method to convert Stream to MemoryStream
public static MemoryStream Read(Stream stream)
{
MemoryStream memStream = new MemoryStream();
byte[] readBuffer = new byte[4096];
int bytesRead;
while ((bytesRead = stream.Read(readBuffer, 0, readBuffer.Length)) > 0)
memStream.Write(readBuffer, 0, bytesRead);
return memStream;
}
then call as below.
var wRes = wReq.GetResponse();
var memstream = Read(wRes.GetResponseStream());
var resgzip = new GZipStream(memstream, CompressionMode.Decompress);
var reader = new StreamReader(resgzip);
var textResponse = reader.ReadToEnd();
I have json string something like this
[ { "SectionDataId": "85247","Product": "0.00"},{ "SectionDataId": "85249","Product": "12.38"},{ "SectionDataId": "85264","Product": "0.00"}]
Which I have compressed using C# code found on this forum
private string encode(string json)
{
byte[] buffer = Encoding.UTF8.GetBytes(json);
MemoryStream ms = new MemoryStream();
using (GZipStream zip = new GZipStream(ms, CompressionMode.Compress, true))
{
zip.Write(buffer, 0, buffer.Length);
}
ms.Position = 0;
MemoryStream outStream = new MemoryStream();
byte[] compressed = new byte[ms.Length];
ms.Read(compressed, 0, compressed.Length);
byte[] gzBuffer = new byte[ms.Length + 4];
System.Buffer.BlockCopy(compressed, 0, gzBuffer, 4, compressed.Length);
System.Buffer.BlockCopy(BitConverter.GetBytes(buffer.Length), 0, gzBuffer, 0, 4);
return Convert.ToBase64String(gzBuffer);
}
This output string I have given as web service output so i can use this string in javascript. but I was not able to uncompress the string using javascript.I there any javascript library or code to solve this problem.
I am writing a C# assembly to take advantage of the GZip functionality in newer versions of the .NET framework, and I am copying code I have used successfully elsewhere.
The way this works is that an application we are using feeds a stream to the assembly, the assembly reads it, compresses it, then returns the compressed result as a string. That string is then put back into the assembly through a stream to uncompress it (normally it will be stored, but I am simply running a basic test).
However, when I feed the string back to the assembly, it errors out while reading it from a byte array that is read from the application's stream. Here is the code:
private void ReadStream(IStream stream, out byte[] data)
{
using (MemoryStream writer = new MemoryStream())
{
IntPtr rwBytes = Marshal.AllocHGlobal(4);
int _rwBytes = 0;
data = new byte[0xafc8];
do
{
stream.Read(data, 0xafc8, rwBytes);
_rwBytes = Marshal.ReadInt32(rwBytes);
writer.Write(data, 0, _rwBytes);
}
while (_rwBytes > 0);
writer.Close();
Marshal.FreeHGlobal(rwBytes);
}
}
public string CompressString([In, MarshalAs(UnmanagedType.AsAny)] object appStream)
{
byte[] buffer = null;
ReadStream(appStream as IStream, out buffer);
MemoryStream ms = new MemoryStream();
using (GZipStream zip = new GZipStream(ms, CompressionMode.Compress, true))
{
zip.Write(buffer, 0, buffer.Length);
}
ms.Position = 0;
MemoryStream outStream = new MemoryStream();
byte[] compressed = new byte[ms.Length];
ms.Read(compressed, 0, compressed.Length);
byte[] gzBuffer = new byte[compressed.Length + 4];
System.Buffer.BlockCopy(compressed, 0, gzBuffer, 4, compressed.Length);
System.Buffer.BlockCopy(BitConverter.GetBytes(buffer.Length), 0, gzBuffer, 0, 4);
string str = Convert.ToBase64String(gzBuffer);
gzBuffer = Convert.FromBase64String(str);
return str;
}
public string DecompressString([In, MarshalAs(UnmanagedType.AsAny)] object appStream)
{
byte[] buffer = null;
ReadStream(appStream as IStream, out buffer);
string compressedText = Encoding.UTF8.GetString(buffer);
byte[] gzBuffer = Convert.FromBase64String(compressedText);
using (MemoryStream ms = new MemoryStream())
{
int msgLength = BitConverter.ToInt32(gzBuffer, 0);
ms.Write(gzBuffer, 4, gzBuffer.Length - 4);
buffer = new byte[msgLength];
ms.Position = 0;
using (GZipStream zip = new GZipStream(ms, CompressionMode.Decompress))
{
zip.Read(buffer, 0, buffer.Length);
}
}
return Encoding.UTF8.GetString(buffer);
}
The string is returned from the CompressString function as:
yK8AAB+LCAAAAAAABADtvQdgHEmWJSYvbcp7f0r1StfgdKEIgGATJNiQQBDswYjN5pLsHWlHIymrKoHKZVZlXWYWQMztnbz33nvvvffee++997o7nU4n99//P1xmZAFs9s5K2smeIYCqyB8/fnwfPyK+ndd5WjRplk6rxarOm6aolmmbN+04TZ/n7cdN2uTU4jwt2vSqqt8241/jR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Pzo+dHz/67nR9j86PnR86PnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89/99/fjSCHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj5//7z49G8KPnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Px///nRCH70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj57/7z8/GsGPnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fP/nef/AYk/UbjIrwAA
When it is read back from the stream in the Encoding.UTF8.GetString call, it is:
yK8AAB+LCAAAAAAABADtvQdgHEmWJSYvbcp7f0r1StfgdKEIgGATJNiQQBDswYjN5pLsHWlHIymrKoHKZVZlXWYWQMztnbz33nvvvffee++997o7nU4n99//P1xmZAFs9s5K2smeIYCqyB8/fnwfPyK+ndd5WjRplk6rxarOm6aolmmbN+04TZ/n7cdN2uTU4jwt2vSqqt8241/jR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Pzo+dHz/67nR9j86PnR86PnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89/99/fjSCHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj5//7z49G8KPnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Px///nRCH70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj57/7z8/GsGPnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fP/nef/AYk/UbjIrwAA
These strings are identical. I can even convert it back and forth in the CompressString function with no problem. Any thoughts out there? This is very strange.
Don't use UTF8, you should use the ascii text encoding instead. When you get the bytes from your string, you'll end up with some extra bytes due to the way UTF strings are encoded.