I have following piece of code, it writes some data but not all. What can cut my data ?
using (var stream = new MemoryStream())
{
XmlTextWriter xwr = new XmlTextWriter(stream, Encoding.UTF8);
service.getXElement.WriteTo(xwr);
byte[] bytes = stream.ToArray();
writeStream.Write(bytes, 0, bytes.Length);
}
Related
I am serializing an object to a stream to store as file and then retrieving and trying to deserialize the object, but get an error parsing. Below is code:
var content = JsonConvert.SerializeObject(data);
var output = new MemoryStream();
var writer = new StreamWriter(output, Encoding.UTF8);
writer.Write(content);
writer.Flush();
//write to some file...
//when reading the file
Stream filestream;
//filestream opens some file stream
byte[] buffer = new byte[4096]
using(MemoryStream ms = new MemoryStream()){
int read;
while((read = filestream.Read(buffer, 0, buffer.Length)) > 0){
ms.Write(buffer, 0, read);
}
var data = Encoding.UTF8.GetString(ms.ToArray());
//encounters error here. I can see that first few chars of the string are question marks.
JsonConvert.DeserializeObject<T>(data);
A couple ideas. Are you getting an exception?
Try encapsulating your write & reads into functions and wrap the memoryStream and streamWriter in using statements as well
using var writer = new StreamWriter(...
Additionally you don't need to set the length of the buffer to 4096, that's probably messing up your encoding. Read your memorystream to an array like so
var buffer = ms.ToArray();
How can I write MemoryStream to byte[]
It looks like you're having an issue copying the memory between the 2 memory streams though. Ideally you would write and read to a file or some other source rather than between 2 memory stream objects but you can copy the contents directly using a copyto
https://learn.microsoft.com/en-us/dotnet/api/system.io.stream.copyto?view=net-6.0
Edit: Adding pseudocode
Assuming your content has been read into
Stream filestream;
using var MemoryStream ms = new MemoryStream();
filestream.CopyTo(ms);
var data = Encoding.UTF8.GetString(ms.ToArray());
return JsonConvert.DeserializeObject<T>(data);
Does anyone know why I'm getting the "Unexpected end of data" error message when un-gzipping the gzip file?
To verify the bytes data is not corrupted, I use the FooTest4.csv to write to file and was able to opened the file successfully.
Both 'FooTest3.csv.gzand 'FooTest2.csv.gz ran into "Unexpected end of data" when un-gzipping.
public static List<byte> CompressFile(List<byte> parmRawBytes)
{
//Initialize variables...
List<byte> returnModifiedBytes = null;
File.WriteAllBytes(#"X:\FooTest4.csv", parmRawBytes.ToArray());
using (var memoryStream = new MemoryStream())
{
using (var gzipStream = new GZipStream(memoryStream, CompressionMode.Compress, false))
{
gzipStream.Write(parmRawBytes.ToArray(), 0, parmRawBytes.ToArray().Length);
gzipStream.Flush();
File.WriteAllBytes(#"X:\FooTest3.csv.gz", memoryStream.ToArray());
returnModifiedBytes = memoryStream.ToArray().ToList();
}
}
File.WriteAllBytes(#"X:\FooTest2.csv.gz", returnModifiedBytes.ToArray());
return returnModifiedBytes;
}
GZipStream needs to be closed so it can write some terminating data to the end of the buffer to complete the gzip encoding.
byte[] inputBytes = ...;
using (var compressedStream = new MemoryStream())
{
using (var compressor = new GZipStream(compressedStream, CompressionMode.Compress))
{
compressor.Write(inputBytes, 0, inputBytes.Length);
}
// get bytes after the gzip stream is closed
File.WriteAllBytes(pathToFile, compressedStream.ToArray());
}
Instead of loading the bytes, compressing and saving them you could do do compression and writing at once. Also I don't know why you're using List<Byte> instead of byte[], maybe this could be it.
void CompressFile(string inputPath, string outputPath)
{
Stream readStream = new FileStream(inputPath, Filemode.Open);
Stream writeStream = new FileStream(outputPath, FileMode.Create);
Stream compressionStream = new GZipStream(writeStream. CompressionMode.Compress);
byte[] data = new byte[readStream.Length];
readStream.Read(data, 0, data.Length);
compressionStream.Write(data, 0, data.Length);
readStream.Close();
writeStream.Close();
}
byte[] CompressFile(string inputPath)
{
byte[] data = File.ReadAllBytes(inputPath);
MemoryStream memStream = new MemoryStream(data);
var gzipStream = new GZipStream(memStream, CompressionMode.Compress);
gzipStream.Write(data, 0, data.Length);
gzipStream.Close();
return gzipStream.ToArray();
}
PS: I wrote the code in the text editor, so there might be errors. Also you say the error is on the "unzippiing", why no show us the unzip code?
I have an XML data which I would like to compress it using GZipStream and upload it to webservice. I would like to create the gzip file in memory instead of of creating it in local disk. I have tried the following:
public string class1(string url, string xml)
{
byte[] data = Encoding.ASCII.GetBytes(xml);
MemoryStream memory = new MemoryStream();
GZipStream gzip = new GZipStream(memory, CompressionMode.Compress, true);
gzip.Write(data, 0, data.Length);
byte[] zip=memory.ToArray();
HttpWebRequest wReq = (HttpWebRequest)WebRequest.Create(url);
wReq.Method = "POST";
wReq.ContentType = "application/zip";
var reqStream = wReq.GetRequestStream();
reqStream.Write(zip,0,zip.Length);
reqStream.Close();
var wRes = wReq.GetResponse();
var resStream = wRes.GetResponseStream();
var resgzip = new GZipStream(resStream, CompressionMode.Decompress);
var reader = new StreamReader(resgzip);
var textResponse = reader.ReadToEnd();
reader.Close();
resStream.Close();
wRes.Close();
return textResponse;
}
After writing data to webservice the server unzips the file and processess it. While the server decompresses the data an exception is thrown in server "Premature end of file". Please help me in this.
Add below method to convert Stream to MemoryStream
public static MemoryStream Read(Stream stream)
{
MemoryStream memStream = new MemoryStream();
byte[] readBuffer = new byte[4096];
int bytesRead;
while ((bytesRead = stream.Read(readBuffer, 0, readBuffer.Length)) > 0)
memStream.Write(readBuffer, 0, bytesRead);
return memStream;
}
then call as below.
var wRes = wReq.GetResponse();
var memstream = Read(wRes.GetResponseStream());
var resgzip = new GZipStream(memstream, CompressionMode.Decompress);
var reader = new StreamReader(resgzip);
var textResponse = reader.ReadToEnd();
I am writing a C# assembly to take advantage of the GZip functionality in newer versions of the .NET framework, and I am copying code I have used successfully elsewhere.
The way this works is that an application we are using feeds a stream to the assembly, the assembly reads it, compresses it, then returns the compressed result as a string. That string is then put back into the assembly through a stream to uncompress it (normally it will be stored, but I am simply running a basic test).
However, when I feed the string back to the assembly, it errors out while reading it from a byte array that is read from the application's stream. Here is the code:
private void ReadStream(IStream stream, out byte[] data)
{
using (MemoryStream writer = new MemoryStream())
{
IntPtr rwBytes = Marshal.AllocHGlobal(4);
int _rwBytes = 0;
data = new byte[0xafc8];
do
{
stream.Read(data, 0xafc8, rwBytes);
_rwBytes = Marshal.ReadInt32(rwBytes);
writer.Write(data, 0, _rwBytes);
}
while (_rwBytes > 0);
writer.Close();
Marshal.FreeHGlobal(rwBytes);
}
}
public string CompressString([In, MarshalAs(UnmanagedType.AsAny)] object appStream)
{
byte[] buffer = null;
ReadStream(appStream as IStream, out buffer);
MemoryStream ms = new MemoryStream();
using (GZipStream zip = new GZipStream(ms, CompressionMode.Compress, true))
{
zip.Write(buffer, 0, buffer.Length);
}
ms.Position = 0;
MemoryStream outStream = new MemoryStream();
byte[] compressed = new byte[ms.Length];
ms.Read(compressed, 0, compressed.Length);
byte[] gzBuffer = new byte[compressed.Length + 4];
System.Buffer.BlockCopy(compressed, 0, gzBuffer, 4, compressed.Length);
System.Buffer.BlockCopy(BitConverter.GetBytes(buffer.Length), 0, gzBuffer, 0, 4);
string str = Convert.ToBase64String(gzBuffer);
gzBuffer = Convert.FromBase64String(str);
return str;
}
public string DecompressString([In, MarshalAs(UnmanagedType.AsAny)] object appStream)
{
byte[] buffer = null;
ReadStream(appStream as IStream, out buffer);
string compressedText = Encoding.UTF8.GetString(buffer);
byte[] gzBuffer = Convert.FromBase64String(compressedText);
using (MemoryStream ms = new MemoryStream())
{
int msgLength = BitConverter.ToInt32(gzBuffer, 0);
ms.Write(gzBuffer, 4, gzBuffer.Length - 4);
buffer = new byte[msgLength];
ms.Position = 0;
using (GZipStream zip = new GZipStream(ms, CompressionMode.Decompress))
{
zip.Read(buffer, 0, buffer.Length);
}
}
return Encoding.UTF8.GetString(buffer);
}
The string is returned from the CompressString function as:
yK8AAB+LCAAAAAAABADtvQdgHEmWJSYvbcp7f0r1StfgdKEIgGATJNiQQBDswYjN5pLsHWlHIymrKoHKZVZlXWYWQMztnbz33nvvvffee++997o7nU4n99//P1xmZAFs9s5K2smeIYCqyB8/fnwfPyK+ndd5WjRplk6rxarOm6aolmmbN+04TZ/n7cdN2uTU4jwt2vSqqt8241/jR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Pzo+dHz/67nR9j86PnR86PnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89/99/fjSCHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj5//7z49G8KPnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Px///nRCH70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj57/7z8/GsGPnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fP/nef/AYk/UbjIrwAA
When it is read back from the stream in the Encoding.UTF8.GetString call, it is:
yK8AAB+LCAAAAAAABADtvQdgHEmWJSYvbcp7f0r1StfgdKEIgGATJNiQQBDswYjN5pLsHWlHIymrKoHKZVZlXWYWQMztnbz33nvvvffee++997o7nU4n99//P1xmZAFs9s5K2smeIYCqyB8/fnwfPyK+ndd5WjRplk6rxarOm6aolmmbN+04TZ/n7cdN2uTU4jwt2vSqqt8241/jR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Pzo+dHz/67nR9j86PnR86PnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89/99/fjSCHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj5//7z49G8KPnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Px///nRCH70/Oj50fOj50fPj54fPT96fvT86PnR86PnR8+Pnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fOj50fPj57/7z8/GsGPnh89P3p+9Pzo+dHzo+dHz4+eHz0/en70/Oj50fP/nef/AYk/UbjIrwAA
These strings are identical. I can even convert it back and forth in the CompressString function with no problem. Any thoughts out there? This is very strange.
Don't use UTF8, you should use the ascii text encoding instead. When you get the bytes from your string, you'll end up with some extra bytes due to the way UTF strings are encoded.
I know this question has been asked before on Stackoverflow, but could not find an explanation.
When I try to read a string from a compressed byte array I get an empty string on the first attempt, on the second I succed and get the string.
Code example:
public static string Decompress(byte[] gzBuffer)
{
if (gzBuffer == null)
return null;
using (var ms = new MemoryStream(gzBuffer))
{
using (var decompress = new GZipStream(ms, CompressionMode.Decompress))
{
using (var sr = new StreamReader(decompress, Encoding.UTF8))
{
string ret = sr.ReadToEnd();
// this is the extra check that is needed !?
if (ret == "")
ret = sr.ReadToEnd();
return ret;
}
}
}
}
All suggestions are appreciated.
- Victor Cassel
I found the bug. It was as Michael suggested in the compression routine. I missed to call Close() on the GZipStream.
public static byte[] Compress(string text)
{
if (string.IsNullOrEmpty(text))
return null;
byte[] raw = Encoding.UTF8.GetBytes(text);
using (var ms = new MemoryStream())
{
using (var compress = new GZipStream (ms, CompressionMode.Compress))
{
compress.Write(raw, 0, raw.Length);
compress.Close();
return ms.ToArray();
}
}
}
What happened was that the data seemed to get saved in a bad state that required two calls to ReadToEnd() in the decompression routine later on to extract the same data. Very odd!
try adding ms.Position = 0 before string ret = sr.ReadToEnd();
Where is gzBuffer coming from? Did you also write the code that is producing the compressed data?
Perhaps the buffer data you have is invalid or somehow incomplete, or perhaps it consists of multiple deflate streams concatenated together.
I hope this helps.
For ByteArray:
static byte[] CompressToByte(string data)
{
MemoryStream outstream = new MemoryStream();
GZipStream compressionStream =
new GZipStream(outstream, CompressionMode.Compress, true);
StreamWriter writer = new StreamWriter(compressionStream);
writer.Write(data);
writer.Close();
return StreamToByte(outstream);
}
static string Decompress(byte[] data)
{
MemoryStream instream = new MemoryStream(data);
GZipStream compressionStream =
new GZipStream(instream, CompressionMode.Decompress);
StreamReader reader = new StreamReader(compressionStream);
string outtext = reader.ReadToEnd();
reader.Close();
return outtext;
}
public static byte[] StreamToByte(Stream stream)
{
stream.Position = 0;
byte[] buffer = new byte[128];
using (MemoryStream ms = new MemoryStream())
{
while (true)
{
int read = stream.Read(buffer, 0, buffer.Length);
if (!(read > 0))
return ms.ToArray();
ms.Write(buffer, 0, read);
}
}
}
You can replace if(!(read > 0)) with if(read <= 0).
For some reason if(read <= 0) isn't displayed corret above.
For Stream:
static Stream CompressToStream(string data)
{
MemoryStream outstream = new MemoryStream();
GZipStream compressionStream =
new GZipStream(outstream, CompressionMode.Compress, true);
StreamWriter writer = new StreamWriter(compressionStream);
writer.Write(data);
writer.Close();
return outstream;
}
static string Decompress(Stream data)
{
data.Position = 0;
GZipStream compressionStream =
new GZipStream(data, CompressionMode.Decompress);
StreamReader reader = new StreamReader(compressionStream);
string outtext = reader.ReadToEnd();
reader.Close();
return outtext;
}
The MSDN Page on the function mentions the following:
If the current method throws an OutOfMemoryException, the reader's position in the underlying Stream object is advanced by the number of characters the method was able to read, but the characters already read into the internal ReadLine buffer are discarded. If you manipulate the position of the underlying stream after reading data into the buffer, the position of the underlying stream might not match the position of the internal buffer. To reset the internal buffer, call the DiscardBufferedData method; however, this method slows performance and should be called only when absolutely necessary.
Perhaps try calling DiscardBufferedData() before your ReadToEnd() and see what it does (I know you aren't getting the exception, but it's all I can think of...)?