Develop Reference

unexpected char when parsing json string

I am serializing an object to a stream to store as file and then retrieving and trying to deserialize the object, but get an error parsing. Below is code:
var content = JsonConvert.SerializeObject(data);
var output = new MemoryStream();
var writer = new StreamWriter(output, Encoding.UTF8);
writer.Write(content);
writer.Flush();
//write to some file...
//when reading the file
Stream filestream;
//filestream opens some file stream
byte[] buffer = new byte[4096]
using(MemoryStream ms = new MemoryStream()){
int read;
while((read = filestream.Read(buffer, 0, buffer.Length)) > 0){
ms.Write(buffer, 0, read);
}
var data = Encoding.UTF8.GetString(ms.ToArray());
//encounters error here. I can see that first few chars of the string are question marks.
JsonConvert.DeserializeObject<T>(data);

A couple ideas. Are you getting an exception?
Try encapsulating your write & reads into functions and wrap the memoryStream and streamWriter in using statements as well
using var writer = new StreamWriter(...
Additionally you don't need to set the length of the buffer to 4096, that's probably messing up your encoding. Read your memorystream to an array like so
var buffer = ms.ToArray();
How can I write MemoryStream to byte[]
It looks like you're having an issue copying the memory between the 2 memory streams though. Ideally you would write and read to a file or some other source rather than between 2 memory stream objects but you can copy the contents directly using a copyto
https://learn.microsoft.com/en-us/dotnet/api/system.io.stream.copyto?view=net-6.0
Edit: Adding pseudocode
Assuming your content has been read into
Stream filestream;
using var MemoryStream ms = new MemoryStream();
filestream.CopyTo(ms);
var data = Encoding.UTF8.GetString(ms.ToArray());
return JsonConvert.DeserializeObject<T>(data);

how to get position and length of unzipped gzipstream in c#?

i'm trying to read .gz files using binary reader by first unzipping with gzipstream, and then creating a new binary reader with the gzipstream. however, when i try to use the BaseStream.Position and BaseStream.Length of BinaryReader (to know when i'm at the end of my file), i get a NotSupportedException, checking the doc for these fields in GZipStream Class shows:
Length
This property is not supported and always throws a NotSupportedException.(Overrides Stream.Length.)
Position
This property is not supported and always throws a NotSupportedException.(Overrides Stream.Position.)
so my question is how can i know when i'm at the end of my file when reading a decompressed GZipStream using BinaryReader? thanks
here is my code:
Stream stream = new MemoryStream(textAsset.bytes);
GZipStream zippedStream = new GZipStream(stream, CompressionMode.Decompress);
using (BinaryReader reader = new BinaryReader(zippedStream))
while(reader.BaseStream.Position != reader.BaseStream.Length)
{
//do stuff with BinaryReader
}
the above throws:
NotSupportedException: Operation is not supported. System.IO.Compression.DeflateStream.get_Position()
due to the BaseStream.Position call in the while()

You can copy your zippedStream to MemoryStream instance, that can be read fully using ToArray function. That is the easiest solution I can think of.
Stream stream = new MemoryStream(textAsset.bytes);
byte[] result;
using (GZipStream zippedStream = new GZipStream(stream, CompressionMode.Decompress))
{
using (MemoryStream reader = new MemoryStream())
{
zippedStream.CopyTo(reader);
result = reader.ToArray();
}
}
Alternatively if you want to read stream in chunks
using (GZipStream zippedStream = new GZipStream(stream, CompressionMode.Decompress))
{
byte[] buffer = new byte[16 * 1024];
int read;
while ((read = zippedStream.Read(buffer, 0, buffer.Length)) > 0)
{
// do work
}
}

Depending on what you are decoding you can read the first type into a byte array using the BinaryReader and then use BitConverter to convert these bytes into the type you want. You can then use BinaryReader as normal until the start of the next record.
byte[] headermarker = new byte[4];
int count;
// if bytes available in underlying stream.
while ((count = br.Read(headermarker, 0, 4) > 0 )
{
Int32 marker = BitConverter.ToInt32(headermarker, 0);
//
// now use Binary Reader for the rest of the record until we loop
//
}

Compressing C# objects using SharpZipLib?

Is it possible to compress a List<T> using SharpZipLib?
Turning the List to a byte array gives me around 60000 bytes (uncompressed).
Compression this with System.IO.Compression.DeflateStream gives me around a 1/3 compression rate but this is far from enough.
The purpose is to store the collections in the (MS SQL) database as a byte[] because saving them as individual rows uses to much space (1 million rows/day).
Thanks
Edit:
List<ItemLog> itemLogs = new List<ItemLog>();
//populate with 1000 ItemLogs
byte[] array = null; //original byte array
BinaryFormatter bf = new BinaryFormatter();
MemoryStream ms = new MemoryStream();
bf.Serialize(ms, itemLogs);
array = ms.ToArray();
the array size is now 60000 bytes
Zip the collection using a ZipOutputStream
MemoryStream outputMemoryStream = new MemoryStream();
ZipOutputStream zipStream = new ZipOutputStream(outputMemoryStream);
zipStream.SetLevel(3);
ZipEntry entry = new ZipEntry("logs");
entry.DateTime = DateTime.Now;
zipStream.PutNextEntry(entry);
StreamUtils.Copy(ms, zipStream, new byte[4096]);
zipStream.CloseEntry();
zipStream.IsStreamOwner = false;
zipStream.Close();
outputMemoryStream.Position = 0;
byte[] compressed = outputMemoryStream.ToArray();
The compressed is now 164 bytes in size. <- length not valid/possible?
Uncompressing gives me a empty array. But as the compression is not right I will skip the uncompression code for now.

I do not see any real problem in your code. The only part, where the problem can be is copying of data. Is the stream of input data at the start of data, that should be stored? Try add the following line:
ms.Seek(0, SeekOrigin.Begin); // added line
StreamUtils.Copy(ms, zipStream, new byte[4096]);
Based on your code I wrote a simple compress function and it works as expected.
private static byte[] Compress(byte[] source)
{
byte[] compressed;
using (var memory = new MemoryStream())
using (var zipped = new ZipOutputStream(memory))
{
zipped.IsStreamOwner = false;
zipped.SetLevel(9);
var entry = new ZipEntry("data")
{
DateTime = DateTime.Now
};
zipped.PutNextEntry(entry);
#if true
zipped.Write(source, 0, source.Length);
#else
using (var src = new MemoryStream(source))
{
StreamUtils.Copy(src, zipped, new byte[4096]);
}
#endif
zipped.Close();
compressed = memory.ToArray();
}
#if false
using (var file = new FileStream("test.zip", FileMode.Create, FileAccess.Write, FileShare.Read))
{
file.Write(compressed, 0, compressed.Length);
}
#endif
return compressed;
}
You have there alternatives how to save the output (array or stream) and there is disabled code to save compressed data to file (to check in a external application the real content of compressed data).
My tested data were a 256 bytes long (data, with small compression rate), and the result was 407 bytes (file).
Try to use array, or check the stream content that is saved.

GZipStream not reading the whole file

I have some code that downloads gzipped files, and decompresses them. The problem is, I can't get it to decompress the whole file, it only reads the first 4096 bytes and then about 500 more.
Byte[] buffer = new Byte[4096];
int count = 0;
FileStream fileInput = new FileStream("input.gzip", FileMode.Open, FileAccess.Read, FileShare.Read);
FileStream fileOutput = new FileStream("output.dat", FileMode.Create, FileAccess.Write, FileShare.None);
GZipStream gzipStream = new GZipStream(fileInput, CompressionMode.Decompress, true);
// Read from gzip steam
while ((count = gzipStream.Read(buffer, 0, buffer.Length)) > 0)
{
// Write to output file
fileOutput.Write(buffer, 0, count);
}
// Close the streams
...
I've checked the downloaded file; it's 13MB when compressed, and contains one XML file. I've manually decompressed the XML file, and the content is all there. But when I do it with this code, it only outputs the very beginning of the XML file.
Anyone have any ideas why this might be happening?

EDIT
Try not leaving the GZipStream open:
GZipStream gzipStream = new GZipStream(fileInput, CompressionMode.Decompress,
false);
or
GZipStream gzipStream = new GZipStream(fileInput, CompressionMode.Decompress);

I ended up using a gzip executable to do the decompression instead of a GZipStream. It can't handle the file for some reason, but the executable can.

Same thing happened to me. In my case only reads up to 6 lines and then reached end of file. So I realized that although the extension is gz, it was compressed by another algorithm not supported by GZipStream. So I used SevenZipSharp library and it worked. This is my code
You can use SevenZipSharp library
using (var input = File.OpenRead(lstFiles[0]))
{
using (var ds = new SevenZipExtractor(input))
{
//ds.ExtractionFinished += DsOnExtractionFinished;
var mem = new MemoryStream();
ds.ExtractFile(0, mem);
using (var sr = new StreamReader(mem))
{
var iCount = 0;
String line;
mem.Position = 0;
while ((line = sr.ReadLine()) != null && iCount < 100)
{
iCount++;
LstOutput.Items.Add(line);
}
}
}
}

Are you calling Close or Flush on fileOutput? (Or just wrap it in a using, which is recommended practice.) If you don't the file might not be flushed to disk when your program ends.

How can I read an Http response stream twice in C#?

I am trying to read an Http response stream twice via the following:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
stream = response.GetResponseStream();
RssReader reader = new RssReader(stream);
do
{
element = reader.Read();
if (element is RssChannel)
{
feed.Channels.Add((RssChannel)element);
}
} while (element != null);
StreamReader sr = new StreamReader(stream);
feed._FeedRawData = sr.ReadToEnd();
However when the StreamReader code executes there is no data returned because the stream has now reached the end. I tried to reset the stream via stream.Position = 0 but this throws an exception (I think because the stream can't have its position changed manually).
Basically, I would like to parse the stream for XML and have access to the raw data (in string format).
Any ideas?

Copy it into a new MemoryStream first. Then you can re-read the MemoryStream as many times as you like:
Stream responseStream = CopyAndClose(resp.GetResponseStream());
// Do something with the stream
responseStream.Position = 0;
// Do something with the stream again
private static Stream CopyAndClose(Stream inputStream)
{
const int readSize = 256;
byte[] buffer = new byte[readSize];
MemoryStream ms = new MemoryStream();
int count = inputStream.Read(buffer, 0, readSize);
while (count > 0)
{
ms.Write(buffer, 0, count);
count = inputStream.Read(buffer, 0, readSize);
}
ms.Position = 0;
inputStream.Close();
return ms;
}

Copying the stream to a MemoryStream as suggested by Iain is the right approach. But since
.NET Framework 4 (released 2010) we have Stream.CopyTo. Example from the docs:
// Create the streams.
MemoryStream destination = new MemoryStream();
using (FileStream source = File.Open(#"c:\temp\data.dat",
FileMode.Open))
{
Console.WriteLine("Source length: {0}", source.Length.ToString());
// Copy source to destination.
source.CopyTo(destination);
}
Console.WriteLine("Destination length: {0}", destination.Length.ToString());
Afterwards you can read destination as many times as you like:
// re-set to beginning and convert stream to string
destination.Position = 0;
StreamReader streamReader = new StreamReader(destination);
string text = streamReader.ReadToEnd();
// re-set to beginning and read again
destination.Position = 0;
RssReader cssReader = new RssReader(destination);
(I have seen Endy's comment but since it is an appropriate, current answer, it should have its own answer entry.)

have you tried resetting the stream position?
if this does not work you can copy the stream to a MemoryStream and there you can reset the position (i.e. to 0) as often as you want.