I'm using Json.Net to consume some seekable streams.
// reset the input stream, in case it was previously read
inputStream.Position = 0;
using (var textReader = new StreamReader(inputStream))
{
using (var reader = new JsonTextReader(textReader))
{
deserialized = serializer.Deserialize(reader, expectedType);
}
}
However, this method 'consumes' the stream, meaning the first contained valid Json token is removed from the stream.
That it very annoying. And meaningless, stream Position is provided to emulate a consumption, and 'reading' generally implies 'not modifying'.
Of course, I can dump the stream into a MemoryStream to protect my precious source stream, but that's a huge overhead, especially when doing trial-and-error on a deserialization.
If there is a way to to just 'read' and not 'read-and-consume', thanks for your help, I could not find documentation about that (and I hope this post will help others to google the solution ^^).
JsonTextReader is a forward-only reader, meaning it cannot be set back to a position earlier in the JSON to re-read a portion of it, even if the underlying stream supports seeking. However, the reader does not actually "consume" the stream, as you said. If you set the CloseInput property on the reader to false to prevent it from closing the underlying reader and stream when it is disposed, you can position the stream back to the beginning and open a new reader on the same stream to re-read the JSON. Here is a short program to demonstrate reading the same stream twice:
class Program
{
static void Main(string[] args)
{
string json = #"{ ""name"": ""foo"", ""size"": ""10"" }";
MemoryStream inputStream = new MemoryStream(Encoding.UTF8.GetBytes(json));
JsonSerializer serializer = new JsonSerializer();
using (var textReader = new StreamReader(inputStream))
{
for (int i = 0; i < 2; i++)
{
inputStream.Position = 0;
using (var reader = new JsonTextReader(textReader))
{
reader.CloseInput = false;
Widget w = serializer.Deserialize<Widget>(reader);
Console.WriteLine("Name: " + w.Name);
Console.WriteLine("Size: " + w.Size);
Console.WriteLine();
}
}
}
}
}
class Widget
{
public string Name { get; set; }
public int Size { get; set; }
}
Output:
Name: foo
Size: 10
Name: foo
Size: 10
Fiddle: https://dotnetfiddle.net/fftZV7
A stream may be consumed once read. The solution could be to copy it to a memory or file stream as below:
MemoryStream ms = new MemoryStream();
inputStream.CopyTo(ms);
ms.Position = 0;
using (var textReader = new StreamReader(ms))
(...)
Please let me know if it works.
Related
I have a memorystream reading a specific part of my data. From the binary I want one ReadInt32 value from position 5-8. How do I achieve this in:
using (var reader = new BinaryReader(stream))
{
somebyte1
somebyte2
somebyte3
//get only this value
int v = reader.ReadInt32;
}
Move the base stream to the position you want to read from:
stream.Seek(4, SeekOrigin.Begin);
using (var reader = new BinaryReader(stream))
{
int v = reader.ReadInt32;
}
In .NET there are stream types that are seekable and types that don't allow seek. This is indicated by the CanSeek property. If your stream allows seek (and a MemoryStream does), you can just move the current position and read the data. If the stream does not allow seek, your only choice is to read and discard the data until you reach the stream position where your desired data is at. So the generalized solution to your problem would be:
const int targetPosition = 4;
BinaryReader reader = new BinaryReader(stream);
using (reader) {
if (stream.CanSeek) {
stream.Position = targetPosition;
}
else {
reader.ReadBytes(targetPosition);
}
int result = reader.ReadInt32();
}
I have created a Web API service that accepts a POSTed JSON file. With the service, I want to parse the JSON file using JSON.NET. I have seen multiple posts on the subject, however, I don't want to save the file to disk. I want to keep the file in memory, parse the file and dispose of the file in memory.
I'm also using .NET Framework 4.0.
EDIT: I should be more clear. When the file is POSTed, it is a file stream. The part I don't know is how to convert the stream to JSON.
public HttpResponseMessage Post()
{
HttpResponseMessage result = null;
int FileLen;
var httpRequest = HttpContext.Current.Request;
if (httpRequest.Files.Count > 0)
{
string MyString = string.Empty;
var postedFile = httpRequest.Files[0];
FileLen = postedFile.ContentLength;
byte[] input = new byte[FileLen];
System.IO.Stream testStream = postedFile.InputStream;
testStream.Read(input, 0, FileLen);
for (int Loop1 = 0; Loop1 < FileLen; Loop1++)
MyString = MyString + input[Loop1].ToString();
CurrentRate.JSONSerializer(MyString);
}
Since JSON.NET has the ability to deserialize a file (converted to a stream) with the following code:
using (StreamReader file = File.OpenText(#"<pathToFile>"))
{
JsonSerializer serializer = new JsonSerializer();
MyObject myObject = (MyObject)serializer.Deserialize(file, typeof(MyObject));
}
By swapping out the stream from File.OpenText with something similar to the following
using (var stream = new MemoryStream(<FileByteArrayFromMemory>);
using (StreamReader file = new StreamReader(stream))
{
JsonSerializer serializer = new JsonSerializer();
MyObject myObject = (MyObject)serializer.Deserialize(file, typeof(MyObject));
}
we never have to deal with an actual file. There may be a more efficient way to create the MemoryStream or you may already have aStream and could initialize the and could skip that step.
EDIT 2: more explicitly showed swapping the StreamReader
I have a class that serializes and deserializes objects for a given type. I am trying to serialize a Custom file object containing the file name, the data in the file and a few other details like created time, modified time etc associated with the file. Additionally the custome class includes some properties or flags I would need at my receiver end (where I deserialize).
When I try to serialize around 30K of such file objects, it does the serialization successfully for a large majority, but throws back an outofmemoryexception for some files.
Below is my serialization class code:
public static string SerializeToByteArray(Type T, object request)
{
DataContractSerializer serializer = new DataContractSerializer(T);
using (MemoryStream memStream = new MemoryStream())
{
using (StreamReader reader = new StreamReader(memStream))
{
serializer.WriteObject(memStream, request);
memStream.Position = 0;
return reader.ReadToEnd();
}
}
}
public static T DeserializeFromByteArray<T>(string xml)
{
DataContractSerializer deserializer = new DataContractSerializer(typeof(T));
using (MemoryStream memStream = new MemoryStream())
{
byte[] data = System.Text.Encoding.UTF8.GetBytes(xml);
memStream.Write(data, 0, data.Length);
memStream.Position = 0;
var newobj = (T)deserializer.ReadObject(memStream);
return newobj;
}
}
My question should be relatively straight forward:
Is it (In any way) possible to create multiple XmlReader objects for the same stream in sequence, without the first reader advancing the stream to the end once it's disposed?
Sample code (Note that the second call to ReadElement will fail because the first reader advanced the stream to the end, for whatever reason):
private static void DoTest()
{
using (var stream = new MemoryStream())
{
WriteElement("Test", stream);
Console.WriteLine("Stream Length after first write: {0}", stream.Length);
WriteElement("Test2", stream);
Console.WriteLine("Stream Length after second write: {0}", stream.Length);
stream.Position = 0;
Console.WriteLine(ReadElement(stream));
Console.WriteLine("Position is now: {0}/{1}", stream.Position, stream.Length);
Console.WriteLine(ReadElement(stream)); // Note that this will fail due to the stream position now being at the end.
}
}
private static string ReadElement(Stream source)
{
string result;
using (var reader = XmlReader.Create(source, new XmlReaderSettings
{
ConformanceLevel = ConformanceLevel.Fragment,
CloseInput = false
}))
{
reader.Read();
result = reader.Name;
reader.Read();
}
return result;
}
private static void WriteElement(string name, Stream target)
{
using (var writer = XmlWriter.Create(target, new XmlWriterSettings
{
ConformanceLevel = ConformanceLevel.Fragment,
WriteEndDocumentOnClose = false,
OmitXmlDeclaration = true,
}))
{
writer.WriteStartElement(name);
writer.WriteEndElement();
}
}
If this is not possible with 'pure .Net', are there any alternative ('Light') Xml parser libraries out there that would support this behaviour?
1. Messy way
If you're able to save the length of each sequence you could do as following:
int len, i = 0; //len is the length of the interval
byte[] buffer = new byte[0xff];
while (len --> 0)
buffer[i++] = stream.ReadByte (); //copies the interval
This separately copies an interval of bytes from the MemoryStream and saves it into a buffer. Then you'd simply jongle with the buffer by assigning it to a new MemoryStream or to a String (see XMLCreate overloads).
The problem is that the first read operation is too GREEDY and eats the whole interval.
2. Original way
Write your one stream to suite your needs!
I am trying to use the WCF DataContractSerializer to serialize a DataContract object into a memoryStream.
Then I use the memoryStream.ToArray to get the serialized content.
Finally, I persist the memoryStream into a file using anther fileStream.
My initial implement is like this. I am missing bytes at the end of the persisted File.
public virtual string SerializeTransient(DataObject data, string targetPath)
{
string securityCode;
using (var memoryStream = new MemoryStream())
{
using (var xmlWriter = XmlWriter.Create(memoryStream, new XmlWriterSettings {Indent = true}))
{
_serializer.WriteObject(xmlWriter, data);
using (var fileStream = new FileStream(targetPath, FileMode.Create))
{
securityCode = CalculateSecurityCode(memoryStream.ToArray());
memoryStream.WriteTo(fileStream);
}
}
}
return securityCode;
}
If I move the persist logic out of the inner using{} block (see below), the output is correct. It almost feels like the WriteObject function didnt finish what it is doing. Could someone please explain to me what is happening there? Thanks.
public virtual string SerializeTransient(DataObject data, string targetPath)
{
string securityCode;
using (var memoryStream = new MemoryStream())
{
using (var xmlWriter = XmlWriter.Create(memoryStream, new XmlWriterSettings {Indent = true}))
{
_serializer.WriteObject(xmlWriter, data);
}
using (var fileStream = new FileStream(targetPath, FileMode.Create))
{
securityCode = CalculateSecurityCode(memoryStream.ToArray());
memoryStream.WriteTo(fileStream);
}
}
return securityCode;
}
XmlWriter has an internal buffer. You should either Close/Dispose XmlWriter or call the XmlWriter.Flush() to force all content to be written to underlying stream (memoryStream).
If memoryStream.ToArray() is called before writer.Flush() then some bytes will possibly remain in internal writer buffer.