I am writing an application which will make a REST HTTP call and receive JSON back in the response body. The response body could be large and the content variable so I am wanting to handle it as a stream and process the content to decide which parts I want to deserialize.
I am able to perform my parsing and deserialize the objects I want using a StreamReader (passing in the HttpResponseMessage.Content, read as a stream) and a JsonTextReader. However, as the JsonTextReader pulls data from the stream, I would also like to trace the raw JSON to a file so that we have the raw response recorded (for diagnostics).
For example, my code:
var serializer = new JsonSerializer()
using (var streamReader = new StreamReader(httpContentStream))
using (var textReader = new JsonTextReader(streamReader))
{
textReader.SupportMultipleContent = true;
while (textReader.Read())
{
// Code which looks at the tokens in the text reader and
// figures out what to throw away, what to deserialize and
// what the next type to deserialize will be (stored in someType)
Type someType = null;
// Deserialize an object that we're interested in.
// This advances the textReader to the next token
// after then end of this object (I.E. More than one token)
var someObject = serializer .Deserialize(textReader, someType);
}
}
Is there a way to also have Json.Net trace out the string that its reading as it pulls characters off the stream? I understand that this could generate a large file!
Thanks in advance for any responses.
Related
I am getting a Gziped JSON which I check for a specific start string and if it's correct I deserialize into my object:
await using var decompressionStream = new GZipStream(externalStream, CompressionMode.Decompress);
var streamReader = new StreamReader(decompressionStream);
string eventString = streamReader.ReadToEnd();
if (!eventString.StartsWith("SomeString"))
{
// Drop it because it can't be deserialized
return;
}
MyObject item = Utf8Json.JsonSerializer.Deserialize<MyObject>(Encoding.UTF8.GetBytes(eventString));
This approach works but it seems to be wasteful to turn the stream into a string and back into an bytes array when Utf8Json can deserialize directly from a stream.
Because 3/4 of incoming streams contain data that is of a different type and can't be deserialized I don't want to put a try/catch block around it because that many exceptions would be too expensive.
Is there a way to peek into the first ~16 chars without consuming them?
This is related to my question HTTPClient Buffer Exceeded 2G; Cannot write more bytes to the buffer but is different enough that IMO it warrants a separate question.
In the other question, I'm trying to figure out how to deal with breaking the 2G request buffer. The idea was to use streaming, but I need to deserialize. In talking to Professor Google, I found that I have to use TextReader to stream/deserialize. so my code for that is:
public async Task<API_Json_Special_Feeds.RootObject> walMart_Special_Feed_Lookup(string url)
{
special_Feed_Lookup_Working = true;
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
using (HttpClient http = new HttpClient(handler))
{
http.DefaultRequestHeaders.AcceptEncoding.Add(new System.Net.Http.Headers.StringWithQualityHeaderValue("gzip"));
http.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
url = String.Format(url);
using (var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
{
Console.WriteLine(response);
var serializer = new JsonSerializer();
using (StreamReader sr = new StreamReader(await response.Content.ReadAsStreamAsync()))
{
using (var jsonTextReader = new JsonTextReader(sr))
{
API_Json_Special_Feeds.RootObject root = (API_Json_Special_Feeds.RootObject)serializer.Deserialize(jsonTextReader);
return root;
}
}
}
}
}
Now, as you can see, the return type is strongly typed. The return type of the method matches. Now, I go to the calling line:
API_Json_Special_Feeds.RootObject Items = await net.walMart_Special_Feed_Lookup(specialFeedsURLs[i].Replace("{apiKey}", Properties.Resources.API_Key_Walmart));
So, we have matching types API_Json_Special_Feeds.RootMethod all the way around.
When run, the calling line throws an InvalidCastException:
Undesired Result:
Unable to cast object of type 'Newtonsoft.Json.Linq.JObject' to type 'RootObject'
I've checked at the end of the method before return, and the result is indeed cast from an object to API_Json_Special_Feeds.RootMethod before being returned.
Question: somewhere between the return statement and the calling line, the object being returned is being converted from an API_Json_Special_Feeds.RootMethod to a Newtonsoft.Json.Linq.JObject. I can't debug it since there's no code in between. If I cast again in the calling line, I get a "Cannot cast" error. How can I prevent the degradation/changing of this object type?
Many, many thinks for your time, consideration, and for any thoughts or suggestions you can provide!
You need to use the generic overload JsonSerializer.Deserialize<T>()
var root = serializer.Deserialize<API_Json_Special_Feeds.RootObject>(jsonTextReader);
Unlike files generated by BinaryFormatter, JSON files generally do not include c# type information, so it's necessary for the receiving system to specify the expected type.
(There are extensions to the JSON standard in which c# type information can be included in a JSON file -- e.g. Json.NET's TypeNameHandling -- but it is still necessary to deserialize the JSON to an appropriate explicit base class.)
See Deserialize JSON from a file for another example of deserializing a strongly typed c# object from a stream.
I am attempting to read a XML document containing elements like the data mentioned below.
Accessing the text node via reader.Value, reader.ReadContentAsString(), reader.ReadContentAsObject() results in the value read being truncated to the last ampersand, so in the case of the data below that would be ISO^urn:ihe:iti:xds:2013:referral. Using XmlDocument the text nodes can be read properly so I am assuming there has to be a way to make this work using the reader as well.
<Slot name="urn:ihe:iti:xds:2013:referenceIdList">
<ValueList>
<Value>123456^^^&orgID&ISO^urn:ihe:iti:xds:2013:referral</Value>
<Value>098765^^^&orgID&ISO^urn:ihe:iti:xds:2013:referral</Value>
</ValueList>
</Slot>
Clarification Edit
After asking the question I was able to determine my issue came from creating an XmlReader from a XPathNavigator instance created from a MessageBuffer executing in the context of a WCF service call. Thus #DarkGray's answer was correct for the original question but did not really address the root of the problem. I provided a second answer which addressed my corner case.
System.ServiceModel.Channels.Message message; // the inbound SOAP message
var buffer = message.CreateBufferedCopy(11 * 1024 * 1024);
var navigator = buffer.CreateNavigator();
var reader = navigator.ReadSubtree();
// advance the reader to the text element
//
// `reader.Value` now produces ISO^urn:ihe:iti:xds:2013:referral
Answer: reader.Value
Output:
123456^^^&orgID&ISO^urn:ihe:iti:xds:2013:referral
098765^^^&orgID&ISO^urn:ihe:iti:xds:2013:referral
Example:
public static void Execute()
{
var xml = #"
<Slot name='urn:ihe:iti:xds:2013:referenceIdList'>
<ValueList>
<Value>123456^^^&orgID&ISO^urn:ihe:iti:xds:2013:referral</Value>
<Value>098765^^^&orgID&ISO^urn:ihe:iti:xds:2013:referral</Value>
</ValueList>
</Slot>
";
var reader = System.Xml.XmlReader.Create(new System.IO.StringReader(xml));
for (; ; )
{
if (!reader.Read())
break;
if (reader.NodeType == System.Xml.XmlNodeType.Text)
Console.WriteLine(reader.Value);
}
}
My question ended up being too broad as the incorrect behavior (truncation when using reader.Value) only manifest when the code was executing within the context of a WCF call. It worked perfectly fine when exercising the logic of the containing class from a unit test.
So the salient setup can be reproduced as follows
The Failing Code
System.ServiceModel.Channels.Message message; // the inbound SOAP message
var buffer = message.CreateBufferedCopy(11 * 1024 * 1024);
var navigator = buffer.CreateNavigator();
var reader = navigator.ReadSubtree();
// advance the reader to the text element
//
// `reader.Value` now produces ISO^urn:ihe:iti:xds:2013:referral
Once this reader instance was created then any XmlText node read from it produced the truncated value when the text contained an character entity reference. The only way I found that allow for the original value to be read in high-fidelity was to eschew the use of the XPathNavigator completely and instead take the hit of creating another Message instance. Note, the fix uses the long way around to write the SOAP envelope to the stream as affected service is using MTOM encoding. Writing to the stream directly from the MessageBuffer resulted in the MIME fences being written out.
The Fix
System.ServiceModel.Channels.Message message; // the inbound SOAP
var buffer = message.CreateBufferedCopy(MaxMessageSize);
var message = buffer.CreateMessage();
using (MemoryStream stream = new MemoryStream())
using (XmlWriter writer = XmlWriter.Create(stream))
{
message.WriteMessage(writer);
writer.Flush();
stream.Position = 0;
using (XmlReader reader = XmlReader.Create(stream))
{
// business logic goes here
}
}
I'm accessing the tomtom json api, and the api either returns me an array of objects, or a single object, when an error has happen.
Example:
[{"driverno": "...
Error Example:
{"errorCode": "8011","errorMsg": "request quota reached, error code: 8011"}
The data is accessed WebRequest, WebResponse and they return a stream, which can then be passed to a DataContractJsonSerializer. However, I can't create a serialization class, which accepts both forms of JSON, and the stream can't be passed twice, because the seek function is not supported.
Is there a way, to create a serialization class which supports both types of JSON input?
I found a workaround, where I copy the Stream to a MemoryStream, which enables seeking. I'm not completly settisfied with th solution, becuase it does a Stream copying and the DataContractJsonSerializer twice.
Sample:
string text = File.ReadAllText(PAHT);
text = Regex.Replace(text, "\\{[\\n\\r ]*\"__type", "{\"__type");
// copy to MemoryStream
using (MemoryStream dataStream = new MemoryStream(Encoding.UTF8.GetBytes(text)))
{
DataContractJsonSerializer errorDeserializer = new DataContractJsonSerializer(typeof(RequestError));
RequestError errorSerilaized = (RequestError)errorDeserializer.ReadObject(dataStream);
// check if an error happened
if (errorSerilaized.errorCode == null)
{
// seek the stream to position 0
dataStream.Position = 0;
DataContractJsonSerializer _deserializer = new DataContractJsonSerializer(typeof(NoneErrorSerializationClass));
NoneErrorSerializationClass tripReportsSerialized = (NoneErrorSerializationClass)_deserializer.ReadObject(dataStream);
// ...
}
else
{
MessageBox.Show(errorSerilaized.errorMsg);
}
}
I can't get the DataContractJsonSerializer object to swallow my stream. When I execute the code with the commented-out line active, I get to see the text provided (and it is a parsable JSON object), so I know that the stream is working fine.
However, for some reason, the compiler complains that the streamReader I'm trying to shove down its throat in ReadObject isn't a Stream. Well, isn't it?!
Argument 1: cannot convert from 'System.IO.StreamReader' to 'System.IO.Stream'
What am I missing and how do I resolve it?
using (StreamReader streamReader = new StreamReader(...))
{
//String responseText = reader.ReadToEnd();
MyThingy thingy = new MyThingy();
DataContractJsonSerializer serializer
= new DataContractJsonSerializer(thingy.GetType());
thingy = serializer.ReadObject(streamReader);
}
I'm adapting this example to work with my stream. Should I approach it from a different angle? If so - how?
You're trying to put in a reader of a stream instead of an actual stream. Skip the using and whatever hides behind the ellipsis (i.e. whatever you put in as an argument when you create an instance of StreamReader), you can probably put that into the ReadObject.
Also, you'll get into problems when reading the data because ReadObject will return an instance of type Object and you'll need to convert it into MyThingy. Since it's a nullable (I'm assuming), you don't have to type cast but rather as-ify it.
MyThingy thingy = new MyThingy();
DataContractJsonSerializer serializer
= new DataContractJsonSerializer(thingy.GetType());
Stream stream = ...;
thingy = serializer.ReadObject(stream) as MyThingy;
You could of course skip the next-to-last line and put the stream directly into the last line.
Courtesy of #JohanLarsson (all Swedes are great, especially those from Stockholm, like me):
In case you can't or don't want to omit the StreamReader declaration in your using statement, I'd suggest that you take a look at BaseStream property to get to it.
You can try this:
using (StreamReader streamReader = new StreamReader(...))
{
DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(MyThingy));
MyThingy thingy = (MyThingy) serializer.ReadObject(streamReader.BaseStream);
}
I've been always using this:
// get stuff here
String json = GetJSON();
List<T> result;
using (var ms = new MemoryStream(Encoding.Unicode.GetBytes(json)))
{
var serializer = new DataContractJsonSerializer(typeof(List<T>));
result = (List<T>)serializer.ReadObject(ms);
}