Abstract
Hi, I'm working on a project where it is needed to send potentially huge json of some object via HttpClient, a 10-20 mb of JSON is a typical size. In order do that efficiently I want to use streams, both with Json.Net to serialize an object plus streams for posting data with HttpClient.
Problem
Here is the snippet for serialization with Json.net, in order to work with streams, Json.net expects a stream that it will write into:
public static void Serialize( object value, Stream writeOnlyStream )
{
StreamWriter writer = new StreamWriter(writeOnlyStream); <-- Here Json.net expects the stream to be already created
JsonTextWriter jsonWriter = new JsonTextWriter(writer);
JsonSerializer ser = new JsonSerializer();
ser.Serialize(jsonWriter, value );
jsonWriter.Flush();
}
While HttpClient expects a stream that it will read from:
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("http://localhost:54359/");
var response = await client.PostAsync("/api/snapshot", new StreamContent(readOnlyStream)); <-- The same thing here, HttpClient expects the stream already to exist
...
}
So eventually this means that both classes expecting the Stream to be created by someone else, but there are no streams both for Json.Net, neither for HttpClient. So the problem seems that can be solved by implementing a stream that would intercept a read requests made to read-only stream, and issue writes upon request from write-only stream.
Question
Maybe someone has stumbled on such situation already, and probably found already implemented solution to this problem. If so, please share it with me,
Thank you in advance!
If you define a subclass of HttpContent :
public class JsonContent:HttpContent
{
public object SerializationTarget{get;private set;}
public JsonContent(object serializationTarget)
{
SerializationTarget=serializationTarget;
this.Headers.ContentType = new MediaTypeHeaderValue("application/json");
}
protected override async Task SerializeToStreamAsync(Stream stream,
TransportContext context)
{
using(StreamWriter writer = new StreamWriter(stream))
using(JsonTextWriter jsonWriter = new JsonTextWriter(writer))
{
JsonSerializer ser = new JsonSerializer();
ser.Serialize(jsonWriter, SerializationTarget );
}
}
protected override bool TryComputeLength(out long length)
{
//we don't know. can't be computed up-front
length = -1;
return false;
}
}
then you can:
var someObj = new {a = 1, b = 2};
var client = new HttpClient();
var content = new JsonContent(someObj);
var responseMsg = await client.PostAsync("http://someurl",content);
and the serializer will write directly to the request stream.
Use PushStreamContent. Rather than have Web API "pull" from a stream, it lets you more intuitively "push" into one.
object value = ...;
PushStreamContent content = new PushStreamContent((stream, httpContent, transportContext) =>
{
using (var tw = new StreamWriter(stream))
{
JsonSerializer ser = new JsonSerializer();
ser.Serialize(tw, value);
}
});
Note that JSON.NET doesn't support async during serialization so while this may be more memory efficient, it won't be very scalable.
I'd recommend trying to avoid such large JSON objects, though. Try to chunk it up, for instance, if you're sending over a large collection. Many clients/servers will flat out reject something so big without special handling.
Related
I send json file to server and want to read that twice.
[HttpPost]
public ActionResult CreateCases(string fileFormat, Guid key)
{
var file = Request.Files[0];
CheckFile(file);
Create(file);
return Json();
}
public object Check(HttpPostedFileBase file)
{
var stream = file.InputStream;
var serializer = new JsonSerializer();
using (var sr = new StreamReader(stream))
using (var jsonTextReader = new JsonTextReader(sr))
{
dynamic json = serializer.Deserialize(jsonTextReader);
...
}
}
public object Create(HttpPostedFileBase file)
{
var stream = file.InputStream;
var serializer = new JsonSerializer();
using (var sr = new StreamReader(stream))
using (var jsonTextReader = new JsonTextReader(sr))
{
dynamic json = serializer.Deserialize(jsonTextReader);
...
}
}
In Check method file.ContentLength = right value
In Create method file.ContentLength = 0 and json variable already = null
What am I doing wrong?
Thanks in advance.
What am I doing wrong?
This:
I [...] want to read that [file] twice
Your client only sends the file to your web application once, so you should only read it once.
Sure, you can rewind the input stream and appear to solve the immediate problem, but that just introduces new problems, because now you have the entire file in memory at once - and your code can only continue once the entire request has been read.
You don't want to read the file twice.
If you want to validate, then process the JSON, then obtain the JSON, store it in a variable, and then validate and process that variable. Yes, this still requires you to read the entire request body, but then that's your requirement.
I have a problem deserializing a JSON file of about 1GB. When I run the following code I get an out of memory exception:
using (FileStream sr = new FileStream("myFile.json", FileMode.Open, FileAccess.Read))
{
using (StreamReader reader = new StreamReader(sr))
{
using (JsonReader jsReader = new JsonTextReader(reader))
{
JsonSerializer serializer = new JsonSerializer();
dataObject = serializer.Deserialize<T>(jsReader);
}
}
}
the exception is thrown by
Newtonsoft.Json.Linq.JTokenWriter.WriteValue(Int64 value)
The serialization works well, here is the code I'm using
using (StreamWriter reader = new StreamWriter("myFile.json"))
{
using (JsonReader jsWriter = new JsonWriter(reader))
{
JsonTextWriter jsonWriter = new JsonTextWriter(jsWriter) { Formatting = Formatting.Indented };
JsonSerializer ser = new JsonSerializer();
ser.Serialize(jsonWriter, dataObject, dataObject.GetType());
jsonWriter.Flush();
}
}}
Am I doing something wrong in the deserialization? Can you help suggesting a way to deserialize big json object?
Thanks
According to Newtonsoft.Json Performance Tips your approach has to work (because you read via stream and it should make portion from your file). I can't figure out why your code doesn't work.
But you can try another approach, that was described in the next article - Parsing Big Records with Json.NET
I can't get the DataContractJsonSerializer object to swallow my stream. When I execute the code with the commented-out line active, I get to see the text provided (and it is a parsable JSON object), so I know that the stream is working fine.
However, for some reason, the compiler complains that the streamReader I'm trying to shove down its throat in ReadObject isn't a Stream. Well, isn't it?!
Argument 1: cannot convert from 'System.IO.StreamReader' to 'System.IO.Stream'
What am I missing and how do I resolve it?
using (StreamReader streamReader = new StreamReader(...))
{
//String responseText = reader.ReadToEnd();
MyThingy thingy = new MyThingy();
DataContractJsonSerializer serializer
= new DataContractJsonSerializer(thingy.GetType());
thingy = serializer.ReadObject(streamReader);
}
I'm adapting this example to work with my stream. Should I approach it from a different angle? If so - how?
You're trying to put in a reader of a stream instead of an actual stream. Skip the using and whatever hides behind the ellipsis (i.e. whatever you put in as an argument when you create an instance of StreamReader), you can probably put that into the ReadObject.
Also, you'll get into problems when reading the data because ReadObject will return an instance of type Object and you'll need to convert it into MyThingy. Since it's a nullable (I'm assuming), you don't have to type cast but rather as-ify it.
MyThingy thingy = new MyThingy();
DataContractJsonSerializer serializer
= new DataContractJsonSerializer(thingy.GetType());
Stream stream = ...;
thingy = serializer.ReadObject(stream) as MyThingy;
You could of course skip the next-to-last line and put the stream directly into the last line.
Courtesy of #JohanLarsson (all Swedes are great, especially those from Stockholm, like me):
In case you can't or don't want to omit the StreamReader declaration in your using statement, I'd suggest that you take a look at BaseStream property to get to it.
You can try this:
using (StreamReader streamReader = new StreamReader(...))
{
DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(MyThingy));
MyThingy thingy = (MyThingy) serializer.ReadObject(streamReader.BaseStream);
}
I've been always using this:
// get stuff here
String json = GetJSON();
List<T> result;
using (var ms = new MemoryStream(Encoding.Unicode.GetBytes(json)))
{
var serializer = new DataContractJsonSerializer(typeof(List<T>));
result = (List<T>)serializer.ReadObject(ms);
}
Presently I need to serialize one of my object which contains more my own classes object.
But the problem is I dont want to save it in a file and then retrieve it into memory stream.
Is there any way to directly serialize my object into stream.
I used BinaryFormatter for seializing.
First I used a MemoryStream directly to take serialize output but it is giving error
at time of deserialization. But later when I serialize it with a file then close it and
again reopen it , it works perfectly. But I want to take it direct into stream because
in my program I need to do it frequently to pass it into network client. And using file
repeatedly might slow down my software.
Hope I clear my problem. Any Sugetion ?
If you're trying to deserialize from the same MemoryStream, have you remembered to seek back to the beginning of the stream first?
var foo = "foo";
var formatter = new BinaryFormatter();
using (var stream = new MemoryStream())
{
// Serialize.
formatter.Serialize(stream, foo);
// Deserialize.
stream.Seek(0, SeekOrigin.Begin);
foo = formatter.Deserialize(stream) as string;
}
Here's a quick and dirty sample, of serializing back and forth a string. Is this what your trying to do?
static void Main(string[] args)
{
var str = "Hello World";
var stream = Serialize(str);
stream.Position = 0;
var str2 = DeSerialize(stream);
Console.WriteLine(str2);
Console.ReadLine();
}
public static object DeSerialize(MemoryStream stream)
{
BinaryFormatter formatter = new BinaryFormatter();
return formatter.Deserialize(stream);
}
public static MemoryStream Serialize(object data)
{
MemoryStream streamMemory = new MemoryStream();
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(streamMemory, data);
return streamMemory;
}
I have a CookieContainer extracted from a HttpWebRequest/HttpWebResponse session named CookieJar. I want my application to store cookies between runs, so cookies collected in the CookieContainer on one run of the program will be used the next run, too.
I think the way to do this would be to somehow write the contents of a CookieContainer to disk. My question is:
How can you write a CookieContainer to the disk? Are there built-in functions for this, or, if not, what are the approaches people have taken? Are there any classes available for simplifying this?
Once you've written a CookieContainer to the disk, how do you load it back in for use?
UPDATE: The first answer has suggested serialization of the CookieContainer. However, I am not very familiar with how to serialize and deserialize such complex objects. Could you provide some sample code? The suggestion was to utilise SOAPFormatter.
This problem was bugging me for ages, nothing I could find worked. I worked it out, so putting that information out into the world.
Answer using BinaryFormatter:
public static void WriteCookiesToDisk(string file, CookieContainer cookieJar)
{
using(Stream stream = File.Create(file))
{
try {
Console.Out.Write("Writing cookies to disk... ");
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(stream, cookieJar);
Console.Out.WriteLine("Done.");
} catch(Exception e) {
Console.Out.WriteLine("Problem writing cookies to disk: " + e.GetType());
}
}
}
public static CookieContainer ReadCookiesFromDisk(string file)
{
try {
using(Stream stream = File.Open(file, FileMode.Open))
{
Console.Out.Write("Reading cookies from disk... ");
BinaryFormatter formatter = new BinaryFormatter();
Console.Out.WriteLine("Done.");
return (CookieContainer)formatter.Deserialize(stream);
}
} catch(Exception e) {
Console.Out.WriteLine("Problem reading cookies from disk: " + e.GetType());
return new CookieContainer();
}
}
I Haven't tried it but it has the attribute Serializable and so can be [de]serialized with .net binary serialization, e.g. SoapFormatter.
Here is the code snippet you asked for.
var formatter = new SoapFormatter();
string file = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "cookies.dat");
using (Stream s = File.Create (file))
formatter.Serialize(s, cookies);
...
CookieContainer retrievedCookies = null;
using (Stream s = File.OpenRead (file))
retrievedCookies = (CookieContainer) formatter.Deserialize(s);
Looking at msdn it seems SoapFormatter is now deprecated in .net 3.5 and it recommends you use Binaryformatter. In the past I have found SoapFormatter useful as the file is readable which helps with diagnosis when deserialization fails! These formatters are sensitive to version changes even in the assembly version (so if you deserialize with one version of the framework upgrade the framework, then it might not deserialize, not sure), but there are ways around this with the Binder property if this becomes a problem. I believe they are primarily designed for short term persistance / remoting, but they might be good enough for you here.
The new DataContractSerializer does not seem to work with it so that is out.
An alternative would be to write a CookieContainerData class to [de]serialize with XmlSerializer and manually convert between this and CookieContainer.
It is interesting to have cookies in text format. Besides being able to be used to write to disk, it can be used for other purposes.
WORKS FOR ME!
Use the LoadCookiesFromFile and SaveCookiesToFile functions to load and write the cookies to the disk respectively.
Or use the GetCookies and SetCookies functions to do the same thing, but to manipulate it as a string.
CookieContainer cookieContainer = new CookieContainer();
void LoadCookiesFromFile(string path)
{
SetCookies(cookieContainer, File.ReadAllText(path));
}
void SaveCookiesToFile(string path)
{
File.WriteAllText(path, GetCookies(cookieContainer));
}
string GetCookies(CookieContainer cookieContainer)
{
using (MemoryStream stream = new MemoryStream())
{
new BinaryFormatter().Serialize(stream, cookieContainer);
var bytes = new byte[stream.Length];
stream.Position = 0;
stream.Read(bytes, 0, bytes.Length);
return Convert.ToBase64String(bytes);
}
}
void SetCookies(CookieContainer cookieContainer, string cookieText)
{
try
{
var bytes = Convert.FromBase64String(cookieText);
using (MemoryStream stream = new MemoryStream(bytes))
{
cookieContainer = (CookieContainer)new BinaryFormatter().Deserialize(stream);
}
}
catch
{
//Ignore if the string is not valid.
}
}
All of the previous answers are outdated since serializing using IFormatter classes was deprecated https://aka.ms/binaryformatter
So the correct method now is serializing it using something else that supports IEnumerable<T>.
Here's an example using System.Text.Json
Serialize
await using var fs = File.OpenWrite("cookies.json");
// Beware: GetAllCookies is available starting with .NET 6
JsonSerializer.Serialize(fs, cookieContainer.GetAllCookies());
Deserialize
var cookieContainer = new CookieContainer();
await using var fs = File.OpenRead("cookies.json");
var cookieCollection = JsonSerializer.Deserialize<CookieCollection>(fs);
cookieContainer.Add(cookieCollection);