Develop Reference

DataContractSerialization - Serialization OutofMemoryException

I have a class that serializes and deserializes objects for a given type. I am trying to serialize a Custom file object containing the file name, the data in the file and a few other details like created time, modified time etc associated with the file. Additionally the custome class includes some properties or flags I would need at my receiver end (where I deserialize).
When I try to serialize around 30K of such file objects, it does the serialization successfully for a large majority, but throws back an outofmemoryexception for some files.
Below is my serialization class code:
public static string SerializeToByteArray(Type T, object request)
{
DataContractSerializer serializer = new DataContractSerializer(T);
using (MemoryStream memStream = new MemoryStream())
{
using (StreamReader reader = new StreamReader(memStream))
{
serializer.WriteObject(memStream, request);
memStream.Position = 0;
return reader.ReadToEnd();
}
}
}
public static T DeserializeFromByteArray<T>(string xml)
{
DataContractSerializer deserializer = new DataContractSerializer(typeof(T));
using (MemoryStream memStream = new MemoryStream())
{
byte[] data = System.Text.Encoding.UTF8.GetBytes(xml);
memStream.Write(data, 0, data.Length);
memStream.Position = 0;
var newobj = (T)deserializer.ReadObject(memStream);
return newobj;
}
}

Xsd2Code serialize file OutOfMemoryException

I m using Xsd2Code to serialize my object in order to generate a Xml file.
It works fine, just when the file contains much data, I get an OutOfMemoryException. Here's the code I used to serialize my object :
/// Serializes current EntityBase object into an XML document
/// </summary>
// <returns>string XML value</returns>
public virtual string Serialize() {
System.IO.StreamReader streamReader = null;
System.IO.MemoryStream memoryStream = null;
try {
memoryStream = new System.IO.MemoryStream();
Serializer.Serialize(memoryStream, this);
memoryStream.Seek(0, System.IO.SeekOrigin.Begin);
streamReader = new System.IO.StreamReader(memoryStream);
return streamReader.ReadToEnd();
}
finally {
if (streamReader != null) {
streamReader.Dispose();
}
if (memoryStream != null) {
memoryStream.Dispose();
}
}
}
My request here, is how can I extend the memory buffer, or how can I avoid such an exception?
Regards.

You don't show the complete ToString() output of the OutOfMemoryException so it's hard to say for sure how much this will help, but one possibility would be to write directly to a StringWriter without creating an intermediate MemoryStream, like so:
public virtual string Serialize()
{
return this.Serialize(Serializer);
}
Using the extension method:
public static class XmlSerializerExtensions
{
class NullEncodingStringWriter : StringWriter
{
public override Encoding Encoding { get { return null; } }
}
public static string Serialize<T>(this T obj, XmlSerializer serializer = null, bool indent = true)
{
if (serializer == null)
serializer = new XmlSerializer(obj.GetType());
// Precisely emulate the output of http://referencesource.microsoft.com/#System.Xml/System/Xml/Serialization/XmlSerializer.cs,2c706ead96e5c4fb
// - Indent by 2 characters
// - Suppress output of the "encoding" tag.
using (var textWriter = new NullEncodingStringWriter())
{
using (var xmlWriter = new XmlTextWriter(textWriter))
{
if (indent)
{
xmlWriter.Formatting = Formatting.Indented;
xmlWriter.Indentation = 2;
}
serializer.Serialize(xmlWriter, obj);
}
return textWriter.ToString();
}
}
}
You might also consider eliminating the formatting and indentation to save more string memory by setting indent = false.
This will reduce your peak memory footprint somewhat, since it completely eliminates the need to have a large MemoryStream in memory at the same time as the resulting string. It won't reduce your peak memory requirement enormously, however, since the memory taken by the MemoryStream will have been proportional to the memory taken by the final XML string.
Beyond that, I can only suggest trying to stream directly to your database.

DataContractSerializer with MemoryStream missing bytes

I am trying to use the WCF DataContractSerializer to serialize a DataContract object into a memoryStream.
Then I use the memoryStream.ToArray to get the serialized content.
Finally, I persist the memoryStream into a file using anther fileStream.
My initial implement is like this. I am missing bytes at the end of the persisted File.
public virtual string SerializeTransient(DataObject data, string targetPath)
{
string securityCode;
using (var memoryStream = new MemoryStream())
{
using (var xmlWriter = XmlWriter.Create(memoryStream, new XmlWriterSettings {Indent = true}))
{
_serializer.WriteObject(xmlWriter, data);
using (var fileStream = new FileStream(targetPath, FileMode.Create))
{
securityCode = CalculateSecurityCode(memoryStream.ToArray());
memoryStream.WriteTo(fileStream);
}
}
}
return securityCode;
}
If I move the persist logic out of the inner using{} block (see below), the output is correct. It almost feels like the WriteObject function didnt finish what it is doing. Could someone please explain to me what is happening there? Thanks.
public virtual string SerializeTransient(DataObject data, string targetPath)
{
string securityCode;
using (var memoryStream = new MemoryStream())
{
using (var xmlWriter = XmlWriter.Create(memoryStream, new XmlWriterSettings {Indent = true}))
{
_serializer.WriteObject(xmlWriter, data);
}
using (var fileStream = new FileStream(targetPath, FileMode.Create))
{
securityCode = CalculateSecurityCode(memoryStream.ToArray());
memoryStream.WriteTo(fileStream);
}
}
return securityCode;
}

XmlWriter has an internal buffer. You should either Close/Dispose XmlWriter or call the XmlWriter.Flush() to force all content to be written to underlying stream (memoryStream).
If memoryStream.ToArray() is called before writer.Flush() then some bytes will possibly remain in internal writer buffer.

Save datacontract serialized object as string to save in sql db

I have a custom type UserSettingConfig I want to save in my database, I want to save it as pure XML as the type might be changed later and migrating pure xml is easier than a binary objet.
public class Serialize
{
private readonly DataContractSerializer _serializer;
public Serialize()
{
_serializer = new DataContractSerializer(typeof(UserSettingConfig));
}
public string SerializeObject(UserSettingConfig userSettingConfig)
{
using (var memoryStream = new MemoryStream())
{
_serializer.WriteObject(memoryStream, userSettingConfig);
string userSettingXml = memoryStream.ToString();
memoryStream.Close();
return userSettingXml;
}
}
public UserSettingConfig DeSerializeObject(string userSettingXml)
{
UserSettingConfig userSettingConfig;
using (var stream = new MemoryStream(userSettingXml))
{
stream.Position = 0;
userSettingConfig = (UserSettingConfig)_serializer.ReadObject(stream);
}
return userSettingConfig;
}
}
This dont work as the Memory Stream want a byte array or int
I want my Serialize to return a string (I can save as varchar(MAX) in my database)

DataContractSerializer.WriteObject has an overload that takes an XmlWriter. You can construct one of those that writes the XML to a StringBuilder:
private static string SerializeToString(object objectToSerialize)
{
var serializer = new DataContractSerializer(objectToSerialize.GetType());
var output = new StringBuilder();
var xmlWriter = XmlWriter.Create(output);
serializer.WriteObject(xmlWriter, objectToSerialize);
xmlWriter.Close();
return output.ToString();
}

You may also consider serializing to JSON instead of XML, using the excellent JSON.NET library which can serialize even the most complex objects easily. JSON is very compact and is still readable.
To serialize:
string json = Newtonsoft.Json.JavaScriptConvert.SerializeObject(anySerializableObject);
To deserialize:
MyClass instance = (MyClass) Newtonsoft.Json.JavaScriptConvert.DeserializeObject(json, typeof(MyClass));

If you need xml without xml declaration, you should use XmlWriterSettings. For instance when you need xml string for node but not entire xml document.
private static string SerializeToString(object objectToSerialize)
{
var serializer = new DataContractSerializer(objectToSerialize.GetType());
var output = new StringBuilder();
var xmlWriter = XmlWriter.Create(output, new XmlWriterSettings() { OmitXmlDeclaration = true});
serializer.WriteObject(xmlWriter, objectToSerialize);
xmlWriter.Close();
return output.ToString();
}

Serialization in C# without using file system

I have a simple 2D array of strings and I would like to stuff it into an SPFieldMultiLineText in MOSS. This maps to an ntext database field.
I know I can serialize to XML and store to the file system, but I would like to serialize without touching the filesystem.
public override void ItemAdding(SPItemEventProperties properties)
{
// build the array
List<List<string>> matrix = new List<List<string>>();
/*
* populating the array is snipped, works fine
*/
// now stick this matrix into the field in my list item
properties.AfterProperties["myNoteField"] = matrix; // throws an error
}
Looks like I should be able to do something like this:
XmlSerializer s = new XmlSerializer(typeof(List<List<string>>));
properties.AfterProperties["myNoteField"] = s.Serialize.ToString();
but that doesn't work. All the examples I've found demonstrate writing to a text file.

StringWriter outStream = new StringWriter();
XmlSerializer s = new XmlSerializer(typeof(List<List<string>>));
s.Serialize(outStream, myObj);
properties.AfterProperties["myNoteField"] = outStream.ToString();

Here's a Generic serializer (C#):
public string SerializeObject<T>(T objectToSerialize)
{
BinaryFormatter bf = new BinaryFormatter();
MemoryStream memStr = new MemoryStream();
try
{
bf.Serialize(memStr, objectToSerialize);
memStr.Position = 0;
return Convert.ToBase64String(memStr.ToArray());
}
finally
{
memStr.Close();
}
}
In your case you could call with:
SerializeObject<List<string>>(matrix);

Use the TextWriter and TextReader classes with the StringWriter.
To Wit:
XmlSerializer s = new XmlSerializer(typeof(whatever));
TextWriter w = new StringWriter();
s.Serialize(w, whatever);
yourstring = w.ToString();

IN VB.NET
Public Shared Function SerializeToByteArray(ByVal object2Serialize As Object) As Byte()
Using stream As New MemoryStream
Dim xmlSerializer As New XmlSerializer(object2Serialize.GetType())
xmlSerializer.Serialize(stream, object2Serialize)
Return stream.ToArray()
End Using
End Function
Public Shared Function SerializeToString(ByVal object2Serialize As Object) As String
Dim bytes As Bytes() = SerializeToByteArray(object2Serialize)
Return Text.UTF8Encoding.GetString(bytes)
End Function
IN C#
public byte[] SerializeToByteArray(object object2Serialize) {
using(MemoryStream stream = new MemoryStream()) {
XmlSerializer xmlSerializer = new XmlSerializer(object2Serialize.GetType());
xmlSerializer.Serialize(stream, object2Serialize);
return stream.ToArray();
}
}
public string SerializeToString(object object2Serialize) {
byte[] bytes = SerializeToByteArray(object2Serialize);
return Text.UTF8Encoding.GetString(bytes);
}