I have added new fields to a persisted class, and need to ensure that they are set to sensible defaults when loading older versions of the XAML serialized file from disk. Previously, with the BinaryFormatter, I would use the OnDeserialization method to work out what defaults I should set if new fields have been added to a persisted class (using the OptionalField atttribute). E.g.:
/// <summary>
/// Runs when the entire object graph has been deserialized.
/// </summary>
/// <param name="sender">The object that initiated the callback. The functionality for this parameter is not currently implemented.</param>
public override void
OnDeserialization
(object sender)
{
I can't seem to find anything equivalent so far when writing to a XAML file, e.g:
using (TextReader reader = File.OpenText(filePath))
{
protocol = (Protocol)XamlServices.Load(reader);
}
I would like to ensure that older files which don't contain the new optional fields within the Protocol type (in the example code above) have sensible default values within. I've hunted around but can't seem to find anything obvious (e.g. https://ludovic.chabant.com/devblog/2008/06/25/almost-everything-you-need-to-know-about-xaml-serialization-part-2/) Is there any equivalent?
XamlServices internally uses a XamlObjectWriter. This type has a XamlObjectWriterSettings parameter that includes various callbacks. These aren't exposed by XamlServices, but its functionality is easily replicated.
I haven't tested this extensively, but this seems to work:
public static object LoadXaml(TextReader textReader)
{
var settings = new XamlObjectWriterSettings
{
AfterBeginInitHandler = (s, e) => Debug.Print($"Before deserializing {e.Instance}"),
AfterEndInitHandler = (s, e) => Debug.Print($"After deserializing {e.Instance}")
};
using (var xmlReader = XmlReader.Create(textReader))
using (var xamlReader = new XamlXmlReader(xmlReader))
using (var xamlWriter = new XamlObjectWriter(xamlReader.SchemaContext, settings))
{
XamlServices.Transform(xamlReader, xamlWriter);
return xamlWriter.Result;
}
}
e.Instance contains the object being deserialized. Not sure which callback is best for your purpose. They're more equivalent to the [OnDeserializing]/[OnDeserialized] attributes because they're called when the individual object is deserialized, rather than after the entire graph is complete like IDeserializationCallback.OnDeserialization.
Here is a more complete implementation of a class that provides events during serialization. XamlObjectReader doesn't support callbacks like XamlObjectWriter does, so this uses a workaround. It only raises an event before but not after serializing an object for the reasons explained in the comments.
public class CallbackXamlService
{
// Default settings that XamlService uses
public XmlWriterSettings XmlWriterSettings { get; set; }
= new XmlWriterSettings { Indent = true, OmitXmlDeclaration = true };
public event EventHandler<XamlObjectEventArgs> BeforeDeserializing;
public event EventHandler<XamlObjectEventArgs> AfterDeserializing;
public event EventHandler<XamlObjectEventArgs> BeforeSerializing;
// AfterSerializing event doesn't seem to be easily possible, see below
public object LoadXaml(TextReader textReader)
{
var settings = new XamlObjectWriterSettings
{
BeforePropertiesHandler = (s, e) => BeforeDeserializing?.Invoke(this, e),
AfterPropertiesHandler = (s, e) => AfterDeserializing?.Invoke(this, e)
};
using (var xmlReader = XmlReader.Create(textReader))
using (var xamlReader = new XamlXmlReader(xmlReader))
using (var xamlWriter = new XamlObjectWriter(xamlReader.SchemaContext, settings))
{
XamlServices.Transform(xamlReader, xamlWriter);
return xamlWriter.Result;
}
}
public string SaveXaml(object instance)
{
var stringBuilder = new StringBuilder();
using (var textWriter = new StringWriter(stringBuilder))
SaveXaml(textWriter, instance);
return stringBuilder.ToString();
}
public void SaveXaml(TextWriter textWriter, object instance)
{
Action<object> beforeSerializing = (obj) => BeforeSerializing?.Invoke(this, new XamlObjectEventArgs(obj));
// There are no equivalent callbacks on XamlObjectReaderSettings
// Using a derived XamlObjectReader to track processed objects instead
using (var xmlWriter = XmlWriter.Create(textWriter, XmlWriterSettings))
using (var xamlXmlWriter = new XamlXmlWriter(xmlWriter, new XamlSchemaContext()))
using (var xamlObjectReader = new CallbackXamlObjectReader(instance, xamlXmlWriter.SchemaContext, null, beforeSerializing))
{
XamlServices.Transform(xamlObjectReader, xamlXmlWriter);
xmlWriter.Flush();
}
}
private class CallbackXamlObjectReader : XamlObjectReader
{
public Action<object> BeforeSerializing { get; }
//private Stack<object> instanceStack = new Stack<object>();
public CallbackXamlObjectReader(object instance, XamlSchemaContext schemaContext, XamlObjectReaderSettings settings, Action<object> beforeSerializing)
: base(instance, schemaContext, settings)
{
BeforeSerializing = beforeSerializing;
}
public override bool Read()
{
if (base.Read())
{
if (NodeType == XamlNodeType.StartObject)
{
//instanceStack.Push(Instance);
BeforeSerializing?.Invoke(Instance);
}
// XamlObjectReader.Instance is not set on EndObject nodes
// EndObject nodes do not line up with StartObject nodes when types like arrays and dictionaries
// are involved, so using a stack to track the current instance doesn't always work.
// Don't know if there is a reliable way to fix this without possibly fragile special-casing,
// the XamlObjectReader internals are horrendously complex.
//else if (NodeType == XamlNodeType.EndObject)
//{
// object instance = instanceStack.Pop();
// AfterSerializing(instance);
//}
return true;
}
return false;
}
}
}
Related
I am using SharpSerializer to serialize/deserialize object.
I want the ability to ignore specific properties when deserializing.
SharpSerializer has an option to ignore properties by attribute or by classes and property name:
SharpSerializerSettings.AdvancedSettings.AttributesToIgnore
SharpSerializerSettings.AdvancedSettings.PropertiesToIgnore
but it seems that these settings are only used to ignore from serialization, not from deserialization (I tested with the GitHub source code and the NugetPackage).
Am I correct?
Is there any way to ignore attributes/properties from deserialization?
P.S.
I'm sure there are other great serialization libraries, but it will take a great amount of effort to change the code and all the existing serialized files.
I opened an issue on the GitHub project, but the project does not seem to be active since 2018.
The object with properties to ignore need not be the root object.
You are correct that SharpSerializer does not implement ignoring of property values when deserializing. This can be verified from the reference source for ObjectFactory.fillProperties(object obj, IEnumerable<Property> properties):
private void fillProperties(object obj, IEnumerable<Property> properties)
{
foreach (Property property in properties)
{
PropertyInfo propertyInfo = obj.GetType().GetProperty(property.Name);
if (propertyInfo == null) continue;
object value = CreateObject(property);
if (value == null) continue;
propertyInfo.SetValue(obj, value, _emptyObjectArray);
}
}
This code unconditionally sets any property read from the serialization stream into the incoming object using reflection, without checking the list of ignored attributes or properties.
Thus the only way to ignore your desired properties would seem to be to create your own versions of XmlPropertyDeserializer or BinaryPropertyDeserializer that skip or filter the unwanted properties. The following is one possible implementation for XML. This implementation reads the properties from XML into a Property hierarchy as usual, then applies a filter action to remove properties corresponding to .NET properties with a custom attribute [SharpSerializerIgnoreForDeserialize] applied, then finally creates the object tree using the pruned Property.
[System.AttributeUsage(System.AttributeTargets.Property, AllowMultiple = false, Inherited = true)]
public class SharpSerializerIgnoreForDeserializeAttribute : System.Attribute { }
public class PropertyDeserializerDecorator : IPropertyDeserializer
{
readonly IPropertyDeserializer deserializer;
public PropertyDeserializerDecorator(IPropertyDeserializer deserializer) => this.deserializer = deserializer ?? throw new ArgumentNullException();
public virtual void Open(Stream stream) => deserializer.Open(stream);
public virtual Property Deserialize() => deserializer.Deserialize();
public virtual void Close() => deserializer.Close();
}
public class CustomPropertyDeserializer : PropertyDeserializerDecorator
{
Action<Property> deserializePropertyAction;
public CustomPropertyDeserializer(IPropertyDeserializer deserializer, Action<Property> deserializePropertyAction = default) : base(deserializer) => this.deserializePropertyAction = deserializePropertyAction;
public override Property Deserialize()
{
var property = base.Deserialize();
if (deserializePropertyAction != null)
property.WalkProperties(p => deserializePropertyAction(p));
return property;
}
}
public static partial class SharpSerializerExtensions
{
public static SharpSerializer Create(SharpSerializerXmlSettings settings, Action<Property> deserializePropertyAction = default)
{
// Adapted from https://github.com/polenter/SharpSerializer/blob/42f9a20b3934a7f2cece356cc8116a861cec0b91/SharpSerializer/SharpSerializer.cs#L139
// By https://github.com/polenter
var typeNameConverter = settings.AdvancedSettings.TypeNameConverter ??
new TypeNameConverter(
settings.IncludeAssemblyVersionInTypeName,
settings.IncludeCultureInTypeName,
settings.IncludePublicKeyTokenInTypeName);
// SimpleValueConverter
var simpleValueConverter = settings.AdvancedSettings.SimpleValueConverter ?? new SimpleValueConverter(settings.Culture, typeNameConverter);
// XmlWriterSettings
var xmlWriterSettings = new XmlWriterSettings
{
Encoding = settings.Encoding,
Indent = true,
OmitXmlDeclaration = true,
};
// XmlReaderSettings
var xmlReaderSettings = new XmlReaderSettings
{
IgnoreComments = true,
IgnoreWhitespace = true,
};
// Create Serializer and Deserializer
var reader = new DefaultXmlReader(typeNameConverter, simpleValueConverter, xmlReaderSettings);
var writer = new DefaultXmlWriter(typeNameConverter, simpleValueConverter, xmlWriterSettings);
var _serializer = new XmlPropertySerializer(writer);
var _deserializer = new CustomPropertyDeserializer(new XmlPropertyDeserializer(reader), deserializePropertyAction);
var serializer = new SharpSerializer(_serializer, _deserializer)
{
//InstanceCreator = settings.InstanceCreator ?? new DefaultInstanceCreator(), -- InstanceCreator not present in SharpSerializer 3.0.1
RootName = settings.AdvancedSettings.RootName,
};
serializer.PropertyProvider.PropertiesToIgnore = settings.AdvancedSettings.PropertiesToIgnore;
serializer.PropertyProvider.AttributesToIgnore = settings.AdvancedSettings.AttributesToIgnore;
return serializer;
}
public static void WalkProperties(this Property property, Action<Property> action)
{
if (action == null || property == null)
throw new ArgumentNullException();
// Avoid cyclic dependencies.
// Reference.IsProcessed is true only for the first reference of an object.
bool skipProperty = property is ReferenceTargetProperty refTarget
&& refTarget.Reference != null
&& !refTarget.Reference.IsProcessed;
if (skipProperty) return;
action(property);
switch (property.Art)
{
case PropertyArt.Collection:
{
foreach (var item in ((CollectionProperty)property).Items)
item.WalkProperties(action);
}
break;
case PropertyArt.Complex:
{
foreach (var item in ((ComplexProperty)property).Properties)
item.WalkProperties(action);
}
break;
case PropertyArt.Dictionary:
{
foreach (var item in ((DictionaryProperty)property).Items)
{
item.Key.WalkProperties(action);
item.Value.WalkProperties(action);
}
}
break;
case PropertyArt.MultiDimensionalArray:
{
foreach (var item in ((MultiDimensionalArrayProperty )property).Items)
item.Value.WalkProperties(action);
}
break;
case PropertyArt.Null:
case PropertyArt.Simple:
case PropertyArt.Reference:
break;
case PropertyArt.SingleDimensionalArray:
{
foreach (var item in ((SingleDimensionalArrayProperty)property).Items)
item.WalkProperties(action);
}
break;
default:
throw new NotImplementedException(property.Art.ToString());
}
}
public static void RemoveIgnoredChildProperties(Property p)
{
if (p.Art == PropertyArt.Complex)
{
var items = ((ComplexProperty)p).Properties;
for (int i = items.Count - 1; i >= 0; i--)
{
if (p.Type.GetProperty(items[i].Name)?.IsDefined(typeof(SharpSerializerIgnoreForDeserializeAttribute), true) == true)
{
items.RemoveAt(i);
}
}
}
}
}
Then, given the following models:
public class Root
{
public List<Model> Models { get; set; } = new ();
}
public class Model
{
public string Value { get; set; }
[SharpSerializerIgnoreForDeserialize]
public string IgnoreMe { get; set; }
}
You would deserialize using the customized XmlPropertyDeserializer as follows:
var settings = new SharpSerializerXmlSettings();
var customSerialzier = SharpSerializerExtensions.Create(settings, SharpSerializerExtensions.RemoveIgnoredChildProperties);
var deserialized = (Root)customSerialzier.Deserialize(stream);
If you need binary deserialization, use the following factory method to create the serializer instead:
public static partial class SharpSerializerExtensions
{
public static SharpSerializer Create(SharpSerializerBinarySettings settings, Action<Property> deserializePropertyAction = default)
{
// Adapted from https://github.com/polenter/SharpSerializer/blob/42f9a20b3934a7f2cece356cc8116a861cec0b91/SharpSerializer/SharpSerializer.cs#L168
// By https://github.com/polenter
var typeNameConverter = settings.AdvancedSettings.TypeNameConverter ??
new TypeNameConverter(
settings.IncludeAssemblyVersionInTypeName,
settings.IncludeCultureInTypeName,
settings.IncludePublicKeyTokenInTypeName);
// Create Serializer and Deserializer
Polenter.Serialization.Advanced.Binary.IBinaryReader reader;
Polenter.Serialization.Advanced.Binary.IBinaryWriter writer;
if (settings.Mode == BinarySerializationMode.Burst)
{
// Burst mode
writer = new BurstBinaryWriter(typeNameConverter, settings.Encoding);
reader = new BurstBinaryReader(typeNameConverter, settings.Encoding);
}
else
{
// Size optimized mode
writer = new SizeOptimizedBinaryWriter(typeNameConverter, settings.Encoding);
reader = new SizeOptimizedBinaryReader(typeNameConverter, settings.Encoding);
}
var _serializer = new BinaryPropertySerializer(writer);
var _deserializer = new CustomPropertyDeserializer(new BinaryPropertyDeserializer(reader), deserializePropertyAction);
var serializer = new SharpSerializer(_serializer, _deserializer)
{
//InstanceCreator = settings.InstanceCreator ?? new DefaultInstanceCreator(), -- InstanceCreator not present in SharpSerializer 3.0.1
RootName = settings.AdvancedSettings.RootName,
};
serializer.PropertyProvider.PropertiesToIgnore = settings.AdvancedSettings.PropertiesToIgnore;
serializer.PropertyProvider.AttributesToIgnore = settings.AdvancedSettings.AttributesToIgnore;
return serializer;
}
}
And do:
var settings = new SharpSerializerBinarySettings();
var customSerialzier = SharpSerializerExtensions.Create(settings, SharpSerializerExtensions.RemoveIgnoredChildProperties);
var deserialized = (Root)customSerialzier.Deserialize(stream);
Notes:
The methods SharpSerializerExtensions.Create() were modeled on SharpSerializer.initialize(SharpSerializerXmlSettings settings) and SharpSerializer.initialize(SharpSerializerBinarySettings settings) by Pawel Idzikowski
The version of SharpSerializer available on nuget, version 3.0.1, only includes commits through 10/8/2017. Submissions since then that add the ability to use Autofac as the instance creator are not available via nuget. My code is based on the version available via nuget, and thus does not initialize SharpSerializer.InstanceCreator which was added in 2018. The project appears not to have updated at all since then.
SharpSerializer.Deserialize() deserializes to the type specified in the serialization stream rather than to a type specified by the caller. It thus appears vulnerable to the sort of type injection attacks described in Alvaro Muñoz & Oleksandr Mirosh's blackhat paper https://www.blackhat.com/docs/us-17/thursday/us-17-Munoz-Friday-The-13th-JSON-Attacks-wp.pdf.
For details see e.g. TypeNameHandling caution in Newtonsoft Json.
If you are willing to fork, modify and build SharpSerializer yourself, you might consider updating ObjectFactory.fillProperties(object obj, IEnumerable<Property> properties) to not set ignored properties.
Demo fiddle #1 here for XML, and #2 here for binary.
I'm trying to create my first SSIS custom source component but I can't get it to save the custom properties into the .dtsx file.
According to https://learn.microsoft.com/en-us/sql/integration-services/extending-packages-custom-objects/persisting-custom-objects , all I needed is to implement the IDTSComponentPersist interface, but this doesn't work, the LoadFromXML and SaveToXML are never called. Neither when I save the file nor when I load the package.
However, if your object has properties that use complex data types, or
if you want to perform custom processing on property values as they
are loaded and saved, you can implement the IDTSComponentPersist
interface and its LoadFromXML and SaveToXML methods. In these methods
you load from (or save to) the XML definition of the package an XML
fragment that contains the properties of your object and their current
values. The format of this XML fragment is not defined; it must only
be well-formed XML.
When I save the SSIS package and look inside the XML, I get this, no data type defined and no values :
Did I miss to set something?
To simplify, I created a small test project. The original project try to save a list of struct with 2 string and 1 integer, but both has the same "incorrect" behavior, SaveToXML and LoadFromXML are never called.
Here's my code:
using System;
using System.Collections.Generic;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Pipeline;
using Microsoft.SqlServer.Dts.Runtime;
using System.Xml;
using System.ComponentModel;
using System.Globalization;
using System.Drawing.Design;
using System.Windows.Forms.Design;
using System.Windows.Forms;
namespace TestCase
{
public class MyConverter : TypeConverter
{
public override bool GetStandardValuesSupported(ITypeDescriptorContext context)
{
return false;
}
public override object ConvertTo(ITypeDescriptorContext context, CultureInfo culture, object value, Type destinationType)
{
if (destinationType.Name.ToUpper() == "STRING")
return string.Join(",", ((List<string>)value).ToArray());
else
return ((string)value).Split(',');
}
public override object ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, object value)
{
if (value.GetType().Name.ToUpper() == "STRING")
return ((string)value).Split(',');
else
return string.Join(",", ((List<string>)value).ToArray());
}
}
class FancyStringEditor : UITypeEditor
{
public override UITypeEditorEditStyle GetEditStyle(ITypeDescriptorContext context)
{
return UITypeEditorEditStyle.Modal;
}
public override object EditValue(ITypeDescriptorContext context, IServiceProvider provider, object value)
{
var svc = (IWindowsFormsEditorService)provider.GetService(typeof(IWindowsFormsEditorService));
List<string> vals = (List<string>)value;
string valsStr = string.Join("\r\n", vals.ToArray());
if (svc != null)
{
using (var frm = new Form { Text = "Your editor here" })
using (var txt = new TextBox { Text = valsStr, Dock = DockStyle.Fill, Multiline = true })
using (var ok = new Button { Text = "OK", Dock = DockStyle.Bottom })
{
frm.Controls.Add(txt);
frm.Controls.Add(ok);
frm.AcceptButton = ok;
ok.DialogResult = DialogResult.OK;
if (svc.ShowDialog(frm) == DialogResult.OK)
{
vals = new List<string>();
vals.AddRange(txt.Text.Split(new string[] { "\r\n" }, StringSplitOptions.RemoveEmptyEntries));
value = vals;
}
}
}
return value;
}
}
[DtsPipelineComponent(ComponentType = ComponentType.SourceAdapter,
CurrentVersion = 0,
Description = "Test class for saving",
DisplayName = "Test class",
IconResource = "None",
NoEditor = false,
RequiredProductLevel = Microsoft.SqlServer.Dts.Runtime.Wrapper.DTSProductLevel.DTSPL_NONE,
SupportsBackPressure = false,
UITypeName = "None")]
public class TestSave : PipelineComponent, IDTSComponentPersist
{
private string _NbBadWordProperty = "NbBadWord";
private string _ListBadWordsProperty = "ListBadWords";
private List<string> _badWords;
public IDTSCustomProperty100 _nb;
public IDTSCustomProperty100 _list;
public TestSave()
{
_badWords = new List<string>();
_badWords.Add("Word1");
_badWords.Add("Word2");
_badWords.Add("Word3");
}
public void LoadFromXML(System.Xml.XmlElement node, IDTSInfoEvents infoEvents)
{
System.Windows.Forms.MessageBox.Show("Oh god! we're inside LoadFromXML!!");
}
public void SaveToXML(System.Xml.XmlDocument doc, IDTSInfoEvents infoEvents)
{
System.Windows.Forms.MessageBox.Show("Oh god! we're inside SaveToXML!!");
XmlElement elementRoot;
XmlNode propertyNode;
// Create a new node to persist the object and its properties.
elementRoot = doc.CreateElement(String.Empty, "NBElement", String.Empty);
XmlAttribute nbEl = doc.CreateAttribute("Nbelement");
nbEl.Value = _badWords.Count.ToString();
elementRoot.Attributes.Append(nbEl);
// Save the three properties of the object from variables into XML.
foreach (string s in _badWords)
{
propertyNode = doc.CreateNode(XmlNodeType.Element, "BadWord", String.Empty);
propertyNode.InnerText = s;
elementRoot.AppendChild(propertyNode);
}
doc.AppendChild(elementRoot);
}
private IDTSCustomProperty100 GetCustomPropertyByName(string name)
{
foreach (IDTSCustomProperty100 prop in this.ComponentMetaData.CustomPropertyCollection)
if (prop.Name.ToUpper() == name)
return prop;
return null;
}
public override DTSValidationStatus Validate()
{
return DTSValidationStatus.VS_ISVALID;
}
public override void ProvideComponentProperties()
{
try
{
base.ProvideComponentProperties();
// reset the component
this.ComponentMetaData.OutputCollection.RemoveAll();
this.ComponentMetaData.InputCollection.RemoveAll();
// Add custom properties
if (GetCustomPropertyByName(_NbBadWordProperty) == null)
{
_nb = this.ComponentMetaData.CustomPropertyCollection.New();
_nb.Name = _NbBadWordProperty;
_nb.Description = "Number of bad word to filter";
_nb.State = DTSPersistState.PS_DEFAULT;
_nb.Value = _badWords.Count;
_nb.ExpressionType = DTSCustomPropertyExpressionType.CPET_NOTIFY;
}
if (GetCustomPropertyByName(_ListBadWordsProperty) == null)
{
IDTSCustomProperty100 _list = this.ComponentMetaData.CustomPropertyCollection.New();
_list.Name = _ListBadWordsProperty;
_list.Description = "List of bad words";
_list.State = DTSPersistState.PS_DEFAULT;
_list.TypeConverter = typeof(MyConverter).AssemblyQualifiedName;
_list.Value = _badWords;
_list.ExpressionType = DTSCustomPropertyExpressionType.CPET_NOTIFY;
_list.UITypeEditor = typeof(FancyStringEditor).AssemblyQualifiedName;
}
// add input objects
// none
// add output objects
IDTSOutput100 o2 = this.ComponentMetaData.OutputCollection.New();
o2.Name = "Dummy output";
o2.IsSorted = false;
foreach (IDTSCustomProperty100 p in this.ComponentMetaData.CustomPropertyCollection)
{
if (p.Name == _ListBadWordsProperty)
{
MyConverter c = new MyConverter();
List<string> l = (List<string>)p.Value;
foreach (string s in l)
{
IDTSOutputColumn100 col1 = o2.OutputColumnCollection.New();
col1.Name = s.Trim();
col1.Description = "Bad word";
col1.SetDataTypeProperties(Microsoft.SqlServer.Dts.Runtime.Wrapper.DataType.DT_WSTR, 500, 0, 0, 0);
}
}
}
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show("Critical error: " + ex.Message);
}
}
}
}
Update1:
Add the TypeConverter and UITypeEditor. Still the same behavior (not saving the "complex" data type).
When I add the source component to a data flow, I got this, everything look fine:
I can edit the property, no problem
But when I save the SSIS package and look at the xml, the property still not saved and still have a datatype of System.NULL:
Thanks!
Important Note - based on Microsoft definition of IDTSComponentPersist Interface and code samples of SaveToXML found on Internet, I suspect that custom persistence can only be implemented on custom SSIS Tasks, Connection Managers and Enumerators.
Well, please choose for yourself whether do you really need to implement custom object persistence. Your custom properties seems to fit well into standard data types Int32 and String.
Important note from Microsoft -
When you implement custom persistence, you must persist all the properties of the object, including both inherited properties and custom properties that you have added.
So, you really have to do a lot of work to persist all properties of component including LocaleID from your sample - in case someone needs to alter it. I would probably do storing ListBadWords custom property as a string without custom XML persistence.
On your code -- the most possible cause of the System.Null data type problem is that ProvideComponentProperties() method is called on initialization of the component, when it is added on the Data Flow. Data type of the property is determined dynamically at this moment, the variable _badwords is not initialized yet and is a reference type, so it is defined as Null reference. The ProvideComponentProperties() method is used to define custom properties and set its default values, to solve your problem - set
if (GetCustomPropertyByName(_ListBadWordsProperty) == null)
{
IDTSCustomProperty100 _list = this.ComponentMetaData.CustomPropertyCollection.New();
_list.Name = _ListBadWordsProperty;
_list.Description = "List of bad words";
_list.State = DTSPersistState.PS_DEFAULT;
_list.TypeConverter = typeof(MyConverter).AssemblyQualifiedName;
// This is the change
_list.Value = String.Empty;
_list.ExpressionType = DTSCustomPropertyExpressionType.CPET_NOTIFY;
_list.UITypeEditor = typeof(FancyStringEditor).AssemblyQualifiedName;
}
If you set yourself up on implementing custom XML persistence - please study Microsoft code sample and other sources. Saving is done a little bit other way. The main difference is that inside elementRoot of the component properties, each property is created under its own XML Node. Node's InnerText is used to store property value, and optional Node's attributes can store additional information.
I would like to have a serialization format that is nearly identical to JSON, except that key-values are represented as <key>="<value>" instead of "<key>":"<value>".
With Newtonsoft I made a custom JsonConverter called TsonConverter that works fairly well, except that it can't "see" an embedded dictionary. Given the following type:
public class TraceyData
{
[Safe]
public string Application { get; set; }
[Safe]
public string SessionID { get; set; }
[Safe]
public string TraceID { get; set; }
[Safe]
public string Workflow { get; set; }
[Safe]
public Dictionary<string, string> Tags {get; set; }
[Safe]
public string[] Stuff {get; set;}
}
And the following code:
TsonConverter weird = new TsonConverter();
JsonSerializerSettings settings = new JsonSerializerSettings();
settings.NullValueHandling = NullValueHandling.Ignore;
settings.Converters.Add(weird);
var tracey = new TraceyData();
tracey.TraceID = Guid.NewGuid().ToString();
tracey.SessionID = "5";
tracey.Tags["Referrer"] = "http://www.sky.net/deals";
tracey.Stuff = new string[] { "Alpha", "Bravo", "Charlie" };
tracey.Application = "Responsive";
string stuff = JsonConvert.SerializeObject(tracey, settings);
I get this:
[Application="Responsive" SessionID="5" TraceID="082ef853-92f8-4ce8-9f32-8e4f792fb022" Tags={"Referrer":"http://www.sky.net/deals"} Stuff=["Alpha","Bravo","Charlie"]]
Obviously I have also overridden the StartObject/EndObject notation, replacing { } with [ ]. Otherwise the results are not bad.
However, there is still the problem of the internal dictionary. In order
to convert the dictionary as well to use my <key>="<value>" format, it looks like I must make a deep dictionary converter.
I'm wondering if there is an easier way to do this.
Perhaps the Newtonsoft tool has a "property generator" and "key-value" generator property that I can set that globally handles this for me?
Any suggestions?
And while we're here, I wonder if there is a StartObject/EndObject formatter property override I can set, which would handle the other customization I've shown above. It would be nice to "skip" making JsonConverter tools for these kinds of simple alterations.
Incidentally:
My custom JsonConverter is choosing properties to serialize based on the [Safe] attribute shown in my sample. This is another nice-to-have. It would be wonderful if the JSon settings could expose an "attribute handler" property that lets me override the usual JSon attributes in favor of my own.
I have no need to de-serialize this format. It is intended as a one-way operation. If someone wishes also to explain how to de-serialize my custom format as well that is an interesting bonus, but definitely not necessary to answer this question.
Appendix
Below is the TraceConverter I had made. It references a FieldMetaData class that simply holds property info.
public class TsonConverter : JsonConverter
{
public override bool CanRead
{
get
{
return false;
}
}
public override bool CanConvert(Type ObjectType)
{
return DataClassifier.TestForUserType(ObjectType);
}
public override void WriteJson(
JsonWriter writer, object value, JsonSerializer serializer)
{
Type objType = value.GetType();
var props = objType.GetProperties(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic);
var propMap = from p in props
from a in p.GetCustomAttributes(typeof(ProfileAttribute), false)
select new FieldMetaData(p, (ProfileAttribute)a);
//writer.WriteStartObject();
writer.WriteStartArray();
bool loopStarted = true;
foreach(var prop in propMap){
object rawValue = prop.GetValue(value);
if (rawValue != null || serializer.NullValueHandling == NullValueHandling.Include)
{
string jsonValue = JsonConvert.SerializeObject(prop.GetValue(value), this);
if (loopStarted)
{
loopStarted = false;
writer.WriteRaw(String.Format("{0}={1}", prop.Name, jsonValue));
}
else
{
writer.WriteRaw(String.Format(" {0}={1}", prop.Name, jsonValue));
}
}
//writer.WriteRaw(String.Format("{0}={1}", prop.Name, prop.GetValue(value)));
//writer.WritePropertyName(prop.Name, false);
//writer.WriteValue(prop.GetValue(value));
}
writer.WriteEndArray();
}
public override object ReadJson(
JsonReader reader, Type objectType,
object existingValue, JsonSerializer serializer)
{
throw new NotImplementedException();
}
}
Rather than creating your own converter, you're going to need to create your own subclass of JsonWriter that writes to your custom file format. (This is how Json.NET implements its BsonWriter.) In your case, your file format is close enough to JSON that you can inherit from JsonTextWriter:
public class TsonTextWriter : JsonTextWriter
{
TextWriter _writer;
public TsonTextWriter(TextWriter textWriter)
: base(textWriter)
{
if (textWriter == null)
throw new ArgumentNullException("textWriter");
QuoteName = false;
_writer = textWriter;
}
public override void WriteStartObject()
{
SetWriteState(JsonToken.StartObject, null);
_writer.Write('[');
}
protected override void WriteEnd(JsonToken token)
{
switch (token)
{
case JsonToken.EndObject:
_writer.Write(']');
break;
default:
base.WriteEnd(token);
break;
}
}
public override void WritePropertyName(string name)
{
WritePropertyName(name, true);
}
public override void WritePropertyName(string name, bool escape)
{
SetWriteState(JsonToken.PropertyName, name);
var escaped = name;
if (escape)
{
escaped = JsonConvert.ToString(name, '"', StringEscapeHandling);
escaped = escaped.Substring(1, escaped.Length - 2);
}
// Maybe also escape the space character if it appears in a name?
_writer.Write(escaped.Replace("=", #"\u003d"));// Replace "=" with unicode escape sequence.
_writer.Write('=');
}
/// <summary>
/// Writes the JSON value delimiter. (Remove this override if you want to retain the comma separator.)
/// </summary>
protected override void WriteValueDelimiter()
{
_writer.Write(' ');
}
/// <summary>
/// Writes an indent space.
/// </summary>
protected override void WriteIndentSpace()
{
// Do nothing.
}
}
Having done this, now all classes will be serialized to your custom format when you use this writer, for instance:
var tracey = new TraceyData();
tracey.TraceID = Guid.NewGuid().ToString();
tracey.SessionID = "5";
tracey.Tags["Referrer"] = "http://www.sky.net/deals";
tracey.Stuff = new string[] { "Alpha", "Bravo", "Charlie" };
tracey.Application = "Responsive";
JsonSerializerSettings settings = new JsonSerializerSettings();
settings.NullValueHandling = NullValueHandling.Ignore;
using (var sw = new StringWriter())
{
using (var jsonWriter = new TsonTextWriter(sw))
{
JsonSerializer.CreateDefault(settings).Serialize(jsonWriter, tracey);
}
Debug.WriteLine(sw.ToString());
}
Produces the output
[Application="Responsive" SessionID="5" TraceID="2437fe67-9788-47ba-91ce-2e5b670c2a34" Tags=[Referrer="http://www.sky.net/deals"] Stuff=["Alpha" "Bravo" "Charlie"]]
As far as deciding whether to serialize properties based on the presence of a [Safe] attribute, that's sort of a second question. You will need to create your own ContractResolver and override CreateProperty, for instance as shown here: Using JSON.net, how do I prevent serializing properties of a derived class, when used in a base class context?
Update
If you want to retain the comma separator for arrays but not objects, modify WriteValueDelimiter as follows:
/// <summary>
/// Writes the JSON value delimiter. (Remove this override if you want to retain the comma separator.)
/// </summary>
protected override void WriteValueDelimiter()
{
if (WriteState == WriteState.Array)
_writer.Write(',');
else
_writer.Write(' ');
}
I know that the same problem is faced by a lot of people in one way or another but what I'm confused about is that how come Newtonsoft JSON Serializer is able to correctly handle this case while JavaScriptSerializer fails to do so.
I'm going to use the same code sample used in one of the other stackoverflow thread (JavascriptSerializer serializing property twice when "new" used in subclass)
void Main()
{
System.Web.Script.Serialization.JavaScriptSerializer serializer = new System.Web.Script.Serialization.JavaScriptSerializer();
var json = serializer.Serialize(new Limited());
Limited status = serializer.Deserialize<Limited>(json); --> throws AmbiguousMatchException
}
public class Full
{
public String Stuff { get { return "Common things"; } }
public FullStatus Status { get; set; }
public Full(bool includestatus)
{
if(includestatus)
Status = new FullStatus();
}
}
public class Limited : Full
{
public new LimitedStatus Status { get; set; }
public Limited() : base(false)
{
Status = new LimitedStatus();
}
}
public class FullStatus
{
public String Text { get { return "Loads and loads and loads of things"; } }
}
public class LimitedStatus
{
public String Text { get { return "A few things"; } }
}
But if I use Newtonsoft Json Serializer, everythings works fine. Why? And is it possible to achieve the same using JavaScriptSerializer?
void Main()
{
var json = JsonConvert.SerializeObject(new Limited());
Limited status = JsonConvert.DeserializeObject<Limited>(json); ----> Works fine.
}
The reason this works in Json.NET is that it has specific code to handle this situation. From JsonPropertyCollection.cs:
/// <summary>
/// Adds a <see cref="JsonProperty"/> object.
/// </summary>
/// <param name="property">The property to add to the collection.</param>
public void AddProperty(JsonProperty property)
{
if (Contains(property.PropertyName))
{
// don't overwrite existing property with ignored property
if (property.Ignored)
return;
JsonProperty existingProperty = this[property.PropertyName];
bool duplicateProperty = true;
if (existingProperty.Ignored)
{
// remove ignored property so it can be replaced in collection
Remove(existingProperty);
duplicateProperty = false;
}
else
{
if (property.DeclaringType != null && existingProperty.DeclaringType != null)
{
if (property.DeclaringType.IsSubclassOf(existingProperty.DeclaringType))
{
// current property is on a derived class and hides the existing
Remove(existingProperty);
duplicateProperty = false;
}
if (existingProperty.DeclaringType.IsSubclassOf(property.DeclaringType))
{
// current property is hidden by the existing so don't add it
return;
}
}
}
if (duplicateProperty)
throw new JsonSerializationException("A member with the name '{0}' already exists on '{1}'. Use the JsonPropertyAttribute to specify another name.".FormatWith(CultureInfo.InvariantCulture, property.PropertyName, _type));
}
Add(property);
}
As you can see above, there is specific code here to prefer derived class properties over base class properties of the same name and visibility.
JavaScriptSerializer has no such logic. It simply calls Type.GetProperty(string, flags)
PropertyInfo propInfo = serverType.GetProperty(memberName,
BindingFlags.Instance | BindingFlags.IgnoreCase | BindingFlags.Public);
This method is documented to throw an exception in exactly this situation:
Situations in which AmbiguousMatchException occurs include the following:
A type contains two indexed properties that have the same name but different numbers of parameters. To resolve the ambiguity, use an overload of the GetProperty method that specifies parameter types.
A derived type declares a property that hides an inherited property with the same name, using the new modifier (Shadows in Visual Basic). To resolve the ambiguity, include BindingFlags.DeclaredOnly to restrict the search to members that are not inherited.
I don't know why Microsoft didn't add logic for this to JavaScriptSerializer. It's really a very simple piece of code; perhaps it got eclipsed by DataContractJsonSerializer?
You do have a workaround, which is to write a custom JavaScriptConverter:
public class LimitedConverter : JavaScriptConverter
{
const string StuffName = "Stuff";
const string StatusName = "Status";
public override object Deserialize(IDictionary<string, object> dictionary, Type type, JavaScriptSerializer serializer)
{
var limited = new Limited();
object value;
if (dictionary.TryGetValue(StuffName, out value))
{
// limited.Stuff = serializer.ConvertToType<string>(value); // Actually it's get only.
}
if (dictionary.TryGetValue(StatusName, out value))
{
limited.Status = serializer.ConvertToType<LimitedStatus>(value);
}
return limited;
}
public override IDictionary<string, object> Serialize(object obj, JavaScriptSerializer serializer)
{
var limited = (Limited)obj;
if (limited == null)
return null;
var dict = new Dictionary<string, object>();
if (limited.Stuff != null)
dict.Add(StuffName, limited.Stuff);
if (limited.Status != null)
dict.Add(StatusName, limited.Status);
return dict;
}
public override IEnumerable<Type> SupportedTypes
{
get { return new [] { typeof(Limited) } ; }
}
}
And then use it like:
try
{
System.Web.Script.Serialization.JavaScriptSerializer serializer = new System.Web.Script.Serialization.JavaScriptSerializer();
serializer.RegisterConverters(new JavaScriptConverter[] { new LimitedConverter() });
var json = serializer.Serialize(new Limited());
Debug.WriteLine(json);
var status = serializer.Deserialize<Limited>(json);
var json2 = serializer.Serialize(status);
Debug.WriteLine(json2);
}
catch (Exception ex)
{
Debug.Assert(false, ex.ToString()); // NO ASSERT.
}
I'm currently trying out the new MemoryCache in .Net 4 to cache a few bits of data in one of our apps. The trouble I have is the objects are updated and the cache appears to be persisting the changes e.g.
public IEnumerable<SomeObject> GetFromDatabase(){
const string _cacheKeyGetDisplayTree = "SomeKey";
ObjectCache _cache = MemoryCache.Default;
var objectInCache = _cache.Get(_cacheKeyGetDisplayTree) as IEnumerable<SomeObject>;
if (objectInCache != null)
return objectInCache.ToList();
// Do something to get the items
_cache.Add(_cacheKeyGetDisplayTree, categories, new DateTimeOffset(DateTime.UtcNow.AddHours(1)));
return categories.ToList();
}
public IEnumerable<SomeObject> GetWithIndentation(){
var categories = GetFromDatabase();
foreach (var c in categories)
{
c.Name = "-" + c.Name;
}
return categories;
}
If I were calling GetWithIndentation() first and then later calling GetFromDatabase() I would expect it to return the original list of SomeObject but instead it returns the modified items (with "-" prefixed on the name).
I thought ToList() destroyed the reference but it still seems to persist the changes. I'm sure it's obvious but can anyone spot where I'm going wrong?
I created a ReadonlyMemoryCache class to solve this problem. It inherits from the .NET 4.0 MemoryCache, but objects are stored readonly (by-value) and cannot be modified. I deep copy the objects before storing using binary serialization.
using System;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.IO;
using System.Runtime.Caching;
using System.Runtime.Serialization.Formatters.Binary;
using System.Threading.Tasks;
namespace ReadOnlyCache
{
class Program
{
static void Main()
{
Start();
Console.ReadLine();
}
private static async void Start() {
while (true)
{
TestMemoryCache();
await Task.Delay(TimeSpan.FromSeconds(1));
}
}
private static void TestMemoryCache() {
List<Item> items = null;
string cacheIdentifier = "items";
var cache = ReadonlyMemoryCache.Default;
//change to MemoryCache to understand the problem
//var cache = MemoryCache.Default;
if (cache.Contains(cacheIdentifier))
{
items = cache.Get(cacheIdentifier) as List<Item>;
Console.WriteLine("Got {0} items from cache: {1}", items.Count, string.Join(", ", items));
//modify after getting from cache, cached items will remain unchanged
items[0].Value = DateTime.Now.Millisecond.ToString();
}
if (items == null)
{
items = new List<Item>() { new Item() { Value = "Steve" }, new Item() { Value = "Lisa" }, new Item() { Value = "Bob" } };
Console.WriteLine("Reading {0} items from disk and caching", items.Count);
//cache for x seconds
var policy = new CacheItemPolicy() { AbsoluteExpiration = new DateTimeOffset(DateTime.Now.AddSeconds(5)) };
cache.Add(cacheIdentifier, items, policy);
//modify after writing to cache, cached items will remain unchanged
items[1].Value = DateTime.Now.Millisecond.ToString();
}
}
}
//cached items must be serializable
[Serializable]
class Item {
public string Value { get; set; }
public override string ToString() { return Value; }
}
/// <summary>
/// Readonly version of MemoryCache. Objects will always be returned in-value, via a deep copy.
/// Objects requrements: [Serializable] and sometimes have a deserialization constructor (see http://stackoverflow.com/a/5017346/2440)
/// </summary>
public class ReadonlyMemoryCache : MemoryCache
{
public ReadonlyMemoryCache(string name, NameValueCollection config = null) : base(name, config) {
}
private static ReadonlyMemoryCache def = new ReadonlyMemoryCache("readonlydefault");
public new static ReadonlyMemoryCache Default {
get
{
if (def == null)
def = new ReadonlyMemoryCache("readonlydefault");
return def;
}
}
//we must run deepcopy when adding, otherwise items can be changed after the add() but before the get()
public new bool Add(CacheItem item, CacheItemPolicy policy)
{
return base.Add(item.DeepCopy(), policy);
}
public new object AddOrGetExisting(string key, object value, DateTimeOffset absoluteExpiration, string regionName = null)
{
return base.AddOrGetExisting(key, value.DeepCopy(), absoluteExpiration, regionName);
}
public new CacheItem AddOrGetExisting(CacheItem item, CacheItemPolicy policy)
{
return base.AddOrGetExisting(item.DeepCopy(), policy);
}
public new object AddOrGetExisting(string key, object value, CacheItemPolicy policy, string regionName = null)
{
return base.AddOrGetExisting(key, value.DeepCopy(), policy, regionName);
}
//methods from ObjectCache
public new bool Add(string key, object value, DateTimeOffset absoluteExpiration, string regionName = null)
{
return base.Add(key, value.DeepCopy(), absoluteExpiration, regionName);
}
public new bool Add(string key, object value, CacheItemPolicy policy, string regionName = null)
{
return base.Add(key, value.DeepCopy(), policy, regionName);
}
//for unknown reasons, we also need deepcopy when GETTING values, even though we run deepcopy on all (??) set methods.
public new object Get(string key, string regionName = null)
{
var item = base.Get(key, regionName);
return item.DeepCopy();
}
public new CacheItem GetCacheItem(string key, string regionName = null)
{
var item = base.GetCacheItem(key, regionName);
return item.DeepCopy();
}
}
public static class DeepCopyExtentionMethods
{
/// <summary>
/// Creates a deep copy of an object. Must be [Serializable] and sometimes have a deserialization constructor (see http://stackoverflow.com/a/5017346/2440)
/// </summary>
public static T DeepCopy<T>(this T obj)
{
using (var ms = new MemoryStream())
{
var formatter = new BinaryFormatter();
formatter.Serialize(ms, obj);
ms.Position = 0;
return (T)formatter.Deserialize(ms);
}
}
}
}
In memory cached objects are stored within the same process space as the cache client process. When a cache client requests a cached object, the client receives a reference to the locally cached object rather than a copy.
The only way to get a clean copy of the object is to implement a custom clone mechanism (ICloneable, Serialization, Automapping, ...). With that copy you will be able to alter the new object without altering the parent object.
Depending on your use case, it is generally not recommanded to update an object in the cache.
You can do it easier if you deserialize and serialize again and get your cache object "By Value".
You can do it like this with Newtonsoft lib (just get it from NuGet)
var cacheObj = HttpRuntime.Cache.Get(CACHEKEY);
var json = JsonConvert.SerializeObject(cacheObj);
var byValueObj = JsonConvert.DeserializeObject<List<string>>(json);
return byValueObj;
Why not just store as json or a string? These are not passed by reference and when you get out of the cache you will get a new copy :) I am here to be challenged as thats what I am doing atm!
Serialization/Deserialization will solve the problem but at the same time it defeats the porpose of having objects in memory. The role of cache is to provide fast access to the stored object and we are adding the deserialization overhead here. Since deserialization is required I would suggest cache as service , something like redis cache, it will be centralized so you don't have to have copy of memory object per worker process and deserialization is anyways done.
The key thing in this case that you chose a fast serialization/deserialization option.