I am trying to write a MongoDb serializer in c# that will allow me to decorate properties via a [Encrypt()] attribute and then at runtime it would allow me to generate an additional property called PropertyName_Encrypted which would contain the encrypted value.
On deserialization, the encrypted property value would be set in the parent property so that the default GET for the property always returns the encrypted value. Users will then call an optional Decrypt() method on the object to get decrypted values.
In doing so, I'm running into some interesting challenges:
How do I add Additional properties to the document when I am serializing current Element? How do I get the current element's name?
Is there a way I can read a specific property from the document/object? For e.g. say I want to pass a symmetric encryption key and read that to encrypt the data while serializing the current element? Is there any way I can do that?
Here are things I have done so far:
I've built an Encrypt Attribute as follows:
[AttributeUsage(AttributeTargets.Property)]
public class EncryptAttribute : Attribute
{
private readonly EncryptedFieldType _fieldType;
private readonly bool _tokenizeDisplay;
private readonly string _encryptedFieldName;
/// <summary>
///
/// </summary>
/// <param name="fieldType">The field type to encrypt. Useful if display needs to show some formatting. If no formatting is necessary, simply set to "Other".</param>
/// <param name="tokenizeDisplay">If set to true, will persist the tokenized value in the original field for display purposes.</param>
/// <param name="encryptedFieldName">Optional. If set, will save the encrypted value in the field name specified. By default all encrypted field values are stored in the corresponding _Encrypted field name. So EmailAddress field if encrypted, would have value under EmailAddress_Encrypted.</param>
public EncryptAttribute(EncryptedFieldType fieldType, bool tokenizeDisplay, string encryptedFieldName = "")
{
_fieldType = fieldType;
_tokenizeDisplay = tokenizeDisplay;
_encryptedFieldName = encryptedFieldName;
}
}
I read this Attribute on Startup and add an Encryption Serializer to the properties that are decorated using this attribute. The code that does that is like so:
var assemblies = AppDomain.CurrentDomain.GetAssemblies()
.Where(x => x.FullName.StartsWith("MongoCustomSerializer"))
.ToList();
var mapper = new Mapper();
foreach (var assembly in assemblies)
{
mapper.Map(assembly);
}
The mapper simply checks which properties in the document have the Encrypt attribute to add the serializer:
public sealed class Mapper
{
public void Map(Assembly assembly)
{
var encryptableTypes = assembly.GetTypes().Where(p =>
typeof(IEncryptable).IsAssignableFrom(p) && p.IsClass && !p.IsInterface && !p.IsValueType &&
!p.IsAbstract).ToList();
if (encryptableTypes.Any())
{
foreach (var encryptableType in encryptableTypes)
{
Map(encryptableType);
}
}
}
private void Map(Type documentType)
{
var properties =
documentType.GetProperties(BindingFlags.Public | BindingFlags.Instance);
if (properties.Length <= 0)
{
return;
}
foreach (var property in properties)
{
RegisterEncrpytionSerializer(property, typeof(EncryptAttribute), documentType);
}
}
private void RegisterEncrpytionSerializer(PropertyInfo property, Type encryptAttributeType, Type documentType)
{
var encryptAttributes = property.GetCustomAttributes(encryptAttributeType, false).ToList();
if (!encryptAttributes.Any()) return;
var memberMap = BsonClassMap.LookupClassMap(documentType).GetMemberMap(property.Name);
memberMap?.SetSerializer(new EncryptionSerializer());
}
}
In my unit tests, I'm getting an error stating that the Bson Class Map is already frozen. Even if I were to figure out a way to bypass that, how would this EncryptionSerializer class work to where I could write an additional property?
Would love to see if someone can assist!
UPDATE 1 - I was able to get the FREEZE error taken care of. It would appear that the LookupClassMap freezes the Member and Class Map info.
This change from the link allows me to take care of that issue:
private void RegisterEncrpytionSerializer(PropertyInfo property, Type encryptAttributeType, Type documentType)
{
var encryptAttributes = property.GetCustomAttributes(encryptAttributeType, false).ToList();
if (!encryptAttributes.Any()) return;
var classMapDefinition = typeof(BsonClassMap<>);
var classMapType = classMapDefinition.MakeGenericType(documentType);
var classMap = (BsonClassMap)Activator.CreateInstance(classMapType);
classMap.AutoMap();
var memberMap = classMap.GetMemberMap(property.Name);
memberMap?.SetSerializer(new KeyVaultEncryptionSerializer(memberMap.ElementName));
}
Are you using a service for saving/retrieving your items that actually call the DB?
I believe you should move the responsibility for writing/reading encrypted values to the calling service (i.e a repository implementation) instead of the BsonSerializer.
It would make sense to me that encryption/decryption is part of the persistence layer and something not handled in the application when needed.
Your implementation targets only the specified property you want to serialize. It doesn't make sense that it creates another property.
A second thought is that your suggested approach with properties that change value based on Decrypt() probably isn't a good idea since it makes your code unpredictable and hard to read. Make your properties dead simple.
What extra security in your code does it really give you if you can decrypt properties by just calling a method anyway?
If you still need to have a Decrypt() would suggest that you create methods for decrypting that return the decrypted value like GetUnencryptedCode() etc, it could just as well be an extension method but still not a readable property.
You should also be looking into using SecureString depending on your use case.
Related
So, I want to save object and then load it and take data from it. I made an class called SaveData, in there I have field isVibrationOn.
Working code below:
public class SaveData
{
public bool isVibratonOn;
}
Here is the code for serialization:
public void SaveGame()
{
SaveData saveData = new SaveData();
BinaryFormatter bf = new BinaryFormatter();
FileStream file = File.Create(Path.Combine(Application.persistentDataPath, FILE_NAME));
SaveData(saveData);
bf.Serialize(file, saveData);
file.Close();
}
private void SaveData(SaveData saveData)
{
saveData.isVibrationOn = VibrationController.controller.isVibrationOn;
}
And here is code for load data:
public void LoadGame()
{
if (File.Exists(Path.Combine(Application.persistentDataPath, FILE_NAME)))
{
FileStream file = File.Open(Path.Combine(Application.persistentDataPath, FILE_NAME), FileMode.Open);
if (file.Length > 0)
{
SaveData saveData = new SaveData();
BinaryFormatter bf = new BinaryFormatter();
saveData = (SaveData)bf.Deserialize(file);
LoadData(saveData);
file.Close();
}
}
}
public LoadData(SaveData saveData)
{
VibrationController.controller.isVibrationOn = saveData.isVibrationOn;
}
My question here is, when I decide to add or remove some fields (lists etc) to SaveData object, my LoadData would look different, but object that is saved on device whould have different fields as well. Simple expample
public LoadData(SaveData saveData)
{
VibrationController.controller.isVibrationOn = saveData.isVibrationOn;
//old save data doesn't have isMusicOn field
//LoadData method is different because I added new field on SaveData object after I saved file.
//This is simple example, but also it could be any changes like list of objects with an object that has different fields added/changed.
MusicController.controller.isMusicOn = saveData.isMusicOn;
}
How would I check if old instance has that field?
As a general recommendation, do not use binaryFormatter. It is slow, inefficient, unsafe and has poor backwardscompatibility.
So if you change the class I would not expect it to be possible to de serialize older data at all, let alone tell you what fields where missing. Switching .net versions can also be an issue with binaryformatter.
There are much better serialization libraries out there. Json.net is the standard for text-based serialization, and I have used protobuf.net for binary serialization. But there are many other libraries that can be used.
To handle missing or optional fields you would typically have some default value, like null, that you can check. It should also be possible to initialize the fields to some other default value if desired.
I would recommend separating your serialization objects from your domain objects, since serialization frameworks may require parameter less constructors or public setters. And separate serialization objects provide a chance to manage differences in object structures between versions.
If your goal is to make sure your code doesn't break because of "missing" field:
If you already have the "old" version rolled out, that's tough - you will have to implement some kind of "migration" from old data to new, probably by keeping the old class as-is, implementing your changes in a new class (possibly derived from old, to keep code duplication to a minimum), and then checking if the data you are deserializing is old (I'll refer to it as "MyClass_Old") or new ("MyClass_New"). If you are able to determine that from some metadata, file attributes or such - great. If not, you could just deserialize it as MyClass_New on purpose and wrap in in try-catch. If you caught a SerializationException, then it's probably MyClass_Old, and then you deserialize it as MyClass_Old and then use it to construct a new MyClass_New instance.
However, if you didn't yet roll out those changes, you can make use of version-tolerant serialization. You can use attributes like [OptionalFieldAttribute] to mark fields that might be missing in different version, [OnDeserializingAttribute] (that goes on method that will be called before deserializing - to maybe set some values in those missing fields), and [OnDeserializedAttribute] to "fix" or validate your deserialized object.
So for your example, knowing that you added the isMusicOn field, you'd mark it as optional (because it might be missing in deserialized data):
[OptionalField(VersionAdded = 2)]
bool isMusicOn;
and then set it to some kind of default value if it's missing. Let's say you want it to be on by default:
[OnDeserializing]
internal void OnDeserializingMethod(StreamingContext context)
{
isMusicOn = true;
}
Note that here you use the "deserializing" attribute, because you do want to keep the value if it is not missing. Since OnDeserializingMethod is called before deserializing, it will be overwritten by deserialized value if it is present. This specific case (you have a field that might be missing, and you want it to have some specific value) is also covered here.
If your goal is to check if this specific field was missing in deserialized object:
With some preparation this same principle (version-tolerant serialization) can also be used to specifically determine if the field was present in deserialized object. You can set it to some predetermined value (one that would not be allowed when serializing) before deserializing (using [OnDeserializing]). Then after deserialization check if that value is still there or it was replaced by something reasonable.
So in your case let's say that allowed values for isMusicOn are true and false, then you can make your isMusicOn a nullable bool:
[OptionalField(VersionAdded = 2)]
bool? isMusicOn;
Then you'll make sure that serialized object cannot have isMusicOn set to null, by using [OnSerializing]:
[OnSerializing()]
internal void OnSerializingMethod(StreamingContext context)
{
if (isMusicOn == null)
isMusicOn = false;
}
Then assign null in a method marked with [OnDeserializing]:
[OnDeserializing]
internal void OnDeserializingMethod(StreamingContext context)
{
isMusicOn = null;
}
and then in [OnDeserialized] check if it is still null (if it was deserialized, it will have changed to proper value like true or false, if not it'll stay null):
[OnDeserialized]
internal void OnDeserializedMethod(StreamingContext context)
{
Console.WriteLine($"isMusicOn {isMusicOn == null ? "wasn't" : "was" } present in deserialized object!");
if (isMusicOn == null)
isMusicOn = false;//set it to some "proper" default value;
}
In this example, actually, nullable value wouldn't even require the setup (it'll be null by default even without the OnDeserializing part), but i'll leave it in as an example.
All this last part is probably more trouble then it's worth. Besides all the hassle with attributes, you have to change your field type to allow for that "super special value", and that might require you to change a lot of other code that depends on it. I suspect the real question is not "how to check if the field was there", but rather "how to make sure my app doesn't break because it wasn't there", and for that you don't need to know if your field was deserialized - just to make sure that it has some reasonable value in it after deserialization.
I'm trying to develop a CodeFluent aspect to set a property of a entity to be a full-text index.
I've found this link, which does something similar to what I'm aiming for.
http://blog.codefluententities.com/2012/11/27/using-the-sql-server-template-producer-to-generate-clustered-indexes/
However this uses a SQL template producer. Are there anyway to set a property to be a full-text index entirely in the aspect itself, so I don't have to install/maintain both template producer and aspect for all projects?
Here's the C# aspect code I have so far:
public class FullTextIndexing : IProjectTemplate
{
public static readonly XmlDocument Descriptor;
public const string Namespace = "http://www.softfluent.com/aspects/samples/FullTextIndexing";
static FullTextIndexing()
{
Descriptor = new XmlDocument();
Descriptor.LoadXml(
#"<cf:project xmlns:cf='http://www.softfluent.com/codefluent/2005/1' defaultNamespace='FullTextIndexing'>
<cf:pattern name='Full Text Indexing' namespaceUri='" + Namespace + #"' preferredPrefix='fti' step='Tables'>
<cf:message class='_doc'>CodeFluent Full Text Indexing Aspect</cf:message>
<cf:descriptor name='fullTextIndexing'
typeName='boolean'
category='Full Text Indexing'
targets='Property'
defaultValue='false'
displayName='Full-Text Index'
description='Determines if property should be full text indexed.' />
</cf:pattern>
</cf:project>");
}
public Project Project { get; set; }
public XmlDocument Run(IDictionary context)
{
if (context == null || !context.Contains("Project"))
{
// we are probably called for meta data inspection, so we send back the descriptor xml
return Descriptor;
}
// the dictionary contains at least these two entries
Project = (Project)context["Project"];
// the dictionary contains at least these two entries
XmlElement element = (XmlElement)context["Element"];
Project project = (Project)context["Project"];
foreach (Entity entity in project.Entities)
{
Console.WriteLine(">>PROPERTY LOGGING FOR ENTITY "+entity.Name.ToUpper()+":<<");
foreach (Property property in entity.Properties)
{
Log(property);
if(MustFullTextIndex(property))
{
Console.WriteLine("CHANGING PROPERTY");
property.TypeName = "bool";
Log(property);
}
}
}
// we have no specific Xml to send back, but aspect description
return Descriptor;
}
private static bool MustFullTextIndex(Property property)
{
return property != null && property.IsPersistent && property.GetAttributeValue("fullTextIndexing", Namespace, false);
}
private static void Log(Property property)
{
Console.WriteLine(property.Trace());
}
}
EDIT ONE:
Following Meziantou's answer, I'm trying to create a template producer, but it's giving me compilation errors when I try to add the new template producer to the project producers list, so I'm probably doing it wrong.
The error says:
Cannot convert type 'CodeFluent.Model.Producer' to 'CodeFluent.Producers.SqlServer.TemplateProducer'
Here's the code I have thus far:
public XmlDocument Run(IDictionary context)
{
if (context == null || !context.Contains("Project"))
{
// we are probably called for meta data inspection, so we send back the descriptor xml
return Descriptor;
}
// the dictionary contains at least these two entries
XmlElement element = (XmlElement)context["Element"];
Project project = (Project)context["Project"];
CodeFluent.Producers.SqlServer.TemplateProducer producer = new CodeFluent.Producers.SqlServer.TemplateProducer();
producer.AddNamespace("CodeFluent.Model");
producer.AddNamespace("CodeFluent.Model.Persistence");
producer.AddNamespace("CodeFluent.Producers.SqlServer");
Console.WriteLine(producer.Element);
//TODO: Need to figure out how to modify the actual template's contents
project.Producers.Add(producer); //Error happens here
// we have no specific Xml to send back, but aspect description
return Descriptor;
}
In the sample code, the aspect is used only because it has a descriptor. Descriptors are used by CodeFluent Entities to populate the property grid:
<cf:descriptor name="IsClusteredIndex" typeName="boolean" targets="Property" defaultValue="false" displayName="IsClusteredIndex" />
So when you set the value of this property to true or false, the xml attribute ns:IsClusteredIndex is added or removed from the xml file.
Then the SQL Template reads the value of the attribute to generate the expected SQL file:
property.GetAttributeValue("sa:IsClusteredIndex", false)
So the aspect is not mandatory, but provides a graphical interface friendly way to add/remove the attribute. If you don't need to integrate into the graphical interface, you can safely remove the aspect.
If your goal is to integrate into the graphical interface, you need an aspect (XML or DLL) or a producer. If you don't want to create a producer, you can embed the template into your aspect. During the build, you can extract the SQL template and add the SQL Template producer to the project, this way everything is located in the aspect.
I am building a C#/.NET 4.5 client for a REST API using JSON.NET. The API supports partial updates; therefore the presence or lack of an attribute in the json on an update has meaning. If the attribute is in the json, the server will set the value accordingly; the the attribute is not passed the server will not update it. This also applies to null values. I have .NET classes for each model; with properties for each JSON attribute (pretty standard).
As an example lets say I have this account object (name, notes) that already exists on the server:
{
'name':'craig',
'notes:'these are notes'
}
If I pass in this json for an update it will update the name, but will leave the notes set to 'these are notes':
var account = api.GetAccount();
account.Name = "bob";
api.UpdateAccount(account);
{
'name':'bob'
}
If I pass this json in for an update, it will set the name and the notes to null on the server:
var account = api.GetAccount();
account.Name = "bob";
account.Notes = null;
api.UpdateAccount(account);
{
'name':'bob',
'notes':null
}
All good up to this point.
My question is how to you get JSON.NET to play along nicely with this. JSON.NET allows control the NullValueHandling which basically says if null values should be serialized or not. However that is not enough in this case. I need to be able to determine if the calling code explicitly set the value to null. Is there a recommended way to handle this?
Ive tried using a Dictionary internal to my models to store the attributes to be serialized via JSON. This allows me to tell if the attribute has been set to anything (including null) via the presence of the key in the dictionary. I found that this approach has some difficulties and I end up rewriting a lot of code that comes standard to JSON.NET (type serialization, generics, nullables, enums...).
Note: I do realize the above example is a bit contrived. In reality the account object returned back from the server would have both name and notes populated, and that when the update happened it would send both back.
The other case where this applies is during creating objects and handling server generated default. For example, lets say the server defaults the account's notes to 'put notes here' when the account is created. If I pass in the Notes attribute with a null value, the server will think the client wants to set it to null. The reality though is the client is not trying to set the Notes to null, and in this case would want the default to be set.
var account = new Account();
account.Name = "bob";
api.CreateAccount(account);
{
'name':'bob',
'notes':null
}
Im always impressed by JSON.NET...
Here is what I ended up with. I used a combination of a ContractResolver, the ShouldSerialize predicate and the NullValueHandling property. This link was very useful. The properties are stored in a Dictionary in a base class ApiModel; that code is straightforward.
Account Model
[JsonProperty("name")]
public string Name
{
get { return this.GetAttributeValue<string>("name"); }
set { this.SetAttributeValue<string>("name", value); }
}
Json Serialization
ApiModel.JsonSerializerSettings = new Newtonsoft.Json.JsonSerializerSettings();
ApiModel.JsonSerializerSettings.NullValueHandling = Newtonsoft.Json.NullValueHandling.Include;
ApiModel.JsonSerializerSettings.ContractResolver = ApiModel.JsonContractResolver;
internal class ApiModelContractResolver : Newtonsoft.Json.Serialization.DefaultContractResolver
{
protected override Newtonsoft.Json.Serialization.JsonProperty CreateProperty(System.Reflection.MemberInfo member, Newtonsoft.Json.MemberSerialization memberSerialization)
{
var property = base.CreateProperty(member, memberSerialization);
property.ShouldSerialize =
instance =>
{
var apiModel = instance as ApiModel;
var hasAttribute = apiModel.HasAttribute(property.PropertyName);
property.NullValueHandling = hasAttribute ? Newtonsoft.Json.NullValueHandling.Include : Newtonsoft.Json.NullValueHandling.Ignore;
return hasAttribute;
};
return property;
}
}
I'm coming from a SQL Server background, and experimenting with Redis in .NET using ServiceStack. I don't mean for Redis to be a full replacement for SQL Server, but I just wanted to get a basic idea of how to use it so I could see where we might make good use of it.
I'm struggling with what I think is a pretty basic issue. We have a list of items that are maintained in a couple of different data stores. For the sake of simplicity, assume the definition of the item is basic: an integer id and a string name. I'm trying to do the following:
Store an item
Retrieve an item if we only know its id
Overwrite an existing item if we only know its id
Show all the items for that specific type
And here's some of the code I've put together:
public class DocumentRepositoryRedis
{
private static string DOCUMENT_ID_KEY_BASE = "document::id::";
public IQueryable<Document> GetAllDocuments()
{
IEnumerable<Document> documentsFromRedis;
using (var documents = new RedisClient("localhost").As<Document>())
{
documentsFromRedis = documents.GetAll();
}
return documentsFromRedis.AsQueryable();
}
public Document GetDocument(int id)
{
Document document = null;
using (var redisDocuments = new RedisClient("localhost").As<Document>())
{
var documentKey = GetKeyByID(document.ID);
if (documentKey != null)
document = redisDocuments.GetValue(documentKey);
}
return document;
}
public void SaveDocument(Document document)
{
using (var redisDocuments = new RedisClient("localhost").As<Document>())
{
var documentKey = GetKeyByID(document.ID);
redisDocuments.SetEntry(documentKey, document);
}
}
private string GetKeyByID(int id)
{
return DOCUMENT_ID_KEY_BASE + id.ToString();
}
}
It all seems to work - except for GetAllDocuments. That's returning 0 documents, regardless of how many documents I have stored. What am I doing wrong?
The typed Redis client also gives you access to the non-typed methods - since Redis ultimately doesn't know or care about your object types. So when you use the client.SetEntry() method, it bypasses some of the typed client's features and just stores the object by a key. You'll want to use the client.Store method since it goes ahead and creates a SET in Redis with all the object IDs related to your type. This SET is important because it's what the GetAll method relies on to serve back all the objects to you. The client.Store method does infer the ID automatically so you'll want to play around with it.
You'd change your GetDocument(int id) and SaveDocument(Document document) methods to use the client.GetById(string id) method, and you'd use client.Store(T value) method. You won't need your GetKeyByID() method anymore. I believe your Document object will need an "Id" property for the typed client to infer your object ID.
I've created a partial class to extend the default spmetal class to handle publishing html fields. As outlined here:
Extending the Object-Relational Mapping
Snippet from public partial class RelatedLinksItem : Item, ICustomMapping:
/// <summary>
/// Read only data is retrieved in this method for each extended SPMetal field
/// Used to Read - CRUD operation performed by SPMetal
/// </summary>
/// <param name="listItem"></param>
[CustomMapping(Columns = new string[] { CONTENT_FIELDtesthtml, CONTENT_FIELDLink })]
public void MapFrom(object listItem)
{
SPListItem item = (SPListItem)listItem;
// link
this.ContentLink = item[CONTENT_FIELDLink] as LinkFieldValue;
// html (does NOT work)
HtmlField html = item[CONTENT_FIELDtesthtml] as HtmlField; // this returns null
// html (does work)
HtmlField html2 = (HtmlField)item.Fields.GetFieldByInternalName(CONTENT_FIELDtesthtml); // this returns object
this.Contenttesthtml = html2;
this.TestHtml = html2.GetFieldValueAsText(item[CONTENT_FIELDtesthtml]); // set property for rendering html
}
Snippet from "webpart":
protected override void CreateChildControls()
{
using (OrganisationalPoliciesDataContext context = new OrganisationalPoliciesDataContext(SPContext.Current.Web.Url))
{
var results = from links in context.RelatedLinks
select links;
foreach (var link in results)
{
// render link
Controls.Add(new LiteralControl(string.Format("<p>Link: {0}</p>", link.ContentLink)));
// render html
Controls.Add(new LiteralControl(string.Format("<p>HTML: {0}</p>", link.TestHtml)));
}
}
}
Two questions:
Why does HtmlField html = item[CONTENT_FIELDtesthtml] as HtmlField; return null, but the item.Fields.GetFieldByInternalName works correctly?
Is there a way to use the GetFieldValueAsText method from within
the webpart or is the approach of storing the value in a custom
property for accessing later acceptable?
You are casting the field value of item[CONTENT_FIELDtesthtml] to the type HtmlField. But HtmlField represents the type of the field and not the type of the field value. Thus HtmlField html will be assigned with null. Check this MSDN page for a reference of all publishing field types and value types.
I am not sure what the field value type of a HtmlField is. Probably just string.
So you should be safe to convert it to string:
string html = Convert.ToString(item[CONTENT_FIELDtesthtml]);
I think storing the value in a property is the way to go. This way you achieve a separation of data layer and presentation layer.