dynamic training/test classes with ML.NET - c#

This is a follow up from the question here
Dynamic classes/objects ML.net's PredictionMoadel<TInput, TOutput> Train()
My system cannot use a predefined class at compile time, therefore I tried to feed a dynamic class into ML.NET like below
// field data type
public class Field
{
public string FieldName { get; set; }
public Type FieldType { get; set; }
}
// dynamic class helper
public class DynamicClass : DynamicObject
{
private readonly Dictionary<string, KeyValuePair<Type, object>> _fields;
public DynamicClass(List<Field> fields)
{
_fields = new Dictionary<string, KeyValuePair<Type, object>>();
fields.ForEach(x => _fields.Add(x.FieldName,
new KeyValuePair<Type, object>(x.FieldType, null)));
}
public override bool TrySetMember(SetMemberBinder binder, object value)
{
if (_fields.ContainsKey(binder.Name))
{
var type = _fields[binder.Name].Key;
if (value.GetType() == type)
{
_fields[binder.Name] = new KeyValuePair<Type, object>(type, value);
return true;
}
else throw new Exception("Value " + value + " is not of type " + type.Name);
}
return false;
}
public override bool TryGetMember(GetMemberBinder binder, out object result)
{
result = _fields[binder.Name].Value;
return true;
}
}
private static void Main(string[] args)
{
var fields = new List<Field>
{
new Field {FieldName = "Name", FieldType = typeof(string)},
new Field {FieldName = "Income", FieldType = typeof(float)}
};
dynamic obj1 = new DynamicClass(fields);
obj1.Name = "John";
obj1.Income = 100f;
dynamic obj2 = new DynamicClass(fields);
obj2.Name = "Alice";
obj2.Income = 200f;
var trainingData = new List<dynamic> {obj1, obj2};
var env = new LocalEnvironment();
var schemaDef = SchemaDefinition.Create(typeof(DynamicClass));
schemaDef.Add(new SchemaDefinition.Column(null, "Name", TextType.Instance));
schemaDef.Add(new SchemaDefinition.Column(null, "Income", NumberType.R4));
var trainDataView = env.CreateStreamingDataView(trainingData, schemaDef);
var pipeline = new CategoricalEstimator(env, "Name")
.Append(new ConcatEstimator(env, "Features", "Name"))
.Append(new FastTreeRegressionTrainer(env, "Income", "Features"));
var model = pipeline.Fit(trainDataView);
}
and got the error: "'No field or property with name 'Name' found in type 'System.Object'". I tried generating the class using Reflection only to run into the same problem.
Is there a workaround? Thanks

For those attempting to do this, I have a working solution that creates the schema and can be used to train data dynamically.
First, grab the code for DynamicTypeProperty and DynamicType from my other answer here.
The following code will create a schema dynamically:
var properties = new List<DynamicTypeProperty>()
{
new DynamicTypeProperty("SepalLength", typeof(float)),
new DynamicTypeProperty("SepalWidth", typeof(float)),
new DynamicTypeProperty("PetalLength", typeof(float)),
new DynamicTypeProperty("PetalWidth", typeof(float)),
};
// create the new type
var dynamicType = DynamicType.CreateDynamicType(properties);
var schema = SchemaDefinition.Create(dynamicType);
You'll then need to create list with the required data. This is done as follows:
var dynamicList = DynamicType.CreateDynamicList(dynamicType);
// get an action that will add to the list
var addAction = DynamicType.GetAddAction(dynamicList);
// call the action, with an object[] containing parameters in exact order added
addAction.Invoke(new object[] {1.1, 2.2, 3.3, 4.4});
// call add action again for each row.
Then you'll need to create an IDataView with the data, this requires using reflection, or the trainers won't infer the correct type.
var mlContext = new MLContext();
var dataType = mlContext.Data.GetType();
var loadMethodGeneric = dataType.GetMethods().First(method => method.Name =="LoadFromEnumerable" && method.IsGenericMethod);
var loadMethod = loadMethodGeneric.MakeGenericMethod(dynamicType);
var trainData = (IDataView) loadMethod.Invoke(mlContext.Data, new[] {dynamicList, schema});
You then, should be able to run the trainData through your pipeline.
Good luck.

Dynamic class doesn't actually create a class definition but it rather provides you with dynamic object.
I looked at the code for SchemaDefinition.Create() it needs an actual class definition to build the schema. So your options are to create and load a class definition dynamically.
You can create your class as string with all dynamic properties and compile it using Microsoft compiler services aka Roslyn. See here. This will generate an assembly (in memory as memory stream or on file system) with your dynamic type.
Now you are only half way there. To get your dynamic type from dynamic assembly you need to load it in the App Domain. See this post.
Once the assembly is loaded you can use 'Activator.CreateInstance()' if it's same domain or if it's your custom domain then you would need yourDomain.CreateInstanceAndUnwrap() to create the object out of dynamically generated Class and to get the type use Assembly.GetType().
Few sample here, A little out of date but will get you on your feet if you are up for this. See CompilerEngine and CompilerService to compile and load the assembly.
Other options: Refelection.Emit() but it requires a great deal of IL level coding. See this post.

Right now I'm using a dummy place holder like this as a workaround
public class TrainingSample
{
public string TextField1;
public string TextField2;
public string TextField3;
public string TextField4;
public string TextField5;
public float FloatField1;
public float FloatField2;
public float FloatField3;
public float FloatField4;
public float FloatField5;
public float FloatField6;
public float FloatField7;
public float FloatField8;
public float FloatField9;
public float FloatField10;
public float FloatField11;
public float FloatField12;
public float FloatField13;
public float FloatField14;
public float FloatField15;
}

Related

Map [name,value] string values to class without reflection

I'm having a huge performance issue about mapping string property names and string property values to classes using reflection.
My issue now:
public class Person
{
public string Property1 { get; set; }
public string Property2 { get; set; }
public string Property3 { get; set; }
public string Property4 { get; set; }
// My class has around 100 properties
public string Property100 { get; set; }
}
I am mapping a key value pair collection to the class using reflection
[{"Property1": "some value"}, {"Property2": "something else"},{"Property3","Property4","value" }.....{"Property100","val"}]
It got to the point that I am now mapping around 10 000 class instances using reflection and the performance is to say it lightly bad.
Any ideas for eliminating the reflection would be greatly appreciated.
I see two options, if you need to avoid reflection for tasks like this(when code could be programatically generated).
First is Expressions I use it often, e.g. I saw some people write something like this
public class A
{
public Prop1 ...
....
public Prop100
public override ToString() => $"{nameof(Prop1)}={Prop1};...";
and so for all 100 properties, and always doing this manually.
And with Expression it can be easily automated, you just need to generate Expression for String.Concat and pass list of properties and names there.
For your example, it is not clear what are your data. How do you do lookup in the list?
Let's assume there is a dictionary<string,string>(you can transform your list of tuples to a dictionary), and all properties are strings as well.
Then we would need to generate a list assignment expressions like this
if(data.ContainsKey("Prop1")) result.Prop1 = data["Prop1"];
And the code would be complicated, anyway it would look like this
private static class CompiledDelegate<T>
{
public static Action<T, Dictionary<string, string>> initObject;
static CompiledDelegate()
{
var i = Expression.Parameter(typeof(Dictionary<string, string>), "i");
var v = Expression.Parameter(typeof(T), "v");
var propertyInfos = typeof(T).GetProperties().ToArray();
var t = new Dictionary<string, string>();
var contains = typeof(Dictionary<string, string>).GetMethod(nameof(Dictionary<string, string>.ContainsKey));
var getter = typeof(Dictionary<string, string>).GetProperties().First(x => x.GetIndexParameters().Length > 0);
var result = new List<Expression>();
foreach (var propertyInfo in propertyInfos)
{
var cst = Expression.Constant(propertyInfo.Name);
var assignExpression =
Expression.IfThen(Expression.Call(i, contains, cst),
Expression.Assign(Expression.PropertyOrField(v, propertyInfo.Name), Expression.MakeIndex(i, getter, new[] { cst })));
result.Add(assignExpression);
}
var block = Expression.Block(result);
initObject = Expression.Lambda<Action<T, Dictionary<string, string>>>(block, new ParameterExpression[] { v, i }).Compile();
}
}
It is an example, it would fail if there were non-string properties.
And it could be used like this
static void Main(string[] args)
{
var tst = new Test();
CompiledDelegate<Test>.initObject(tst, new Dictionary<string, string>
{
{ "S3", "Value3" },
{ "S2", "Value2" },
});
CompiledDelegate<Test>.initObject(tst, new Dictionary<string, string>
{
{ "S3", "Value3" },
{ "S1", "Value1" },
});
Console.ReadKey();
}
The second option is, actually, what it should be ideally imlemented like Using source generators I think such things do have to be done just in build time.
There is a lot of articles on msdn, for instance with samples. But it turned out to be not very easy to implement, even just a sample.
I can say, it didn't work for me, while I tried to do it according to samples.
In order to get it work I had to change TargetFramework to netstandard2.0, do something else...
But after all, when build was green, Visual Studio still showed an error.
Ok, it disappeared after VS restart, but still, that doesn't look very usable.
So, this is a generator, that creates a converter for every class with attribute.
It is again a sample, it doesn't check many things.
[Generator]
public class ConverterGenerator : ISourceGenerator
{
private static string mytemplate = #"using System.Collections.Generic;
using {2};
namespace GeneratedConverters
{{
public static class {0}Converter
{{
public static {0} Convert(Dictionary<string, string> data)
{{
var result = new {0}();
{1}
return result;
}}
}}
}}";
public static string GetNamespaceFrom(SyntaxNode s)
{
if (s.Parent is NamespaceDeclarationSyntax namespaceDeclarationSyntax)
{
return namespaceDeclarationSyntax.Name.ToString();
}
if (s.Parent == null)
return "";
return GetNamespaceFrom(s.Parent);
}
public void Execute(GeneratorExecutionContext context)
{
GetMenuComponents(context, context.Compilation);
}
private static void GetMenuComponents(GeneratorExecutionContext context, Compilation compilation)
{
var allNodes = compilation.SyntaxTrees.SelectMany(s => s.GetRoot().DescendantNodes());
var allClasses = allNodes.Where(d => d.IsKind(SyntaxKind.ClassDeclaration)).OfType<ClassDeclarationSyntax>();
var classes = allClasses
.Where(c => c.AttributeLists.SelectMany(a => a.Attributes).Select(a => a.Name).Any(s => s.ToString().Contains("DictionaryConverter")))
.ToImmutableArray();
foreach (var item in classes.Distinct().Take(1))
{
context.AddSource(item.Identifier.Text + "Converter", String.Format(mytemplate, item.Identifier.Text, SourceText.From(GenerateProperties(item)), GetNamespaceFrom(item)));
}
}
private static string GenerateProperties(ClassDeclarationSyntax s)
{
var properties = s.Members.OfType<PropertyDeclarationSyntax>();
return String.Join(Environment.NewLine,
properties.Select(p =>
{
var name = p.Identifier.Text;
return $"if(data.ContainsKey(\"{name}\")) result.{name} = data[\"{name}\"];";
}));
}
public void Initialize(GeneratorInitializationContext context)
{
}
}
and
static void Main(string[] args)
{
var t1 = GeneratedConverters.TestConverter.Convert(new Dictionary<string, string>
{
{ "S3", "Value3" },
{ "S2", "Value2" },
});
}
Best performance without reflection would be manual mapping.
It seems your key/value pair collection is regular JSON. So you could use the JSONTextReader from JSON.NET and read the string. Then manually map the JSON properties to the class properties.
Like so:
JsonTextReader reader = new JsonTextReader(new StringReader(jsonString));
while (reader.Read())
{
if (reader.Value != null)
{
// check reader.Value.ToString() and assign to correct class property
}
}
More info can be found on the JSON.NET website : https://www.newtonsoft.com/json/help/html/ReadingWritingJSON.htm

Converting an anonymous object's properties and values to a Dictionary

I am creating a library for an existing API. I currently have QueryParameter classes for each request class. The QueryParameter classes are simple but they do vary (not all requests take the same query parameters).
Here is an example of a QueryParameter class:
public class ApiRequestAQueryParameters
{
public string Name { get; set; }
public int Start { get; set; }
public int Stop { get; set; }
}
I am interested in a way to convert a class like this into a Dictionary that I can feed to our Web client. I am hoping to have a reusable method like:
private Dictionary<string, string> GenerateQueryParameters(object queryParametersObject)
{
// perform conversion
}
This way I won't have to pull out the QueryParameter properties for each request (there will be dozens of requests)
The reason that I am using QueryParameter classes instead of making QueryParameter a Dictionary property of each API request class is to be developer friendly. I want to make it so that others can build these API requests by looking at the classes.
There are 2 ways: 1) use reflection and 2) serialize to json and back.
Here is the 1st method:
private Dictionary<string, string> GenerateQueryParameters(object queryParametersObject)
{
var res = new Dictionary<string, string>();
var props = queryParametersObject.GetType().GetProperties();
foreach (var prop in props)
{
res[prop.Name] = prop.GetValue(queryParametersObject).ToString();
}
return res;
}
You can do something like this:
private Dictionary<string, string> GenerateQueryParameters(object queryParameters)
{
var startStop = new StartStop() { Start = queryParameters.Start, Stop = queryParameters.Stop};
var result = new Dictionary<string, string>();
result.Add(queryParameters.Name, startStop);
return result;
}
public class StartStop
{
public int Start { get; set; }
public int Stop { get; set; }
}
This may be the perfect case to utilize ExpandoObjects. An ExpandoObject is a dynamic type, whose properties can be created at run time. ExpandoObject implements IDictionary < string, object > so it's easy to convert to a Dictionary < string, object > .
In the example below, an ExpandoObject is created and converted to a Dictionary < string, object > and then converted to a Dictionary < string, string >.
dynamic apiVar = new ExpandoObject();
apiVar.Name = "Test";
apiVar.Start = 1;
apiVar.Stop = 2;
var iDict = (IDictionary<string, object>) apiVar;
/* if you can utilize a Dictionary<string, object> */
var objectDict = iDict.ToDictionary(i => i.Key, i => i.Value);
/* if you need a Dictionary<string, string> */
var stringDict = iDict.ToDictionary( i=>i.Key, i=> i.Value.ToString());
There are also different ways of setting properties on an ExpandoObject. Below is an example of setting a property by a variable name.
dynamic apiVar = new ExpandoObject();
var propertyName = "Name";
apiVar[propertyName] = "Test";
propertyName = "Start";
apiVar[propertyName] = 1;
propertyName = "Stop";
apiVar[propertyName] = 2;
I always reuse the RouteValueDictionary class for this. It has a constructor that accepts any object and the class itself implements IDictionary.
It's available in the System.Web dll
private Dictionary<string, string> GenerateQueryParameters(object queryParametersObject)
{
return new RouteValueDictionary(queryParametersObject).ToDictionary(d => d.Key, d => Convert.ToString(d.Value));
}

C# Pass Dictionary of Class Types Into Generic

I have a dictionary of class types that looks like this:
private static Dictionary<int, Type> GetArrayOfClassInstances()
{
var availability = new LeisureLinkBaseProduct.Availability();
var baseRate = new LeisureLinkBaseProduct.BaseRate();
var stayRestrictions = new LeisureLinkBaseProduct.StayRestrictions();
var checkInInformation = new LeisureLinkBaseProduct.CheckInInformation();
var specials = new LeisureLinkBaseProduct.Specials();
var taxes = new LeisureLinkBaseProduct.Taxes();
var fees = new LeisureLinkBaseProduct.Fees();
var classDictionary = new Dictionary<int, Type>();
classDictionary.Add(1, availability.GetType());
classDictionary.Add(2, baseRate.GetType());
classDictionary.Add(3, stayRestrictions.GetType());
classDictionary.Add(4, checkInInformation.GetType());
classDictionary.Add(5, specials.GetType());
classDictionary.Add(6, taxes.GetType());
classDictionary.Add(7, fees.GetType());
return classDictionary;
}
And I want to pass it into this generic method that looks like this:
var classes = GetArrayOfClassInstances();
foreach (var instance in classes)
{
var request = RequestBuilder.BuildAdditionalDataRequest(link.href);
var response = Api<instance.value>(request.Endpoint);
}
but I get the intellisense error "instance is a variable but is used like a type" how do I pass this class type into the generic then? Is this possible? Thanks!
The following ran for me and is based on this post:
Api Class:
public class Api<T>
{
public Api(Type E){}
}
Create Api Method:
private static void CreateApis()
{
var classes = GetArrayOfClassInstances();
foreach (KeyValuePair<int, Type> instance in classes)
{
Type apiType = typeof(Api<>).MakeGenericType(instance.Value);
var api = Activator.CreateInstance(apiType, new[]{typeof(System.Net.EndPoint)});
}
}
You will need reflection as the type is not known at compile-time:
typeof(Api<>).MakeGenericType(instance.value)
.GetConstructor(new[] {typeof(Endpoint)})
.Invoke(new[]{request.Endpoint});

Dynamic property assigned to object

Is it possible to attach dynamic property to an object of user-defined class?
public class Room
{
public int NumberOfDoors { get; set; }
public int NumberOfWindows { get; set; }
}
then from other context:
Room room = new Room();
dynamic contact = new ExpandoObject();
contact.NumberOfWalls = 4;
and then somehow associate NumberOfWalls with room, as its property?
Update (Larger Picture):
as per #nawfal's suggestion
I have a cached List<Room> being iterated in a razor view (outside the themes folder), calling a
particular partial view (from the current theme) for each element. One of the theme needs an extra
property in Room. I only have access to modify code in that particular theme folder, the partial
views cshtml files (don't ask why).
So its basically:
(psuedocode)
Room.NoOfWalls = SomeHeavyLiftingProcess(Room.NoOfWindows, Room.NoOfDoors)
I am looking for a way o update the List<Room> rooms object with NoOfWalls in
HttpRuntime.Cache["rooms"] to avoid calling SomeHeavyLiftingProcess() with each request. The
goal is to inject a property in cached object. Unfortuntely HttpRuntime.Cache["rooms"] is object
type and doesn't allow me to do this:
HttpRuntime.Cache["rooms"][3]["NoOfWalls"] = SomeHeavyLiftingProcess(..)
So I am thinking, for the first request (when cache is empty or invalid):
Unpackig: Retrieve (List<Room>)HttpRuntime.Cache["room"], inject NoOfWalls in the current room object.
Repacking: Update List<Room> room with the new object and assign it back to HttpRuntime.Cache.
For the subsequent requests, the value of NoOfWalls will come from cached object #Model.NoOfWalls.
You cannot add properties not defined in a class to an existing instance, without using a dynamic object like ExpandoObject.
If you need to add members to an existing class, you can create a child class with a special constructor:
public class SpecialRoom : Room
{
public SpecialRoom() { }
public SpecialRoom(Room copy)
{
this.NumberOfDoors = copy.NumberOfDoors;
this.NumberOfWindows = copy.NumberOfWindows;
}
public int NumberOfJacuzzis { get; set; }
}
Usage:
var room = new Room();
room.NumberOfDoors = 3;
var specialRoom = new SpecialRoom(room)
{
NumberOfJacuzzis = 7
};
Or:
var listOfRooms = new List<Room>();
// ...
var listOfSpecialRooms = listOfRooms.Select(x => new SpecialRoom(x));
listOfSpecialRooms.ForEach(x => x.NumberOfJacuzzis = ComplexCalculation(x));
If you have an existing concrete object (like an instance of the Room class), you can convert it to a dynamic object with a method like this:
public static dynamic ConvertObjectToDynamic(object value)
{
if (value == null)
{
return null;
}
IDictionary<string, object> dynamicObject = new ExpandoObject();
var properties = value.GetType().GetProperties(
BindingFlags.Public | BindingFlags.Instance);
foreach (var propertyInfo in properties)
{
if (propertyInfo.GetIndexParameters().Length == 0)
{
var propertyValue = propertyInfo.GetValue(value);
dynamicObject[propertyInfo.Name] = propertyValue;
}
}
return dynamicObject;
}
Usage:
var room = new Room();
room.NumberOfDoors = 3;
dynamic dynamicObject = ConvertToDynamic(room);
dynamicObject.WhateverYouWant = 7;
Now dynamicObject.NumberOfDoors will be 3, and dynamicObject.WhateverYouWant will be 7.

How to add an attribute to a property at runtime

//Get PropertyDescriptor object for the given property name
var propDesc = TypeDescriptor.GetProperties(typeof(T))[propName];
//Get FillAttributes methodinfo delegate
var methodInfo = propDesc.GetType().GetMethods(BindingFlags.Instance | BindingFlags.Public |
BindingFlags.NonPublic)
.FirstOrDefault(m => m.IsFamily || m.IsPublic && m.Name == "FillAttributes");
//Create Validation attribute
var attribute = new RequiredAttribute();
var attributes= new ValidationAttribute[]{attribute};
//Invoke FillAttribute method
methodInfo.Invoke(propDesc, new object[] { attributes });
Hi I am trying to add Validation attribute at runtime using the above code. However I am getting the below exception:
Collection was of a fixed size
Don't let someone tell you that you can't do it. You can run for president if you want :-)
For your convenience, this is a fully working example
public class SomeAttribute : Attribute
{
public SomeAttribute(string value)
{
this.Value = value;
}
public string Value { get; set; }
}
public class SomeClass
{
public string Value = "Test";
}
[TestMethod]
public void CanAddAttribute()
{
var type = typeof(SomeClass);
var aName = new System.Reflection.AssemblyName("SomeNamespace");
var ab = AppDomain.CurrentDomain.DefineDynamicAssembly(aName, AssemblyBuilderAccess.Run);
var mb = ab.DefineDynamicModule(aName.Name);
var tb = mb.DefineType(type.Name + "Proxy", System.Reflection.TypeAttributes.Public, type);
var attrCtorParams = new Type[] { typeof(string) };
var attrCtorInfo = typeof(SomeAttribute).GetConstructor(attrCtorParams);
var attrBuilder = new CustomAttributeBuilder(attrCtorInfo, new object[] { "Some Value" });
tb.SetCustomAttribute(attrBuilder);
var newType = tb.CreateType();
var instance = (SomeClass)Activator.CreateInstance(newType);
Assert.AreEqual("Test", instance.Value);
var attr = (SomeAttribute)instance.GetType()
.GetCustomAttributes(typeof(SomeAttribute), false)
.SingleOrDefault();
Assert.IsNotNull(attr);
Assert.AreEqual(attr.Value, "Some Value");
}
Sorry, I'm very late to the party. This is for those who might come here later. The top answer is awesome. Recently, a library has been developed, that abstracts away all of that complexity, and gives you something as simple as this:
var attributeType = typeof(CustomAAttribute);
var attributeParams = new object[] { "Jon Snow" };
var typeExtender = new TypeExtender("ClassA");
typeExtender.AddProperty("IsAdded", typeof(bool), attributeType, attributeParams);
To work with. details of how to install and use the library can be found here
Disclaimer: I developed this library and I've been using it for a lot of projects and it works like magic
Use FastDeepCloner which I developed.
public class test{
public string Name{ get; set; }
}
var prop = DeepCloner.GetFastDeepClonerProperties(typeof(test)).First();
prop.Attributes.Add(new JsonIgnoreAttribute());
// now test and se if exist
prop = DeepCloner.GetFastDeepClonerProperties(typeof(test)).First();
bool containAttr = prop.ContainAttribute<JsonIgnoreAttribute>()
// or
JsonIgnoreAttribute myAttr = prop.GetCustomAttribute<JsonIgnoreAttribute>();
It is not working because the FillAttributes method expects a parameter of the IList type and you are passing an array. Below is the implementation of MemberDescriptor.FillAttributes:
protected virtual void FillAttributes(IList attributeList) {
if (originalAttributes != null) {
foreach (Attribute attr in originalAttributes) {
attributeList.Add(attr);
}
}
}
As you can see, FillAttributes just fills the attributeList parameter with all attributes of your property. And to make your code work, change the var attributes= new ValidationAttribute[]{attribute}; line with:
var attributes = new ArrayList { attribute };
This code has nothing with adding attributes to property of type at runtime. This is "adding attribute to the PropertyDescriptor" extracted from type and has no sense unless you are trying to build a type at runtime which is based on an already existing type.
It is not possible to add Attributes in run-time. Attributes are static and cannot be added or removed.
Similar questions:
Can attributes be added dynamically in C#?
Remove C# attribute of a property dynamically

Categories

Resources