So I'm creating kubernetes cluster with dotnet aws cdk.
Now I want to automatic create some kubernetes manifest in the cluster.
Based on AWS doc, manifest object should be created with Dictionary<string,Object>
However some yaml manifest contain array of object value, which contain a duplicate key.
Example:
AllowedRegions:
- key: key1
value: val1
- key: key2
value: val2
How to represent above dictionary object?
I have try to use plain string but the object is not in array format which causing deployment error.
Aws doc reference:
https://docs.aws.amazon.com/cdk/api/v2/dotnet/api/Amazon.CDK.AWS.EKS.KubernetesManifest.html
Finally got it resolved. Using
Dictionary<string,object>[]
"AllowedRegions", new Dictionary<string, Object>[]
{
new()
{
{ "key", "key1" },
{ "value", "value1" }
},
new()
{
{ "key", "key2" },
{ "value", "value2" }
}
}
Related
This is I guess a bit more sophisticated. I have this model successfully created with EF Core 7 (but I guess it's for all Core versions same behavior) by a code model.
The culprit is the entity SubJourney that appears as child of TrustFrameworkPolicies and as a child of Candidates. I can create the model and the database looks fine in regard to the created schema.
I need to add the data in one step, because in real life it's one single XML import.
However, adding entities has limitations. Let's assume this code to add the data in one step:
var guid = Guid.NewGuid();
var sj1 = new SubJourney {
DbKey = guid,
Type = SubJourneyTYPE.Transfer
};
var sj2 = new SubJourney {
DbKey = guid,
Type = SubJourneyTYPE.Transfer
};
var trustFrameworkPolicy = new TrustFrameworkPolicy {
UserJourneys = new List<UserJourney> {
new UserJourney {
Id = "Journey1",
OrchestrationSteps = new List<OrchestrationStepUserJourney> {
new OrchestrationStepUserJourney {
JourneyList = new List<Candidate> {
new Candidate {
SubJourneyReferenceId = "Test",
SubJourney = sj1
}
}
}
}
}
},
SubJourneys = new List<SubJourney> {
sj2
}
};
context.Set<TrustFrameworkPolicy>().Add(trustFrameworkPolicy);
context.SaveChanges();
As you can see, the objects sj1 and sj2 are identical. That's how they appear in the XML import. However, from perspective of the database it's the same (I want to treat it as the same, actually).
To get it working I just need to use the same object, like so:
var sj = new SubJourney
{
Type = SubJourneyTYPE.Transfer
};
If I reference in both positions just this sj EF Ccore treats it as one. However, because the object is created by a serializer (and it contains hundreds of entities, then), this is not feasible.
The Errors
If I enforce the same primary key for both I get this:
System.InvalidOperationException: The instance of entity type 'SubJourney' cannot be tracked because another instance with the key value '{DbKey: d44948dc-d514-4928-abea-3450150c26c4}' is already being tracked. When attaching existing entities, ensure that only one entity instance with a given key value is attached.
I read: The primary key must not be the same to have the entity twice.
If I do not enforce the same primary key I get this:
The INSERT statement conflicted with the FOREIGN KEY constraint "FK_Candidates_SubJourneys_SubJourneyDbKey"
I read: The primary key should be same to fulfil the constraint.
In the debugger the graph shows properly with the object inserted twice:
The Question
How does EF Core recognizes the object as "same"? The two errors are mutually exclusive. What I want is to add the entity twice (enforced by serializer) and still treat the entity as one (enforced by my schema).
What I tried
I read https://learn.microsoft.com/en-us/ef/core/change-tracking/identity-resolution
The text suggests to use ReferenceLoopHandling, but XmlSerializer doesn't know such an option. I tried to serialize the graph as JSON and deserialized the JSON back using the suggested options, but NewtonsoftJson doesn't see this as loop, because the objects reference each other indirectly. Finally, same error.
Setting its primary key doesn't work as shown. Overriding GetHashcode/Equals doesn't work either.
I have also tried to manipulate the ChangeTracker:
context.ChangeTracker.TrackGraph(trustFrameworkPolicy, node =>
{
//Console.WriteLine($"***** Tracking {node.Entry.Entity.GetType().Name}");
if (node.Entry.Entity is SubJourney subJourney)
{
Console.WriteLine("*** Recognized Subjourney ***");
var keyValue = node.Entry.Property(nameof(SubJourney.Id)).CurrentValue;
// Key is another property I know is set and unique (not the PK)
var entityType = node.Entry.Metadata;
var existingEntity = node.Entry.Context.ChangeTracker.Entries()
.FirstOrDefault(
e => Equals(e.Metadata, entityType)
&& Equals(e.Property(nameof(SubJourney.Id)).CurrentValue, keyValue));
if (existingEntity == null)
{
node.Entry.State = EntityState.Added;
} else
{
// Just ignore (in debugger I see the state is in fact "Detached")
}
} else {
node.Entry.State = EntityState.Added;
}
});
Still same error (foreign key constraint issue).
Now I run a bit out options. Any pointer how to deal with this would be appreciated.
As a playground I created a simple demo project (console app) with code (and all tries) with SqlLocalDb reference for use with VS 2022:
https://github.com/joergkrause/StackoverflowEFCoreIssue
Thanks for reading through this post :-)
EF works with tracked references. When you have two untracked classes, whether the values are identical or different, they are treated as 2 distinct records. When you associate them to new parent records, EF will attempt to insert them as brand new rows, resulting in either duplicate data (if the PKs are overwritten by Identity columns) or you get exceptions like "A entity with the same Id is already tracked" or unique constraint violations when EF attempts to insert a duplicated row.
When performing operations with imported/transformed data, you need to take care to account for data records that might exist, or at minimum, references that the DbContext may already be tracking. This means that given a set of DTOs you cannot simply map them into a set of Entities and then Add/Update them in the DbContext, especially as even with new top-level entities these will often reference existing records especially in the case of many-to-one relationships.
Take for example I have a list of Orders which contain a Customer reference. I might have 3 orders, two associated with Customer ID 1, one with Customer ID 2. In the serialized data I might get something like:
orders [
{
Number: "10123"
Customer:
{
Id: 1,
Name: "Han Solo"
}
},
{
Number: "10124"
Customer:
{
Id: 1,
Name: "Han Solo"
}
},
{
Number: "10125"
Customer:
{
Id: 2,
Name: "Luke Skywalker"
}
}]
The orders might be expected to be uniquely new though anything unique like Order Number should be verified before inserting, however the Customer might be new, or someone that already exists.
If we use Automapper or such to create Order and Customer entities we would get 3 distinct references for the Customers, even though two of the records reference the same customer. Instead we should be explicit about the entities we actually know we want to insert vs. any relations we should check and use:
foreach(var orderDto in orderDtos)
{
// Handle situation where a duplicate record might already exist.
if (_context.Orders.Any(x => x.OrderNumber == orderDto.OrderNumber))
throw new InvalidOperation("Order already exists");
var order = Mapper.Map<Order>(orderDto);
var customer = _context.Customers.SingleOrDefault(x => x.Id == orderDto.Customer.Id);
if (customer != null)
order.Customer = customer;
_context.Orders.Add(order);
}
This assumes that Automapper would create a Customer when mapping an Order. If the Customer is expected to exist, then I would use Single rather than SingleOrDefault and the call would throw if given a Customer ID that doesn't exist. Beyond this you would also want to consider how to scope when work is committed to the DB, whether each order insert is a Unit of Work or the whole batch. Any existing references need to be resolved and overwrite any created entities. The DbContext will check it's local tracking cache first then look to the DB if necessary but it's the best way to guarantee existing records are referenced to avoid duplicate data or exceptions.
As stated in the post I can't control the graph due to the serializer used. However, JSON serializer is more powerful. The links and answers were helpful for further research. I found that a ReferenceResolver shall work it out. In relation to the code in question I got this:
internal class SubJourneyResolver : IReferenceResolver
{
private readonly IDictionary<string, SubJourney> _sjCache = new Dictionary<string, SubJourney>();
public void AddReference(object context, string reference, object value)
{
if (value is SubJourney sj)
{
var id = reference;
if (!_sjCache.ContainsKey(id))
{
_sjCache.Add(id, sj);
}
}
}
public string GetReference(object context, object value)
{
if (value is SubJourney sj)
{
_sjCache[sj.Id] = sj;
return sj.Id;
}
return null;
}
public bool IsReferenced(object context, object value)
{
if (value is SubJourney sj)
{
return _sjCache.ContainsKey(sj.Id);
}
return false;
}
public object ResolveReference(object context, string reference)
{
var id = reference;
_sjCache.TryGetValue(id, out var sj);
return sj;
}
}
In the JSON it add $id properties for read objects and replaces the copied object with $ref property. The graph now looks like this:
{
"$id": null,
"UserJourneys": {
"$id": null,
"$values": [
{
"$id": null,
"Policy": null,
"OrchestrationSteps": {
"$id": null,
"$values": [
{
"$id": null,
"Journey": null,
"JourneyList": {
"$id": null,
"$values": [
{
"$id": null,
"SubJourney": {
"$id": "k1",
"Policy": null,
"Id": "k1",
"Type": 0,
"DbKey": "00000000-0000-0000-0000-000000000000"
},
"SubJourneyReferenceId": "Test",
"DbKey": "00000000-0000-0000-0000-000000000000"
}
]
},
"Type": 0,
"DbKey": "00000000-0000-0000-0000-000000000000"
}
]
},
"Id": "Journey1",
"DbKey": "00000000-0000-0000-0000-000000000000"
}
]
},
"SubJourneys": {
"$id": null,
"$values": [
{
"$ref": "k1"
}
]
},
"DbKey": "00000000-0000-0000-0000-000000000000"
}
I used another unique property (id) that is independent of the primary key. Now I'm going to deserialize the thing back to .NET object graph. Full code here (Newtonsoft.Json needs to be referenced):
var serialized = JsonConvert.SerializeObject(
trustFrameworkPolicy,
new JsonSerializerSettings
{
ReferenceLoopHandling = ReferenceLoopHandling.Ignore,
PreserveReferencesHandling = PreserveReferencesHandling.All,
TypeNameHandling = TypeNameHandling.Auto,
ReferenceResolver = new SubJourneyResolver() // solution!
});
var deserialized = JsonConvert.DeserializeObject<TrustFrameworkPolicy>(serialized, new JsonSerializerSettings
{
ReferenceLoopHandling = ReferenceLoopHandling.Ignore,
PreserveReferencesHandling = PreserveReferencesHandling.All,
TypeNameHandling = TypeNameHandling.Auto
});
The deserialized object is now added to EF context and saved properly.
I still dislike the idea of an additional serialization step for the sole purpose of object handling. For big graphs it's a lot memory consumption. However, I see the advantage of extreme control the JSON serializer provides.
Maybe in the future EF Core provides a similar way to add a "ReferenceResolver" natively to smooth a complex graph.
I have the following structure of additional information where I need to update the value of one of the tokens in the structure. The data is an array of JTokens with a parent called 'additionalFields' as follows:
{{"additionalFields":
[
{ "name": "NAME1", "value": "VALUE1" },
{ "name": "NAME2", "value": "VALUE2" },
{ "name": "NAME3", "value": "VALUE3" },
{ "name": "NAME4", "value": "VALUE4" }
]}
I'm trying to update the value of one of the tokens e.g. to change VALUE1 to VALUE10.
Once I have located the token I need to update my code removes it as follows.
additionalField.Remove();
I then create a new token to replace the one I have removed (containing the new value) using the following functions.
public static JToken CreateNewToken(string name, string value)
{
var stringToken = CreateNewStringToken(name, value);
var token = JToken.Parse(stringToken);
return (JToken) token;
}
private static string CreateNewStringToken(string name, string value)
{
return $"{{\"name\":\"{name}\",\"value\":\"{value}\"}}";
}
I then add the new token as follows.
additionalFields.AddAfterSelf(updatedToken);
Putting it all together we have the following
foreach (var additionalField in additionalFields)
{
//is this the key we are looking for?
var keyToken = additionalField.First;
if (keyToken?.First == null) continue;
if (string.Equals(keyToken.First.ToString(), "newname", StringComparison.CurrentCultureIgnoreCase))
{
//remove the current token
additionalField.Remove();
//add the updated token
var updatedToken = CreateNewToken("newname", "newvalue");
additionalFields.AddAfterSelf(updatedToken); <-- error occurs here!!
}
}
However after adding the token I'm getting the following error
Newtonsoft.Json.Linq.JProperty cannot have multiple values
I can see in the debugger that the token has been removed (as the token.Count is reduced by 1) so cannot understand why I'm getting an error adding the replacement token.
I was able to reproduce your problem here: https://dotnetfiddle.net/JIVCVB
What is going wrong
additionalFields refers to the JArray of JObjects containing name and value JProperties. You are looping through this JArray to try to find the first JObject having a name property with a certain value, and when you find it you attempt to replace the JObject with a whole new JObject. You successfully remove the old JObject from the JArray, but when you are doing AddAfterSelf to insert the new JObject, you are referencing additionalFields (plural) not additionalField (singular). Recall that additionalFields is the JArray. So you are saying that you want to add the new JObject after the array. The array's parent is a JProperty called additionalFields. A JProperty can only have one value, so AddAfterSelf fails with the error you see.
How to fix your code
I think what you intended to do was additionalField.AddAfterSelf(updatedToken). However, this, too, will fail, for a different reason: you already removed the additionalField from the JArray at that point, so it no longer has a parent context. You would need to AddAfterSelf before you remove the item you are trying to insert after. If you fix that, you still have another problem: your loop doesn't break out after you've done the replacement, so then you will get an error about modifying the collection while looping over it.
Here is the relevant section of code with the corrections:
if (string.Equals(keyToken.First.ToString(), "NAME1", StringComparison.CurrentCultureIgnoreCase))
{
//add the updated token
var updatedToken = CreateNewToken("newname", "newvalue");
additionalField.AddAfterSelf(updatedToken);
//remove the current token
additionalField.Remove();
// we found what we were looking for so no need to continue looping
break;
}
Fiddle: https://dotnetfiddle.net/KcFsZc
A simpler approach
You seem to be jumping through a lot of hoops to accomplish this task. Instead of looping, you can use FirstOrDefault to find the object you are looking for in the array. Once you've found it, you don't need to replace the whole object; you can just update the property values directly.
Here's how:
var rootObject = JToken.Parse(json);
// Get a reference to the array of objects as before
var additionalFields = rootObject["additionalFields"];
// Find the object we need to change in the array
var additionalField = additionalFields.FirstOrDefault(f =>
string.Equals((string)f["name"], "NAME1", StringComparison.CurrentCultureIgnoreCase);
// if the object is found, update its properties
if (additionalField != null)
{
additionalField["name"] = "newname";
additionalField["value"] = "newvalue";
}
Working demo: https://dotnetfiddle.net/ZAKRmi
I wish to write some C# which allows the client to provide a JSON string and query string. The query string would then be used to address values in the JSON object.
For example, if I had this JSON:
{
"head": "big",
"fingers": [
"one", "thumb",
"two", "ring"
],
"arm": {
"elbow", "locked"
}
}
And this query string:
"fingers.two"
I would want to return the value "ring".
Is this (or something like it) possible in C#?
I have tried using the ExpandoObject class, but this does not allow dynamic runtime inspection:
var json = JsonConvert.DeserializeObject<ExpandoObject>(jsonStr);
As far as I can tell, the discovery of values on the json variable needs to be done at code time, rather than runtime, which means I cannot dynamically find values being queried for.
JSONPath does this
Assuming the following JSON (fixed a few syntax errors in the original)
{
"head": "big",
"fingers": {
"one":"thumb",
"two":"ring"
},
"arm": {
"elbow": "locked"
}
}
And this query
MyJObjectOrToken.SelectToken("fingers.two")
You will get the following output:
[
"ring"
]
It should be trivial then to extract the value as a string using JSON.Net methods and return the result to your user.
Support for JSONPath is built into JSON.Net
https://www.newtonsoft.com/json/help/html/SelectToken.htm
I'm trying to update a property of a specific object in an array in my document.
For example:
{
_id: #####
Items: [
{ Key: 1, Value: "Something" },
{ Key: 2, Value: "Foo" },
{ Key: 1, Value: "Bar" },
]
}
I'm using the MongoDB C# 2.0 driver, and this is what I have for my filter (although I'm pretty sure this will match the entire document, not the sub document).
FilterDefinition<GroupDto> filter = Builders<GroupDto>.Filter.Eq(i => i.Id, groupId) &
Builders<GroupDto>.Filter.ElemMatch(i => i.Items, u => u.Key == key);
Effectively, what I'm trying to achieve, is to match the document by Id, then find the object in the Items array where the 'Key' matches, and then update 'Value' property for that specific array object only. So I match Key: 2, I can update the 'Value' field for Key: 2 only, and Key: 1 and Key: 3 remain unchanged.
Is this even possible?
Cheers,
Justin
Actually, after reading the question posted, it's not quite a duplicate. In the example in the other question, the entire sub document is replaced, where as I just wanted to update a single field.
So I found the answer in a Jira ticket for the MongoDB CSHARP driver here: https://jira.mongodb.org/browse/CSHARP-531
You can use -1 in the indexer to specify using the positional operator.
From ticket:
This is only applicable to the new API in 2.0.0. -1 can be used as an indexer or with GetElementAt to indicate to us to use the positional operator. For example:
Builders<Entity>.Update.Set(x => x.MyArray[-1].Value, 10);
// will yield {$set: { "MyArray.$.Value", 10 } }
Cheers,
Justin
I have an object from my elasticsearch resultset;
that I'm iterating true this via this foreach:
foreach (Nest.IHit<dynamic> temp in result.Hits) {
}
one temp it's source looks like this (just rightclicked in visual studio and clicked on "copy value")
{{
"charSet": "UTF-8",
"executionTime": 927,
"parentUrl": "http://wfintranetdvlp.sidmar.be/sdg/",
"#timestamp": "2015-08-05T13:50:40.721Z",
"method": "GET",
"contentLength": 31575,
"mimeType": "text/html",
"text": "You are here: Home Productie Productie Productie Sidgal Sidgal 1 Campagneplan Dagverslag PamBrowser Sidgal 2 Campagneplan Dagverslag PamBrowser Sidgal 3 Campagneplan Dagverslag PamBrowser Alle Lijnen Stilstanden Productierapporten Autonoom Onderhoud JAP AO-zones Uitgevoerde AO-activiteit afgelopen jaar Kalender audits AO",
"title": "Productie",
"url": "http://wfintranetdvlp.sidmar.be/sdg/productie-2/",
"httpStatusCode": 200
}}
now in my code I can access the params like following temp.Source.title or temp.Source.url but when I want to access the #timestamp it returns null
any idea on how I can access the timestamp?
C# identifiers can not start with #. You're actually trying to access timestamp - the # is called the verbatim specifier, and it allows you to use keywords as identifiers, e.g. you can have a local named #this, which is actually the this identifier.
The only way would be to access the variable by name, something like yourvar["#timestamp"].
I deleted my original answer as I found this SO answer after trying something in my code for you.
The relevant code from the answer is this:
static object GetDynamicMember(object obj, string memberName)
{
var binder = Binder.GetMember(CSharpBinderFlags.None, memberName, obj.GetType(),
new[] { CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.None, null) });
var callsite = CallSite<Func<CallSite, object, object>>.Create(binder);
return callsite.Target(callsite, obj);
}
It uses reflection to build up the call to get the value back, and the "#timestamp" can easily be passed in as a string.