I am trying to perform a bulkwrite with c# https://mongodb.github.io/mongo-csharp-driver/2.7/reference/driver/crud/writing/#bulk-writes - I could have a list of documents from 70-80K.
var correlationIdFilter = Builders<BsonDocument>.Filter.AnyIn("CorrelationId", ninKeysRecon);
var missingData = collection.Find(correlationIdFilter).ToList();
Missing Data Data Sample
{
"_id" : ObjectId("61dd323bfe35f25bb2dcde8e"),
"CorrelationId" : "17bd621d-e47f-4ab1-9004-9543294a4549",
"Key" : "123",
"Date" : "2016-06-28T00:00:00",
"CurrentDate" : ISODate("2022-01-11T07:31:07.011+0000"),
"SourceSystem" : "abc",
"SourceEntity" : "source"
},
{
"_id" : ObjectId("61dd323bfe35f25bb2dcd123"),
"CorrelationId" : "18bd621d-e47f-4ab1-9004-9543294a4549",
"Key" : "123232324",
"Date" : "2016-06-28T00:00:00",
"CurrentDate" : ISODate("2022-01-11T07:31:07.011+0000"),
"SourceSystem" : "abc",
"SourceEntity" : "source"
},
.
.
.
{100K Documents}
Then I createoptions and bulkOps
var options = new UpdateOptions { IsUpsert = true };
var bulkOps = new List<WriteModel<BsonDocument>>();
I dont know if I need to loop through missingData to create a new UpdateOneModel() but I am struggling to see where or how to create a filter because when I set _id as a filter, I get the error _id field is immutable. I do not want to overwrite the _id but I simply want to achieve something like this
collection.UpdateMany(correlationIdFilter, missingBson, options);
Or if I have to create a for loop, I have tried:
foreach(var data in missingBson)
{
var upsert = new UpdateOneModel<BsonDocument>(
new BsonDocument("_id", 1),
new BsonDocument("$set", data)) { IsUpsert = true };
bulkOps.Add(upsert);
}
collection.BulkWrite(bulkOps);
I get the error:
WriteErrors: [ { Category : "Uncategorized", Code : 66, Message : "Performing an update on the path '_id' would modify the immutable field '_id'" } ].'
removing - { IsUpsert = true } runs fine but doesn't do any upserting which I need.
Thank you
The first argument passed to UpdateOneModel is the filter. The filter tells MongoDB which document you want to update. You pass new BsonDocument("_id", 1), which tells MongoDB you want to update the document with an _id of 1. _id is a special field in MongoDB documents:
The field name _id is reserved for use as a primary key; its value must be unique in the collection, is immutable, and may be of any type other than an array. If the _id contains subfields, the subfield names cannot begin with a ($) symbol
_id is immutable, meaning it cannot be changed. This is an issue because the data you are passing to $set contains an _id field that is not 1.
Instead, in the filter document you should be passing the _id of the document you are trying to insert.
foreach(var data in missingBson)
{
var upsert = new UpdateOneModel<BsonDocument>(
new BsonDocument("_id", data._id),
new BsonDocument("$set", data)) { IsUpsert = true };
bulkOps.Add(upsert);
}
This way the _id fields match, and you are not attempting to update a document with an _id of 1 for each write operation.
try this, with bulkWriteAsycn func. You can use this func. with a timer
var requests = new List<WriteModel<T>>();
foreach(var data in dataList)
{
requests.Add(
new UpdateOneModel<T>
(
filter,
Builders<T>.Update(x=>x.key, data.key) // custom updates
)
{IsUpsert=true};
)
}
await collection.BulkWriteAsync(requests);
this filter is your custom filter for updated fields.
Needs attention for new record (all fields are possibly)
Filter Ex:
Builders.Filter.Where(item=> item.id == data.id)
see UpdateOneModel.IsUpsert Property
Related
I have a list of BsonDocument:
var list = db.GetCollection<BsonDocument>(collectionName);
var myIds = list.Find(_ => true)
.Project(Builders<BsonDocument>.Projection.Include("_id"))
.ToList();
that contains:
myIds = "{
{ "_id" : "cc9d9282-c9d2-4cba-a776-ffddsds274d5" },
{ "_id" : "2c1ddd82-c9d2-4dda-afr6-d79ff1274d56" },
{ "_id" : "ss969281-c9d2-4cba-a776-d79ffds274d5" }
}"
And want to query like this:
var deleted =list.DeleteMany(Builders<MessageExchange>.Filter.In("_id", myIds));
I also have tried the following:
var filter = new BsonDocument("_id", new BsonDocument("$in", new BsonArray(myIds)));
var deleted = list.DeleteMany(filter);
Returns the attribute DeletedCount = 0
Could somebody point what seems to be wrong about the filter?
You'll have to extract the _id from the BsonDocument like this:
var extractedIds = myIds.Select(x => x["_id"].ToString()).ToList();
After which you can use it in the filter.
list.DeleteMany(Builders<MessageExchange>.Filter.In("_id", extractedIds));
Make sure that the _id part of the filter matches that of the MessageExchange class
Another way to do so is by making it strong typed:
list.DeleteMany(Builders<MessageExchange>.Filter.In(x => x.Id, extractedIds));
This works as well (based on Skami's answer):
var filter = new BsonDocument("_id", new BsonDocument("$in", new BsonArray(extractedIds)));
list.DeleteMany(filter);
therefore is not tied to the MessageExchange class.
The data:
The collection contains a list of audit records and I want to return the last modified items from the collection.
For example:
So the query needs to return Audit 1235 and 1237 Only.
The following statement works in Mongo Shell and returns the data sub-millisecond, I just need to also figure out how to return the entire Collection item instead of just the Id.
db.Forms.aggregate(
{ $group: { _id: "$Id", lastModifiedId: { $last: "$_id" } } }
)
However, I need to convert this to the C# Driver's syntax.
I have the following at the moment but it's not working and returns (for lack of a better term) weird data (see screencap under the statement).
var results = collection.Aggregate()
.Group(new BsonDocument { { "_id", "$Id" }, { "lastModifiedId", new BsonDocument("$last", "_id") } })
.ToListAsync().Result.ToList();
My current solution gets the full collection back and then runs it through an extension method to get the latest records (where list is the full collection):
var lastModifiedOnlyList =
from listItem in list.OrderByDescending(_ => _.AuditId)
group listItem by listItem.Id into grp
select grp.OrderByDescending(listItem => listItem.AuditId)
.FirstOrDefault();
While this code works, it is EXTREMELY slow because of the sheer amount of data that is being returned from the collection, so I need to do the grouping on the list as part of the collection get/find.
Please let me know if I can provide any additional information.
Update: With Axel's help I managed to get it resolved:
var pipeline = new[] { new BsonDocument { { "$group", new BsonDocument { { "_id", "$Id" }, { "LastAuditId", new BsonDocument { { "$last", "$_id" } } } } } } };
var lastAuditIds = collection.Aggregate<Audit>(pipeline).ToListAsync().Result.ToList().Select(_=>_.LastAuditId);
I moved that to it's own method and then use the IDs to get the collection items back, with my projection working as well:
var forLastAuditIds = ForLastAuditIds(collection);
var limitedList = (
projection != null
? collection.Find(forLastAuditIds & filter, new FindOptions()).Project(projection)
: collection.Find(forLastAuditIds & filter, new FindOptions())
).ToListAsync().Result.ToList();
"filter" in this case is either an Expression or a BsonDocument. The performance is great as well - sub-second for the whole thing. Thanks for the help, Axel!
I think you're doing an extra OrderBy, this should do:
var lastModifiedOnlyList =
from listItem in list
group listItem by listItem.Id into grp
select grp.OrderByDescending(listItem => listItem.AuditId)
.FirstOrDefault();
EDIT:
To gain performance in the query, you could use the Aggregate function differently:
var match = new BsonDocument
{
{
"$group",
new BsonDocument
{
{ "_id", "$Id" },
{ "lastModifiedId", new BsonDocument
{
{
"$last", "$_id"
}
}}
}
}
};
var pipeline = new[] { match };
var result = collection.Aggregate(pipeline);
That should be the equivalent of your Mongo Shell query.
I "accidentally" had some documents stored in RavenDB with all the properties camel-cased because of a setting made with the json serializer ( json.SerializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver();
).
Now I'm storing new documents using Pascal-casing. Can I search for documents ignoring case of the property name?
Eg.
Old document format:
{
name : "foo",
someStuff : "buzz"
}
New document format:
{
Name : "bar",
SomeStuff : "baz"
}
Thanks!
You would need to use the LuceneQuery to manually set the property name for the query.
I have the following document in a mongodb collection
{
"_id" : "52bbd9bef2ba1f37f4f010c1",
"Name" : "Some Name",
"TypeId" : 1
}
I now want to edit this document using the c# driver
BsonDocument doc = BsonDocument.Parse(json);
IMongoQuery query = Query.And(Query.EQ("_id", doc["_id"]));
UpdateBuilder ub = MongoDB.Driver.Builders.Update.Set('Name', 'New Name');
WriteConcernResult updatedBook = _collection.Update(query, ub);
if (!updatedBook.UpdatedExisting)
{
// I always end here
}
It is also weird that the docs tell that _collection.Update(query, ub) should return a BsonDocument while they return a WriteConcernResult. Anyway if I use another property then _id I find the record.
It seems MongoDB is expecting a ObjectId() function wrapper around a ObjectId, isn't it? At least I need this in my tests while using MongoVUE.
Edit
If I try to find a record with this query
{
"_id" : "52bbd9bef2ba1f37f4f010c1"
}
I get nothing. If I use this
{
"_id" : ObjectId("52bbd9bef2ba1f37f4f010c1"),
}
it worked. So much for the Javascript way. AFAIK I have two ways to update a record with the native .net driver:
FindAndModifyResult
WriteConcernResult
But both ways expecting a BsonValue as value where ObjectId seems not to be a valid one...
So I tried using
IMongoQuery query = Query.And(Query.EQ("_id", doc["_id"]));
as described above but that is the same like
{
"_id" : "52bbd9bef2ba1f37f4f010c1"
}
So I wonder how to update a document by it's ObjectId?
When using the currently available Nuget published MongoDB driver for .NET, I can confirm that this code as an example works properly:
MongoClient client = new MongoClient(); // connect to localhost
MongoServer server = client.GetServer();
MongoDatabase test = server.GetDatabase("test");
MongoCollection examples = test.GetCollection("examples");
string id = "52bc958ca45026c2ff24f90b";
IMongoQuery query = Query.EQ("_id", ObjectId.Parse(id));
UpdateBuilder ub = Update.Set("Name", "George");
WriteConcernResult updatedBook = examples.Update(query, ub);
Using this collection data:
> db.examples.remove()
> db.examples.insert({Name:"Larry"})
> db.examples.find()
{ "_id" : ObjectId("52bc958ca45026c2ff24f90b"), "Name" : "Larry" }
Then, run the C# code above and check the collection again:
> db.examples.find()
{ "Name" : "George", "_id" : ObjectId("52bc958ca45026c2ff24f90b") }
While the WriteConcernResult class itself isn't a BsonDocument, the Response property contains the response from the command.
I also removed the unnecessary Query.And, as it's doesn't do anything when there's only one condition:
query = { "_id" : ObjectId("52bc958ca45026c2ff24f90b") }
ub = { "$set" : { "Name" : "George" } }
In MongoDB, if I have a document structure as follows:
{ "_id" : { "$binary" : "jchPoPd7PUS1w+sR7is23w==", "$type" : "03" },
"companies" :
[
{ "_id" : { "$binary" : "jchPoPd7PUS1w+sR7is23w==", "$type" : "03" },
"name" : "Google" },
{ "_id" : { "$binary" : "jchPoPd7PUS1w+sR7is23w==", "$type" : "03" },
"name" : "Greenfin" },
{ "_id" : { "$binary" : "jchPoPd7PUS1w+sR7is23w==", "$type" : "03" },
"name" : "Zynet" }
],
"firstname" : "Peter",
"surname" : "Smith" }
(i.e. a Person document with a Companies array embedded within the person document), then how do I update ALL occurrences of a specific Company (targetting via the company _id) with a single query+update?
I have tried the following:
MongoCollection personCollection = mdb.GetCollection("person");
BsonBinaryData bin = new BsonBinaryData(new Guid("0AE91D6B-A8FA-4D0D-A94A-91D6AC9EE343"));
QueryComplete query = Query.EQ("companies._id", bin);
var update = Update.Set("companies.name", "GreenfinNewName");
SafeModeResult result = personCollection.Update(query, update, UpdateFlags.Multi);
but it doesn't work. I guess my question boils down to two questions: how do I target the embedded companies in the initial query, and then how do I set the new company name in the Set statement. Note: In this example I have chosen to "denormalise" the company data, and embed it in each person document, so there may be several person documents with the same company id and name. Thanks very much.
UPDATE:
Using Bugai13's technique I've got closer. This works:
MongoCollection personCollection = mdb.GetCollection("person");
QueryComplete query = Query.EQ("companies.name", "Bluefin");
var update = Update.Set("companies.$.companynotes", "companynotes update via name worked");
SafeModeResult result = personCollection.Update(query, update, UpdateFlags.Multi, SafeMode.True);
But this doesn't work:
MongoCollection personCollection = mdb.GetCollection("person");
BsonBinaryData bin = new BsonBinaryData(new Guid("0AE91D6B-A8FA-4D0D-A94A-91D6AC9EE343"));
Builders.QueryComplete query = Query.EQ("companies._id", bin);
var update = Update.Set("companies.$.companynotes", "companynotes update via id worked");
SafeModeResult result = personCollection.Update(query, update, UpdateFlags.Multi, SafeMode.True);
So, I can't yet update using the primary key, which is what I need to do...
I suppose use you should take a look into positional operator in mongodb:
var update = MongoDB.Driver.Builders.Update
.Set("companies.$.name", "GreenfinNewName");
^^
all magic here
Note: above code will update only first matched item in array. So if you have two company in nested array with name = GreenfinNewName above code will update only first matched.
Note: UpdateFlags.Multi means multiple documents, but not multiple items in nested array.
Update:
QueryComplete query = Query.EQ("companies._id",
BsonValue.Create(new Guid("0AE91D6B-A8FA-4D0D-A94A-91D6AC9EE343")));
var update = Update.Set("companies.$.companynotes", "companynotes");
personCollection.Update(query, update, UpdateFlags.Multi, SafeMode.True);
Hope this help!
Just wanted to say thanks to you both. Am learning Mongo and this thread helped.
I'm using the 10gen C# driver, and for reference this is my code:
MongoServer mongo = MongoServer.Create();
MongoDatabase db = mongo.GetDatabase("test");
MongoCollection<BsonDocument> coll = db["contacts"];
BsonDocument doc = new BsonDocument();
doc["FirstName"] = "Daniel";
doc["LastName"] = "Smith";
doc["Address"] = "999 Letsby Avenue";
doc["City"] = "London";
doc["County"] = "Greater London";
doc["Postcode"] = "N13";
coll.Insert<BsonDocument>(doc);
QueryComplete qSel = Query.EQ("Postcode", "N13");
MongoCursor<BsonDocument> cur = coll.Find(qSel);
foreach (BsonDocument bdoc in cur)
{
Console.WriteLine(bdoc["FirstName"] + ":" + bdoc["Address"]);
}
UpdateBuilder docTwo = Update.Set("Postcode", "MK10");
coll.Update(qSel, docTwo, UpdateFlags.Multi);
QueryDocument qSel2 = new QueryDocument("FirstName", "Daniel");
MongoCursor<BsonDocument> cur2 = coll.Find(qSel2);
foreach (BsonDocument bsdoc in cur2)
{
Console.WriteLine(bsdoc["FirstName"] + " : " + bsdoc["Postcode"]);
}