I am using .Net MongoDB.Driver 2.2.3 and MongoDB 3.2. I want to get the _id of the document I insert or replace with collection.ReplaceOneAsync(), because I use it later on.
However the UpsertedId of the result is null if the operation was an update (it is set, if an insert happened).
What is the best way to determine the _id of the updated document?
var collection = GetMongoDatabase().GetCollection<BsonDocument>("foos");
var document = new BsonDocument
{
{"foo", "new bar"}
};
var result = await collection.ReplaceOneAsync(
Builders<BsonDocument>.Filter.Eq("foo", "bar"),
document,
new UpdateOptions {IsUpsert = true});
// I want to use result.UpsertedId here;
If I take a look at result after an update in Visual Studio's Immediate Window, I see:
?result
{MongoDB.Driver.ReplaceOneResult.Acknowledged}
[MongoDB.Driver.ReplaceOneResult.Acknowledged]: {MongoDB.Driver.ReplaceOneResult.Acknowledged}
IsAcknowledged: true
IsModifiedCountAvailable: true
MatchedCount: 1
ModifiedCount: 1
UpsertedId: null
As far I know ReplaceOneAsync method doesn't return the UpsertedId when the document is replaced, you will see that field with a value when the document is inserted. If you replace an existing document, you can check in your DB that the replacement document is going to have the same _id of the old document:
var collection = GetMongoDatabase().GetCollection<BsonDocument>("foos");
var d=collection.Find<BsonDocument>(Builders<BsonDocument>.Filter.Eq("foo", "bar")).FirstOrDefault();
var id = d["_id"].ToString();//Check your id here
var document = new BsonDocument
{
{"foo", "new bar"}
};
var result = await collection.ReplaceOneAsync(
Builders<BsonDocument>.Filter.Eq("foo", "bar"),
document,
new UpdateOptions {IsUpsert = true});
var d1=collection.Find<BsonDocument>(Builders<BsonDocument>.Filter.Eq("foo", "new bar")).FirstOrDefault();
var id1 = d1["_id"].ToString(); // You will see the same _id
Probably the method you are looking for is FindOneAndReplaceAsync:
var d =collection.FindOneAndReplace<BsonDocument>(Builders<BsonDocument>.Filter.Eq("foo", "new bar"),
document,
new FindOneAndReplaceOptions<BsonDocument, BsonDocument>()
{ IsUpsert=true,
ReturnDocument=ReturnDocument.After
});
var id = d["_id"].ToString();
Related
Using ElasticSearch NEST .Net package 7.13.2 in Visual Studio 2019
For a list of products I am currently updating existing documents in my product index by using the following code:
var productIndex = "productindex";
foreach (var product in products)
{
productClassIdScript = $"ctx._source.productClassId = \"{product.ProductClassId}\"; ";
elasticClient.Update<productIndex, object>(product.Id,
q => q.Script(s => s.Source(productClassIdScript).Lang("painless")));
}
I do this for more than 10000 products and it takes about 2 hours.
I know I can insert new documents with the Bulk API.
Can I do the updates with the BulkAll method ?
Something like this:
var bulkAllObservable = elasticClient.BulkAll<Product>(myBulkAllRequest)
.Wait(TimeSpan.FromMinutes(15), next =>
{
// do something e.g. write number of pages to console
});
How should I construct myBulkAllRequest ?
Any help is much appreciated.
Bulk index will drastically reduce your indexing / updating time, so this is a good way to go.
You can still use BulkAll for updates, in case elasticsearch already has
document with provided id, the document will be updated.
var bulk = elasticClient.BulkAll<EsDocument>(new List<EsDocument> { new EsDocument { Id = "1", Name = "1" }}, d => d);
using var subscribe = bulk.Subscribe(new BulkAllObserver(onNext: response => Console.WriteLine("inserted")));
bulk.Wait(TimeSpan.FromMinutes(1), response => Console.WriteLine("Bulk insert done"));
var bulk2 = elasticClient.BulkAll<EsDocument>(new List<EsDocument> { new EsDocument { Id = "1", Name = "1_updated" }}, d => d);
using var subscribe2 = bulk2.Subscribe(new BulkAllObserver(onNext: response => Console.WriteLine("inserted")));
bulk2.Wait(TimeSpan.FromMinutes(1), response => Console.WriteLine("Bulk insert done"));
First BulkAll will insert document with Id "1" second, will update document with Id "1".
Index state after the first bulkd
and after second one
I am trying to update a bson document based on a filter query however instead of updating the record filtered, it creates a new file and then updates that file from then on. Please can someone help with this? See below code example:
Note - this behaviour doesn't usually occur when not filtering by a nested element.
//Filter update by user's Id
var filter = Builders<BsonDocument>.Filter.Eq("UserID", userID);
//Filter update by nested bookmark Id to update
var arrayFilters = new List<ArrayFilterDefinition>
{ new BsonDocumentArrayFilterDefinition<BsonDocument>(new
BsonDocument("i.bookmarkID", bookmarkID ))
};
//variables to update
var update = Builders<BsonDocument>.Update
.Set("userBookmarks.$[i].Name", viewName)
...
;
var updateOptions = new UpdateOptions { ArrayFilters = arrayFilters };
collection.UpdateOne(filter, update, updateOptions);
await App.NavigationPage.Navigation.PushAsync(new homepage());
I have a table in DynamoDB.
Table name test-vod
Primary partition key guid (String)
Primary sort key -
With additional attributes as you can see below.
The goal is to query the table using one of the columns that are not a primary key srcVideo, to accomplish that we created a second local index.
And using the low-level API from DynamoDB SDK NuGet package we query with the below code (open to other options instead of low-level API).
var queryRequest = new QueryRequest
{
TableName = $"{_environmentName}-vod",
IndexName = "srcVideo-index",
ScanIndexForward = true,
KeyConditionExpression = "srcVideo = :v_srcVideo",
ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
{
{":v_srcVideo",new AttributeValue {S = inputMediaKey}}
}
};
var response = await _client.QueryAsync(queryRequest, cancellationToken);
// Does not exist
var hlsUrl = response.Items
.SelectMany(p => p)
.SingleOrDefault(p => p.Key.Equals("hlsUrl"));
I am interested to retrieve 3 attributes (fields) from the response hlsUrl, dashUrl, workflowsStatus but all 3 missing, the response contains a Dictionary with a count of keys 27, these are only 27 out of the 35 available columns.
I have tried using ProjectionExpression and other query combinations with no success.
You don't show the CREATE TABLE you've used...
Sounds like your index wasn't created with the Projection attribute you really want...
Default is , KEYS_ONLY. Sounds like you want ALL or maybe INCLUDE just selected attributes...GlobalSecondaryIndex - Projection
Local secondary indexes work the same way...
It is interesting but I made it work with the below code, even if the key/value is not present in the dictionary when inspecting the debugger you can still retrieve it.
var queryRequest = new QueryRequest
{
TableName = tableName,
IndexName = "srcVideo-index",
ScanIndexForward = true,
KeyConditionExpression = "srcVideo = :v_srcVideo",
ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
{
{":v_srcVideo", new AttributeValue {S = inputMediaKey}}
}
};
var response = await _client.QueryAsync(queryRequest, cancellationToken);
if (response.Items.AnyAndNotNull())
{
var dictionary = response.Items.First().ToDictionary(p => p.Key, x => x.Value.S);
return Result.Ok (new VodDataInfo(
dictionary["srcBucket"],
dictionary["srcVideo"],
dictionary["destBucket"],
dictionary.ContainsKey("dashUrl")
? dictionary["dashUrl"]
: default,
dictionary.ContainsKey("hlsUrl")
? dictionary["hlsUrl"]
: default,
dictionary["workflowStatus"]));
}
When data from a device goes into the elastic there are duplicates. I like to avoid this duplicates. I'm using a object of IElasticClient, .NET and NEST to put data.
I searched for a method like ElasticClient.SetDocumentId(), but cant find.
_doc doc = (_doc)obj;
HashObject hashObject = new HashObject { DataRecordId = doc.DataRecordId, TimeStamp = doc.Timestamp };
// hashId should be the document ID.
int hashId = hashObject.GetHashCode();
ElasticClient.IndexDocumentAsync(doc);
I would like to update the data set inside the Elastic instead of adding one more same object right now.
Assuming the following set up
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(pool)
.DefaultIndex("example")
.DefaultTypeName("_doc");
var client = new ElasticClient(settings);
public class HashObject
{
public int DataRecordId { get; set; }
public DateTime TimeStamp { get; set; }
}
If you want to set the Id for a document explicitly on the request, you can do so with
Fluent syntax
var indexResponse = client.Index(new HashObject(), i => i.Id("your_id"));
Object initializer syntax
var indexRequest = new IndexRequest<HashObject>(new HashObject(), id: "your_id");
var indexResponse = client.Index(indexRequest);
both result in a request
PUT http://localhost:9200/example/_doc/your_id
{
"dataRecordId": 0,
"timeStamp": "0001-01-01T00:00:00"
}
As Rob pointed out in the question comments, NEST has a convention whereby it can infer the Id from the document itself, by looking for a property on the CLR POCO named Id. If it finds one, it will use that as the Id for the document. This does mean that an Id value ends up being stored in _source (and indexed, but you can disable this in the mappings), but it is useful because the Id value is automatically associated with the document and used when needed.
If HashObject is updated to have an Id value, now we can just do
Fluent syntax
var indexResponse = client.IndexDocument(new HashObject { Id = 1 });
Object initializer syntax
var indexRequest = new IndexRequest<HashObject>(new HashObject { Id = 1});
var indexResponse = client.Index(indexRequest);
which will send the request
PUT http://localhost:9200/example/_doc/1
{
"id": 1,
"dataRecordId": 0,
"timeStamp": "0001-01-01T00:00:00"
}
If your documents do not have an id field in the _source, you'll need to handle the _id values from the hits metadata from each hit yourself. For example
var searchResponse = client.Search<HashObject>(s => s
.MatchAll()
);
foreach (var hit in searchResponse.Hits)
{
var id = hit.Id;
var document = hit.Source;
// do something with them
}
Thank you very much Russ for this detailed and easy to understand description! :-)
The HashObject should be just a helper to get a unique ID from my real _doc object. Now I add a Id property to my _doc class and the rest I will show with my code below. I get now duplicates any more into the Elastic.
public void Create(object obj)
{
_doc doc = (_doc)obj;
string idAsString = doc.DataRecordId.ToString() + doc.Timestamp.ToString();
int hashId = idAsString.GetHashCode();
doc.Id = hashId;
ElasticClient.IndexDocumentAsync(doc);
}
Using MongoDB C# driver (http://github.com/samus/mongodb-csharp), seems that I'm unable to get the data by ObjectId. Below the command that I'm using:
var spec = new Document { { "_id", id } };
var doc = mc.FindOne(spec);
I also tried this:
var spec = new Document { { "_id", "ObjectId(\"" + id + "\")" } };
var doc = mc.FindOne(spec);
Both return nothing. Meanwhile, if I query it from the mongo console, it returns the expected result.
My question is, does that driver actually support the lookup by ObjectId?
Thanks..
It does support fetching by object ID. Your id variable should be an Oid. Is it the correct type?
Here is a complete program that will
Connect to Mongo
Insert a document
Fetch the document back using its ID
Print the document's details.
// Connect to Mongo
Mongo db = new Mongo();
db.Connect();
// Insert a test document
var insertDoc = new Document { { "name", "my document" } };
db["database"]["collection"].Insert(insertDoc);
// Extract the ID from the inserted document, stripping the enclosing quotes
string idString = insertDoc["_id"].ToString().Replace("\"", "");
// Get an Oid from the ID string
Oid id = new Oid(idString);
// Create a document with the ID we want to find
var queryDoc = new Document { { "_id", id } };
// Query the db for a document with the required ID
var resultDoc = db["database"]["collection"].FindOne(queryDoc);
db.Disconnect();
// Print the name of the document to prove it worked
Console.WriteLine(resultDoc["name"].ToString());
var spec = new Document { { "_id", ObjectId.Parse(id) } };
var doc = mc.FindOne(spec);