MongoDB was harder than I remembered! I've tried various versions of if-exists-replace-else-insert with various functions and options. It should be easy, shouldn't it?
It's my personal opinion that the following should work.
var collection = storageClient.GetCollection<Observer>("observers");
await collection.Indexes.CreateOneAsync(Builders<Observer>.IndexKeys.Ascending(_ => _.MyId), new CreateIndexOptions { Unique = true });
foreach (var observer in _observers)
{
observer.Timestamp = DateTime.Now;
var res = await collection.FindAsync(o => o.MyId == observer.MyId);
if (res==null ||res.Current == null) {
await collection.InsertOneAsync(observer); //Part 1, fails 2nd time solved with res.MoveNextAsync()
}
else
{
observer.ID = res.Current.Single().ID;
var res2 = await collection.ReplaceOneAsync(o=>o.MyId==observer.MyId, observer);
var res3 = await collection.FindAsync(o => o.MyId == observer.MyId);
await res3.MoveNextAsync();
Debug.Assert(res3.Current.Single().Timestamp == observer.Timestamp); //Part 2, Assert fails.
}
}
Observer looks approximately like this:
public class Observer : IObserver
{
[BsonId]
public Guid ID { get; set; }
public int MyId { get; set; }
public DateTime Timestamp { get; set; }
}
The second time I run this with the exact same collection I unexpectedly get:
E11000 duplicate key error index: db.observers.$MyId_1 dup key: { : 14040 }
Edit:
Added original part two code: replacement.
Edit 2:
Now my code looks like this. Still fails.
var collection = storageClient.GetCollection<Observer>("gavagai_mentions");
await collection.Indexes.CreateOneAsync(Builders<Observer>.IndexKeys.Ascending(_ => _.MyID), new CreateIndexOptions { Unique = true });
foreach (var observer in _observers)
{
observer.Timestamp = DateTime.Now;
// Create a BsonDocument version of the POCO that we can manipulate
// and then remove the _id field so it can be used in a $set.
var bsonObserver = observer.ToBsonDocument();
bsonObserver.Remove("_id");
// Create an update object that sets all fields on an insert, and everthing
// but the immutable _id on an update.
var update = new BsonDocument("$set", bsonObserver);
update.Add(new BsonDocument("$setOnInsert", new BsonDocument("_id", observer.ID)));
// Enable the upsert option to create the doc if it's not found.
var options = new UpdateOptions { IsUpsert = true };
var res = await collection.UpdateOneAsync(o => o.MyID == observer.MyID,
update, options);
var res2 = await collection.FindAsync(o => o.MyID == observer.MyID);
await res2.MoveNextAsync();
Debug.Assert(res2.Current.Single().Timestamp == observer.Timestamp); //Assert fails, but only because MongoDB stores dates as UTC, or so I deduce. It works!!
}
You can do this atomically with UpdateOneAsync by using the IsUpsert option to create the doc if it doesn't already exist.
foreach (var observer in _observers)
{
// Create a BsonDocument version of the POCO that we can manipulate
// and then remove the _id field so it can be used in a $set.
var bsonObserver = observer.ToBsonDocument();
bsonObserver.Remove("_id");
// Create an update object that sets all fields on an insert, and everthing
// but the immutable _id on an update.
var update = new BsonDocument("$set", bsonObserver);
update.Add(new BsonDocument("$setOnInsert", new BsonDocument("_id", observer.ID)));
// Enable the upsert option to create the doc if it's not found.
var options = new UpdateOptions { IsUpsert = true };
var res = await collection.UpdateOneAsync(o => o.MyId == observer.MyId,
update, options);
}
Ok, it's just been a long time since I worked with cursors.
await res.MoveNextAsync();
helped.
This stuff is not hard to Google, but the fact of the matter is, I failed, so I'm going to leave the question up.
If you have an answer for part two https://stackoverflow.com/questions/32586064/replace-poco-with-mongodb-net-driver-2, then and would really like to post it here then I'll be happy two edit the questions.
As was pointed out in the comments the information is in fact readily available in the docs http://mongodb.github.io/mongo-csharp-driver/2.0/reference/driver/crud/reading/#finding-documents.
Related
I am trying to add one property in some of my documents (later I will be updating it).
My code looks like below:
public async Task<bool> InsertTagsToPeople(IEnumerable<People> toInsert)
{
var collection = _mongoClient.GetPeopleDetailsCollection();
try
{
var updates = new List<WriteModel<People>>();
foreach (var doc in toInsert)
{
var filter = Builders<People>.Filter.And(
Builders<People>.Filter.Eq(x => x.Address, doc.Address),
Builders<People>.Filter.Eq(x => x.Type, doc.Type));
var update = Builders<People>.Update.Set(x => x.Tags, doc.Tags);
updates.Add(new UpdateOneModel<People>(filter, update){IsUpsert = false});
}
var result = await collection.BulkWriteAsync(updates);
return true;
}
}
Problem with this code is that it is not setting Tags property. Nothing on Database is changing.
In result object I have properties like this:
When I change IsUpsert = true it throws error
Category : "DuplicateKey", Code : 11000, Message : "E11000 duplicate
key error collection:
I don`t understand why it is trying to update whole model instead just one property. But I suppose that IsUpsert do not matter here. Update.Set should do the work.
I have an Entity Framework 6 class called Materials, which is reflected in my database as a table with the same name. Using a parent parameter, I need to return a sorted list of materials from a SQL Query, so that I can later check that edits the user makes do not affect the order. My SQL is a stored procedure that looks like this:
CREATE PROC [dbo].[GET_SortedMaterials](#FinishedGoodCode VARCHAR(50))
AS
SELECT
ROW_NUMBER() OVER (ORDER BY Component.Percentage_of_Parent DESC,Material.Material) AS _sortField
,Material.*
FROM
Components AS Component
INNER JOIN Materials AS Material ON Component.Child_Material = Material.Material
WHERE
Component.Parent_Code = #FinishedGoodCode
ORDER BY
Component.Percentage_of_Parent DESC
,Material.Material
As you can see, the orderby field is not included in the Material. For this reason, I felt I could not return just a set of Material objects and still keep the sorting - I have performed the ordering in SQL and added the _sortField (I think that field may be a bad idea).
My C# code to read the SQL looks like this:
public async Task<SortedList<int, Materials>> GET_SortedMaterials(IProgress<Report> progress, string finishedGoodCode)
{
try
{
var report = new Report { Message = "Retrieving Sorted Materials", NewLine = true, StatusCode = Enums.StatusCode.Working };
progress.Report(report);
using (var context = new DBContext())
{
var ingredientList = await context.Database.SqlQuery<(int _sortField,Materials mat)>("[app].[GET_Customers]").ToListAsync();
var sorted = new SortedList<int, Raw_Materials>();
foreach (var (_sortField, mat) in ingredientList.OrderBy(x=>x._sortField))
{
sorted.Add(_sortField, mat);
}
return sorted;
}
}
catch (Exception ex)
{ [EXCLUDED CODE]
}
}
When the code executes, I get the correct number of rows returned, but I do not get a Sorted list where the Key corresponds to the _sortField value and the Value to the Material value. I have tried various different versions of basically the same code and I cannot get the script to return a list of materials with information about their sorting, instead, the conversion to EF class fails entirely and I only get null values back:
Any advice about how to return a sorted list from SQL and maintain the sorting in C#, when the sort field is not in the return values would be very gratefully received.
use
var ingredientList = await context.Database.SqlQuery<Materials>("[app].[GET_Customers]").Select((mat, _sortField) => (_sortField, mat)).ToDictionary(x => x._sortField, x => x.mat);
or if you want async load use
var ingredientList = await context.Database.SqlQuery<Materials>("[app].[GET_Customers]").ToListAsync().Result.Select((mat, _sortField) => (_sortField, mat)).ToDictionary(x => x._sortField, x => x.mat);
full code
public async Task<SortedList<int, Materials>> GET_SortedMaterials(IProgress<Report> progress, string finishedGoodCode)
{
try
{
var report = new Report { Message = "Retrieving Sorted Materials", NewLine = true, StatusCode = Enums.StatusCode.Working };
progress.Report(report);
using (var context = new DBContext())
{
var ingredientList = await context.Database.SqlQuery<Materials>("[app].[GET_Customers]").ToListAsync().Result.Select((mat, _sortField) => (_sortField, mat)).ToDictionary(x => x._sortField, x => x.mat);
var sorted = new SortedList<int, Raw_Materials>();
foreach (var item in ingredientList.OrderBy(x => x.Key))
{
sorted.Add(item.Key, item.Value);
}
return sorted;
}
}
catch (Exception ex)
{
[EXCLUDED CODE]
}
}
I'm having a problem trying, what boils down to, incrementing a field in a document or inserting an entire document. The context is "trying to insert an initial document for a sequence or incrementing the sequence number for an existing sequence".
This code:
private async Task<int> GetSequenceNumber(string sequenceName)
{
var filter = new ExpressionFilterDefinition<Sequence>(x => x.Id == sequenceName);
var builder = Builders<Sequence>.Update;
var update = builder
.SetOnInsert(x => x.CurrentValue, 1000)
.Inc(x => x.CurrentValue, 1);
var sequence = await _context.SequenceNumbers.FindOneAndUpdateAsync(
filter,
update,
new FindOneAndUpdateOptions<Sequence>
{
IsUpsert = true,
ReturnDocument = ReturnDocument.After,
});
return sequence.CurrentValue;
}
results in the exception
MongoDB.Driver.MongoCommandException: Command findAndModify failed: Updating the path 'currentvalue' would create a conflict at 'currentvalue'.
at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ProcessResponse(ConnectionId connectionId, CommandMessage responseMessage)
Removing the SetOnInsert results in no errors, but inserts a document with the currentValue equal to 1 instead of the expected 1000.
It almost appears if SetOnInsert is not being honored, and that what's happening is a default document is inserted and then currentValue is incremented via Inc atomically as the new document is created.
How do I overcome these issues? A non-C# solution would also be welcome, as I could translate that...
Ok thanks to #dododo in the comments, I now realize that both an Inc and a SetOnInsert can't be applied at the same time. It's unintuitive because you'd think the former would apply on update only and the latter on insert only.
I went with the solution below, which suffers more than one round-trip, but at least works, and appears to work with my concurrency based tests.
public async Task<int> GetSequenceNumber(string sequenceName, int tryCount)
{
if (tryCount > 5) throw new InvalidOperationException();
var filter = new ExpressionFilterDefinition<Sequence>(x => x.Id == sequenceName);
var builder = Builders<Sequence>.Update;
// optimistically assume value was already initialized
var update = builder.Inc(x => x.CurrentValue, 1);
var sequence = await _context.SequenceNumbers.FindOneAndUpdateAsync(
filter,
update,
new FindOneAndUpdateOptions<Sequence>
{
IsUpsert = true,
ReturnDocument = ReturnDocument.After,
});
if (sequence == null)
try
{
// we have to try to save a new sequence...
sequence = new Sequence { Id = sequenceName, CurrentValue = 1001 };
await _context.SequenceNumbers.InsertOneAsync(sequence);
}
// ...but something else could beat us to it
catch (MongoWriteException e) when (e.WriteError.Code == DuplicateKeyCode)
{
// ...so we have to retry an update
return await GetSequenceNumber(sequenceName, tryCount + 1);
}
return sequence.CurrentValue;
}
I'm sure there are other options. It may be possible to use an aggregation pipeline, for example.
When data from a device goes into the elastic there are duplicates. I like to avoid this duplicates. I'm using a object of IElasticClient, .NET and NEST to put data.
I searched for a method like ElasticClient.SetDocumentId(), but cant find.
_doc doc = (_doc)obj;
HashObject hashObject = new HashObject { DataRecordId = doc.DataRecordId, TimeStamp = doc.Timestamp };
// hashId should be the document ID.
int hashId = hashObject.GetHashCode();
ElasticClient.IndexDocumentAsync(doc);
I would like to update the data set inside the Elastic instead of adding one more same object right now.
Assuming the following set up
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(pool)
.DefaultIndex("example")
.DefaultTypeName("_doc");
var client = new ElasticClient(settings);
public class HashObject
{
public int DataRecordId { get; set; }
public DateTime TimeStamp { get; set; }
}
If you want to set the Id for a document explicitly on the request, you can do so with
Fluent syntax
var indexResponse = client.Index(new HashObject(), i => i.Id("your_id"));
Object initializer syntax
var indexRequest = new IndexRequest<HashObject>(new HashObject(), id: "your_id");
var indexResponse = client.Index(indexRequest);
both result in a request
PUT http://localhost:9200/example/_doc/your_id
{
"dataRecordId": 0,
"timeStamp": "0001-01-01T00:00:00"
}
As Rob pointed out in the question comments, NEST has a convention whereby it can infer the Id from the document itself, by looking for a property on the CLR POCO named Id. If it finds one, it will use that as the Id for the document. This does mean that an Id value ends up being stored in _source (and indexed, but you can disable this in the mappings), but it is useful because the Id value is automatically associated with the document and used when needed.
If HashObject is updated to have an Id value, now we can just do
Fluent syntax
var indexResponse = client.IndexDocument(new HashObject { Id = 1 });
Object initializer syntax
var indexRequest = new IndexRequest<HashObject>(new HashObject { Id = 1});
var indexResponse = client.Index(indexRequest);
which will send the request
PUT http://localhost:9200/example/_doc/1
{
"id": 1,
"dataRecordId": 0,
"timeStamp": "0001-01-01T00:00:00"
}
If your documents do not have an id field in the _source, you'll need to handle the _id values from the hits metadata from each hit yourself. For example
var searchResponse = client.Search<HashObject>(s => s
.MatchAll()
);
foreach (var hit in searchResponse.Hits)
{
var id = hit.Id;
var document = hit.Source;
// do something with them
}
Thank you very much Russ for this detailed and easy to understand description! :-)
The HashObject should be just a helper to get a unique ID from my real _doc object. Now I add a Id property to my _doc class and the rest I will show with my code below. I get now duplicates any more into the Elastic.
public void Create(object obj)
{
_doc doc = (_doc)obj;
string idAsString = doc.DataRecordId.ToString() + doc.Timestamp.ToString();
int hashId = idAsString.GetHashCode();
doc.Id = hashId;
ElasticClient.IndexDocumentAsync(doc);
}
I try to get all data from collection into MongoDB server using C# driver.
The idea is connect to the server and get all collection than insert into list of class.
List<WatchTblCls> wts;
List<UserCls> users;
List<SymboleCls> syms;
public WatchTbl()
{
InitializeComponent();
wts = new List<WatchTblCls>();
users = new List<UserCls>();
syms = new List<SymboleCls>();
}
public async void getAllData()
{
client = new MongoClient("mongodb://servername:27017");
database = client.GetDatabase("WatchTblDB");
collectionWatchtbl = database.GetCollection<WatchTbl>("Watchtbl");
collectionUser = database.GetCollection<UserCls>("Users");
collectionSymbole = database.GetCollection<SymboleCls>("Users");
var filter = new BsonDocument();
using (var cursor = await collectionWatchtbl.FindAsync(filter))
{
while (await cursor.MoveNextAsync())
{
var batch = cursor.Current;
foreach (var document in batch)
{
wts.Add(new WatchTblCls(document["_id"], document["userId"], document["wid"], document["name"], document["Symboles"]));
}
}
}
}
I get this error under
wts.Add(new WatchTblCls(document["_id"], document["userId"], document["wid"], document["name"], document["Symboles"]));
Cannot apply indexing with [] to an expression of type 'WatchTbl'
I don't understand the reason behind using WatchTbl and WatchTblCls both together. Is WatchTblCls a model for the entity WatchTbl here? Im not sure.
In any case. If you go for aggregation and want to convert WatchTbl collection to WatchTblCls list, your desired solution might look like the following. I don't know the defiitions of the classes so I'm assuming:
var client = new MongoClient("mongodb://servername:27017");
var database = client.GetDatabase("WatchTblDB");
var collectionWatchtbl = database.GetCollection<WatchTbl>("Watchtbl");
var collectionUser = database.GetCollection<UserCls>("Users");
var collectionSymbole = database.GetCollection<SymboleCls>("Users");
var list = collectionWatchtbl.AsQueryable().Select(x => new WatchTblCls() {
id = x.id,
userId = x.userId,
.....
});
If you can use the same WatchTbl class and still want to load the full collection to a local List (which is definitely not a good idea):
List<WatchTbl> list = await collectionWatchtbl.Find(x => true).ToListAsync();