Problem statement:
I have a collection in MongoDB that has a field with the type Int32. I would like to add a document to this collection. I need to increment the value by 1 for each insert as that field is indexed and must be unique.
Options:
[preferable] Increment the value on the DB side. That is, not specifying a new (higher) value. Just instruct MongoDB to auto increment upon insert.
Reading first. Executing a find query against the DB to find the current (before insert) highest value first, incrementing in memory, and inserting the new doc. This might fail due to racing conditions (the operation is not atomic).
keeping an index counter in memory. Not an option for me as there are multiple apps writing to the same collection (legacy limitation).
Other Ideas?
Example:
{
_id: ....
index: 123,
open: true
}
await collection.InsertOneAsync(record.ToBsonDocument());
The new doc inserted should have index value of 124
Language:
C#
Questions:
Can you provide a sample code (C#) to achieve the first option?
Extra info:
I do not have access to the code of the other app (which keeps its own index number). So having another collection and adding an sequence resolver function will not work as this will trigger a change to the legacy app.
MongoDB has a default tutorial on how to achieve that here
1 - Create a counters collections and insert the id there:
db.counters.insert(
{
_id: "userid",
seq: 0
}
)
2 - Create a custom function to retrieve the next value:
function getNextSequence(name) {
var ret = db.counters.findAndModify(
{
query: { _id: name },
update: { $inc: { seq: 1 } },
new: true
}
);
return ret.seq;
}
Use the getNextSequence to retrieve the next value:
db.users.insert(
{
_id: getNextSequence("userid"),
name: "Sarah C."
}
)
db.users.insert(
{
_id: getNextSequence("userid"),
name: "Bob D."
}
)
I had to do this in a project using MongoDB C# Driver.
Here's what I did: I created a separated collection called Sequence, with the name and value of it and I also created a repository for it.
Here is the code of class Sequence:
public class Sequence
{
[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
[BsonElement("_id")]
public string Id { get; set; }
public string SequenceName { get; set; }
public int SequenceValue { get; set; }
}
And now the code of the method to generate the sequence value:
public class SequenceRepository
{
protected readonly IMongoDatabase _database;
protected readonly IMongoCollection<Sequence> _collection;
public SequenceRepository(IMongoDatabase database)
{
_database = database;
_colecao = _database.GetCollection<Sequence>(typeof(Sequence).Name);
}
public int GetSequenceValue(string sequenceName)
{
var filter = Builders<Sequence>.Filter.Eq(s => s.SequenceName, sequenceName);
var update = Builders<Sequence>.Update.Inc(s => s.SequenceValue , 1);
var result = _colecao.FindOneAndUpdate(filter, update, new FindOneAndUpdateOptions<Sequence, Sequence> { IsUpsert = true, ReturnDocument = ReturnDocument.After });
return result.SequenceValue;
}
}
Finally I called this method before insert some document:
public void Inserir(Order order)
{
order.Code = new SequenceRepository(_database).GetSequenceValue("orderSequence");
_collection.InsertOne(order);
}
You can create a Mongo Sequence in a separate collection counter
db.counter.insert({ _id: "mySeq", seq: 0 })
You can encapsulate sequence logic in a simple function like this
function getNextMySeq(name) {
var ret = db.counter.findAndModify({
query: { _id: name },
update: { $inc: { seq: 1 } },
new: true
});
return ret.seq;
}
Now simply use the function call during the insert
db.collection.insert({
index: getNextMySeq("mySeq")
})
Related
I've got this entity:
internal record OrderBookEntity
{
[BsonId]
public required AssetDefinition UnderlyingAsset { get; init; }
public required List<OrderEntity> Orders { get; init; }
}
which leads to this sort of document:
{
_id: {
Class: 0,
Symbol: 'EURUSD'
},
Orders: [
{
_id: 'a611ffb1-c3e7-43d6-8238-14e311122125',
Price: '-10.000000101',
Amount: '30.000000003',
OrderAction: 1,
EffectiveTime: ISODate('2022-10-14T06:33:02.872Z')
},
{
_id: 'a611ffb1-c3e7-43d6-8238-14e311122126',
Price: '-10.000000101',
Amount: '30.000000003',
OrderAction: 1,
EffectiveTime: ISODate('2022-10-14T06:33:08.264Z')
}
]
}
I can add and remove from the Orders set without updating the whole document with:
Builders<OrderBookEntity>.Update.AddToSet(...);
Builders<OrderBookEntity>.Update.Pull(...);
I can't, however, see a way to modify one of those in place.
How would I go about changing the Amount on saying a611ffb1-c3e7-43d6-8238-14e311122125 without having to read the document, modify the collection, and update the whole thing, or just pulling and reading the order... Neither of which seems particularly performant.
You can work with FieldDefinition by providing the field to be updated in string instead of Func expression.
MongoDB query
db.collection.update({
"Orders._id": "a611ffb1-c3e7-43d6-8238-14e311122125"
},
{
$set: {
"Orders.$.Amount": "100"
}
})
Demo # Mongo Playground
MongoDB .NET Driver syntax
var filter = new BsonDocument
{
{ "Orders._id", "a611ffb1-c3e7-43d6-8238-14e311122125" }
};
var update = Builders<OrderBookEntity>.Update.Set("Orders.$.Amount", /* value */);
UpdateResult result = await _collection.UpdateOneAsync(filter, update);
Demo
How can I create an array of unique elements in MongoDB c# driver,
I don't want to check every time if this element already in the array or not.
suppose :
list=[1,2,3,4]
then I shouldn't be able to add duplicate element (such as 3 )
You can use the AddToSet or AddToSetEach method, every time you create or update the array, as mentioned in the comments:
var update = Builders<Entity>.Update.AddToSetEach(e => e.Items, new [] {1, 2});
collection.UpdateOne(new BsonDocument(), update, new UpdateOptions { IsUpsert = true });
And you can define a schema validation when creating the collection, to ensure that duplicate items will never be allowed (an error would be thrown on insert/update, “Document failed validation”).
You can define the schema in the MongoDB shell, or here is how to do it in C#:
var options = new CreateCollectionOptions<Entity>
{
ValidationAction = DocumentValidationAction.Error,
ValidationLevel = DocumentValidationLevel.Strict,
Validator = new FilterDefinitionBuilder<Entity>().JsonSchema(new BsonDocument
{
{ "bsonType", "object" },
{ "properties", new BsonDocument("Items", new BsonDocument
{
{ "type" , "array" },
{ "uniqueItems", true }
})
}
})
};
database.CreateCollection("entities", options, CancellationToken.None);
where Entity is an example class like this:
public class Entity
{
public ObjectId Id { get; set; }
public int[] Items { get; set; }
}
Here are the API docs for CreateCollectionOptions and in the unit tests you can see examples of usage - e.g. JsonSchema(). Unfortunately I don't see anything in the reference docs with more thorough explanations.
Is it possible (preferably using the C# Builders) to add a new item to a deeply nested array I.e. an array within an array within an array.
My data model looks something like :
public class Company
{
public string Id { get; set; }
public string Name { get; set; }
public IEnumerable<Department> Departments { get; set; }
}
public class Department
{
public string Id { get; set; }
public string Name { get; set; }
public IEnumerable<Managers> Managers { get; set; }
}
public class Manager
{
public string Id { get; set; }
public string Name { get; set; }
public IEnumerable<Employee> Employees { get; set; }
}
public class Employee
{
public string Id { get; set; }
public string Name { get; set; }
}
Which translates to:
{
"Id": 12345,
"Name": "Company Ltd",
"Departments": [
{
"Id": 1,
"Name": "Development",
"Managers" : [
{
"Id" : 5555,
"Name" : "The Boss",
"Employees": [
{
"Id" : 123,
"Name" : "Developer 1"
},
{
"Id" : 124,
"Name" : "Developer 2"
}
]
}
]
}
]
}
If I wanted to add another employee under a specific manager how would I go about doing that?
In order to push to a nested array, you must make use of the positional operator $ in order to specify a matching outer array element to apply the operation to. For example:
db.collection.update(
{"my_array._id": myTargetId},
{$push: {"my_array.$.my_inner_array": myArrayElem}}
);
This breaks down, however, for traversing nested arrays--that is, you can only use the positional operator on the single array, not any nested ones. This is a well-defined problem as noted in the MongoDB documentation.
If you absolutely need to perform these kinds of nested array operations, then you have a couple of options available to you:
The first, and preferred, is to update your document structure and avoid nesting arrays more than one level deep. This will avoid the issue altogether, but will require any existing data to be migrated to the new structure and additional efforts to be made to structure the data in the way you need on the fly on retrieval. Separate client and server representations of your data will end up being required.
The second is to perform a series of less-reliable steps:
1. Retrieve the original document.
2. Locate the indexes for each array where your target element is located manually.
3. Attempt an update on the specific index chain and attempt to match that index chain as well.
4. Check the result of the update attempt--if it fails, then it's possible that the document was changed while the indexes were being calculated.
For example, if you wanted to update manager with ID 5555 to have the additional employee, you'd perform the following query after retrieving the indexes:
// Index chain found to be Departments.0 and Managers.0
db.collection.update(
{
"Id": 12345,
"Departments.0.Managers.0.Id": 5555 // Specify index chain 0,0 and ensure that our target still has Id 5555.
},
{ $push: {
"Departments.0.Managers.0.Employees": myNewEmployee // Push to index chain 0,0
}}
);
Use positional operator for each of the arrays except the one you want to push to.
Use array filters in update options to specify the department and manager ids. The letter used in the array filters should match the letter used as the positional operator in the update definition. So "d.Id" -> "Departments.$[d]"
If you want to match on more than one property you can use a dictionary in the array filter.
private IMongoCollection<Company> _collection;
public async Task AddEmployee()
{
var filter = Builders<Company>.Filter.Where(d => d.Id == "companyId");
var update = Builders<Company>.Update
.Push("Departments.$[d].Managers.$[m].Employees", new Employee { Id = "employeeId", Name = "employeeName" });
var updateOptions = new UpdateOptions
{
ArrayFilters = new List<ArrayFilterDefinition>
{
new BsonDocumentArrayFilterDefinition<BsonDocument>(new BsonDocument("d.Id", "departmentId")),
new BsonDocumentArrayFilterDefinition<BsonDocument>(new BsonDocument("m.Id", "managerId")),
}
};
await _collection.UpdateOneAsync(filter, update, updateOptions);
}
The problem here is that you need to use strings in the update definition and filters but not sure how to manage it without strings.
To remove an employee from the array is similar but you will have to specify an extra filter for the employee that you want to remove.
public async Task FireEmployee()
{
var filter = Builders<Company>.Filter.Where(d => d.Id == "companyId");
var employeeFilter = Builders<Employee>.Filter.Where(e => e.Id == "employeeId");
var update = Builders<Company>.Update
.PullFilter("Departments.$[d].Managers.$[m].Employees", employeeFilter);
var updateOptions = new UpdateOptions
{
ArrayFilters = new List<ArrayFilterDefinition>
{
new BsonDocumentArrayFilterDefinition<BsonDocument>(new BsonDocument("d.Id", "departmentId")),
new BsonDocumentArrayFilterDefinition<BsonDocument>(new BsonDocument("m.Id", "managerId")),
}
};
await _collection.UpdateOneAsync(filter, update, updateOptions);
}
I am using Newtonsoft.Json 11.0.2 in .Net core 2.0.
If i use JObject, i am able to SelectToken like so:
JObject.Parse("{\"context\":{\"id\":42}}").SelectToken("context.id")
Returns
42
However, if i use JRaw, i get null for the same path?
new JRaw("{\"context\":{\"id\":42}}").SelectToken("context.id")
returns
null
Due to how my code is setup, my model is already in JRaw, and converting it to JObject just to select this token seems like a waste of RAM (this call is on the hot path).
UPDATE
Ok, my actual data comes down in a model where only one of the properties is JRaw, so i need something like the below to work:
JsonConvert.DeserializeObject<Dictionary<string, JRaw>>(
"{\"a\":{\"context\":{\"id\":42}}}")["a"].SelectToken("context.id")
The above returns null again.
Title might be a bit misleading, but basically what the OP needs is a way to parse an existing (and large) JRaw object without consuming too much memory.
I ran some tests and I was able to find a solution using a JsonTextReader.
I don't know the exact structure of the OP's json strings, so I'll assume something like this:
[
{
"context": {
"id": 10
}
},
{
"context": {
"id": 20
}
},
{
"context": {
"id": 30
}
}
]
Result would be an integer array with the id values (10, 20, 30).
Parsing method
So this is the method that takes a JRaw object as a parameter and extracts the Ids, using a JsonTextReader.
private static IEnumerable<int> GetIds(JRaw raw)
{
using (var stringReader = new StringReader(raw.Value.ToString()))
using (var textReader = new JsonTextReader(stringReader))
{
while (textReader.Read())
{
if (textReader.TokenType == JsonToken.PropertyName && textReader.Value.Equals("id"))
{
int? id = textReader.ReadAsInt32();
if (id.HasValue)
{
yield return id.Value;
}
}
}
}
}
In the above example I'm assuming there is one and only one type of object with an id property.
There are other ways to extract the information we need - e.g. we can check the token type and the path as follows:
if (textReader.TokenType == JsonToken.Integer && textReader.Path.EndsWith("context.id"))
{
int id = Convert.ToInt32(textReader.Value);
yield return id;
}
Testing the code
I created the following C# classes that match the above json structure, for testing purposes:
public class Data
{
[JsonProperty("context")]
public Context Context { get; set; }
public Data(int id)
{
Context = new Context
{
Id = id
};
}
}
public class Context
{
[JsonProperty("id")]
public int Id { get; set; }
}
Creating a JRaw object and extracting the Ids:
class Program
{
static void Main(string[] args)
{
JRaw rawJson = CreateRawJson();
List<int> ids = GetIds(rawJson).ToList();
Console.Read();
}
// Instantiates 1 million Data objects and then creates a JRaw object
private static JRaw CreateRawJson()
{
var data = new List<Data>();
for (int i = 1; i <= 1_000_000; i++)
{
data.Add(new Data(i));
}
string json = JsonConvert.SerializeObject(data);
return new JRaw(json);
}
}
Memory Usage
Using Visual Studio's Diagnostic tools I took the following snapshots, to check the memory usage:
Snapshot #1 was taken at the beginning of the console application (low memory as expected)
Snapshot #2 was taken after creating the JRaw object
JRaw rawJson = CreateRawJson();
Snapshot #3 was taken after extracting the ids
List ids = GetIds(rawJson).ToList();
I have over 128 documents in my Raven database of type Foo:
class Foo {
public string Id {get; set;}
public string Name {get; set;}
}
For two documents, the Name property has value "MyName".
With an IDocumentSession session, if I perform session.query<Foo>().Where(f => f.Name.equals("MyName")), I get zero results. This appears to be because the two documents that match "MyName" are not returned in the 128 documents returned from the RavenDB server (which is the default client-side page size). So, the client API filters by Name=="MyName" on the 128 documents returned, but since my two matching documents were not among those first 128, no matching documents are found. I verified this hypothesis by 1. looking at my RavenDb studio in my browser and verifying that these two documents exist, and 2. by implementing an unbounded, streaming query and successfully retrieving these two documents:
var results = new List<Foo>();
var query = session.Query<Foo>().Where(f => f.Name.equals("MyName");
using (var enumerator = session.Advanced.Stream(query){
while (enumerator.MoveNext()){
results.Add(enumerator.Current.Document);
}
}
However, the streaming solution is not ideal for me. My question is the following: is there a way to ask RavenDB to perform the filter on Name on the server, before returning 128 documents to the client? I want to search through all documents in my database for my given Where filter, but once the filter is applied, I am perfectly content to have the server return <= 128 documents to the client API.
Your assumption is not correct. The default page size applies to the result of the query and not to the document collection you are querying on (if this was really true it would cause ugly problems left and right as you have no control over what comes first and what comes last in the collection).
Are you actually executing the query (i.e. calling query.ToList() or something similiar)? - If you do, please provide further code showing your query and assigning the result.
EDIT
So this here works as expected on my machine:
[TestFixture]
public class UnitTest3
{
public class Foo
{
public string Id { get; set; }
public string Name { get; set; }
}
private readonly IDocumentStore _documentStore;
public UnitTest3()
{
_documentStore = new EmbeddableDocumentStore
{
Configuration =
{
RunInUnreliableYetFastModeThatIsNotSuitableForProduction = true,
RunInMemory = true,
}
}.Initialize();
}
public void InsertDummies()
{
using (IDocumentSession session = _documentStore.OpenSession())
{
for (int i = 0; i < 1000; i++)
{
Foo foo = new Foo { Name = "Foo" + i };
session.Store(foo);
}
Foo fooA = new Foo { Name = "MyName"};
session.Store(fooA);
Foo fooB = new Foo { Name = "MyName" };
session.Store(fooB);
session.SaveChanges();
}
}
[Test]
public void Query()
{
List<Foo> result;
InsertDummies();
using (IDocumentSession session = _documentStore.OpenSession())
{
result = session.Query<Foo>().Where(f => f.Name.Equals("MyName")).ToList();
}
Assert.AreEqual(2, result.Count);
}
}
Did you check whether the index might be stale? - https://ravendb.net/docs/article-page/3.0/csharp/indexes/stale-indexes