I got one place in my application where I want to use Mongo (3.6) as a lock of multiple threads (on different servers). Essentially something like "if one thread started work, other threads should see it through mongo and dont start the same work in parallel".
From the documentation I learned
Combined with "majority" write concern, "linearizable" read concern enables multiple threads to perform reads and writes on a single document as if a single thread performed these operations in real time;
So this sounded good to me, I insert a certain document if one thread starts work, and other threads check if such document already exists and dont start if so, but it does not work for my case.
I prepared two tests - one non-parallel that successfully blocks the second thread - but the parallel test fails and I get two of these RebuildLog documents.
using System;
using System.Threading.Tasks;
using FluentAssertions;
using Xunit;
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.Driver;
namespace FindOneAndUpdateTests
{
public class FindOneAndUpdateTests
{
private static IMongoDatabase GetDatabase()
{
var dbName = "test";
var client = new MongoClient("mongodb://localhost:45022");
return client.GetDatabase(dbName);
}
private IMongoCollection<RebuildLog> GetCollection()
{
return GetDatabase().GetCollection<RebuildLog>("RebuildLog");
}
[Fact]
public async Task FindOneAndUpdate_NotParallel_Test()
{
var dlpId = Guid.NewGuid();
var first = await FindOneAndUpdateMethod(dlpId);
var second = await FindOneAndUpdateMethod(dlpId);
first.Should().BeFalse();
second.Should().BeTrue();
}
[Fact]
public async Task FindOneAndUpdate_Parallel_Test()
{
var dlpId = Guid.NewGuid();
var taskFirst = FindOneAndUpdateMethod(dlpId);
var taskSecond = FindOneAndUpdateMethod(dlpId);
var first = await taskFirst;
var second = await taskSecond;
first.Should().BeFalse();
second.Should().BeTrue();
}
private async Task<bool> FindOneAndUpdateMethod(Guid dlpId)
{
var mongoCollection = GetCollection();
var filterBuilder = Builders<RebuildLog>.Filter;
var filter = filterBuilder.Where(w => w.DlpId == dlpId);
var creator = Builders<RebuildLog>.Update
.SetOnInsert(w => w.DlpId, dlpId)
.SetOnInsert(w => w.ChangeDate, DateTime.UtcNow)
.SetOnInsert(w => w.BuildDate, DateTime.UtcNow)
.SetOnInsert(w => w.Id, Guid.NewGuid());
var options = new FindOneAndUpdateOptions<RebuildLog>
{
IsUpsert = true,
ReturnDocument = ReturnDocument.Before
};
var result = await mongoCollection
.WithWriteConcern(WriteConcern.WMajority)
.WithReadConcern(ReadConcern.Linearizable)
.FindOneAndUpdateAsync(filter, creator, options);
return result != null;
}
}
[BsonIgnoreExtraElements]
public class RebuildLog
{
public RebuildLog()
{
Id = Guid.NewGuid();
}
public Guid Id { get; set; }
public DateTime ChangeDate { get; set; }
public string ChangeUser { get; set; }
public Guid DlpId { get; set; }
public string Portal { get; set; }
public DateTime? BuildDate { get; set; }
}
}
My suspicion is that my idea with the atomic handcrafted GetOrInsert (see the FindOneAndUpdate with IsUpsert) breaks the constraint of "on a single document" in the documentation. Any idea to fix this or is it just not possible?
It is interesting. May be you have no unique index on DlpId? That's why mongo decides that sequential execution of these operations is not necessary because in your case it's no write-then-read pattern (as it pointed in "Client Sessions and Causal Consistency Guarantees"). It is update-or-create two times concurrently.
What about this? :
public class SyncDocument
{
// ...
[BsonElement("locked"), BsonDefaultValue(false)]
public bool Locked { get; set; }
}
In client code:
var filter = Builders<SyncDocument>.Filter.Eq(d => d.Locked, false);
var update = Builders<SyncDocument>.Update.Set(d => d.Locked, true);
var result = collection.UpdateOne(filter, update);
if (result.ModifiedCount == 1) {
Console.WriteLine("Lock acquired");
}
Document with Locked field should be created before applications startup (if it is applicable for your task).
Related
I am using MongoDb .net driver in which i have to update document based on certain condition.
Here is how my find query looks like in c# mongo driver
DbService.conversations.Find (new BsonDocument ("_id", new ObjectId ("obje-id-here"))).FirstOrDefault ();
How can i update specific field of document based on certain _id using Mongodb .net driver without using Builders?
**Note : **
I have tried this update query
var updateResultFromQuery = await DbService.conversations.UpdateOneAsync(Builders<RawBsonDocument>.Filter.Eq("_id", "5e01a89e5f317324780b7f83"),Builders<RawBsonDocument>.Update.Set("visitorName", "Guest41815"));
Console.WriteLine("after update response --- "+updateResultFromQuery.ToJson());
But it's not updating the value even i am receiving update response like this
{ "_t" : "Acknowledged" }
You could simply ReplaceOne object instead of updating it , that way you wont be forced to use builders but you will be doing a find and replace instead. 2 database operations instead of one. You can then update your object in memory instead.
Collection.ReplaceOne(filter, replacement, new UpdateOptions() { IsUpsert = false });
here's an example of create/update/find as requested:
using MongoDB.Entities; //Install-Package MongoDB.Entities
using MongoDB.Entities.Core;
using System;
using System.Linq;
namespace StackOverflow
{
public class Customer : Entity
{
public string Name { get; set; }
public Agent Agent { get; set; }
}
public class Agent
{
public string Name { get; set; }
public string Email { get; set; }
}
public class Program
{
private static void Main()
{
new DB("test", "localhost");
// create a customer with embedded agent
var customer = new Customer
{
Name = "Customer A",
Agent = new Agent { Name = "Agent Uno", Email = "uno#youknow.com" }
};
customer.Save();
// update customer name
DB.Update<Customer>()
.Match(c =>
c.ID == customer.ID &&
c.Agent.Email == "uno#youknow.com")
.Modify(c =>
c.Name, "Updated Customer")
.Execute();
// find updated customer
var cst = DB.Find<Customer>()
.Match(customer.ID)
.Execute()
.Single();
Console.WriteLine($"Customer Name: {cst.Name}");
Console.Read();
}
}
}
I achieved updating my document without using any third party library using these options
var filter = new BsonDocument(new Dictionary<string, dynamic> () {
{
"_id", new BsonObjectId("object-id-here")
},
{
"assigned_to.email" , agentEmail
}
});
var updateDoc = new BsonDocument(new Dictionary<string, dynamic> () {
{
"$set", new BsonDocument("assigned_to.$.avgResponseTime", Convert.ToDouble(obj.agentChatAvgResponseTime))
}
});
var updateQueryResult = DbService.conversations.UpdateOne(filter, updateDoc, new UpdateOptions {IsUpsert = false });
I have some data in documentdb with following structure:
{
id:1,
key:"USA",
states:["New York","New Jersey", "Ohio", "Florida" ]
}
I need to check if "California" is available in the above document with a query from C# using CreateDocumentQuery which returns true/false. How can I read the boolean value from CreateDocumentQuery?
Assuming you have a DTO looking something like this:
class Item
{
[JsonProperty("id")]
public string Id { get; set; }
[JsonProperty("key")]
public string Key { get; set; }
[JsonProperty("states")]
public List<string> States { get; set; }
}
You can use the Contains method on an array to check if the value exists or not. The LINQ to SQL converter will turn this into a sql query for you. All you have to do is to add any other predicates in the where clause and run it. I am intentionally only selecting the Id of the document to make save some RUs and make it run faster.
If you add the partition key in the expression present in the where clause, or if you know what the partition key value is, please set it in the feed options to enhance performance. Cross partition queries are not really recommended as part of your day to day workflow.
You can use the CountAsync() extension method of the DocumentQueryable to get the count of doucments matching the predicate and then do a > 0 to see if it exists.
Here is the code for that:
public async Task<bool> ArrayContainsAsync()
{
var documentClient = new DocumentClient(new Uri("https://localhost:8081"), "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==");
var collectionUri = UriFactory.CreateDocumentCollectionUri("dbname", "collectionName");
return (await documentClient
.CreateDocumentQuery<Item>(collectionUri, new FeedOptions { EnableCrossPartitionQuery = true })
.Where(x => x.States.Contains("California")).CountAsync()) > 0;
}
However, the code below will take control of the query and exit on first match which will be way more efficient than the code above.
public async Task<bool> ArrayContainsAsync()
{
var documentClient = new DocumentClient(new Uri("https://localhost:8081"), "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==");
var collectionUri = UriFactory.CreateDocumentCollectionUri("dbname", "collectionName");
var query = documentClient
.CreateDocumentQuery<Item>(collectionUri, new FeedOptions { EnableCrossPartitionQuery = true })
.Where(x => x.States.Contains("California")).Select(x=> x.Id).AsDocumentQuery();
while (query.HasMoreResults)
{
var results = await query.ExecuteNextAsync();
if (results.Any())
return true;
}
return false;
}
My 1st question, so please be kind... :)
I'm using the C# HttpClient to invoke Jobs API Endpoint.
Here's the endpoint: Jobs API Endpoint (doesn't require key, you can click it)
This gives me JSON like so.
{
"count": 1117,
"firstDocument": 1,
"lastDocument": 50,
"nextUrl": "\/api\/rest\/jobsearch\/v1\/simple.json?areacode=&country=&state=&skill=ruby&city=&text=&ip=&diceid=&page=2",
"resultItemList": [
{
"detailUrl": "http:\/\/www.dice.com\/job\/result\/90887031\/918715?src=19",
"jobTitle": "Sr Security Engineer",
"company": "Accelon Inc",
"location": "San Francisco, CA",
"date": "2017-03-30"
},
{
"detailUrl": "http:\/\/www.dice.com\/job\/result\/cybercod\/BB7-13647094?src=19",
"jobTitle": "Platform Engineer - Ruby on Rails, AWS",
"company": "CyberCoders",
"location": "New York, NY",
"date": "2017-04-16"
}
]
}
I've pasted a complete JSON snippet so you can use it in your answer. The full results are really long for here.
Here's are the C# classes.
using Newtonsoft.Json;
using System.Collections.Generic;
namespace MyNameSpace
{
public class DiceApiJobWrapper
{
public int count { get; set; }
public int firstDocument { get; set; }
public int lastDocument { get; set; }
public string nextUrl { get; set; }
[JsonProperty("resultItemList")]
public List<DiceApiJob> DiceApiJobs { get; set; }
}
public class DiceApiJob
{
public string detailUrl { get; set; }
public string jobTitle { get; set; }
public string company { get; set; }
public string location { get; set; }
public string date { get; set; }
}
}
When I invoke the URL using HttpClient and deserialize using JSON.NET, I do get the data back properly.
Here's the code I am calling from my Console App's Main method (hence the static list, I think this could be better refactored??)
private static List<DiceApiJob> GetDiceJobs()
{
HttpClient httpClient = new HttpClient();
var jobs = new List<DiceApiJob>();
var task = httpClient.GetAsync("http://service.dice.com/api/rest/jobsearch/v1/simple.json?skill=ruby")
.ContinueWith((taskwithresponse) =>
{
var response = taskwithresponse.Result;
var jsonString = response.Content.ReadAsStringAsync();
jsonString.Wait();
var result = JsonConvert.DeserializeObject<DiceApiJobWrapper>(jsonString.Result);
if (result != null)
{
if (result.DiceApiJobs.Any())
jobs = result.DiceApiJobs.ToList();
if (result.nextUrl != null)
{
//
// do this GetDiceJobs again in a loop? How?? Any other efficient elegant way??
}
}
});
task.Wait();
return jobs;
}
But now, how do I check if there are more jobs using the nextUrl field? I know I can check to see if it's not null, and if if not, that means there are more jobs to pull down.
Results from my debugging and stepping through
How do I do this recursively, and without hanging and with some delays so I don't cross the API limits? I think I have to use TPL ( Task Parallel Library) but am quite baffled.
Thank you!
~Sean
If you are concerned about response time of your app and would like to return some results before you actually get all pages/data from the API, you could run your process in a loop and also give it a callback method to execute as it gets each page of data from the API.
Here is a sample:
public class Program
{
public static void Main(string[] args)
{
var jobs = GetDiceJobsAsync(Program.ResultCallBack).Result;
Console.WriteLine($"\nAll {jobs.Count} jobs displayed");
Console.ReadLine();
}
private static async Task<List<DiceApiJob>> GetDiceJobsAsync(Action<DiceApiJobWrapper> callBack = null)
{
var jobs = new List<DiceApiJob>();
HttpClient httpClient = new HttpClient();
httpClient.BaseAddress = new Uri("http://service.dice.com");
var nextUrl = "/api/rest/jobsearch/v1/simple.json?skill=ruby";
do
{
await httpClient.GetAsync(nextUrl)
.ContinueWith(async (jobSearchTask) =>
{
var response = await jobSearchTask;
if (response.IsSuccessStatusCode)
{
string jsonString = await response.Content.ReadAsStringAsync();
var result = JsonConvert.DeserializeObject<DiceApiJobWrapper>(jsonString);
if (result != null)
{
// Build the full list to return later after the loop.
if (result.DiceApiJobs.Any())
jobs.AddRange(result.DiceApiJobs.ToList());
// Run the callback method, passing the current page of data from the API.
if (callBack != null)
callBack(result);
// Get the URL for the next page
nextUrl = (result.nextUrl != null) ? result.nextUrl : string.Empty;
}
}
else
{
// End loop if we get an error response.
nextUrl = string.Empty;
}
});
} while (!string.IsNullOrEmpty(nextUrl));
return jobs;
}
private static void ResultCallBack(DiceApiJobWrapper jobSearchResult)
{
if (jobSearchResult != null && jobSearchResult.count > 0)
{
Console.WriteLine($"\nDisplaying jobs {jobSearchResult.firstDocument} to {jobSearchResult.lastDocument}");
foreach (var job in jobSearchResult.DiceApiJobs)
{
Console.WriteLine(job.jobTitle);
Console.WriteLine(job.company);
}
}
}
}
Note that the above sample allows the callback method to access each page of data as it is received by the GetDiceJobsAsync method. In this case, the console, displays each page as it becomes available. If you do not want the callback option, you can simply pass nothing to GetDiceJobsAsync.
But the GetDiceJobsAsync also returns all the jobs when it completes. So you can choose to act on the whole list at the end of GetDiceJobsAsync.
As for reaching API limits, you can insert a small delay within the loop, right before you repeat the loop. But when I tried it, I did not encounter the API limiting my requests so I did not include it in the sample.
We're currently trying to use the Task<IAsyncEnumerator<StreamResult<T>>> StreamAsync<T>(IQueryable<T> query, CancellationToken token = null), running into some issues.
Our document look something like:
public class Entity
{
public string Id { get; set; }
public DateTime Created { get; set; }
public Geolocation Geolocation { get; set; }
public string Description { get; set; }
public IList<string> SubEntities { get; set; }
public Entity()
{
this.Id = Guid.NewGuid().ToString();
this.Created = DateTime.UtcNow;
}
}
In combination we've a view model, which is also the model were indexing:
public class EntityViewModel
{
public string Id { get; set; }
public DateTime Created { get; set; }
public Geolocation Geolocation { get; set; }
public string Description { get; set; }
public IList<SubEntity> SubEntities { get; set; }
}
And ofcourse, the index, with the resulttype inheriting from the viewmodel, to enable that SubEntities are mapped and output correctly, while enabling the addition of searchfeatures such as fulltext etc.:
public class EntityWithSubentitiesIndex : AbstractIndexCreationTask<Entity, EntityWithSubentitiesIndex.Result>
{
public class Result : EntityViewModel
{
public string Fulltext { get; set; }
}
public EntityWithSubentitiesIndex ()
{
Map = entities => from entity in entities
select new
{
Id = entity.Id,
Created = entity.Created,
Geolocation = entity.Geolocation,
SubEntities = entity.SubEntities.Select(x => LoadDocument<SubEntity>(x)),
Fulltext = new[]
{
entity.Description
}.Concat(entity.SubEntities.Select(x => LoadDocument<SubEntity>(x).Name)),
__ = SpatialGenerate("__geolokation", entity.Geolocation.Lat, entity.Geolocation.Lon)
};
Index(x => x.Created.Date, FieldIndexing.Analyzed);
Index(x => x.Fulltext, FieldIndexing.Analyzed);
Spatial("__geolokation", x => x.Cartesian.BoundingBoxIndex());
}
}
Finally we're querying like this:
var query = _ravenSession.Query<EntityWithSubentitiesIndex.Result, EntityWithSubentitiesIndex>()
.Customize(c =>
{
if (filter.Boundary == null) return;
var wkt = filter.Boundary.GenerateWkt().Result;
if (!string.IsNullOrWhiteSpace(wkt))
{
c.RelatesToShape("__geolokation", wkt, SpatialRelation.Within);
}
})
.AsQueryable();
// (...) and several other filters here, removed for clarity
var enumerator = await _ravenSession.Advanced.StreamAsync(query);
var list = new List<EntityViewModel>();
while (await enumerator.MoveNextAsync())
{
list.Add(enumerator.Current.Document);
}
When doing so we're getting the following exception:
System.InvalidOperationException: The query results type is 'Entity'
but you expected to get results of type 'Result'. If you want to
return a projection, you should use
.ProjectFromIndexFieldsInto() (for Query) or
.SelectFields() (for DocumentQuery) before calling to
.ToList().
According to the documentation, the Streaming API should support streaming via an index, and querying via an IQueryable at once.
How can this be fixed, while still using an index, and the streaming API, to:
Prevent having to page through the normal query, to work around the default pagesize
Prevent having to load the subentities one at a time when querying
Thanks in advance!
Try to use:
.As<Entity>()
(or .OfType<Entity>()) in your query. That should work in the regular stream.
This is a simple streaming query using "TestIndex" that is an index over an entity Test and I'm using a TestIndex.Result to look like your query. Note that this is actually not what the query will return, it's only there so you can write typed queries (ie. .Where(x => x.SomethingMapped == something))
var queryable = session.Query<TestIndex.Result, TestIndex>()
.Customize(c =>
{
//do stuff
})
.As<Test>();
var enumerator = session.Advanced.Stream(queryable);
while (enumerator.MoveNext())
{
var entity = enumerator.Current.Document;
}
If you instead want to retrieve the values from the index and not the actual entity being indexed you have to store those as fields and then project them into a "view model" that matches your mapped properties. This can be done by using .ProjectFromIndexFieldsInto<T>() in your query. All the stored fields from the index will be mapped to the model you specify.
Hope this helps (and makes sense)!
Edit: Updated with a, for me, working example of the Streaming API used with ProjectFromIndexFieldsInto<T>() that returns more than 128 records.
using (var session = store.OpenAsyncSession())
{
var queryable = session.Query<Customers_ByName.QueryModel, Customers_ByName>()
.Customize(c =>
{
//just to do some customization to look more like OP's query
c.RandomOrdering();
})
.ProjectFromIndexFieldsInto<CustomerViewModel>();
var enumerator = await session.Advanced.StreamAsync(queryable);
var customerViewModels = new List<CustomerViewModel>();
while (await enumerator.MoveNextAsync())
{
customerViewModels.Add(enumerator.Current.Document);
}
Console.WriteLine(customerViewModels.Count); //in my case 504
}
The above code works great for me. The index has one property mapped (name) and that property is stored. This is running the latest stable build (3.0.3800).
As #nicolai-heilbuth stated in the comments to #jens-pettersson's answer, it seems to be a bug in the RavenDB client libraries from version 3 onwards.
Bug report filed here: http://issues.hibernatingrhinos.com/issue/RavenDB-3916
I'm following this guide and I am getting an error. Can anyone help me?
The code for my datamodel is below
namespace Datalayer {
public class DataModel {
public DataModel()
{
using (btWholesaleDataContext db = new btWholesaleDataContext()) {
//! requires auth
var MACRequestList = from r in db.btRequests
select new Models.BT.Request {
ID = r.ID,
Date = r.DateTime,
StatusCode = 3,
Status = r.Status
};
MACRequests = MACRequestList.AsQueryable();
}
}
public IQueryable<Models.BT.Request> MACRequests { get; private set; }
}
}
The web service gives the error
Cannot access a disposed
object.Object name: 'DataContext
accessed after Dispose.'
When I access MACRequests
I have only posted the code I think is broken. If you want to see more just let me know.
Your data context is being disposed at the end of your constructor, at the end of the using { } block. However when you use the IQueryable MACRequests property, it needs that underlying context, which has since been disposed.
One possible way to handle this is to make your class IDisposable and dispose the context that way:
public class DataModel : IDisposable {
private btWholesaleDataContext wholesaleDataContext;
public DataModel()
{
wholesaleDataContext = new btWholesaleDataContext();
//! requires auth
var MACRequestList = ... ;
MACRequests = MACRequestList.AsQueryable();
}
public IQueryable<Models.BT.Request> MACRequests { get; private set; }
public void Dispose() {
if(wholesaleDataContext != null)
wholesaleDataContext.Dispose();
}
}
Then you have to make sure that DataModel is properly disposed by whatever uses it.
Another alternative is to make MACRequests the actual list of items instead of the IQueryable:
public class DataModel {
public DataModel()
{
using (btWholesaleDataContext db = new btWholesaleDataContext()) {
//! requires auth
var MACRequestList = ... ;
MACRequests = MACRequestList.ToList(); // ToList reads the records now, instead of later.
}
}
public List<Models.BT.Request> MACRequests { get; private set; }
}
I think its because you are using an IQueryable<>. Its lazily queries the service.
Use List<> instead so that it queries immediately
Or make "btWholesaleDataContext db" into a member variable
Queries to MACRequests are deferred - once you're out of the using block and your DataContext is disposed you're not going to be able to make the query you want.
You're creating the data context in a using block in the constructor of your DataModel...so by the time you access the MACRequests, the data context has been disposed.
Consider the following:
public class DataModel : IDisposable {
btWholesaleDataContext db = new btWholesaleDataContext();
public void Dispose()
{
btWholesaleDataContext.Dipose();
}
public IQueryable<Models.BT.Request> MACRequests {
get {
return from r in db.btRequests
select new Models.BT.Request {
ID = r.ID,
Date = r.DateTime,
StatusCode = 3,
Status = r.Status
};
}
}
}
Note that this usage will work:
using (var dm = new DataModel())
{
dm.MACRequests.ToArray();
}
but this will fail for the same reason as the original:
IQueryable<Models.BT.Request> requests = null;
using (var dm = new DataModel())
{
requests = dm.MACRequests;
}
// this will fail because the context is disposed by the time we force enumeration of the query
requests.ToArray();
...Alternatively, since WCF data services can't filter on projections, and thus all you can really do with the query
from r in db.btRequests
select new Models.BT.Request {
ID = r.ID,
Date = r.DateTime,
StatusCode = 3,
Status = r.Status
};
is execute it...
just consider changing your original code to return an array or a list instead of leaving it as a queryable.