How can I convert this Mongo Shell script to MongoDB C# Driver?
var myItems = []
var myCursor = db.Tickets.aggregate(
[
{ $match : { TicketProjectID : 49 } },
{ $project: { TicketProjectID:1, TicketID:1, concatValue: { $concat: [ "$Status", " - ", "$Name" ] } } }
// I will have a list of fields that I need to concatenate at run time. So C# query should support concatenation for "N" number of fields at run-time.
//{ $group: { _id: null, count: { $sum: 1 } } }
],
{ allowDiskUse: true }
)
//This seems like a ugly performance approach when we are working against 100k results with above match
while (myCursor.hasNext()) {
var item = myCursor.next();
if(item.concatValue.search(/mysearchkey/i) > -1)
{
myItems.push(item.TicketID)
}
}
myItems
or is there a better way to do the string search in concatenated projection instead of foreach in cursor, as some quires might get 50k records.
This is what I have tried so far, (Not using Aggregation)
Note: Trimmed this code to suite for public Q&A sites. So please consider this as Pseudo-code
var tickets = ticketsCollection.FindSync(filter).ToList();
string concatinatedValue = string.Empty;
foreach (var ticket in tickets)
{
foreach (var field in customFieldsForThisProject)
concatinatedValue += ticket[field.Replace(" ", "_")];
if(concatinatedValue.StripHtml().contains("MysearchWord"))
{
TikectIdList.Add(ticket["TicketID"])
}
}
Thanks to #Nikola.Lukovic, working on his pseudo-code, I came up with this working solution.
Approach one: fully using C# Driver
var ticketsCollection = _mongoConnect.Database.GetCollection<BsonDocument>("Tickets");
var dbResult = from ticket in ticketsCollection.AsQueryable()
select new
{
TicketProjectID = ticket["TicketProjectID"],
TicketID = ticket["TicketID"],
ConcatValue = ticket["Status"] + (string) ticket["Name"]
};
var matches = from dbr in dbResult
where dbr.ConcatValue.Contains(searchKey)
where dbr.ConcatValue.StartsWith(searchKey)
select dbr;
This will not work for my scenario as fields I am trying to
concatenate are if type string, but $add will only work with
numeric and date types.
Approach two: using RunCommand and passing straight Shell command. This will work for all datatypes. And works for my need as well.
var projectCommand =
BsonDocument.Parse(
"{ $project: { _id: -1, TicketProjectID:1, TicketID:1, concatValue: { $concat: [ \"$Status\", \" - \", \"$Name\" ] } } }");
var matchCommand =
BsonDocument.Parse("{ $match: {concatValue: { $regex: '" + searchKey + "', $options: 'i'} } }");
var pipeline = new[] {projectCommand, matchCommand};
var result = ticketsCollection.Aggregate<BsonDocument>(pipeline).ToList();
if (result.Count > 0)
return result.Select(x => (int)x["TicketID"]).ToList();
return null;
Edited according to the given comment
If you can use AsQueryable() you can get the values like this:
var dbResult = from ticket in ticketsCollection.AsQueryable()
where ticket.TicketProjectID == 49
select new
{
TicketProjectID = ticket.TicketProjectID,
TicketID = ticket.TicketID,
ConcatValue = ticket.Status + " - " + ticket.Name
};
and than later you can do something like this:
var result = from dbr in dbResult
where dbr.ConcatValue.Contains("something") //or
where dbr.ConcatValue.StartsWith("something")//or you can use regex
select dbr;
Note: For some reason both Status and Name properties from type Ticket need to be of a type String for concatenation to work since mongo driver won't recognize the call to ToString() from some other type.
If you want to concatenate properties from some other types you could get them separately from the db and than concat them locally.
note, i'm not that good with mongo shell i could mess something up but you can see in which way you could go
Alternatively you could write your shell command like this and put it in a string:
var command = #"db.Tickets.aggregate(
[
{ $project: { TicketProjectID:1, TicketID:1, concatValue: { $concat: [ "$Status", " - ", "$Name" ] } } },
{ $match : { TicketProjectId : 49, concatValue : { $regex : /mysearchkey/i } } }
],
{ allowDiskUse : true }
);";
then execute it in c# with RunCommandAsync method from MongoDatabase.
var result = await mongoDatabase.RunCommandAsync<BsonDocument>(BsonDocument.Parse(command));
Related
As the following image, the collection is Posts. The _ids for the post and the comment are known.
So, How to check if a given value d exists in the AgreedUserIds?
Try this for single character search:
db.Posts.find(
{
_id: "b4d5...ff79",
Comments: {
$elemMatch: {
_id: "c1ea...d45",
AgreedUserIds: "d"
}
}
}
);
And this for multi character search:
db.Posts.find(
{
_id: "b4d5...ff79",
Comments: {
$elemMatch: {
_id: "c1ea...d45",
AgreedUserIds: { $in: ["c", "d"] }
}
}
}
);
If you want to keep it BsonDocument you have options to combine filters like this>
var client = new MongoClient();
var database = client.GetDatabase("test");
var collection = database.GetCollection<BsonDocument>("posts");
var postId = new BsonObjectId(new ObjectId("604b5ff389ff6887d1b91a93"));
var commentId = new BsonObjectId(new ObjectId("604b5ff389ff6887d1b91a92"));
var userId = "a";
//var userId = "e"; // not found
var postIdMatches = Builders<BsonDocument>.Filter.Eq("_id", postId);
var commentIdMatches = Builders<BsonDocument>.Filter.Eq("_id", commentId);
var userIdInside = Builders<BsonDocument>.Filter.AnyEq("AgreedUserIds", userId); // "a" in checking
var commentCheck = Builders<BsonDocument>.Filter.ElemMatch("Comments", userIdInside & commentIdMatches); // & combines filters. Also ElemMatch on any of the Comments is passing the criteria
var rows = (await collection.FindAsync<BsonDocument>(postIdMatches & commentCheck)).ToList();
You could also GetCollection<Post> with C# Post class describing the Post business object, and use .Any Linq query on that.
I'm stuck at a problem, apparently I might have misunderstood something, but here is the problem.
I've got some testdata in a database, but when I run this:
Connection to elastic
private string uri = "http://localhost:9200/";
private string testindex = "testconnectiones";
private static ElasticClient GetClient(string testindex, string uri)
{
var pool = new SingleNodeConnectionPool(new Uri(uri));
var connectionSettings =
new ConnectionSettings(pool, sourceSerializer: JsonNetSerializer.Default).DefaultIndex(testindex);
return new ElasticClient(connectionSettings);
}
Then
public void TestCreateIndexBaseOnOrderId1()
{
var client = GetClient(testindex, uri);
var searchResponse = client.Search<TestLogs>(s => s
.AllTypes()
.From(0)
.Size(1000)
.Analyzer("standard")
.Query(q => q
.Match(m => m
.Field(f => f.OrderID)
.Query("")
)
)
);
var eventTestArray = searchResponse.Documents.Select(x => new {x.OrderID }).ToArray();
Console.WriteLine("searchResponse.Documents.Count: " + searchResponse.Documents.Count());
var i = 0;
var j = 0;
foreach (var s in eventTestArray)
{
Console.WriteLine($"{i}: " + s);
i++;
}
Assert.AreNotEqual(eventTestArray, null);
}
the output is only:
searchResponse.Documents.Count: 0.
There should be 10 orders in the search response.
I just wonder if I've misunderstood something.
The data in the database looks for example like this
{
"_index": "testconnectiones",
"_type": "logs",
"_id": "1",
"_version": 1,
"found": true,
"_source": {
"OrderId": 1,
"Event": "CreateOrder"
}
}
When using the type TestLogs, how does the client know to target testconnectiones index?
There are a few things to consider:
Ensuring that testconnectiones index is queried
Ensuring that f => f.OrderID will serialize to a string value to target a field that exists for the type and index in Elasticsearch. By default, NEST camel cases property names when serializing
Ensuring that the query has an input so that it is not considered "conditionless", or using .Verbatim() to serialize the query exactly as is
Ensuring that indexed documents are available for search
A part of the problem was now solved went by.
QueryContainer query = new TermQuery()
{
Field = "OrderId",
Value = "1"
};
var searchRequest = new SearchRequest(index: "testindex")
{
Query = query
};
var searchResult = client.Search<TestLogs>(searchRequest);
foreach (var s in orderIdArray)
{
Console.WriteLine($"{i}: OrderId:" + s.OrderID + " Event: " + s.Event + " Time: " + s.TimeStamp);
i++;
}
Now is the problem only that if i change
QueryContainer query = new TermQuery()
{
Field = "OrderId",
Value = "1"
};
to...
QueryContainer query = new TermQuery()
{
Field = "OrderId",
Value = "1"
};
It wont give any output is Test method ELK_algorithmsTests.TestIndexCreation.testToMakeSameSelectionButWithSelectedEvent threw exception:
System.IndexOutOfRangeException: ..
any suggestion where is should look? because im out of ideas right now..
Can someone please help me translate the following into code for the mongodb 2.4 C# driver? I have been trying to get this to work but I'm having a hard time figuring it out. When I run the following mongo query through the C# drive, I get no records. When I run through the mongo command line I get back 2 records.
I just need to understand what I am doing that is causing the difference in behavior when using the C# driver.
So far I have the following for the C# driver:
public IEnumerable<ComponentRecordDataModel> GetComponentRecords(
)
{
var componentTypeFilter = Builders<ComponentRecordDataModel>.Filter.Eq(x => x.ComponentType, "Investment");
var authorizedUserFilter =
Builders<ComponentRecordDataModel>.Filter.AnyEq(x => x.AuthorizedUserIds, "59e8c1d35e13de1494887658");
var componentFieldFilters = new List<FieldValueDataModel>
{
new FieldValueDataModel()
{
FieldId = "59d664d1c153f67518f98888",
Value = "gov"
},
new FieldValueDataModel()
{
FieldId = "59d664d1c153f67518f98889",
Value = "azure"
}
};
var componentFilters = new List<FilterDefinition<FieldValueDataModel>>();
foreach (var componentFieldFilter in componentFieldFilters)
{
var bsonRegex = new BsonRegularExpression(Regex.Escape(componentFieldFilter.Value), "i");
componentFilters.Add(Builders<FieldValueDataModel>.Filter.And(new[]
{
Builders<FieldValueDataModel>.Filter.Eq(x => x.FieldId, componentFieldFilter.FieldId),
Builders<FieldValueDataModel>.Filter.Eq(x => x.Value, bsonRegex)
}));
}
var fieldFilter = Builders<ComponentRecordDataModel>.Filter.ElemMatch(
x => x.Fields, Builders<FieldValueDataModel>.Filter.Or(componentFilters)
);
var filter = componentTypeFilter & authorizedUserFilter & fieldFilter;
var renderedFilter = filter.ToJson();
return MongoContext.Find(filter).ToList();
}
/The mongodb query I am trying to replication is shown below/
db.getCollection('componentRecords').aggregate(
[{
$match: {
"componentType": "Investment"
}
},
{
$match: {
"authorizedUserIds": {
$elemMatch: {
$in: ["59e8c1d35e13de1494887658"]
}
}
}
},
{
$match: {
"fields": {
$elemMatch: { $or:
[{ $and : [
{ fieldId: ObjectId("59d664d1c153f67518f98888") },
{ value: { "$regex": "gov", '$options' : 'i' } }
]},
{ $and : [
{ fieldId: ObjectId("59d664d1c153f67518f98889") },
{ value: { "$regex": "azure", '$options' : 'i' } }
]}
]
}
}
}
}
])
Have you tried using linq ? it maybe easier to understand and write:
var arr = new [] { "59e8c1d35e13de1494887658" };
var id1 = ObjectId("59d664d1c153f67518f98888");
var gov = "gov";
var id2 = ObjectId("59d664d1c153f67518f98889");
var azure = "azure";
collection.AsQuerable().Where(w =>
w.componentType == "Investment" &&
w.authorizedUserIds.Any(a => arr.Contains(a.X)) &&
(
(w.fieldId == id1 && w.value.ToLower() == go".ToLower()) ||
(w.fieldId == id2 && w.value.ToLower() == azure.ToLower()) ||
)
).ToList();
Actually I didn't try this code in real example, but worked with linq in mongodb.
I am trying to write a query in neo4jclient to add/update all parts of a hierarchy tree into the database in 1 hit.
I am having trouble when it comes to using merge on collections though.
I see the foreach function but haven't been successful getting it to work.
So far I've tried the below
I've tried writing it as a full string but keep getting an exception of 'Neo4jClient.NeoException: SyntaxException: Invalid input '.': expected an identifier character, whitespace, '}' or ':' (line 6, column 90 (offset: 266))'
Below the string is how I imagine the code to work if the foreach function worked like c# but don't know the correct syntax.
Any help would be really appreciated.
public void Save(CypherAspirationsViewModel aspirations, string emailAddress)
{
var query = graphClient.Cypher
.Match("(user:User)")
.Where((CypherUser user) => user.EmailAddress == emailAddress)
// Aspirations
.Merge("user" + CypherRelationships.Aspirations + ">(aspirations:Aspirations {Guid: {aspirationsGuid}})")
.OnCreate()
.Set("aspirations = {aspirations}")
.WithParams(new
{
aspirationsGuid = aspirations.CypherAspirations.Guid,
aspirations = aspirations.CypherAspirations
});
// Permanent Remuneration
if (aspirations.CypherPermanentRemuneration != null) {
query = query.Merge("aspirations" + CypherRelationships.PermanentRemuneration + ">(permanentRemuneration:Remuneration {Guid: {permanentRemunerationGuid}})")
.OnCreate()
.Set("permanentRemuneration = {permanentRemuneration}")
.WithParams(new
{
permanentRemunerationGuid = aspirations.CypherPermanentRemuneration.Guid,
permanentRemuneration = aspirations.CypherPermanentRemuneration
});
}
// Contract Remuneration
if (aspirations.CypherContractRemuneration != null) {
query = query.Merge("aspirations" + CypherRelationships.ContractRemuneration + ">(contractRemuneration:Remuneration {Guid: {contractRemunerationGuid}})")
.OnCreate()
.Set("contractRemuneration = {contractRemuneration}")
.WithParams(new
{
contractRemunerationGuid = aspirations.CypherContractRemuneration.Guid,
contractRemuneration = aspirations.CypherContractRemuneration
});
}
// Consultant Remuneration
if(aspirations.CypherConsultantRemuneration != null)
{
query = query.Merge("aspirations" + CypherRelationships.ConsultantRemuneration + ">(consultantRemuneration:Remuneration {Guid: {consultantRemunerationGuid}})")
.OnCreate()
.Set("consultantRemuneration = {consultantRemuneration}")
.WithParams(new
{
consultantRemunerationGuid = aspirations.CypherConsultantRemuneration.Guid,
consultantRemuneration = aspirations.CypherConsultantRemuneration
});
}
// Locations
if (aspirations.CypherLocations != null)
{
//string forEachString = "(n in {locations} | merge (aspirations " + CypherRelationships.Location + ">(location:Location {Guid: {n.Guid}})) on create set location.Name = n.Name, location.Longitude = n.Longitude, location.Latitude = n.Latitude)";
//query = query
// .ForEach(forEachString)
// .WithParam("locations", aspirations.CypherLocations);
query = query.ForEach("CypherLocation location in aspirations.CypherLocations")
.Merge("aspirations" + CypherRelationships.Location + ">(location:Location {Guid: {locationGuid}})")
.OnCreate()
.Set("location = {location}")
.WithParams(new
{
locationGuid = location.Guid,
location = location
});
}
query.ExecuteWithoutResults();
}
Cypher Query:
MATCH (user:User)
WHERE (user.EmailAddress = "email")
MERGE user-[:ASPIRATIONS]->(aspirations:Aspirations {Guid: "0d700793-4702-41ee-99f1-685472e65e51"})
ON CREATE
SET aspirations = {
"FullTime": true,
"PartTime": false,
"Permanent": false,
"Contract": false,
"Consultant": false,
"WillingToRelocate": false,
"CommuteEnum": 40,
"Guid": "0d700793-4702-41ee-99f1-685472e65e51"
}
FOREACH (n in [
{
"Name": "Location1",
"Longitude": 10.0,
"Latitude": 1.0,
"Guid": "a9f25fda-9559-4723-80ec-d8711a260adc"
}
] |
merge (aspirations -[:LOCATION]->(location:Location {Guid: {n.Guid}}))
on create set location.Name = n.Name, location.Longitude = n.Longitude, location.Latitude = n.Latitude)
Ok, I think I've got it nailed. There are a few Cypher changes needed (depending on server version)
First off, the original error is from the final Merge in the ForEach - you have {Guid: {n.Guid}} but you don't need the extra {} - so it should be: {Guid: n.Guid}.
Once you've got that, if you're going against a 3.0 DB, you'll need to add some parentheses in your merge methods, for example:
.Merge("user" + CypherRelationships.Aspirations + ">(aspirations:Aspirations {Guid: {aspirationsGuid}})")
should become:
.Merge("user" + CypherRelationships.AspirationsWithClosingParentheses + ">(aspirations:Aspirations {Guid: {aspirationsGuid}})")
where AspirationsWithClosingParentheses is something like:
var AspirationsWithClosingParentheses = "Aspirations)--"
You'll need to do that for every merge, as 3.0 requires the identifiers to be surrounded!
With some earlier help, I created a C# script in SSIS to retrieve data from MongoDB to SQL Server. While regular documents are retrieved easily, nested documents and arrays are problematic.
Problem 1: I have shipping_address.country that returns results by using
this.UserDBBuffer.SCountry = document["shipping_address"].AsBsonDocument["country"].ToString();
However, mlocation.address gives me an error '"country" not found' using the same code:
this.UserDBBuffer.Country = document["mlocation"].AsBsonDocument["country"].ToString();
Problem 2: Retrieving items from arrays. I have an array that looks like "devices -> Document -> device_data -> model" or "devices -> Document -> device_data -> brand". How do I retrieve "model" or "brand" values in my code?
Thanks a lot for your help. Below is my entire code:
public override void CreateNewOutputRows()
{
string connectionString = "mongodb://localhost";
MongoServer myMongo = MongoServer.Create(connectionString);
myMongo.Connect();
var db = myMongo.GetDatabase("UserDB");
//Declaring variables for Date Created conversions
string DateCreatedString;
DateTime DateCreatedDateUTC;
DateTime DateCreatedDateLocal;
var fields = Fields.Include("mlocation.country", "mlocation", "_id", "primary_email", "gender", "date_created");
var collection = db.GetCollection<BsonDocument>("users");
foreach (var document in collection.FindAll().SetFields(fields))
{
this.UserDBBuffer.AddRow();
this.UserDBBuffer.ID = document["_id"] == null ? "" : document["_id"].ToString();
this.UserDBBuffer.Country = document["mlocation"].AsBsonDocument["country"].ToString();
this.UserDBBuffer.PrimaryEmail = document["primary_email"] == null ? "" : document["primary_email"].ToString();
this.UserDBBuffer.Gender = document["gender"] == null ? "" : document["gender"].ToString();
//Importing Date Created as String for data manipulation
DateCreatedString = document["date_created"] == null ? "" : document["date_created"].ToString();
//First, making sure that we have a UTC datetime
DateCreatedDateUTC = DateTime.Parse(DateCreatedString).ToUniversalTime();
//Second, converting to Local Time
DateCreatedDateLocal = DateTime.Parse(DateCreatedString).ToLocalTime();
//Finally, assigning variables to rows
this.UserDBBuffer.DateTimeCreatedUTC = DateCreatedDateUTC;
this.UserDBBuffer.DateTimeCreatedLocal = DateCreatedDateLocal;
}
myMongo.Disconnect();
}
For Problem 2, I found a Java Script that someone used; if I can convert it to C#, it might help a lot:
count = 0;
function user_list(){
var cursor = db.users.find()
//var cursor = db.users.find({"devices": {"$ne":[]}})
cursor.forEach(function(user) {
var deviceInfo = "";
if (user.devices){
if (user.devices[0]){
dd = user.devices[0].device_data;
if (dd) {
deviceInfo = dd.model + "," + dd.brand + "," + dd.model + "," + dd.device + "," + dd.pixel_height + "," + dd.pixel_width + "," + dd.pixel_format;
}
}
}
var location = "";
if (user.mlocation) location = user.mlocation.country;
print(user._id + "," + location + "," + user.primary_email + "," + user.date_created + "," + deviceInfo);
count++;
});
}
user_list();
print(count);
For problem 1, are you sure all docs contain a field mlocation that is a document containing the country field. I was able to reproduce the "Element country not found" with a document that is missing the value.
e.g. with
db.users.find()
{ "_id" : ObjectId("4f04c56a0f8fa4413bed1078"), "primary_email" : "email#email.com", "shipping_address" : [ {"country" : "USA", "city" : "San Francisco" }, { "country" : "IN", "city" : "Chennai" } ], "mlocation" : { "country" : "Canada", "city" : "Montreal" } }
{ "_id" : ObjectId("4f04d1605ab5a3805aaa8666"), "primary_email" : "incorrect#email.com", "shipping_address" : [ { "country" : "MX", "city" : "Cabo San Lucas" } ], "mlocation" : { "city" : "Montreal" } }
the 2nd document throws the exception. You can either check for its existance or use the default value option
document["mlocation"].AsBsonDocument.GetValue("country", null)
For problem 2, you cannot cast a BsonArray as a document. So for the above e.g to get shipping_address.country you can do something like
foreach (var addr in document["shipping_address"].AsBsonArray)
{
var country = addr.AsBsonDocument["country"].AsString;
}
Assuming the devices element is an array, just drill your way down into the element you are looking for, like this:
BsonDocument document; // assume this comes from somewhere
var devices = document["devices"].AsBsonArray;
var device = devices[0].AsBsonDocument; // first element of array
var deviceData = device["device_data"].AsBsonDocument;
var model = deviceData["model"].AsString;
var brand = deviceData["brand"].AsString;
I've broken it down into steps for clarity, but you can combine some of these steps into longer statements if you want.
To clarify your comment to Robert's answer, you can use BsonDocument.Contains to check if a given BsonDocument contains a field of the specified name before getting its value (http://api.mongodb.org/csharp/current/html/6181f23f-f6ce-fc7d-25a7-fc682ffd3c04.htm)
Instead of:
var mlocation = document["mlocation"].AsBsonDocument;
var country = "";
if (mlocation != null && mlocation.Contains("country"))
{
country = mlocation.AsBsonDocument.GetValue("country").ToString();
}
I would write:
var mlocation = document["mlocation"].AsBsonDocument;
var country = "";
if (mlocation.Contains("country"))
{
country = mlocation["country"].AsString;
}
And instead of:
var devices = document["devices"].AsBsonArray;
if (devices.ToList().Count > 0)
{
if (devices[0].AsBsonDocument != null)
{
var deviceinfo = devices[0].AsBsonDocument;
if (deviceinfo["device_data"].AsBsonDocument != null)
{
var deviceData = deviceinfo["device_data"].AsBsonDocument;
model = deviceData.GetValue("model", null).AsString;
}
}
}
I would write:
var devices = document["devices"].AsBsonArray;
if (devices.Count > 0)
{
var deviceinfo = devices[0].AsBsonDocument;
if (deviceinfo.Contains("device_data"))
{
var deviceData = deviceinfo["device_data"].AsBsonDocument;
var model = deviceData.GetValue("model", "").AsString; // "" instead of null
}
}