With some earlier help, I created a C# script in SSIS to retrieve data from MongoDB to SQL Server. While regular documents are retrieved easily, nested documents and arrays are problematic.
Problem 1: I have shipping_address.country that returns results by using
this.UserDBBuffer.SCountry = document["shipping_address"].AsBsonDocument["country"].ToString();
However, mlocation.address gives me an error '"country" not found' using the same code:
this.UserDBBuffer.Country = document["mlocation"].AsBsonDocument["country"].ToString();
Problem 2: Retrieving items from arrays. I have an array that looks like "devices -> Document -> device_data -> model" or "devices -> Document -> device_data -> brand". How do I retrieve "model" or "brand" values in my code?
Thanks a lot for your help. Below is my entire code:
public override void CreateNewOutputRows()
{
string connectionString = "mongodb://localhost";
MongoServer myMongo = MongoServer.Create(connectionString);
myMongo.Connect();
var db = myMongo.GetDatabase("UserDB");
//Declaring variables for Date Created conversions
string DateCreatedString;
DateTime DateCreatedDateUTC;
DateTime DateCreatedDateLocal;
var fields = Fields.Include("mlocation.country", "mlocation", "_id", "primary_email", "gender", "date_created");
var collection = db.GetCollection<BsonDocument>("users");
foreach (var document in collection.FindAll().SetFields(fields))
{
this.UserDBBuffer.AddRow();
this.UserDBBuffer.ID = document["_id"] == null ? "" : document["_id"].ToString();
this.UserDBBuffer.Country = document["mlocation"].AsBsonDocument["country"].ToString();
this.UserDBBuffer.PrimaryEmail = document["primary_email"] == null ? "" : document["primary_email"].ToString();
this.UserDBBuffer.Gender = document["gender"] == null ? "" : document["gender"].ToString();
//Importing Date Created as String for data manipulation
DateCreatedString = document["date_created"] == null ? "" : document["date_created"].ToString();
//First, making sure that we have a UTC datetime
DateCreatedDateUTC = DateTime.Parse(DateCreatedString).ToUniversalTime();
//Second, converting to Local Time
DateCreatedDateLocal = DateTime.Parse(DateCreatedString).ToLocalTime();
//Finally, assigning variables to rows
this.UserDBBuffer.DateTimeCreatedUTC = DateCreatedDateUTC;
this.UserDBBuffer.DateTimeCreatedLocal = DateCreatedDateLocal;
}
myMongo.Disconnect();
}
For Problem 2, I found a Java Script that someone used; if I can convert it to C#, it might help a lot:
count = 0;
function user_list(){
var cursor = db.users.find()
//var cursor = db.users.find({"devices": {"$ne":[]}})
cursor.forEach(function(user) {
var deviceInfo = "";
if (user.devices){
if (user.devices[0]){
dd = user.devices[0].device_data;
if (dd) {
deviceInfo = dd.model + "," + dd.brand + "," + dd.model + "," + dd.device + "," + dd.pixel_height + "," + dd.pixel_width + "," + dd.pixel_format;
}
}
}
var location = "";
if (user.mlocation) location = user.mlocation.country;
print(user._id + "," + location + "," + user.primary_email + "," + user.date_created + "," + deviceInfo);
count++;
});
}
user_list();
print(count);
For problem 1, are you sure all docs contain a field mlocation that is a document containing the country field. I was able to reproduce the "Element country not found" with a document that is missing the value.
e.g. with
db.users.find()
{ "_id" : ObjectId("4f04c56a0f8fa4413bed1078"), "primary_email" : "email#email.com", "shipping_address" : [ {"country" : "USA", "city" : "San Francisco" }, { "country" : "IN", "city" : "Chennai" } ], "mlocation" : { "country" : "Canada", "city" : "Montreal" } }
{ "_id" : ObjectId("4f04d1605ab5a3805aaa8666"), "primary_email" : "incorrect#email.com", "shipping_address" : [ { "country" : "MX", "city" : "Cabo San Lucas" } ], "mlocation" : { "city" : "Montreal" } }
the 2nd document throws the exception. You can either check for its existance or use the default value option
document["mlocation"].AsBsonDocument.GetValue("country", null)
For problem 2, you cannot cast a BsonArray as a document. So for the above e.g to get shipping_address.country you can do something like
foreach (var addr in document["shipping_address"].AsBsonArray)
{
var country = addr.AsBsonDocument["country"].AsString;
}
Assuming the devices element is an array, just drill your way down into the element you are looking for, like this:
BsonDocument document; // assume this comes from somewhere
var devices = document["devices"].AsBsonArray;
var device = devices[0].AsBsonDocument; // first element of array
var deviceData = device["device_data"].AsBsonDocument;
var model = deviceData["model"].AsString;
var brand = deviceData["brand"].AsString;
I've broken it down into steps for clarity, but you can combine some of these steps into longer statements if you want.
To clarify your comment to Robert's answer, you can use BsonDocument.Contains to check if a given BsonDocument contains a field of the specified name before getting its value (http://api.mongodb.org/csharp/current/html/6181f23f-f6ce-fc7d-25a7-fc682ffd3c04.htm)
Instead of:
var mlocation = document["mlocation"].AsBsonDocument;
var country = "";
if (mlocation != null && mlocation.Contains("country"))
{
country = mlocation.AsBsonDocument.GetValue("country").ToString();
}
I would write:
var mlocation = document["mlocation"].AsBsonDocument;
var country = "";
if (mlocation.Contains("country"))
{
country = mlocation["country"].AsString;
}
And instead of:
var devices = document["devices"].AsBsonArray;
if (devices.ToList().Count > 0)
{
if (devices[0].AsBsonDocument != null)
{
var deviceinfo = devices[0].AsBsonDocument;
if (deviceinfo["device_data"].AsBsonDocument != null)
{
var deviceData = deviceinfo["device_data"].AsBsonDocument;
model = deviceData.GetValue("model", null).AsString;
}
}
}
I would write:
var devices = document["devices"].AsBsonArray;
if (devices.Count > 0)
{
var deviceinfo = devices[0].AsBsonDocument;
if (deviceinfo.Contains("device_data"))
{
var deviceData = deviceinfo["device_data"].AsBsonDocument;
var model = deviceData.GetValue("model", "").AsString; // "" instead of null
}
}
Related
I am trying to write a query in neo4jclient to add/update all parts of a hierarchy tree into the database in 1 hit.
I am having trouble when it comes to using merge on collections though.
I see the foreach function but haven't been successful getting it to work.
So far I've tried the below
I've tried writing it as a full string but keep getting an exception of 'Neo4jClient.NeoException: SyntaxException: Invalid input '.': expected an identifier character, whitespace, '}' or ':' (line 6, column 90 (offset: 266))'
Below the string is how I imagine the code to work if the foreach function worked like c# but don't know the correct syntax.
Any help would be really appreciated.
public void Save(CypherAspirationsViewModel aspirations, string emailAddress)
{
var query = graphClient.Cypher
.Match("(user:User)")
.Where((CypherUser user) => user.EmailAddress == emailAddress)
// Aspirations
.Merge("user" + CypherRelationships.Aspirations + ">(aspirations:Aspirations {Guid: {aspirationsGuid}})")
.OnCreate()
.Set("aspirations = {aspirations}")
.WithParams(new
{
aspirationsGuid = aspirations.CypherAspirations.Guid,
aspirations = aspirations.CypherAspirations
});
// Permanent Remuneration
if (aspirations.CypherPermanentRemuneration != null) {
query = query.Merge("aspirations" + CypherRelationships.PermanentRemuneration + ">(permanentRemuneration:Remuneration {Guid: {permanentRemunerationGuid}})")
.OnCreate()
.Set("permanentRemuneration = {permanentRemuneration}")
.WithParams(new
{
permanentRemunerationGuid = aspirations.CypherPermanentRemuneration.Guid,
permanentRemuneration = aspirations.CypherPermanentRemuneration
});
}
// Contract Remuneration
if (aspirations.CypherContractRemuneration != null) {
query = query.Merge("aspirations" + CypherRelationships.ContractRemuneration + ">(contractRemuneration:Remuneration {Guid: {contractRemunerationGuid}})")
.OnCreate()
.Set("contractRemuneration = {contractRemuneration}")
.WithParams(new
{
contractRemunerationGuid = aspirations.CypherContractRemuneration.Guid,
contractRemuneration = aspirations.CypherContractRemuneration
});
}
// Consultant Remuneration
if(aspirations.CypherConsultantRemuneration != null)
{
query = query.Merge("aspirations" + CypherRelationships.ConsultantRemuneration + ">(consultantRemuneration:Remuneration {Guid: {consultantRemunerationGuid}})")
.OnCreate()
.Set("consultantRemuneration = {consultantRemuneration}")
.WithParams(new
{
consultantRemunerationGuid = aspirations.CypherConsultantRemuneration.Guid,
consultantRemuneration = aspirations.CypherConsultantRemuneration
});
}
// Locations
if (aspirations.CypherLocations != null)
{
//string forEachString = "(n in {locations} | merge (aspirations " + CypherRelationships.Location + ">(location:Location {Guid: {n.Guid}})) on create set location.Name = n.Name, location.Longitude = n.Longitude, location.Latitude = n.Latitude)";
//query = query
// .ForEach(forEachString)
// .WithParam("locations", aspirations.CypherLocations);
query = query.ForEach("CypherLocation location in aspirations.CypherLocations")
.Merge("aspirations" + CypherRelationships.Location + ">(location:Location {Guid: {locationGuid}})")
.OnCreate()
.Set("location = {location}")
.WithParams(new
{
locationGuid = location.Guid,
location = location
});
}
query.ExecuteWithoutResults();
}
Cypher Query:
MATCH (user:User)
WHERE (user.EmailAddress = "email")
MERGE user-[:ASPIRATIONS]->(aspirations:Aspirations {Guid: "0d700793-4702-41ee-99f1-685472e65e51"})
ON CREATE
SET aspirations = {
"FullTime": true,
"PartTime": false,
"Permanent": false,
"Contract": false,
"Consultant": false,
"WillingToRelocate": false,
"CommuteEnum": 40,
"Guid": "0d700793-4702-41ee-99f1-685472e65e51"
}
FOREACH (n in [
{
"Name": "Location1",
"Longitude": 10.0,
"Latitude": 1.0,
"Guid": "a9f25fda-9559-4723-80ec-d8711a260adc"
}
] |
merge (aspirations -[:LOCATION]->(location:Location {Guid: {n.Guid}}))
on create set location.Name = n.Name, location.Longitude = n.Longitude, location.Latitude = n.Latitude)
Ok, I think I've got it nailed. There are a few Cypher changes needed (depending on server version)
First off, the original error is from the final Merge in the ForEach - you have {Guid: {n.Guid}} but you don't need the extra {} - so it should be: {Guid: n.Guid}.
Once you've got that, if you're going against a 3.0 DB, you'll need to add some parentheses in your merge methods, for example:
.Merge("user" + CypherRelationships.Aspirations + ">(aspirations:Aspirations {Guid: {aspirationsGuid}})")
should become:
.Merge("user" + CypherRelationships.AspirationsWithClosingParentheses + ">(aspirations:Aspirations {Guid: {aspirationsGuid}})")
where AspirationsWithClosingParentheses is something like:
var AspirationsWithClosingParentheses = "Aspirations)--"
You'll need to do that for every merge, as 3.0 requires the identifiers to be surrounded!
How can I convert this Mongo Shell script to MongoDB C# Driver?
var myItems = []
var myCursor = db.Tickets.aggregate(
[
{ $match : { TicketProjectID : 49 } },
{ $project: { TicketProjectID:1, TicketID:1, concatValue: { $concat: [ "$Status", " - ", "$Name" ] } } }
// I will have a list of fields that I need to concatenate at run time. So C# query should support concatenation for "N" number of fields at run-time.
//{ $group: { _id: null, count: { $sum: 1 } } }
],
{ allowDiskUse: true }
)
//This seems like a ugly performance approach when we are working against 100k results with above match
while (myCursor.hasNext()) {
var item = myCursor.next();
if(item.concatValue.search(/mysearchkey/i) > -1)
{
myItems.push(item.TicketID)
}
}
myItems
or is there a better way to do the string search in concatenated projection instead of foreach in cursor, as some quires might get 50k records.
This is what I have tried so far, (Not using Aggregation)
Note: Trimmed this code to suite for public Q&A sites. So please consider this as Pseudo-code
var tickets = ticketsCollection.FindSync(filter).ToList();
string concatinatedValue = string.Empty;
foreach (var ticket in tickets)
{
foreach (var field in customFieldsForThisProject)
concatinatedValue += ticket[field.Replace(" ", "_")];
if(concatinatedValue.StripHtml().contains("MysearchWord"))
{
TikectIdList.Add(ticket["TicketID"])
}
}
Thanks to #Nikola.Lukovic, working on his pseudo-code, I came up with this working solution.
Approach one: fully using C# Driver
var ticketsCollection = _mongoConnect.Database.GetCollection<BsonDocument>("Tickets");
var dbResult = from ticket in ticketsCollection.AsQueryable()
select new
{
TicketProjectID = ticket["TicketProjectID"],
TicketID = ticket["TicketID"],
ConcatValue = ticket["Status"] + (string) ticket["Name"]
};
var matches = from dbr in dbResult
where dbr.ConcatValue.Contains(searchKey)
where dbr.ConcatValue.StartsWith(searchKey)
select dbr;
This will not work for my scenario as fields I am trying to
concatenate are if type string, but $add will only work with
numeric and date types.
Approach two: using RunCommand and passing straight Shell command. This will work for all datatypes. And works for my need as well.
var projectCommand =
BsonDocument.Parse(
"{ $project: { _id: -1, TicketProjectID:1, TicketID:1, concatValue: { $concat: [ \"$Status\", \" - \", \"$Name\" ] } } }");
var matchCommand =
BsonDocument.Parse("{ $match: {concatValue: { $regex: '" + searchKey + "', $options: 'i'} } }");
var pipeline = new[] {projectCommand, matchCommand};
var result = ticketsCollection.Aggregate<BsonDocument>(pipeline).ToList();
if (result.Count > 0)
return result.Select(x => (int)x["TicketID"]).ToList();
return null;
Edited according to the given comment
If you can use AsQueryable() you can get the values like this:
var dbResult = from ticket in ticketsCollection.AsQueryable()
where ticket.TicketProjectID == 49
select new
{
TicketProjectID = ticket.TicketProjectID,
TicketID = ticket.TicketID,
ConcatValue = ticket.Status + " - " + ticket.Name
};
and than later you can do something like this:
var result = from dbr in dbResult
where dbr.ConcatValue.Contains("something") //or
where dbr.ConcatValue.StartsWith("something")//or you can use regex
select dbr;
Note: For some reason both Status and Name properties from type Ticket need to be of a type String for concatenation to work since mongo driver won't recognize the call to ToString() from some other type.
If you want to concatenate properties from some other types you could get them separately from the db and than concat them locally.
note, i'm not that good with mongo shell i could mess something up but you can see in which way you could go
Alternatively you could write your shell command like this and put it in a string:
var command = #"db.Tickets.aggregate(
[
{ $project: { TicketProjectID:1, TicketID:1, concatValue: { $concat: [ "$Status", " - ", "$Name" ] } } },
{ $match : { TicketProjectId : 49, concatValue : { $regex : /mysearchkey/i } } }
],
{ allowDiskUse : true }
);";
then execute it in c# with RunCommandAsync method from MongoDatabase.
var result = await mongoDatabase.RunCommandAsync<BsonDocument>(BsonDocument.Parse(command));
I have the following method
public static ProjectListItemViewModel CreateViewModel(P.Project project)
{
return new ProjectListItemViewModel
{
Id = project.Id,
ExpectedResult = Regex.Replace(project.ExpectedResult, #"<[^>]+>| ", string.Empty).Trim().Length > 100 ? Regex.Replace(project.ExpectedResult, #"<[^>]+>| ", string.Empty).Trim().Substring(0, 100) + "..." : Regex.Replace(project.ExpectedResult, #"<[^>]+>| ", string.Empty).Trim(),
Initiator = project.Initiator.FullName,
Status = project.Status.Name,
ProjectManager = project.ProjectManager != null ? project.ProjectManager.FullName :"",
};
}
My question is how to apply Regex to ExpectedResult only if it is not null in the shortest possible way?
I'd change your style a bit, to only execute the regex once:
// Define other methods and classes here
public static ProjectListItemViewModel CreateViewModel(P.Project project)
{
var rex = new Regex(#"<[^>]+>| ");
var expected = rex.Replace(project.ExpectedResult ?? string.Empty, string.Empty).Trim();
return new ProjectListItemViewModel
{
Id = project.Id,
ExpectedResult = expected.Length > 100 ? expected.Length.Substring(0, 100) : expected,
Initiator = project.Initiator.FullName,
Status = project.Status.Name,
ProjectManager = project.ProjectManager != null ? project.ProjectManager.FullName : "",
};
}
Try Like This:
String Result = !ExpectedResult ?? YourRegEx;
In this way It directly applies if the value is not null.
This is not tested
where as this works fine in general c#
String Result = ExpectedResult ?? YourRegEx;
Ok, so I am trying to send POST commands over an http connection, and using JSON formatting to do so. I am writing the program to do this in C#, and was wondering how I would format an array of values to be passed as JSON to the server.
Currently I have this:
new {name = "command" , index = "X", optional = "0"}
Which translates to this in JSON:
"name": "command",
"index": "X",
"optional": "0"
And I want to make an array, called items, where each element contains these three values. So it would essentially be an array of objects, in which the object contains a name, an index, and an optional field.
My guess was that it would be something along the lines of this:
new {items = [(name = "command" , index = "X", optional = "0"),
(name = "status" , index = "X", optional = "0")]}
Which, if it were correct syntax, would translate to this in JSON:
"items":
[
{
"name": "command",
"index": "X",
"optional": "0"
},
{
"name": "status",
"index": "X",
"optional": "0"
}
]
But, evidently I'm doing it wrong. Ideas? Any help is appreciated.
You're close. This should do the trick:
new {items = new [] {
new {name = "command" , index = "X", optional = "0"},
new {name = "command" , index = "X", optional = "0"}
}}
If your source was an enumerable of some sort, you might want to do this:
new {items = source.Select(item => new
{
name = item.Name, index = item.Index, options = item.Optional
})};
You'd better create some class for each item instead of using anonymous objects. And in object you're serializing you should have array of those items. E.g.:
public class Item
{
public string name { get; set; }
public string index { get; set; }
public string optional { get; set; }
}
public class RootObject
{
public List<Item> items { get; set; }
}
Usage:
var objectToSerialize = new RootObject();
objectToSerialize.items = new List<Item>
{
new Item { name = "test1", index = "index1" },
new Item { name = "test2", index = "index2" }
};
And in the result you won't have to change things several times if you need to change data-structure.
p.s. Here's very nice tool for complex jsons
Also , with Anonymous types ( I prefer not to do this) -- this is just another approach.
void Main()
{
var x = new
{
items = new[]
{
new
{
name = "command", index = "X", optional = "0"
},
new
{
name = "command", index = "X", optional = "0"
}
}
};
JavaScriptSerializer js = new JavaScriptSerializer(); //system.web.extension assembly....
Console.WriteLine(js.Serialize(x));
}
result :
{"items":[{"name":"command","index":"X","optional":"0"},{"name":"command","index":"X","optional":"0"}]}
new {var_data[counter] =new [] {
new{ "S NO": "+ obj_Data_Row["F_ID_ITEM_MASTER"].ToString() +","PART NAME": " + obj_Data_Row["F_PART_NAME"].ToString() + ","PART ID": " + obj_Data_Row["F_PART_ID"].ToString() + ","PART CODE":" + obj_Data_Row["F_PART_CODE"].ToString() + ", "CIENT PART ID": " + obj_Data_Row["F_ID_CLIENT"].ToString() + ","TYPES":" + obj_Data_Row["F_TYPE"].ToString() + ","UOM":" + obj_Data_Row["F_UOM"].ToString() + ","SPECIFICATION":" + obj_Data_Row["F_SPECIFICATION"].ToString() + ","MODEL":" + obj_Data_Row["F_MODEL"].ToString() + ","LOCATION":" + obj_Data_Row["F_LOCATION"].ToString() + ","STD WEIGHT":" + obj_Data_Row["F_STD_WEIGHT"].ToString() + ","THICKNESS":" + obj_Data_Row["F_THICKNESS"].ToString() + ","WIDTH":" + obj_Data_Row["F_WIDTH"].ToString() + ","HEIGHT":" + obj_Data_Row["F_HEIGHT"].ToString() + ","STUFF QUALITY":" + obj_Data_Row["F_STUFF_QTY"].ToString() + ","FREIGHT":" + obj_Data_Row["F_FREIGHT"].ToString() + ","THRESHOLD FG":" + obj_Data_Row["F_THRESHOLD_FG"].ToString() + ","THRESHOLD CL STOCK":" + obj_Data_Row["F_THRESHOLD_CL_STOCK"].ToString() + ","DESCRIPTION":" + obj_Data_Row["F_DESCRIPTION"].ToString() + "}
}
};
I'm trying to query a DynamoDB that i have created using code from the Amazon docs with a few simple modifications. I'm trying to take the data i get and write it to a log file as strings. But all i can seem to get is this:
2013-02-22 20:21:37.9268|Trace|[System.Collections.Generic.Dictionary2+KeyCollection[System.String,Amazon.DynamoDB.Model.AttributeValue] System.Collections.Generic.Dictionary2+ValueCollection[System.String,Amazon.DynamoDB.Model.AttributeValue]]|
I've tried a few different things but all return either the same thing, or something very similar.
The code i'm using:
private static void GetCallsForRange()
{
AmazonDynamoDBConfig config = new AmazonDynamoDBConfig();
config.ServiceURL = "http://dynamodb.us-west-2.amazonaws.com";
AmazonDynamoDBClient client = new AmazonDynamoDBClient(config);
DateTime startDate = DateTime.Today.AddDays(-21);
string start = startDate.ToString("G", DateTimeFormatInfo.InvariantInfo);
DateTime endDate = DateTime.Today;
string end = endDate.ToString("G", DateTimeFormatInfo.InvariantInfo);
QueryRequest request = new QueryRequest
{
TableName = "Inquiry",
HashKeyValue = new AttributeValue { S = "+15555555555" },
RangeKeyCondition = new Condition
{
ComparisonOperator = "BETWEEN",
AttributeValueList = new List<AttributeValue>()
{
new AttributeValue { S = start },
new AttributeValue { S = end }
}
}
};
QueryResponse response = client.Query(request);
QueryResult result = response.QueryResult;
foreach (Dictionary<string, AttributeValue> item in response.QueryResult.Items)
{
string logMsg = String.Format("[{0} {1}]", item.Keys, item.Values);
Logging.LogTrace(logMsg);
}
}
You will need to iterate over each item in the response.QueryResult.Items. You could rewrite your loop like this (taken from the Amazon DynamoDB documentation):
foreach (Dictionary<string, AttributeValue> item in response.QueryResult.Items)
{
LogItem(item);
}
private void LogItem(Dictionary<string, AttributeValue> attributeList)
{
foreach (KeyValuePair<string, AttributeValue> kvp in attributeList)
{
string attributeName = kvp.Key;
AttributeValue value = kvp.Value;
string logValue =
(value.S == null ? "" : value.S) +
(value.N == null ? "" : value.N.ToString()) +
(value.B == null ? "" : value.B.ToString()) +
(value.SS == null ? "" : string.Join(",", value.SS.ToArray())) +
(value.NS == null ? "" : string.Join(",", value.NS.ToArray())) +
(value.BS == null ? "" : string.Join(",", value.BS.ToArray()));
string logMsg = string.Format("[{0} {1}]", attributeName, logValue);
Logging.LogTrace(logMsg);
}
}
Essentially, you need to discover the "type" of the AttributeValue(String, Number, Binary, StringSet, NumberSet, BinarySet) and then output that to your log.
I hope that helps!