Related
I am extremely new to RavenDB and HangFire and I'm creating a background task but I get an indexing error when I'm querying ravendb documents as shown below :
public static void UpdateDoNotDisturb()
{
IList<HotelRoom> rooms = new List<HotelRoom>();
Hotel project;
IQueryable<HotelRoom> queryableObj;
//Update Rooms with DND status and DndUntil greater than now.
using (var session = MvcApplication.DocumentStore.OpenSession())
{
queryableObj = session.Query<HotelRoom, HotelRoom_Search>().Where(x => x.Status == "DND" && x.DndUntilUtc < DateTime.Now).OfType<HotelRoom>();
//queryableObj = session.Query<HotelRoom, HotelRoom_Search>().Where(x => x.Status == "DND").OfType<HotelRoom>();
using (var enumerator = session.Advanced.Stream(queryableObj))
{
while (enumerator.MoveNext())
{
var room = enumerator.Current.Document;
rooms.Add(room);
}
}
}
}
on the document :
{
"HotelId": "hotels/1474",
"RoomNumber": "407",
"Floor": "4th Floor",
"Status": "DND",
"RoomType": "Guest Room",
"RoomTemplate": "1474_Install_SBBs.json",
"Json": {
"main": {
"picRoomNumber": "Images/Photos/d5/d5de2652-08c3-49c7-a75a-446934962a1d.jpg",
"qrCode": "http://2d.my/36tn"
},
"taskSBB1": {
"deviceTvModelValue": "Other",
"tvLocation": "None",
"deviceTvModel": "Other",
"itemSbbRemoved": 0,
"deviceSbbQr": "http://2d.my/36tm",
"deviceTvMountType": "Articulating wall mount",
"deviceTvMountTypeValue": "Articulating wall mount",
"tvLocationValue": "",
"taskTechName": "Anton Anskin",
"taskName": "Install SBB",
"deviceTvSn": "703rmcj83595",
"taskStatus": "In Progress",
"deviceSbbPhotoMount": "Images/Photos/84/8410b77b-20f2-4146-b9cb-19e9dd3259ef.jpg",
"taskStarted": "2017-06-06T14:50:34.046000Z"
},
"tasks": [
"taskSBB1"
],
"survey": {
"sbbCount": "1",
"damageBoolean": 0
}
},
"LastUpdatedBy": "Mark",
"LastUpdatedUtc": "2017-07-10T18:46:27.9305296Z",
"StartedBy": null,
"StartedUtc": "2017-06-06T14:50:34.0460000Z",
"CompletedBy": null,
"CompletedUtc": "0001-01-01T00:00:00.0000000",
"CompletedDate": "0001-01-01T00:00:00.0000000",
"DndBy": "Mark",
"DndUntilUtc": "2017-07-10T21:46:00.0000000Z",
"RoomPhotos": [
{
"Key": "main",
"Name": "picRoomNumber",
"ImagePath": "Images/Photos/d5/d5de2652-08c3-49c7-a75a-446934962a1d.jpg",
"AddedBy": "Anton",
"AddedUtc": "2017-06-06T14:51:16.6010049Z"
},
{
"Key": "taskSBB1",
"Name": "deviceSbbPhotoMount",
"ImagePath": "Images/Photos/84/8410b77b-20f2-4146-b9cb-19e9dd3259ef.jpg",
"AddedBy": "Anton",
"AddedUtc": "2017-06-06T14:51:17.2572542Z"
}
],
"Issues": [],
"Scans": [],
"ChangeLog": [],
"CurrentPhase": null
}
And I'm getting the error :
"Error": "System.ArgumentException: The field 'DndUntilUtc' is not indexed, cannot query on fields that are not indexed\r\n at Raven.Database.Indexing.Index.AssertQueryDoesNotContainFieldsThatAreNotIndexed(IndexQuery indexQuery, AbstractViewGenerator viewGenerator)\r\n at Raven.Database.Indexing.Index.IndexQueryOperation.<Query>d__57.MoveNext()\r\n at Raven.Database.Util.ActiveEnumerable`1..ctor(IEnumerable`1 enumerable)\r\n at Raven.Database.DocumentDatabase.<>c__DisplayClass96.<Query>b__8b(IStorageActionsAccessor actions)\r\n at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action`1 action, EsentTransactionContext transactionContext)\r\n at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action)\r\n at Raven.Database.DocumentDatabase.Query(String index, IndexQuery query, CancellationToken externalCancellationToken, Action`1 headerInfo, Action`1 onResult)\r\n at Raven.Database.Server.Responders.QueryStreams.Respond(IHttpContext context)\r\n at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx)\r\n at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx)"
}
Source=Raven.Client.Lightweight
StackTrace:
at Raven.Client.Connection.HttpJsonRequest.RawExecuteRequest()
at Raven.Client.Connection.ServerClient.StreamQuery(String index, IndexQuery query, QueryHeaderInformation& queryHeaderInfo)
at Raven.Client.Document.DocumentSession.Stream[T](IDocumentQuery`1 query, QueryHeaderInformation& queryHeaderInformation)
at Raven.Client.Document.DocumentSession.Stream[T](IQueryable`1 query, QueryHeaderInformation& queryHeaderInformation)
at Raven.Client.Document.DocumentSession.Stream[T](IQueryable`1 query)
I'm curious as to how to fix this error and why. Thank you.
You're querying the index 'HotelRoom_Search' with a property 'DndUntilUtc' but the index doesn't contain it.
In lucene/ravendb world, if you query the index by specific property [, although the collection (or table in sql world) contains it], it would work only if the index contains it.
simply add the field 'DndUntilUtc' to your index.
your didn't included your index definition so i'll guess it's something like this :
public class HotelRoom_Search : AbstractIndexCreationTask<HotelRoom>
{
public HotelRoom_Search()
{
Map = rooms => from room in rooms
select new
{
//bla bla
room.DndUntilUtc // add this line
//bla bla
};
Indexes.Add(x => x.DndUntilUtc, FieldIndexing.Analyzed);
}
}
and don't forget to run this, so the server would be updated of the index definition :
var store = new DocumentStore { //....}
new HotelRoom_Search().Execute(Store);
UPDATE:
to overcome this you could also query the object HotelRoom without specifying the index (pass just the entity as T to session.Query). this would make ravendb create an Auto index according to the properties you use in your query.
ravendb docs:
Indexes are used by server to satisfy queries, whenever a user issues a query RavenDB will use an existing index if it matches query or create a new one if no match is found.
Indexes created by issuing a query are calleddynamic or Auto indexes and can be easily identified, due to their name that starts withAuto/ prefix.
Indexes created explicitly by user are calledstatic.
usually you want to control the way raven indexes (and retrieves) docs to tune performance, so it's not a good option but good as default.
of course that wouldn't fix the bug.
So I have this object model:
string Name; // name of the person
int Age; // age of the person
string CreatedBy; // operator who created person
My query sounds like this: all documents WHERE Age > 40 AND CreatedBy == 'callum' AND Name contains 'll'
CreatedBy is a necessary, scope of control.
Age is also a necessary (but isn't a security issue)
Name is where it can get fuzzy, because that is what the user is querying. Akin to sort of contains
The query below works for the first two parts:
"query": {
"bool": {
"must": [
{
"range": {
"age": {
"gt": 40
}
}
},
{
"match": {
"createdBy": "Callum"
}
}
]
}
}
I tried adding a multi_match because ultimately it maybe a search across Name, Address and other bits of information. I couldn't make sense of where to fit it in.
In my, nested queries would be useful. So first filter out all irrelevant users, then filter out irrelevant ages. Then do some fuzzier matching on relevant fields.
So, the answer to this isn't straightforward.
First of all you need to create an Analyser for Compound Words.
So in the .NET client it looks like:
this.elasticClient.CreateIndex("customer", p => p
.Settings(s => s
.Analysis(a => a
.TokenFilters(t => t
.NGram("bigrams_filter", ng => ng
.MaxGram(2)
.MinGram(2)))
.Analyzers(al => al
.Custom("bigrams", l => l
.Tokenizer("standard")
.Filters("lowercase", "bigrams_filter"))))));
this.elasticClient.Map<Person>(m => m
.Properties(props => props
.String(s => s
.Name(p => p.Name)
.Index(FieldIndexOption.Analyzed)
.Analyzer("bigrams"))
.String(s => s
.Name(p => p.CreatedBy)
.NotAnalyzed())
.Number(n => n
.Name(p => p.Age))));
Which is a sort of direct translation of the the first link provided. This now means that all names will be broken into their bigram representation:
Callum
ca
al
ll
lu
um
Then you need the actual query to take advantage of this. Now this is bit I like, because we've set up that index on the name column, it means that all term queries can have partial words in them, so take this for example (Sense query):
GET customer/_search
{
"query": {
"filtered": {
"query": {
"multi_match": {
"query": "ll",
"fields": ["name"]
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"age": {
"gt": 40
}
}
},
{
"match": {
"createdBy": "Callum"
}
}
]
}
}
}
}
}
Here, we have a filtered query. So the query is always the first to be run (can't find documentation yet to cite that, but I have read it), and this will be the partial terms match. Then we simply filter - which is done after the query - to get the subset of results we need.
Because the ngrams analyser is only set on name that is the only column that will be partially matched against. So CreatedBy won't and thus we get our security around the results.
Basically what you can do is put the query into two blocks:
"query": {
"filter":{
"bool":
{
"must":[
{
"range": {
"age": {
"gt": 40
}
}
}
]
}
},
"query":{
"bool": {
"must": [
{
"multi_match" : {
"query": "ll",
"fields": [ "createdBy", "Address","Name" ] ,
"fuzziness":2
}
}
]
}
}
}
What you can do is in filter you can use condtions to filter things out, on then with the filtered data you can apply you multi-match query. The main reason why I included age in filter is because you dont need to perform any kind of free text search, you just need to check with a static value, you can include more conditions within the must block of filter.
You can also look into this article, which might give you some overview.
https://googleweblight.com/?lite_url=https://www.elastic.co/blog/found-optimizing-elasticsearch-searches&ei=EBaRAJDx&lc=en-IN&s=1&m=75&host=www.google.co.in&ts=1465153335&sig=APY536wHUUfGEjoafiVIzGx2H77aieiymw
Hope it helps!
How do I perform the SQL Join equivalent in MongoDB?
For example say you have two collections (users and comments) and I want to pull all the comments with pid=444 along with the user info for each.
comments
{ uid:12345, pid:444, comment="blah" }
{ uid:12345, pid:888, comment="asdf" }
{ uid:99999, pid:444, comment="qwer" }
users
{ uid:12345, name:"john" }
{ uid:99999, name:"mia" }
Is there a way to pull all the comments with a certain field (eg. ...find({pid:444}) ) and the user information associated with each comment in one go?
At the moment, I am first getting the comments which match my criteria, then figuring out all the uid's in that result set, getting the user objects, and merging them with the comment's results. Seems like I am doing it wrong.
As of Mongo 3.2 the answers to this question are mostly no longer correct. The new $lookup operator added to the aggregation pipeline is essentially identical to a left outer join:
https://docs.mongodb.org/master/reference/operator/aggregation/lookup/#pipe._S_lookup
From the docs:
{
$lookup:
{
from: <collection to join>,
localField: <field from the input documents>,
foreignField: <field from the documents of the "from" collection>,
as: <output array field>
}
}
Of course Mongo is not a relational database, and the devs are being careful to recommend specific use cases for $lookup, but at least as of 3.2 doing join is now possible with MongoDB.
We can merge/join all data inside only one collection with a easy function in few lines using the mongodb client console, and now we could be able of perform the desired query.
Below a complete example,
.- Authors:
db.authors.insert([
{
_id: 'a1',
name: { first: 'orlando', last: 'becerra' },
age: 27
},
{
_id: 'a2',
name: { first: 'mayra', last: 'sanchez' },
age: 21
}
]);
.- Categories:
db.categories.insert([
{
_id: 'c1',
name: 'sci-fi'
},
{
_id: 'c2',
name: 'romance'
}
]);
.- Books
db.books.insert([
{
_id: 'b1',
name: 'Groovy Book',
category: 'c1',
authors: ['a1']
},
{
_id: 'b2',
name: 'Java Book',
category: 'c2',
authors: ['a1','a2']
},
]);
.- Book lending
db.lendings.insert([
{
_id: 'l1',
book: 'b1',
date: new Date('01/01/11'),
lendingBy: 'jose'
},
{
_id: 'l2',
book: 'b1',
date: new Date('02/02/12'),
lendingBy: 'maria'
}
]);
.- The magic:
db.books.find().forEach(
function (newBook) {
newBook.category = db.categories.findOne( { "_id": newBook.category } );
newBook.lendings = db.lendings.find( { "book": newBook._id } ).toArray();
newBook.authors = db.authors.find( { "_id": { $in: newBook.authors } } ).toArray();
db.booksReloaded.insert(newBook);
}
);
.- Get the new collection data:
db.booksReloaded.find().pretty()
.- Response :)
{
"_id" : "b1",
"name" : "Groovy Book",
"category" : {
"_id" : "c1",
"name" : "sci-fi"
},
"authors" : [
{
"_id" : "a1",
"name" : {
"first" : "orlando",
"last" : "becerra"
},
"age" : 27
}
],
"lendings" : [
{
"_id" : "l1",
"book" : "b1",
"date" : ISODate("2011-01-01T00:00:00Z"),
"lendingBy" : "jose"
},
{
"_id" : "l2",
"book" : "b1",
"date" : ISODate("2012-02-02T00:00:00Z"),
"lendingBy" : "maria"
}
]
}
{
"_id" : "b2",
"name" : "Java Book",
"category" : {
"_id" : "c2",
"name" : "romance"
},
"authors" : [
{
"_id" : "a1",
"name" : {
"first" : "orlando",
"last" : "becerra"
},
"age" : 27
},
{
"_id" : "a2",
"name" : {
"first" : "mayra",
"last" : "sanchez"
},
"age" : 21
}
],
"lendings" : [ ]
}
I hope this lines can help you.
This page on the official mongodb site addresses exactly this question:
https://mongodb-documentation.readthedocs.io/en/latest/ecosystem/tutorial/model-data-for-ruby-on-rails.html
When we display our list of stories, we'll need to show the name of the user who posted the story. If we were using a relational database, we could perform a join on users and stores, and get all our objects in a single query. But MongoDB does not support joins and so, at times, requires bit of denormalization. Here, this means caching the 'username' attribute.
Relational purists may be feeling uneasy already, as if we were violating some universal law. But let’s bear in mind that MongoDB collections are not equivalent to relational tables; each serves a unique design objective. A normalized table provides an atomic, isolated chunk of data. A document, however, more closely represents an object as a whole. In the case of a social news site, it can be argued that a username is intrinsic to the story being posted.
You have to do it the way you described. MongoDB is a non-relational database and doesn't support joins.
With right combination of $lookup, $project and $match, you can join mutiple tables on multiple parameters. This is because they can be chained multiple times.
Suppose we want to do following (reference)
SELECT S.* FROM LeftTable S
LEFT JOIN RightTable R ON S.ID = R.ID AND S.MID = R.MID
WHERE R.TIM > 0 AND S.MOB IS NOT NULL
Step 1: Link all tables
you can $lookup as many tables as you want.
$lookup - one for each table in query
$unwind - correctly denormalises data , else it'd be wrapped in arrays
Python code..
db.LeftTable.aggregate([
# connect all tables
{"$lookup": {
"from": "RightTable",
"localField": "ID",
"foreignField": "ID",
"as": "R"
}},
{"$unwind": "R"}
])
Step 2: Define all conditionals
$project : define all conditional statements here, plus all the variables you'd like to select.
Python Code..
db.LeftTable.aggregate([
# connect all tables
{"$lookup": {
"from": "RightTable",
"localField": "ID",
"foreignField": "ID",
"as": "R"
}},
{"$unwind": "R"},
# define conditionals + variables
{"$project": {
"midEq": {"$eq": ["$MID", "$R.MID"]},
"ID": 1, "MOB": 1, "MID": 1
}}
])
Step 3: Join all the conditionals
$match - join all conditions using OR or AND etc. There can be multiples of these.
$project: undefine all conditionals
Complete Python Code..
db.LeftTable.aggregate([
# connect all tables
{"$lookup": {
"from": "RightTable",
"localField": "ID",
"foreignField": "ID",
"as": "R"
}},
{"$unwind": "$R"},
# define conditionals + variables
{"$project": {
"midEq": {"$eq": ["$MID", "$R.MID"]},
"ID": 1, "MOB": 1, "MID": 1
}},
# join all conditionals
{"$match": {
"$and": [
{"R.TIM": {"$gt": 0}},
{"MOB": {"$exists": True}},
{"midEq": {"$eq": True}}
]}},
# undefine conditionals
{"$project": {
"midEq": 0
}}
])
Pretty much any combination of tables, conditionals and joins can be done in this manner.
You can join two collection in Mongo by using lookup which is offered in 3.2 version. In your case the query would be
db.comments.aggregate({
$lookup:{
from:"users",
localField:"uid",
foreignField:"uid",
as:"users_comments"
}
})
or you can also join with respect to users then there will be a little change as given below.
db.users.aggregate({
$lookup:{
from:"comments",
localField:"uid",
foreignField:"uid",
as:"users_comments"
}
})
It will work just same as left and right join in SQL.
As others have pointed out you are trying to create a relational database from none relational database which you really don't want to do but anyways, if you have a case that you have to do this here is a solution you can use. We first do a foreach find on collection A( or in your case users) and then we get each item as an object then we use object property (in your case uid) to lookup in our second collection (in your case comments) if we can find it then we have a match and we can print or do something with it.
Hope this helps you and good luck :)
db.users.find().forEach(
function (object) {
var commonInBoth=db.comments.findOne({ "uid": object.uid} );
if (commonInBoth != null) {
printjson(commonInBoth) ;
printjson(object) ;
}else {
// did not match so we don't care in this case
}
});
Here's an example of a "join" * Actors and Movies collections:
https://github.com/mongodb/cookbook/blob/master/content/patterns/pivot.txt
It makes use of .mapReduce() method
* join - an alternative to join in document-oriented databases
$lookup (aggregation)
Performs a left outer join to an unsharded collection in the same database to filter in documents from the “joined” collection for processing. To each input document, the $lookup stage adds a new array field whose elements are the matching documents from the “joined” collection. The $lookup stage passes these reshaped documents to the next stage.
The $lookup stage has the following syntaxes:
Equality Match
To perform an equality match between a field from the input documents with a field from the documents of the “joined” collection, the $lookup stage has the following syntax:
{
$lookup:
{
from: <collection to join>,
localField: <field from the input documents>,
foreignField: <field from the documents of the "from" collection>,
as: <output array field>
}
}
The operation would correspond to the following pseudo-SQL statement:
SELECT *, <output array field>
FROM collection
WHERE <output array field> IN (SELECT <documents as determined from the pipeline>
FROM <collection to join>
WHERE <pipeline> );
Mongo URL
It depends on what you're trying to do.
You currently have it set up as a normalized database, which is fine, and the way you are doing it is appropriate.
However, there are other ways of doing it.
You could have a posts collection that has imbedded comments for each post with references to the users that you can iteratively query to get. You could store the user's name with the comments, you could store them all in one document.
The thing with NoSQL is it's designed for flexible schemas and very fast reading and writing. In a typical Big Data farm the database is the biggest bottleneck, you have fewer database engines than you do application and front end servers...they're more expensive but more powerful, also hard drive space is very cheap comparatively. Normalization comes from the concept of trying to save space, but it comes with a cost at making your databases perform complicated Joins and verifying the integrity of relationships, performing cascading operations. All of which saves the developers some headaches if they designed the database properly.
With NoSQL, if you accept that redundancy and storage space aren't issues because of their cost (both in processor time required to do updates and hard drive costs to store extra data), denormalizing isn't an issue (for embedded arrays that become hundreds of thousands of items it can be a performance issue, but most of the time that's not a problem). Additionally you'll have several application and front end servers for every database cluster. Have them do the heavy lifting of the joins and let the database servers stick to reading and writing.
TL;DR: What you're doing is fine, and there are other ways of doing it. Check out the mongodb documentation's data model patterns for some great examples. http://docs.mongodb.org/manual/data-modeling/
There is a specification that a lot of drivers support that's called DBRef.
DBRef is a more formal specification for creating references between documents. DBRefs (generally) include a collection name as well as an object id. Most developers only use DBRefs if the collection can change from one document to the next. If your referenced collection will always be the same, the manual references outlined above are more efficient.
Taken from MongoDB Documentation: Data Models > Data Model Reference >
Database References
Before 3.2.6, Mongodb does not support join query as like mysql. below solution which works for you.
db.getCollection('comments').aggregate([
{$match : {pid : 444}},
{$lookup: {from: "users",localField: "uid",foreignField: "uid",as: "userData"}},
])
You can run SQL queries including join on MongoDB with mongo_fdw from Postgres.
MongoDB does not allow joins, but you can use plugins to handle that. Check the mongo-join plugin. It's the best and I have already used it. You can install it using npm directly like this npm install mongo-join. You can check out the full documentation with examples.
(++) really helpful tool when we need to join (N) collections
(--) we can apply conditions just on the top level of the query
Example
var Join = require('mongo-join').Join, mongodb = require('mongodb'), Db = mongodb.Db, Server = mongodb.Server;
db.open(function (err, Database) {
Database.collection('Appoint', function (err, Appoints) {
/* we can put conditions just on the top level */
Appoints.find({_id_Doctor: id_doctor ,full_date :{ $gte: start_date },
full_date :{ $lte: end_date }}, function (err, cursor) {
var join = new Join(Database).on({
field: '_id_Doctor', // <- field in Appoints document
to: '_id', // <- field in User doc. treated as ObjectID automatically.
from: 'User' // <- collection name for User doc
}).on({
field: '_id_Patient', // <- field in Appoints doc
to: '_id', // <- field in User doc. treated as ObjectID automatically.
from: 'User' // <- collection name for User doc
})
join.toArray(cursor, function (err, joinedDocs) {
/* do what ever you want here */
/* you can fetch the table and apply your own conditions */
.....
.....
.....
resp.status(200);
resp.json({
"status": 200,
"message": "success",
"Appoints_Range": joinedDocs,
});
return resp;
});
});
You can do it using the aggregation pipeline, but it's a pain to write it yourself.
You can use mongo-join-query to create the aggregation pipeline automatically from your query.
This is how your query would look like:
const mongoose = require("mongoose");
const joinQuery = require("mongo-join-query");
joinQuery(
mongoose.models.Comment,
{
find: { pid:444 },
populate: ["uid"]
},
(err, res) => (err ? console.log("Error:", err) : console.log("Success:", res.results))
);
Your result would have the user object in the uid field and you can link as many levels deep as you want. You can populate the reference to the user, which makes reference to a Team, which makes reference to something else, etc..
Disclaimer: I wrote mongo-join-query to tackle this exact problem.
playORM can do it for you using S-SQL(Scalable SQL) which just adds partitioning such that you can do joins within partitions.
Nope, it doesn't seem like you're doing it wrong. MongoDB joins are "client side". Pretty much like you said:
At the moment, I am first getting the comments which match my criteria, then figuring out all the uid's in that result set, getting the user objects, and merging them with the comment's results. Seems like I am doing it wrong.
1) Select from the collection you're interested in.
2) From that collection pull out ID's you need
3) Select from other collections
4) Decorate your original results.
It's not a "real" join, but it's actually alot more useful than a SQL join because you don't have to deal with duplicate rows for "many" sided joins, instead your decorating the originally selected set.
There is alot of nonsense and FUD on this page. Turns out 5 years later MongoDB is still a thing.
I think, if You need normalized data tables - You need to try some other database solutions.
But I've foun that sollution for MOngo on Git
By the way, in inserts code - it has movie's name, but noi movie's ID.
Problem
You have a collection of Actors with an array of the Movies they've done.
You want to generate a collection of Movies with an array of Actors in each.
Some sample data
db.actors.insert( { actor: "Richard Gere", movies: ['Pretty Woman', 'Runaway Bride', 'Chicago'] });
db.actors.insert( { actor: "Julia Roberts", movies: ['Pretty Woman', 'Runaway Bride', 'Erin Brockovich'] });
Solution
We need to loop through each movie in the Actor document and emit each Movie individually.
The catch here is in the reduce phase. We cannot emit an array from the reduce phase, so we must build an Actors array inside of the "value" document that is returned.
The code
map = function() {
for(var i in this.movies){
key = { movie: this.movies[i] };
value = { actors: [ this.actor ] };
emit(key, value);
}
}
reduce = function(key, values) {
actor_list = { actors: [] };
for(var i in values) {
actor_list.actors = values[i].actors.concat(actor_list.actors);
}
return actor_list;
}
Notice how actor_list is actually a javascript object that contains an array. Also notice that map emits the same structure.
Run the following to execute the map / reduce, output it to the "pivot" collection and print the result:
printjson(db.actors.mapReduce(map, reduce, "pivot"));
db.pivot.find().forEach(printjson);
Here is the sample output, note that "Pretty Woman" and "Runaway Bride" have both "Richard Gere" and "Julia Roberts".
{ "_id" : { "movie" : "Chicago" }, "value" : { "actors" : [ "Richard Gere" ] } }
{ "_id" : { "movie" : "Erin Brockovich" }, "value" : { "actors" : [ "Julia Roberts" ] } }
{ "_id" : { "movie" : "Pretty Woman" }, "value" : { "actors" : [ "Richard Gere", "Julia Roberts" ] } }
{ "_id" : { "movie" : "Runaway Bride" }, "value" : { "actors" : [ "Richard Gere", "Julia Roberts" ] } }
We can merge two collection by using mongoDB sub query. Here is example,
Commentss--
`db.commentss.insert([
{ uid:12345, pid:444, comment:"blah" },
{ uid:12345, pid:888, comment:"asdf" },
{ uid:99999, pid:444, comment:"qwer" }])`
Userss--
db.userss.insert([
{ uid:12345, name:"john" },
{ uid:99999, name:"mia" }])
MongoDB sub query for JOIN--
`db.commentss.find().forEach(
function (newComments) {
newComments.userss = db.userss.find( { "uid": newComments.uid } ).toArray();
db.newCommentUsers.insert(newComments);
}
);`
Get result from newly generated Collection--
db.newCommentUsers.find().pretty()
Result--
`{
"_id" : ObjectId("5511236e29709afa03f226ef"),
"uid" : 12345,
"pid" : 444,
"comment" : "blah",
"userss" : [
{
"_id" : ObjectId("5511238129709afa03f226f2"),
"uid" : 12345,
"name" : "john"
}
]
}
{
"_id" : ObjectId("5511236e29709afa03f226f0"),
"uid" : 12345,
"pid" : 888,
"comment" : "asdf",
"userss" : [
{
"_id" : ObjectId("5511238129709afa03f226f2"),
"uid" : 12345,
"name" : "john"
}
]
}
{
"_id" : ObjectId("5511236e29709afa03f226f1"),
"uid" : 99999,
"pid" : 444,
"comment" : "qwer",
"userss" : [
{
"_id" : ObjectId("5511238129709afa03f226f3"),
"uid" : 99999,
"name" : "mia"
}
]
}`
Hope so this will help.
How do I perform the SQL Join equivalent in MongoDB?
For example say you have two collections (users and comments) and I want to pull all the comments with pid=444 along with the user info for each.
comments
{ uid:12345, pid:444, comment="blah" }
{ uid:12345, pid:888, comment="asdf" }
{ uid:99999, pid:444, comment="qwer" }
users
{ uid:12345, name:"john" }
{ uid:99999, name:"mia" }
Is there a way to pull all the comments with a certain field (eg. ...find({pid:444}) ) and the user information associated with each comment in one go?
At the moment, I am first getting the comments which match my criteria, then figuring out all the uid's in that result set, getting the user objects, and merging them with the comment's results. Seems like I am doing it wrong.
As of Mongo 3.2 the answers to this question are mostly no longer correct. The new $lookup operator added to the aggregation pipeline is essentially identical to a left outer join:
https://docs.mongodb.org/master/reference/operator/aggregation/lookup/#pipe._S_lookup
From the docs:
{
$lookup:
{
from: <collection to join>,
localField: <field from the input documents>,
foreignField: <field from the documents of the "from" collection>,
as: <output array field>
}
}
Of course Mongo is not a relational database, and the devs are being careful to recommend specific use cases for $lookup, but at least as of 3.2 doing join is now possible with MongoDB.
We can merge/join all data inside only one collection with a easy function in few lines using the mongodb client console, and now we could be able of perform the desired query.
Below a complete example,
.- Authors:
db.authors.insert([
{
_id: 'a1',
name: { first: 'orlando', last: 'becerra' },
age: 27
},
{
_id: 'a2',
name: { first: 'mayra', last: 'sanchez' },
age: 21
}
]);
.- Categories:
db.categories.insert([
{
_id: 'c1',
name: 'sci-fi'
},
{
_id: 'c2',
name: 'romance'
}
]);
.- Books
db.books.insert([
{
_id: 'b1',
name: 'Groovy Book',
category: 'c1',
authors: ['a1']
},
{
_id: 'b2',
name: 'Java Book',
category: 'c2',
authors: ['a1','a2']
},
]);
.- Book lending
db.lendings.insert([
{
_id: 'l1',
book: 'b1',
date: new Date('01/01/11'),
lendingBy: 'jose'
},
{
_id: 'l2',
book: 'b1',
date: new Date('02/02/12'),
lendingBy: 'maria'
}
]);
.- The magic:
db.books.find().forEach(
function (newBook) {
newBook.category = db.categories.findOne( { "_id": newBook.category } );
newBook.lendings = db.lendings.find( { "book": newBook._id } ).toArray();
newBook.authors = db.authors.find( { "_id": { $in: newBook.authors } } ).toArray();
db.booksReloaded.insert(newBook);
}
);
.- Get the new collection data:
db.booksReloaded.find().pretty()
.- Response :)
{
"_id" : "b1",
"name" : "Groovy Book",
"category" : {
"_id" : "c1",
"name" : "sci-fi"
},
"authors" : [
{
"_id" : "a1",
"name" : {
"first" : "orlando",
"last" : "becerra"
},
"age" : 27
}
],
"lendings" : [
{
"_id" : "l1",
"book" : "b1",
"date" : ISODate("2011-01-01T00:00:00Z"),
"lendingBy" : "jose"
},
{
"_id" : "l2",
"book" : "b1",
"date" : ISODate("2012-02-02T00:00:00Z"),
"lendingBy" : "maria"
}
]
}
{
"_id" : "b2",
"name" : "Java Book",
"category" : {
"_id" : "c2",
"name" : "romance"
},
"authors" : [
{
"_id" : "a1",
"name" : {
"first" : "orlando",
"last" : "becerra"
},
"age" : 27
},
{
"_id" : "a2",
"name" : {
"first" : "mayra",
"last" : "sanchez"
},
"age" : 21
}
],
"lendings" : [ ]
}
I hope this lines can help you.
This page on the official mongodb site addresses exactly this question:
https://mongodb-documentation.readthedocs.io/en/latest/ecosystem/tutorial/model-data-for-ruby-on-rails.html
When we display our list of stories, we'll need to show the name of the user who posted the story. If we were using a relational database, we could perform a join on users and stores, and get all our objects in a single query. But MongoDB does not support joins and so, at times, requires bit of denormalization. Here, this means caching the 'username' attribute.
Relational purists may be feeling uneasy already, as if we were violating some universal law. But let’s bear in mind that MongoDB collections are not equivalent to relational tables; each serves a unique design objective. A normalized table provides an atomic, isolated chunk of data. A document, however, more closely represents an object as a whole. In the case of a social news site, it can be argued that a username is intrinsic to the story being posted.
You have to do it the way you described. MongoDB is a non-relational database and doesn't support joins.
With right combination of $lookup, $project and $match, you can join mutiple tables on multiple parameters. This is because they can be chained multiple times.
Suppose we want to do following (reference)
SELECT S.* FROM LeftTable S
LEFT JOIN RightTable R ON S.ID = R.ID AND S.MID = R.MID
WHERE R.TIM > 0 AND S.MOB IS NOT NULL
Step 1: Link all tables
you can $lookup as many tables as you want.
$lookup - one for each table in query
$unwind - correctly denormalises data , else it'd be wrapped in arrays
Python code..
db.LeftTable.aggregate([
# connect all tables
{"$lookup": {
"from": "RightTable",
"localField": "ID",
"foreignField": "ID",
"as": "R"
}},
{"$unwind": "R"}
])
Step 2: Define all conditionals
$project : define all conditional statements here, plus all the variables you'd like to select.
Python Code..
db.LeftTable.aggregate([
# connect all tables
{"$lookup": {
"from": "RightTable",
"localField": "ID",
"foreignField": "ID",
"as": "R"
}},
{"$unwind": "R"},
# define conditionals + variables
{"$project": {
"midEq": {"$eq": ["$MID", "$R.MID"]},
"ID": 1, "MOB": 1, "MID": 1
}}
])
Step 3: Join all the conditionals
$match - join all conditions using OR or AND etc. There can be multiples of these.
$project: undefine all conditionals
Complete Python Code..
db.LeftTable.aggregate([
# connect all tables
{"$lookup": {
"from": "RightTable",
"localField": "ID",
"foreignField": "ID",
"as": "R"
}},
{"$unwind": "$R"},
# define conditionals + variables
{"$project": {
"midEq": {"$eq": ["$MID", "$R.MID"]},
"ID": 1, "MOB": 1, "MID": 1
}},
# join all conditionals
{"$match": {
"$and": [
{"R.TIM": {"$gt": 0}},
{"MOB": {"$exists": True}},
{"midEq": {"$eq": True}}
]}},
# undefine conditionals
{"$project": {
"midEq": 0
}}
])
Pretty much any combination of tables, conditionals and joins can be done in this manner.
You can join two collection in Mongo by using lookup which is offered in 3.2 version. In your case the query would be
db.comments.aggregate({
$lookup:{
from:"users",
localField:"uid",
foreignField:"uid",
as:"users_comments"
}
})
or you can also join with respect to users then there will be a little change as given below.
db.users.aggregate({
$lookup:{
from:"comments",
localField:"uid",
foreignField:"uid",
as:"users_comments"
}
})
It will work just same as left and right join in SQL.
As others have pointed out you are trying to create a relational database from none relational database which you really don't want to do but anyways, if you have a case that you have to do this here is a solution you can use. We first do a foreach find on collection A( or in your case users) and then we get each item as an object then we use object property (in your case uid) to lookup in our second collection (in your case comments) if we can find it then we have a match and we can print or do something with it.
Hope this helps you and good luck :)
db.users.find().forEach(
function (object) {
var commonInBoth=db.comments.findOne({ "uid": object.uid} );
if (commonInBoth != null) {
printjson(commonInBoth) ;
printjson(object) ;
}else {
// did not match so we don't care in this case
}
});
Here's an example of a "join" * Actors and Movies collections:
https://github.com/mongodb/cookbook/blob/master/content/patterns/pivot.txt
It makes use of .mapReduce() method
* join - an alternative to join in document-oriented databases
$lookup (aggregation)
Performs a left outer join to an unsharded collection in the same database to filter in documents from the “joined” collection for processing. To each input document, the $lookup stage adds a new array field whose elements are the matching documents from the “joined” collection. The $lookup stage passes these reshaped documents to the next stage.
The $lookup stage has the following syntaxes:
Equality Match
To perform an equality match between a field from the input documents with a field from the documents of the “joined” collection, the $lookup stage has the following syntax:
{
$lookup:
{
from: <collection to join>,
localField: <field from the input documents>,
foreignField: <field from the documents of the "from" collection>,
as: <output array field>
}
}
The operation would correspond to the following pseudo-SQL statement:
SELECT *, <output array field>
FROM collection
WHERE <output array field> IN (SELECT <documents as determined from the pipeline>
FROM <collection to join>
WHERE <pipeline> );
Mongo URL
It depends on what you're trying to do.
You currently have it set up as a normalized database, which is fine, and the way you are doing it is appropriate.
However, there are other ways of doing it.
You could have a posts collection that has imbedded comments for each post with references to the users that you can iteratively query to get. You could store the user's name with the comments, you could store them all in one document.
The thing with NoSQL is it's designed for flexible schemas and very fast reading and writing. In a typical Big Data farm the database is the biggest bottleneck, you have fewer database engines than you do application and front end servers...they're more expensive but more powerful, also hard drive space is very cheap comparatively. Normalization comes from the concept of trying to save space, but it comes with a cost at making your databases perform complicated Joins and verifying the integrity of relationships, performing cascading operations. All of which saves the developers some headaches if they designed the database properly.
With NoSQL, if you accept that redundancy and storage space aren't issues because of their cost (both in processor time required to do updates and hard drive costs to store extra data), denormalizing isn't an issue (for embedded arrays that become hundreds of thousands of items it can be a performance issue, but most of the time that's not a problem). Additionally you'll have several application and front end servers for every database cluster. Have them do the heavy lifting of the joins and let the database servers stick to reading and writing.
TL;DR: What you're doing is fine, and there are other ways of doing it. Check out the mongodb documentation's data model patterns for some great examples. http://docs.mongodb.org/manual/data-modeling/
There is a specification that a lot of drivers support that's called DBRef.
DBRef is a more formal specification for creating references between documents. DBRefs (generally) include a collection name as well as an object id. Most developers only use DBRefs if the collection can change from one document to the next. If your referenced collection will always be the same, the manual references outlined above are more efficient.
Taken from MongoDB Documentation: Data Models > Data Model Reference >
Database References
Before 3.2.6, Mongodb does not support join query as like mysql. below solution which works for you.
db.getCollection('comments').aggregate([
{$match : {pid : 444}},
{$lookup: {from: "users",localField: "uid",foreignField: "uid",as: "userData"}},
])
You can run SQL queries including join on MongoDB with mongo_fdw from Postgres.
MongoDB does not allow joins, but you can use plugins to handle that. Check the mongo-join plugin. It's the best and I have already used it. You can install it using npm directly like this npm install mongo-join. You can check out the full documentation with examples.
(++) really helpful tool when we need to join (N) collections
(--) we can apply conditions just on the top level of the query
Example
var Join = require('mongo-join').Join, mongodb = require('mongodb'), Db = mongodb.Db, Server = mongodb.Server;
db.open(function (err, Database) {
Database.collection('Appoint', function (err, Appoints) {
/* we can put conditions just on the top level */
Appoints.find({_id_Doctor: id_doctor ,full_date :{ $gte: start_date },
full_date :{ $lte: end_date }}, function (err, cursor) {
var join = new Join(Database).on({
field: '_id_Doctor', // <- field in Appoints document
to: '_id', // <- field in User doc. treated as ObjectID automatically.
from: 'User' // <- collection name for User doc
}).on({
field: '_id_Patient', // <- field in Appoints doc
to: '_id', // <- field in User doc. treated as ObjectID automatically.
from: 'User' // <- collection name for User doc
})
join.toArray(cursor, function (err, joinedDocs) {
/* do what ever you want here */
/* you can fetch the table and apply your own conditions */
.....
.....
.....
resp.status(200);
resp.json({
"status": 200,
"message": "success",
"Appoints_Range": joinedDocs,
});
return resp;
});
});
You can do it using the aggregation pipeline, but it's a pain to write it yourself.
You can use mongo-join-query to create the aggregation pipeline automatically from your query.
This is how your query would look like:
const mongoose = require("mongoose");
const joinQuery = require("mongo-join-query");
joinQuery(
mongoose.models.Comment,
{
find: { pid:444 },
populate: ["uid"]
},
(err, res) => (err ? console.log("Error:", err) : console.log("Success:", res.results))
);
Your result would have the user object in the uid field and you can link as many levels deep as you want. You can populate the reference to the user, which makes reference to a Team, which makes reference to something else, etc..
Disclaimer: I wrote mongo-join-query to tackle this exact problem.
playORM can do it for you using S-SQL(Scalable SQL) which just adds partitioning such that you can do joins within partitions.
Nope, it doesn't seem like you're doing it wrong. MongoDB joins are "client side". Pretty much like you said:
At the moment, I am first getting the comments which match my criteria, then figuring out all the uid's in that result set, getting the user objects, and merging them with the comment's results. Seems like I am doing it wrong.
1) Select from the collection you're interested in.
2) From that collection pull out ID's you need
3) Select from other collections
4) Decorate your original results.
It's not a "real" join, but it's actually alot more useful than a SQL join because you don't have to deal with duplicate rows for "many" sided joins, instead your decorating the originally selected set.
There is alot of nonsense and FUD on this page. Turns out 5 years later MongoDB is still a thing.
I think, if You need normalized data tables - You need to try some other database solutions.
But I've foun that sollution for MOngo on Git
By the way, in inserts code - it has movie's name, but noi movie's ID.
Problem
You have a collection of Actors with an array of the Movies they've done.
You want to generate a collection of Movies with an array of Actors in each.
Some sample data
db.actors.insert( { actor: "Richard Gere", movies: ['Pretty Woman', 'Runaway Bride', 'Chicago'] });
db.actors.insert( { actor: "Julia Roberts", movies: ['Pretty Woman', 'Runaway Bride', 'Erin Brockovich'] });
Solution
We need to loop through each movie in the Actor document and emit each Movie individually.
The catch here is in the reduce phase. We cannot emit an array from the reduce phase, so we must build an Actors array inside of the "value" document that is returned.
The code
map = function() {
for(var i in this.movies){
key = { movie: this.movies[i] };
value = { actors: [ this.actor ] };
emit(key, value);
}
}
reduce = function(key, values) {
actor_list = { actors: [] };
for(var i in values) {
actor_list.actors = values[i].actors.concat(actor_list.actors);
}
return actor_list;
}
Notice how actor_list is actually a javascript object that contains an array. Also notice that map emits the same structure.
Run the following to execute the map / reduce, output it to the "pivot" collection and print the result:
printjson(db.actors.mapReduce(map, reduce, "pivot"));
db.pivot.find().forEach(printjson);
Here is the sample output, note that "Pretty Woman" and "Runaway Bride" have both "Richard Gere" and "Julia Roberts".
{ "_id" : { "movie" : "Chicago" }, "value" : { "actors" : [ "Richard Gere" ] } }
{ "_id" : { "movie" : "Erin Brockovich" }, "value" : { "actors" : [ "Julia Roberts" ] } }
{ "_id" : { "movie" : "Pretty Woman" }, "value" : { "actors" : [ "Richard Gere", "Julia Roberts" ] } }
{ "_id" : { "movie" : "Runaway Bride" }, "value" : { "actors" : [ "Richard Gere", "Julia Roberts" ] } }
We can merge two collection by using mongoDB sub query. Here is example,
Commentss--
`db.commentss.insert([
{ uid:12345, pid:444, comment:"blah" },
{ uid:12345, pid:888, comment:"asdf" },
{ uid:99999, pid:444, comment:"qwer" }])`
Userss--
db.userss.insert([
{ uid:12345, name:"john" },
{ uid:99999, name:"mia" }])
MongoDB sub query for JOIN--
`db.commentss.find().forEach(
function (newComments) {
newComments.userss = db.userss.find( { "uid": newComments.uid } ).toArray();
db.newCommentUsers.insert(newComments);
}
);`
Get result from newly generated Collection--
db.newCommentUsers.find().pretty()
Result--
`{
"_id" : ObjectId("5511236e29709afa03f226ef"),
"uid" : 12345,
"pid" : 444,
"comment" : "blah",
"userss" : [
{
"_id" : ObjectId("5511238129709afa03f226f2"),
"uid" : 12345,
"name" : "john"
}
]
}
{
"_id" : ObjectId("5511236e29709afa03f226f0"),
"uid" : 12345,
"pid" : 888,
"comment" : "asdf",
"userss" : [
{
"_id" : ObjectId("5511238129709afa03f226f2"),
"uid" : 12345,
"name" : "john"
}
]
}
{
"_id" : ObjectId("5511236e29709afa03f226f1"),
"uid" : 99999,
"pid" : 444,
"comment" : "qwer",
"userss" : [
{
"_id" : ObjectId("5511238129709afa03f226f3"),
"uid" : 99999,
"name" : "mia"
}
]
}`
Hope so this will help.
I am trying to write a search query on an elastic index that will return me results from any part of the field value.
I have a Path field that contains values like C:\temp\ab-cd\abc.doc
I want the ability to send a query that will return my any matching part from what I wrote
QueryContainer currentQuery = new QueryStringQuery
{
DefaultField = "Path",
Query = string.Format("*{0}*", "abc"),
};
The above will return results, this will not:
QueryContainer currentQuery = new QueryStringQuery
{
DefaultField = "Path",
Query = string.Format("*{0}*", "ab-cd"),
};
The same goes for any other special character like ##$%^&* and so on.
Is there some generic way to send a query and to find exactly what I searched?
Each of my fields are multi-fields and I can use the *.raw options but do not exactly know how or if I should
Use nGrams to split the text in smaller chunks and use term filter to query it. Pro: it should be faster. Con: the size of the index (disk space) will be larger because more terms (from nGram filter) are generated.
PUT /test
{
"settings": {
"analysis": {
"analyzer": {
"my_ngram_analyzer": {
"tokenizer": "keyword",
"filter": [
"substring"
]
}
},
"filter": {
"substring": {
"type": "nGram",
"min_gram": 1,
"max_gram": 50
}
}
}
},
"mappings": {
"test": {
"properties": {
"Path": {
"type": "string",
"index_analyzer": "my_ngram_analyzer",
"search_analyzer": "keyword"
}
}
}
}
}
And the query:
GET /test/test/_search
{
"query": {
"term": {
"Path": {
"value": "\temp"
}
}
}
}
If you wish, you can use the config above as a sub-field for whatever mapping you already have.
If you want to use query_string there one thing you need to be aware: you need to escape special characters. For example -, \ and : (complete list here). Also, when indexing, the \ char needs escaping, otherwise it will issue an error. This is what I tested especially with query_string: https://gist.github.com/astefan/a52fa4989bf5298102d1