Develop Reference

Having poor performance when getting data from cosmos db by code

We have aroung 140K documents in a collection, and a typical search returns around 6 MB of data.
This process is taking around 30 to 40 seconds, and I would like to know why, and how can I improve it?
Sample data to see how the structure looks like: https://pastebin.com/0eJJfVKB
_id is the primarmy key, and there's a compound index on HotelID, GiataCityID, and CountryCode.
This is how I'm getting my data from cosmos by code:
The first part is to only get these data , as I figured out why return everything, just get what I need, that should speed things up.
Would appreciate any input, and ready to provide more details.
var hoteldetails = new List<HotelDetails>();
var projection = Builders<HotelDetails>.Projection.Expression(
t => new HotelDetails
{
Id = t.Id,
PropertyName = t.PropertyName,
GiataCityId = t.GiataCityId,
City = t.City,
CityId = t.CityId,
SupplierList = t.SupplierList,
CountryCode = t.CountryCode,
CountryName = t.CountryName,
longitude = t.longitude,
latitude = t.latitude,
Images = t.Images,
HotelId = t.HotelId,
SupplierRoomList = t.SupplierRoomList
});
this.GetDBClient();
var database = this.client.GetDatabase(this.settings.DataBase);
var collection = database.GetCollection<HotelDetails> (MongoCollections.HotelDetails.ToString());
if (HotelIDs != null && HotelIDs.Count > 0)
{
hoteldetails = await collection
.Aggregate()
.Match(x => x.CountryCode == CountryCode && (x.CityId == CityId || x.GiataCityId ==CityId.ToString())
&& x.SupplierList.Any(y => HotelIDs.Contains(y.HotelId) && y.SupplierName == SupplierNameType))
.Project(projection)
.Limit(Limit)
.ToListAsync();
}
The RU data page
RUs
I tried to use projecting to only get the properties i need, that sped things up a bit but not by a whole lot.
I tried to play with the filter, but really to no effect.

Show only active records and remove expired records using linq c#

RecordExpiration is a datetime field. I have to show only active records which are not expired.
var triggerEvents = ctx.CDB.Admin_FlexTriggerEvents.AsQueryable().Where(x => x.RecordID == RecordID)
.Select(x => new TriggerEvent
{
RecordID = x.RecordID,
FlexTriggerType_RecordID = x.FlexTriggerType_RecordID,
Condition = x.FlexTriggerType.Condition,
MergeData = x.FlexTriggerType.FlexTriggerEntityType.MergeData,
FlexTriggerEventNotifications = x.FlexTriggerEventNotifications,
FlexTriggerEventEmployeeMessages = x.FlexTriggerEventEmployeeMessages,
FlexTriggerEventNudgeEmails = x.FlexTriggerEventNudgeEmails,
FlexTriggerEventDataExchanges = x.FlexTriggerEventDataExchanges,
FlexTriggerEventCreateGroupComms = x.FlexTriggerEventCreateGroupComms,
Name = x.Name,
RecordExpiration = x.RecordExpiration
}).ToList();
Not sure where can I add a condition. All of the records before today can come in expired records.

I have to show only active records which are not expired.
You forgot to tell us what an active record is. And how do you know that a record is not expired. Is that in property RecordExpiration?
int recordId = ...
var activeNonExpiredRecords = ctx.CDB.Admin_FlexTriggerEvents
.Where(flexTriggerEvent => flexTriggerEvent.RecordID == recordID
&& !flexTriggerEvent.RecordExpiration
&& flexTriggerEvent.IsActive)
.Select(record => ...)
In words: from all FlexTriggerEvents, keep only those FlexTriggerEvents that have a value for property RecordId that equals the value of recordId, AND that have a false value for property RecordExpiration AND that have a true value for IsActive. From the remaining records, select ...
If you don't have a property IsActive, create a Boolean expression that represents IsActive:
For instance: active if less than a week old:
&& DateTime.Today - flexTriggerEvent.EventDate < TimeSpan.FromDays(7)

LInq query return value specific field

I have 2 tables as follows :
CatTable
CatCode
CatName
DogTable
DogCode
CatCode
NameCode
So I would like to write down a query where return me the list of all data present in the table CatTable
PLUS if in the Table DogTable the value in the field “CatCode” is the same of CatTable.CatCode and the field “NameCode” is Empty then should return a value "false" in the field “DogTable.CatCode”
Example
var query = from c in CatTable
from d in DogTable
where c.CatCode == d.CatCode
select new { c.CatCode, c.CatName, d.CatCode }
Do you have any suggestion about that?

a simple version is to use the Any to return the boolean of the logic you want on that last field
var query = CatTable.Select(ct => new
{
CatCode = ct.CatCode,
CatName = ct.CatName,
DogCatCode = !DogTable.Any(dt => dt.CatCode == ct.CatCode && dt.NameCode == "")
});

C# Pulling the Count of tables Associated with ID

Hello I have two tables that look like the following:
PeriodValue
Id (PK)
Name
Description
StartDate
EndDate
ActiveFlg
AcademicTerm
Id (PK)
Year Id (FK)
Name
Start Date
End Date
The objective is to pull the count of terms associated with every period value. This is my code to do so.
public async Task<PeriodValueDTO> GetSchoolYearPeriodValueDTOById (int periodValueId)
{
var value = await Db.PeriodValues.FindAsync(periodValueId);
return new PeriodValueDTO()
{
id = periodValueId,
Name = value.Name,
StartDate = value.Date.ToShortDateString(),
EndDate = value.EndDate.ToShortDateString(),
Description = value.Description
};
}
This method calls the one above
public async Task<List<PeriodValueDTO>> GettAllPeriodValueDTOsByType(int periodTypeId)
{
var toReturn = new List<PeriodValueDTO>();
var pvs = await Db.PeriodValues.Where(x => x.PeriodTypeId == periodTypeId).ToListAsync();
var pvIds = pvs.Select(x => x.Id).ToList();
var periodPeriodVal = await Db.Period_PeriodValue.Where(x => pvIds.Contains(x.PeriodValueId)).ToListAsync();
foreach (var ppv in periodPeriodVal)
{
var periodValue = pvs.FirstOrDefault(x => x.Id == ppv.PeriodValueId);
var value = await GetSchoolYearPeriodValueDTOById(periodTypeId);
var rightId = value.id; //Added this
var terms = Db.AcademicTerms.Where(x => x.YearId == rightId).ToArray(); //Changed This
var dto = new PeriodValueDTO()
{
id = periodValue.Id,
Name = periodValue.Name,
StartDate = periodValue.Date.ToShortDateString(),
EndDate = periodValue.EndDate.ToShortDateString(),
Description = periodValue.Description,
Count = terms.Length //Changed this
};
toReturn.Add(dto);
};
return toReturn;
}
However I am getting this error:
Failed to load resource: the server responded with a status of 500 (Internal Server Error)
Error: Resolving failed with a reason [object Object], but no resolveFailed provided for segment SchoolYear
If I comment out the lines that include var terms, var value, and Count it runs. When they are included I get the error above.
Still a novice to this. Any help would be great.

It's likely because of this line:
var terms = Db.AcademicTerms.Where(x => x.YearId == value.id);
Here, you are trying to generate an expression to be translated into SQL and executed on the database (because Db.AcademicTerms is IQueryable). You are not executing this query in memory. The query parser tries to convert value to a SQL parameter and fails because it is not a primitive object. So you have two choices:
Save value.id into a separate variable and then use that in your query, or better:
Save all AcademicTerms into memory before your loop (I assume there are not hundreds of them) by calling .ToArray() and then query against this. This also resolves the additional N+1 Selects antipattern you have, while allowing greater flexibility.

var value = await GetSchoolYearPeriodValueDTOById(periodTypeId);
should be :
var value = await GetSchoolYearPeriodValueDTOById(periodValue.Id);
no ?

Linq help - sub query gives null pointer exception

I writing a email system where we have a table of users "tblUsers" and a table of messages. A user can have many messages (from other users in tblusers) in his or her inbox (one:many).
In tblUsers table, I have a column called ImageURL (string) that contains the URL to the user's avatar. In this case, I'm looping through the messages in an inbox belonging to a user and what I'm trying to do is, once I get the message, walk up the tree to the tblUser and get the value in the ImageURL column for the owner of that message as marked "SenderAvatar" below.
Here's what I tried. The problem is that the sub linq for SenderAvatar below is throwing a nullpointer exception even though I have confirmed that there is a value for ImageURL (this is dev so there's only three users). Somehow my logic and linq's logic is at odds here. Can someone please help? Thanks!
Edit
I found two bugs. The first bug is Dzienny pointed me to the right direction where I was comparing apples and oranges. The second bug is FromUserId = ux.tblUserId, where I'm setting the current user id to FromUserId Guys, thank you for all your help on this.
public List<UserInboxMsg> GetUserInboxMsg(IKASLWSEntities conx, int userid)
{
var u = (from m in conx.tblUsers where m.Id == userid select m).FirstOrDefault();
if (u != null)
{
return (from ux in u.tblInboxes
orderby ux.CreationTS descending
select new UserInboxMsg
{
CreationTS = ux.CreationTS,
ExpirationDate = ux.ExpirationDate,
FromUserId = ux.tblUserId,
HasImage = ux.HasImage,
ImageId = ux.ImageId ?? 0,
IsDeleted = ux.IsDeleted,
IsRead = ux.IsRead,
MsgId = ux.Id,
MsgSize = ux.MessageSize,
ParentId = ux.ParentId,
Title = ux.Title,
ToUserId = userid,
FromUserName = ux.Title,
SenderAvatar = conx.tblMessages.Where(mu=>mu.Id == ux.Id).FirstOrDefault().tblUser.ImageURL,
Message = ux.Message
}).ToList<UserInboxMsg>();
}
else
{
return new List<UserInboxMsg>();
}
}
}

If in the entity-framework, there is a foreign key reference between the two tables you could probably do this:
SenderAvatar = conx.tblMessages.FirstOrDefault( mu=>mu.Id == ux.Id).ImageURL,

Try this.
public List<UserInboxMsg> GetUserInboxMsg(IKASLWSEntities conx, int userid)
{
var u = (from m in conx.tblUsers where m.Id == userid select m).FirstOrDefault();
if (u != null && conx != null)
{
return (from ux in u.tblInboxes
orderby ux.CreationTS descending
select new UserInboxMsg
{
...
...
SenderAvatar = conx.tblMessages.Any(mu=>mu.Id == ux.Id) ? (conx.tblMessages.First(mu=>mu.Id == ux.Id).tblUser != null? conx.tblMessages.First(mu=>mu.Id == ux.Id).tblUser.ImageURL : null) : null,
Message = ux.Message
}).ToList<UserInboxMsg>();
}
else
{
return new List<UserInboxMsg>();
}
}
}
if you are getting null for the Avatar, it is either because there are no entries in tblMessages where mu.Id equals ux.Id or the tblMessage entry is there but the tblUser property is null

There are several problems here.
The first is that the second statement is executed in memory, while it's possible to make the whole query run as SQL:
from u in conx.tblUsers where m.Id == userid
from ux in u.tblInboxes
orderby ux.CreationTS descending
select new UserInboxMsg
{
CreationTS = ux.CreationTS,
ExpirationDate = ux.ExpirationDate,
FromUserId = ux.tblUserId,
HasImage = ux.HasImage,
ImageId = ux.ImageId ?? 0,
IsDeleted = ux.IsDeleted,
IsRead = ux.IsRead,
MsgId = ux.Id,
MsgSize = ux.MessageSize,
ParentId = ux.ParentId,
Title = ux.Title,
ToUserId = userid,
FromUserName = ux.Title,
SenderAvatar = conx.tblMessages.Where(mu => mu.Id == ux.Id)
.FirstOrDefault().tblUser.ImageURL,
Message = ux.Message
}
This has three benefits:
you fetch less data from the database
you get rid of the null reference exception, because SQL doesn't have null references. It just returns null if a record isn't found.
you can return the result of this statement without the if-else.
Second, less important, is that you should use a navigation property like Inbox.Messages in stead of joining (sort of) the inbox and its messages. This makes it less likely that you use the wrong join variables and it condenses your code:
SenderAvatar = ux.Messages.
.FirstOrDefault().User.ImageURL,
Now if there is no avatar, there is no avatar. And there's no null reference exception.
(By the way, you can see that I hate these prefixes in class and property names).

I can only guess this part of your code is wrong : SenderAvatar = conx.tblMessages.Where(mu=>mu.Id == ux.Id).FirstOrDefault().tblUser.ImageURL
I think for example you should use (mu=>mu.UserId == ux.Id) instead of (mu=>mu.Id == ux.Id). In your code, you are comparing "Id" of a table to "Id" of another table which normally in one to many relations is wrong. (only works in one to one relations)
I said I can guess because you didn't mention any information about tblInboxes and tblMessages fields. If you could provide me more information about their structure, I could answer in more detail.
By the way to make your code more clear you can use:
var u = conx.tblUsers.FirstOrDefault(m=>m.Id == userid);
instead of
var u = (from m in conx.tblUsers where m.Id == userid select m).FirstOrDefault();
OR
conx.tblMessages.FirstOrDefault(mu=>mu.Id == ux.Id)
instead of
conx.tblMessages.Where(mu=>mu.Id == ux.Id).FirstOrDefault()