I'm using the Microsoft oData client to retrieve user information from a third-party API. I have the following code.
var query = Context.Users.Where(x => x.username.EndsWith("10"));
var result = query.ToList().Select(x => x.username);
However, this would only return 100 records. I can use the following code to retrieve all the records without the condition;
DataServiceCollection<User> users = new DataServiceCollection<User>(
Context.Users
);
while (users.Continuation != null)
{
//use the token to query for more users
//and load the results back into the collection
users.Load(
Context.Execute<User>(users.Continuation)
);
//print the current count of users retrieved
Console.WriteLine(users.Count);
}
How can I combine the two? i.e. to retrieve all the records with condition (x.username.EndsWith("10")).
I would say that it is a good practice to get only 100 rows at a time. But you can use the oDatas $top to get more than 100 rows. Microsoft oData client uses the the Take function for that: https://learn.microsoft.com/en-us/odata/client/query-options
I should be able to combine the two simply by:
var result= Context.Users.Where(x => x.username.EndsWith("10")).Take(1000).Select(x => x.username);
I'm using queries like this in oData Client version 7.6.2, but I think it will work in newer versions too.
var users = new List<User>();
DataServiceQueryContinuation<User> nextLink = null;
var query = Context.Users.Where(x => x.username.EndsWith("10")) as DataServiceQuery<User>;
var response = await query.ExecuteAsync() as QueryOperationResponse<User>;
do
{
if (nextLink != null)
{
response = await Context.ExecuteAsync<User>(nextLink) as QueryOperationResponse<User>;
}
users.AddRange(response);
}
while ((nextLink = response.GetContinuation()) != null);
Related
I don't have a problem currently, but I want to make sure, that the performance is not too shabby for my issue. My search on Microsofts documentation was without any success.
I have a Entity of the name Reservation. I now want to add some statistics to the program, where I can see some metrics about the reservations (reservations per month and favorite spot/seat in particular).
Therefore, my first approach was the following:
public async Task<ICollection<StatisticElement<Seat>>> GetSeatUsage(Company company)
{
var allReservations = await this.reservationService.GetAll(company);
return await this.FetchGroupedSeatData(allReservations, company);
}
public async Task<ICollection<StatisticElement<DateTime>>> GetMonthlyReservations(Company company)
{
var allReservations = await this.reservationService.GetAll(company);
return this.FetchGroupedReservationData(allReservations);
}
private async Task<ICollection<StatisticElement<Seat>>> FetchGroupedSeatData(
IEnumerable<Reservation> reservations,
Company company)
{
var groupedReservations = reservations.GroupBy(r => r.SeatId).ToList();
var companySeats = await this.seatService.GetAll(company);
return (from companySeat in companySeats
let groupedReservation = groupedReservations.FirstOrDefault(s => s.Key == companySeat.Id)
select new StatisticElement<Seat>()
{
Value = companySeat,
StatisticalCount = groupedReservation?.Count() ?? 0,
}).OrderByDescending(s => s.StatisticalCount).ToList();
}
private ICollection<StatisticElement<DateTime>> FetchGroupedReservationData(IEnumerable<Reservation> reservations)
{
var groupedReservations = reservations.GroupBy(r => new { Month = r.Date.Month, Year = r.Date.Year }).ToList();
return groupedReservations.Select(
groupedReservation => new StatisticElement<DateTime>()
{
Value = new DateTime(groupedReservation.Key.Year, groupedReservation.Key.Month, 1),
StatisticalCount = groupedReservation.Count(),
}).
OrderBy(s => s.Value).
ToList();
}
To explain the code a little bit: With GetSeatUsage and GetMonthlyReservations I can get the above mentioned data of a company. Therefore, I fetch ALL reservations at first (with reservationService.GetAll) - this is the point, where I think the performance will be a problem in the future.
Afterwards, I call either FetchGroupedSeatData or FetchGroupedReservationData, which first groups the reservations I previously fetched from the database and then converts them in a, for me, usable format.
As I said, I think the group by after I have read ALL the data from the database MIGHT be a problem, but I cannot find any information regarding performance in the documentation.
My other idea was, that I create a new method in my ReservationService, which then already returns the grouped list. But, again, I can't find the information, that the EF adds the GroupBy to the DB Query or basically does it after all of the data has been read from the database. This method would look something like this:
return await this.Context.Set<Reservation>.Where(r => r.User.CompanyId == company.Id).GroupBy(r => r.SeatId).ToListAsync();
Is this already the solution? Where can I check that? Am I missing something completely obvious?
I am using NEST client to get data from elastic search. As I have already 300000 data there request are so long. I followed this answer
My method looks like:
public static async Task<IList<T>> RockAndScroll<T>(this IElasticClient client,
string indexName,
string scrollTimeoutMinutes = "2m",
int scrollPageSize = 1000
) where T : class
{
var searchResponse = await client.SearchAsync<T>(sd => sd
.Index(indexName)
.From(0)
.Take(scrollPageSize)
.MatchAll()
.Scroll(scrollTimeoutMinutes));
var results = new List<T>();
while (true)
{
if (!searchResponse.IsValid || string.IsNullOrEmpty(searchResponse.ScrollId))
throw new Exception($"Search error: {searchResponse.ServerError.Error.Reason}");
if (!searchResponse.Documents.Any())
break;
results.AddRange(searchResponse.Documents);
searchResponse = await client.ScrollAsync<T>(scrollTimeoutMinutes, searchResponse.ScrollId);
}
await client.ClearScrollAsync(new ClearScrollRequest(searchResponse.ScrollId));
return results;
}
Filtering so slow as well:
var data = await _client.RockAndScroll<ElasticRequest>("taikunrequests");
var queryable = data.AsQueryable();
queryable = TaikunRequestListExtension.RoleManagement(request, _userAccessor.GetCurrentRole(), currentOrganization, partnerOrgs, queryable);
queryable = TaikunRequestListExtension.Sort(request, queryable);
queryable = TaikunRequestListExtension.Search(request, queryable);
queryable = TaikunRequestListExtension.Range(request, queryable);
queryable = TaikunRequestListExtension.Filter(request, queryable);
var count = queryable.Count();
var result = queryable
.Skip(request.Offset ?? 0).Take(request.Limit ?? 50).ToList();
return new TaikunRequestList(result, count);
If I use from and size from nest client it is max 1000, and 10000 for search after, and that was reason I used with scroll api as I need to filter option inside all data.
Is it any way to improve performance of requests or using another solution?
Is it any way to improve performance of requests or using another solution?
Yes. The current code is reading every single document (MatchAll) and then querying them in-memory (AsQueryable). This approach is going to have horrible performance for real-world datasets.
What you should do is build a NEST query and send that to ElasticSearch. That way ElasticSearch does the actual query/filter and only returns the matching documents.
In a web application with a SQL Server database, I have implemented the data access layer using the "Repository Pattern". For filtering a User based on its email, I'm using an expression like this:
var emailFilter = "user#example.com";
var query = _dbContext.Set<User>().Where(x => x.Email.Normalize() == emailFilter.Normalize());
var result = query.ToListAsync();
But EF Core throws an exception which says:
... could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync()
I have more than 200k users in the users' table and I don't want to filter data on the client-side.
The mentioned code is only an example but I mean other use cases with more complex methods.
Now, how can I use complex functions to filter data on the server-side?
I suggest to you, first normalize your email in db by bulk update :
by Z.EntityFramework.Plus.EFCore
_dbContext.Set<User>().update(x => new User {Email = x.Email.Normalize()} ;
then
emailFilter = ("user#example.com").Normalize();
var query = _dbContext.Set<User>().Where(x => x.Email == emailFilter);
var result = query.ToListAsync();
it's an instance solution
For non standard go with plain sql queries...
var emailFilter = "user#example.com";
var result = await _dbContext.Users.FromSqlRaw($"SELECT * FROM dbo.Users WHERE LOWER(Email) = {emailFilter}").ToListAsync();
Or use Functions
var emailFilter = "user#example.com";
var result = await _dbContext.Users.Where(x => EF.Functions.Like(x.Email, $"%{emailFilter}%")).ToListAsync();
With the penalty of the performance...
Okay, so this is probably from not knowing how to use EF core correctly as this is my 2nd day using but I appear to have to run .Include() in order to query against the "inner join" this method creates.
I've got this bit working and filtering down my results, but I don't want to return the include in the model as it's making a 256kb JSON response turn into a 2.8Mb JSON response.
public IEnumerable<Property> GetFeaturedProperties()
{
var properties = _db.Properties
.Include(p => p.GeoInfo)
.Include(p => p.FeaturedUntil)
.ToArray();
var featuredProperties = new List<Property>();
var DateTimeNow = DateTime.Now;
var midnightToday = DateTimeNow.AddHours(-DateTimeNow.Hour)
.AddMinutes(-DateTimeNow.Minute)
.AddSeconds(-DateTimeNow.Second-1);
for (var i = 0; i < properties.Count(); i++)
{
var property = properties[i];
if(property.FeaturedUntil.Any(p => p.FeaturedDate >= midnightToday))
featuredProperties.Add(property);
}
return featuredProperties;
}
So the offending line is .Include(p => p.FeaturedUntil). As this is an Array of dates that can be up anything from 10-1000 rows per joined row. It includes ALL data, even historical so this is really racking up data.
Can I run my query and then run something to .RemoveInclude(p => p.FeaturedUntil)?
You don't need to load the navigation properties in order to apply filtering. When you access a navigation property inside LINQ to Entities query, it's translated to the corresponding SQL construct, including JOINs. No real object/collections are involved. The whole query (with some exceptions) executes at server (database) side.
In your case, the following simple query will do the job:
public IEnumerable<Property> GetFeaturedProperties()
{
var DateTimeNow = DateTime.Now;
var midnightToday = DateTimeNow.AddHours(-DateTimeNow.Hour)
.AddMinutes(-DateTimeNow.Minute)
.AddSeconds(-DateTimeNow.Second-1);
return _db.Properties
.Include(p => p.GeoInfo) // assuming you want to return this data
.Where(p => p.FeaturedUntil.Any(p => p.FeaturedDate >= midnightToday))
.ToList();
}
For more info, see How Queries Work documentation topic.
I am trying to fetch the followers using Linq To Twitter latest version 3.0.2..But It is not returning back any response either not throw any error. Please suggest
var followers = GetTwitterFollowers
(twitterUserId, auth, maxFollowers);
var foll = followers.ContinueWith(f =>
{
return f.Result;
}).Result;
"GetTwitterFollowers" method defined as:
private static async Task<List<TwitterData>> GetTwitterFollowers(
ulong twitterUserId, SingleUserAuthorizer auth, int? maxFollowers)
{
var follower = maxFollowers ?? 15000;
try
{
var twitterCtx = new TwitterContext(auth);
var followers = await twitterCtx.Friendship
.Where(f => f.Type == FriendshipType.FollowersList
&& f.UserID == twitterUserId.ToString())
.Select(f => new TwitterData()
{
Followers = f.Users.Where(t => !t.Protected).Take(follower).Select(s => s).ToList()
}).SingleOrDefaultAsync();
return GetFollowersList(followers.Followers);
}
catch (Exception ex)
{
return null;
}
}
Get followers from Twitter first:
var friendship =
await
(from friend in twitterCtx.Friendship
where friend.Type == FriendshipType.FollowersList &&
friend.UserID == twitterUserId.ToString()
select friend)
.SingleOrDefaultAsync();
Then you can do your custom projection with an LINQ to Objects query on the results.
Update:
You should await the call to GetTwitterFollowers, like this:
var followers = await GetTwitterFollowersAsync(twitterUserId, auth, maxFollowers);
Then, you don't need the ContinueWith. It isn't working because GetTwitterFollowers returns immediately after hitting the await. The ContinueWith lambda might execute, but there isn't anything keeping the routine from returning and you don't get the result of the lambda execution. It might help to spend a little time looking at async/await to make this easier. Here's something that might help:
http://msdn.microsoft.com/en-us/library/hh191443.aspx