I'm facing a strange issue with Elastisearch and scroll. I'm trying to get 50 192 records from my index but scrollId is returning 0 so my loop doesn't execute. There is my code :
var results = new List<MyModelList>();
var scanResults = client.Search<MyModel>(s => s
.Index("myindex")
.Type("MyIndexType")
.From(0)
.Size(2000)
.Query(q => q.QueryString(qs => qs.Query(query)))
.Scroll("5m")
);
var resultsscroll = client.Scroll<MyModel>("10m", scanResults.ScrollId);
while (resultsscroll.Documents.Any())
{
foreach (var doc in results)
{
results.Add(doc);
}
resultsscroll = client.Scroll<MyModel>("10m", resultsscroll.ScrollId);
}
return results;
When putting a break point, I can see that scanResults is set to 2000, but next, resultsscroll is at 0. What am I doing wrong? Followed the official doc, and make some search, seems everyone is using it this way. If someone as an idea of why it is not working it would be nice.
You will see that I've removed :
.SearchType(Nest.SearchType.Scan)
From my query, because if I let it, scanResults is set to 0. Don't understand why neither.
This is how if fixes my issue (also here on Aman's demand).
//Settings
var maxQuery = 50000;
//Connection
var client = ConnectElasticClient();
//Prepare Result
var results = new List<ModelDL>();
var countHits = (int)client.Search<Model>(s => s.Index("MyIndex").Type("MyType")
.Query(q => q.QueryString(qs => qs.Query(query)))).Total;
//First Query
var response =
client.Search<Model>(s => s.Index("MyIndex").Type("MyType").Size(maxQuery)
.Query(q => q.QueryString(qs => qs.Query(query))));
if (countHits <= maxQuery)
{
return results;
}
var cpt = maxQuery;
for (var i = 0; i < ((countHits - maxQuery) / maxQuery); i++)
{
var res =
client.Search<Model>(s => s.Index("MyIndex").Type("MyType").From(cpt).Size(maxQuery)
.Query(q => q.QueryString(qs => qs.Query(query))));
cpt += maxQuery;
results.AddRange(res.Documents.Select(x => new EntrepriseSireneDL(x)));
}
var lastScrollSize = countHits - cpt;
var r =
client.Search<Model>(s => s.Index("MyIndex").Type("MyType").From(cpt).Size(lastScrollSize)
.Query(q => q.QueryString(qs => qs.Query(query))));
results.AddRange(r.Documents.Select(x => new ModelDL(x)));
return results;
This way, i get everything i want as i want it.
Related
Easily say, why those two qqq are different in the breakpoint?
TodayInformation and YesterdayInformation actually are type of List<MyClass>
It is very strange that as usually, YesterdayInformation couldn't be any changes through this part, but actually YesterdayInformation is updated as TodayInformation I never use any pointer or reference?
var qqq = YesterdayInformation;
var TodayInformation = YesterdayInformation;
TodayInformation.Select(o =>
{
o.Signal = SignalpairList.Where(p => p.pair == o.pair).Select(p => p.signal).First();
o.SigmaMove = SigmaMovepairList.Where(p => p.pair == o.pair).Select(p => p.SigmaMove).First();
o.Date = Today;
return o;
}).ToList();
qqq = YesterdayInformation;
Because TodayInformation and YesterdayInformation are the same reference. You have a query of TodayInormation, but you modified each item in .Select method. Your query does something like:
var TodayInformation = YesterdayInformation;
foreach(var o in TodayInformation)
{
//you are modifying each item of TodayInformation collection
o.Signal = SignalpairList.Where(p => p.pair == o.pair).Select(p => p.signal).First();
o.SigmaMove = SigmaMovepairList.Where(p => p.pair == o.pair).Select(p => p.SigmaMove).First();
o.Date = Today;
}
I think you want this:
var TodayInformation = YesterdayInformation;
var result = TodayInformation.Select(o => new
{
Signal = SignalpairList.Where(p => p.pair == o.pair).Select(p => p.signal).First(),
SigmaMove = SigmaMovepairList.Where(p => p.pair == o.pair).Select(p => p.SigmaMove).First(),
Date = Today
}).ToList();
This query loops over the collection TodayInformation, loads the data to an anonymous object without modifying the original item, and set the result list into a variable.
I'm not exactly sure what you are saying or asking, but the code as-in is basically a loop updating the items.
This would be a lot clearer, and equivalent to your code.
var TodayInformation = YesterdayInformation;
foreach (var oin TodayInformation) {
o.Signal = SignalpairList.Where(p => p.pair == o.pair).Select(p => p.signal).First();
o.SigmaMove = SigmaMovepairList.Where(p => p.pair == o.pair).Select(p => p.SigmaMove).First();
o.Date = Today;
}
I'm trying to translate a query I've written to Linq for the past few days I can't seem to make it work. This is the query I'm trying to translate:
SELECT
hsd.CoveragePeriodBeginDate,
RateTotal = SUM(hsd.Rate),
ReimbursementTotal = SUM(hsd.TotalReimbursement),
AdjustmentsTotal = SUM(hsd.Adjustments)
FROM
( SELECT
CoveragePeriodBeginDate,
PaidDate,
Rate = TotalClaimCharge,
TotalReimbursement = ReimbursementAmount,
Adjustments = SUM(BaseRateChangeAmount)
FROM
dbo.HsdMonthlyCapitatation
WHERE
MemberID = 12345678
GROUP BY
CoveragePeriodBeginDate,
PaidDate,
TotalClaimCharge,
ReimbursementAmount
) hsd
GROUP BY
hsd.CoveragePeriodBeginDate
ORDER BY
hsd.CoveragePeriodBeginDate
What I need to do is translate this into Linq. I have tried many different ways, but can't seem to make it work right. It always seems to aggregate too much.
Here's the closest I've come.
var rawCapData = db.HsdMonthlyCapitations.Where(x => x.MemberID == memberID)
.Select(x => new {
CoveragePeriod = x.CoveragePeriodBeginDate,
TotalCharge = x.TotalClaimCharge,
Reimbursement = x.ReimbursementAmount,
PaidDate = x.PaidDate,
Adjust = x.BaseRateChangeAmount
})
.GroupBy(x => new {
CoverageDate = x.CoveragePeriod,
Paid = x.PaidDate,
Rate = x.TotalCharge,
Reimburse = x.Reimbursement
})
.GroupBy(x => new {
Coverage = x.Key.CoverageDate,
DhsRate = x.Sum(y => y.TotalCharge),
ReimbursementTotal = x.Sum(y => y.Reimbursement),
Adjustments = x.Sum(y => y.Adjust)
})
.Select(x => new {
CapMonthYear = x.Key.Coverage,
DhsRate = x.Key.DhsRate,
TotalReimbursement = x.Key.ReimbursementTotal,
AdjustmentsTotal = x.Key.Adjustments
});
I should say I have gotten it to work, but I feel it's rather cludgey and a mix of regular LINQ and lambda expressions, and I would prefer to code it all with lambda expressions, if at all possible. Here's the code I have gotten to work:
var rawCapitationData = from capitation
in db.HsdMonthlyCapitations
where capitation.MemberID == memberID
group capitation by new
{
capitation.CoveragePeriodBeginDate,
capitation.TotalClaimCharge,
capitation.ReimbursementAmount,
capitation.PaidDate
} into cap
select new {
CapitationMonthYear = cap.Key.CoveragePeriodBeginDate,
TotalReimbursement = cap.Key.TotalClaimCharge,
DhsCapitationAmount = cap.Key.ReimbursementAmount,
PaidDate = cap.Key.PaidDate,
DhsAdjustments = cap.Sum(x => x.BaseRateChangeAmount)
};
var capitationData = rawCapitationData.GroupBy(cap => cap.CapitationMonthYear)
.Select(data => new {
CapitationDate = data.Key,
TotalReimbursement = data.Sum(x => x.TotalReimbursement),
DhsCapitationAmount = data.Sum(x => x.DhsCapitationAmount),
DhsAdjustments = data.Sum(x => x.DhsAdjustments)
});
My preference is to do this all in one statement. Is it even possible? I feel I'm close with the lambda expressions, but I know I'm missing something.
Any help or advice is greatly appreciated.
Not sure what are you trying to achieve, but I've ended up with this:
return db.HsdMonthlyCapitations
.Where(x => x.MemberID == memberID)
.GroupBy(x => new {x.CoveragePeriodBeginDate, x.PaidDate, x.TotalClaimCharge, x.ReimbursementAmount})
.Select(x => new
{
x.Key.CoveragePeriodBeginDate,
x.Key.PaidDate,
Rate = x.Key.TotalClaimCharge,
TotalReimbursement = x.Key.ReimbursementAmount,
Adjustments = x.Sum(m => m.BaseRateChangeAmount)
})
.GroupBy(x => x.CoveragePeriodBeginDate)
.Select(x => new
{
CoveragePeriodBeginDate = x.Key,
RateTotal = x.Sum(m => m.Rate),
ReimbursementTotal = x.Sum(m => m.TotalReimbursement),
AdjustmentsTotal = x.Sum(m => m.Adjustments),
})
.OrderBy(x => x.CoveragePeriodBeginDate);
I want to count some filtered data in List of objects (TSource), while processing it in result list (TResult). My method looks like:
int ratedUsersCount = 0;
return sourceList.
.GroupBy(x => x.UserId)
.Select(u => {
var count = u.Count();
if (count >= 10) ratedUsersCount ++;
return new ... //some new object with calculations
}
)
.OrderByDescending(_ => some field)
.ThenByDescending(_ => some other field)
.Take(20)
.Select((u, i) => {
u.Index = i + 1;
u.Rating = i / ratedUsersCount ;
return u;
};
);
So, does the variable ratedUsersCount use incorrectly?
How can I fix the issue?
Try splitting your code up, it will become much cleaner, easier to understand, and won't produce side effects from within LINQ.
var groupedUsers = sourceUsers.GroupBy(x => x.UserId);
var minimumCount = 10;
var ratedUserCount = grouperUsers.Count(x => x.Take(minimumCount + 1).Count() >= minimumCount);
var intermediateUsers = groupedUsers.Select(x => new { /* ... */ });
var filteredUsers = intermediateUsers.OrderByDescending(/* ... */).ThenByDescending(/* ... */).Take(20);
var finalUsers = filteredUsers.Select(/* ... */);
Note that this may be a little slower, since grouperUsers.Count(x => x.Count() >= 10) will enumerate the IEnumerable in order to apply the counting function.
I have the following code
var profilesWithMailLists = profilesWithCf.GroupJoin(
profile2Maillist,
p => p.SqlId,
p2l => p2l.ProfileId,
(p, mailLists) =>
{
var p2lmongo = mailLists.Select<TProfile2MailList, Profile2MailList>(p2l =>
{
return new Profile2MailList
{
MailListId = p2l.MailListId,
Status = p2l.Status,
SubscriptionDate = p2l.SubscriptionDate
};
});
p.MailLists = p2lmongo.ToArray();
return p;
});
Is it possible to skip iteration and returning of profile if p2lmongo collection will contain 0 elements ?
Try using GroupBy and a Join instead of GroupJoin.
Some code to illustrate (not tested or type-checked):
var profilesWithMailListsAlt =
profilesWithCf.Join(
profile2Maillist.GroupBy(p2l => p2l.ProfileId),
p => p.SqlId,
p2lgroup => p2lgroup.Key,
(p, mailListGroup) =>
{
var p2lmongo = mailListGroup.Select<TProfile2MailList, Profile2MailList>(p2l =>
{
return new Profile2MailList
{
MailListId = p2l.MailListId,
Status = p2l.Status,
SubscriptionDate = p2l.SubscriptionDate
};
});
p.MailLists = p2lmongo.ToArray();
return p;
});
If that still doesn't do what you want, try explicitly excluding empty groups by changing
profile2Maillist.GroupBy(p2l => p2l.ProfileId)
to
profile2Maillist.GroupBy(p2l => p2l.ProfileId).Where(group => group.Any())
Just shooting from the hip here.
I'm trying to translate this sql statement
SELECT row, SUM(value) as VarSum, AVG(value) as VarAve, COUNT(value) as TotalCount
FROM MDNumeric
WHERE collectionid = 6 and varname in ('C3INEV1', 'C3INEVA2', 'C3INEVA3', 'C3INVA11', 'C3INVA17', 'C3INVA19')
GROUP BY row
into an EF 4 query using lambda expressions and am missing something.
I have:
sumvars = sv.staticvararraylist.Split(',');
var aavresult = _myIFR.MDNumerics
.Where(r => r.collectionid == _collid)
.Where(r => sumvars.Contains(r.varname))
.GroupBy(r1 =>r1.row)
.Select(rg =>
new
{
Row = rg.Key,
VarSum = rg.Sum(p => p.value),
VarAve = rg.Average(p => p.value),
TotalCount = rg.Count()
});
where the staticvararraylist has the string 'C3INEV1', 'C3INEVA2', 'C3INEVA3', 'C3INVA11', 'C3INVA17', 'C3INVA19' (without single quotes) and the _collid variable = 6.
While I'm getting the correct grouping, my sum, average, & count values aren't correct.
You didn't post your error message, but I suspect it's related to Contains. I've found that Any works just as well.
This should get you quite close:
var result =
from i in _myIFR.MDNumerics
where i.collectionid == _collid && sumvars.Any(v => i.varname == v)
group i by i.row into g
select new {
row = g.Key,
VarSum = g.Sum(p => p.value),
VarAve = g.Average(p => p.value),
TotalCount = g.Count()
};
Try this:
var aavresult = _myIFR.MDNumerics
.Where(r => r.collectionid == _collid && sumvars.Contains(r.varname))
.GroupBy(r1 =>r1.row,
(key,res) => new
{
Row = key,
VarSum = res.Sum(r1 => r1.value),
VarAve = res.Average(r1 => r1.value),
TotalCount = res.Count()
});