Develop Reference

Comparing two lists to see if I need to update the database or even add to

I have a simple class which holds a primary key of which I don't know what type it will be before it runs, as i'm getting the data from COM. It will either be an int or string.
I basically just need to fill up my toUpdateList & toAddList. This was working fine below with not too many records to play around with. However now the mongoDBList returns around 65k records and it's all turned very slow and it's taking 15+ minutes to resolve toUpdateList.
I'm pretty new to C# so I'm likely missing something.
I basically just need to compare one list to another and see if the RecordRevision is higher in the toUpdateList. For the toAddList this ones pretty simple as if it doesn't exist it needs to be added.
Thanks for looking I appreciate it!
class KeyRevision
{
public dynamic RecordRevision;
public dynamic PrimaryKey;
}
List<KeyRevision> potentialUpdateList = new List<KeyRevision>();
List<KeyRevision> mongoDBList = new List<KeyRevision>();
List<KeyRevision> toUpdateList = new List<KeyRevision>();
List<KeyRevision> toAddList = new List<KeyRevision>();
var sql = env.ExecuteSQL(sqlQuery);
sql.First();
// Loop over them and add to array
do
{
if (sql.RecordCount > 0)
{
//Console.WriteLine(sql.GetPropertyValue(primaryKey).ToString() + " : " + sql.RecordRevision);
var o = new KeyRevision();
o.PrimaryKey = sql.GetPropertyValue(primaryKey);
o.RecordRevision = sql.RecordRevision;
potentialUpdateList.Add(o);
}
sql.Next();
} while (!sql.EOF);
// Ask mongo for docs
var docs = collection1.Find(_ => true).Project("{RecordRevision: 1}").ToList();
// Get them into our type
mongoDBList = docs.ConvertAll(x => new KeyRevision()
{
PrimaryKey = x.GetValue("_id"),
RecordRevision = x.GetValue("RecordRevision")
});
// Finds which records we need to update
toUpdateList = potentialUpdateList.Where(x =>
mongoDBList.Any(y => y.PrimaryKey == x.PrimaryKey && y.RecordRevision < x.RecordRevision)).ToList();
// Finds the records we need to add
toAddList = potentialUpdateList.Where(x =>
mongoDBList.FindIndex(y => y.PrimaryKey == x.PrimaryKey) < 0).ToList();
Console.WriteLine($"{toUpdateList.Count} need to be updated");
Console.WriteLine($"{toAddList.Count} need to be updated");

How to quickly cut up and reassemble a C# List?

The setup:
I have a session variable that carries a list of IDs, pipe-delimited. The IDs are related to views in my site, and related to a breadcrumb builder.
Session["breadCrumb"] = "1001|1002|1003|1004";
If I'm on the view that corresponds to 1002, I'd like to cut everything AFTER that id out of the session variable.
I'd thought to use something like:
var curView = "1002";
if (Session["breadCrumb"] != null) {
var crumb = Session["breadCrumb"].ToString().Split('|').ToList();
var viewExists = crumb.Any(c => c.Value == curView);
if (viewExists) {
//remove everything after that item in the array.
}
}
But I'm wide open to methodologies.

You could use TakeWhile to get back only the items from the splitted list that precede the currentView.
var curView = "1002";
if (Session["breadCrumb"] != null)
{
var crumb = Session["breadCrumb"].ToString().Split('|').ToList();
var viewExists = crumb.TakeWhile(c => c != curView).ToList();
viewExists.Add(curView);
string result = string.Join("|",viewExists);
}
While this approach works I think that also the previous answer (now wrongly deleted from Mr. Andrew Whitaker) was correct. Using IndexOf should be faster with less splitting, looping, joining strings. I suggest Mr Whitaker to undelete its answer.
EDIT
This is from the deleted answer from Mr.Whitaker.
I will repost here because I think that its approach is simpler and should give better perfomances, so future readers could see also this option.
var crumb = Session["breadCrumb"].ToString()
int index = crumb.IndexOf(curView);
if (index >= 0)
{
Session["breadCrumb"] = crumb.Substring(0, index + curView.Length);
}
If Andrew decide to undelete its answer I will be glad to remove this part. Just let me know.

You could just store a List<string> in the Session directly. This saves you from having to split/concat the string manually. I know this does not answer the question directly, but I believe it is a superior solution to that.
var curView = "1002";
var crumb = Session["breadCrumb"] as List<string>;
if (crumb != null) {
var viewExists = crumb.Any(c => c.Value == curView);
if (viewExists) {
// remove everything after that item in the array.
}
}

I almost regret this, but frankly I'd just go for a regular expression:
var result = Regex.Replace(input, "(?<=(\\||^)" + current + ")(?=\\||$).*", "");
This does not directly tell you if the current view existed in the input, but even though this is also possible with the regex in this particular instance another, dead simple test exists:
var viewExists = result.Length != current.Length;

Concurent foreach iteration of two list strings

Let's say I have two List<string>. These are populated from the results of reading a text file
List owner contains:
cross
jhill
bbroms
List assignee contains:
Chris Cross
Jack Hill
Bryan Broms
During the read from a SQL source (the SQL statement contains a join)... I would perform
if(sqlReader["projects.owner"] == "something in owner list" || sqlReader["assign.assignee"] == "something in assignee list")
{
// add this projects information to the primary results LIST
list_by_owner.Add(sqlReader["projects.owner"],sqlReader["projects.project_date_created"],sqlReader["projects.project_name"],sqlReader["projects.project_status"]);
// if the assignee is not null, add also to the secondary results LIST
// logic to determine if assign.assignee is null goes here
list_by_assignee.Add(sqlReader["assign.assignee"],sqlReader["projects.owner"],sqlReader["projects.project_date_created"],sqlReader["projects.project_name"],sqlReader["projects.project_status"]);
}
I do not want to end up using nested foreach.
The FOR loop would probably suffice. Someone had mentioned ZIP to me but wasn't sure if that would be a preferable route to go in my situation.

One loop to iterate through both lists (assuming both have same count):
for (int i = 0; i < alpha.Count; i++)
{
var itemAlpha = alpha[i] // <= your object of list alpha
var itemBeta = beta[i] // <= your object of list beta
//write your code here
}

From what you describe, you don't need to iterate at all.
This is what you need:
http://msdn.microsoft.com/en-us/library/bhkz42b3.aspx
Usage:
if ((listAlpga.contains(resultA) || (listBeta.contains(resultA)) {
// do your operation
}
List Iteration will happen implicitly inside the contains method. And thats 2n comparisions, vs n*n for nested iteration.
You would be better off with sequential iteration in each list one after the other, if at all you need to go that route.

This list is maybe better represented as a List<KeyValuePair<string, string>> which would pair the two list values together in a single list.

There are several options for this. The least "painful" would be plain old for loop:
for (var index = 0; index < alpha.Count; index++)
{
var alphaItem = alpha[index];
var betaItem = beta[index];
// Do something.
}
Another interesting approach is using the indexed LINQ methods (but you need to remember they get evaluated lazily, you have to consume the resulting enumerable), for example:
alpha.Select((alphaItem, index) =>
{
var betaItem = beta[index];
// Do something
})
Or you can enumerate both collection if you use the enumerator directly:
using (var alphaEnumerator = alpha.GetEnumerator())
using (var betaEnumerator = beta.GetEnumerator())
{
while (alphaEnumerator.MoveNext() && betaEnumerator.MoveNext())
{
var alphaItem = alphaEnumerator.Current;
var betaItem = betaEnumerator.Current;
// Do something
}
}

Zip (if you need pairs) or Concat (if you need combined list) are possible options to iterate 2 lists at the same time.

I like doing something like this to enumerate over parallel lists:
int alphaCount = alpha.Count ;
int betaCount = beta.Count ;
int i = 0 ;
while ( i < alphaCount && i < betaCount )
{
var a = alpha[i] ;
bar b = beta[i] ;
// handle matched alpha/beta pairs
++i ;
}
while ( i < alphaCount )
{
var a = alpha[i] ;
// handle unmatched alphas
++i ;
}
while ( i < betaCount )
{
var b = beta[i] ;
// handle unmatched betas
++i ;
}

How can I make this code faster?? List<custom_class>().Find(lambda_expression)

I need to match email sends with email bounces so I can find if they were delivered or not. The catch is, I have to limit the bounce to within 4 days of the send to eliminate matching the wrong send to the bounce. Send records are spread over a period of 30 days.
LinkedList<event_data> sent = GetMyHugeListOfSends(); //for example 1M+ records
List<event_data> bounced = GetMyListOfBounces(); //for example 150k records
bounced = bounced.OrderBy(o => o.event_date).ToList(); //this ensures the most accurate match of bounce to send (since we find the first match)
List<event_data> delivered = new List<event_data>();
event_data deliveredEmail = new event_data();
foreach (event_data sentEmail in sent)
{
event_data bounce = bounced.Find(item => item.email.ToLower() == sentEmail.email.ToLower() && (item.event_date > sentEmail.event_date && item.event_date < sentEmail.event_date.AddDays(deliveredCalcDelayDays)));
//create delivered records
if (bounce != null)
{
//there was a bounce! don't add a delivered record!
}
else
{
//if sent is not bounced, it's delivered
deliveredEmail.sid = siteid;
deliveredEmail.mlid = mlid;
deliveredEmail.mid = mid;
deliveredEmail.email = sentEmail.email;
deliveredEmail.event_date = sentEmail.event_date;
deliveredEmail.event_status = "Delivered";
deliveredEmail.event_type = "Delivered";
deliveredEmail.id = sentEmail.id;
deliveredEmail.number = sentEmail.number;
deliveredEmail.laststoretransaction = sentEmail.laststoretransaction;
delivered.Add(deliveredEmail); //add the new delivered
deliveredEmail = new event_data();
//remove bounce, it only applies to one send!
bounced.Remove(bounce);
}
if (bounced.Count() == 0)
{
break; //no more bounces to match!
}
}
So I did some testing and it's processing about 12 sent records per second. At 1M+ records, it will take 25+ hours to process!
Two questions:
How can I find the exact line that is taking the most time?
I am assuming it's the lambda expression finding the bounce that is taking the longest since this was much faster before I put that in there. How can I speed this up?
Thanks!
Edit
---Ideas---
One idea I just had is to sort the sends by date like I did the bounces so that the search through the bounces will be more efficient, since an early send would be likely to hit an early bounce as well.
Another idea I just had is to run a couple of these processes in parallel, although I would hate to multi-thread this simple application.

I would be reasonably confident in saying that yes it is your find that is taking the time.
It looks like you are certain that the find method will return 0 or 1 records only (not a list) in which case the way to speed this up would be to create a lookup (a dictionary) instead of creating a List<event_data> for your bounced var, create a Dictionary<key, event_data> instead, then you can just look-up the value by key instead of doing a find.
The trick is in creating your key (I don't know enough about your app to help with that) but essentially the same criteria that is in your find.
EDIT. (adding some pseudo code)
void Main()
{
var hugeListOfEmails = GetHugeListOfEmails();
var allBouncedEmails = GetAllBouncedEmails();
IDictionary<string, EmailInfo> CreateLookupOfBouncedEmails = CreateLookupOfBouncedEmails(allBouncedEmails);
foreach(var info in hugeListOfEmails)
{
if(CreateLookupOfBouncedEmails.ContainsKey(info.emailAddress))
{
// Email is bounced;
}
else
{
// Email is not bounced
}
}
}
public IEnumerable<EmailInfo> GetHugeListOfEmails()
{
yield break;
}
public IEnumerable<EmailInfo> GetAllBouncedEmails()
{
yield break;
}
public IDictionary<string, EmailInfo> CreateLookupOfBouncedEmails(IEnumerable<EmailInfo> emailList)
{
var result = new Dictionary<string, EmailInfo>();
foreach(var e in emailList)
{
if(!result.ContainsKey(e.emailAddress))
{
if(//satisfies the date conditions)
{
result.Add(e.emailAddress, e);
}
}
}
return result;
}
public class EmailInfo
{
public string emailAddress { get; set; }
public DateTime DateSent { get; set; }
}

You should improve by using ToLookup method to create lookup table for email address
var bouncedLookup = bounced.ToLookup(k => k.email.ToLower());
and use this in the loop to lookup by the email first
var filteredBounced = bouncedLookup[sent_email.email.ToLower()];
// mini optimisation here
var endDate = sentEmail.event_date.AddDays(deliveredCalcDelayDays);
event_data bounce = filteredBounced.Find(item => item.event_date > sentEmail.event_date && item.event_date < endDate));
I could not compile it but I think that should do. Please try it.

You are finding items in a list. That means it has to traverse the whole list so it is an order (n) operation. Could you not store those sent emails in a Dictionary with the key being the email address you are searching on. The go through the bounces linking back to the emails in the dictionary. The lookup will be constant time and the you will go through the bounces so it will be order (n) overall. You current method is order (n squared)

Converting bounced to sortedlist might be a good solution
SortedList<string,data> sl = new SortedList<string,event_data>(bounced.ToDictionary(s=>s.email,s=>s));
and to find a bounce use
sl.Select(c=>c.Key.Equals(item => item.email,StringComparison.OrdinalIgnoreCase) && ...).FirstOrDefault();

There's another concern about your code, that I want to point out.
Memory consumption. I don't know your machine configuration, but here are some thoughts about the code:
Initially you are allocating space for 1,2M+ objects of event_data
type. I can't see event_data full type definition, but assuming
that emails are all unique and seeing that the type has quite many
properties, I can assume that such a collection is rather heavy
(hundreds of Meg possibly).
Next you are allocating another bunch of event_data objects
(almost 1M if I've counted it right). It's getting even more heavy
in terms of memory consumption
I don't know about other objects, that are present in data-model of your application, but considering all things I've mentioned, you can easily get close to memory limit
for 32-bit process and thus force GC to work very often. In fact
you can easily have a GC collecting after each call
bounced.Remove(bounce); And it's really would significantly slow down your app.
So, even if you are having a plenty of memory left and/or your app is 64-bit, I would try to minimize memory consumption. Pretty sure it would get your code run faster. For example, you can do complete processing of deliveredEmail, without storing it, or load your initial event_data in chunks etc.

On Consideration, the number of bounces is relatively small, so,
Why not pre omptimise the bounce lookup as much as possible, this code makes a delegate for each possible bounce and groups them into a dictionary for access by the e-mail key.
private static DateInRange(
DateTime sendDate,
DateTime bouncedDate,
int deliveredCalcDelayDays)
{
if (sentDate < bouncedDate)
{
return false;
}
return sentDate < bouncedDate.AddDays(deliveredCalcDelayDays);
}
static IEnumerable<event_data> GetDeliveredMails(
IEnumerable<event_data> sent,
IEnumerable<event_data> bounced,
int siteId,
int mlId,
int mId,
int deliveredCalcDelayDays)
{
var grouped = bounced.GroupBy(
b => b.email.ToLowerInvariant());
var lookup = grouped.ToDictionary(
g => g.Key,
g => g.OrderBy(e => e.event_date).Select(
e => new Func<DateTime, bool>(
s => DateInRange(s, e.event_date, deliveredCalcDelayDays))).ToList());
foreach (var s in sent)
{
var key = s.email.ToLowerInvariant();
List<Func<DateTime, nool>> checks;
if (lookup.TryGetValue(key, out checks))
{
var match = checks.FirstOrDefault(c => c(s.event_date));
if (match != null)
{
checks.Remove(match);
continue;
}
}
yield return new event_data
{
.sid = siteid;
.mlid = mlid;
.mid = mid;
.email = s.email;
.event_date = s.event_date;
.event_status = "Delivered";
.event_type = "Delivered";
.id = s.id;
.number = s.number;
.laststoretransaction = s.laststoretransaction
};
}
}
You could try pre-compiling the delegates in lookup if this is not fast enough.

Ok the final solution I found was a Dictionary for the bounces.
The sent LinkedList was sorted by sent_date so it would loop through in chronological order. That's important because I have to match the right send to the right bounce.
I made a Dictionary<string,<List<event_data>>, so the key was email, and the value was a List of all <event_data> bounces for the email address. The List was sorted by event_date since I wanted to make sure the first bounce was matched to the send.
Final result...it went from processing 700 records/minute to 500k+ records/second.
Here is the final code:
LinkedList sent = GetMyHugeListOfSends();
IEnumerable sentOrdered = sent.OrderBy(send => send.event_date);
Dictionary> bounced = GetMyListOfBouncesAsDictionary();
List delivered = new List();
event_data deliveredEmail = new event_data();
List bounces = null;
bool matchedBounce = false;
foreach (event_data sentEmail in sentOrdered)
{
matchedBounce = false;
//create delivered records
if (bounced.TryGetValue(sentEmail.email, out bounces))
{
//there was a bounce! find out if it was within 4 days after the send!
foreach (event_data bounce in bounces)
{
if (bounce.event_date > sentEmail.event_date &&
bounce.event_date <= sentEmail.event_date.AddDays(4))
{
matchedBounce = true;
//remove the record because a bounce can only match once back to a send
bounces.Remove(bounce);
if(bounces.Count == 0) //no more bounces for this email
{
bounced.Remove(sentEmail.email);
}
break;
}
}
if (matchedBounce == false) //no matching bounces in the list!
{
//if sent is not bounced, it's delivered
deliveredEmail.sid = siteid;
deliveredEmail.mlid = mlid;
deliveredEmail.mid = mid;
deliveredEmail.email = sentEmail.email;
deliveredEmail.event_date = sentEmail.event_date;
deliveredEmail.event_status = "Delivered";
deliveredEmail.event_type = "Delivered";
deliveredEmail.id = sentEmail.id;
deliveredEmail.number = sentEmail.number;
deliveredEmail.laststoretransaction = sentEmail.laststoretransaction;
delivered.Add(deliveredEmail); //add the new delivered
deliveredEmail = new event_data();
}
}
else
{
//if sent is not bounced, it's delivered
deliveredEmail.sid = siteid;
deliveredEmail.mlid = mlid;
deliveredEmail.mid = mid;
deliveredEmail.email = sentEmail.email;
deliveredEmail.event_date = sentEmail.event_date;
deliveredEmail.event_status = "Delivered";
deliveredEmail.event_type = "Delivered";
deliveredEmail.id = sentEmail.id;
deliveredEmail.number = sentEmail.number;
deliveredEmail.laststoretransaction = sentEmail.laststoretransaction;
delivered.Add(deliveredEmail); //add the new delivered
deliveredEmail = new event_data();
}
if (bounced.Count() == 0)
{
break; //no more bounces to match!
}
}

Use a numeric value in a linq dynamic query string

I am trying to make a dynamic linq query that will check for values based on a string.
First of all, here's the query:
objQry = from o in m_Db.OBJECTS.Where(whereConditions)
select o;
if(!objQry.Any())
{
return null;
}
The whereConditions variable is a string I build and pass as parameter to find out the values I need. Here's examples of valid string:
OBJ_NAME == \"Sword\" and OBJ_OWNER == \"Stan\"
This will return any item whose name is "Sword" and owner is "Stan;
OBJ_COLOR == \"Blue\" OR OBJ_COLOR == \"Red\"
This will return any item which color is either blue or red.
Up to there, I'm fine, but now I have a problem: I need to check a decimal field. So I've tried this string:
OBJ_NUMBER == 1
But the query returns null even if there are objects which OBJ_NUMBER value is 1. It's a decimal. How can I indicate the query that they need to check for a decimal value?
**** EDIT ****
I have tried to "modify" the value passed so that it looks like this:
"CARD_NUMBER == Convert.ToDecimal(1)"
And now I have a different kind of error telling me this:
LINQ to Entities does not recognize the method 'System.Decimal ToDecimal(Int32)' method, and this method cannot be translated into a store expression.
Any clues anyone? I'm still looking for a way to do this. Thanks!
EDIT 2
You can get an example of how my code is shaped by looking at this question.
Let's come back at this problem. I want to check decimal values. Let's say that OBJ_NUMBER is a decimal field.
Using Dynamic Linq, I tried to read the decimal field. Say that I want to get each object which number is 1.27. The whereConditions field would then be shaped like this:
OBJ_NUMBER == 1.27
But then I would get an Invalid real literal '1.27' error. I don't know why.
So I have tried Gert Arnold's solution and done this instead:
decimal bDecimal = decimal.Parce(valueToParse);
param = new ObjectParameter("cardNumber", typeof(decimal)) { Value = bDecimal };
valuesToUse.Add("CARD_NUMBER == #cardNumber");
listParams.Add(param);
But I ended up having 2 problems:
The first problem is that my whereConditions string is shaped this way:
CARD_NUMBER == #cardNumber
But I get the following error:
No property or field 'cardNumber' exists in type 'CARD'
Leading me to believe that it cannot make the link between the object parameter and the string used to do the query.
As you can see, I have a list of Params. This is because I cannot know for sure how many parameters the user will chose. So each time the user enters a new search field, I have to create a new ObjectParameter and store it in a list. Here's how I try to do the thing after:
ObjectParameter[] arrayParameters = listParams.ToArray();
// Convert the list to an array
And then, when I try to make the query:
cardQry = from c in mDb.CARD.Where(whereConditions, arrayParameters)
select c;
But to no avail.
RESULTS
Based on the answered question below, I have developped something "awful", yet functional.
First of all, I ignore every decimal fields because I could never reach them with dynamic linq. Instead, I do this:
var valuesToParse = keyValuePair.Value.Split(new[] {' '}, StringSplitOptions.RemoveEmptyEntries);
// Here I parse the value and, if that's the case, the symbol.
decimal baseValue = decimal.Parse(valuesToParse[0]);
if (valuesToParse.Count() > 1)
{
string baseMethod = valuesToParse[1];
if (baseMethod == ">" || baseMethod == ">=")
{
if (baseMethod == ">=")
{
baseValue--;
}
// The list is actually like this: Dictionary<string, object> list = new Dictionary<string, object>();
list.Add("low", baseValue);
// I kind of activate a tag telling me that the user is looking for a higher value.
cardHigher = true;
}
else
{
if (baseMethod == "<=")
{
baseValue++;
}
list.Add("low", baseValue);
cardLower = true;
}
}
else
{
//lowParam = new ObjectParameter("dec", typeof(decimal)) { Value = baseValue };
list.Add("low", baseValue);
}
cardNumberActivated = true;
At the end, when I get the list of objects, I do this:
if (list.Count > 0)
{
(example)
if (cardNumberActivated)
{
if (cardHigher)
{
q = mDb.CARD.Where("CARD_NUMBER >= #0", list["low"]).ToList();
}
else if (cardLower)
{
q = mDb.CARD.Where("CARD_NUMBER <= #0", list["low"]).ToList();
}
else
{
q = mDb.CARD.Where("CARD_NUMBER == #0", list["low"]).ToList();
}
}
}
// Here we get the orinalData with the basic filters.
listToReturn.AddRange(cardQry);
if (q != null)
{
//listToReturn.AddRange(q);
for (int i = 0; i < listToReturn.Count; i++)
{
var priceList1 = listToReturn[i];
if (!q.Any(_item => _item.CARD_NUMBER == priceList1.CARD_NUMBER))
{
listToReturn.RemoveAt(i);
i--;
}
}
}
And it works. This is not an elegant way to make it work, but I can validate the fields the way I wanted, and for this, I am thankful at last.

You should not build a query string with inline predicate values. Use parameters in stead. Then will also be able to specify the type:
var whereConditions= "it.CARD_NUMBER = #cardNumber";
var param = new ObjectParameter("cardNumber", typeof(decimal)) { Value = 1 };
objQry = from o in m_Db.OBJECTS.Where(whereConditions, param);
Edit
I don't know what doesn't work in your code. Here's just a random piece of working code derived from one of my own projects:
var param1 = new ObjectParameter("dec", typeof(decimal)) { Value = 90000m };
var param2 = new ObjectParameter("int", typeof(int)) { Value = 90000 };
var q = ValueHolders.Where("it.DecimalValue >= #dec OR it.IntegerValue > #int",
param1, param2).ToList();
Note that param1, param2 could also be an array of ObjectParameter.