Get result from function in batches - c#

I have a function that gets x numbers og items from a SharePoint list. It takes the items in batches. After each batch I do something with the items, destroys everything and take the next batch for calculation. I currently consider to use events. So have an event for each batch. Is this the correct strategy or is there a better way to do it? I am thinking anonymous functions or something similar?
public static List<Item> GetAllItems(this List list, int rowLimit, List<string> fields, bool includeRoleAssignments, ILogger logger)
{
var result = new List<Item>();
var ctx = list.Context;
ListItemCollectionPosition position = null;
var camlQuery = new CamlQuery();
camlQuery.ViewXml =
#"<View Scope='RecursiveAll'>
<Query>
<OrderBy Override='TRUE'><FieldRef Name='ID'/></OrderBy>
</Query>
<ViewFields></ViewFields>" +
"<RowLimit Paged='TRUE'>" + rowLimit + "</RowLimit>" +
"</View>";
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
do
{
try
{
using (var clonedCtx = ctx.Clone(ctx.Url))
{
List listWithClonedContext = clonedCtx.Web.Lists.GetByTitle(list.Title);
clonedCtx.Load(listWithClonedContext);
clonedCtx.ExecuteQuery();
ListItemCollection listItems = null;
camlQuery.ListItemCollectionPosition = position;
listItems = listWithClonedContext.GetItems(camlQuery);
foreach (string field in fields)
{
clonedCtx.Load(listItems, includes => includes.Include(i => i[field]));
}
if (!includeRoleAssignments) {
clonedCtx.Load(listItems, item => item.ListItemCollectionPosition);
}
else {
clonedCtx.Load(listItems, item =>
item.ListItemCollectionPosition,
item => item.Include(
i => i.RoleAssignments.Include(
ra => ra.Member,
ra => ra.Member.LoginName,
ra => ra.RoleDefinitionBindings.Include(rd => rd.Description, rd => rd.Name))));
}
clonedCtx.Load(listItems, item => item.ListItemCollectionPosition);
clonedCtx.ExecuteQueryWithIncrementalRetry(3, 1, logger);
// here i want to do something with items before next loop/batch
position = listItems.ListItemCollectionPosition;
if (position != null)
{
logger.WriteTrace(string.Format("Iteration on getting items performed: {0}", position.PagingInfo), SeverityLevel.Verbose);
}
else
{
logger.WriteTrace("Getting all items finished.", SeverityLevel.Verbose);
}
logger.Flush();
}
}
catch (Exception ex)
{
logger.WriteException(ex);
}
}
while (position != null);
return result;
}

Maybe events are an option but there might also be an easier way to "stream" them out instead of returning all at once with a list. Therefore use yield and change to IEnumerable<Item>:
public static IEnumerable<Item> EnumerateItems(this List list, int rowLimit, List<string> fields, bool includeRoleAssignments, ILogger logger)
{
// ...
do
{
try
{
using (var clonedCtx = ctx.Clone(ctx.Url))
{
//...
camlQuery.ListItemCollectionPosition = position;
listItems = listWithClonedContext.GetItems(camlQuery);
// ...
foreach(Item x in listItems)
{
yield return x;
}
position = listItems.ListItemCollectionPosition;
// ...
}
while (position != null);
}
On this way you can start processing them while you are still fetching them or you could filter them, for example with Where,Skip or Take without loading all into memory first.

Related

How do I pick out values between a duplicate value in a collection?

I have a method that returns a collection that has a duplicate value.
static List<string> GenerateItems()
{
var _items = new List<string>();
_items.Add("Tase");
_items.Add("Ray");
_items.Add("Jay");
_items.Add("Bay");
_items.Add("Tase");
_items.Add("Man");
_items.Add("Ran");
_items.Add("Ban");
return _items;
}
I want to search through that collection and find the first place that duplicate value is located and start collecting all the values from the first appearance of the duplicate value to its next appearance. I want to put this in a collection but I only want the duplicate value to appear once in that collection.
This is what I have so far but.
static void Main(string[] args)
{
string key = "Tase";
var collection = GenerateItems();
int index = collection.FindIndex(a => a == key);
var matchFound = false;
var itemsBetweenKey = new List<string>();
foreach (var item in collection)
{
if (item == key)
{
matchFound = !matchFound;
}
if (matchFound)
{
itemsBetweenKey.Add(item);
}
}
foreach (var item in itemsBetweenKey)
{
Console.WriteLine(item);
}
Console.ReadLine();
}
There must be an easier way of doing this. Perhaps with Indexing or a LINQ query?
You can do something like that
string key = "Tase";
var collection = GenerateItems();
int indexStart = collection.FindIndex(a => a == key);
int indexEnd = collection.FindIndex(indexStart+1, a => a == key);
var result = collection.GetRange(indexStart, indexEnd-indexStart);
You can use linq select and group by to find the first index and last index of all duplicates (Keep in mind if something is in the list more then 2 times it would ignore the middle occurences.
But I personally think the linq for this seems overcomplicated. I would stick with simple for loops and if statements (Just turn it into a method so it reads better)
Here is a solution with Linq to get all duplicate and all values between those duplicates including itself once as you mentioned.
var collection = GenerateItems();
var Duplicates = collection.Select((x,index) => new { index, value = x })
.GroupBy(x => x.value)//group by the strings
.Where(x => x.Count() > 1)//only take duplicates
.Select(x=>new {
Value = x.Key,
FirstIndex = x.Min(y=> y.index),//take first occurenc
LastIndex = x.Max(y => y.index)//take last occurence
}).ToList();
var resultlist = new List<List<string>>();
foreach (var duplicaterange in Duplicates)
resultlist .Add(collection.GetRange(duplicaterange.FirstIndex, duplicaterange.LastIndex - duplicaterange.FirstIndex));
Try this function
public List<string> PickOut(List<string> collection, string key)
{
var index = 0;
foreach (var item in collection)
{
if (item == key)
{
return collection.Skip(index).TakeWhile(x=> x != key).ToList();
}
index++;
};
return null;
}
First finding the duplicate key then find the second occurrence of the item and then take result.
var firstduplicate = collection.GroupBy(x => x)
.Where(g => g.Count() > 1)
.Select(g => g.Key).First();
var indices = collection.Select((b, i) => b == firstduplicate ? i : -1).Where(i => i != -1).Skip(1).FirstOrDefault();
if (indices>0)
{
var result = collection.Take(indices).ToList();
}

Group list of strings with common prefixes

Suppose I have a list of strings [city01, city01002, state02, state03, city04, statebg, countryqw, countrypo]
How do I group them in a dictionary of <string, List<Strings>> like
city - [city01, city04, city01002]
state- [state02, state03, statebg]
country - [countrywq, countrypo]
If not code, can anyone please help with how to approach or proceed?
As shown in other answers you can use the GroupBy method from LINQ to create this grouping based on any condition you want. Before you can group your strings you need to know the conditions for how a string is grouped. It could be that it starts with one of a set of predefined prefixes, grouped by whats before the first digit or any random condition you can describe with code. In my code example the groupBy method calls another method for every string in your list and in that method you can place the code you need to group the strings as you want by returning the key to group the given string under. You can test this example online with dotnetfiddle: https://dotnetfiddle.net/UHNXvZ
using System;
using System.Collections.Generic;
using System.Linq;
public class Program
{
public static void Main()
{
List<string> ungroupedList = new List<string>() {"city01", "city01002", "state02", "state03", "city04", "statebg", "countryqw", "countrypo", "theFirstTown"};
var groupedStrings = ungroupedList.GroupBy(x => groupingCondition(x));
foreach (var a in groupedStrings) {
Console.WriteLine("key: " + a.Key);
foreach (var b in a) {
Console.WriteLine("value: " + b);
}
}
}
public static string groupingCondition(String s) {
if(s.StartsWith("city") || s.EndsWith("Town"))
return "city";
if(s.StartsWith("country"))
return "country";
if(s.StartsWith("state"))
return "state";
return "unknown";
}
}
You can use LINQ:
var input = new List<string>()
{ "city01", "city01002", "state02",
"state03", "city04", "statebg", "countryqw", "countrypo" };
var output = input.GroupBy(c => string.Join("", c.TakeWhile(d => !char.IsDigit(d))
.Take(4))).ToDictionary(c => c.Key, c => c.ToList());
i suppose you have a list of references you are searching in the list:
var list = new List<string>()
{ "city01", "city01002", "state02",
"state03", "city04", "statebg", "countryqw", "countrypo" };
var tofound = new List<string>() { "city", "state", "country" }; //references to found
var result = new Dictionary<string, List<string>>();
foreach (var f in tofound)
{
result.Add(f, list.FindAll(x => x.StartsWith(f)));
}
In the result, you have the dictionary wanted. If no value are founded for a reference key, the value of key is null
Warning: This answer has a combinatorial expansion and will fail if your original string set is large. For 65 words I gave up after running for a couple of hours.
Using some IEnumerable extension methods to find Distinct sets and to find all possible combinations of sets, you can generate a group of prefixes and then group the original strings by these.
public static class IEnumerableExt {
public static bool IsDistinct<T>(this IEnumerable<T> items) {
var hs = new HashSet<T>();
foreach (var item in items)
if (!hs.Add(item))
return false;
return true;
}
public static bool IsEmpty<T>(this IEnumerable<T> items) => !items.Any();
public static IEnumerable<IEnumerable<T>> AllCombinations<T>(this IEnumerable<T> start) {
IEnumerable<IEnumerable<T>> HelperCombinations(IEnumerable<T> items) {
if (items.IsEmpty())
yield return items;
else {
var head = items.First();
var tail = items.Skip(1);
foreach (var sequence in HelperCombinations(tail)) {
yield return sequence; // Without first
yield return sequence.Prepend(head);
}
}
}
return HelperCombinations(start).Skip(1); // don't return the empty set
}
}
var keys = Enumerable.Range(0, src.Count - 1)
.SelectMany(n1 => Enumerable.Range(n1 + 1, src.Count - n1 - 1).Select(n2 => new { n1, n2 }))
.Select(n1n2 => new { s1 = src[n1n2.n1], s2 = src[n1n2.n2], Dist = src[n1n2.n1].TakeWhile((ch, n) => n < src[n1n2.n2].Length && ch == src[n1n2.n2][n]).Count() })
.SelectMany(s1s2d => new[] { new { s = s1s2d.s1, s1s2d.Dist }, new { s = s1s2d.s2, s1s2d.Dist } })
.Where(sd => sd.Dist > 0)
.GroupBy(sd => sd.s.Substring(0, sd.Dist))
.Select(sdg => sdg.Distinct())
.AllCombinations()
.Where(sdgc => sdgc.Sum(sdg => sdg.Count()) == src.Count)
.Where(sdgc => sdgc.SelectMany(sdg => sdg.Select(sd => sd.s)).IsDistinct())
.OrderByDescending(sdgc => sdgc.Sum(sdg => sdg.First().Dist)).First()
.Select(sdg => sdg.First())
.Select(sd => sd.s.Substring(0, sd.Dist))
.ToList();
var groups = src.GroupBy(s => keys.First(k => s.StartsWith(k)));

check discontinuity of multiple ranges in a list

I would like to ask you if there's a way by Linq to check discontinuity of multiple ranges, for example we have a class AgeRange:
public class AgeRange
{
public int firstValue {get;set;}
public int secondValue {get;set;}
}
var ageRange1 = new AgeRange(0,2); // interval [0,2]
var ageRange2 = new AgeRange(4,10); // interval [4,10]
var ageRange3 = new AgeRange(11,int.MaxValue); // interval [11,+oo[
var ageRangeList = new List<AgeRange>();
ageRangeList.Add(ageRange1);
ageRangeList.Add(ageRange2);
ageRangeList.Add(ageRange3);
in this example we have a discontinuity between first range and second range.
is there a way in Linq to check discontinuity between elements in ageRangeList ?
Thanks for you help.
Assuming firstValue always <= secondValue (for the same element), you can try to use Aggregate:
var start = ageRangeList
.OrderBy(a => a.firstValue).Dump()
.First();
var result = ageRangeList
.OrderBy(a => a.firstValue)
.Aggregate(
(hasGap: false, s: start.secondValue),
(tuple, range) =>
{
if (tuple.hasGap)
{
return tuple;
}
else
{
var max = Math.Max(tuple.s, tuple.s+1); //hacky overflow protection
if (max < range.firstValue)
{
return (true, tuple.s);
}
else
{
return (false, Math.Max(tuple.s, range.secondValue));
}
}
})
.hasGap;
The downside of such approach is that it still will need to loop through all age ranges.
If you want to find first discontinuity and use that information elsewhere
public static IEnumerable<AgeRange> FindDiscontinuity(List<AgeRange> ageRangeList) {
foreach(var ageRange in ageRangeList.Zip(ageRangeList.Skip(1), (a, b) => new {Prev = a, Current = b})) {
if(ageRange.Prev.SecondValue != ageRange.Current.FirstValue) {
yield return ageRange.Prev;
yield return ageRange.Current;
break;
}
}
}
public static void Main()
{
var ageRange1 = new AgeRange(0, 2);
var ageRange2 = new AgeRange(4, 10);
var ageRange3 = new AgeRange(11, int.MaxValue);
var ageRangeList = new List<AgeRange>();
ageRangeList.Add(ageRange1);
ageRangeList.Add(ageRange2);
ageRangeList.Add(ageRange3);
var result = FindDiscontinuity(ageRangeList);
foreach(var ageRange in result) {
Console.WriteLine("{0}, {1}", ageRange.FirstValue, ageRange.SecondValue);
}
}
You can change the function so it can return boolean value instead of data.

Merge two or more T in List<T> based on condition

I have the below class:
public class FactoryOrder
{
public string Text { get; set; }
public int OrderNo { get; set; }
}
and collection holding the list of FactoryOrders
List<FactoryOrder>()
here is the sample data
FactoryOrder("Apple",20)
FactoryOrder("Orange",21)
FactoryOrder("WaterMelon",42)
FactoryOrder("JackFruit",51)
FactoryOrder("Grapes",71)
FactoryOrder("mango",72)
FactoryOrder("Cherry",73)
My requirement is to merge the Text of FactoryOrders where orderNo are in sequence and retain the lower orderNo for the merged FactoryOrder
- so the resulting output will be
FactoryOrder("Apple Orange",20) //Merged Apple and Orange and retained Lower OrderNo 20
FactoryOrder("WaterMelon",42)
FactoryOrder("JackFruit",51)
FactoryOrder("Grapes mango Cherry",71)//Merged Grapes,Mango,cherry and retained Lower OrderNo 71
I am new to Linq so not sure how to go about this. Any help or pointers would be appreciated
As commented, if your logic depends on consecutive items so heavily LINQ is not the easiest appoach. Use a simple loop.
You could order them first with LINQ: orders.OrderBy(x => x.OrderNo )
var consecutiveOrdernoGroups = new List<List<FactoryOrder>> { new List<FactoryOrder>() };
FactoryOrder lastOrder = null;
foreach (FactoryOrder order in orders.OrderBy(o => o.OrderNo))
{
if (lastOrder == null || lastOrder.OrderNo == order.OrderNo - 1)
consecutiveOrdernoGroups.Last().Add(order);
else
consecutiveOrdernoGroups.Add(new List<FactoryOrder> { order });
lastOrder = order;
}
Now you just need to build the list of FactoryOrder with the joined names for every group. This is where LINQ and String.Join can come in handy:
orders = consecutiveOrdernoGroups
.Select(list => new FactoryOrder
{
Text = String.Join(" ", list.Select(o => o.Text)),
OrderNo = list.First().OrderNo // is the minimum number
})
.ToList();
Result with your sample:
I'm not sure this can be done using a single comprehensible LINQ expression. What would work is a simple enumeration:
private static IEnumerable<FactoryOrder> Merge(IEnumerable<FactoryOrder> orders)
{
var enumerator = orders.OrderBy(x => x.OrderNo).GetEnumerator();
FactoryOrder previousOrder = null;
FactoryOrder mergedOrder = null;
while (enumerator.MoveNext())
{
var current = enumerator.Current;
if (mergedOrder == null)
{
mergedOrder = new FactoryOrder(current.Text, current.OrderNo);
}
else
{
if (current.OrderNo == previousOrder.OrderNo + 1)
{
mergedOrder.Text += current.Text;
}
else
{
yield return mergedOrder;
mergedOrder = new FactoryOrder(current.Text, current.OrderNo);
}
}
previousOrder = current;
}
if (mergedOrder != null)
yield return mergedOrder;
}
This assumes FactoryOrder has a constructor accepting Text and OrderNo.
Linq implementation using side effects:
var groupId = 0;
var previous = Int32.MinValue;
var grouped = GetItems()
.OrderBy(x => x.OrderNo)
.Select(x =>
{
var #group = x.OrderNo != previous + 1 ? (groupId = x.OrderNo) : groupId;
previous = x.OrderNo;
return new
{
GroupId = group,
Item = x
};
})
.GroupBy(x => x.GroupId)
.Select(x => new FactoryOrder(
String.Join(" ", x.Select(y => y.Item.Text).ToArray()),
x.Key))
.ToArray();
foreach (var item in grouped)
{
Console.WriteLine(item.Text + "\t" + item.OrderNo);
}
output:
Apple Orange 20
WaterMelon 42
JackFruit 51
Grapes mango Cherry 71
Or, eliminate the side effects by using a generator extension method
public static class IEnumerableExtensions
{
public static IEnumerable<IList<T>> MakeSets<T>(this IEnumerable<T> items, Func<T, T, bool> areInSameGroup)
{
var result = new List<T>();
foreach (var item in items)
{
if (!result.Any() || areInSameGroup(result[result.Count - 1], item))
{
result.Add(item);
continue;
}
yield return result;
result = new List<T> { item };
}
if (result.Any())
{
yield return result;
}
}
}
and your implementation becomes
var grouped = GetItems()
.OrderBy(x => x.OrderNo)
.MakeSets((prev, next) => next.OrderNo == prev.OrderNo + 1)
.Select(x => new FactoryOrder(
String.Join(" ", x.Select(y => y.Text).ToArray()),
x.First().OrderNo))
.ToList();
foreach (var item in grouped)
{
Console.WriteLine(item.Text + "\t" + item.OrderNo);
}
The output is the same but the code is easier to follow and maintain.
LINQ + sequential processing = Aggregate.
It's not said though that using Aggregate is always the best option. Sequential processing in a for(each) loop usually makes for better readable code (see Tim's answer). Anyway, here's a pure LINQ solution.
It loops through the orders and first collects them in a dictionary having the first Id of consecutive orders as Key, and a collection of orders as Value. Then it produces a result using string.Join:
Class:
class FactoryOrder
{
public FactoryOrder(int id, string name)
{
this.Id = id;
this.Name = name;
}
public int Id { get; set; }
public string Name { get; set; }
}
The program:
IEnumerable<FactoryOrder> orders =
new[]
{
new FactoryOrder(20, "Apple"),
new FactoryOrder(21, "Orange"),
new FactoryOrder(22, "Pear"),
new FactoryOrder(42, "WaterMelon"),
new FactoryOrder(51, "JackFruit"),
new FactoryOrder(71, "Grapes"),
new FactoryOrder(72, "Mango"),
new FactoryOrder(73, "Cherry"),
};
var result = orders.OrderBy(t => t.Id).Aggregate(new Dictionary<int, List<FactoryOrder>>(),
(dir, curr) =>
{
var prevId = dir.SelectMany(d => d.Value.Select(v => v.Id))
.OrderBy(i => i).DefaultIfEmpty(-1)
.LastOrDefault();
var newKey = dir.Select(d => d.Key).OrderBy(i => i).LastOrDefault();
if (prevId == -1 || curr.Id - prevId > 1)
{
newKey = curr.Id;
}
if (!dir.ContainsKey(newKey))
{
dir[newKey] = new List<FactoryOrder>();
}
dir[newKey].Add(curr);
return dir;
}, c => c)
.Select(t => new
{
t.Key,
Items = string.Join(" ", t.Value.Select(v => v.Name))
}).ToList();
As you see, it's not really straightforward what happens here, and chances are that it performs badly when there are "many" items, because the growing dictionary is accessed over and over again.
Which is a long-winded way to say: don't use Aggregate.
Just coded a method, it's compact and quite good in terms of performance :
static List<FactoryOrder> MergeValues(List<FactoryOrder> dirtyList)
{
FactoryOrder[] temp1 = dirtyList.ToArray();
int index = -1;
for (int i = 1; i < temp1.Length; i++)
{
if (temp1[i].OrderNo - temp1[i - 1].OrderNo != 1) { index = -1; continue; }
if(index == -1 ) index = dirtyList.IndexOf(temp1[i - 1]);
dirtyList[index].Text += " " + temp1[i].Text;
dirtyList.Remove(temp1[i]);
}
return dirtyList;
}

Searching a nested List<T>

I have this data structure:
class Conference
{
private List<List<string>>_orgs;
public List<List<string>> Orgs
{
set { _orgs = value; } get { return _orgs; }
}
}
Data in this collection:
List<string> sublist = new List<string>();
sublist.Add("university");
sublist.Add("organization");
List<List<string>> list = new List<List<string>>();
list.Add(sublist);
Then:
Conference c = new Conference();
c.Orgs = list;
I have collection of conference objects:
List<Conference> listConferences = new List<Conference>();
listConferences.Add(c);
I want search a string like "uni" and find collection of conference have orgs like "uni". How can I do this?
You can do this:
var selection = listConferences
.Where(x => x.Orgs.SelectMany(y => y).Any(y => y.Contains("uni")))
.ToList();
Note:
the trailing ToList() might not be necessary depending on your needs (e.g. if you iterate selection only once you can skip it).
Please use this code, below;
instead of third one, you may use your own conference list. you can now use similar to like keyword.
List<string> first = new List<string>();
first.Add("University");
first.Add("Standard");
List<List<string>> second = new List<List<string>>();
second.Add(first);
List<List<List<string>>> third = new List<List<List<string>>>();
third.Add(second);
var e = third.Find(delegate(List<List<string>> r)
{
bool isValid = false;
if(r.Count > 0)
{
foreach(List<string> s in r)
{
if(s.Count > 0 )
{
isValid = s.FindAll(delegate(string t){ return t.StartsWith("uni", StringComparison.OrdinalIgnoreCase);}).Count > 0;
}
}
}
return isValid;
});
Done, one more workout using linq. You should be feeling comfortable with this:
var univ = from p in c.Orgs
select p.FindAll(r => r.FindAll(s => s.StartsWith("univ", StringComparison.OrdinalIgnoreCase)));

Categories

Resources