Develop Reference

Time series LINQ query

I have a central repository for IoT device logs. So as the logs arrive they have a timestamp. The problem I want to solve is, over a given time span, the same device might send multiple logs regarding its interaction with a specific catalyst. I want to consider that set of logs as a single event and not 5 disparate logs. I want to count the number of interactions. and not the number of logs.
Data Set
public class Data
{
public Guid DeviceId {get; set;}
public DateTime StartTime { get; set; }
public DateTime EndDateTime { get; set; }
public int Id { get; set; }
public int Direction { get; set;}
}
Data d1 = new Data();// imagine it's populated
Data d2 = new Data();// imagine it's populated
I am looking for a LINQ query that would yield something along the lines of
If ((d1.DeviceId == d2.DeviceId ) && (d1.Id == d2.Id) && (d1.Direction == d2.Direction) && (d1.StartTime - d2.StartTime < 15 minutes ))
If i know that the same IoT device is interacting with the same Id (catalyst) and the Direction is the same, and all of those logs occur within a 15 minute time span, It can be presumed that they correspond to the same catalyst event.
I do not control the log creation so ... no i cannot update the data to include "something" that would indicate the relationship.
Data per request... nothing fancy. I am sure most people suspect that I have 30+ properties and I only provide the one impacted by the calculation, but this is a simple set of possibilities
class SampleData
{
public List<Data> GetSampleData()
{
Guid device1 = Guid.NewGuid();
List<Data> dataList = new List<Data>();
Data data1 = new Data();
data1.DeviceId = device1;
data1.Id = 555;
data1.Direction = 1;
data1.StartTime = new DateTime(2010, 8, 18, 16, 32, 0);
data1.EndDateTime = new DateTime(2010, 8, 18, 16, 32, 30);
dataList.Add(data1);
//so this data point should be excluded in the final result
Data data2 = new Data();
data1.DeviceId = device1;
data1.Id = 555;
data1.Direction = 1;
data1.StartTime = new DateTime(2010, 8, 18, 16, 32, 32);
data1.EndDateTime = new DateTime(2010, 8, 18, 16, 33, 30);
dataList.Add(data2);
//Should be included because ID is different
Data data3 = new Data();
data1.DeviceId = device1;
data1.Id = 600;
data1.Direction = 1;
data1.StartTime = new DateTime(2010, 8, 18, 16, 32, 2);
data1.EndDateTime = new DateTime(2010, 8, 18, 16, 32, 35);
dataList.Add(data3);
//exclude due to time
Data data4 = new Data();
data1.DeviceId = device1;
data1.Id = 600;
data1.Direction = 1;
data1.StartTime = new DateTime(2010, 8, 18, 16, 32, 37);
data1.EndDateTime = new DateTime(2010, 8, 18, 16, 33, 40);
dataList.Add(data4);
//include because time > 15 minutes
Data data5 = new Data();
data1.DeviceId = device1;
data1.Id = 600;
data1.Direction = 1;
data1.StartTime = new DateTime(2010, 8, 18, 16, 58, 42);
data1.EndDateTime = new DateTime(2010, 8, 18, 16, 58, 50);
dataList.Add(data5);
return dataList;
}

This turned out to be more complex than I hoped for.
I used a custom LINQ extension method I have called ScanPair which is a variation of my Scan method, which is an version of the APL scan operator (which is like Aggregate, but returns the intermediate results). ScanPair returns the intermediate results of the operation along with each original value. I think I need to think about how to make all of these more general purpose, as the pattern is used by a bunch of other extension methods I have for grouping by various conditions (e.g. sequential, runs, while test is true or false).
public static class IEnumerableExt {
public static IEnumerable<(TKey Key, T Value)> ScanPair<T, TKey>(this IEnumerable<T> src, Func<T, TKey> seedFn, Func<(TKey Key, T Value), T, TKey> combineFn) {
using (var srce = src.GetEnumerator()) {
if (srce.MoveNext()) {
var seed = (seedFn(srce.Current), srce.Current);
while (srce.MoveNext()) {
yield return seed;
seed = (combineFn(seed, srce.Current), srce.Current);
}
yield return seed;
}
}
}
}
Now, you can use a tuple as an intermediate result to track the initial timestamp and the group number, and increment to the next (timestamp, group number) when the interval goes over 15 minutes. If you first group by the interaction, and then count the less than 15-minute groups per interaction, you get the answer:
var ans = interactionLogs.GroupBy(il => new { il.DeviceId, il.Id, il.Direction })
.Select(ilg => new {
ilg.Key,
Count = ilg.OrderBy(il => il.Timestamp)
.ScanPair(il => (firstTimestamp: il.Timestamp, groupNum: 1), (kvp, cur) => (cur.Timestamp - kvp.Key.firstTimestamp).TotalMinutes <= 15 ? kvp.Key : (cur.Timestamp, kvp.Key.groupNum + 1))
.GroupBy(ilkvp => ilkvp.Key.groupNum, ilkvp => ilkvp.Value)
.Count()
});
Here is a portion of a sample of intermediate results from ScanPair - the actual result is a ValueTuple with two fields, where the Key is the intermediate result (which is the ValueTuple of firstTimestamp,groupNum) and Value is the corresponding source (log) item. Using the function seeded version puts the first source item into the seed function to begin the process.
Key_firstTimestamp Key_groupNum Timestamp
7:58 PM 1 7:58 PM
7:58 PM 1 8:08 PM
7:58 PM 1 8:12 PM
8:15 PM 2 8:15 PM
8:15 PM 2 8:20 PM

Conditionally Include XElement in LINQ Select Operation?

I have working code in which I create a new var of type EnumerableRowCollection < XElement >. I have a new requirement where one of the XElements representing an Address must be conditionally included based on a Document Type value.
public class Taxes
{
public int DocumentType { get; set; }
private XElement BuildBodyXML()
{
// other stuff
Address billAddrObj = GetBillTo(dt);
Address buyerAddrObj = GetBuyerPrimary(dt);
var xBillTo = BuildAddress(billAddrObj, "BILL_TO");
var xBuyer = BuildAddress(buyerAddrObj, "BUYER_PRIMARY");
var INVOICE = from row in dt.AsEnumerable()
select new XElement(tcr + "INVOICE",
xBillTo, // This one needs to be conditionally included based on DocumentType
xBuyer,
// ... other elements ...
new XElement(tcr + "INVOICE_NUMBER", row.Field<string>("DOCNUMBR").Trim()));
// other stuff
return INVOICE;
}
public XElement BuildAddress(Address anAddress, string Name)
{
var xAddress = new XElement(tcr + Name);
// other stuff
return xAddress;
}
}
The Bill To XElement must be included conditionally based on the value of DocumentType. Can you help me achieve this?
UPDATE (Solution derived from answer by tinstaafl): I used the following code:
(new[] { 1, 2, 3, 4, 5, 6, 7, 8, 16, 17, 18, 19, 20, 21, 22, 23, 24 }.Contains(DocumentType) ? xBillTo : null),

You could use the ternary operator and set the conditional response that doesn't require the xBillTo object to null.

C# merge multiple lists based on timestamp

I have a data set that comes as a list of objects in C#, looking something below.
public class MotorDataModel
{
public DateTime timestamp { set; get; }
public decimal MotorSpeed { set; get; }
public decimal MotorTemp { set; get; }
public decimal MotorKw{ set; get; }
}
public class MotorModel
{
public string MotorName { set; get; }
public List<MotorDataModel> MotorData { set; get; }
}
When I do the query, I will have 1 or more MotorModel records coming back (say motor 1, 2, 3, ...), each with their own timestamps, and various data points at those time stamps.
I am then sending this data to a javascript charting library, which takes the data in as a data table (e.g. spreadsheet like format), such as:
TimeStamp | Motor1:kW | Motor1:Speed | Motor1:Temp | Motor2:kW |Motor2:Speed ...
with the data following in rows. The data will be grouped on the timestamp, which should be the within a couple minutes of each other, in a consistent increment (say 15 minutes).
The plan is to transform the data in C#, convert it to JSON, and and send it to the chart library (Google Chart).
I don't have to format this in C#, and could convert the Object list data in C# to JSON, and reformat it in javascript on the client, but it seems better to transform it at the server.
Either way, I am struggling on how to transform the data from a multiple list of objects to a "datatable" like view.
This answer via LINQ seems to be close, but I have multiple lists of equipment, not a defined number.
I have also looked at just looping through and building the data table (or array), but unsure of what structure makes the most sense.
So, if anyone has done something similar, or has any feedback, it would be much appreciated.
Suggested format for providing sample data
Below is some sample data provided by BlueMonkMN. Please update the question providing sample data representative of your actual question.
List<MotorModel> allData = new List<MotorModel>() {
new MotorModel() {MotorName="Motor1", MotorData = new List<MotorDataModel> {
new MotorDataModel(){timestamp=new DateTime(2016, 9, 18, 2, 56, 0), MotorSpeed=20.0M, MotorTemp=66.2M, MotorKw=5.5M},
new MotorDataModel(){timestamp=new DateTime(2016, 9, 18, 3, 10, 30), MotorSpeed=10.0M, MotorTemp=67.0M, MotorKw=5.5M},
new MotorDataModel(){timestamp=new DateTime(2016, 9, 18, 3, 25, 45), MotorSpeed=17.5M, MotorTemp=66.1M, MotorKw=5.8M},
new MotorDataModel(){timestamp=new DateTime(2016, 9, 18, 3, 40, 23), MotorSpeed=22.2M, MotorTemp=65.8M, MotorKw=5.4M}
}},
new MotorModel() {MotorName="Motor2", MotorData = new List<MotorDataModel> {
new MotorDataModel(){timestamp=new DateTime(2016, 9, 18, 2, 58, 0), MotorSpeed=21.0M, MotorTemp=67.2M, MotorKw=5.6M},
new MotorDataModel(){timestamp=new DateTime(2016, 9, 18, 3, 11, 30), MotorSpeed=11.0M, MotorTemp=68.0M, MotorKw=5.6M},
new MotorDataModel(){timestamp=new DateTime(2016, 9, 18, 3, 24, 45), MotorSpeed=18.5M, MotorTemp=67.1M, MotorKw=5.9M},
new MotorDataModel(){timestamp=new DateTime(2016, 9, 18, 3, 39, 23), MotorSpeed=23.2M, MotorTemp=66.8M, MotorKw=5.5M}
}}
};

One possibility is to iterate through all the data, and build the table, as you suggest. I would suggest using a Dictionary, with the timestamp as the key. For each timestamp there can be multiple MotorData's, so it could have a list, like this:
Dictionary<DateTime, List<MotorDataModel>>
A code snippet to build this table would look like this:
List<MotorModel> motorModels; // filled in previously
// build result structure in this dictionary:
Dictionary<DateTime, List<MotorDataModel>> table = new Dictionary<DateTime, List<MotorDataModel>>();
// iterate through all motors and their data, and fill in the table
foreach(MotorModel m in motorModels)
{
foreach(MotorDataModel md in m.MotorData)
{
DateTime ts = md.timestamp;
// if this is the first occurance of the timestamp, create new 'row'
if (!table.ContainsKey(ts)) table[ts] = new List<MotorDataModel>();
// add the data to the 'row' of this timestamp
table[ts].Add(md);
}
}
// output the table
foreach(DateTime ts in table.Keys)
{
...
foreach(MotorDataModel md in table[ts])
{
...
}
}

I'd use Json.NET from NewtonSoft.
JObject o = new JObject();
foreach (MotorModel mm in allData) {
foreach (MotorDataModel mdm : mm.MotorData()) {
string key = mdm.TimeStamp.ToString(); // Or do your own format
o[key][mm.MotorName + ":kW"] = mdm.MotorKw;
o[key][mm.MotorName + ":Speed"] = mdm.MotorSpeed;
o[key][mm.MotorName + ":TEmp"] = mdm.MotorTemp;
}
}

Could you try something like this to compute your data:
var motorData = allData.SelectMany(x => x.MotorData).ToArray();
var starting = motorData.Min(x => x.timestamp);
var ending = motorData.Max(x => x.timestamp);
var duration = ending.Subtract(starting);
var blocks = (int)Math.Ceiling(duration.TotalMinutes / 15.0);
var query =
from b in Enumerable.Range(0, blocks)
let s = starting.AddMinutes(b * 15.0)
let e = starting.AddMinutes((b + 1.0) * 15.0)
select new
{
Timestamp = s,
MotorSpeedAverage =
motorData
.Where(x => x.timestamp >= s && x.timestamp < e)
.Average(x => x.MotorSpeed),
};
I get this result:

List to array (or?) for graphing purposes

I would like to know how to take one element from each item in a list and put it into an array for graphing purposes. I was wondering if, since I want to graph each element in each row of the list, could I do it all at once or would I have to separately pull each element out into its own array?
A few lines of a code example is my preferred learning method and would be much appreciated.
Thanks!

You can conceptually put the logic to do this in one place.
class DataIterator<T> : IEnumerable<T[]> {
private readonly IList<IList<T>> _lists;
public DataIterator(IList<IList<T>> lists) {
Contract.Assert(lists.All(l => l.Count == lists[0].Count));
_lists = lists;
}
public IEnumerator<T[]> GetEnumerator() {
var value = new List<T>(_lists.Count);
for (var i = 0; i < _lists[0].Count; i++) {
value.AddRange(_lists.Select(t => t[i]));
yield return value.ToArray();
value = new List<T>(_lists.Count);
}
}
IEnumerator IEnumerable.GetEnumerator() {
return GetEnumerator();
}
}
This implements IEnumerable<T> so you can use it which foreach loops.
An example of use:
var lists = new List<IList<int>> {
new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 },
new List<int> { 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 },
new List<int> { 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 }
};
var iter = new DataIterator<int>(lists);
foreach (var items in iter) {
Array.ForEach(items, i => {
Console.Write("{0:D2} ", i);
});
Console.WriteLine();
}
outputs:
01 11 21
02 12 12
...

c# loop through all fields of enum assigning values from string array

I am building a Soap Body for a web-service, and there are dozens of optional fields.
Currently I have been handling these like this:
wsSoapBody.OrderType = aMessage[(int)cardCreate.OrderType].ToString();
wsSoapBody.ActivateFlag = Convert.ToInt32(aMessage[(int)cardCreate.ActivateFlag].ToString()); //P-02925;
if (aMessage[(int)cardCreate.ShipDate].ToString() != ""){
wsSoapBody.ShipmentDate = Convert.ToDateTime(aMessage[(int)cardCreate.ShipDate].ToString()); //P-02925;
}
wsSoapBody.ShipmentMethodCard = aMessage[(int)cardCreate.ShipMethodCard].ToString();
wsSoapBody.ShipmentMethodPin = aMessage[(int)cardCreate.ShipMethodPIN].ToString();
The CardCreate you see in those value assignments is an enumerated constant in the class cardCreate defined as below:
namespace EvryCardManagement
{
class CardCreate
{
#region Variables
private DCSSCardCreateType req;
private DCSSCardCreateResponseType rsp;
private DCSSCardCreate_V3_0Service stub;
public string tokenID { get; set; }
private enum cardCreate
{
MsgType = 0,
MsgVersion = 1,
WSName = 2,
ReplyTo = 3,
SourceSystem = 4,
Timestamp = 5,
UniqueMessageID = 6,
SFDCContext = 7,
InstitutionID = 8,
CardNumber = 9,
Version = 10,
ProductID = 11,
AccountNumber = 12,
CustomerID = 13,
CustomerNumber = 14,
EmbossName1 = 15,
Expiry = 16,
FeeMonth = 17,
ChargeAccountNo = 18,
PINMethod = 19,
CardFlag = 20,
AddressTypeCard = 21,
AddressTypePIN = 22,
OrderType = 23,
ActivateFlag = 24,
ShipDate = 25,
ShipMethodCard = 26,
ShipMethodPIN = 27,
FirstName = 28,
LastName = 29,
CardAddress1 = 30,
CardAddress2 = 31,
CardAddress3 = 32,
CardAddress4 = 33,
CardAddress5 = 34,
CardAddress6 = 35,
CardPostCode = 36,
CardCity = 37,
CardCountry = 38,
PINName = 39,
PINAddress1 = 40,
PINAddress2 = 41,
PINAddress3 = 42,
PINAddress4 = 43,
PINAddress5 = 44,
PINAddress6 = 45,
PINPostCode = 46,
PINCity = 47,
PINCountry = 48,
Validfrom = 49,
Note = 50,
MakeCheckStatus = 51,
EmbossName2 = 52,
PAmount = 53,
PAmountLength = 54,
GKIndicator = 55,
CreditLimit = 56,
CardDesignNo = 57,
ExtPictureID = 58,
BulkID = 59,
AccountNo2 = 60
}
so, rather than doing them all one by one as I have been doing, is it possible to loop through the wsSoapBody (which is defined in the web-service) and for each one, get the corresponding value from the aMessage (which is defined as an array like this string[] aMessage)
EDIT
I have the below code to loop through, but I want to assign to the wsSoapBody and I am stuck:
foreach (cardCreate cItem in (cardCreate[])Enum.GetValues(typeof(cardCreate)))
{
}
(the above correction was suggested as an edit by Steve Lillis that was rejected due to a conflict)
so I don't know how then to assign the values to each element for example I want to set
wsSoapBody[cItem].value = aMessage[(int)CardCreate[cItem]`
or I also tried:
wsSoapBody[cItem] = aMessage[(int)cItem].ToString();
but am having trouble making it work (or even compile) due to lack of knowledge
EDIT #2:
I have also looked at GetNames as possibly I want the names and tried:
foreach (string name in Enum.GetNames(typeof(cardCreate)))
{
wsSoapBody[name] = aMessage[(int)name].ToString();
}
But I cannot apply indexing with [] to an expression of type 'DCSSCardCreateType'
thanks

Why not place the values onto the enum itself and then enumerate?
For example using System.ComponentModel Description attribute we can add that information to the enum itself such as:
public enum cardCreate
{
[Description("General Response")]
MsgType = 0,
[Description("V2.0")]
WSName = 2,
[Description("OmegaMan")]
ReplyTo = 3,
[Description("Windows 10")]
SourceSystem = 4,
}
So when we call a special method to enumerate the enum where we can extract that text and use it appropriately later such as:
myextensions.GetEnumValues<cardCreate>()
.Select (ceEnum => new
{
Original = ceEnum,
IndexValue = (int)ceEnum,
Text = ceEnum.GetAttributeDescription()
})
The dynamic entity will look like this after the projection (the select):
Sweet! Now we have all the information in a easy consumable entity which provides all the information needed.
What? You need more than a string description? Then create a custom attribute on the enum and have all items/types of data to return as needed. For that see my blog article C# Using Extended Attribute Information on Objects.
Here are the extension methods used in the above example:
public static class myextensions
{
public static IEnumerable<T> GetEnumValues<T>()
{
Type type = typeof( T );
if (!type.IsEnum)
throw new Exception( string.Format("{0} is not an enum.", type.FullName ));
FieldInfo[] fields =
type.GetFields( BindingFlags.Public | BindingFlags.Static );
foreach (var item in fields)
yield return (T)item.GetValue( null );
}
/// <summary>If an attribute on an enumeration exists, this will return that
/// information</summary>
/// <param name="value">The object which has the attribute.</param>
/// <returns>The description string of the attribute or string.empty</returns>
public static string GetAttributeDescription( this object value )
{
string retVal = string.Empty;
try
{
retVal = value.GetType()
.GetField( value.ToString() )
.GetCustomAttributes( typeof( DescriptionAttribute ), false )
.OfType<DescriptionAttribute>()
.First()
.Description;
}
catch (NullReferenceException)
{
//Occurs when we attempt to get description of an enum value that does not exist
}
finally
{
if (string.IsNullOrEmpty( retVal ))
retVal = "Unknown";
}
return retVal;
}
}