I do have a database table with millions of archived value rows that consist of a value (single), two integer id fields, and a datetime field (and some other fields that are not relevant for the cache I want to build). The table structure is fixed, I can't change it. Now I want to read all rows into an array of objects of a simple class that I want to keep in memory for caching purposes.
In order to keep the memory consumption low, I want to use a unix timestamp instead of a datetime object. This is also useful because the frontend charts etc. that will consecutively work with this cache also natively work with Unix Timestamps.
For the cache creation I want to select directly into an ArchiveElement. This works well for most fields, but I don't know how to create a Unix Timestamp on the fly WITHIN the select statement:
ArchiveCache = db.ArchiveValues.Select(x => new ArchiveElement() {
DataPointId = (UInt16)x.DataPointId,
StationId = (UInt16)x.StationId,
Value = (Single)x.Value,
DateValue = x.DateValue // <-- Here I want to cast into Unix Timestamp
});
This is the ArchiveElement class:
public class ArchiveElement
{
public UInt32 DateValue;
public UInt16 DataPointId;
public UInt16 StationId;
public Single Value;
}
I do have a function in my application that can convert a DateTime into a Unix TimeStamp, but that does not work within the scope of the select statement. So I need to find another way.
One way would be to insert a .ToList() before the .Select(..) statement, so that I have access to my own function, but that would be an ugly workaround as it fetches a lot of unneccessary fields from the DB.
Is there a way to somehow convert the datetime "in place"?
So first you declare the Unix Epoch time:
var unixEpoch = DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
Now in your Select, just subtract the epoch from your date value which gives a TimeSpan type from which you can get the TotalSeconds which is the value you need:
ArchiveCache = db.ArchiveValues.Select(x => new ArchiveElement() {
DataPointId = (UInt16)x.DataPointId,
StationId = (UInt16)x.StationId,
Value = (Single)x.Value,
DateValue = (x.DateValue - unixEpoch).TotalSeconds
});
Note: This assumes that your DateValue property is a double, you may need to cast it to a long.
Edit
To cope with Entity Framework, this may work:
ArchiveCache = db.ArchiveValues.Select(x => new ArchiveElement() {
DataPointId = (UInt16)x.DataPointId,
StationId = (UInt16)x.StationId,
Value = (Single)x.Value,
DateValue = SqlFunctions.DateDiff("ss", unixEpoch, x.DateValue)
});
This may require a reference/import for System.Data.Entity.SqlServer.
DATEDIFF(SECOND,{d '1970-01-01'}, GETUTCDATE())
This will return the unix seconds in sql.
Related
I am converting ADODB namespace code to SqlClient. I am trying to replace code that utilized recordset.Value. I am having problems converting this line of code. How can I subtract DateTime.Now from the value in the time_of_lock (Datetime data type) column in SQL?
else if (DateTime.Now - rs.Fields["time_of_lock"].Value < TimeSpan.FromMinutes(15))
I am assuming you can get two DateTime values from your code. If so, try pop both into the following function:
private bool TimesApartNoMoreThan(DateTime first, DateTime second, int threshold)
{
return (second - first).TotalMinutes > threshold;
}
when you subtract two DateTimes, you get a TimeSpan. Having obtained that, you can represent it as whatever unit of duration you need (in this case, minutes):
From your statement, I am assuming rs.Fields["time_of_lock"] is of DateTime datatype. If that is the case, please try this:
DateTime.Now .Subtract(rs.Fields["time_of_lock"].Value).TotalMinutes < 15
I saw lots of duplicate post of this But for me it's something different.
I have a Datetime object and get time portion & assign another time to that.When i'm going to assign it it raise those error.
in here newStartDateGroup is a DateTime Object
in here OpenTime is a TimeSpan
Property or indexer cannot be assigned to — it is read only
else if(newStartDateGroup.TimeOfDay < i.OpenTime && newEndDateGroup.TimeOfDay > i.CloseTime) // < >
{
newStartDateGroup.TimeOfDay = i.OpenTime;
DateTime struct in .NET is immutable, so instead of changing it's values you have to create new one:
newStartDateGroup = newStartDateGroup.Date.Add(i.OpenTime);
You can only create new DateTime instances.
According to your shown code, you want to set only the day time.
This should do it:
newStartDateGroup = DateTime.Today.Add(i.OpenTime);
DateTime.Today is today at 00:00.
Just assign the hours, minutes and seconds manually.
DateTime temp = DateTime(newStartDateGroup.Year, newStartDateGroup.Month, newStartDateGroup.Day, i.OpenTime.Hours, i.OpenTime.Minutes, i.OpenTime.Seconds);
newStartDateGroup = temp;
I've dates array with values below:
"07/07/2011", "08/05/2011", "09/07/2011", "12/07/2011"
Using this as input in my C# program, I need to build a new collection which will have missing dates..ie. 10/07/2011, 11/07/2011.
Is recursion the best way to achieve this?
Thanks.
Not at all. This should be a straightforward process. You have a starting date, you have an interval... You start walking the array and if the next value does not match your previous value plus the interval you insert a new value into the new array. If it does match, you copy that value.
If you need more data (metadata) about each entry then create a class that holds the date and whatever metadata you find useful (e.g. a bool like this_value_was_inserted_artificially)
Using recursion would unnecessarily complicate things.
No recursion needed. This can probably be optimized, but should do the job:
public static IEnumerable<DateTime> FindMissingDates(IEnumerable<DateTime> input)
{
// get the range of dates to check
DateTime from = input.Min();
DateTime to = input.Max();
// how many days?
int numberOfDays = to.Subtract(from).Days;
// create an IEnumerable<DateTime> for all dates in the range
IEnumerable<DateTime> allDates = Enumerable.Range(0, numberOfDays)
.Select(n => from.AddDays(n));
// return all dates, except those found in the input
return allDates.Except(input);
}
You can pull this off very nicely with Linq:
var dates=new[]{
DateTime.Parse("07/07/2011"),
DateTime.Parse("08/07/2011"),
DateTime.Parse("09/07/2011"),
DateTime.Parse("12/07/2011")};
var days=(dates.Max()-dates.Min()).Days;
var otherDays=
Enumerable
.Range(0,days)
.Select(d=>dates.Min().AddDays(d))
.Except(dates);
It might be a simple fix, but I can't for the life of me think of how to do this. I compute a bunch of StartDates and End Dates into a bunch of arrays of dates using this query:
this.Reserved = unit.Reservations.Where(r => r.Active.HasValue && r.Active.Value).SelectMany(r => Utilities.DateRangeToArray(r.StartDate, r.EndDate)).ToArray();
Utilities.DateRangeToArray() is defined as follows:
public static IEnumerable<DateTime> DateRangeToArray(DateTime start, DateTime end) {
DateTime curDate = start;
while (curDate <= end) {
yield return curDate;
curDate.AddDays(1);
}
}
Is there a way to make this less memory intensive?
Thanks!
Your code is broken - AddDays doesn't change the existing value, it returns a new value. You're ignoring that new value, thus creating an infinite loop.
Change your code to:
public static IEnumerable<DateTime> DateRangeToArray(DateTime start,
DateTime end) {
DateTime curDate = start;
while (curDate <= end) {
yield return curDate;
curDate = curDate.AddDays(1);
}
}
Another hint: unit testing can help you find this sort of problem long before you try to use the method in a LINQ query. I'd also change the name, given that it's not returning an array.
You're sure you don't have any reservations where r.StartDate > r.EndDate, right? If you do, you'll get an infinite loop, I think.
I assume the out of memory is when converting the result to the array. Two points:
The output will contain duplicate dates for overlapping reservations.
Perhaps Reserved should be a collection of date ranges (start,end) rather than containing every date?
Hello everyone I'm currently having 2 issues with the code below:
Upon return of result1 I'm trying to complete a check to see if it is != null and if it is't it will begin to delete the records selected. The issue is that even when result1 returns nothing and defaults the if statement doesn't pick this up so I guess I'm missing something but what?
I'm wishing to return only values which are over 10 mintues old (this will later be scaled to 12 hours) to do this I'm checking against a.DateTime which is a DateTime value stored in a database. However if i use the <= or >= operators it doesn't work so again what am I missing?
DateTime dateTime = DateTime.Now.Subtract(new TimeSpan(0, 0, 10, 0));
var result1 = (from a in cpuInfo
where a.DateTime <= dateTime
select a).DefaultIfEmpty(null);
if (result1 != null)
{
foreach (TblCPUInfo record1 in result1)
{
localDB.TblCPUInfo.DeleteOnSubmit(record1);
localDB.SubmitChanges();
}
}
Philippe has talked about the sequence side of things - although you don't even need the call to Any(). After all, if there are no changes the loop just won't do anything.
Do you really want to submit the changes on each iteration? It would probably make more sense to do this once at the end. Additionally, you can use DateTime.AddMinutes to make the initial "10 minutes ago" simpler, and if you're only filtering by a Where clause I'd use dot notation.
After all these changes (and making the variable names more useful), the code would look like this:
DateTime tenMinutesAgo = DateTime.Now.AddMinutes(-10);
var entriesToDelete = cpuInfo.Where(entry => entry.DateTime <= tenMinutesAgo);
foreach (var entry in entriesToDelete)
{
localDB.TblCPUInfo.DeleteOnSubmit(entry);
}
localDB.SubmitChanges();
Now, as for why <= isn't working for you... is it possible that you need the UTC time instead of the local time? For example:
DateTime tenMinutesAgo = DateTime.UtcNow.AddMinutes(-10);
If that still isn't working, I suggest you have a look at the generated query and play with it in a SQL tool (e.g. Enterprise Manager or SQL Server Management Studio) to work out why it's not returning any results.
DefaultIfEmpty will return a single item with the content you provided, so in your case a collection with a single value "null".
You should check for elements in the collection using the Any() extension method. In your case:
DateTime dateTime = DateTime.Now.Subtract(new TimeSpan(0, 0, 10, 0));
var result1 = from a in cpuInfo
where a.DateTime <= dateTime
select a;
if (result1.Any())
{
foreach (TblCPUInfo record1 in result1)
{
localDB.TblCPUInfo.DeleteOnSubmit(record1);
localDB.SubmitChanges();
}
}
But if this is really your code, you can skip the Any() check completely, because the foreach loop will not run if there are no elements in result1.