The problem I am trying to solve is pretty simple: I need to make sure my timestamps are stored in the database as UTC. I use DateTime in my model objects, which maps to timespamp column in Postgres DB.
I'd like to do 2 things:
When a value is saved to the database, I want to check the DateTime.Kind field and throw up if it is not UTC. This will help me to capture errors if I try to feed a non-UTC time to a timestamp field. I do not want to do any transformations, just assert.
When loading from DB, I need those DateTime instances come out with DateTime.Kind set to UTC, so the rest of my application can recognize they are UTC and convert to local or whatever, as needed.
I only need to apply this logic to selected timestamp fields, not to all DateTime-s, because I have other DateTime values in my model, which need to be stored as they are.
I've tried this:
public class UtcTimestampMapper: SqlMapper.TypeHandler<DateTime>
{
public override void SetValue(IDbDataParameter parameter, DateTime dt)
{
if (dt.Kind == DateTimeKind.Utc)
parameter.Value = dt;
else
throw new InvalidOperationException("UTC timestamp expected");
}
public override DateTime Parse(object value)
{
return DateTime.SpecifyKind((DateTime)value, DateTimeKind.Utc);
}
}
This solution has 2 issues:
It applies to all DateTime values. With SetValue I can do the trick checking the column name in IDbDataParameter, maybe... But Parse method just gets a naked value, there is nothing I can do to determine where it goes.
The SetValue method won't be called at all, unless I remove the standard handler for DateTime. See this SO answer for details: Dapper TypeHandler.SetValue() not being called Again, if I replace the handler, I replace it globally, there is no way to choose where it applies.
So, is there a way to solve my problem with the facilities Dapper provides to customize mappings?
When DataTime field is exposed via OData in .NET 5 Web API, it has both Date and Time components. We are mainly using it for exposing Dates so there is now a requirement to format the cumbersome yyyy-MM-ddThh:mm:ss+timezone to just the date part yyyy-MM-dd. For this I've found, that OData support converting DateTime to Edm.Date with this exact reason behind.
Following this article (https://learn.microsoft.com/en-us/odata/webapi/date-timeofday-with-entity-framework) I can do the conversion in two ways:
decorate the DateTime property in the result entity DTO with Column attribute:
[Column(TypeName = "date")]
public DateTime StartDate{ get; set; }
configure the date property in the model configuration by adding "AsDate()":
Builder.EntitySet<ResultDto>("ExampleDataSet").EntityType
.HasKey(o => o.Id)
.Property(c => c.StartDate).AsDate();
It works quite well and I get what I need, but the problem starts when the DateTime property is a key or a part of the key. In that case:
the Column attribute is ignored and the date format doesn't change.
If configuration is changed, an exception is thrown at the serialization and in the results instead of getting the nextLink with the skip token, I get an exception with the message "The type 'System.DateTime' is not supported when converting to a URI literal."
I tried to override DefaultSkipTokenHandler class with my own implementation, but the exception happens in the GenerateNextPagLink method which means I have to write my own NextLink Generator. Before I try doing that, I want to check, that I have not messed up somewhere else or am trying to achieve things in a backwards way. If not, is this a bug in the OData library?
Libraries used:
Microsoft.OData.Core v7.9.0
Microsoft.AspNetCore.OData.Versioning.ApiExplorer v5.0.0
I'm using the extension below for MongoDB date fields because of MongoDB stores times in UTC by default.
public static class DateTimeExtensions {
public static DateTime AdjustUtcDiffByOffset(this DateTime time) {
return time.AddHours(DateTimeOffset.Now.Offset.Hours);
}
}
}
All of them cause different problems although I have tried a few ways like attributes or serialization methods in .NET on the application level. I have decided to use this extension in .NET for now but I think that this is not a complete and exact solution too.
Is there any solution for this problem on the database level without being dependent on programming language wtih an adjust or something else?
EDIT
I think that I should an explain more after comments below. I already know MongoDB stores times in UTC that linked in this post as you see above. This can be useful but I don't need any UTC time zone difference in my app and I don't want to deal in Presentation Layer for every programming language separately. And also, I don't even want only one extra row or function in the other layers because of move away than base logic or business.
Let the my architecture undertaking this. I'm pretty lazy, the life is really short and the birds are flying outside :) I don't want the different fields as like as string convertings unnecessarily. I need a datetime type in the db due to I'm doing many time calculation in the app.
I'm using .NET now and therefore MongoDB .NET driver. I have tried different serialization methods but it cause another problems in the Data Access architecture. In conclusion, I can use UTC in my another app but I don't want it now and I prefer the local time when I assign to the field. I have decided to use the encapsulation below for C# especially.
private DateTime _startTime;
public DateTime StartTime {
get => _startTime;
set => _startTime = new DateTime(value.Ticks, DateTimeKind.Utc);
}
I don't think it's possible from the db level. What I did, was to write custom setter for date properties which will force mongoDB to assume the time is already in UTC, thus avoid a conversion like below:
private DateTime _createdUTC;
public DateTime CreatedUtc
{
get
{
return _createdUTC;
}
set
{
_createdUTC = new DateTime(value.Ticks, DateTimeKind.Utc);
}
}
MongoDB always saves dates in UTC.
What you can do is indicate to the client that he must convert the date to the local date when retrieving data from the database.
You can register a serializer before instantiating the client:
BsonSerializer.RegisterSerializer(DateTimeSerializer.LocalInstance);
var client = new MongoClient(options) ...
After lot of debugging I came to know exact cause of the crash. Firstly
Test 1:
I am loading an XML file from drive deserializing it against a MOTORCLASS and using the MOTORCLASS properties, later serializing back again to XML It works fine.
Test 2:
I have a Datatable and all its rows are mapped to the MOTORCLASS properties and now when serializing to XML a crash occurs..
When looking into MOTORCLASS property
`public object APPOINTMENT
{
get
{
return this.aPPOINTMENTField;
}
set
{
this.aPPOINTMENTField = value;
}
}`
On run time the TEST 1 sets APPOINTMENT as Xmlnode whereas
TEST 2 assign APPOINTMENT as Datetime.
I think if I convert Datetime to Xmlnode than it should solve the problem. But not sure how to achieve it. I have tried
[System.Xml.Serialization.XmlElementAttribute("APPOINTMENT")].
But it still Datetime. Can anyone shed some light here.
TEST 1:
TEST 2:
The problem isn't the type of value itself that happens to be assigned to the property; It's the XmlSerializer that needs to know beforehand what type it can expect.
If APPOINTMENT should always be a DateTime, just change the property type to DateTime. Besides fixing the xml serialization problem, this will also prevent bugs and improve your application's performance.
If APPOINTMENT can be different things, you can supply type candidates to the serializer:
[XmlElement("AppointmentAsDateTime", Type = typeof(DateTime))]
[XmlElement("AppointmentAsOtherType", Type = typeof(OtherType))]
public object APPOINTMENT { get; set; }
This allows the serializer to be able to deal with situations where APPOINTMENT is a DateTime or an OtherType.
(Credit to that solution goes to Marc Gravell: How to serialize property of type Object with XmlSerializer)
I'm developing a WCF RESTful service with C#, .NET Framework 4.0 and Entity Framework Code First.
I have this class (that represents a table on database):
[DataContract]
public class PostLine
{
public int PostLineId { get; set; }
[DataMember]
public int? UserId { get; set; }
[DataMember]
public string Description { get; set; }
[DataMember]
public string DateUtc { get; set; }
public User Author { get; set; }
}
And I'm trying to do this:
DateTime fourDaysAgo = DateTime.Now.Date.AddDays(-4);
var postLines =
context.PostLines.Where(p => DateTime.Compare(DateTime.Parse(p.DateUtc), fourDaysAgo) > 0).Include("Author");
But I get the following error:
{System.NotSupportedException: LINQ to Entities doesn't recognize the method 'System.DateTime Parse(System.String)', which can not be converted into an expression of the repository.
I need that PostLine.DateUtc to be a string because I'm going to use it on my web service, and send it as JSON, so its better for me to store it as a string.
If I use DateTime type, I'm going to get something like this on JSON response:
{
"DateUtc": "/Date(1380924000000+0200)/",
"Description": "post_1",
"UserId": 1
}
Do you know how can I compare a string with a DateTime on a LINQ expression?
I think the best approach will be to split the property in two.
Entity Framework wants a DateTime property. That makes perfect sense.
For serialization you want a string property. That also makes perfect sense.
However, you're trying to use a single property for both, and that doesn't make sense, and isn't necessary.
[DataContract]
public class PostLine
{
...
public DateTime DateUtcAsDateTime { get; set; }
[DataMember, NotMapped]
public string DateUtcAsString {
get { return DateUtcAsDateTime.ToString(); }
set { DateUtcAsDateTime = DateTime.Parse(value); }
}
...
}
Now, DateUtcAsDateTime will be used by Entity Framework, and DateUtcAsString will be ignored by Entity Framework as it has a NotMapped attribute.
DateUtcAsString, on the other hand, is the only one of these properties that has a DataMember attribute, so should be the only one that gets serialized.
You can of course rename one of these properties back to DateUtc, if you want.
Update: as Matt Johnson points out, an improvement would be to specify the format in a way that always results in the exact same string. This ensures your strings don't change, just because your code gets moved to another server that happens to have different regional settings.
[DataMember, NotMapped]
public string DateUtcAsString {
get { return DateUtcAsDateTime.ToString("o"); }
set { DateUtcAsDateTime = DateTime.Parse(value, "o", null, DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal); }
}
Note that I am using DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal instead of the DateTimeStyles.RoundTripKind that he suggested, because the name DateUtc strongly suggests you always want UTC, never local time. And I'm not specifying any culture explicitly, as the "o" format is already independent of culture.
You could, if it's easier for your other code to handle, use "r" instead of "o" with the same benefits.
This is certainly an XY problem if I ever saw one. You're asking about comparing datetime as strings in Entity Framework, while it seems the real problem is that you don't like the default date format of the DataContractJsonSerializer that WCF uses by default.
Part of the problem is that you are mixing local and UTC. You're getting /Date(1380924000000+0200)/, which contains the local time zone offset of the server. This is because you started from DateTime.Now, which has a .Kind of DateTimeKind.Local.
If you instead used DateTime.UtcNow, it would have a .Kind of DateTimeKind.Utc, and would be serialized as /Date(1380924000000)/. And yes, the numerical portion of the format would be the same. Even when there is an offset specified, the number part is still related to UTC.
That's just one problem with this format. The other is that while DataContractJsonSerializer writes the local offset during serialization, it doesn't use it properly during deserialization. It just assumes that if any offset is provided, that the time should be local - even if the computer doing the deserialization has a completely different offset.
The better format to use in JSON is the ISO8601 format. For example, this UTC value would look like 2013-10-04T22:00:00.000Z. And while you can easily pass in a different date format to DataContractJsonSerializer if you use it directly, WCF doesn't easily expose this to you.
You could go down the route of changing the DataContractJsonSerializer settings via a custom WCF message formatter, such as described here, here and here, but it gets complicated very quickly. Be careful if you do this, and be sure to test thoroughly!
Another idea would be to write a custom WCF message formatter that uses JSON.Net instead of DataContractJsonSerializer. JSON.Net uses the ISO8601 format by default, so you would be set.
But honestly, the best solution is to not try to use WCF to build your REST endpoints. The date format issue is just the beginning. There are all sorts of other problems that can pop up along the way. Instead, use a modern framework that is designed for this purpose, such as ASP.Net WebAPI, or ServiceStack.
ASP.Net WebAPI uses JSON.Net, and ServiceStack has it's own JSON serializer called ServiceStack.Text. JSON.Net uses ISO8601 as it's default date format. If you use Service Stack, you'll need to set JsConfig.DateHandler = JsonDateHandler.ISO8601;
So in recap, you have two XY problems:
First, you chose WCF to build your rest endpoints, which was so problematic that the industry developed other solutions.
Second, you couldn't get DateTime to emit the correct format, so you decided to treat it as a string, which then you couldn't compare back to a DateTime in an EF query.
And yes, I realize I didn't answer your question of how to compare a DateTime input against a string property in Entity Framework, but I think now you can see why. If you really want to go down that road, you might find something useful in SqlFunctions or EntityFunctions - but even then, how are you going to build an efficient index on this column in your database? Querying will be really slow if you get a lot of data. I would advise against it.
I don't know how you are instantiating your DataContractJsonSerializer. If you are directly instantiating it...you can pass a DataContractJsonSerializerSettings, with a DateTimeFormat for setting how DateTime are serialized.
If you are using a behavior to instantiate your Serializer things are a little more complicated.
If you really want to go down the route of using a string for your class when transfering your data to your client, you should have a separate DTO class.
You can then use a library like AutoMapper to map your PostLine class to PostLineDto with an expression.
However the next problem you will face then is that your Expression<Func<PostLine,PostLineDto>> will contain Expression.MethodCall(DateTime.Parse) and you will have to inject an ExpressionVisitor that can convert...
p => p.DateUtc > DateTime.Parse("2013 Aug 1") - bool
into
p => p.DateUtc > new DateTime(1, Aug, 2013) - bool
Which is a real pain.