My table is set up like this
PK | SK | CreatedAt (DateTime)(Secondary Global Index) | Random Attributes
USER | 0xFF | Today | 1
USER | 0xFE | Yesterday | 2
So my partition key is a general group name for the objects. "USER" just tells me it's a user object. Later I will have "ITEM" or other random objects, or links between objects for quick queries. The sort key is the actual ID of the user which is randomly generated (uuid). The secondary global index is the "CreatedAt" which is a DateTime object.
I just want to find the latest user (I only really care about the SK).
I tried this;
QueryRequest qr = new QueryRequest
{
TableName = "mytable",
IndexName = "CreatedAt-index",
ScanIndexForward = true,
KeyConditionExpression = "PK = :v_pk",
FilterExpression = "CreatedAt BETWEEN :v_timeOldest AND :v_timeNow",
ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
{
{":v_pk", new AttributeValue{S = "USER" } },
{":v_timeNow", new AttributeValue{S = DateTime.Now.ToString(AWSSDKUtils.ISO8601DateFormat)} },
{":v_timeOldest", new AttributeValue{S = DateTime.Now.AddYears(-1).ToString(AWSSDKUtils.ISO8601DateFormat)} }
},
Limit = 1
};
QueryResponse res = await _client.QueryAsync(qr);
The error is;
Query key condition not supported
I assume it's because it expects CreatedAt to be the partition key, and it needs to be unique so it can't have a BETWEEN comparison. But that isn't useful for this case due to duplicate dates for many objects and the fact that it can be between other dates.
I tried it the other way where the KeyCondition is the CreatedAt statement and the FilterExpression is the PK but same error.
What I expect is to just gather all objects with the partition key "USER" and then with those sort ascending/descending based on the GSI partition key and pick the one at the top of the list.
What am I missing here? Is my understanding of the GSI conceptually flawed?
From the docs for KeyConditionExpression
You must specify the partition key name and value as an equality condition.
You can optionally provide a second condition for the sort key (if present).
In this specific query, you are trying to query the CreatedAt-index index and specifying PK in the KeyConditionExpression. However, your index uses CreatedAt as the primary key, not the attribute named PK. You said
I tried it the other way where the KeyCondition is the CreatedAt statement and the FilterExpression is the PK but same error.
You didn't show the query, but it sounds like you did this
...
KeyConditionExpression = "CreatedAt BETWEEN :v_timeOldest AND :v_timeNow",
FilterExpression = "PK = :v_pk",
...
which will not work because (from the docs) you must specify the partition key name and value as an equality condition. In other words, you can't do a range query on a partition key.
You don't show how you've defined the secondary index, but it sounds like you've defined a partition key using createdAt with no sort key on the index. Logically, that would look like this:
This will not help you very much, since you cannot perform a range query on a partition key.
However, if you were to define a secondary index with a partition key of USER and a sort key of createdAt. That index would look like this
This index would allow you to perform a query operation as you describe.
As an aside, the base table you've described does not sound very useful. You are storing all users in a single partition with a UUID as the sort key. The sort order of UUIDs doesn't sound particularly useful to your application. You may want to consider KSUIDs for your user id's. KSUIDs have the useful feature of being unique (like a UUID) and sortable by creation time. This is particularly useful when you need a unique identifier and need to order them by date created.
I am new to entity framework and I have been searching a while for an answer to this question and I can't find anything that directally addresses this.
Here is the problem. I have a table in Oracle. In this table there are 2 fields(there are more but not important to this question). Card_Transaction_Id and Parent_Card_Transaction_ID. The Parent_Card_Transaction_Id field is constrained by the Card_Transaction_Id field and I am using a Oracle sequence via a trigger to populate the Card_Transaction_Id field.
In my code, I am using Entity Framework(Version 5) to connect using the Code First Approach.
The issue is when I try to create a new record. I need to know what the next sequence value is in order to populate the Parent_Card_Transaction_Id. My mapping for card transactions:
public class CardTransactionMap : EntityTypeConfiguration<CardTransaction>
{
public CardTransactionMap(string schema)
{
ToTable("CARD_TRANSACTION", schema);
// Mappings & Properties
// Primary Key
HasKey(t => t.CardTransactionId);
Property(t => t.CardTransactionId)
.HasColumnName("CARD_TRANSACTION_ID")
.HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity);
Property(t => t.ParentCardTransactionId)
.HasColumnName("PARENT_CARD_TRANSACTION_ID");
Property(t => t.CardProfileId)
.HasColumnName("CARD_PROFILE_ID");
}
}
The question is - is there any way to get the next sequence number before I save the record?
My current work arround is to use the following method:
public static decimal GetNextCardTransactionSequenceValue()
{
using (var context = new Context(new OracleConnectionFactory().GetConnection()))
{
var sequence = context.Database.SqlQuery<int>("select card_transaction_id from card_transaction").ToList();
return sequence[0];
}
}
Using that method, I get the next value and then just populate my table. This works but I don't like doing it this way. I feel that there must be a better way to do it.
Thanks in advance.
You have to do this by navigation properties.
By fetching the next value from a sequence before actually using it in the same session you create yourself a concurrency issue: another user can increment the index (by an insert) in the time between drawing its next value and assigning it to the child record. Now the child will belong to the other user's record!
If your CardTransaction class has a parent reference like this:
int ParentCardTransaction { get; set; }
[ForeignKey("ParentCardTransaction")]
CardTransaction ParentCardTransaction { get; set; }
you can create a parent and child in one go and call SaveChanges without worrying about setting FK values yourself:
var parent = new CardTransaction { ... };
var child = new CardTransaction { ParentCardTransaction = parent, ... };
SaveChanges();
Now EF wil fetch the new CardTransactionId from the parent and assign it to the FK of the child. So generating and getting the parent Id happens all in one session, so it is guaranteed to be the same value.
Apart from preventing concurrency issues, of course it is much easier anyway to let EF do the heavy lifting of getting and assiging key values.
Create a Stored Procedure or Query that will return you the next Value from the Table here is an Example
SELECT NVL(MAX(card_transaction_id + 1), 0) AS MAX_VAL
FROM card_transaction T;
or Create a Trigger - for OracleDB
Change your table definition to this :
CREATE TABLE t1 (c1 NUMBER GENERATED BY DEFAULT ON NULL AS IDENTITY,
c2 VARCHAR2(10));
as per the information in the link i provided in the comment.
after the update ef will automatically query the value for the id that is inserted, there is no need to fill in the id before the insert. ef will generate an insert sql query without id.
I know I'm close with this one...
My structure is:
I have some companies
Each company has a primary user (account)
And a list of all users (accounts).
I've implemented this in the db as a companies table that has "Primary" as a foreign key to the Accounts table, and the Accounts table has a CompanyId which is a foreign key to the Companies table.
So there's only ever one primary user, and each user is associated with one company.
I want to retrieve a list of all companies, and plonk them in a c# object. What I have so far is:
public IEnumerable<Company> GetAllCompaniesList()
{
var allData = database.Companies.All()
.Join(database.Accounts).On(database.Accounts.Id == database.Companies.Primary)
.Join(database.Accounts).On(database.Accounts.CompanyId == database.Companies.Id)
.ToList<Company>();
return allData;
}
and it works in my tests using an in-memory adapter with the relevant keys set up, but not in the real version, crashing out with
The objects "dbo.Accounts" and "dbo.Accounts" in the FROM clause have
the same exposed names. Use correlation names to distinguish them.
I think this means I need to name each join (e.g. to make sql like "join a on a = b as c"), but I can't find the syntax to do that. Anyone know?
You can alias table names using the .As method. In addition to that, the Join method has an optional second out parameter which puts the reference to the aliased table in a dynamic variable; it just makes referring to the table easier.
So try this:
public IEnumerable<Company> GetAllCompaniesList()
{
dynamic primary;
var allData = database.Companies.All()
.Join(database.Accounts.As("PrimaryAccounts"), out primary)
.On(primary.Id == database.Companies.Primary)
.Join(database.Accounts)
.On(database.Accounts.CompanyId == database.Companies.Id)
.ToList<Company>();
return allData;
}
That should work fine.
I'm using entity framework to connect with the database. I've one little problem:
I've one table which have one varbinary(MAX) column(with filestream).
I'm using SQL request to manage the "Data" part, but EF for the rest(metadata of the file).
I've one code which has to get all files id, filename, guid, modification date, ... of a file. This doesn't need at all the "Data" field.
Is there a way to retrieve a List but without this column filled?
Something like
context.Files.Where(f=>f.xyz).Exclude(f=>f.Data).ToList();
??
I know I can create anonymous objects, but I need to transmit the result to a method, so no anonymous methods. And I don't want to put this in a list of anonymous type, and then create a list of my non-anonymous type(File).
The goal is to avoid this:
using(RsSolutionsEntities context = new RsSolutionsEntities())
{
var file = context.Files
.Where(f => f.Id == idFile)
.Select(f => new {
f.Id, f.MimeType, f.Size, f.FileName, f.DataType,
f.DateModification, f.FileId
}).FirstOrDefault();
return new File() {
DataType = file.DataType, DateModification = file.DateModification,
FileId = file.FileId, FileName = file.FileName, Id = file.Id,
MimeType = file.MimeType, Size = file.Size
};
}
(I'm using here the anonymous type because otherwise you will get a NotSupportedException: The entity or complex type 'ProjectName.File' cannot be constructed in a LINQ to Entities query.)
(e.g. this code throw the previous exception:
File file2 = context.Files.Where(f => f.Id == idFile)
.Select(f => new File() {Id = f.Id, DataType = f.DataType}).FirstOrDefault();
and "File" is the type I get with a context.Files.ToList(). This is the good class:
using File = MyProjectNamespace.Common.Data.DataModel.File;
File is a known class of my EF datacontext:
public ObjectSet<File> Files
{
get { return _files ?? (_files = CreateObjectSet<File>("Files")); }
}
private ObjectSet<File> _files;
Is there a way to retrieve a List but without this column filled?
Not without projection which you want to avoid. If the column is mapped it is natural part of your entity. Entity without this column is not complete - it is different data set = projection.
I'm using here the anonymous type because otherwise you will get a
NotSupportedException: The entity or complex type 'ProjectName.File'
cannot be constructed in a LINQ to Entities query.
As exception says you cannot project to mapped entity. I mentioned reason above - projection make different data set and EF don't like "partial entities".
Error 16 Error 3023: Problem in mapping fragments starting at line
2717:Column Files.Data in table Files must be mapped: It has no
default value and is not nullable.
It is not enough to delete property from designer. You must open EDMX as XML and delete column from SSDL as well which will make your model very fragile (each update from database will put your column back). If you don't want to map the column you should use database view without the column and map the view instead of the table but you will not be able to insert data.
As a workaround to all your problems use table splitting and separate the problematic binary column to another entity with 1 : 1 relation to your main File entity.
I'd do something like this:
var result = from thing in dbContext.Things
select new Thing {
PropertyA = thing.PropertyA,
Another = thing.Another
// and so on, skipping the VarBinary(MAX) property
};
Where Thing is your entity that EF knows how to materialize. The resulting SQL statement shouldn't include the large column in its result set, since it's not needed in the query.
EDIT: From your edits, you get the error NotSupportedException: The entity or complex type 'ProjectName.File' cannot be constructed in a LINQ to Entities query. because you haven't mapped that class as an entity. You can't include objects in LINQ to Entities queries that EF doesn't know about and expect it to generate appropriate SQL statements.
You can map another type that excludes the VarBinary(MAX) column in its definition or use the code above.
you can do this:
var files = dbContext.Database.SqlQuery<File>("select FileId, DataType, MimeType from Files");
or this:
var files = objectContext.ExecuteStoreQuery<File>("select FileId, DataType, MimeType from Files");
depending on your version of EF
I had this requirement because I have a Document entity which has a Content field with the content of the file, i.e. some 100MB in size, and I have a Search function that I wanted to return the rest of the columns.
I chose to use projection:
IQueryable<Document> results = dbContext.Documents.Include(o => o.UploadedBy).Select(o => new {
Content = (string)null,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
FileName = o.FileName,
Id = o.Id,
// etc. even with related entities here like:
UploadedBy = o.UploadedBy
});
Then my WebApi controller passes this results object to a common Pagination function, which applies a .Skip, .Take and a .ToList.
This means that when the query gets executed, it doesn't access the Content column, so the 100MB data is not being touched, and the query is as fast as you'd want/expect it to be.
Next, I cast it back to my DTO class, which in this case is pretty much exactly the same as the entity class, so this might not be a step you need to implement, but it's follows my typical WebApi coding pattern, so:
var dtos = paginated.Select(o => new DocumentDTO
{
Content = o.Content,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
FileName = o.FileName,
Id = o.Id,
UploadedBy = o.UploadedBy == null ? null : ModelFactory.Create(o.UploadedBy)
});
Then I return the DTO list:
return Ok(dtos);
So it uses projection, which might not fit the original poster's requirements, but if you're using DTO classes, you're converting anyway. You could just as easily do the following to return them as your actual entities:
var dtos = paginated.Select(o => new Document
{
Content = o.Content,
ContentType = o.ContentType,
DocumentTypeId = o.DocumentTypeId,
//...
Just a few extra steps but this is working nicely for me.
For EF Core 2
I implemented a solution like this:
var files = context.Files.AsNoTracking()
.IgnoreProperty(f => f.Report)
.ToList();
The base idea is to turn for example this query:
SELECT [f].[Id], [f].[Report], [f].[CreationDate]
FROM [File] AS [f]
into this:
SELECT [f].[Id], '' as [Report], [f].[CreationDate]
FROM [File] AS [f]
you can see the full source code in here:
https://github.com/aspnet/EntityFrameworkCore/issues/1387#issuecomment-495630292
I'd like to share my attempts to workaround this problem in case somebody else is in the same situation.
I started with what Jeremy Danyow suggested, which to me is the less painful option.
// You need to include all fields in the query, just make null the ones you don't want.
var results = context.Database.SqlQuery<myEntity>("SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName");
In my case, I needed a IQueryable<> result object so I added AsQueryable() at the end. This of course let me add calls to .Where, .Take, and the other commands we all know, and they worked fine. But there's a caveat:
The normal code (basically context.myEntity.AsQueryable()) returned a System.Data.Entity.DbSet<Data.DataModel.myEntity>, while this approach returned System.Linq.EnumerableQuery<Data.DataModel.myEntity>.
Apparently this means that my custom query gets executed "as is" as soon as needed and the filtering I added later is done afterwards and not in the database.
Therefore I tried to mimic Entity Framework's object by using the exact query EF creates, even with those [Extent1] aliases, but it didn't work. When analyzing the resulting object, its query ended like
FROM [dbo].[TableName] AS [Extent1].Where(c => ...
instead of the expected
FROM [dbo].[TableName] AS [Extent1] WHERE ([Extent1]...
Anyway, this works, and as long as the table is not huge, this method will be fast enough. Otherwise you have no option than to manually add the conditions by concatenating strings, like classic dynamic SQL. A very basic example in case you don't know what I'm talking about:
string query = "SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName";
if (parameterId.HasValue)
query += " WHERE Field1 = " + parameterId.Value.ToString();
var results = context.Database.SqlQuery<myEntity>(query);
In case your method sometimes needs this field, you can add a bool parameter and then do something like this:
IQueryable<myEntity> results;
if (excludeBigData)
results = context.Database.SqlQuery<myEntity>("SELECT Field1, Field2, Field3, HugeField4 = NULL, Field5 FROM TableName").AsQueryable();
else
results = context.myEntity.AsQueryable();
If anyone manages to make the Linq extensions work properly like if it was the original EF object, please comment so I can update the answer.
I'm using here the anonymous type because otherwise you will get a
NotSupportedException: The entity or complex type 'ProjectName.File'
cannot be constructed in a LINQ to Entities query.
var file = context.Files
.Where(f => f.Id == idFile)
.FirstOrDefault() // You need to exeucte the query if you want to reuse the type
.Select(f => new {
f.Id, f.MimeType, f.Size, f.FileName, f.DataType,
f.DateModification, f.FileId
}).FirstOrDefault();
And also its not a bad practice to de-normalize the table into further, i.e one with metadata and one with payload to avoid projection. Projection would work, the only issue is, need to edit any time a new column is added to the table.
I tried this:
From the edmx diagram (EF 6), I clicked the column I wanted to hide from EF and on their properties you can set their getter and setter to private. That way, for me it works.
I return some data which includes a User reference, so I wanted to hide the Password field even though it's encrypted and salted, I just didn't want it on my json, and I didn't want to do a:
Select(col => new {})
because that's a pain to create and maintain, especially for big tables with a lot of relationships.
The downside with this method is that if you ever regenerate your model, you would need to modify their getter and setter again.
Using Entity Framework Power Tools you can do the following in efpt.config.json:
"Tables": [
{
"ExcludedColumns": [
"FileData"
],
"Name": "[dbo].[Attachment]",
"ObjectType": 0
}
]
I want to update my database using a LINQ2SQL query.
However this seems for some reason to be a very ugly task compared to the otherwise lovely LINQ code.
The query needs to update two tables.
tbl_subscription
(
id int,
sub_name nvarchar(100),
sub_desc nvarchar(500),
and so on.
)
tbl_subscription2tags
(
sub_id (FK to tbl_subscription)
tag_id (FK to a table called tbl_subscription_tags)
)
Now down to my update function a send a tbl_subscription entity with the tags and everything.
I can't find a pretty way to update my database..
I can only find ugly examples where I suddenly have to map all attributes..
There most be a smart way to perform this. Please help.
C# Example if possible.
I have tried this with no effect:
public void UpdateSubscription(tbl_subscription subscription)
{
db.tbl_subscriptions.Attach(subscription);
db.Refresh(System.Data.Linq.RefreshMode.OverwriteCurrentValues, subscription);
db.SubmitChanges(System.Data.Linq.ConflictMode.FailOnFirstConflict);
}
Source for this code is here:
http://skyeyefive.spaces.live.com/blog/cns!6B6EB6E6694659F2!516.entry
Why don't just make the changes to the objects and perform a SubmitChanges to the DataContext?
using(MyDataContext dc = new MyDataContext("ConnectionString"))
{
foreach(var foo in dc.foo2)
{
foo.prop1 = 1;
}
dc.SubmitChanges();
}
Otherwise you need to tell us more about the lifecycle of the object you want to manipulate
edit: forgot to wrap in brackets for using
Unless I'm misunderstanding your situation, I think that citronas is right.
The best and easiest way that I've found to update database items through LINQ to SQL is the following:
Obtain the item you want to change from the data context
Change whatever values you want to update
Call the SubmitChanges() method of the data context.
Sample Code
The sample code below assumes that I have a data context named DBDataContext that connects to a database that has a Products table with ID and Price parameters. Also, a productID variable contains the ID of the record you want to update.
using (var db = new DBDataContext())
{
// Step 1 - get the item from the data context
var product = db.Products.Where(p => p.ID == productID).SingleOrDefault();
if (product == null) //Error checking
{
throw new ArgumentException();
}
// Step 2 - change whatever values you want to update
product.Price = 100;
// Step 3 - submit the changes
db.SubmitChanges();
}
I found out that you can use "Attach" as seen in my question to update a table, but apparently not the sub tables. So I just used a few Attach and it worked without having to run through parameters!