I have the table in PostgreSQL with many records and unique key
CREATE TABLE parcels
(
Id SERIAL PRIMARY KEY NOT NULL,
Number CITEXT NOT NULL,
UserId INT REFERENCES Stations(Id) NOT NULL,
TimeStampUtc TIMESTAMP WITHOUT TIME ZONE NOT NULL
);
CREATE UNIQUE INDEX ON parcels (Number, UserId, (TimeStampUtc::date));
then I import data from Excel and map it to list. At the end I have something like
await _dbContext.Parcels.AddRangeAsync(parcels);
await _dbContext.SaveChangesAsync();
parcels contains about 20,000 records and can contains records witch violates the unique constraint. In such case I need to skip that records and continue to insert.
Now I got an expected error
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred
while updating the entries. See the inner exception for details. --->
Npgsql.PostgresException: 23505: duplicate key value violates unique
constraint "parcels_number_userid_timestamputc_idx"
How to ignore it and continue to insert?
I found some similar questions like this and this but I don't want to load table to memory, because there are a lot of records to compare
You should be able to do is the following:
create a stored procedure that receives the parcels and does an insert paired with a join on the column with the unique key constraint (= filter out all that already have a value in the where condition)
call the stored procedure from your code and pass the parcels
in case you need an artificial id, there are two ways to do it:
return the ID from the insert function (returning id) that the database created automatically for you
generate the Id on your client (usually only works well when you have GUID columns)
Related
I use the MS Sync Framework to sync my SQL Server instance with a local SQL CE file to make it possible working offline with my Windows app.
I use GUIDs as keys. On my table I have a unique index on 2 columns: user_id and setting_id:
usersettings table
------------------
id PK -> I also tried it without this column. Same result
user_id FK
setting_id FK
value
Now I do the following:
I create a new record in this table in both databases - SQL Server and SQL CE with the same user_id and setting_id.
This should work and merge the data together since this can happen in real life. But I get an error when syncing saying the unique key constraint led to an error. The key pair already exists in the table.
A duplicate value cannot be inserted into a unique index. [ Table name = user_settings,Constraint name = unique_userid_settingid ]
Why can't MS sync handle that? It should not try to insert the key pair again. It should update the value if needed.
The issue is if you add the same key pair to different copies of the table, they get different IDs (GUIDs) as primary keys in this usersettings table.
As this is simply a many-to-many table between Users and Settings, there is no need to have that ID as a PK (or even a column at all).
Instead, just use a concatenated key of the two FKs e.g.,
CREATE TABLE [dbo].[usersettings](
[user_id] [UNIQUEIDENTIFIER] NOT NULL,
[setting_id] [UNIQUEIDENTIFIER] NOT NULL,
[value] [varchar](50) NOT NULL,
CONSTRAINT [PK_usersettings] PRIMARY KEY CLUSTERED ([user_id] ASC, [setting_id] ASC) );
Of course, include appropriate field settings (e.g., if you use VARCHARs to store the IDs) and relevant FKs.
As the rows inserted should now be identical on the two copies, it should merge fine.
If you must have a single column as a unique identifier for the table, you could make it meaningful e.g.,
the PK (ID) becomes a varchar (72)
it gets filled with CONCAT(user_ID, setting_id)
As the User_ID and Setting_ID are FKs, you should already have them generated so concatenating them should be easy enough.
Do you get the error during sync, then it should appear as a conflict, that you must solve in code.
https://learn.microsoft.com/en-us/previous-versions/sql/synchronization/sync-framework-2.0/bb734542(v=sql.105)
I also see this in the manual: By default, the following objects are not copied to the client database: FOREIGN KEY constraints, UNIQUE constraints, DEFAULT constraints, and the SQL Server ROWGUIDCOL property. This indicates poor support for your scenario
I suggest you remove the unique constraint from the device table.
This question already has answers here:
Duplicate entry for key 'PRIMARY' .Ignoring the spaces for strings
(2 answers)
Closed 5 years ago.
I have the following structure, mapped with Entity Framework 6 using the Database First principle:
Here is the source database:
CREATE TABLE `Foo` (
`Guid` VARCHAR(36),
`Name` VARCHAR(500) NOT NULL,
`Author` VARCHAR(100) NOT NULL,
PRIMARY KEY (`Guid`),
UNIQUE KEY `unique_fooname` (`Name`,`Author`));
CREATE TABLE `FooVersion` (
`Guid` VARCHAR(36),
`Version` INT,
`RefFooGuid` VARCHAR(36) NOT NULL,
PRIMARY KEY (`Guid`),
UNIQUE KEY `unique_fooversion` (`Version`,`RefFooGuid`),
CONSTRAINT `fk_foo_version`
FOREIGN KEY (`RefFooGuid`)
REFERENCES `Foo` (`Guid`)
ON DELETE NO ACTION
ON UPDATE NO ACTION);
CREATE TABLE `FooVersionPart` (
`Name` VARCHAR(250) NOT NULL,
`RefFooVersionGuid` VARCHAR(36) NOT NULL,
PRIMARY KEY (`Name`, `RefFooVersionGuid`),
INDEX `fk_fooversion_fooversionpart_idx` (`RefFooVersionGuid` ASC),
CONSTRAINT `fk_fooversion_fooversionpart`
FOREIGN KEY (`RefFooVersionGuid`)
REFERENCES `FooVersion` (`Guid`)
ON DELETE NO ACTION
ON UPDATE NO ACTION);
At one point in my code, I am creating a new foo like this:
var dbContext = new DbContext();
var newVersion = new FooVersion();
newVersion.Guid = Guid.NewGuid().ToString()
newVersion.Parts = sourceParts.Select(s => new FooVersionPart
{
Name = s.Name,
RefFooVersionGuid = newVersion.Guid
}.ToList();
var foo = new Foo
{
Author = "Me"
Guid = Guid.NewGuid().ToString(),
Name = "Foo"
};
dbContext.Foos.Add(foo);
foo.Versions.Add(newVersion);
dbContext.SaveChanges();
I am getting the following error during the SaveChanges:
Duplicate entry 'dim.proran.db.tmp.dataCallistHDay -9e6620f4-227d-44de-b781-5fd67' for key 'PRIMARY'
The errors occurs more specifically when EF is trying to insert one of the FooVersionPart (dim.proran.db.tmp.dataCallistHDay is the Name of that part and 9e6620f4-227d-44de-b781-5fd67 is the -truncated- RefFooVersionGuid of that part).
I have the absolute certainty sourceParts have no duplicate and neither in the database.
Here is the generated SQL:
INSERT INTO Foo [..];
INSERT INTO FooVersion [..];
INSERT INTO FooVersionPart [..];
INSERT INTO FooVersionPart [..];
INSERT INTO FooVersionPart [..];
INSERT INTO FooVersionPart [..];
Etc
The exception always occurs on the same FooVersionPart (dim.proran.db.tmp.dataCallistHDay). It is the 1910th elements of 2435. So EF is not trying to insert twice all parts, just one at the middle.
The weirdest thing is that it worked a while ago, and it does not work anymore with no changes in all the related stuff (no changes in the schema, no library update, no changes in the code). It works well in one of my environment, and it does not work with the same code in my dev environment.
One last thing, it is not specific to that Guid. At each attempt, the Guid is different (not the other inputs, so it still fails on dim.proran.db.tmp.dataCallistHDay), and at each attempt I get the same error.
Do you have any idea of what could cause that?
The exception message (Duplicate entry 'dim.proran.db.tmp.dataCallistHDay -9e6620f4-227d-44de-b781-5fd67' for key 'PRIMARY') combined with the primary key for the table it refers to (PRIMARY KEY ('Name', 'RefFooVersionGuid')) tells us that you are attempting to insert duplicate data into the table, specifically multiple FooVersionParts with the same name.
Now you say you have done a duplicate check on your source data, but what you may not know is that many (all?) SQL database don't count trailing spaces as part of the record. For example this query will actually return a record:
SELECT 1
WHERE 'abc' = 'abc '
So, as you confirmed, your data does have duplicate(s) that won't be spotted by a C# GroupBy but will be caught by the database engine. An easy solution is to trim the data before you group it, a good habit to get into, particularly with data inputted manually.
what is the best way to prevent duplicate records in a SQL Server database? Using triggers? Using a unique constraint?
Use unique constraints on one or more columns in the table.
Example:
CREATE TABLE Persons
(
P_Id int NOT NULL UNIQUE,
LastName varchar(255) NOT NULL UNIQUE,
FirstName varchar(255) NOT NULL UNIQUE,
Address varchar(255),
City varchar(255)
)
Alter existing table as below
ALTER TABLE Persons
ADD CONSTRAINT uc_PersonID UNIQUE (P_Id,LastName)
If you are using a front-end application to populate the table in the database. Do your validation select query from the application on the database to check for duplicates before inserting into the database. Using constraints will prevent duplicates by throwing an exception.
Note: The above example is SQL SERVER, Oracle, ms access
For much more indepth solution see How to prevent duplicate records being inserted with SqlBulkCopy when there is no primary key
If you don't want error throw from unique constraint and you also want database to receive duplicated data but insert nothing. You may look at merge statement
https://technet.microsoft.com/en-us/library/bb522522%28v=sql.105%29.aspx?f=255&MSPPError=-2147217396
I have this DbContext object which consists of -
- Employee
- CompanyAddress (PK: AddressFirstLine, City)
Note: one Employee can have many CompanyAddress
Records are added to CompanyAddress table only if some address doesn't exists in CompanyAddress table.
If I have two DBContext objects from database say Snapshot1, Snapshot2.
Say when both these snapshots were taken, there were no records in CompanyAddress table.
When changes were made to Snapshot1 and saved - records are written to CompanyAddress table.
When changes were made to Snapshot2 and saved using
mydataContext.SaveChanges();
exception occurs:
System.Data.Entity.Infrastructure.DbUpdateException: An error occurred while updating the entries
System.Data.SqlClient.SqlException: Violation of PRIMARY KEY constraint 'PK_CompanyAddress'. Cannot insert duplicate key in object 'dbo.CompanyAddress'
It seems saving of Snapshot1 made Snapshot2 dirty because when they are saved back to database, both had same CompanyAddress records.
What other call/settings I can make on dbContext object to avoid this error?
Thank you!
Your error has got nothing to do with the DbContext objects. Your problem is that you are trying to insert a record with duplicating primary key. That is what your exception message says.
Look at how you create your CompanyAddress objects and what are the keys when you save them - this will give you the clues.
Edit: And it is a bad idea to have primary key to be a natural key, i.e. you should not assign city and address as primary keys. You should have either Guid or Integer to be primary key that is not dependent on the information stored in DB.
And to enforce uniqueness, before you save to DB, you check if that record exists, and can add a unique index to database table based on the unique constraints.
I am building an API with OrmLite from ServiceStack.
When populating it with test data I get the following error:
The INSERT statement conflicted with the FOREIGN KEY constraint
"FK_Order_Customer_CustomerId". The conflict occurred in database
"C:\USERS\ALECTAYLOR\SOCIALBOOTSTRAPAPI\SRC\SOCIALBOOTSTRAPAPI\APP_DATA\USERAUTH.MDF",
table "dbo.Customer", column 'Id'. The statement has been terminated.
Code (lines 213-236 + CreateOrders function): http://pastebin.com/Njhz7sD2
Profiler output: http://fiddle.jshell.net/cTen2/1/show/
Thanks for any advice on how to rectify this issue.
FOREIGN KEY constraint generally means that you are trying to insert a value into a table that doesn't exist in the reference table. Take a look at MSDN article on Foreign Keys for more info about what they are and how they work. You need to have a look at the actual structure of the data tables order and customer.
I would guess that you are inserting a customerId into the orders table that doesn't exist in the customers table.
since this is the insert that's failing, the only logical explanation is that customer number 1 doesn't exist. I saw that you insert 3 customers a few lines before. Maybe the transaction was not committed between the moment the customers were inserted and the order is inserted.
INSERT INTO "Order" ("CustomerId", "ShopId", "ShippingAddress",
"OrderDate", "RequiredDate", "ShippedDate", "Total") VALUES (1, 0,
'{line1:440 Crescent St, line2:South Melbourne, postCode:7416,
city:Melbourne, country:Australia}', '20120430 07:43:18.686', NULL,
NULL, 0);
Try to commit the insert after you insert the clients and before you insert the orders
Alright, got it to work.
Needed to set the ShopId of Order and orderId of the orderDetails List.
http://pastebin.com/TbrW150T