Develop Reference

SQL Server allow duplicates in any column, but not all columns

I've searched through numerous threads to try to find an answer to this but any answer I've found suggests using a unique constraint on a single column, or multiple columns.
My problem is, I'm writing an application in C# with a SQL Server back end. One of the features is to allow a user to import a .CSV file into the database after a little bit of pre-processing. I need to find the quickest method to prevent the user from importing the same data more than once. The data will look something like
ID -- will be auto-generated in SQL Server (PK)
Date Time(datetime)
Machine(nchar)
...
...
...
Name(nchar)
Age(int)
I want to allow any number of the columns to be duplicate values, a long as the entire record is not.
I was thinking of creating another column in the database, obtained by hashing all of the columns together and making it unique but want sure if that was the most efficient method, or if the resulting hash would be guaranteed unique. The CSV files will only be around 60 MB, but there will be tens of thousands of them.
Any help would be appreciated.
Thanks

You should be able to resolve this by creating a unique constraint which includes all the columns.
create table #a (col1 varchar(10), col2 varchar(10))
ALTER TABLE #a
ADD CONSTRAINT UQ UNIQUE NONCLUSTERED
(col1, col2)
-- Works, duplicate entries in columns
insert into #a (col1, col2)
values ('a', 'b')
,('a', 'c')
,('b', 'c')
-- Fails, full duplicate record:
insert into #a (col1, col2)
values ('a1', 'b1')
,('a1', 'b1')

The code below can work to ensure that you don't duplicate the [Date Time], Machine, [Name] and Age columns when you insert the data.
It's important to ensure that at the time of running the code, each row of the incoming dataset has a unique ID on it. This code just fails to shift any rows where the ID gets selected because all four other values are already duplicated in the destination table.
INSERT INTO MAIN_TABLE ([Date Time],Machine,[Name],Age)
SELECT [Date Time],Machine,[Name],Age
FROM IMPORT_TABLE WHERE ID NOT IN
(
SELECT I.ID FROM IMPORT_TABLE I INNER JOIN MAIN_TABLE M
ON I.[Date Time]=M.[Date Time]
AND I.Machine=M.Machine
AND I.[Name]=M.[Name]
AND I.Age=M.Age
)

ALTER TABLE DROP COLUMN failed because one or more objects access this column

I am trying to do this:
ALTER TABLE CompanyTransactions DROP COLUMN Created
But I get this:
Msg 5074, Level 16, State 1, Line 2
The object 'DF__CompanyTr__Creat__0CDAE408' is dependent on column 'Created'.
Msg 4922, Level 16, State 9, Line 2
ALTER TABLE DROP COLUMN Created failed because one or more objects access this column.
This is a code first table. Somehow the migrations have become all messed up and I am trying to manually roll back some changed.
I have no idea what this is:
DF__CompanyTr__Creat__0CDAE408

You must remove the constraints from the column before removing the column. The name you are referencing is a default constraint.
e.g.
alter table CompanyTransactions drop constraint [df__CompanyTr__Creat__0cdae408];
alter table CompanyTransactions drop column [Created];

The #SqlZim's answer is correct but just to explain why this possibly have happened. I've had similar issue and this was caused by very innocent thing: adding default value to a column
ALTER TABLE MySchema.MyTable ADD
MyColumn int DEFAULT NULL;
But in the realm of MS SQL Server a default value on a colum is a CONSTRAINT. And like every constraint it has an identifier. And you cannot drop a column if it is used in a CONSTRAINT.
So what you can actually do avoid this kind of problems is always give your default constraints a explicit name, for example:
ALTER TABLE MySchema.MyTable ADD
MyColumn int NULL,
CONSTRAINT DF_MyTable_MyColumn DEFAULT NULL FOR MyColumn;
You'll still have to drop the constraint before dropping the column, but you will at least know its name up front.

As already written in answers you need to drop constraints (created automatically by sql) related to all columns that you are trying to delete.
Perform followings steps to do the needful.
Get Name of all Constraints using sp_helpconstraint which is a system stored procedure utility - execute following exec sp_helpconstraint '<your table name>'
Once you get the name of the constraint then copy that constraint name and execute next statement i.e alter table <your_table_name>
drop constraint <constraint_name_that_you_copied_in_1> (It'll be something like this only or similar format)
Once you delete the constraint then you can delete 1 or more columns by using conventional method i.e Alter table <YourTableName> Drop column column1, column2 etc

When you alter column datatype you need to change constraint key for every database
alter table CompanyTransactions drop constraint [df__CompanyTr__Creat__0cdae408];

You need to do a few things:
You first need to check if the constrain exits in the information schema
then you need to query by joining the sys.default_constraints and sys.columns
if the columns and default_constraints have the same object ids
When you join in step 2, you would get the constraint name from default_constraints. You drop that constraint. Here is an example of one such drops I did.
-- 1. Remove constraint and drop column
IF EXISTS(SELECT *
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = N'TABLE_NAME'
AND COLUMN_NAME = N'LOWER_LIMIT')
BEGIN
DECLARE #sql NVARCHAR(MAX)
WHILE 1=1
BEGIN
SELECT TOP 1 #sql = N'alter table [TABLE_NAME] drop constraint ['+dc.name+N']'
FROM sys.default_constraints dc
JOIN sys.columns c
ON c.default_object_id = dc.object_id
WHERE dc.parent_object_id = OBJECT_ID('[TABLE_NAME]') AND c.name = N'LOWER_LIMIT'
IF ##ROWCOUNT = 0
BEGIN
PRINT 'DELETED Constraint on column LOWER_LIMIT'
BREAK
END
EXEC (#sql)
END;
ALTER TABLE TABLE_NAME DROP COLUMN LOWER_LIMIT;
PRINT 'DELETED column LOWER_LIMIT'
END
ELSE
PRINT 'Column LOWER_LIMIT does not exist'
GO

In addition to accepted answer, if you're using Entity Migrations for updating database, you should add this line at the beggining of the Up() function in your migration file:
Sql("alter table dbo.CompanyTransactions drop constraint [df__CompanyTr__Creat__0cdae408];");
You can find the constraint name in the error at nuget packet manager console which starts with FK_dbo.

I had the same problem and this was the script that worked for me with a table with a two part name separated by a period ".".
USE [DATABASENAME]
GO
ALTER TABLE [TableNamePart1].[TableNamePart2] DROP CONSTRAINT [DF__ TableNamePart1D__ColumnName__5AEE82B9]
GO
ALTER TABLE [TableNamePart1].[ TableNamePart1] DROP COLUMN [ColumnName]
GO

I needed to replace an INT primary key with a Guid. After a few failed attempts, the EF code below worked for me. If you hyst set the defaultValue... you end up with a single Guid a the key for existing records.
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropUniqueConstraint("PK_Payments", "Payments");
migrationBuilder.DropColumn(
name: "PaymentId",
table: "Payments");
migrationBuilder.AddColumn<Guid>(
name: "PaymentId",
table: "Payments",
type: "uniqueidentifier",
defaultValueSql: "NewId()",
nullable: false);
}

Copy the default constraint name from the error message and type it in the same way as the column you want to delete.

I had the same problem, I could not remove migrations, it would show error that something is already applied, so i changed my DB name in appsettings, removed all migrations, and then added new migration and it worked. Dont understand issue completely, but it worked

I fixed by Adding Dropping constraint inside migration.
migrationBuilder.DropForeignKey(
name: "FK_XX",
table: "TableX").
and below recreates constraint.
migrationBuilder.AddForeignKey(
name: "FK_XX",
table: "TableX",
column: "ColumnX",
onDelete: ReferentialAction.Restrict);

How to handle duplicates in SQL Server

I have a table in SQL Server called Test_Table with two columns: ID and Name
The table looks something like this:
ID NAME
--------
1 John
2 Jane
Now I have a stored procedure which inserts records into this.
INSERT INTO Test_Table
VALUES (#Id,#Name)
And I'm passing this values from my c# code. Now I want to modify this so that the table does not have duplicates. Where should I check this, In the code or the DB? I'm very weak in DB side stuff. So how can I handle duplicates before inserting values in my table

The "right" way to do that is in DB because:
Don't need to read all DB
Need to pass all data to C# which increase the IO
Concurrency - if you have more than 1 C# application you will need to sync them vs in DB it would be simpler
You can define the column as unique or key, which will prevent duplicate values ,DB will take care of it
If you use MSSQL use UNIQUE Constraints
Read this good answer about avoid duplicates

You should do this check in the database. Always, if you want it to be true of the data.
I'm not sure what you consider a duplicate. Normally, an id column would be an identity column that is automatically incremented for each value. This would prevent duplicates. You would define it as:
create table test_table (
id int not null identity(1, 1),
. . .
Then, you would insert into it using:
insert into test_table(name)
values (#Name);
The id would be assigned automatically.
If you want no duplicates just for name, then create a unique index or unique constraint (really the same thing). You can do this in the table definition just by adding unique to the column:
create table test_table (
id int not null identity(1, 1),
name varchar(255) unique
. . .
Or by creating a unique index after you have created the table:
create index test_table_name on test_table(name)
(Or by explicitly creating a constraint, which is another method.)

In either case ,you will have to access to database to check wheteher values exist already.
IF NOT EXISTS (SELECT * FROM Test_Table WHERE ID= #ID AND Name=#Name)
BEGIN
INSERT INTO Test_Table
VALUES (#Id,#Name)
END
If it is possible to make ID column as unique you can avoid checking as insertion would.t be allowed for repeating ID values , in that case you will have to handle error.
See this thread how to handle violation of Unique key constraint.

If you don't want repeating IDs you'll have to set the ID as the Primary Key, which is pretty much obligatory.
If you don't want the Name to repeat, you could populate a list with the Names the table contains, and then you would only insert whatever name is not in that List.
Here is an example, instead of using a list I used a dictionary:
Dictionary<int, string> Names = new Dictionary<int, string> ();
using (SqlCommand command = new SqlCommand ("SELECT * FROM TestTable", con))
using (SqlDataReader reader = command.ExecuteReader ()) {
while (reader.Read ()) {
Names.Add (reader["ID"], reader["NAME"]);
}
}
if (!Names.ContainsValue ("ValueYouWantToInsert")) {
//do stuff
}

You should check it in DB, Also you can make ID as Primary Key
Which is mostly used, because people can have duplicate name.

You can modify your Id with the Unique key constraint or you can also make it Primary key.
Try like this:
alter table Test_Table add primary key (ID)
and
alter table Test_Table add unique key (Name)

IF NOT EXISTS (SELECT * FROM Test_Table WHERE ID= #ID AND Name=#Name)
BEGIN
INSERT INTO Test_Table
VALUES (#Id,#Name)
END
ELSE
BEGIN
UPDATE Test_Table
SET ID= #ID,NAME = #Name
WHERE ID= #ID AND Name=#Name
END

How to insert row into table or add value if row already exists

I'm making a C# based program which is using MySQL database to store data. I want to make an option to add row to table OR add value to row if it already exists. I tried to write this but nothing worked. I'm pretty new to MySQL commands so this might be obvious but I can't find an answer to my question. I think it's something like this (but like I said, I might be completely wrong):
if exists(update database.table set column = column + 1 where anothercolumn = something)
and then something to make the database do:
(insert into database.table (column1, column2, column3) values (a, b, c);

If you have a unique key (unique index or primary key) defined on the table, you can use an INSERT ... ON DUPLICATE KEY UPDATE statement
INSERT INTO mytable (col1, col2, col3) VALUES ('a','b','c')
ON DUPLICATE KEY UPDATE col3 = col3 + 1;
Otherwise, you'll need to run two separate SQL statements, an INSERT to add a row, or an UPDATE to modify a row (just like you show except for the "if exists" stuff.)
The column already "exists" in the row, the question is what value is stored.

error while inserting primary key

I'm trying (SQL Server Compact) to add primary key constraint on existing table that has some rows in it. While adding primary key I'm getting the error:
"A duplicate key cannot be inserted into a unique index"
I don't what this is, can anyone help me with this?

Make sure the data in the table respects the contraint you're trying to set on the table. If the column you are making primary has duplicate entries, it won't be able to work as primary key, hence the error.
You could try and find the rows with duplicate entries, with something like this:
select Id, Count(*) from myTable
having Count(*) > 1
group by Id

Try this
select id_column, count(*) from your_table group by id_column having count(*) > 1
If there are any records returned from this above query you cannot add a primary key on id_column since duplicate IDs exist.
Of course you will need to replace id_column and your_table with the appropriate names.