I have a table with 3 columns - id (pk), pageId (fk), name. I have a php script which dumps about 5000 records into the table, with about half being duplicates, with same pageId and name. Combination of pageId and name should be unique. What is the best way to prevent duplicates being saved to the table as I loop through the script in php?
First step would be to set a unique key on the table:
ALTER TABLE thetable ADD UNIQUE INDEX(pageid, name);
Then you have to decide what you want to do when there's a duplicate. Should you:
ignore it?
INSERT IGNORE INTO thetable (pageid, name) VALUES (1, "foo"), (1, "foo");
Overwrite the previously entered record?
INSERT INTO thetable (pageid, name, somefield)
VALUES (1, "foo", "first")
ON DUPLICATE KEY UPDATE (somefield = 'first')
INSERT INTO thetable (pageid, name, somefield)
VALUES (1, "foo", "second")
ON DUPLICATE KEY UPDATE (somefield = 'second')
Update some counter?
INSERT INTO thetable (pageid, name)
VALUES (1, "foo"), (1, "foo")
ON DUPLICATE KEY UPDATE (pagecount = pagecount + 1)
You can also ignore the error with mysql: INSERT IGNORE INTO TABLE ... it will ignore the key error, skip over that insert and move on to the next.
From a mysql point you can do
alter table YOURTABLE add unique index(pageId, name);
If your wording is correct and you want to do it from php you can do
$already_done = array();
foreach ($records as $record)
{
$unique_hash = md5($record['name'].$record['pageId']);
if (!in_array($unique_hash, $already_done))
{
$already_done[] = $unique_hash;
// sql insert here
}
}
either way those should do you just fine.
You can set the PageID and Name to a Unique index in the MySQL database. This way when you insert the rows, it will cause an error, which can be ignored by PHP, and you can just go to the next row.
This assumes you are inserting rows individually. AKA:
foreach($large_data as $fields)
{
mysql_query("INSERT INTO TABLE (`Something`) VALUES('".$fields['something']."');
}
Related
I'm working with Dapper and .Net 6.0... I have to do an insert from table1 to table2... in the insert the columns match each other... the only column is "toID" of table1... which does NOT match the column of table2 (that's why I put a 4 in it) but I have to make it auto-increment so that for each insert there is an incrementing sequence
var sql =
$"INSERT INTO table1 (toId,teId,dateShift,SectorOrigen) SELECT 4,teID,#dateModify,#LastSector FROM table2";
That is to say... that when I generate an insert the ID = 1, then another ID = 2 and so on continuously
Any advice??
You can create and use the sequence https://learn.microsoft.com/en-us/sql/t-sql/statements/create-sequence-transact-sql?view=sql-server-ver16
I want to enter unique index value into my database when insert other table data. In this case my table index column ID is unique but not auto-increment.
I also want to check the maximum of index value in target table. If the table is empty then index value starts from 0, OR if it contains 1 row then the index value starts from 2.
Already I can successfully do this operation using a stored procedure. But I want to do this operation using Entity Framework when saving data.
My stored procedure is:
ALTER PROCEDURE [dbo].[CreatePerson]
#Name Nvarchar(50),
#Code Nvarchar(50)
AS
BEGIN
Declare #Id int
SET #Id = ISNULL(((SELECT MAX([id]) FROM dbo.tbl_Person)+1),'1')
Insert Into dbo.tbl_Person([Id], [Name], [Code])
Values (#id, #Name, #Code)
END
Can anyone tell me what the LINQ Command for this index value save process?
A straight translation of your SQL stored procedure into Linq would basically be something like this:
var newId = ctx.Persons.Any() ? ctx.Persons.Select(p => p.Id).Max() + 1 : 1;
var newPerson = new Person { Id = newId, Name = someName, Code = someCode };
ctx.Persons.Add(newPerson);
ctx.SaveChanges();
Note: this solution is certainly NOT recommended for inserting unique id's in a database, consider carefully the suggestions of Sharped and Magnus in the comments below the question to better use auto-increment or random guids to solve your problem.
With an auto-incremented id column your code would look like this:
var newPerson = new Person { Name = someName, Code = someCode };
ctx.Persons.Add(newPerson);
ctx.SaveChanges();
var newId = newPerson.Id;
Note that EF will update the id column of the newly created entity for you in code automagically.
In a c# desktop application I am getting this list of data which I am reading by barcode into text file; here is the result;
R900, 27674T07, 27438T17, 27736T21, 26609T08,
R901, 27770T12, 27833T07, 26402T12, 27771T09, 26003T13,
R902, 26003T14, 26402T11, 26246T17,
R904, 28055T09, 25356T08, 25825T07, 25556T09,
and I am transforming it to update queries;
UPDATE TABLE SET NUMBER = R900 WHERE id in ( 27674T07, 27438T17, 27736T21, 26609T08)
UPDATE TABLE SET NUMBER = R901 WHERE id in ( 27770T12, 27833T07, **26402T12**, **27771T09**, 26003T13)
UPDATE TABLE SET NUMBER = R902 WHERE id in ( 26003T14, **26402T11**, 26246T17)
UPDATE TABLE SET NUMBER = R904 WHERE id in ( 28055T09, 25356T08, 25825T07, **25556T09**)
Finally I am executing this SQL query. But the problem is I don't know which id is not found in IN clause in database. I need to report back to user which id didn't found with its NUMBER
For example the bold id's are not found in database, and couldn't update. So expected result is:
NUMBER id
R901 26402T12
R901 27771T09
R902 26402T11
R903 25556T09
how can I return this?
You could do something like this
declare #mytable as TABLE
(
Id nvarchar(20)
)
UPDATE TABLE SET NUMBER = R900
OUTPUT INSERTED.Id into #mytable
WHERE id in ( 27674T07, 27438T17, 27736T21, 26609T08)
Select * from #mytable
#mytable will contain updated Ids only.
Hope this helps.
create a temp table to store the splitted value into it.
then
SELECT temp.number, temp.Id
FROM #temp temp
LEFT OUTER JOIN TABLE ON temp.id = TABLE.id
WHERE TABLE.id is null
Is it possible to perform an insert or update with the following constraints :
Dapper is used in the project
The Primary Key is auto-incrementing positive
The newly inserted data may not have a PK value (or has a value of 0)
The data needs to have the PK value on a newly inserted row.
The query is being generated procedurally, and generalizing it would be preferable
Something like :
int newId = db.QueryValue<int>( <<insert or update>>, someData );
I have read about different solutions, and the best solution seems to be this one :
merge tablename as target
using (values ('new value', 'different value'))
as source (field1, field2)
on target.idfield = 7
when matched then
update
set field1 = source.field1,
field2 = source.field2,
...
when not matched then
insert ( idfield, field1, field2, ... )
values ( 7, source.field1, source.field2, ... )
but
it seems to fail on the third constraint and
it does not guarantee to return the newly generated id.
Because of the 5th constraint (or preferance), a stored procedure seems overly complicated.
What are the possible solutions? Thanks!
If your table has an auto-increment field, you can't assign a value to that field when inserting a record. OK you can, but it's normally a bad idea :)
Using the T-SQL MERGE statement you can put all of the values into the source table, including your default invalid identity value, then write the insert clause as:
when not matched then
insert (field1, field2, ...)
values (source.field1, source.field2, ...)
: and use the output clause to get the inserted identity value:
OUTPUT inserted.idfield
That said, I think you might be complicating your SQL code generation a little, especially for tables with a lot of fields. It is often better to generate distinct UPDATE and INSERT queries... especially if you've got some way of tracking the changes to the object so that you can only update the changed fields.
Assuming you're working on MS SQL, you can use SCOPE_IDENTITY() function after the INSERT statement to get the value of the identity field for the record in a composite statement:
INSERT INTO tablename(field1, field2, ...)
VALUES('field1value', 'field2value', ...);
SELECT CAST(SCOPE_IDENTITY() AS INT) ident;
When you execute this SQL statement you'll get back a resultset with the inserted identity in a single column. Your db.QueryValue<int> call will then return the value you're after.
For standard integer auto-increment fields the above is fine. For other field types, or for a more general case, try casting SCOPE_IDENTITY() result to VARCHAR(MAX) and parse the resultant string value to whichever type your identity column expects - GUID, etc.
In the general case, try this in your db class:
public string InsertWithID(string insertQuery, params object[] parms)
{
string query = insertQuery + "\nSELECT CAST(SCOPE_IDENTITY() AS VARCHAR(MAX)) ident;\n";
return this.QueryValue<string>(insertQuery, parms);
}
And/or:
public int InsertWithIntID(string insertQuery, params object[] parms)
{
string query = insertQuery + "\nSELECT CAST(SCOPE_IDENTITY() AS INT) ident;\n";
return this.QueryValue<int>(query, parms);
}
That way you can just prepare your insert query and call the appropriate InsertWithID method to get the resultant identity value. That should satisfy your 5th constraint with luck :)
Here is the my problem: I have a SQL Server database called emp. It has an employee table (with userid int column). I need to retrieve the last record of the userid in employee table with increment userid value + 1. At the moment I did it on the my GUI. So how do I write a sql query for it?
To return the the record with the highest userid, you can do:
SELECT TOP 1 userid
FROM employee
ORDER BY userid DESC
or...
SELECT MAX(userid)
FROM employee
If your plan is to then increment the userid manually and insert a new record with that new ID, I'd recommend against doing that - what if 2 processes try to do it at the same time? Instead, use an IDENTITY column as userid and let the incrementing be handled for you automatically
You shouldn't be manually incrementing the userid column, use an IDENTITY column instead. That will automatically add 1 for you for every new row.
CREATE TABLE Employees (
UserId INT IDENTITY PRIMARY KEY NOT NULL,
UserName NVARCHAR(255) NOT NULL,
// etc add other columns here
)
If you really really have to select the highest userid it is a very simple query:
SELECT MAX(UserId) + 1
FROM Employees
[Edit]
Based on your comments, you should use the SELECT MAX(UserId) + 1 FROM Employees query. But be aware that this does not guarantee the number will be the ID. Normally you would not show an Id value until after the record has been saved to the database.
This will give you last inserted record, If you don't have Identity column.
EXECUTE ('DECLARE GETLAST CURSOR DYNAMIC FOR SELECT * FROM [User]')
OPEN GETLAST
FETCH LAST FROM GETLAST
CLOSE GETLAST
DEALLOCATE GETLAST
If you have set identity than you can use following.
SELECT top(1) ID from [YourTable] order by ID desc
To have the new userid before saving, create a NextId table.
Before inserting the user, get the new value from NextId:
UserId = SELECT Coalesce(NextId, 0) + 1 from NextId
Then update the NextID table:
UPDATE NEXTID SET NextId = IserID
And then use that value in your user creation code
You can get gaps, there are more complicated methods to avoid them; but I think this will do