I have a list of items with different ids which represent a SQL table's PK values.
Is there any way to build an efficient and safe statement?
Since now I've always prepared a string representing the statement and build it as I traversed the list via a foreach loop.
Here's an example of what I'm doing:
string update = "UPDATE table SET column = 0 WHERE";
foreach (Line l in list)
{
update += " id = " + l.Id + " OR";
}
// To remove last OR
update.Remove(update.Length - 3);
MySqlHelper.ExecuteNonQuery("myConnectionString", update);
Which feels very unsafe and looks very ugly.
Is there a better way for this?
So yeah, in SQL you've got the 'IN' keyword which allows you to specify a set of values.
This should accomplish what you would like (syntax might be iffy, but the idea is there)
var ids = string.Join(',', list.Select(x => x.Id))
string update = $"UPDATE table SET column = 0 WHERE id IN ({ids})";
MySqlHelper.ExecuteNonQuery("myConnectionString", update);
However, the way you're performing your SQL can be considered dangerous (you should be fine as this just looks like ids from a DB, who knows, better to be safe than sorry). Here you're passing parameters straight into your query string, which is a potential risk to SQL injection which is very dangerous. There are ways around this, and using the inbuilt .NET 'SqlCommand' object
https://www.w3schools.com/sql/sql_injection.asp
https://learn.microsoft.com/en-us/dotnet/api/system.data.sqlclient.sqlcommand?view=dotnet-plat-ext-6.0
It would be more efficient to use IN operator:
string update = "UPDATE table SET column = 0 WHERE id IN (";
foreach (Line l in list)
{
update += l.Id + ",";
}
// To remove last comma
update.Remove(update.Length - 1);
// To insert closing bracket
update += ")";
If using .NET Core Framework, see the following library which creates parameters for a WHERE IN. The library is a port from VB.NET which I wrote in Framework 4.7 years ago. Clone the repository, get SqlCoreUtilityLibrary project for creating statements.
Setup.
public void UpdateExample()
{
var identifiers = new List<int>() { 1, 3,20, 2, 45 };
var (actual, exposed) = DataOperations.UpdateExample(
"UPDATE table SET column = 0 WHERE id IN", identifiers);
Console.WriteLine(actual);
Console.WriteLine(exposed);
}
Just enough code to create the parameterizing SQL statement. Note ActualCommandText method is included for development, not for production as it reveals actual values for parameters.
public static (string actual, string exposed) UpdateExample(string commandText, List<int> identifiers)
{
using var cn = new SqlConnection() { ConnectionString = GetSqlConnection() };
using var cmd = new SqlCommand() { Connection = cn };
cmd.CommandText = SqlWhereInParamBuilder.BuildInClause(commandText + " ({0})", "p", identifiers);
cmd.AddParamsToCommand("p", identifiers);
return (cmd.CommandText, cmd.ActualCommandText());
}
For a real app all code would be done in the method above rather than returning the two strings.
Results
UPDATE table SET column = 0 WHERE id IN (#p0,#p1,#p2,#p3,#p4)
UPDATE table SET column = 0 WHERE id IN (1,3,20,2,45)
Related
I'm using Dapper to work with sql database.
I have some search logic in my website project.
My search gets list of string parameters.
//filter is list of strings
var sql = new StringBuilder();
sql.Append("SELECT LibraryDocumentId FROM LibraryDocumentKeywords WHERE ");
sql.Append(string.Join("OR ", filter.Select(f => string.Format("LOWER(Keyword) LIKE '%{0}%'", f)).ToList()));
var isList = conn.Query<int>(sql.ToString()).ToList();
Actually I don't want to use this approach of generating dynamic SQL query, because Dapper will cache every single query. I would prefer to pass filter with parameter. Can someone help me with that? Any idea ?
What you have at the moment is also a huge SQL injection risk. You might want to use DynamicParameters here, i.e. (completely untested, you may need to tweak slightly):
var sql = new StringBuilder(
"SELECT LibraryDocumentId FROM LibraryDocumentKeywords");
int i = 0;
var args = new DynamicParameters();
foreach(var f in filter) {
sql.Append(i == 0 ? " WHERE " : " OR ")
.Append("LOWER(Keyword) LIKE #p").Append(i);
args.Add("p" + i, "%" + f + "%");
i++;
}
var data = conn.Query<int>(sql.ToString(), args);
This should cache fairly cleanly (one cache item per number of filters, regardless of their contents).
I want to create simple database in runtime, fill it with data from internal resource and then read each record through loop. Previously I used LiteDb for that but I couldn't squeeze time anymore so
I choosed SQLite.
I think there are few things to improve I am not aware of.
Database creation process:
First step is to create table
using var create = transaction.Connection.CreateCommand();
create.CommandText = "CREATE TABLE tableName (Id TEXT PRIMARY KEY, Value TEXT) WITHOUT ROWID";
create.ExecuteNonQuery();
Next insert command is defined
var insert = transaction.Connection.CreateCommand();
insert.CommandText = "INSERT OR IGNORE INTO tableName VALUES (#Id, #Record)";
var idParam = insert.CreateParameter();
var valueParam = insert.CreateParameter();
idParam.ParameterName = "#" + IdColumn;
valueParam.ParameterName = "#" + ValueColumn;
insert.Parameters.Add(idParam);
insert.Parameters.Add(valueParam);
Through loop each value is inserted
idParameter.Value = key;
valueParameter.Value = value.ValueAsText;
insert.Parameters["#Id"] = idParameter;
insert.Parameters["#Value"] = valueParameter;
insert.ExecuteNonQuery();
Transaction commit transaction.Commit();
Create index
using var index = transaction.Connection.CreateCommand();
index.CommandText = "CREATE UNIQUE INDEX idx_tableName ON tableName(Id);";
index.ExecuteNonQuery();
And after that i perform milion selects (to retrieve single value):
using var command = _connection.CreateCommand();
command.CommandText = "SELECT Value FROM tableName WHERE Id = #id;";
var param = command.CreateParameter();
param.ParameterName = "#id";
param.Value = id;
command.Parameters.Add(param);
return command.ExecuteReader(CommandBehavior.SingleResult).ToString();
For all select's one connection is shared and never closed. Insert is quite fast (less then minute) but select's are very troublesome here. Is there a way to improve them?
Table is quite big (around ~2 milions records) and Value contains quite heavy serialized objects.
System.Data.SQLite provider is used and connection string contains this additional options: Version=3;Journal Mode=Off;Synchronous=off;
If you go for performance, you need to consider this: each independent SELECT command is a roundtrip to the DB with some extra costs. It's similar to a N+1 select problem in case of parent-child relations.
The best thing you can do is to get a LIST of items (values):
SELECT Value FROM tableName WHERE Id IN (1, 2, 3, 4, ...);
Here's a link on how to code that: https://www.mikesdotnetting.com/article/116/parameterized-in-clauses-with-ado-net-and-linq
You could have the select command not recreated for every Id but created once and only executed for every Id. From your code it seems every select is CreateCommand/CreateParameters and so on. See this for example: https://learn.microsoft.com/en-us/dotnet/api/system.data.idbcommand.prepare?view=net-5.0 - you run .Prepare() once and then only execute (they don't need to be NonQuery)
you could then try to see if you can be faster with ExecuteScalar and not having reader created for one data result, like so: https://learn.microsoft.com/en-us/dotnet/api/system.data.idbcommand.executescalar?view=net-5.0
If scalar will not prove to be faster then you could try to use .SingleRow instead of .SingleResult in your ExecuteReader for possible performance optimisations. According to this: https://learn.microsoft.com/en-us/dotnet/api/system.data.commandbehavior?view=net-5.0 it might work. I doubt that but if first two don't help, why not try it too.
How can I use dynamic queries in C# ? From what I've searched its similiar to when we use SqlCommand with parameters to prevent sql injection(example below).
using (SQLiteConnection DB_CONNECTION = new SQLiteConnection(connectionString))
{
DB_CONNECTION.Open();
string sqlquery = "UPDATE table SET Name =#Name, IsComplete=#IsComplete WHERE Key =#Key;";
int rows = 0;
using (SQLiteCommand command = new SQLiteCommand(sqlquery, DB_CONNECTION))
{
SQLiteParameter[] tableA = { new SQLiteParameter("#Key", todo.Key), new SQLiteParameter("#Name", table.Name), new SQLiteParameter("#IsComplete", table.IsComplete) };
command.Parameters.AddRange(tableA);
rows = command.ExecuteNonQuery();
}
DB_CONNECTION.Close();
return (rows);
}
I'm new to c# and i wondering how can I make this work, thanks in advance.
Basically just build up the string sqlQuery based on a set of conditions and ensure that the appropriate parameters have been set. For example, here is some psuedo-C# (not tested for bugs):
//Set to true, so our queries will always include the check for SomeOtherField.
//In reality, use some check in the C# code that you would want to compose your query.
//Here we set some value we want to compare to.
string someValueToCheck = "Some value to compare";
using (SQLiteConnection DB_CONNECTION = new SQLiteConnection(connectionString))
{
DB_CONNECTION.Open();
string sqlquery = "UPDATE MyTable SET Name =#Name, IsComplete=#IsComplete WHERE Key =#Key";
//Replace this with some real condition that you want to use.
if (!string.IsNullOrWhiteSpace(someValueToCheck))
{
sqlquery += " AND SomeOtherField = #OtherFieldValue"
}
int rows = 0;
using (SQLiteCommand command = new SQLiteCommand(sqlquery, DB_CONNECTION))
{
//Use a list here since we can't add to an array - arrays are immutable.
List<SQLiteParameter> tableAList = {
new SQLiteParameter("#Key", todo.Key),
new SQLiteParameter("#Name", table.Name),
new SQLiteParameter("#IsComplete", table.IsComplete) };
if (!string.IsNullOrWhiteSpace(someValueToCheck)) {
//Replace 'someValueToCheck' with a value for the C# that you want to use as a parameter.
tableAList.Add(new SQLiteParameter("#OtherFieldValue", someValueToCheck));
}
//We convert the list back to an array as it is the expected parameter type.
command.Parameters.AddRange(tableAList.ToArray());
rows = command.ExecuteNonQuery();
}
DB_CONNECTION.Close();
return (rows);
}
In this day and age it would probably be worth looking into LINQ to Entities, as this will help you to compose queries dynamically in your code - for example https://stackoverflow.com/a/5541505/201648.
To setup for an existing database - also known as "Database First" - see the following tutorial:
https://msdn.microsoft.com/en-au/data/jj206878.aspx
You can skip step 1 since you already have a database, or do the whole tutorial first as practice.
Here is some psuedo-C# LINQ code to perform roughly the same update as the previous example:
//The context you have setup for the ERP database.
using (var db = new ERPContext())
{
//db is an Entity Framework database context - see
//https://msdn.microsoft.com/en-au/data/jj206878.aspx
var query = db.MyTable
.Where(c => c.Key == todo.Key);
if (!string.IsNullOrWhiteSpace(someValueToCheck))
{
//This where is used in conjunction to the previous WHERE,
//so it's more or less a WHERE condition1 AND condition2 clause.
query = query.Where(c => c.SomeOtherField == someValueToCheck);
}
//Get the single thing we want to update.
var thingToUpdate = query.First();
//Update the values.
thingToUpdate.Name = table.Name;
thingToUpdate.IsComplete = table.IsComplete;
//We can save the context to apply these results.
db.SaveChanges();
}
There is some setup involved with Entity Framework, but in my experience the syntax is easier to follow and your productivity will increase. Hopefully this gets you on the right track.
LINQ to Entites can also map SQL stored procedures if someone one your team objects to using it for performance reasons:
https://msdn.microsoft.com/en-us/data/gg699321.aspx
OR if you absolutely ust compose custom queries in the C# code this is also permitted in Entity Framework:
https://msdn.microsoft.com/en-us/library/bb738521(v=vs.100).aspx
I have an application I need to create which, given some user input in the form of CSV, needs to parse and generate this CSV into multiple formats. One of these formats is a series of SQL INSERT statements (as a string) for each line of CSV.
(At this point you can assume I've already parsed the CSV into a list of values or something, so that is not the point of the question)
Given that this input could contain vulnerabilities, I wish to generate the INSERT statements which have been validated and sanitised.
I am familiar with creating an SqlCommand object and adding values to its list of Parameters, but looking over a similar question it doesn't appear to work in way I had hoped.
So is there a way to generate sanitised SQL statements, as strings, in the way I need to?
EDIT: This is an example what I want to do.
CSV:
id,name,country
1,Alice,France
2,Bob,Austria
3,Carol,Germany
SQL:
...
INSERT INTO Users (id, name, country) VALUES (1, 'Alice', 'France');
INSERT INTO Users (id, name, country) VALUES (2, 'Bob', 'Austria');
INSERT INTO Users (id, name, country) VALUES (3, 'Carol', 'Germany');
...
As there are no data types given in the CSV, the application has to determine that as well.
Not so much an answer, as a cautionary note. If you end up needing to go the 'classic' escaping route to do this, and really need safety (i.e. the data is coming in from untrusted source), don't forget it's not just simple escaping you need to worry about.
Basic character escaping we hear about all the time:
' -> '' apostrophe's and stuff are quite obvious and documented ad-nauseum
; multiple-commands in one statement - not always allowed by the DB, but dangerous
If you're parsing for "nefarious behaviour" though, have you thought about:
SELECT/*not important*/1/*really...*/FROM/*im serious*/users
SELECT%09FIELDS%09FROM%0dTABLE_NAME
WHERE username=CONCAT(CHAR(97),CHAR(100),CHAR(109),CHAR(105),CHAR(110))
SELECT passwd FROM users WHERE username=0x61646d696e
In summary: Here Be Dragons.
http://www.ihteam.net/papers/blind-sqli-regexp-attack.pdf
http://ferruh.mavituna.com/sql-injection-cheatsheet-oku/#HexbasedSamples
Well chances are if you don't wan't to use SQL objects then you would have to sanatize the entries yourself. I'm not aware of any recommended format for SQL however for MySQL the following would work. I've changed it to work with SQL however I cant garantee it has covered all possible injection attacks.
public string sqlEscape(string VAL)
{
if (VAL == null)
{
return null;
}
return "\"" + Regex.Replace(VAL, #"[\r\n\x00\x1a\\'""]", #"\$0") + "\"";
}
to use you would then do (assuming your CSV line is stored in an array called csv):
string query = #"INSERT INTO Users (id, name, country) VALUES (" + sqlEscape(csv[0]) + ", " + sqlEscape(csv[1]) + ", " + sqlEscape(csv[2]) + ");";
if anyone can enhance this let me know!
Because i don't know how you've stored your variables, i'll show you a complete, possible implementation with your sample data using a List<Dictionary<String, Object>>():
Add your sample-data:
var tableName = "Users";
var records = new List<Dictionary<String, Object>>();
var recordFields = new Dictionary<String, Object>();
recordFields.Add("id", 1);
recordFields.Add("name", "Alice");
recordFields.Add("country", "France");
records.Add(recordFields);
recordFields = new Dictionary<String, Object>();
recordFields.Add("id", 2);
recordFields.Add("name", "Bob");
recordFields.Add("country", "Austria");
records.Add(recordFields);
recordFields = new Dictionary<String, Object>();
recordFields.Add("id", 3);
recordFields.Add("name", "Carol");
recordFields.Add("country", "Germany");
records.Add(recordFields);
Generate the parametrized insert statements:
using (var con = new SqlConnection(Settings.Default.ConnectionString))
{
con.Open();
foreach (var record in records)
{
String insertSql = String.Format("INSERT INTO {0} ({1}) VALUES ({2});"
, tableName
, String.Join(",", record.Select(r => r.Key))
, String.Join(",", record.Select(r => "#" + r.Key)));
using (var insertCommand = new SqlCommand(insertSql, con))
{
foreach (var field in record)
{
var param = new SqlParameter("#" + field.Key, field.Value);
insertCommand.Parameters.Add(param);
}
insertCommand.ExecuteNonQuery();
}
}
}
Note that this is not really tested(it compiles and looks good) but it should help you anyway.
Edit:
#Oded: I think the problem is that daniel doesn't want to run the
inserts, but show/output/save them. So using the SqlCommand parameters
is no good, because you don't see the generated SQL.
#Cylindric That's correct
That's not possible and a contradiciton, you cannot use a String with SqlParameters. So i'm afraid you're open to Sql-Injection if you would run these inserts later. I would suggest to use above code when you're actually running the statemenets.
I have a simple problem with a not so simple solution... I am currently inserting some data into a database like this:
kompenzacijeDataSet.KompenzacijeRow kompenzacija = kompenzacijeDataSet.Kompenzacije.NewKompenzacijeRow();
kompenzacija.Datum = DateTime.Now;
kompenzacija.PodjetjeID = stranka.id;
kompenzacija.Znesek = Decimal.Parse(tbZnesek.Text);
kompenzacijeDataSet.Kompenzacije.Rows.Add(kompenzacija);
kompenzacijeDataSetTableAdapters.KompenzacijeTableAdapter kompTA = new kompenzacijeDataSetTableAdapters.KompenzacijeTableAdapter();
kompTA.Update(this.kompenzacijeDataSet.Kompenzacije);
this.currentKompenzacijaID = LastInsertID(kompTA.Connection);
The last line is important. Why do I supply a connection? Well there is a SQLite function called last_insert_rowid() that you can call and get the last insert ID. Problem is it is bound to a connection and .NET seems to be reopening and closing connections for every dataset operation. I thought getting the connection from a table adapter would change things. But it doesn't.
Would anyone know how to solve this? Maybe where to get a constant connection from? Or maybe something more elegant?
Thank you.
EDIT:
This is also a problem with transactions, I would need the same connection if I would want to use transactions, so that is also a problem...
Using C# (.net 4.0) with SQLite, the SQLiteConnection class has a property LastInsertRowId that equals the Primary Integer Key of the most recently inserted (or updated) element.
The rowID is returned if the table doesn't have a primary integer key (in this case the rowID is column is automatically created).
See https://www.sqlite.org/c3ref/last_insert_rowid.html for more.
As for wrapping multiple commands in a single transaction, any commands entered after the transaction begins and before it is committed are part of one transaction.
long rowID;
using (SQLiteConnection con = new SQLiteConnection([datasource])
{
SQLiteTransaction transaction = null;
transaction = con.BeginTransaction();
... [execute insert statement]
rowID = con.LastInsertRowId;
transaction.Commit()
}
select last_insert_rowid();
And you will need to execute it as a scalar query.
string sql = #"select last_insert_rowid()";
long lastId = (long)command.ExecuteScalar(sql); // Need to type-cast since `ExecuteScalar` returns an object.
last_insert_rowid() is part of the solution. It returns a row number, not the actual ID.
cmd = CNN.CreateCommand();
cmd.CommandText = "SELECT last_insert_rowid()";
object i = cmd.ExecuteScalar();
cmd.CommandText = "SELECT " + ID_Name + " FROM " + TableName + " WHERE rowid=" + i.ToString();
i = cmd.ExecuteScalar();
I'm using Microsoft.Data.Sqlite package and I do not see a LastInsertRowId property. But you don't have to create a second trip to database to get the last id. Instead, combine both sql statements into a single string.
string sql = #"
insert into MyTable values (null, #name);
select last_insert_rowid();";
using (var cmd = conn.CreateCommand()) {
cmd.CommandText = sql;
cmd.Parameters.Add("#name", SqliteType.Text).Value = "John";
int lastId = Convert.ToInt32(cmd.ExecuteScalar());
}
There seems to be answers to both Microsoft's reference and SQLite's reference and that is the reason some people are getting LastInsertRowId property to work and others aren't.
Personally I don't use an PK as it's just an alias for the rowid column. Using the rowid is around twice as fast as one that you create. If I have a TEXT column for a PK I still use rowid and just make the text column unique. (for SQLite 3 only. You need your own for v1 & v2 as vacuum will alter rowid numbers)
That said, the way to get the information from a record in the last insert is the code below. Since the function does a left join to itself I LIMIT it to 1 just for speed, even if you don't there will only be 1 record from the main SELECT statement.
SELECT my_primary_key_column FROM my_table
WHERE rowid in (SELECT last_insert_rowid() LIMIT 1);
The SQLiteConnection object has a property for that, so there is not need for additional query.
After INSERT you just my use LastInsertRowId property of your SQLiteConnection object that was used for INSERT command.
Type of LastInsertRowId property is Int64.
Off course, as you already now, for auto increment to work the primary key on table must be set to be AUTOINCREMENT field, which is another topic.
database = new SQLiteConnection(databasePath);
public int GetLastInsertId()
{
return (int)SQLite3.LastInsertRowid(database.Handle);
}
# How about just running 2x SQL statements together using Execute Scalar?
# Person is a object that has an Id and Name property
var connString = LoadConnectionString(); // get connection string
using (var conn = new SQLiteConnection(connString)) // connect to sqlite
{
// insert new record and get Id of inserted record
var sql = #"INSERT INTO People (Name) VALUES (#Name);
SELECT Id FROM People
ORDER BY Id DESC";
var lastId = conn.ExecuteScalar(sql, person);
}
In EF Core 5 you can get ID in the object itself without using any "last inserted".
For example:
var r = new SomeData() { Name = "New Row", ...};
dbContext.Add(r);
dbContext.SaveChanges();
Console.WriteLine(r.ID);
you would get new ID without thinking of using correct connection or thread-safety etc.
If you're using the Microsoft.Data.Sqlite package, it doesn't include a LastInsertRowId property in the SqliteConnection class, but you can still call the last_insert_rowid function by using the underlying SQLitePCL library. Here's an extension method:
using Microsoft.Data.Sqlite;
using SQLitePCL;
public static long GetLastInsertRowId(this SqliteConnection connection)
{
var handle = connection.Handle ?? throw new NullReferenceException("The connection is not open.");
return raw.sqlite3_last_insert_rowid(handle);
}