Develop Reference

Query specific column on entire tables in a database

I have a SQLite database contains several tables which some of them (not all) have a specific column called 'attachment'. I wonder if there is a query that get values of the 'attachment' column from all the tables which have this column. I can get all tables' name from the database, and then, query all tables individually. But I think there must be a single query to do this.

You can use SQLite code to get the sql SELECT statement as a string and execute it to get all the values of the column attachment in all tables of the database:
SELECT GROUP_CONCAT('SELECT ' || pti.name || ' FROM ' || sm.name, ' UNION ALL ') sql
FROM sqlite_master sm CROSS JOIN pragma_table_info(sm.name) pti
WHERE sm.type = 'table' AND pti.name = 'attachment';
The above query returns a resultset with only 1 row and 1 column aliased as sql with a value of a string like this:
SELECT attachment FROM table1
UNION ALL
SELECT attachment FROM table2
UNION ALL
SELECT attachment FROM table4
You can change UNION ALL to UNION inside the function GROUP_CONCAT(), depending on your requirement.
See a simplified demo.

You can get the values of every attachment field in every table that has this field defined using the GetSchema method on the connection to find all the relevant tables then you can use the sql stament UNION to extract in a single command all the attachments values.
This should be the code using only standard ADO.NET commands and methods:
using(SQLiteConnection cnn = new SQLiteConnection(#"Data Source=E:\\temp\\mydb.db;Version=3"))
{
cnn.Open();
// We want all columns with the name as "attachment"
DataTable dt = cnn.GetSchema("COLUMNS", new string[] {null, null, null, "attachment"});
// Now we prepare the single commands that extract the attachments value
// from every row returned by the previous query.
// The TABLE_NAME field contains the name of the relevant table
var s = dt.AsEnumerable().Select(x => $"SELECT attachment FROM [{x.Field<string>("TABLE_NAME")}]");
// We join together the single commands separating them with the UNION statement
string command = string.Join(" UNION ", s);
// Finally we can construct and execute a command loading a datatable
// with all the attachements values from every table.
SQLiteCommand cmd = new SQLiteCommand(command, cnn);
dt = new DataTable();
dt.Load(cmd.ExecuteReader());
// Here you can work on the single field in the _dt_ table
....
}
Note, UNION will extract a distinct value for each attachment. This means that if you have two attachments with the same name, it will be listed only one time. If you want to keep everything then change UNION to UNION ALL (with spaces before and after the statement)

Batch insert to SQL Server table from DataTable using ODBC Connection

I have been asked to look at finding the most efficient way to take a DataTable input and write it to a SQL Server table using C#. The snag is that the solution must use ODBC Connections throughout, this rules out sqlBulkCopy. The solution must also work on all SQL Server versions back to SQL Server 2008 R2.
I am thinking that the best approach would be to use batch inserts of 1000 rows at a time using the following SQL syntax:
INSERT INTO dbo.Table1(Field1, Field2)
SELECT Value1, Value2
UNION
SELECT Value1, Value2
I have already written the code the check if a table corresponding to the DataTable input already exists on the SQL Server and to create one if it doesn't.
I have also written the code to create the INSERT statement itself. What I am struggling with is how to dynamically build the SELECT statements from the rows in the data table. How can I access the values in the rows to build my SELECT statement? I think I will also need to check the data type of each column in order to determine whether the values need to be enclosed in single quotes (') or not.
Here is my current code:
public bool CopyDataTable(DataTable sourceTable, OdbcConnection targetConn, string targetTable)
{
OdbcTransaction tran = null;
string[] selectStatement = new string[sourceTable.Rows.Count];
// Check if targetTable exists, create it if it doesn't
if (!TableExists(targetConn, targetTable))
{
bool created = CreateTableFromDataTable(targetConn, sourceTable);
if (!created)
return false;
}
try
{
// Prepare insert statement based on sourceTable
string insertStatement = string.Format("INSERT INTO [dbo].[{0}] (", targetTable);
foreach (DataColumn dataColumn in sourceTable.Columns)
{
insertStatement += dataColumn + ",";
}
insertStatement += insertStatement.TrimEnd(',') + ") ";
// Open connection to target db
using (targetConn)
{
if (targetConn.State != ConnectionState.Open)
targetConn.Open();
tran = targetConn.BeginTransaction();
for (int i = 0; i < sourceTable.Rows.Count; i++)
{
DataRow row = sourceTable.Rows[i];
// Need to iterate through columns in row, getting values and data types and building a SELECT statement
selectStatement[i] = "SELECT ";
}
insertStatement += string.Join(" UNION ", selectStatement);
using (OdbcCommand cmd = new OdbcCommand(insertStatement, targetConn, tran))
{
cmd.ExecuteNonQuery();
}
tran.Commit();
return true;
}
}
catch
{
tran.Rollback();
return false;
}
}
Any advice would be much appreciated. Also if there is a simpler approach than the one I am suggesting then any details of that would be great.

Ok since we cannot use stored procedures or Bulk Copy ; when I modelled the various approaches a couple of years ago, the key determinant to performance was the number of calls to the server. So batching a set of MERGE or INSERT statements into a single call separated by semi-colons was found to be the fastest method. I ended up batching my SQL statements. I think the max size of a SQL statement was 32k so I chopped up my batch into units of that size.
(Note - use StringBuilder instead of concatenating strings manually - it has a beneficial effect on performance)
Psuedo-code
string sqlStatement = "INSERT INTO Tab1 VALUES {0},{1},{2}";
StringBuilder sqlBatch = new StringBuilder();
foreach(DataRow row in myDataTable)
{
sqlBatch.AppendLine(string.Format(sqlStatement, row["Field1"], row["Field2"], row["Field3"]));
sqlBatch.Append(";");
}
myOdbcConnection.ExecuteSql(sqlBatch.ToString());
You need to deal with batch size complications, and formatting of the correct field data types in the string-replace step, but otherwise this will be the best performance.

Marked solution of PhillipH is open for several mistakes and SQL injection.
Normally you should build a DbCommand with parameters and execute this instead of executing a self build SQL statement.
The CommandText must be "INSERT INTO Tab1 VALUES ?,?,?" for ODBC and OLEDB, SqlClient needs named parameters ("#<Name>").
Parameters should be added with the dimensions of underlaying column.

How to copy a MySql database schema using C#?

I'm trying to use C# & MySql in order to copy an empty table (recreate the Schema really). The structure looks like this:
> TableTemplate (schema)
+ Tables
> FirstTable (table)
> second table (table)
> ...
> SomeOtherTable
+ Tables
> ...
What I would like is to copy the TableTemplate into a new Schema with the user name.
The first obvious path to oblivion was trying CREATE TABLE #UserName LIKE TableTemplate, swiftly learning that sql parameters are supposed to be used for values and not table names (all hail jon skeet, again: How to pass a table as parameter to MySqlCommand?).
So that leaves us with manual validation of the user names in order to build the table names (robert's a prime example).
Next, it seems that even CREATE TABLE UserID LIKE TableTemplate; won't work (even from MySQL Workbench), since TableTemplate isn't a table.
So It's down to writing a loop that will create a table LIKE each table in TableTemplate, after creating a UserID Schema (after manual validation of that string), or trying other options like dumping the database and creating a new one, as seen in these questions:
C# and mysqldump
Slow performance using mysqldump from C#
But I would prefer avoid running a process, dumping the database, and creating it from there every time I add a user.
Any suggestions would be highly appreciated.

I think mysqldump would be better. but if you want to do in one process. try this.
SELECT
CONCAT ("CREATE TABLE SomeOtherTable.",
TABLE_NAME ," AS SELECT * FROM TableTemplate.", TABLE_NAME
) as creation_sql
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = 'TableTemplate';
the output will be like
CREATE TABLE SomeOtherTable.tbl_name AS SELECT * FROM TableTemplate.tbl_name;
then iterate result and execute CREATE TABLE ....

Ended up using something like this, in a method where aName is passed for the table name:
using (MySqlCommand cmd = new MySqlCommand(string.Format("CREATE DATABASE {0} ;", aName), connection))
{
cmd.ExecuteNonQuery(); // Create the database with the given user name
// Building the sql query that will return a "create table" per table in some_db template DB.
cmd.CommandText = (string.Format("SELECT CONCAT (\"CREATE TABLE {0}.\", TABLE_NAME ,\" "
+ "LIKE some_other_db.\", TABLE_NAME ) as creation_sql "
+ "FROM information_schema.TABLES WHERE TABLE_SCHEMA = 'some_db';"
, aName));
try // Building the inner tables "create table" sql strings
{
using (MySqlDataReader reader = cmd.ExecuteReader())
{
while (reader.Read())
createInnerTablesList.Add(reader.GetString(0));
}
}
catch (MySqlException mysql_ex) { ... } // handle errors
foreach (var sql_insert_query in createInnerTablesList)
{
try // Insert the tables into the user database
{
cmd.CommandText = sql_insert_query;
cmd.ExecuteNonQuery();
}
catch (Exception e) { ... } // handle errors
}
}
The reasons for using LIKE vs AS like Jungsu suggested is that even though the AS will create the tables, it will not keep any of the constraints and keys defined (primary key, etc).
Using the LIKE will replicate them with the constraints.
I'm still not too happy about this, since I feel I'm missing something though ...

How to count columns of SQL Server 2008's table using C#?

I am developing an application in C# in Visual Studio 2008. I connected a SQL Server 2008 database with it.
I want to count the number of columns so that I can loop around them to get the particular data.
I can figure it out columns by going to the database but I am joing 4-5 tables in my programs so I want to know if I can count the columns.
Can anyone help me in this?
Thank you
Shyam

select count(*) from INFORMATION_SCHEMA.columns where TABLE_NAME = 'YourTableName'

Something like this ?
SELECT COUNT(*)
FROM sys.columns c
INNER JOIN sys.tables t ON c.object_id = t.object_id
WHERE t.name = 'yourTable'
See this page provided wy TaronPro to know how to retrieve the result.

If you are using SQLConnection object to connect to DB, use its GetSchema method to get list of all columns without querying.
using (SqlConnection connection = new SqlConnection(connectionString))
{
// Connect to the database then retrieve the schema information.
connection.Open();
DataTable table = connection.GetSchema("Tables");
..
..
..
If you want to know columns for specific owner, table or table type, use restriction within GetSchema method.
string[] restrictions = new string[4];
restrictions[1] = "dbo";
DataTable table = connection.GetSchema("Tables", restrictions);
for more information refer this link.

What I did in a similar situation is that when I executed the query I retrieved all the data into a DataSet.
When I got the DataSet I opened the first table (ds.Tables[0]). Obviously you check first for existance.
When you have the table then its as simple as performing a
dt.Columns.Count;
In summary DS.Tables[0].Columns.Count
To find a specific column by name you loop through and find the one that
for (z=0; z < dt.Columns.Count; z++)
{
// check to see if the column name is the required name passed in.
if (dt.Columns[z].ColumnName == fieldName)
{
// If the column was found then retrieve it
//dc = dt.Columns[z];
// and stop looking the rest of the columns
requiredColumn = z;
break;
}
}
Then to find the data you need I would then loop through the rows of the table and get the field for that column...
ie...
string return = dr.Field<string>(requiredColumn);
Probalby not the best way of doing it but it works. Obviously if the data contained in the field is not string then you need to pass the appropriate type...
dr.Field<decimal>(requiredColumn)
dr.Field<int>(requiredColumn)
etc
Rgds
George

The reader itself gives you the number of columns. This is useful when you don't want to know the number rows of a specific table or view but from an ad-hoc query.
You can dump the columns like this
string sql = "SELECT * FROM my query";
SqlCommand cmd = new SqlCommand(sql, connection);
using (SqlDataReader reader = cmd.ExecuteReader()) {
while (reader.Read()) {
for (int i = 0; i < reader.FieldCount; i++) {
Console.WriteLine("{0} = {1}",
reader.GetName(i),
reader.IsDBNull(i) ? "NULL" : reader.GetValue(i));
}
Console.WriteLine("---------------");
}
}

you can use Microsoft.SqlServer.Management.Smo namespace to get the number of columns in a specified table as follows
1 . add Microsoft.SqlServer.Management.Smo dll in your project and use the namespace Microsoft.SqlServer.Management.Smo
2 . write the follwing code
private int colCount()
{
Server server=new Server(".\\SQLEXPRESS");
Database database=Server.Databases["your database name"];
Table table=database.Tables["your table name"];
return (table.Columns.Count);
}

SQL Insert one row or multiple rows data?

I am working on a console application to insert data to a MS SQL Server 2005 database. I have a list of objects to be inserted. Here I use Employee class as example:
List<Employee> employees;
What I can do is to insert one object at time like this:
foreach (Employee item in employees)
{
string sql = #"INSERT INTO Mytable (id, name, salary)
values ('#id', '#name', '#salary')";
// replace #par with values
cmd.CommandText = sql; // cmd is IDbCommand
cmd.ExecuteNonQuery();
}
Or I can build a balk insert query like this:
string sql = #"INSERT INTO MyTable (id, name, salary) ";
int count = employees.Count;
int index = 0;
foreach (Employee item in employees)
{
sql = sql + string.format(
"SELECT {0}, '{1}', {2} ",
item.ID, item.Name, item.Salary);
if ( index != (count-1) )
sql = sql + " UNION ALL ";
index++
}
cmd.CommandType = sql;
cmd.ExecuteNonQuery();
I guess the later case is going to insert rows of data at once. However, if I have
several ks of data, is there any limit for SQL query string?
I am not sure if one insert with multiple rows is better than one insert with one row of data, in terms of performance?
Any suggestions to do it in a better way?

Actually, the way you have it written, your first option will be faster.
Your second example has a problem in it. You are doing sql = + sql + etc. This is going to cause a new string object to be created for each iteration of the loop. (Check out the StringBuilder class). Technically, you are going to be creating a new string object in the first instance too, but the difference is that it doesn't have to copy all the information from the previous string option over.
The way you have it set up, SQL Server is going to have to potentially evaluate a massive query when you finally send it which is definitely going to take some time to figure out what it is supposed to do. I should state, this is dependent on how large the number of inserts you need to do. If n is small, you are probably going to be ok, but as it grows your problem will only get worse.
Bulk inserts are faster than individual ones due to how SQL server handles batch transactions. If you are going to insert data from C# you should take the first approach and wrap say every 500 inserts into a transaction and commit it, then do the next 500 and so on. This also has the advantage that if a batch fails, you can trap those and figure out what went wrong and re-insert just those. There are other ways to do it, but that would definately be an improvement over the two examples provided.
var iCounter = 0;
foreach (Employee item in employees)
{
if (iCounter == 0)
{
cmd.BeginTransaction;
}
string sql = #"INSERT INTO Mytable (id, name, salary)
values ('#id', '#name', '#salary')";
// replace #par with values
cmd.CommandText = sql; // cmd is IDbCommand
cmd.ExecuteNonQuery();
iCounter ++;
if(iCounter >= 500)
{
cmd.CommitTransaction;
iCounter = 0;
}
}
if(iCounter > 0)
cmd.CommitTransaction;

In MS SQL Server 2008 you can create .Net table-UDT that will contain your table
CREATE TYPE MyUdt AS TABLE (Id int, Name nvarchar(50), salary int)
then, you can use this UDT in your stored procedures and your с#-code to batch-inserts.
SP:
CREATE PROCEDURE uspInsert
(#MyTvp AS MyTable READONLY)
AS
INSERT INTO [MyTable]
SELECT * FROM #MyTvp
C# (imagine that records you need to insert already contained in Table "MyTable" of DataSet ds):
using(conn)
{
SqlCommand cmd = new SqlCommand("uspInsert", conn);
cmd.CommandType = CommandType.StoredProcedure;
SqlParameter myParam = cmd.Parameters.AddWithValue
("#MyTvp", ds.Tables["MyTable"]);
myParam.SqlDbType = SqlDbType.Structured;
myParam.TypeName = "dbo.MyUdt";
// Execute the stored procedure
cmd.ExecuteNonQuery();
}
So, this is the solution.
Finally I want to prevent you from using code like yours (building the strings and then execute this string), because this way of executing may be used for SQL-Injections.

look at this thread,
I've answered there about table valued parameter.

Bulk-copy is usually faster than doing inserts on your own.
If you still want to do it in one of your suggested ways you should make it so that you can easily change the size of the queries you send to the server. That way you can optimize for speed in your production environment later on. Query times may v ary alot depending on the query size.

The batch size for a SQL Server query is listed at being 65,536 * the network packet size. The network packet size is by default 4kbs but can be changed. Check out the Maximum capacity article for SQL 2008 to get the scope. SQL 2005 also appears to have the same limit.