Develop Reference

Is there any way to make this query faster and build where clause outside of loop?

I have the following code, (and I am completely aware about parameterized queries and SQL Injection):
foreach(var item in items)
{
string query = "select sum(convert(decimal(18,3),tbl.Price)) p, sum(convert(decimal(18,2),tbl.Sale)) s from table1 tbl " +
$"where tbl.ID = {item .ID}";
Execute(query);
//Do stuff with query result
}
The problem is I have a lot of items and I have to execute the query for each of the items because the where clause will be complete in each step. I think if I will be able to make my query out side of my loop, my query will be faster. But I don't know how. Is there any way to do this?

Instead of executing the query for every item. You can add group by to your query and execute only once.
string query = "select tbl.ID, sum(convert(decimal(18,3),tbl.Price)) p, sum(convert(decimal(18,2),tbl.Sale)) s from table1 tbl group by tbl.ID ";
var result = Execute(query);
foreach(var item in items)
{
var row = result.Select(r => r.ID == item.ID).FirstOrDefault();
//Do stuff with query result
}

Do not execute the query for each ID separately. Instead, execute a single query for all Ids using group by to get the p and s values for each id and a parameterized in clause (or better yet, a stored procedure with a table valued parameter).
Here is the IN version of the query:
select Id,
sum(convert(decimal(18,3),tbl.Price)) p,
sum(convert(decimal(18,2),tbl.Sale)) s
from table1 tbl
Where Id IN(<1,2,3,4....>)
group by Id
Replace <1,2,3,4....> with parameters like described in this answer.
Here is the table valued parameter version of the query:
select tbl.Id,
sum(convert(decimal(18,3),tbl.Price)) p,
sum(convert(decimal(18,2),tbl.Sale)) s
from table1 tbl
inner join #items i on tbl.Id = i.Id
group by tbl.Id
For a detailed explanation about using table valued parameters, read this answer.

C# Dynamic full outer join on 2 datatables

I need to do a full outer join on 2 datatables dinamically, I don't know what columns are in the datatables but they will be the same in each table, I also only have the name of the column I need to do the join on in a variable. Is there a way of doing this?
What I need to do is join 2 datatables in a C# script. I'm using a Dataflow in an SSIS to get data from a couple of files, and at the end I need to compare the 2 final sets of data. I need to to this on whatever 2 datatables as long as they have the same columns, so I can't finish the process in an SSIS as I need to specify the columns.
The GetData() I just use it in case I need to compare 2 tables but donnesPROD and donnesDEV are filled from object variables in the SSIS.
Here's my code so far :
DataTable donnesPROD = GetData(connectionPROD, sql_request);
DataTable donnesDEV = GetData(connectionDEV, sql_request);
Here's the code for GetData :
DataTable GetData(string cs, string query)
{
OleDbConnection conn = new OleDbConnection(cs);
conn.Open();
OleDbCommand cmd = new OleDbCommand(query, conn);
DataTable dt = new DataTable();
dt.Load(cmd.ExecuteReader());
conn.Close();
return dt;
}
I have the list of the columns in another datatable, and I have the name of the primary key in a string variable key. From here I need to be able to do a fullouterjoin of donnesPROD and donnesDEV on key. Can this be done this way? Or is there a way of generating the script code it self dynamically and then execute it?

You have two options.
Conditional joins
If you don't know the specific column name, but you do have some idea what the column name might be, you could do a conditional join like this:
CREATE PROCEDURE ExampleDynamicJoin(#JoinColumn AS VarChar(40))
AS
BEGIN
SELECT *
FROM TableA
JOIN TableB ON (#JoinColumn = 'ColumnA' AND TableA.ColumnA = TableB.ColumnA)
OR (#JoinColumn = 'ColumnB' AND TableA.ColumnB = TableB.ColumnB)
OR (#JoinColumn = 'ColumnC' AND TableA.ColumnC = TableB.ColumnC)
END
You may not get the best performance out of this (the conditional joins will confuse the query engine and it may not pick the best index, if it picks one at all). If the table is very large you could also do something like this. It is a bit painful-looking but will get better performance:
CREATE PROCEDURE ExampleDynamicJoin(#JoinColumn AS VarChar(40))
AS
BEGIN
IF (#JoinColumn = 'ColumnA') BEGIN
SELECT *
FROM TableA
JOIN TableB ON TableA.ColumnA = TableB.ColumnA
END
IF (#JoinColumn = 'ColumnB') BEGIN
SELECT *
FROM TableA
JOIN TableB ON TableA.ColumnB = TableB.ColumnB
END
IF (#JoinColumn = 'ColumnC') BEGIN
SELECT *
FROM TableA
JOIN TableB ON TableA.ColumnC = TableB.ColumnC
END
END
If TableA or TableA are part of a larger query, and you'd end up duplicating tons of SQL, you could always extract the resultset for just TableA and TableB into a temporary table, then use the temporary table in the larger query.
Dynamic SQL
If you don't have the foggiest about the column name and there are tons of possibilities, you could construct the SQL as a string and join that way. You should validate the column name that is passed in; not only will that make sure the column actually exists, but it will prevent the dynamic SQL from being constructed when #JoinColumn contains an injection attack, since legal column names do not contain SQL statements. Example:
CREATE PROCEDURE ExampleDynamicJoin(#JoinColumn AS VarChar(40))
AS
BEGIN
DECLARE #Sql AS VarChar(MAX)
IF NOT EXISTS
(
SELECT 0
FROM syscolumns c
JOIN sysobjects o ON o.id = c.id
WHERE o.Name = 'TableA'
AND c.Name = #JoinColumn
)
RAISERROR (15600,-1,-1, 'ExampleDynamicJoin'); //Throw error if column doesn't exist
SET #Sql =
'SELECT *
FROM TableA
JOIN TableB ON TableA.' + #JoinColumn + ' = TableB.' + #JoinColumn
sp_ExecuteSql #Sql
END
Or, if you don't use stored procedures,
DataTable ExampleDynamicJoin(string joinColumn)
{
if (!ValidateColumn(joinColumn)) throw new ArgumentException();
var sql = String.Format(
#"SELECT *
FROM TableA
JOIN TableB ON TableA.{0} = TableB.{0}",
joinColumn
);
using (var connection = GetConnectionFromSomewhere())
{
using (var cmd = new SqlCommand
{
CommandText = sql,
CommandType = CommandType.Text,
Connection = connection
})
{
var reader = cmd.ExecuteReader();
var table = new DataTable();
table.Load(reader);
return table;
}
}
}
When using dynamic SQL you should always use parameters if possible. But you can't use parameters as a column name, so in this case you have to concatenate. When concatenating, ALWAYS white list the inputs. That is why I included a function named ValidateColumn which could look like this:
bool ValidateColumn(string columnName)
{
switch columnName.ToUpper()
{
case "COLUMNA":
case "COLUMNB":
case "COLUMNC":
return true;
default:
return false;
}
}

While Selecting based on ROWID - Distinct or Group By doesn't work in oracle

I have query like below , I tried to filter out duplicate columns by using Group BY
SELECT contacts.rowid AS ROW_PASS,
duty_rota.rowid AS ROW_PASS_ROTA,
duty_rota.duty_type AS DUTY_TYPE
FROM duty_rota,
duty_types,
contacts
WHERE duty_rota.duty_type = duty_types.duty_type
AND duty_rota.duty_officer = contacts.duty_id
AND sname IS NOT NULL
GROUP BY contacts.rowid,
duty_rota.rowid,
duty_rota.duty_type
ORDER BY duty_date
After playing with the query little bit I came to know we can't filter out distinct using group by while using ROWID. So can somebody please help me to write code (in SQL) with a logic that
if (any row is completely identical with another row of the query o/p)
{
then display only one column
}
I will be using the output as gridview's data source in C#, so if not in SQL - can you help me whether somehow in C# I can achieve to display only identical columns?

If you want to filter duplicate rows, you can use this query:
SELECT Max(duty_rota.rowid) AS ROW_PASS_ROTA,
duty_rota.duty_type AS DUTY_TYPE
FROM duty_rota,
duty_types,
contacts
WHERE duty_rota.duty_type = duty_types.duty_type
AND duty_rota.duty_officer = contacts.duty_id
AND sname IS NOT NULL
GROUP BY duty_rota.duty_type
ORDER BY DUTY_TYPE

Here you go: http://sqlfiddle.com/#!2/2a038/2
Take out the ROWID's. Example: If your table has 3 columns (colA, colB, colC) you could find exact row dups this way...
select a.* from
(
select count(*) dupCnt, colA, colB, colC from myTable
group by colA, colB, colC
) a
where dupCnt > 1

First, the ROWID is a unique field for each row, so using this field you will never have duplicates. The only solution here is to not use it. It's data does not hold anything you would want to display anyway.
Simply put, if you want no duplicates, you need the DISTINCT keyword:
SELECT DISTINCT field1,
field2
FROM table1,
table2
WHERE table1.key1 = table2.key1;
This will select all Field1, Field2 combinations from the two tables. Due to the DISTINCT keyword, each line will only be in the result list once. Duplicates will not be in the result list.
SELECT DISTINCT duty_rota.duty_type AS DUTY_TYPE
FROM duty_rota,
duty_types,
contacts
WHERE duty_rota.duty_type = duty_types.duty_type
AND duty_rota.duty_officer = contacts.duty_id
AND sname IS NOT NULL
ORDER BY duty_date
You will only need to GROUP BY if you need further operations on the result set, like counting the duplicates. If all you need is "no duplicates", the DISTINCT keyword is exactly what you are looking for.
Edit:
In case I misread your question and you want to see only those, that are duplicates, you need to group and you need to filter based on the groups criteria. You can do that using the HAVING clause. It's kind of an additional WHERE of the groups criteria:
SELECT FIELD1, FIELD2, COUNT(*)
FROM TABLE1, TABLE2
WHERE TABLE1.KEY1 = TABLE2.KEY1
GROUPB BY FIELD1, FIELD2
HAVING COUNT(*) > 1

Get total row count in Entity Framework

I'm using Entity Framework to get the total row count for a table. I simply want the row count, no where clause or anything like that. The following query works, but is slow. It took about 7 seconds to return the count of 4475.
My guess here is that it's iterating through the entire table, just like how IEnumerable.Count() extension method works.
Is there a way I can get the total row count "quickly"? is there a better way?
public int GetLogCount()
{
using (var context = new my_db_entities(connection_string))
{
return context.Logs.Count();
}
}

You can even fire Raw SQL query using entity framework as below:
var sql = "SELECT COUNT(*) FROM dbo.Logs";
var total = context.Database.SqlQuery<int>(sql).Single();

That is the way to get your row count using Entity Framework. You will probably see faster performance on the second+ queries as there is an initialization cost the first time that you run it. (And it should be generating a Select Count() query here, not iterating through each row).
If you are interested in a faster way to get the raw row count in a table, then you might want to try using a mini ORM like Dapper or OrmLite.
You should also make sure that your table is properly defined (at the very least, that it has a Primary Key), as failure to do this can also affect the time to count rows in the table.

If you have access to do so, it would be much quicker to query the sys tables to pull this information.
E.g.
public Int64 GetLogCount()
{
var tableNameParam = new SqlParameter("TableName", "Logs");
var schemaNameParam = new SqlParameter("SchemaName", "dbo");
using (var context = new my_db_entities(connection_string))
{
var query = #"
SELECT ISNULL([RowCount],0)
FROM (
SELECT SchemaName,
TableName,
Sum(I.rowcnt) [RowCount]
FROM sysindexes I
JOIN sysobjects O (nolock) ON I.id = o.id AND o.type = 'U'
JOIN (
SELECT so.object_id,
ss.name as SchemaName,
so.name as TableName
FROM sys.objects SO (nolock)
JOIN sys.schemas SS (nolock) ON ss.schema_id = so.schema_id
) SN
ON SN.object_id = o.id
WHERE I.indid IN ( 0, 1 )
AND TableName = #TableName AND SchemaName = #SchemaName
GROUP BY
SchemaName, TableName
) A
";
return context.ExecuteStoreQuery<Int64>(query, tableNameParam, schemaNameParam).First();
}
}

Linq Query to Get Distinct Records from Two Tables

I have two Tables - tblExpenses and tblCategories as follows
tblExpenses
ID (PK),
Place,
DateSpent,
CategoryID (FK)
tblCategory
ID (PK),
Name
I tried various LINQ approaches to get all distinct records from the above two tables but not with much success. I tried using UNION and DISTINCT but it didnt work.
The above two tables are defined in my Model section of my project which in turn will create tables in SQLite. I need to retrieve all the distinct records from both the tables to display values in gridview.
Kindly provide me some inputs to accomplish this task. I did some research to find answer to this question but nothing seemed close to what I wanted. Excuse me if I duplicated this question.
Here is the UNION, DISTINCT approaches I tried:
DISTINCT # ==> Gives me Repetitive values
(from exp in db.Table<tblExpenses >()
from cat in db.Table<tblCategory>()
select new { exp.Id, exp.CategoryId, exp.DateSpent, exp.Expense, exp.Place, cat.Name }).Distinct();
UNION # ==> Got an error while using UNION

I think union already does the distict when you join the two tables you can try somethin like
var query=(from c in db.tblExpenses select c).Concat(from c in
db.tblCategory select c).Distinct().ToList();

You will always get DISTINCT records, since you are selecting the tblExpenses.ID too. (Unless there are multiple categories with the same ID. But that of course would be really, really bad design.)
Remember, when making a JOIN in LINQ, both field names and data types should be the same. Is the field tblExpenses.CategoryID a nullable field?
If so, try this JOIN:
db.Table<tblExpenses>()
.Join(db.Table<tblCategory>(),
exp => new { exp.CategoryId },
cat => new { CategoryId = (int?)cat.ID },
(exp, cat) => new {
exp.Id,
exp.CategoryId,
exp.DateSpent,
exp.Expense,
exp.Place,
cat.Name
})
.Select(j => new {
j.Id,
j.CategoryId,
j.DateSpent,
j.Expense,
j.Place,
j.Name
});

You can try this queries:
A SELECT DISTINCT query like this:
SELECT DISTINCT Name FROM tblCategory INNER JOIN tblExpenses ON tblCategory.categoryID = tblExpenses.categoryID;
limits the results to unique values in the output field. The query results are not updateable.
or
A SELECT DISTINCTROW query like this:
SELECT DISTINCTROW Name FROM tblCategory INNER JOIN tblExpenses ON tblCategory.categoryID = tblExpenses.categoryID;<br/><br/>
looks at the entire underlying tables, not just the output fields, to find unique rows.
reference:http://www.fmsinc.com/microsoftaccess/query/distinct_vs_distinctrow/unique_values_records.asp