SSIS - Dynamic Script Component - c#

I want to build a dynamic script component as source and destionation for multible tables. I just started by sample at Microsoft Page. It is working for specific table but now need to convert it to dynamic for all tables but need your helps.
First I thought to generate 'MyAddressOutputBuffer.Column' part dynamicly by below sql query.
SqlCommand cmdMeta = new SqlCommand(#"
select ColumnMapping = ' MyAddressOutputBuffer.Column'+cast(RANK() OVER(ORDER BY c.name)-1 as varchar(5))+' = sqlReader.GetValue(' + cast(RANK() OVER(ORDER BY c.name) - 1 as varchar(5)) + ').ToString();'
from sys.tables t
inner join sys.columns c on t.object_id = c.object_id
where t.name = 'Address'", sqlConn);
sqlReaderMeta = cmdMeta.ExecuteReader();
Above code give expected result but not sure how replace with hardcoded mapping with below part :
public override void CreateNewOutputRows()
{
while (sqlReader.Read())
{
{
MyAddressOutputBuffer.AddRow();
MyAddressOutputBuffer.AddressID = sqlReader.GetValue(0).ToString(); // I use getVAlue for all data types.
MyAddressOutputBuffer.City = sqlReader.GetValue(1).ToString();
}
}
}
Is there any way to say C# to run dynamic query which return from above sql instead of hardcoded?

Related

C# SqlCommandBuilder , CommandUpdate - how to write correct update based on select with outer join tables

I want is to update 2 fields: p.FlagaWaznosci and p.Notatka
My select looks like:
Select DISTINCT p.id,p.Model_Number,p.Product_Name,p.Website_Link,p.Entry_date,p.LastUpdate_date,p.PrzydzialRozmiarow_ID,p.FlagaWaznosci,p.Notatka,pr.NazwaRozmiarowki,wd.LINK_StockX
from Products p with(nolock)
left outer join Widok_Model_Sklep_Stockx_Linki wd with(nolock) on wd.Product_ID = p.id
left outer join PrzydzialRozmiarow pr with(nolock) on pr.id = p.PrzydzialRozmiarow_ID
inner join Shops s with(nolock) on s.ID = p.Shop_ID
There is just outer joins to get correct data that I need to be displayed in gridview. And now when values p.FlagaWaznosci or p.Notatka is changed I want to save update in my database.
I try to use
//loads dataand fill to gridview
DataTable WszystkieProduktyDlaDanegoSklepu;
SqlDataAdapter sda555123 = new SqlDataAdapter("here is my select", conn123);
sda555123.Fill(WszystkieProduktyDlaDanegoSklepu);
//later update table Prooducts and save changed on p.Notatka and p.FlagaWaznosci
cmdbl = new SqlCommandBuilder(sda555123);
cmdbl.ConflictOption = ConflictOption.OverwriteChanges;
sda555123.Update(WszystkieProduktyDlaDanegoSklepu);
But this way I have error
So I searched a lot and found: I have to write own CommandUpdate.
So ... sda555123.UpdateCommand and I don't have idea how can I write own update for it in update command.
The update in SQL Server should looks like:
Update Products
set FlagaWaznosci = #Flagawaznosci from my sda555123,
Notatka = #Notatka from my sda555123
where id = # p.ID from my sda555123
How my command update should looks like here?
EDIT 1 :
i try added : WszystkieProduktyDlaDanegoSklepu.PrimaryKey = new DataColumn[] { WszystkieProduktyDlaDanegoSklepu.Columns["id"] }
but nothing . Still this error.
I would solve the problem by changing the approach instead of mutating the update command of the SqlDataAdapter.
Given that Products.id in your query is unique within the result set:
1- Create a temporary table (local or global), having its columns same as the result of the query with id as primary key.
2- Insert data into the temporary table using your select statement.
3- DataAdatper.selectQuery.commandText is set to "select * from TempTable"
4- The update command is now based on a simple select statement, consequently any change in the datagridview/datatable can be updated to the temptable using dataadapter.update(datatable)
5- As for the final database update, you could use the below statement
Update Prd
set Prd.FlagaWaznosci = TempTable.FlagaWaznosci ,Prd.Notatka = TempTable.Notatka etc.. all the fields that need to be updated
from my Products as Prd
Inner Join TempTable on TempTable.id = Prd.id
Note that the update in (5) will affect all rows, even unchanged ones.
To address this issue you can proceed as below
1- Save changed ids in a list.
List<string> lst = new List<string>();
foreach(DataRow dr in datatable.GetChanges(DataRowState.Modified))
{
lst.add(dr["id"].ToString());
}
2- Convert your list to a string value to be concatenated with the query in (5)
String strchange = String.Join(",",lst); //will give you id1,id2,...
//The update query becomes
Update Prd
set Prd.FlagaWaznosci = TempTable.FlagaWaznosci ,Prd.Notatka =
TempTable.Notatka etc.. all the fields that need to be updated
from my Products as Prd
Inner Join TempTable on TempTable.id = Prd.id
Where Prd.id In ( strchange )
Kindly update your tables separately because in join you just seen two or more than two tables into one table form . but you cant do any crud operation on

Use local variables and multi query command with mysql and C#

I am trying to use a two line query to a mysql database like this with C#:
set #var = 1; select id from table where id = #var`;
Executing this in C# does not give any error, the reader has columns, but don't have rows in it, so, no data is retrieved.
Running the same queries in the workbench retrieve the expected data.
Running one query to calculate the variable and other to use it replaced works.
I want to use a variable to store references and use it to filter queries that provide an union.
SET #my_value = (select min(id) from (select id from datatable limit 2000) as a);
and a complex select query, that i can resume in:
select * from
datatable as a
left join (
(select databalt2 where id > #my_value) as ba union
(select datatable3 where id > #my_value) as bb) as b
on a.id = b.id
where a.id > #my_value;
The datareader should obtain the results, but instead reports that it has no rows, but has the correct amount of columns.
And, obviously, shows no results, where it should.
Edit 1: C# code
string query = #"
SET #`my_value` = (select min(id) from (select id from datatable limit 2000) as a);
select * from
datatable as a
left join (
(select databalt2 where id > #`my_value`) as ba union
(select datatable3 where id > #`my_value`) as bb) as b
on a.id = b.id
where a.id > #`my_value`;
";
try
{
MySqlCommand command = Connection.CreateCommand();
MySqlDataReader reader;
command.CommandTimeout = commandTimeout;
command.CommandText = query;
reader = command.ExecuteReader();
}
catch (Exception ex)
{
}
These queries using local variables are possible, the local variable has to be inside like #`var`, because it can be identifyied as a local parameter.
I was missing a in a local variable call. So ... a problem between the screen and the chair!

C# Dynamic full outer join on 2 datatables

I need to do a full outer join on 2 datatables dinamically, I don't know what columns are in the datatables but they will be the same in each table, I also only have the name of the column I need to do the join on in a variable. Is there a way of doing this?
What I need to do is join 2 datatables in a C# script. I'm using a Dataflow in an SSIS to get data from a couple of files, and at the end I need to compare the 2 final sets of data. I need to to this on whatever 2 datatables as long as they have the same columns, so I can't finish the process in an SSIS as I need to specify the columns.
The GetData() I just use it in case I need to compare 2 tables but donnesPROD and donnesDEV are filled from object variables in the SSIS.
Here's my code so far :
DataTable donnesPROD = GetData(connectionPROD, sql_request);
DataTable donnesDEV = GetData(connectionDEV, sql_request);
Here's the code for GetData :
DataTable GetData(string cs, string query)
{
OleDbConnection conn = new OleDbConnection(cs);
conn.Open();
OleDbCommand cmd = new OleDbCommand(query, conn);
DataTable dt = new DataTable();
dt.Load(cmd.ExecuteReader());
conn.Close();
return dt;
}
I have the list of the columns in another datatable, and I have the name of the primary key in a string variable key. From here I need to be able to do a fullouterjoin of donnesPROD and donnesDEV on key. Can this be done this way? Or is there a way of generating the script code it self dynamically and then execute it?
You have two options.
Conditional joins
If you don't know the specific column name, but you do have some idea what the column name might be, you could do a conditional join like this:
CREATE PROCEDURE ExampleDynamicJoin(#JoinColumn AS VarChar(40))
AS
BEGIN
SELECT *
FROM TableA
JOIN TableB ON (#JoinColumn = 'ColumnA' AND TableA.ColumnA = TableB.ColumnA)
OR (#JoinColumn = 'ColumnB' AND TableA.ColumnB = TableB.ColumnB)
OR (#JoinColumn = 'ColumnC' AND TableA.ColumnC = TableB.ColumnC)
END
You may not get the best performance out of this (the conditional joins will confuse the query engine and it may not pick the best index, if it picks one at all). If the table is very large you could also do something like this. It is a bit painful-looking but will get better performance:
CREATE PROCEDURE ExampleDynamicJoin(#JoinColumn AS VarChar(40))
AS
BEGIN
IF (#JoinColumn = 'ColumnA') BEGIN
SELECT *
FROM TableA
JOIN TableB ON TableA.ColumnA = TableB.ColumnA
END
IF (#JoinColumn = 'ColumnB') BEGIN
SELECT *
FROM TableA
JOIN TableB ON TableA.ColumnB = TableB.ColumnB
END
IF (#JoinColumn = 'ColumnC') BEGIN
SELECT *
FROM TableA
JOIN TableB ON TableA.ColumnC = TableB.ColumnC
END
END
If TableA or TableA are part of a larger query, and you'd end up duplicating tons of SQL, you could always extract the resultset for just TableA and TableB into a temporary table, then use the temporary table in the larger query.
Dynamic SQL
If you don't have the foggiest about the column name and there are tons of possibilities, you could construct the SQL as a string and join that way. You should validate the column name that is passed in; not only will that make sure the column actually exists, but it will prevent the dynamic SQL from being constructed when #JoinColumn contains an injection attack, since legal column names do not contain SQL statements. Example:
CREATE PROCEDURE ExampleDynamicJoin(#JoinColumn AS VarChar(40))
AS
BEGIN
DECLARE #Sql AS VarChar(MAX)
IF NOT EXISTS
(
SELECT 0
FROM syscolumns c
JOIN sysobjects o ON o.id = c.id
WHERE o.Name = 'TableA'
AND c.Name = #JoinColumn
)
RAISERROR (15600,-1,-1, 'ExampleDynamicJoin'); //Throw error if column doesn't exist
SET #Sql =
'SELECT *
FROM TableA
JOIN TableB ON TableA.' + #JoinColumn + ' = TableB.' + #JoinColumn
sp_ExecuteSql #Sql
END
Or, if you don't use stored procedures,
DataTable ExampleDynamicJoin(string joinColumn)
{
if (!ValidateColumn(joinColumn)) throw new ArgumentException();
var sql = String.Format(
#"SELECT *
FROM TableA
JOIN TableB ON TableA.{0} = TableB.{0}",
joinColumn
);
using (var connection = GetConnectionFromSomewhere())
{
using (var cmd = new SqlCommand
{
CommandText = sql,
CommandType = CommandType.Text,
Connection = connection
})
{
var reader = cmd.ExecuteReader();
var table = new DataTable();
table.Load(reader);
return table;
}
}
}
When using dynamic SQL you should always use parameters if possible. But you can't use parameters as a column name, so in this case you have to concatenate. When concatenating, ALWAYS white list the inputs. That is why I included a function named ValidateColumn which could look like this:
bool ValidateColumn(string columnName)
{
switch columnName.ToUpper()
{
case "COLUMNA":
case "COLUMNB":
case "COLUMNC":
return true;
default:
return false;
}
}

C# query using a generic Type

I am trying to make a generic function to do a database call (see code below). I have put the function in a separate solution, so that i can use it in different projects.
the line:
var data = d.Database.SqlQuery<T> (sql).First();
gives me the error:
Invalid object name 'VM_MailData'
(VM_MailData is the type I add as generic type T)
public static void ProcessData<T>(string Group, int Id)
{
string ConnectionString = "SomeConnectionStringName";
string sql = "select top 1 * from " + (typeof (T).Name) + " where " + Group + "Id = " + Id + ";";
DbContext d = new DbContext(ConnectionString);
var data = d.Database.SqlQuery<T> (sql).First();
//Do some stuff with the data...
html = "some tekst...";
foreach (var sourceProperty in data.GetType().GetProperties())
{
html = html.Replace("{#" + sourceProperty.Name + "#}", sourceProperty.GetValue(data, new object[] { }) == null ? "" : sourceProperty.GetValue(data, new object[] { }).ToString());
//enter code here
}
}
You most probably need to call ProcessData<T>(string Group, int Id) with some base type of VM_MailData. I assume that VM stands for "view model" and you have something like
public class MailData
{
}
public class VM_MailData : MailData
{
}
Where MailData class is actually the one that has a corresponding table.
So instead of calling ProcessData<VM_MailData >("some group", 1) you need to call ProcessData<MailData>("some group", 1).
P.S. You really should use parameterized queries to avoid SQL Injection!
I can see where you're heading :) so...
You have to create a map (dictionary) that maps from Type to appropriate table name. You can built that using some kind of convention based on the name of the type or the tables.
Use those to get available tables and views in the dabatase
SELECT SCHEMA_NAME(schema_id) as SchemaName, name as Name FROM sys.tables
SELECT SCHEMA_NAME(schema_id) as SchemaName, name as Name FROM sys.views
You can use the following script to get the primary key column of the table.
DECLARE #origin_table_name AS VARCHAR(50)
SET #origin_table_name = 'Your_table_name_goes_here'
SELECT
s.name AS TABLE_SCHEMA
, t.name AS TABLE_NAME
, c.name AS COLUMN_NAME
, k.name AS CONSTRAINT_NAME
, ic.key_ordinal AS ORDINAL_POSITION
FROM
sys.key_constraints AS k
JOIN sys.tables AS t ON t.object_id = k.parent_object_id
JOIN sys.schemas AS s ON s.schema_id = t.schema_id
JOIN sys.index_columns AS ic ON ic.object_id = t.object_id
AND ic.index_id = k.unique_index_id
JOIN sys.columns AS c ON c.object_id = t.object_id
AND c.column_id = ic.column_id
WHERE
k.type = 'PK'
AND t.name = #origin_table_name
Good luck

Get total row count in Entity Framework

I'm using Entity Framework to get the total row count for a table. I simply want the row count, no where clause or anything like that. The following query works, but is slow. It took about 7 seconds to return the count of 4475.
My guess here is that it's iterating through the entire table, just like how IEnumerable.Count() extension method works.
Is there a way I can get the total row count "quickly"? is there a better way?
public int GetLogCount()
{
using (var context = new my_db_entities(connection_string))
{
return context.Logs.Count();
}
}
You can even fire Raw SQL query using entity framework as below:
var sql = "SELECT COUNT(*) FROM dbo.Logs";
var total = context.Database.SqlQuery<int>(sql).Single();
That is the way to get your row count using Entity Framework. You will probably see faster performance on the second+ queries as there is an initialization cost the first time that you run it. (And it should be generating a Select Count() query here, not iterating through each row).
If you are interested in a faster way to get the raw row count in a table, then you might want to try using a mini ORM like Dapper or OrmLite.
You should also make sure that your table is properly defined (at the very least, that it has a Primary Key), as failure to do this can also affect the time to count rows in the table.
If you have access to do so, it would be much quicker to query the sys tables to pull this information.
E.g.
public Int64 GetLogCount()
{
var tableNameParam = new SqlParameter("TableName", "Logs");
var schemaNameParam = new SqlParameter("SchemaName", "dbo");
using (var context = new my_db_entities(connection_string))
{
var query = #"
SELECT ISNULL([RowCount],0)
FROM (
SELECT SchemaName,
TableName,
Sum(I.rowcnt) [RowCount]
FROM sysindexes I
JOIN sysobjects O (nolock) ON I.id = o.id AND o.type = 'U'
JOIN (
SELECT so.object_id,
ss.name as SchemaName,
so.name as TableName
FROM sys.objects SO (nolock)
JOIN sys.schemas SS (nolock) ON ss.schema_id = so.schema_id
) SN
ON SN.object_id = o.id
WHERE I.indid IN ( 0, 1 )
AND TableName = #TableName AND SchemaName = #SchemaName
GROUP BY
SchemaName, TableName
) A
";
return context.ExecuteStoreQuery<Int64>(query, tableNameParam, schemaNameParam).First();
}
}

Categories

Resources