I need to transform one single column values(columnA) from TABLE A in one database to another column B in TABLE B in some other database.
When i transfer columnA for example has (Employee names) but in the destination TABLE B it should be the (Employee ID). I have a lookup table which has the employeeID for the matching employee name.
Conditions:
I need only one single column gets updated in the destination TABLE B. without affecting any other columns.(Is it possible to insert in that way ? i know insertion involves all the columns to be involved)
I know this can be done in SSIS and i have created
* source oledb
*Lookup Transform
*Destination OLEDB
But the problem is at the destination output TABLE B the lookup transform is inserting NULL values to the unmatched column.
Can someone please guide me which is the best way to do this ?
Similarly i need to include various flows for the destination table from various databases.
You need to use a SQL Command as your destination, and have it do an UPDATE command to update your destination table to update the value of the EmployeeID column with the ID from the lookup table. If they are both on the same database, you can do a join in the command, otherwise you should use a Lookup transformation to get the ID based on the name.
Related
I was given a task to insert over 1000 rows with 4 columns. The table in question does not have a PK or FK. Let's say it contains columns ID, CustomerNo, Description. The records needed to be inserted can have the same CustomerNo and Description values.
I read about importing data to a temporary table, comparing it with the real table, removing duplicates, and moving new records to the real table.
I also could have 1000 queries that check if such a record already exists and insert data if it does not. But I'm too ashamed to try that out for obvious reasons.
I'm not expecting any specific code, because I did not give any specific details. What I'm hoping for is some pseudocode or general advice for completing such tasks. I can't wait to give some upvotes!
So the idea is, you don't want to insert an entry if there's already an entry with the same ID?
If so, after you import your data into a temporary table, you can accomplish what you're looking for in the where clause of a select statement:
insert into table
select ID, CustomerNo, Description from #data_source
where (#data_source.ID not in (select table.ID from table))
I would suggest to you to load the data into a temp table or variable table. Then you can do a "Select Into" using the distinct key word which will removed the duplicated records.
you will always need to read the target table, unless you bulk load the target table into a temp table(in this point you will have two temp tables) compare both, eliminate duplicates and then insert in target table, but even this is not accurate, because you can have a new insert in the target table while you do this.
I am currently learning the Azure storage table. I did some learning from http://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-table-storage-20/
It seems a key/value pair storage. It has partition key, row key, and include up to 252 properties.
From the link above, it can only filter the query result by row key. Can we query the result by filtering one of the property.
My project is to migrate our current sql database to the Azure storage table.
Current SQL database has three columns
Column a, Column b, Column c
Currently, our function need to support to query both by filtering column a and column b. When we migrate to Azure storage table, we need to keep the function
I think we only can set one of the two columns as the row key and the others should be in the properties of the Azure table. Also I don't think the column b can be the row key since it can not uniquely identify a row in our previous sql table (My understanding is correct?)
So can I query by both Column a and b if I only set column a as the row key and column b as a property when I migrate the table to Azure storage. If not, is there any suggestions to solve it?
Thanks so much
Yes, you can filter by any attribute. However please keep in mind that only PartitionKey and RowKey are indexed thus any query operation which doesn't include at least PartitionKey will result in full table scan. This will become an issue if your table contains lots of entities.
I'm trying to store a DataTable into a single column in a SQL Server table. The idea behind this is a user runs a SQL command and the returned output is stored into a datatable, then I want that datatable to be stored into a SQL Server logging table. Later on I want to be able to retrieve that entire datatable back for displaying on a logging aspx page.
Currently I'm just storing it as a big string but that doesn't give me column headers and the formatting is kinda funky as well as being inefficient.
TIA
I would probably convert the datatable to XML and store it into an XML field type if I was going to do what you are trying to do.
Hello you can try with WriteXml
this link give you sample interessant : http://msdn.microsoft.com/fr-fr/library/system.data.datatable.writexml.aspx
Another Idea is to create two tables in your database. It is slightly complex.
One Table contains two columns, Let name the table PTable.
Columns:
ID and ColumnID
ID is the primary key and ColumnID contains the name of your column in datatable
After creating this table create another table. It will consists of three fields. Let name it STable. This table stores the columns of you datatable.
Columns:
stblID, PtblID and PtColumnID
StbID is the primary key in this table, PtblID is the Primary key of PTable and PtColumnID is the ColumnID of PTable. This table stores the rows of table. Now store the data in this table and receive the data when you need it.
and the simplest idea is to create a table in your datbabase having an xml column and store your datatable as an xml.
I am doing a conversion with SqlBulkCopy. I currently have an IList collection of classes which basically i can do a conversion to a DataTable for use with SqlBulkCopy.
Problem is that I can have 3 records with the same ID.
Let me explain .. here are 3 records
ID Name Address
1 Scott London
1 Mark London
1 Manchester
Basically i need to insert them sequentially .. hence i insert record 1 if it doesn't exist, then the next record if it exists i need to update the record rather than insert a new 1 (notice the id is still 1) so in the case of the second record i replace both columns Name And Address on ID 1.
Finally on the 3rd record you notice that Name doesn't exist but its ID 1 and has an address of manchester so i need to update the record but NOT CHANGING Name but updating Manchester.. hence the 3rd record would make the id1 =
ID Name Address
1 Mark Manchester
Any ideas how i can do this? i am at a loss.
Thanks.
EDIT
Ok a little update. I will manage and merge my records before using SQLbulkCopy. Is it possible to get a list of what succeeded and what failed... or is it a case of ALL or nothing? I presume there is no other alternative to SQLbulkCopy but to do updates?
it would be ideal to be able to Insert everything and the ones that failed are inserted into a temp table ... hence i only need to worry about correcting the ones in my failed table as the others i know are all OK
Since you need to process that data into a DataTable anyway (unless you are writing a custom IDataReader), you should merge the records before giving them to SqlBulkCopy; for example (in pseudo code):
/* create empty data-table */
foreach(row in list) {
var row = /* try to get exsiting row from data-table based on id */
if(row == null) { row = /* create and append row to data-table */ }
else { merge non-trivial properties into existing row */
}
then pass the DataTable to SqlBulkCopy once you have the desired data.
Re the edit; in that scenario, I would upload to a staging table (just a regular table that has a schema like the uploaded data, but typically no foreign keys etc), then use regular TSQL to move the data into the transactional tables. In addition to full TSQL support this also allows better logging of operations. In particular, perhaps look at the OUTPUT clause of INSERT which can help complex bulk operations.
You can't do updates with bulk copy (bulk insert), only insert. Hence the name.
You need to fix the data before you insert them. If this means you have updates to pre-existing rows, you can't insert those as that will generate the key conflict.
You can either bulk insert into a temporary table, and run the appropriate insert or update statements, only insert the new rows and issue update statements for the rest, or delete the pre-existing rows after fetching them and fixing the data before reinserting.
But there's no way to persuade bulk copy to update an existing row.
Is there a simple solution for duplicating table rows in SQL Server as well as all table rows with foreign keys pointing to the cloned table row? I've got a "master" table and a bunch of "child" tables which have a foreign key into the ID of the master table. I need to not only create a perfect copy of the master table, but clone each and every child table referencing the master table. Is there a simpler way to do this than creating a new row in the master table, copying in the information from the row to be cloned, then going through each child table and doing the same with each row pointing to the cloned row in the master table?
I'm using a SQL Server 2005 Database accessed through C# ASP.net MVC 1.0.
If by "simple" you mean is there is a procedure that can be called to do it, no there is not. However, you can use the INFORMATION_SCHEMA views such as INFORMATION_SCHEMA.COLUMNS and INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS to query for the columns in a table or the list of related tables and dynamically build your INSERT statements to copy one row to another. Of course, this does not account for other uniqueness constraints that might be on the tables (e.g. a table with a Name column which requires that the values be unique).