I have a DataTable with a lot of rows (Over a hundred million) and am writing an application that needs to insert into that table.
I will be using OleDbDataAdapter for the job and I am puzzled whats the best way to do this. I only need to insert into this enormous table, however I don't want to hard code the insert statement into application.
I figured I could use
OleDbDataAdapter adapter = new OleDbDataAdapter();
adapter.SelectCommand = new OleDbCommand("select * from table_name");
OleDbCommandBuilder cb = new OleDbCommandBuilder(adapter);
...
adapter.Fill(data_set_name, "Table_name");
But this would be really bad since I don't need/want the data and the memory usage would be awful. So I was interesting if I could alter SelectCommand with TOP? It would look like so:
adapter.SelectCommand = new OleDbCommand("select TOP 1 * from table_name");
Now the Fill command would be really fast and I would have the data I needed for all the future insert statements. I could add rows to datatable and then just call
adapter.Update(data_set_name, "Table_name");
Would this work? And is this a valid / recommended way of doing this? It is really important that the application is fast and uses only the necessary resources. Is there a better way of doing this?
Thank you for your input!
If you don't need the data you can change the select command to
SELECT * FROM Table_Name WHERE 1=2
Then you won't get any rows back
IMO, the best way would be to:
Use the OleDbDataAdapter.FillSchema(data_set_name, SchemaType.Source) method to create the DataTable with a structure mapped from the datasource. You are basically trying to do the same thing by pulling a single row in your Select statement. Your Select statement in this case could remain "select * from table_name". I believe that you do not need to call the OleDbDataAdapter.Fill method now.
Instead of using a CommandBuilder, create your InsertCommand statement yourself.
Related
For example, I have a script that contains
insert into #temp from TABLES
insert into #temp1 from #temp
select * from #temp
I already have a C# MVC application. I just want to be able to execute the script above to display the #temp table on my front-end. I have seen the function ExecuteNonQuery(), however, it only shows the rows affected. Is there anyone that can point me a guide to solve this issue?
EDIT: I just want to run the script above and obtain the results, i don't want to rewrite the script in a framework.
Here is a sample for ExecuteNonQuery:
private void ExecuteNonQuery(string queryString, string connectionString)
{
using (SqlConnection connection = new SqlConnection(
connectionString))
{
SqlCommand command = new SqlCommand(queryString, connection);
command.Connection.Open();
command.ExecuteNonQuery();
}
}
From MSDN
And you can call your inserts like this: ExecuteNonQuery("insert into #temp from TABLES", connectionString);
For reading you can use ExecuteReader (MSDN) But if your table has many columns, this can be tedious to display all the columns. I prefer to use micro ORM (or data mapper otherwise) like Dapper - it is easy to use, minimum set up and with plenty of documentation.
Some people advise to use Entity Framework, but for this case it looks like will be an overkill. Also it has a steep learning curve.
If you want to read data from temporary table so use the ExecuteReader() and refer the following code.
using(SqlDataReader rdr = cmd.ExecuteReader())
{
while (rdr.Read())
{
var myString = rdr.GetString(0); //The 0 stands for "the 0'th column", so the
first column of the result.
// Do somthing with this rows string, for example to put them in to a list
listDeclaredElsewhere.Add(myString);
}
}
For select you have can use DataSet object and using DataAdapter fill data to DataSet
DataSet ds=new DataSet();
SqlDataAdapter da=new SqlDataAdapter(commandObject);
da.fill(ds);
DataSet ds=new DataSet();
SqlDataAdapter da=new SqlDataAdapter(commandObject);
da.fill(ds);
yes, that would work as well. but be aware that you are working with a DataSet (not DataTable on purpose?), and both DataSet and it's matching DataTable(s) are all dynamic objects without type safety (i.e. you are working with ds.Tables[0].Rows[0][2] as string; ).
good luck with the runtime exceptions.
I know this error/question has been posted a lot. Iv been researching for a while now and still cannot figure how to properly update my database with changes made in my DataTable.
I have a DataTable filled with information from 2 different tables in my database. Changes are only being made to the columns that come from 1 table though. The second table I need to JOIN for calculations.
So when I try to update my database, changes will only be made to the 1 table. I am still need to ado.net, so im having a hard time applying the correct way to apply these changes based on information from MSDN and StackOverflow.
This is the code I have:
using (SqlCommand cmd = new SqlCommand("SELECT INCIDENT_NO, report_dt,MIN(theStart) as The_Start, Resolved_DT, Resolution_TIME, resolution_met,response_met,max_response_time,max_resolution_time,impact FROM TICKET_REPORT JOIN TimeTracker ON Incident_NO=theTask WHERE Resolved_DT != '' GROUP BY INCIDENT_NO,report_dt,Resolved_DT,Resolution_Time, resolution_met,response_met,max_response_time,max_resolution_time,impact", sqlConn))
{
var calc = new Calculation(holidays, new OpenHours("09:00;17:00"));
SqlDataAdapter da = new SqlDataAdapter(cmd);
dtResolution.Columns.Clear();
dtResolution.Clear();
da.Fill(dtResolution);
//-------Making calculation changes to columns from one table in datatable here---------
try
{
da.Update(dtResolution);
Console.WriteLine("Update successful");
}
catch(Exception ex)
{
Console.WriteLine("Updated failed");
Console.WriteLine(ex.Message);
}
da.Dispose();
}
I leaned that CommandBuilder will create the update command for me, but MSDN says I can only use CommandBuilder if my SELECT statement only includes 1 table, I have 2 two.
Other StackOverflow answers are saying that if im using a JOIN, im on my own.
How can I properly accomplish this update? Can I do this in C#? Should I pass the data table to a stored procedure in sqlserver instead?
I found it easier to just create a merge statement inside of a stored procedure, and just pass the datatable to the stored procedure.
My code is construct to read data in a datagridView named (dg) from my database.
Its actually work well whit a SqlDataAdapter.
First Is it a good idea to change my SqlDataAdapter for a SqlCommand ?
If YES
I want to use this for change my SqlDataAdapter.
//SqlCommand cmd = new SqlCommand("Command String", con);
//SqlDataReader readdata;
CODE
SqlConnection con = new SqlConnection(dc.Con);
SqlDataAdapter da = new SqlDataAdapter();
con.Open();
da.SelectCommand = new SqlCommand("SELECT * FROM tblContacts", con);
DataTable dt = new DataTable();
da.Fill(dt);
con.Close();
dg.DataSource = dt;
I typically use the DataAdapter for data access when I need to do data-binding to controls. It's very convenient and efficient in those scenarios. Otherwise, I use the Command objects directly. Performance-wise, I'm inclined to agree w/ punzki. There shouldn't be much difference between the two.
http://msforums.ph/forums/p/9057/9057.aspx
Actually, from what I remember, SqlDataAdapter uses SqlDataReader to retrieve records. So It's always good to use SQLDataReader when you're going to just retrieve data from the backend. But if you're going to retrieve data and then update (insert, update, delete) data later on, then it's better to use SqlDataAdapter. I think it's more efficient that way.
http://msforums.ph/forums/t/29256.aspx
There IS an effect on performance.
SqlDataReader is no doubt faster than a SqlDataAdapter as the DataReader reads data in a forward only mode and you can get a specific type of value returned back to you, such as a string or int etc... however with the SqlDataAdapter, it will fill a datatable or dataset will records it finds in your select statement, taking with it the correct value type for the columns and is a disconnected representation of in memory database and is ideal and easier to use if you are going to show large amounts of records to a binding source, as with a SqlDataReader, it is not possible but to only obtain a value for a column you specify per row.
The SqlDataAdapter also allows you to Update, Delete or Insert rows into the Dataset/DataTable which is an advantage and will execute the appropriate command, if you implemented it correctly, based on how the rows were modified in the Dataset/DataTable.
SqlDataAdapter is expensive compared to a fast forward read on the SqlDataReader, and has more advantages but entirely depends on your solution and what you require.
You are stating that you are going to show alot of records, whilst that is all very well, it would be even better for the benefit of the performance and memory usage to only obtain records that you require to be shown and a SqlDataAdapter would be suitable for this also but still you are required to select records which are the ones you will most likely show to the user, either by input search criteria, or perhaps by paging.
http://social.msdn.microsoft.com/Forums/en-US/adodotnetdataproviders/thread/c2d762fd-f4a0-4875-8bb8-42f7480e97c8/
I have a DataSet that I fill with values from a XML-file. I would like to insert the values then into a SQL table. How do I do that?
Here is how I fill my DataSet:
DataSet dataset = new DataSet();
dataset.ReadXml(xmlfile);
customer.DataSource = dataset.Tables[0];
If all you're ever going to do is inserts, then the simplest way is to just loop through the rows in the DataTable and create and execute a DbCommand for each row. The specific syntax to use depends on the kind of database you're using; for SQL Server, it might look like this:
string sql = "INSERT INTO T (A, B, C) VALUES (#A, #B, #C)";
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
foreach (DataRow r in myTable.Rows)
{
SqlCommand cmd = conn.CreateCommand();
cmd.CommandText = sql;
cmd.Parameters.AddWithValue("#A", r["A"]);
cmd.Parameters.AddWithValue("#B", r["B"]);
cmd.Parameters.AddWithValue("#C", r["C"]);
cmd.ExecuteNonQuery();
}
}
This glosses over many, many possible complicating factors, e.g. exception handling, filtering out rows with the wrong DataRowState, calling AcceptChanges on the DataRow, updating identity columns with the value assigned by the database, and so on. There's a lot to understand in ADO.NET, and that stuff's in there for a reason. But if all you're going to do is insert rows, and the DataColumn objects in your DataTable have the right names and types and proper values for Size and AllowDbNull, and there are no foreign-key relations involved, and you're not going to encounter duplicate primary keys with existing data in the table, the above should work.
My suggestion would be to create a stored procedure for your insert and then create a method that traverses the data set, and sets the parameters of your stored proc and then executes it.
I think you would need to loop through the dataset and perform inserts.
You can obviously do this with a sproc, and you just pass along the parameters, or with LINQ.
Either way should work.
This completely depends on you existing Data access Layer.
If one doesn't exist i would look more towards LinqtoSQL rather than using Stored procedures, Stored procedures are more difficult to manage
Simple,
Just use the DataSet.ReadXml() method.
http://msdn.microsoft.com/en-us/library/system.data.dataset.readxml.aspx
Now if your xml is different from your schema, then your most likely going to have to loop and fill your schema specific dataset, then save as needed.
I know this might be a bit awkward but I am trying to modify the order of certain columns in a MS Access database in C# with OLE DB. How can I commit a certain change in the order of the columns of a datatable in a dataset? If that is not possible, how can I reorder columns of database table by using a dataset?
Here is a sample of what I have (in C#):
command.Connection = conn;
command.CommandText = tableName;
command.CommandType = CommandType.TableDirect;
adapter = new OleDbDataAdapter(command);
dataset = new DataSet(tableName);
adapter.Fill(dataset, tableName);
dataset.Tables[0].Columns[dataset.Tables[0].Columns.Count-1].SetOrdinal(CB_PositionCol.SelectedIndex);
dataset.Tables[0].AcceptChanges();
The AcceptChanges does not seem to work for what I wish to do since I believe it only commits changes in DataRows...
Thank you for any help!
Use an ALTER TABLE statement with an OleDbCommand. DataSet, DataTable and DataTableAdapters are meant to be transparent to the underlying structure of the data. You can actually use Table Adapters to transform data from two different structures.
http://www.functionx.com/vbnet/oledb/Lesson02.htm