I am trying to set up a synchronization routine in C# to send data from a ms access database to a sql server. MS Access is not my choice it's just the way it is.
I am able to query the MS Access database and get OleDbDataReader record set. I could potentially read each individual record and insert it onto SQL Server but it seems so wasteful.
Is there a better way to do this. I know I could do it in MS Access linking to sql server and perform the update easy but this is for end users and I don't want them messing with access.
EDIT:
Just looking at SqlBulkCopy I think that may be the answer if I get my results into DataRow[]
I found a solution in .NET that I am very happy with. It allows me to give the access to the sync routine to any user within my program. It involves the SQLBulkCopy class.
private static void BulkCopyAccessToSQLServer
(CommandType commandType, string sql, string destinationTable)
{
using (DataTable dt = new DataTable())
{
using (OleDbConnection conn = new OleDbConnection(Settings.Default.CurriculumConnectionString))
using (OleDbCommand cmd = new OleDbCommand(sql, conn))
using (OleDbDataAdapter adapter = new OleDbDataAdapter(cmd))
{
cmd.CommandType = commandType;
cmd.Connection.Open();
adapter.SelectCommand.CommandTimeout = 240;
adapter.Fill(dt);
adapter.Dispose();
}
using (SqlConnection conn2 = new SqlConnection(Settings.Default.qlsdat_extensionsConnectionString))
{
conn2.Open();
using (SqlBulkCopy copy = new SqlBulkCopy(conn2))
{
copy.DestinationTableName = destinationTable;
copy.BatchSize = 1000;
copy.BulkCopyTimeout = 240;
copy.WriteToServer(dt);
copy.NotifyAfter = 1000;
}
}
}
}
Basically this puts the data from MS Access into a DataTable it then uses the second connection conn2 and the SqlBulkCopy class to send the data from this DataTable to the SQL Server. It's probably not the best code but should give anyone reading this the idea.
You should harness the power of SET based queries over RBAR efforts.
Look into a SSIS solution to synchronize the data and then schedule the package to run at regular intervals using SQL Server Agent.
You can call an SSIS package from the command line so you can effectively do it from MS Access or from C#.
Also, the SQL Server, the MS Access DB and the SSIS package do not have to be on the same machine. As long as your calling program can see the SSIS package, and the package can connect to the SQL Server and the MS Access DB, you can transfer data from one place to another.
It sounds like what you are doing is ETL. There are several tools that are built to do this and to me, there is little reason to reinvent the functionality. You have SQL Server, therefore you have SSIS. It has a ton of tools for automated transformations, cleanups, lookups, etc. that you can use out of the box.
Unless this is a real cut-and-dry data load and there is absolutely no scope for the complexity of the upload to increase later on (yeah, right!) I would go with a tried and tested ETL tool.
If SQL Server Integration Services isn't an option, you could write out to a temporary text file the data that you read from Access and then call bcp.exe to load it to the database.
I have done something like this before.
I used
OleDbConnection aConnection = new OleDbConnection(String.Format("Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0}", fileName));
aConnection.Open();
to open the access db. Then
OleDbCommand aCommand = new OleDbCommand(String.Format("select * from {0}", accessTable), aConnection);
OleDbDataReader aReader = aCommand.ExecuteReader();
to execute the read from the table. Then
int fieldCount = aReader.FieldCount;
to get the field count
while (aReader.Read())
to loop the records and
object[] values = new object[fieldCount];
aReader.GetValues(values);
to retrieve the values.
There are several ways to sync but it can give a problem when you change a field name in sql server or add a new column or delete. The best option would be:
Create connections for sql server and oledb.
Write custom query to fetch record from one connection and save it to another.
Before executing make sure to program in a way that you update all table definitions.
In my case this helped in a way because the load on sql server became down.
can you not transfer Access file to the server and delete it once sync is complete?
You can create windows service for that..
Related
I have a local SQL Server 2017 database, and I need to copy two tables to an Azure SQL Server database; one table has over 100 million rows of data including a "geography" type column. How do I do that?
I am right now running a bulk copy:
using (SqlConnection streamsConnection = new SqlConnection(streamsConnectionString))
{
streamsConnection.Open();
using (SqlConnection cloudConnection = new SqlConnection(cloudConnectionString))
{
cloudConnection.Open();
using (SqlCommand cmd = streamsConnection.CreateCommand())
using (SqlBulkCopy bcp = new SqlBulkCopy(cloudConnection))
{
bcp.DestinationTableName = "GroundDataNodes";
bcp.BatchSize = 200000;
bcp.BulkCopyTimeout = 1200;
bcp.NotifyAfter = 100000;
bcp.SqlRowsCopied += new SqlRowsCopiedEventHandler(s_SqlRowsCopied);
cmd.CommandText = "SELECT [Id],[nodeid],[latlon],[type],[label],[code],[lat],[lon]FROM[dbo].[GroundDataNodes]";
using (SqlDataReader reader = cmd.ExecuteReader())
{
bcp.WriteToServer(reader);
}
Console.WriteLine("Finished!");
Console.ReadLine();
}
}
}
But I'm quite new to the bulk load side and am wondering how I can improve this so it doesn't take weeks to run...
Give Azure Database Migration Services a Try. I Migrated an on-premise SQL Server with 10 Million Rows in a Days Time. But of course it also largely depends on your bandwidth.
Also:
Use the multi CPU General Purpose Pricing Tier when you create your service instance to allow the service to take advantage of multiple vCPUs for parallelization and faster data transfer.
Temporarily scale up your Azure SQL Database target instance to the Premium tier SKU during the data migration operation to minimize Azure SQL Database throttling that may impact data transfer activities when using lower-level SKUs.
so I have tried all sorts of upload stuff like the bulk upload, migration, backup etc but all had the same problem and that was my upload speed isn't up to it. It would work but take days to run. So I decided to write a server side bit of code to just populate it from there directly onto the database therefore taking my upload speeds out of the equation. I imagine if I had better upload speeds the migration tool would have worked fine even with the Geographic field etc. not quick but it would work.
Right, I have been tasked with developing a new application in MVC3 that unfortunately has to integrate very slightly with a classic asp web site. This won't be forever as the old site will get an update at some point, but not yet. In the mean time however the new MVC3 application will need a little bit of access to the database for the old site, which is a old MS Access .mdb whereas the new app will be using sql server 2008.
I would greatly appreciate it if someone could give me some examples of how to connect to the access db, aswell as how to execute sql queries (i am fine writing the sql, just got no idea how to execute against the database from my mvc3 app).
thanks in advance
EDIT: I've not got much experience with the old site, but it appears to use the JET adaptor if that helps! ;-)
Your question requires an answer too extensive to be given in detail
I will give you a check list of things and class to research
Define the connection string used to reach your database [see
here]
Create and open the OleDbConnection
Define your OleDbCommand and the command text to be executed
Create and use an OleDbDataReader to read your data line by line
Create and use an OleDbDataAdapter to read your data and load a
DataSet or DataTable
Now don't forget to close your connection and use parametrized query
string connectionString = Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\mydatabase.mdb;Jet OLEDB:Database Password=MyDbPassword;
public void InsertRow(string connectionString, string insertSQL)
{
using (OleDbConnection connection = new OleDbConnection(connectionString))
{
// The insertSQL string contains a SQL statement that
// inserts a new row in the source table.
OleDbCommand command = new OleDbCommand(insertSQL);
// Set the Connection to the new OleDbConnection.
command.Connection = connection;
// Open the connection and execute the insert command.
try
{
connection.Open();
command.ExecuteNonQuery();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
// The connection is automatically closed when the
// code exits the using block.
}
}
Firstly, For Those who would like to ask WHY on earth am I DOWNSIZING from SQL SERVER to ACCESS, let me tell you the scenario.
There are some PC's with very low configuration, (256 MB RAM, 2GHzProcessor) I cannot install SQL Server. Hence I want major operations to carry out on SQL server and some data retrieving and auditing work to be done on Slower machine.
So here we go:
I want to copy table from SQL Server to MS Access 2007. I tried:
1)Connect to sql server, fill a datatable object by reading table from sql server.
2) Create a connection to MS Access, and use Dataadapter.Update method to update table to MS Access database.
However 2nd step is not working although its not throwing any error. Here is my code:
SqlConnection cnn = new SqlConnection(#"initial catalog=DBTempleERM;user id=aditya;password=Aditya_ravi$;Data Source=adityalappy\sqlexpress");
SqlCommand cmd = new SqlCommand("SELECT * FROM donationdetails", cnn);
cnn.Open();
System.Data.SqlClient.SqlDataAdapter sDA = new SqlDataAdapter(cmd);
DataTable donationdetails = new DataTable();
sDA.MissingSchemaAction = MissingSchemaAction.AddWithKey;
sDA.Fill(donationdetails);
MessageBox.Show(donationdetails.Rows.Count.ToString());
OleDbConnection oleConn = new OleDbConnection(#"Provider=Microsoft.ACE.OLEDB.12.0;Jet OLEDB:Database Password=Aditya_ravi$;Data Source=C:\dbt.accdb");
oleConn.Open();
OleDbCommand oleComm = new OleDbCommand();
OleDbDataAdapter oDA = new OleDbDataAdapter(oleComm);
OleDbCommandBuilder oCb = new OleDbCommandBuilder(oDA);
oDA.Update(donationdetails);
No error is thrown at the end of the execution, but I cannot see any records copied from SQL Server to MS Access.
I learnt that SQL Bulk copy cannot be used to copy from SQL Server to Access.
I also want to add the primary key from SQL Server table to MS Access table.
Why dont you use SSIS to do this for you.
You can create a SSIS package to copy a sql table to MS access.
If you want to initiate by .NET then create a SSIS package and call it from .NET
For details
At this point, oDA is not connected to any table on the Access side:
oDA.Update(donationdetails);
So even though you have all the data in a DataTable, you haven't got a target to copy it into.
I don't think this is the best approach, but that's the core of why your code isn't working as it is.
Ancient question, but I'm betting the RowState of all your rows in donationdetails were Unchanged, so the DataAdapter treats them as "I don't need to do anything with this row."
You can use dataset object instead of datatable.
DataSet ds=new DataSet();
sDA.Fill(ds,tablename);
oDA.Update(ds);
I'm trying to create a windows form application that manipulates data from several tables stored on a SQL server.
What's the best way to store the data locally, while the application is running? I had a previous program that only modified one table, and that was set up to use a datagridview. However, as I don't necessarily want to view all the tables, I am looking for another way to store the data retrieved by the SELECT * FROM ... query.
Is it better to load the tables, make changes within the C# application, and then update the modified tables at the end, or simply perform all operations on the database, remotely (retrieving the tables each time they are needed)?
You can take in one table at a time using a ConnectionString and assign it to a DataTable. You can then make changes to the DataTable in any form you want. Once you are finished making the changes you can commit the changes back to Database by using a DataAdapter.
Here's how to get a table:
DataTable table;
using (SqlDbConnection connection = new SqlDbConnection(connectionString))
{
connection.Open();
using (SqlDbCommand command = new SqlDbCommand(tableName, connection))
{
command.CommandType = CommandType.TableDirect;
SqlDbDataReader dr = command.ExecuteReader(CommandBehavior.CloseConnection);
table = new DataTable(tableName);
routeID.Load(dr);
}
}
Here's how to commit the table after changes, make sure to assign your DataTable to a DataSet in order to get the changes for commit:
DataSet dataSet = new DataSet();
dataSet.add(table);
using (var adapter = new SqlDbDataAdapter("SELECT * FROM " + tableName, connection))
{
using (var builder = new SqlDbCommandBuilder(adapter))
{
adapter.Fill(dataSet, tableName);
using (DataSet newSet = dataSet.GetChanges(DataRowState.Added))
{
builder.QuotePrefix = "[";
builder.QuoteSuffix = "]";
adapter.InsertCommand = builder.GetInsertCommand();
adapter.Update(newSet, tableName);
}
}
}
There may be a few miss Types, I didn't compile to check for Errors. Good Luck.
The DataSet object has methods that give you the ability to manipulate data from multiple tables in memory, and then save those changes back to the database. Whether that's a good idea depends on your requirements and data--are there multiple simultaneous users, how do you need to handle conflicting updates, etc. I usually prefer to write changes back to the DB immediately, but then, I usually work in a web app context.
How can I get a DataSet with all the data from a SQL Express server using C#?
Thanks
edit: To clarify, I do want all the data from every table. The reason for this, is that it is a relatively small database. Previously I'd been storing all three tables in an XML file using DataSet's abilities. However, I want to migrate it to a database.
You can use the GetSchema method to get all the tables in the database and then use a data adapter to fill a dataset. Something like this (I don't know if it compiles, I just paste some code and change it a bit):
DbProviderFactory factory = DbProviderFactories.GetFactory("System.Data.SqlClient");
DataTable tables = null;
DataSet database = new DataSet();
using (DbConnection connection = factory.CreateConnection())
{
connection.ConnectionString = "Data Source=(local);Initial Catalog=Northwind;Integrated Security=True";
string[] restrictions = new string[4];
// Catalog
restrictions[0] = "Northwind";
// Owner
restrictions[1] = "dbo";
// Table - We want all, so null
restrictions[2] = null;
// Table Type - Only tables and not views
restrictions[3] = "BASE TABLE";
connection.Open();
// Here is my list of tables
tables = connection.GetSchema("Tables", restrictions);
// fill the dataset with the table data
foreach (DataRow table in tables.Rows)
{
string tableName = table["TABLE_NAME"].ToString();
DbDataAdapter adapter = factory.CreateDataAdapter();
DbCommand command = factory.CreateCommand();
command.Connection = connection;
command.CommandType = CommandType.Text;
command.CommandText = "select * from [" + tableName + "]";
adapter.SelectCommand = command;
adapter.Fill(database, tableName);
}
}
EDIT:
Now I refactored it a bit and now it's working as it should. The use of DbConnection and DbProviderFactories is for database engine abstraction, I recommend using it so you can change the database engine changing this line and the connection string:
DbProviderFactory factory = DbProviderFactories.GetFactory("System.Data.OracleClient");
The GetSchema method will retrive all tables from your database to a DataTable and then we get all the data from each table to the DataSet using the DataAdapter.
I think you need to narrow down the question somewhat... All the data? You mean, all the data in every table in every database? Well, the only answer to that is, a lot of code.
To connect to and talk to a SQL Server Express database engine, use the classes in the System.Data.SqlClient namespace, namely:
SqlConnection: Connect to the database
SqlCommand: Talk to the database
SqlDataReader: Iterate over data retrieved from the database
You can check the MSDN pages for all of these classes by clicking on the links above.
Here are some overview-links with more information:
CodeProject: Beginners guide to accessing SQL Server through C#
DevHood: Accessing SQL Server Data in C# with ADO.NET
Note that by and large, you use a SQL Server Express database engine the same way as the full SQL Server product, the difference is more in the tools you get with it, and some limitations in the express engine. Other than that you can just use the classes and language that you would use for a normal SQL Server database engine installation.
If this post didn't answer your question, please elaborate, and you have a higher chance of getting the answer you seek.
This can be done by using dataAdapter class.