I want to load millions of records from an Oracle table into a Postgres table in the quickest and most convenient way using a C# console app. Ideally I don't want to COPY out to a .CSV file and then load back in as I am not sure about the necessary permissions on the target server. Is it possible to do it via an Oracle datareader? This is what I have tried so far (The SQL query columns names are named as such to show the datatypes in Oracle)
public void RunExportPackage()
{
var sql = "SELECT NUMBER(10,0), CHAR(3), VARCHAR2(10), NUMBER, CHAR(1) FROM TABLE";
using var sourceConnection = new OracleConnection(_appSettingsOptions.ConnectionStrings.OracleConnection);
OracleCommand com = new OracleCommand(sql, sourceConnection);
sourceConnection.Open();
using (OracleDataReader oracleReader = com.ExecuteReader())
{
using (NpgsqlConnection targetConnection = new NpgsqlConnection(_appSettingsOptions.ConnectionStrings.PostgresConnectionDEV))
{
targetConnection.Open();
using (var writer = targetConnection.BeginBinaryImport("COPY TABLE FROM STDIN BINARY"))
{
writer.WriteRow(oracleReader);
}
}
}
}
When I try this I am getting this error.
IEnumerable parameters are not supported, pass an array or List instead
The column types in the postgres table are
If someone could just point me in the right direction that would be great :)
Related
I've got a console application that uses an API to get data which is then saved into a csv file in the following format:
Headers:
TicketID,TicketTitle,TicketStatus,CustomerName,TechnicianFullName,TicketResolvedDate
Body:
String values. where TicketResolvedDate is written as: YYYY-MM-DDTHH:mm:ssZ
Now I want to import this csv file into my mssql express database using the same console application and make sure the TicketID is imported as a integer datatype and the TicketResolvedDate as a SQL datetime datatype.
I've made the following code:
List<TicketCSV> tickets = new List<TicketCSV>();
using var reader1 = new StreamReader(OutputClosedTickets);
using var reader2 = new StreamReader(OutputWorkhours);
using var csv1 = new CsvReader((IParser)reader1);
{
csv1.Configuration.Delimiter = ",";
csv1.Configuration.MissingFieldFound = null;
csv1.Configuration.PrepareHeaderForMatch = (string header, int index) => header.ToLower();
csv1.ReadHeader();
while (csv1.Read())
{
var record = new TicketCSV
{
TicketID = csv1.GetField<int>("TicketID"),
TicketTitle = csv1.GetField("TicketTitle"),
TicketStatus = csv1.GetField("TicketStatus"),
CustomerName = csv1.GetField("CustomerName"),
TechnicianFullName = csv1.GetField("TechnicianFullName"),
TicketResolvedDate = SqlDateTime.Parse(csv1.GetField("TicketResolvedDate"))
};
tickets.Add(record);
}
}
using (var bulkCopy = new SqlBulkCopy(connectionString))
{
bulkCopy.DestinationTableName = "GeslotenTickets";
bulkCopy.WriteToServer((IDataReader)csv1);
bulkCopy.DestinationTableName = "WerkUren";
bulkCopy.WriteToServer((IDataReader)reader2);
}
But I'm not sure if this is remotely near the idea i should have to establish this
You're on the right track, but there are a couple issues with your code. You're reading the CSV data into objects, but then passing the CsvReader to the bulk copy operation. At that point all the CSV data in the reader has already been consumed, because you read it all when you were creating objects. Thus the SqlBulkCopy won't see any data in the reader.
The next issue that I think you're going to have is that the "schema" of the data reader needs to match the schema of the target SQL table. If the schemas don't match, you'll typically get some cryptic error message out of the SqlBulkCopy operation, that some type can't be converted.
I maintain a library that I've specifically designed to work well in this scenario: Sylvan.Data.Csv. It allows you to apply a schema to the "untyped" CSV data.
Here is an example of how you could write CSV data to a table in SqlServer:
using Sylvan.Data.Csv;
using System.Data.SqlClient;
static void LoadTableCsv(SqlConnection conn, string tableName, string csvFile)
{
// read the column schema of the target table
var cmd = conn.CreateCommand();
cmd.CommandText = $"select top 0 * from {tableName}"; // beware of sql injection
var reader = cmd.ExecuteReader();
var colSchema = reader.GetColumnSchema();
reader.Close();
// apply the column schema to the csv reader.
var csvSchema = new CsvSchema(colSchema);
var csvOpts = new CsvDataReaderOptions { Schema = csvSchema };
using var csv = CsvDataReader.Create(csvFile, csvOpts);
using var bulkCopy = new SqlBulkCopy(conn);
bulkCopy.DestinationTableName = tableName;
bulkCopy.EnableStreaming = true;
bulkCopy.WriteToServer(csv);
}
You still might encounter errors if the CSV data doesn't correctly match the schema, or has invalid or broken records, but this should work if your csv files are clean and valid.
I am using this method to upload data to SQL.
private void button5_Click(object sender, EventArgs e)
{
string filepath = textBox2.Text;
string connectionString_i = string.Format(#"Provider=Microsoft.Jet.OleDb.4.0; Data Source={0};Extended Properties=""Text;HDR=YES;FMT=Delimited""", Path.GetDirectoryName(filepath));
using (OleDbConnection connection_i = new OleDbConnection(connectionString_i))
{
connection_i.Open();
OleDbCommand command = new OleDbCommand ("Select * FROM [" + Path.GetFileName(filepath) +"]", connection_i);
command.CommandTimeout = 180;
using (OleDbDataReader dr = command.ExecuteReader())
{
string sqlConnectionString = MyConString;
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(sqlConnectionString))
{
SqlBulkCopy bulkInsert = new SqlBulkCopy(sqlConnectionString);
bulkInsert.BulkCopyTimeout = 180;
bulkInsert.DestinationTableName = "Table_Name";
bulkInsert.WriteToServer(dr);
MessageBox.Show("Upload Successful!");
}
}
connection_i.Close();
}
}
I have an Excel sheet in .CSV format of about 1,048,313 entries. That bulk copy method is just working for about 36000 to 60000 entries. I want to ask if there is any way that I can select the first 30000 entries from Excel and upload them to a SQL Server table, then again select next chunk of 30000 rows and upload those to SQL Server, and so on until the last entry has been stored.
Create a datatable to store the values from your csv file that needs to be inserted into your target table. Each column in the datatable would correspond to a data column in the csv file.
Create a custom data type (table-valued) on SQL Server to match your data table, including data type and length. As this post was tagged sql-server and not access, as your sample connection string seems to contradict that.
Using a text reader and a counter variable, populate your datatable with 30,000 records.
Pass the data table to your insert query or stored procedure. The pararameter type is SqlDbType.Structured.
In the event that the job fails and you need to restart, the first step could be to determine the last inserted value from a predefined key in your field. Your could also use a left outer join as part of your insert query to only insert records that do not exist on the table. These are just a few of the more common techniques to restart a failed ETL job.
This technique has some tactical advantages over the bulk copy as it adds flexibility and is less coupled to the target table, thus changes to the table could be less volatile, depending on the nature of the change.
I have a sqlite database. When I query the db directly via the command line, the schema_version result is 3078.
But when I do this in .NET, the result is always '1'.
Here's a snippet:
public static long GetInternalSchemaVersion(this SQLiteConnection con)
{
using (var cmd = con.CreateCommand())
{
cmd.CommandText = string.Format("PRAGMA schema_version;");
DataTable results;
using (results = new DataTable())
{
using (SQLiteDataAdapter da = new SQLiteDataAdapter(cmd))
{
da.Fill(results);
return results.ToEnumerable().First().GetInt32("schema_version");
}
}
}
}
Am I doing something wrong?
Ideally you would show the connection string you're using. The most likely case is that when you're connecting via .Net, you're not connecting to the same database. The schema version is auto-incremented by SQLite on schema change, so the fact that it's returning 1 suggests this is a new database. To verify, run another query to pull back some other data you expect to be there. You'll probably find that the db is empty.
I am new to C# and have been using PHP for Years.
The problem I am facing is that I want to create a method/function that will take an "sql query" as parameter and return whatever data is retrieved so that it can be used using the key/index/column_name.
What I have achieved
I have achieved is that I am able to access the data through index within the same function.
but I wont know the no. of columns that will return so I am unable to store it in an array.
Code
string ConnectionString = ConfigurationManager.ConnectionStrings["sqlcon"].ConnectionString;
MySqlConnection db;
db = new MySql.Data.MySqlClient.MySqlConnection(ConnectionString);
try
{
db.Open();
MySqlCommand cmd = new MySqlCommand();
cmd.Connection = db;
cmd.CommandText = sql;
MySqlDataReader rdr = cmd.ExecuteReader();
while (rdr.Read())
{
Console.WriteLine(rdr['areaId'].toString());
}
db.Close();
return true;
}
The problems
1. I do not know the number of Columns.
2. I cannot access the index or Column name.
I want to create an associative array of data with the column name as index and data as its data
EDIT
I am trying to build an application on C# using WPF with MySQL Database.
My first suggestion would be to use one of the micro ORMs like Dapper.net
This can be fetched via nuget. If you wish to retrieve an unknown array of data from a table, or number of tables, you could fetch it into an ExpandoObject and then enumerate over that. An example of looping over an ExpandoObject can be found here.
public Foo FetchSomeData(string query)
{
using (var cn = Connection)
{
var result = cn.Query(query);
//do something with result, and return it
return new Foo(result);
}
}
I'd have concerns about passing raw sql into the DB without at least checking it or using parameters as it could be a major security risk, especially if you allow users to generate the query and you don't vet it first.
as new to C# and SQLite world, how can i possibly display data from SQLite in the DataGridView in C#?
I really don't have any idea on how to do it. Even I tried some tutorials from the internet but I don't get it right. And most of them are using MySQL.
You can use System.Data.SQLite ADO.NET provider for connecting to your sqlite db.
When the references are added, you can write queries, like
using (var conn = new SQLiteConnection(#"Data Source=test.db3"))
using (var cmd = conn.CreateCommand())
{
conn.Open();
cmd.CommandText = "SELECT * FROM random_table_name";
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
string name = reader.GetString(reader.GetOrdinal("name"));
...
}
}
}
Put the read values into a List or some kind of collection.
After finishing, you can bind your Data Grid View into your list. It's a good msdn page.