I have a bunch of 2 line (with header row) '|' delimited text files. I need to import this into a specific SQL table and I'm having a hard time with the command.
string sqltable = ("dbo.SLT_C" + "60" + "Staging");
string[] importfiles= Directory.GetFiles(#"K:\jl\load\dest", "*.txt")
SqlConnection con = new SqlConnection("Data Source=" + "Cove" + ";Initial Catalog=" + "GS_Ava_MCase"+ ";Integrated Security=" + "SSPI");
con.Open();
foreach (string importfile in importfiles)
{
}
or maybe I am going about this the whole wrong way.
You could look at a ready-made solution, like FileHelpers. This FREE library allows you to define the structure of your file by means of a class describing the fields in your file, and then you can easily load the whole file into an array of that class type.
Once that's done, just simply iterate through the objects, and save them to your SQL Server.
Or check out the SQL Bulkcopy options:
bcp command line utility
SqlBulkCopy class in ADO.NET - also see this article at SQL Team
If you want to do it in "straight" ADO.NET, use something like this approach:
string sqltable = "dbo.SLT_C60Staging";
string[] importfiles = Directory.GetFiles(#"K:\jl\load\dest", "*.txt");
// try to wrap your ADO.NET stuff into using() statements to automatically
// dispose of the SqlConnection after you're done with it
using(SqlConnection con = new SqlConnection("Data Source=Cove;Initial Catalog=GS_Ava_MCase;Integrated Security=SSPI"))
{
// define the SQL insert statement and use parameters
string sqlStatement =
"INSERT INTO dbo.YourTable(DateField, TimeField, TextField) VALUES(#Date, #Time, #Text)";
// define the SqlCommmand to do the insert - use the using() approach again
using(SqlCommand cmd = new SqlCommand(sqlStatement, con))
{
// define the parameters for the SqlCommand
cmd.Parameters.Add("#Date", SqlDbType.DateTime);
cmd.Parameters.Add("#Time", SqlDbType.DateTime);
cmd.Parameters.Add("#Text", SqlDbType.VarChar, 1000);
// loop through all files found
foreach (string importfile in importfiles)
{
// read the lines from the text file
string[] allLines = File.ReadAllLines(importfile);
con.Open();
// start counting from index = 1 --> skipping the header (index=0)
for (int index = 1; index < allLines.Length; index++)
{
// split up the data line into its parts, using "|" as separator
// items[0] = date
// items[1] = time
// items[2] = text
string[] items = allLines[index].Split(new char[] { '|' });
cmd.Parameters["#Date"].Value = items[0];
cmd.Parameters["#Time"].Value = items[1];
cmd.Parameters["#Text"].Value = items[2];
cmd.ExecuteNonQuery();
}
con.Close();
}
}
}
That should work - you're question was too vague to know exactly what data will be in the lines, and what kind of SQL insert statement you'd need...
Using the text ODBC driver might work as well. In the ODBC administrator, you can choose the "Microsoft Access Text Driver". It allows you to choose the delimiter type. After setting up the data source, import to a data table. From there, it should be fairly simple to move the data into a SQL Server table.
Related
I'm trying to import data into a SQL Server database from a .csv file. I have just one problem: for the money row, I am throwing Format.Exception due to wrong format of money variable.
I tried to convert to double I change the period instead of comma, I change in split(); method also semicolon ; instead of comma , but the exception didn't go away. Does anyone know what to do about this?
It is just an experiment.
My .csv file looks like this:
Database table's columns are:
name, second_Name, nickname, money
Code:
public void Import()
{
SqlCommand command = null;
var lineNumber = 0;
using (SqlConnection conn = DatabaseSingleton.GetInstance())
{
// conn.Open();
using (StreamReader reader = new StreamReader(#"C:\Users\petrb\Downloads\E-Shop\E-Shop\dataImport.csv"))
{
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
if (lineNumber != 0)
{
var values = line.Split(',');
using (command = new SqlCommand("INSERT INTO User_Shop VALUES (#name, #second_Name, #nickname, #money", conn))
{
command.Parameters.Add(new SqlParameter("#name", values[0].ToString()));
command.Parameters.Add(new SqlParameter("#second_Name", values[1].ToString()));
command.Parameters.Add(new SqlParameter("#nickname", values[2].ToString()));
command.Parameters.Add(new SqlParameter("#money", Convert.ToDecimal(values[3].ToString())));
command.Connection = conn;
command.ExecuteNonQuery();
}
}
lineNumber++;
}
}
conn.Close();
}
Console.WriteLine("Products import completed");
Console.ReadLine();
}
I maintain a package Sylvan.Data.Csv that makes it very easy to bulk import CSV data into SQL Server, assuming the shape of your CSV file matches the target table.
Here is some code that demonstrates how to do it:
SqlConnection conn = ...;
// Get the schema for the target table
var cmd = conn.CreateCommand();
cmd.CommandText = "select top 0 * from User_Shop";
var reader = cmd.ExecuteReader();
var tableSchema = reader.GetColumnSchema();
// apply the schema of the target SQL table to the CSV data.
var options =
new CsvDataReaderOptions {
Schema = new CsvSchema(tableSchema)
};
using var csv = CsvDataReader.Create("dataImport.csv", options);
// use sql bulk copy to bulk insert the data
var bcp = new SqlBulkCopy(conn);
bcp.BulkCopyTimeout = 0;
bcp.DestinationTableName = "User_Shop";
bcp.WriteToServer(csv);
On certain .NET framework versions GetColumnSchema might not exist, or might throw NotSupportedException. The Sylvan.Data v0.2.0 library can be used to work around this. You can call the older GetSchemaTable API, then use the Sylvan.Data.Schema type to convert it to the new-style schema IReadOnlyCollection<DbColumn>:
DataTable schemaDT = reader.GetSchemaTable();
var tableSchema = Schema.FromSchemaTable(schemaDT);
Try this:
SqlParameter moneyParam = new SqlParameter("#money", SqlDbType.Money);
moneyParam.Value = SqlMoney(Convert.ToDecimal(values[3].ToString()))
command.Parameters.Add(moneyParam)
Not sure if it'll work, but it seems to make sense to me.
The problem being that when you use the constructor for the SQL parameter, I think it assumes the type of your variable, so in this case, as you're passing a double or whatever, the equivalent DB type is 'decimal', however your DB schema will be using the DB type 'money', so explicitly setting the DB type in your parameter constructor should help.
I have an WPF application that reads data from a file like so:
foreach (String line in File.ReadAllLines(file, Encoding.UTF8))
{}
Each line is then parsed and displayed on the screen which all works fine. Some of the data has cyrillic alphabet in it and the strings that I'm using to store this data in are also displayed fine on the screen in the app window.
However, after that I'm using those same strings to insert them into MySQL database. I'm building a query and firing it up MySqlCommand cmd = new MySqlCommand(query, conn); which successfully inserts a new line in the database with the appropriate information. Numbers are all fine, however all the strings that go into the database and have cyrillic letters are displayed as ????????
Database engine is InnoDB and the encoding of the table and all varchar fields in it is utf_general_ci so any idea what is going on and how can I save the correct string in the database?
EDIT:
Per request, here's some code. Database connection:
conn = new MySqlConnection();
conn.ConnectionString = "//censored//";
And the file reading / db loading, shortened for the purposes of this code snippet:
foreach (String line in File.ReadAllLines(file, Encoding.UTF8))
{
string[] tokens = line.Split('|');
string query = "INSERT INTO myTable SET first_name = '" + tokens[0] + "'" + ", last_name = '" + tokens[1] + "'";
MessageBox.Show(tokens[0]);
MySqlCommand cmd = new MySqlCommand(query, conn);
cmd.ExecuteNonQuery();
}
The message box shows the name as it should be but what goes into the database is ???????.
After some headbanging I did figure out where the problem is so posting an answer for all to see:
The key part is the way you establish your connection to the database:
conn.ConnectionString = #"Server = YOURSERVER; Database = YOURDB; Uid = YOURUSER ; Pwd = 'YOURPASSWORD'; charset=utf8;";
I was missing the charset=utf8; part before so I assume all kinds of non-utf8 junk was going to the database regardless of the fact that I was encoding in UTF8 on both sides of the connection. Hope this helps!
I am working on sharp nlp where i am extracting all the adjective now i need to store this in database and i have successfully added this to database but the problem is with me that i want to store adjective separately to database how can i store the adjective separately or for example we have string and we want to store each word separately into database and we have only one column how can we do this? .please help me out
here is my code.
private void button2_Click(object sender, EventArgs e)
{
try
{
string cleaned = richTextBox1.Text.Trim();
string st = "INSERT INTO TABLE1(adjective)VALUES('" + cleaned + "')";
SqlConnection con = new SqlConnection("Data Source=ZAZIKHAN\\SQLEXPRESS;Initial Catalog=mis;Integrated Security=True");
con.Open();
SqlCommand cmd = new SqlCommand(st, con);
if (cmd.ExecuteNonQuery() == 1)
{
MessageBox.Show(" succesfully added");
}
else
{
MessageBox.Show("Sorry we couldnt add the Values Please try Again");
}
con.Close();
}
catch (Exception ex)
{
MessageBox.Show("" + ex);
}
}
now i have this data to be stored which is in richtextbox.
local/JJ daily/JJ n/price/rate/JJ human/JJ possible/JJ correct/JJ exact/JJ local/JJ
local/JJ daily/JJ n/price/rate/JJ human/JJ possible/JJ correct/JJ exact/JJ local/JJ
dry/JJ nd/JJ
new/JJ full/JJ OK/JJ final/JJ white/JJ OK/JJ howajaa/JJ smajder/JJR agaa/JJ nasambd/JJ Live/JJ
final/JJ
great/JJ s3/JJ final/JJ
resistant/JJ Z1/JJ white/JJ new/JJ black/JJ amaa.Kintact/JJ possible/JJ main/JJ mobile/JJ rapak/JJ mil/JJ
important/JJ mil/JJ smart/JJ
35-h/JJ OK/JJ full/JJ
Want/JJ complete/JJ white/JJ same/JJ
available/JJ perfect/JJ
interested/JJ
First off, the lines
string cleaned = richTextBox1.Text.Trim();
string st = "INSERT INTO TABLE1(adjective)VALUES('" + cleaned + "')";
create a massive security hole known as SQL Injection.
In order to store the adjectives separately in a properly denormalized database, you would have a parent table where e.g. the original sentence is stored, and a child table with a 1:N relationship to the parent where you store one row per adjective plus the appropriate ID of the parent row.
Since you only have one column available, you can use any convenient format to store the array of adjectives in a single column. You could serialize that array (to Binary, XML, JSON, etc) and store it, or since you know you have a limited input character set, you could even store it as a comma separated list.
You can prefix your words with some characters to indicate whether they are verb , noun , adjective and tehn insert those value in database
eg
N_JonSkeet - Noun
V_Solve - Verb
A_Highest - Adjective
string cleaned = extractAdjective(richTextBox1.Text.Trim());
string st = "INSERT INTO TABLE1(word) VALUES(#Word)";
SqlConnection con = new SqlConnection("Data Source=ZAZIKHAN\\SQLEXPRESS;Initial Catalog=mis;Integrated Security=True");
con.Open();
SqlCommand cmd = new SqlCommand(st, con);
SqlParameter param = new SqlParameter("#Word", SqlDbType.NChar);
param.Value = "A_"+cleaned;
cmd.Parameters.Add(param);
I would separate the string into a list and then iterate over the list and insert into your DB:
var vals = "local/JJ daily/JJ n/price/rate/JJ human/JJ possible/JJ...";
var results = vals.Replace(" ", "")
.Replace("/JJ", "|")
.Replace("/", "|")
.Split('|')
.Distinct();
while(var result in results)
{
// DO insert here
}
Here is the background of my problem: I have a combobox that when users start typing, it should retrieve suggested items from a column in database table. The user starts inputing name and the program should suggest names by looking at both first and last names (database has separate tables for both )
Here is the code that I had:
try{
String temp = nameCBox.Text;
AutoCompleteStringCollection namesSuggestion = new AutoCompleteStringCollection();
OleDbConnection conn = new OleDbConnection("Provider=Microsoft.ACE.Oledb.12.0;Data Source=C:\\LogEntry\\LogEntry.accdb; Persist Security Info = False;");
OleDbDataReader reader;
conn.Open();
String text2send = "Select Name from [Teachers] where FName like '" + temp + "' OR LName like '" + temp + "' Group by [Name]";
OleDbCommand cmd = new OleDbCommand(text2send, conn);
reader = cmd.ExecuteReader();
if (reader.HasRows == true)
{
while (reader.Read())
namesSuggestion.Add(reader["temp"].ToString());
}
reader.Close();
nameCBox.AutoCompleteCustomSource = namesSuggestion;
conn.Close();
}
errors:
1) I see no suggestions in the combo box
2) When I type in the combo box, it highlights the text and when I type something else again, it will write on the previous typed character.
Please Help
desktopmaker
Erm
What's this doing
namesSuggestion.Add(reader["temp"].ToString());
reader is returning a column called Name..
I wouldunless the user is typing it in expect the search string to contain a wild card.
E.g. Like 'John%', or Like '%Smith' in standard sql, access uses * instead of % I seem to remember.
Oh and presumably you aren't worried about sql injection attacks??
Since you are using Like operator, try to make the input between look like this : %input%
To get something like :
Take a look at this, it may help
var sql = String.Format("Select Name from [Teachers] WHERE FName Like '%{0}%' OR LName Like '%{0}%'", temp);
Other points may be helpful about your current code :
Use the using statement in your code, it dispose the resources, and for the Connection, it close it.
using(var conn = new OleDbConnection("ConnectionStrong"))
{
//code
}
The best, is to use a parameterized query Link
What I have is an extremely large text file that needs to go into a specific column at a specific raw. The file is around 100k lines. So what I want to do is read the whole file, and for each line append into that specific SQL column the line. Here's what I have but i really need help on the SQL query
string[] primaryfix = File.ReadAllLines(dinfo+"\\"+filex);
string filename = filex.ToString();
string[] spltifilename = filename.Split('.');
foreach (string primary in primaryfix)
{
string sqltable = ("dbo.amu_Textloadingarea");
string sql = "update " + sqltable + " set [Text] = [Text] + '" + primary + "' where begbates = '" + spltifilename[0] + "'";
SqlConnection con = new SqlConnection("Data Source= Corvette ;Initial Catalog= GSK_Avandia_SSECASE;Integrated Security= SSPI");
con.Open();
SqlCommand cmd = new SqlCommand(sql, con);
SqlDataReader reader = cmd.ExecuteReader();
con.Close();
}
everything is fine except for the string sql, it doesn't update the way I would like it to.
Any help is always appreciated.
Look likes you're trying to read from the database with that code inside the loop. SqlDataReader provides a way to read rows from the database, but not the other way around.
Replace
SqlDataReader reader = cmd.ExecuteReader();
with
cmd.ExecuteNonQuery();
The first thing I see is that you are not escaping the input from the text file; any SQL escape characters (like a single quote) will break that command. I'd recommend using parameters so you needn't worry about escapes at all.
Other than that, nothing pops to mind that would suggest why the command isn't working, but I do wonder if it might not cause fewer problems if it's such a large file to read it line-by-line rather than all at once.