Develop Reference

Using MySqlBulkLoader to upload a DataSet content : Issue with the filename

I'd like to try the performance of MySqlBulkLoader knowing that the Adapter.update() method i'm using is taking roughly 30 mn to run.
I understand you have to go through a file to do it so here is my code :
private void button14_Click(object sender, EventArgs e)
{
string fileName = #"C:\Users\Utilisateur\ds.txt";
if (File.Exists(fileName))
{
File.Delete(fileName);
}
using (StreamWriter sw = File.CreateText(fileName))
{
foreach (DataRow row in Globals.ds.Tables[0].Rows)
{
foreach (object item in row.ItemArray)
{
string itemstr = item.ToString();
sw.Write((string)itemstr + "\t");
}
sw.WriteLine();
}
}
using (var conn = new MySqlConnection(Globals.connString))
{
conn.Open();
MySqlCommand comm = new MySqlCommand("TRUNCATE Song",conn);
comm.ExecuteNonQuery();
var bl = new MySqlBulkLoader(conn)
{
TableName = Globals.ds.Tables[0].ToString(),
Timeout = 600,
FieldTerminator = "\t",
LineTerminator = "\n",
FileName = fileName
};
var numberOfInsertedRows = bl.Load();
Console.WriteLine(numberOfInsertedRows);
}
}
The file is generated ok. but at the var numberOfInsertedRows = bl.Load(); line, i have the following error at run time :
MySql.Data.MySqlClient.MySqlException: 'Can't get stat of '/var/packages/MariaDB10/target/mysql/disk/C:\Users\Utilisateur\ds.txt' (Errcode: 2 "No such file or directory")'
I tried to put "/" instead of "\" in the fileName but it's the same error.
I have no idea what's going on, anyone can help ?
Thanks

By default, MySqlBulkLoader loads a file from the server's file system. To use a local file, set bl.Local = true; before calling bl.Load().
To enable this, you will need to set AllowLoadLocalInfile = True in your connection string; see https://mysqlconnector.net/troubleshooting/load-data-local-infile/
Finally, if you switch to MySqlConnector, you can use its MySqlBulkCopy API to load data directly from a DataTable, instead of first saving it to a local CSV file, then loading that file.

C# Error: Microsoft Excel cannot access the file '...'. There are several possible reasons

I've developed an ASP.Net MVC application, that is running on a IIS sever. I've wrote a code that reads a CSV and insert the rows of it in a database.
[HttpPost]
public ActionResult InsertPosition(int id, HttpPostedFileBase position)
{
var posicoesExistentes = db.tbPositions.Where(s => s.id_unique == id).AsEnumerable();
foreach (tbPosition posicao in posicoesExistentes)
{
db.tbPositions.Remove(posicao);
}
if (!Directory.Exists(Server.MapPath("~/App_Data/")))
{
System.IO.Directory.CreateDirectory(Server.MapPath("~/App_Data/"));
}
string excelPath = Server.MapPath("~/App_Data/" + position.FileName);
if (System.IO.File.Exists(excelPath))
{
System.IO.File.Delete(excelPath);
}
position.SaveAs(excelPath);
string tempPath = Server.MapPath("~/App_Data/" + "tmp_" + position.FileName);
System.IO.File.Copy(excelPath, tempPath, true);
Excel.Application application = new Excel.Application();
Excel.Workbook workbook = application.Workbooks.Open(tempPath, ReadOnly: true,Editable:false);
Excel.Worksheet worksheet = workbook.ActiveSheet;
Excel.Range range = worksheet.UsedRange;
application.Visible = true;
for (int row = 1; row < range.Rows.Count - 1; row++)
{
tbPosition p = new tbPosition();
p.position = (((Excel.Range)range.Cells[row, 1]).Text == "") ? null : Convert.ToInt32(((Excel.Range)range.Cells[row, 1]).Text);
p.left = ((Excel.Range)range.Cells[row, 2]).Text;
p.right = ((Excel.Range)range.Cells[row, 3]).Text;
p.paper = ((Excel.Range)range.Cells[row, 4]).Text;
p.denomination = ((Excel.Range)range.Cells[row, 5]).Text;
p.material = ((Excel.Range)range.Cells[row, 6]).Text;
p.norme = ((Excel.Range)range.Cells[row, 7]).Text;
p.finalized_measures = ((Excel.Range)range.Cells[row, 8]).Text;
p.observation = ((Excel.Range)range.Cells[row, 9]).Text;
p.id_unique = id;
db.tbPositions.Add(p);
db.SaveChanges();
}
workbook.Close(true, Type.Missing, Type.Missing);
application.Quit();
System.IO.File.Delete(tempPath);
return Json("Success", JsonRequestBehavior.AllowGet);
}
but in return I got the error ' Microsoft Excel cannot access the file '...'. There are several possible reasons' when I try to open the requested excel file.
I've already tried to open the file as readonly, I've already tried to give permissions to the specifieds folders, multiples ways of close the excel file, and create an copy file of the original and read him. But unsuccessful in each one of these solutions. What have I missed here?

Unsupported
The short answer is that trying to programatically manipulate an Excel document using the Automation API is not supported outside of a UI context. You will come across all sorts of frustrations (for example, the API is permitted to show dialogs - how are you going to click on "OK" if it's running on a web-server?).
Microsoft explicitly state this here
Microsoft does not recommend or support server-side Automation of Office.
So what do I use?
I would recommend using the OpenXML SDK - this is free, fully supported and much faster than the Automation API.
Aspose also has a set of products, but they are not free, and I've not used them.
But I HAVE to do it this way
However, if you absolutely have to use the COM API then the following might help you:
HERE BE DRAGONS
The big problem with automation in Excel is that you need to ensure you close every single reference whenever you use them (by calling ReleaseComObject on it).
For example, the following code will cause Excel to stay open:
var range;
range = excelApplication.Range("A1");
range = excelApplication.Range("A2");
System.Runtime.InteropServices.Marshal.ReleaseComObject(range)
range = Nothing
This is because there is still a reference left over from the call to get range "A1".
Therefore, I would recommend writing a wrapper around the Excel class so that any access to, e.g., a range frees any previous ranges accessed before accessing the new range.
For reference, here is the code I used to release COM objects in the class I wrote:
Private Sub ReleaseComObject(ByVal o As Object)
Try
If Not IsNothing(o) Then
While System.Runtime.InteropServices.Marshal.ReleaseComObject(o) > 0
'Wait for COM object to be released.'
End While
End If
o = Nothing
Catch exc As System.Runtime.InteropServices.COMException
LogError(exc) ' Suppress errors thrown here '
End Try
End Sub

Try this
protected void ImportCSV(object sender, EventArgs e)
{
importbtn();
}
public class Item
{
public Item(string line)
{
var split = line.Split(',');
string FIELD1 = split[0];
string FIELD2 = split[1];
string FIELD3 = split[2];
string mainconn = ConfigurationManager.ConnectionStrings["ConnectionString"].ConnectionString;
using (SqlConnection con = new SqlConnection(mainconn))
{
using (SqlCommand cmd = new SqlCommand("storedProcedureName", con))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.AddWithValue("#FIELD1", SqlDbType.VarChar).Value = FIELD1;
cmd.Parameters.AddWithValue("#FIELD2", SqlDbType.VarChar).Value = FIELD2;
cmd.Parameters.AddWithValue("#FIELD3", SqlDbType.VarChar).Value = FIELD3;
con.Open();
cmd.ExecuteNonQuery();
}
}
}
}
private void importbtn()
{
try
{
string csvPath = Server.MapPath("~/Files/") + Path.GetFileName(FileUpload1.PostedFile.FileName);
FileUpload1.SaveAs(csvPath);
var listOfObjects = File.ReadLines(csvPath).Select(line => new Item(line)).ToList();
DataTable dt = new DataTable();
dt.Columns.AddRange(new DataColumn[3] { new DataColumn("FIELD1", typeof(string)),
new DataColumn("FIELD2", typeof(string)),
new DataColumn("FIELD3",typeof(string)) });
string csvData = File.ReadAllText(csvPath);
foreach (string row in csvData.Split('\n'))
{
if (!string.IsNullOrEmpty(row))
{
dt.Rows.Add();
int i = 0;
//Execute a loop over the columns.
foreach (string cell in row.Split(','))
{
dt.Rows[dt.Rows.Count - 1][i] = cell;
i++;
}
}
}
GridView1.DataSource = dt;
GridView1.DataBind();
Label1.Text = "File Attached Successfully";
}
catch (Exception ex)
{
Message.Text = "Please Attach any File" /*+ ex.Message*/;
}
}

I want to show file upload progress in c# application

I want to show a file upload progress in c# windows application As I am using online database it takes a bit more time to save or upload.I am using following Code.I have searched but It does not shows exact percent wise progress.Also I Want to change file name before moving to another lOcation.
Thanks In advance.
OpenFileDialog dlg = new OpenFileDialog();
DialogResult dlgRes = dlg.ShowDialog();
Application.DoEvents();
label14.Visible = true;
if (DialogResult.OK != DialogResult.Cancel)
{
foreach (string file in dlg.FileNames)
{
try
{
newpath = Path.Combine(textBox2.Text, Path.GetFileName(file)).ToString();
filenametemp = "tush" + Path.GetFileName(file).ToString();
if (String.IsNullOrEmpty(textBox2.Text))
{
MessageBox.Show("please select folder to save");
}
else
{
File.Copy(file, Path.Combine(textBox2.Text, Path.GetFileName(file)).ToString());
if (dlgRes != DialogResult.Cancel)
{
//Provide file path in txtFilePath text box.
txtFilePath.Text = dlg.FileName;
}
//string folderpath = System.IO.Directory.GetParent(Environment.CurrentDirectory).Parent.FullName + textBox2.Text;
//string filePath = textBox2.Text + System.IO.Path.GetFileName(dlg.FileName);
//System.IO.File.Copy(dlg.FileName, folderpath, true);
try
{
//Read File Bytes into a byte array
byte[] FileData = ReadFile(txtFilePath.Text);
//Initialize SQL Server Connection
SqlConnection CN = new SqlConnection(ConfigurationManager.ConnectionStrings["Copymanagment"].ConnectionString);
//Set insert query
string qry = "insert into fileinfo (File_Id,File_Path,date,OriginalPath,Billnumber,Billdate,FileData) values(#FileId,#Filepath,#Date,#OriginalPath,#Billnumber,#Billdate,#FileData)";
//Initialize SqlCommand object for insert.
SqlCommand SqlCom = new SqlCommand(qry, CN);
string path = textBox2.Text;
string[] arrpath = null;
int count;
arrpath = path.Split('\\');
for (count = 0; count <= arrpath.Length - 1; count++)
{
// MessageBox.Show(arrpath[count]);
}
string folder = arrpath[arrpath.Length - 1] + databaseid().ToString();
//We are passing Original File Path and File byte data as sql parameters.
string fileid = Path.GetDirectoryName(dlg.FileName);
SqlCom.Parameters.Add(new SqlParameter("#FileId", (object)folder));
SqlCom.Parameters.Add(new SqlParameter("#Filepath", (object)newpath));
SqlCom.Parameters.Add(new SqlParameter("#Date", (object)System.DateTime.Now.ToString("dd-MM-yyyy hh:mm:ss tt")));
SqlCom.Parameters.Add(new SqlParameter("#OriginalPath", (object)txtFilePath.Text));
SqlCom.Parameters.Add(new SqlParameter("#Billnumber", (object)textbillno.Text));
SqlCom.Parameters.Add(new SqlParameter("#Billdate", (object)dateTimePicker1.Value.ToString("dd-MM-yyyy")));
SqlCom.Parameters.Add(new SqlParameter("#FileData", (object)FileData));
//Open connection and execute insert query.
CN.Open();
SqlCom.ExecuteNonQuery();
CN.Close();
label14.Visible = false;
//Close form and return to list or Files.
MessageBox.Show("File saved Succsesfully");
txtFilePath.Text = "";
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
}
}
catch
{
MessageBox.Show("File already available");
}
}
}

The way you have it done, a single statement in inserting the whole data in one shot and blocks until done without a chance to update tge UI. You can upload your BLOB data in chuncks using UPDATETEXT. see https://msdn.microsoft.com/en-us/library/vstudio/3517w44b(v=vs.100).aspx for full explanations on how to do that. Doing that, you will have the opportunity to update the value of a prigress bar.

You can use asp.net ajax control toolkit. There is a ajax file upload control.
Note Do not get confused with asynFile upload control in the same library(that control does not have progress bar).
You can get info here http://www.codingfusion.com/Post/AjaxFileUpload-example-to-upload-multiple-files-in
To download the library go here http://www.codingfusion.com/Post/3-Different-ways-to-add-AjaxControlToolkit-in-Asp
for windows app try following https://stackoverflow.com/a/9446524/4810628

SSIS and The Excel Files Saga

I have a Excel File (xls) that has a column called Money. In the Money column all the columns are formatted as number, except for some that have that marker saying formatted as text against them. I convert the Excel file to CSV using a c# script that uses IMEX=1 in the connection string to open it. The fields that are marked with stored as text do not come through to the csv file. The file is large, about 20MB. So this means 100 values like 33344 etc do not come thro the csv file.
I tried to put a delay in where I open the Excel File. This worked on my PC but not the Development machine.
Have any idea how to get round this without manually intervention, like format all columns with mixed data types as number etc ? I am looking for an automated solution that works every time . This is on SSIS 2008.
static void ConvertExcelToCsv(string excelFilePath, string csvOutputFile, int worksheetNumber = 1) {
if (!File.Exists(excelFilePath)) throw new FileNotFoundException(excelFilePath);
if (File.Exists(csvOutputFile)) throw new ArgumentException("File exists: " + csvOutputFile);
// connection string
var cnnStr = String.Format("Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0};Extended Properties=\"Excel 8.0;IMEX=1;HDR=NO\"", excelFilePath);
var cnn = new OleDbConnection(cnnStr);
// get schema, then data
var dt = new DataTable();
try {
cnn.Open();
var schemaTable = cnn.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
if (schemaTable.Rows.Count < worksheetNumber) throw new ArgumentException("The worksheet number provided cannot be found in the spreadsheet");
string worksheet = schemaTable.Rows[worksheetNumber - 1]["table_name"].ToString().Replace("'", "");
string sql = String.Format("select * from [{0}]", worksheet);
var da = new OleDbDataAdapter(sql, cnn);
da.Fill(dt);
}
catch (Exception e) {
// ???
throw e;
}
finally {
// free resources
cnn.Close();
}
// write out CSV data
using (var wtr = new StreamWriter(csvOutputFile)) {
foreach (DataRow row in dt.Rows) {
bool firstLine = true;
foreach (DataColumn col in dt.Columns) {
if (!firstLine) { wtr.Write(","); } else { firstLine = false; }
var data = row[col.ColumnName].ToString().Replace("\"", "\"\"");
wtr.Write(String.Format("\"{0}\"", data));
}
wtr.WriteLine();
}
}
}

My solution was to specify a format for the incoming files which said no columns with mixed data types. Solution was from business side and not technology.

Can the server use a file from a client without uploading it to the server? ASP.NET

I was wondering, at the moment, the client uploads a file to a directory on the server where the server can then use the file (csv file) to update a SQL database.
Is that the only way the server can use that file? For it to first be uploaded to the server? Or can you use that file without uploading it to a directory of the server?
(using visual studio 2012, C# and asp.net)
Code update---
asp Uploading the file. (I know the code can be cleaner an written better, but my testing code is never clean)
//Uplod file to the server
FileUpload1.SaveAs(serverUploadDir + FileUpload1.FileName);
//Use the Uploaded File to update the sql table
dbConn.UploadCSVToSql(serverUploadDir + FileUpload1.FileName);
The UploadCSVToSql
public void UploadCSVToSql(string filepath)
{
//string filepath = "C:\\Copy of ABI Employee List.csv";
StreamReader sr = new StreamReader(filepath);
string line = sr.ReadLine();
string[] value = line.Split(',');
DataTable dt = new DataTable();
DataRow row;
foreach (string dc in value)
{
dt.Columns.Add(new DataColumn(dc));
}
while (!sr.EndOfStream)
{
value = sr.ReadLine().Split(',');
if (value.Length == dt.Columns.Count)
{
row = dt.NewRow();
row.ItemArray = value;
dt.Rows.Add(row);
}
}
SqlBulkCopy bc = new SqlBulkCopy(conn.ConnectionString, SqlBulkCopyOptions.TableLock);
bc.DestinationTableName = "MainDump";
bc.BatchSize = dt.Rows.Count;
conn.Open();
bc.WriteToServer(dt);
bc.Close();
conn.Close();

I don't see how you would get a file from the client to the server without uploading it, but you don't need to store it to a folder. If you use the
<asp:FileUpload ID="fuMyUpload" runat="server" />
control. you can get the data in a stream and store it in memory.
if (!fuMyUpload.HasFile)
{
lblWarning.Text = "No file Selected";
return;
}
var csvData = Encoding.UTF8.GetString(fuCircuitCsv.FileBytes);
using (var reader = new StringReader(csvData))
{
var headers = reader.ReadLine().Split(',');
while ((line = reader.ReadLine()) != null)
{
var fields = line.Split(',');
}
}

If you have a background job running on the server which monitors the folder for any CSV file, then it needs to be uploaded to the server. If that is not the case, then you should be able to process the file in C# only and perform database update.