Develop Reference

SSIS C# Script Task: (Bulk) Insert data from .csv into SQL Server database (preferably using OLEDB)

Problem Stement
I am trying to completely automate (via parametrization) my SSIS package. It uses the data flow that reads a .csv file and inserts its contents into SQL Server table. So, I need to do this without using the data flow task.
New setup
I have replaced the data flow task with a script task that does the same thing.
The .csv file is loaded into the DataTable object and then inserted into the destination table using SqlBulkCopy class and SqlConnection instance.
public void Main()
{
var atlas_source_application = (string)Dts.Variables["$Project::Atlas_SourceApplication"].Value;
var ssis_package_name = (string)Dts.Variables["System::PackageName"].Value;
var csv_path = (string)Dts.Variables["$Project::SVM_Directory"].Value;
var atlas_server_name = (string)Dts.Variables["$Project::AtlasProxy_ServerName"].Value;
var atlas_init_catalog_name = (string)Dts.Variables["$Project::AtlasProxy_InitialCatalog"].Value;
var connname = #"Data Source=" + atlas_server_name + ";Initial Catalog=" + atlas_init_catalog_name + ";Integrated Security=SSPI;";
var csv_file_path = #"" + csv_path + "\\" + ssis_package_name + ".csv";
try
{
DataTable csvData = new DataTable();
// Part I - Read
string contents = File.ReadAllText(csv_file_path, System.Text.Encoding.GetEncoding(1252));
TextFieldParser parser = new TextFieldParser(new StringReader(contents));
parser.HasFieldsEnclosedInQuotes = true;
parser.SetDelimiters(",");
string[] fields;
while (!parser.EndOfData)
{
fields = parser.ReadFields();
if (csvData.Columns.Count == 0)
{
foreach (string field in fields)
{
csvData.Columns.Add(new DataColumn(string.IsNullOrWhiteSpace(field.Trim('\"')) ? null : field.Trim('\"'), typeof(string)));
}
}
else
{
csvData.Rows.Add(fields.Select(item => string.IsNullOrWhiteSpace(item.Trim('\"')) ? null : item.Trim('\"')).ToArray());
}
}
parser.Close();
// Part II - Insert
using (SqlConnection dbConnection = new SqlConnection(connname))
{
dbConnection.Open();
using (SqlBulkCopy s = new SqlBulkCopy(dbConnection))
{
s.DestinationTableName = "[" + atlas_source_application + "].[" + ssis_package_name + "]";
foreach (var column in csvData.Columns)
{
s.ColumnMappings.Add(column.ToString(), column.ToString());
}
s.WriteToServer(csvData);
}
}
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception ex)
{
Dts.Events.FireError(0, "Something went wrong ", ex.ToString(), string.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
This setup works perfectly fine on my local computer. However, once the package is deployed on the server, the insertion part breaks since the database is nowhere to be found (at least that's what it tells me).
Therefore, I tried to imitate the visual SSIS component inside the data flow task [Destination OLE DB] that uses a connection manager.
Old Setup
OLE DB connection manager setup
OLE DB destination setup
This setup uses OLE DB driver with "SQL Server Native Client 11.0" provider (or simply SQLNCLI11.1), "SSPI" integrated security, "Table or view - fast load" mode of access to data. This setup works perfectly fine locally and on the server.
Desired Setup
Armed with this knowledge I have tried to use OleDbConnection and OleDbCommand classes using this stackoverflow question, but I can't see how to use these components to bulk insert data into the DB.
I have also tried to use the visual SSIS component which is called "Bulk Insert Task", but lo luck there either.
How can I possibly insert in bulk using OLE DB?

How to use Windows Search Service on a local and on a remote machine both?

I am working on a C# WPF project which can search files on specified directory paths. These paths can be on a local and on a remote machine as well, so I have to solve it on both.
At first I tried to use a query which has a local path, but the OleDbDataReader class returns with zero records. I am sure that I have adjusted everything on the corresponding way, e.g. Indexing Options contains the correct path: - or in file explorer: - and File Types contains txt file in the list:
Code to the local searching:
CSearchManager manager = new CSearchManager();
CSearchCatalogManager catalogManager = manager.GetCatalog("SystemIndex");
CSearchQueryHelper queryHelper = catalogManager.GetQueryHelper();
queryHelper.QuerySelectColumns = "System.ItemName,System.FileName,System.Author,System.ItemUrl,System.ItemType";
queryHelper.QueryWhereRestrictions = #"AND SCOPE='file:D:\testIndexing' AND System.FileName LIKE '*.txt'";
string userQuery = "SELECT System.FileName FROM SystemIndex WHERE SCOPE='file:D:\testIndexing' AND System.FileName LIKE '*.txt'";
string sqlQuery = queryHelper.GenerateSQLFromUserQuery(userQuery);
OleDbConnection conn = new OleDbConnection(queryHelper.ConnectionString);
conn.Open();
OleDbCommand command = new OleDbCommand(sqlQuery, conn);
using (OleDbDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
Console.WriteLine(reader.GetString(0));
Console.WriteLine(reader[0]);
}
}
conn.Close();
Later I tried to connect to a Windows Server. The server path is correct because I can browse the server files via File Explorer - this is a VPN connection -. The Indexing Options are correctly adjusted here as well. But the problem is that I can not use the connection method above, because the GetCatalog methot is not supports the remote connection - so I tried THIS method. I found an example which can see below, but here I got an exeption like:
System.UnauthorizedAccessException: 'Retrieving the COM class factory for remote component with CLSID {[CLSID]} from machine [SERVER NAME] failed due to the following error: 80070005 [SERVER NAME].'
The problem is when I tried to add the CLSID. I have no idea where can I get this ID because I didn't find any concrete solution for this. Here I want some help like what is the CLSID in my case, and how to get it?
Code to the remote searching:
Guid guid = new Guid("{[CLSID]}"); // this is what I don't know how to provide
Type managerType = Type.GetTypeFromCLSID(guid, "SERVER_NAME", true);
var comManager = Activator.CreateInstance(managerType);
CSearchManagerClass manager = (CSearchManagerClass)System.Runtime.InteropServices.Marshal.CreateWrapperOfType(comManager, typeof(CSearchManagerClass));
CSearchCatalogManager catalogManager = manager.GetCatalog("SERVER_NAME.SystemIndex");
CSearchQueryHelper queryHelper = catalogManager.GetQueryHelper();
queryHelper.QuerySelectColumns = "System.ItemName,System.FileName,System.Author,System.ItemUrl,System.ItemType";
queryHelper.QueryWhereRestrictions = #"AND SCOPE='file:\\SERVER_NAME\path\to\scope' AND System.FileName LIKE '*.txt'";
string userQuery = #"SELECT System.FileName FROM SystemIndex WHERE SCOPE='file:\\SERVER_NAME\path\to\scope' System.FileName LIKE '*.txt'";
string sqlQuery = queryHelper.GenerateSQLFromUserQuery(userQuery);
OleDbConnection conn = new OleDbConnection(queryHelper.ConnectionString);
conn.Open();
OleDbCommand command = new OleDbCommand(sqlQuery, conn);
using (OleDbDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
Console.WriteLine(reader.GetString(0));
Console.WriteLine(reader[0]);
}
}
conn.Close();

MS Access OleDB object reference not set to an instance of an object

I have created a .Net 4.5 application that reads and writes to MS Access database. While running the application on my machine (which has MS office) everything is fine but when I run this on my server (which does not has MS Office but has ACE ) I get the object reference not set to an instance of an object.
This happens when I click the button. Following is the code
try
{
OleDbConnection conn = new OleDbConnection(connectionString);
conn.Open();
est_index = getTableIndex(conn,estimateTableIndex);
for (int i = 0; i < inputValues.Count; i++)
{
string FieldName = "";
string FieldValue = "";
if (inputValues[i] is System.Web.UI.WebControls.TextBox)
{
FieldName = inputValues[i].ClientID;
FieldValue = ((System.Web.UI.WebControls.TextBox)(inputValues[i])).Text;
}
else
{
FieldName = inputValues[i].ClientID;
FieldValue = ((System.Web.UI.WebControls.DropDownList)(inputValues[i])).SelectedValue;
}
//everything fine till here..then throws the error below
string my_querry = "INSERT INTO Estimate VALUES(#Est_id,#FieldName,#FieldValue)";
OleDbCommand cmd = new OleDbCommand(my_querry, conn);
cmd.Parameters.AddWithValue("#Est_id", est_index);
cmd.Parameters.AddWithValue("#FieldName", FieldName);
cmd.Parameters.AddWithValue("#FieldValue", FieldValue);
cmd.ExecuteNonQuery();
}
conn.Close();
}
catch (Exception e)
{
Console.Write(e.InnerException.Message);
}
Not sure why this is happening ?
EDIT: I created a virtual machine and downloaded the ACE and the same error occurred that clients face
EDIT 2: The connection string looks like "Provider = Microsoft.ACE.OLEDB.12.0; Data Source = C:\cohber.accdb"
I tried to debug and see where the code broke and it broke at cmd.ExecuteNonQuery();. The message at exception was operation must use an updateable query
Edit 3: the MS Access file is located in c driver. I gave all the users full permission but still get the error

You need to set read/ write permissions to the folder containing the access .mdb file

OleDbDataAdapter.Fill(DataTable) not working with big xls file size

I'm trying to get a DataTable, reading a .xls file.
If I run the below code with a .xls file with size 25kb, it works fine, but if I load a bigger size file (7,52MB), it doesn't work.
string filenamePath = System.IO.Path.Combine(Server.MapPath("/Uploads"), FileUpload1.FileName);
FileUpload1.SaveAs(filenamePath);
string[] validFileTypes = { "xls", "xlsx"};
string ext = System.IO.Path.GetExtension(FileUpload1.PostedFile.FileName);
bool isValidFile = false;
string fileName = FileUpload1.FileName;
for (int i = 0; i < validFileTypes.Length; i++)
{
if (ext == "." + validFileTypes[i])
{
isValidFile = true;
break;
}
}
if (isValidFile)
{
DataTable dt = ConvertXLSTpXLM.convertXLSToDb(filenamePath, "ELEMENT", "ELEMENTS", true, fileName);
}
This is the convertXLSToDb method
public static DataTable convertXLSToDb(string filePath, string firstElement, string secondElement, bool hasHeaders, string filename)
{
DataTable dtexcel = new DataTable();
string HDR = hasHeaders ? "Yes" : "No";
string strConn;
if (filePath.Substring(filePath.LastIndexOf('.')).ToLower() == ".xls")
strConn = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + filePath + ";Extended Properties=\"Excel 12.0;HDR=" + HDR + ";IMEX=0\"";
else
strConn = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + filePath + ";Extended Properties=\"Excel 8.0;HDR=" + HDR + ";IMEX=0\"";
OleDbConnection conn = new OleDbConnection(strConn);
conn.Open();
DataTable schemaTable = conn.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, new object[] { null, null, null, "TABLE" });
DataRow schemaRow = schemaTable.Rows[0];
string sheet = schemaRow["TABLE_NAME"].ToString();
if (!sheet.EndsWith("_"))
{
string query = "SELECT * FROM [" + sheet + "]";
OleDbDataAdapter daexcel = new OleDbDataAdapter(query, conn);
dtexcel.Locale = CultureInfo.CurrentCulture;
daexcel.Fill(dtexcel);
}
conn.Close();
return dtexcel;
}
No error occurs, but on the "daexcel.Fill(dtexcel);" the page runs infinitely.

if I run the below code with a .xls file with size 25kb, it works fine, but if I load a bigger size file (7,52MB), it doesn't work.
Putting the fact that you don't Dispose a lot of IDisposable resources to one side, it's possible that there's nothing wrong with the size of the file, rather the problem is the contents of the file. That is to say, you might have a row with "curious" data and if you were to get that curious data into the 25kb file it would not work, either.
Microsoft explicitly state that using ACE in a service environment is a bad idea:
The Access Database Engine 2010 Redistributable is not intended:
...
To be used by a system service or server-side program where the code will run under a system account, or will deal with multiple users identities concurrently, or is highly reentrant and expects stateless behavior. Examples would include a program that is run from task scheduler when no user is logged in, or a program called from server-side web application such as ASP.NET, or a distributed component running under COM+ services.
Similar caveats apply to JET but they're not documented quite as explicitly.
To diagnose where the problem is, get the code out of ASP .NET and put it into an interactive console application and run it.
If the code still exhibits problems you could consider pausing the process with the debugger and trying to inspect where the code has "frozen". Being paused on GetOleDbSchemaTable instead of Fill might be illuminating.
Beyond that, given that neither JET nor ACE are recommended in a server environment you might consider giving up .xsl support and using something like EPPlus to read your .xlsx files.

The File Upload control can by default upload files upto 4MB only. Refer the Note on the page below.
http://msdn.microsoft.com/en-us/library/ms227669(v=vs.90).aspx
You can change the httpRuntime maxRequestLength in the web config as mentioned below.
How to programmatically set (using GET SET property) "httpRuntime maxRequestLength" in ASP.NET with C# as code behind
Hope this helps!

C# Open DBF file

I'm having a problem opening a DBF file - I need to open it, read everything and process it. I tried several solutions (ODBC/OLEDB), several connection string, but nothing worked so far.
The problem is, when I execute the SQL command to get everything from the file, nothing gets returned - no rows. What's even more odd, the content of the DBF file being opened get deleted.
See the code I have:
public override bool OpenFile(string fileName, string subFileName = "")
{
OleDbConnection con = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + Path.GetDirectoryName(fileName) + ";Extended Properties=dBASE IV;User ID=;Password=;");
try
{
if (con.State == ConnectionState.Closed) { con.Open(); }
OleDbDataAdapter da = new OleDbDataAdapter("select * from " + Path.GetFileName(fileName), con);
DataSet ds = new DataSet();
da.Fill(ds);
con.Close();
int i = ds.Tables[0].Rows.Count;
return true;
}
catch
{
return false;
}
}
I debugged the code and watched the file being opend in Windows Explorer. When it reached this line:
da.Fill(ds);
the size of the file dropped to only a few Bytes (from hundreds of kB).
My next thought was to make the DBF file read only. That however cause an "unexpected exception from an external driver".
So my question is - what the heck? I'm sure the file is not corrupt, it is a direct export from some DB. (No, I do not have access to that DB). I can also open that file in MS Office no problem.
I cannot share the DBF file - it contains confidential data.

Two things... just because its a .DBF file extension might night mean its a Dbase IV file. It might actually be that of Visual Foxpro. That said, I would look into downloading and installing the Visual Foxpro OleDB driver from Microsoft download. Next, the OleDbConnection is pointing to the path that has the actual tables (you already have that).
The query itself, shouldn't care about the extension, so I would change your call to get just then name via "Path.GetFileNameWithoutExtension"
It might be a combination of the two.
Connection string for VFP provider
"Provider=VFPOLEDB.1;Data Source=" + FullPathToDatabase

This is not the exact answer but it will help you to find the issue.
Try to give an inline connection string and select query to make sure problem is not with building those. Catch the exception and check the details of it.
OleDbConnection con = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=c:\folder;Extended Properties=dBASE IV;User ID=;Password=;"); // give your path directly
try
{
con.Open();
OleDbDataAdapter da = new OleDbDataAdapter("select * from tblCustomers.DBF", con); // update this query with your table name
DataSet ds = new DataSet();
da.Fill(ds);
con.Close();
int i = ds.Tables[0].Rows.Count;
return true;
}
catch(Exception e)
{
var error = e.ToString();
// check error details
return false;
}