Develop Reference

SQL job failed to run a C#.net console application with an status message: 403

I have a console application that uses credential to connect Share Point Online from "Windows credential manager" and request Share Point Online to download all items from list and further this information is used to start the procedure on the SQL Server
I'm not experienced in writing such application, but particular this application works normally when it executes from Visual Studio or by executing file. But it fails with error code 403 when it executes by SQL Agent on the same computer.
For clarification: SQL Server runs on the same computer where I develop the application and where is Share Point credential is stored.
I start job under my windows account using proxy, so I suppose there is no problem with credential. My windows account has sysadmin permissions on the SQL Server and admin persmissions on the OS.
But after searching for whole day I have no idea where the mistake could be.
Please help me with advice and tell me where I made a mistake.
Thanks in advance.
There is output error from SQL Job:
Message
Executed as user: LAPTOP\username. Unhandled Exception:System.Net.WebException:
The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.SharePoint.Client.SPWebRequestExecutor.Execute()
at Microsoft.SharePoint.Client.ClientContext.GetFormDigestInfoPrivate()
at Microsoft.SharePoint.Client.ClientContext.EnsureFormDigest()
at Microsoft.SharePoint.Client.ClientContext.ExecuteQuery()
at SharePointTrigger.Program.GetNewItem(String targetSiteUrl, String listName,
String filedName) in
C:\Work\Korus\Pernod\Vista\Repose\ETL_LoadDataCSV\SharePointTrigger\Program.cs:line 114
at SharePointTrigger.Program.Main(String[] args) in
C:\Work\Korus\Pernod\Vista\Repose\ETL_LoadDataCSV\SharePointTrigger\Program.cs:line 40.
Process Exit Code -532462766. The step failed.
And there is console application (there is no App.config file and DatabaseOperation class. However DatabaseOperation class contains only method for executing procedure on the SQL Server):
using System;
using System.Collections.Generic;
using System.Data.SqlClient;
using System.Linq;
using System.Security;
using System.Text;
using System.Threading.Tasks;
using System.Web.UI.WebControls;
using Microsoft.Graph;
using Microsoft.SharePoint.Client;
using OfficeDevPnP.Core.Utilities;
using SP = Microsoft.SharePoint.Client;
using System.Configuration;
using System.Collections.Specialized;
using static SharePointTrigger.DatabaseOperation;
namespace SharePointTrigger
{
class Program
{
static void Main(string[] args)
{
//To get values from App.config
string SQLServerName = ConfigurationManager.AppSettings.Get("SQLServerName");
string SQLDatabaseName = ConfigurationManager.AppSettings.Get("SQLDatabaseName");
string SQLProcedureName = ConfigurationManager.AppSettings.Get("SQLProcedureName");
string SQLJobName = ConfigurationManager.AppSettings.Get("SQLJobName");
string SPSiteURL = ConfigurationManager.AppSettings.Get("SPSiteURL");
string SPListName = ConfigurationManager.AppSettings.Get("SPListName");
string SPStatusColumnName = ConfigurationManager.AppSettings.Get("SPStatusColumnName");
string SPLoadIdColumnName = ConfigurationManager.AppSettings.Get("SPLoadIdColumnName");
string SSISVariable_name = ConfigurationManager.AppSettings.Get("SSISVariable_name");
string SSISEnvironment_name = ConfigurationManager.AppSettings.Get("SSISEnvironment_name");
string SSISFolder_name = ConfigurationManager.AppSettings.Get("SSISFolder_name");
int itemCounter = 0;
JobResult jobResult = new JobResult();
List<MyItem> myItemList = GetNewItem(SPSiteURL, SPListName, SPStatusColumnName);
myItemList.Sort((x, y) => y.id.CompareTo(x.id));
if (myItemList.Count() > 0)
{
//To avoid reload the same items
foreach (MyItem item in myItemList)
{
List<FieldValue> fieldValueList = new List<FieldValue>();
fieldValueList.Add(new FieldValue() { fieldName = SPStatusColumnName, fieldValue = "In progress" });
UpdateListItem(SPSiteURL, SPListName, item.id, fieldValueList);
}
//To start jon
MyItem myItem = myItemList[0];
int SPListId = myItem.id;
SqlConnection sqlconnection = DatabaseOperation.GetSqlConnection(SQLServerName, SQLDatabaseName);
jobResult = DatabaseOperation.StartSQLJob(sqlconnection, SQLProcedureName, SQLJobName, SSISVariable_name, SSISEnvironment_name, SSISFolder_name, SPListId);
}
//Update items fields Status, Loadid
itemCounter = 0;
foreach (MyItem myItem in myItemList)
{
if (itemCounter == 0)
{
List<FieldValue> fieldValueList = new List<FieldValue>();
fieldValueList.Add(new FieldValue() { fieldName = SPStatusColumnName, fieldValue = jobResult.result });
fieldValueList.Add(new FieldValue() { fieldName = SPLoadIdColumnName, fieldValue = jobResult.loadId });
UpdateListItem(SPSiteURL, SPListName, myItem.id, fieldValueList);
}
else
{
List<FieldValue> fieldValueList = new List<FieldValue>();
fieldValueList.Add(new FieldValue() { fieldName = SPStatusColumnName, fieldValue = "Missed" });
UpdateListItem(SPSiteURL, SPListName, myItem.id, fieldValueList);
}
itemCounter += 1;
}
}
//Get all items fro the list
private static List<MyItem> GetNewItem(string targetSiteUrl, string listName, string filedName)
{
List<MyItem> myItemList = new List<MyItem>();
using (ClientContext context = new ClientContext(targetSiteUrl))
{
context.Credentials = CredentialManager.GetSharePointOnlineCredential(targetSiteUrl);
Web myWeb = context.Web;
SP.List myList = myWeb.Lists.GetByTitle(listName);
SP.ListItemCollection listItemCollection = myList.GetItems(CamlQuery.CreateAllItemsQuery());
context.Load(listItemCollection,
eachItem => eachItem.Include(
item => item,
item => item["Title"],
item => item["ID"],
item => item[filedName] //Field: "Status"
)
);
// ExecuteQuery will pull all data from SharePoint
// which has been staged to Load()
context.ExecuteQuery();
foreach (SP.ListItem listItem in listItemCollection)
{
if ((string)listItem[filedName] == "New")
{
MyItem myItem = new MyItem();
myItem.id = (int)listItem["ID"];
myItem.title = (string)listItem["Title"];
myItem.status = (string)listItem[filedName];
myItemList.Add(myItem);
}
}
}
return myItemList;
}
//Update items fields
static void UpdateListItem(string targetSiteUrl, string listName, int itemId, List<FieldValue> fieldValuesList)
{
using (ClientContext context = new ClientContext(targetSiteUrl))
{
context.Credentials = CredentialManager.GetSharePointOnlineCredential(targetSiteUrl);
//List
SP.List announcementsList = context.Web.Lists.GetByTitle(listName);
//List item
SP.ListItem listItem = announcementsList.GetItemById(itemId);
foreach (FieldValue fieldValue in fieldValuesList)
{
//Change field value
listItem[fieldValue.fieldName] = fieldValue.fieldValue;
}
listItem.Update();
context.ExecuteQuery();
}
}
private class MyItem
{
public int id;
public string title;
public string status;
}
private class FieldValue
{
public string fieldName;
public string fieldValue;
}
}
}
Job step settings look like:
CMD step

When VS is used it uses normally the build-in webserver as fa as I can remember - deploying to a real IIS may end in trouble. Please have a look here
https://en.it1352.com/article/530a9f2f660b4088a9637b7d294194af.html
Anyway - in some cases TLS missing for authentication cause trouble

Have you checked the user permission on SQL?
(here a guide to create and configure an user to use SQL Server Agent Job)

The problem is that I doesn't use access token to enter the site. The application works fine in Visual Studio because previously received access token was kept in the cache. I've rewrite this part using example from here https://www.c-sharpcorner.com/article/sharepoint-csom-for-net-standard/

How to update a SharePoint Online list if a new record is inserted in the database?

I have a MySQL database that holds personal information. Whenever a new employee gets hired he/she fills out some personal information and that data gets stored in a table.
After some research (and since I don't have access to the other systems -only the database) the plan is to build a C# console app that retrieves the data and check it against the SharePoint list. I want to update the list (create a new item) if a new record is in the database that does not exist in the SharePoint list from before.
Note that if the SharePoint list contains more columns, then the table with additional manual information.
I have posted the connection code against the database and how I retrieve the data.
How can I check if the item exists in the SharePoint list? Would anybody be able to provide an answer that includes code for creating and inserting the new item? I have two columns (in both the database and SP list) that could work as a primary key.
There is a REST API that supports CRUD so I guess this should be a no-brainer.
SharePoint list:
using System;
using System.Windows;
public class DbConnection
{
private String databaseName;
private String serverAddress;
private String pwd;
private String userName;
private Boolean connected;
private MySql.Data.MySqlClient.MySqlConnection conn;
public DbConnection(String databaseName, String serverAddress, String pwd, String userName)
{
this.databaseName = databaseName;
this.serverAddress = serverAddress;
this.pwd = pwd;
this.userName = userName;
connected = false;
}
public void Connect()
{
if (connected == true)
{
Console.Write("There is already a connection");
}
else
{
connected = false;
String connectionString = "server=" + serverAddress + ";" + "database=" + databaseName + ";" + "uid=" + userName + ";" + "pwd=" + pwd + ";";
Console.WriteLine(connectionString);
try
{
conn = new MySql.Data.MySqlClient.MySqlConnection(connectionString);
conn.Open();
Console.Write("Connection was succesfull");
}
catch (MySql.Data.MySqlClient.MySqlException ex)
{
MessageBox.Show(ex.Message);
}
}
}
public Boolean IsConnected()
{
return connected;
}
public MySql.Data.MySqlClient.MySqlConnection getConnection()
{
return conn;
}
public void Close()
{
conn.Close();
connected = false;
}
}
Then I retrieve the data like so:
using MySql.Data.MySqlClient;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace daily_CC_SP_update
{
class Program
{
static void Main()
{
DbConnection mySQLConn = new DbConnection(dbName, serverAddress, pwd, userName);
mySQLConn.Connect();
string sqlQuery = "SELECT * FROM tbl_CC_SP";
MySqlCommand sqlCom = new MySqlCommand(sqlQuery, mySQLConn.getConnection());
MySqlDataReader reader = sqlCom.ExecuteReader();
Console.WriteLine("Following output from DB");
if(reader.Read())
{
Console.WriteLine(reader.GetString(0));
}
//Keep the console alive until enter is pressed, for debugging
Console.Read();
mySQLConn.Close();
}
}
}
I will create a view in the database to retrieve the correct data.

First just to clarify :).. You are using SharePoint on-prem right? So we can use Farm solutions.
If yes then I would resolve this case with the fallowing solution.
I would develop an SPJob (SharePoint Timer job). It may only be included in Farm solution. Basically it looks like this:
create Farm project in solution
add class that inherits from SPJobDefinition and put Your logic in Execute method which You need to override (in this method create a standard SQL connection and Query this table from mySQL db then compare with Your SPList and do the work :) ) (also maybe here a good approach would be to store some credentials for this connection string in some SPList on some config site or somewhere... not to hardcode it ;))
For e.g.
public class CustomJob : SPJobDefinition
{
public CustomJob() : base() { }
public CustomJob(string jobName, SPService service) : base(jobName, service, null, SPJobLockType.None)
{
this.Title = jobName;
}
public CustomJob(string jobName, SPWebApplication webapp) : base(jobName, webapp, null, SPJobLockType.ContentDatabase)
{
this.Title = jobName;
}
public override void Execute(Guid targetInstanceId)
{
SPWebApplication webApp = this.Parent as SPWebApplication;
try
{
// Your logic here
}
catch (Exception ex)
{
SPDiagnosticsService.Local.WriteTrace(0, new SPDiagnosticsCategory("CustomJob - Execute", TraceSeverity.Unexpected, EventSeverity.Error), TraceSeverity.Unexpected, ex.Message, ex.StackTrace);
}
}
}
add new feature to the solution that scope is webApplication and add event receiver to this feature
on feature active register Your Timer job (remember to remove it on deactive :))
public class Feature2EventReceiver : SPFeatureReceiver
{
const string JobName = "CustomJob";
public override void FeatureActivated(SPFeatureReceiverProperties properties)
{
try
{
SPSecurity.RunWithElevatedPrivileges(delegate ()
{
// add job
SPWebApplication parentWebApp = (SPWebApplication)properties.Feature.Parent;
DeleteExistingJob(JobName, parentWebApp);
CreateJob(parentWebApp);
});
}
catch (Exception ex)
{
SPDiagnosticsService.Local.WriteTrace(0, new SPDiagnosticsCategory("CustomJob-FeatureActivated", TraceSeverity.Unexpected, EventSeverity.Error), TraceSeverity.Unexpected, ex.Message, ex.StackTrace);
}
}
public override void FeatureDeactivating(SPFeatureReceiverProperties properties)
{
lock (this)
{
try
{
SPSecurity.RunWithElevatedPrivileges(delegate ()
{
// delete job
SPWebApplication parentWebApp = (SPWebApplication)properties.Feature.Parent;
DeleteExistingJob(JobName, parentWebApp);
});
}
catch (Exception ex)
{
SPDiagnosticsService.Local.WriteTrace(0, new SPDiagnosticsCategory("CustomJob-FeatureDeactivating", TraceSeverity.Unexpected, EventSeverity.Error), TraceSeverity.Unexpected, ex.Message, ex.StackTrace);
}
}
}
private bool CreateJob(SPWebApplication site)
{
bool jobCreated = false;
try
{
// schedule job for once a day
CustomJob job = new CustomJob(JobName, site);
SPDailySchedule schedule = new SPDailySchedule();
schedule.BeginHour = 0;
schedule.EndHour = 1;
job.Schedule = schedule;
job.Update();
}
catch (Exception)
{
return jobCreated;
}
return jobCreated;
}
public bool DeleteExistingJob(string jobName, SPWebApplication site)
{
bool jobDeleted = false;
try
{
foreach (SPJobDefinition job in site.JobDefinitions)
{
if (job.Name == jobName)
{
job.Delete();
jobDeleted = true;
}
}
}
catch (Exception)
{
return jobDeleted;
}
return jobDeleted;
}
}
deploy and activate Your feature on the web app (I think the best would be to configure the job to run every day or every hour)
a nice article with some example how to do that all may be found here (I know the article is for SP 2010 but it will work the same for 2013, 2016 and probably also 2019 (with this on-prem version I don't have much exp) :)
another article with same solution here (this for SP 2013)
** Update **
for SharePoint Online the above solution will not work as it is a farm solution. In Online the solution is as always something 'external' :).
For sure You already have some kind of server were You store solutions for SP online (like SP apps that are provider hosted.. etc).
My approach would be to develop a simple C# console app. First in this app do a SQL connection to mySql and query the table to get the data.. then using CSOM query SharePoint List to do the compare.
something like this
using (var clientContext = new ClientContext("url"))
{
CamlQuery camlQuery = new CamlQuery();
string query = "add some query here";
camlQuery.ViewXml = query;
collListItem = list.GetItems(camlQuery);
clientContext.Load(collListItem, items => items.Include( item => item["Title"], item => .... // add other columns You need here);
clientContext.ExecuteQuery();
if (collListItem.Count > 0)
{
// Your code here :)
}
}
Also be aware that You can run CSOM with credentials of different user (like some kind of admin) giving the network credentials like this:
NetworkCredential _myCredentials = new NetworkCredential("user", "password", "companydomain");
Also please be aware about threshold... in CSOM You can always use paginated query were You first get 5000 items after that under 5000 etc etc. until the collection is empty :).
After You run this console app manually a couple of times to be sure it is working properly, simply add this console app to task Scheduler on this server as a new task in task library. Also there You can provide trigger time like run every hour or day etc. here is a nice stack overflow post how to add this kind of task
.. I hope the answer now is better for Your environment :)

So I have had a great progress with my c# program. I have a fully functional connection between MySql database and SharePoint online using MySql.Data CSOM. Can manipulate and control all lists and data inside.
However I have one problem, nut not sure if this can be solved. Little to non information I could find around this topic.
I create a new ListItem. Set all the fields to a value. But there is one column that is of type "Person". Every employee has his own site where this links to, like Lookup. When adding a value to this field the server gives me the following error:
Microsoft.SharePoint.Client.ServerException: Invalid data has been used to update the list item. The field you are trying to update may be read only.
at Microsoft.SharePoint.Client.ClientRequest.ProcessResponseStream(Stream responseStream)
at Microsoft.SharePoint.Client.ClientRequest.ProcessResponse()
at Microsoft.SharePoint.Client.ClientRequest.ExecuteQueryToServer(ChunkStringBuilder sb)
at Microsoft.SharePoint.Client.ClientContext.ExecuteQuery()
at SPList.CreateNewItem(String userName, Int32 employeeNumber, String fullName, String firstName, String lastName, DateTime emplymentStart, DateTime employmentEnd, String department, String mobile, String address, String postcode, String postTown, String email) in C:\Users\fjs\source\repos\daily_CC_SP_update\SPList.cs:line 153
SharePoint field spec
Here is the code where I am creating the new Item.
using System;
using Microsoft.SharePoint.Client;
using System.Linq;
using System.Net;
public class SPList
{
private readonly ClientContext context;
private readonly List list;
private readonly ListItemCollection items;
private readonly Web web;
//Credentials may be needed, its commented out!
public SPList(String siteUrl, String listName, NetworkCredential netCred)
{
try
{
//NetworkCredential _myCredentials = netCred;
context = new ClientContext(siteUrl);
list = context.Web.Lists.GetByTitle(listName);
items = list.GetItems(CamlQuery.CreateAllItemsQuery());
web = context.Web;
context.Load(items);
context.Load(list);
context.Load(context.Web.Lists, lists => lists.Include(list => list.Title));
context.ExecuteQuery();
Console.WriteLine("Connected to SharePoint successfully!");
}
catch(Exception e)
{
Console.WriteLine(e);
}
}
public void CreateNewItem(String userName, int employeeNumber, String fullName, String firstName, String lastName, DateTime emplymentStart, DateTime employmentEnd, String department, String mobile, String address, String postcode, String postTown, String email)
{
try
{
ListItemCreationInformation newItemSepc = new ListItemCreationInformation();
ListItem newItem = list.AddItem(newItemSepc);
newItem["Title"] = userName;
newItem["Employee_x0020_Number"] = employeeNumber;
newItem["Full_x0020_Name"] = fullName;
newItem["First_x0020_Name"] = firstName;
newItem["Last_x0020_Name"] = lastName;
newItem["_x000a_Employment_x0020_start_x0"] = emplymentStart.Date;
newItem["Employment_x0020_end_x0020_date"] = employmentEnd.Date;
newItem["Department"] = department;
newItem["Mobile"] = mobile;
newItem["Adress"] = address;
newItem["Postcode"] = postcode;
newItem["Post_x0020_town"] = postTown;
newItem["Email"] = email;
newItem["Person"] = fullName;
newItem.Update();
context.ExecuteQuery();
}
catch(Exception e)
{
Console.WriteLine(e);
}
}
}
If i comment newItem["Person"] = fullName; out the code works fine. Can this somehow be fixed? Otherwise I have to eddid the item in SharePoint and add the value :/
The weird field names is because SharePoint stores it this way for some reason

The solution is to set the items["LockUpColumn"] not to a string but a lockup field

Take too long time to write in to Access Data Base using C# WPF application, Visual Studio,

I have a WPF application using C# and VS.
And I am using an Access database.
I have a loop that has to run in a maximum time of 500MS, But its take 570+-
In my program, I have a wait time of ~340MS in total and more ~160MS that I can to optimize
After checking with a Stopwatch I found that when I write my data to my Access Database Its take about ~50MS (I have a 3 writes to there).
And I have no Idea how to optimize my Database write
My Class that connect and using the database is an external DLL file
that look like that (I also give an example of one method that take a 50MS of runtime, named as "AddDataToLocalHeaderResult"):
namespace DataBaseManager
{
public class LocalPulserDBManager
{
private string localConnectionString;
private string databaseName = $#"C:\Pulser\LocalPulserDB.mdb";
private readonly int _30DaysBack = -30;
private static readonly Lazy<LocalPulserDBManager> lazy =new Lazy<LocalPulserDBManager>(() => new LocalPulserDBManager());
public static LocalPulserDBManager LocalPulserDBManagerInstance { get { return lazy.Value; } }
private void CreateConnectionString()
{
localConnectionString = $#"Provider=Microsoft.Jet.OLEDB.4.0;Data Source={databaseName};Persist Security Info=True";
}
private LocalPulserDBManager()
{
CreateConnectionString();
}
public void AddDataToLocalHeaderResult(string reportNumber,string reportDescription,
string catalog,string workerName,int machineNumber, Calibration c,string age)
{
if (IsHeaderLocalDataExist(reportNumber, catalog, machineNumber, c) == false)
{
using (OleDbConnection openCon = new OleDbConnection(localConnectionString))
{
string query = "INSERT into [HeaderResult] ([ReportNumber],[ReportDescription],[CatalogNumber], " +
"[WorkerName], [LastCalibrationDate], [NextCalibrationDate], [MachineNumber], [Age]) " +
"VALUES (#report ,#reportDescription ,#catalog, #workerName," +
" #LastCalibrationDate, #NextCalibrationDate, #machineNumber, #age)";
using (OleDbCommand command = new OleDbCommand(query))
{
command.Parameters.AddWithValue("#report", reportNumber);
command.Parameters.AddWithValue("#reportDescription", reportDescription);
command.Parameters.AddWithValue("#catalog", catalog);
command.Parameters.AddWithValue("#workerName", workerName);
command.Parameters.AddWithValue("#LastCalibrationDate", c.LastCalibrationDate);
command.Parameters.AddWithValue("#NextCalibrationDate", c.NextCalibrationDate);
command.Parameters.AddWithValue("#machineNumber", machineNumber);
command.Parameters.AddWithValue("#age", age);
command.Connection = openCon;
openCon.Open();
int recordsAffected = command.ExecuteNonQuery();
openCon.Close();
}
}
}
}
....
....
METHODS
....
}
}
In my executable program I use that like that :
I have usings as that : using static DataBaseManager.LocalPulserDBManager;
and in my code I exeute the method like that LocalPulserDBManagerInstance.AddDataToLocalHeaderResult(ReportNumber, Date_Description,CatalogNumber, WorkerName, (int)MachineNumber, calibrationForSave, AgeCells);
One of my access database table look like that :
One row in that table look like that:
50MS it is normal runtime in that situation?
If here is missing any information please tell me...
********************* EDITING **************************
I have change my AddDataToLocalHeaderResult method as the first command told me
I got the same result
public void AddDataToLocalHeaderResult(string reportNumber,string reportDescription,
string catalog,string workerName,int machineNumber, Calibration c,string age)
{
if (IsHeaderLocalDataExist(reportNumber, catalog, machineNumber, c) == false)
{
using (OleDbConnection openCon = new OleDbConnection(localConnectionString))
{
string query = "INSERT into [HeaderResult] ([ReportNumber],[ReportDescription],[CatalogNumber], " +
"[WorkerName], [LastCalibrationDate], [NextCalibrationDate], [MachineNumber], [EditTime], [Age]) " +
"VALUES (#report ,#reportDescription ,#catalog, #workerName," +
" #LastCalibrationDate, #NextCalibrationDate, #machineNumber,#edittime, #age)";
DateTime dt = DateTime.Now;
DateTime edittime = new DateTime(dt.Year, dt.Month, dt.Day, dt.Hour, dt.Minute, dt.Second);
using (OleDbCommand command = new OleDbCommand(query))
{
command.Parameters.AddWithValue("#report", reportNumber);
command.Parameters.AddWithValue("#reportDescription", reportDescription);
command.Parameters.AddWithValue("#catalog", catalog);
command.Parameters.AddWithValue("#workerName", workerName);
command.Parameters.AddWithValue("#LastCalibrationDate", c.LastCalibrationDate);
command.Parameters.AddWithValue("#NextCalibrationDate", c.NextCalibrationDate);
command.Parameters.AddWithValue("#machineNumber", machineNumber);
command.Parameters.AddWithValue("#edittime", edittime);
command.Parameters.AddWithValue("#age", age);
command.Connection = openCon;
openCon.Open();
int recordsAffected = command.ExecuteNonQuery();
openCon.Close();
}
}
}
}

Using the method you're showing here, you're adding one row at a time. So the server is opening a connection to the db, the data's being written to memory, then to the physical file (mdb), then the indexes are being updated. To that's a full four steps per row you're trying to execute. Worse than that, the data write to the physical file is time consuming.
I think that if you use a different approach, do these four steps (connection, memory, data write, re-index) for the entire set of data you're trying to insert. So, let's say you're adding 1000 records, rather than 4000 steps (4x1000), you could reduce this processing to 1400 processing steps (1 connection, super-fast 1000 memory writes, 1 data file write, 1 index revision).
The following code gives the rough idea of what I'm talking about:
class Program
{
static void Main(string[] args)
{
//memory-only list for data loading
List<HeaderResult> mylist = new List<HeaderResult>(){ new HeaderResult("report1","desc of report","ete"), new HeaderResult("report2", "desc of report2", "ete2")};
var tableForInsert = new DataTable();
using (SqlDataAdapter dataAdapter = new SqlDataAdapter("SELECT * from HeaderResult", "my conneciton string")) {
dataAdapter.Fill(tableForInsert);
//now I have a live copy of the table into which I want to insert data
//blast in the data
foreach (HeaderResult hr in mylist) {
tableForInsert.Rows.Add(hr);
}
//now all the data is written at once and sql will take care of the indexes after the datat's written
dataAdapter.Update(tableForInsert);
}
}
//class should have same fields as your table
class HeaderResult
{
string report;
string reportDescription;
string etc;
public HeaderResult(string rpt, string desc, string e)
{
report = rpt;
reportDescription = desc;
etc = e;
}
}

C# Converting a var/string to action

My command system works off a database where the entire command is stored.
In the Command creation, the
.SetAction((t, m, c) => { //code });
is what the command actually does.
Pulling from the db, this comes in as a var.
How would I go about converting this to actual code for the Action to actually handle?
Reading through a couple pages from Google leaves me to believe I'll need to use Reflection which I've only touched on with Interfaces.
Snippet of the SetAction
public Command SetAction(Action<string, Message, DiscordClient> action)
{
this.action = action;
return this;
}
The code I am using looks like this:
using (var command = sql.CreateCommand())
{
command.CommandText = "SELECT * FROM commands";
using (var reader = command.ExecuteReader())
{
var actionCol = reader.GetOrdinal("action");
while (reader.Read())
{
var action = reader.GetValue(acionCol);
cm.AddCommand(new Command()
.SetAction((t, m, c,) => { }));
)
}
}

Pooling MySQL Connections with Microsoft Enterprise Library

My setup is MySql.Data.MySqlClient v6.9.8.0 and Microsoft.Practices.EnterpriseLibrary.Data v6.0.0.
The program is a long running program that runs continuously listening for tasks and then performs the job with some form of database action (depending on what the request was.) Sometimes the requests will be one after the other, sometimes there will be several hours between them.
I've tried using Pooling=true in the connection string but it causes me a lot of problems (not all the time - these are intermittent problems.)
Here is an example:
[MySqlException (0x80004005): Authentication to host 'localhost' for user 'root' using method 'mysql_native_password' failed with message: Reading from the stream has failed.]
Turning off pooling fixes the problem but at the same time it makes the queries slower because we can't reuse connections. I've searched online and a lot of people have this same issue and the only fix/workaround I've found is Pooling=false which I'd rather avoid if possible.
Here is an example of my query code:
Database db = this.GetDatabase(databaseName);
List<dynamic> results = new List<dynamic>();
// Run the sql query
using (DbCommand dbCommand = db.GetSqlStringCommand(query))
{
foreach (var parameter in inParameters)
{
db.AddInParameter(dbCommand, parameter.Key, parameter.Value.Item1, parameter.Value.Item2);
}
foreach (var parameter in outParameters)
{
db.AddOutParameter(dbCommand, parameter.Key, parameter.Value.Item1, parameter.Value.Item2);
}
using (IDataReader dataReader = db.ExecuteReader(dbCommand))
{
IDictionary<string, object> instance;
do
{
// Read each row
while (dataReader.Read())
{
instance = new ExpandoObject() as IDictionary<string, object>;
// Populate the object on the fly with the data
for (int i = 0; i < dataReader.FieldCount; i++)
{
instance.Add(dataReader.GetName(i), dataReader[i]);
}
// Add the object to the results list
results.Add(instance);
}
} while (dataReader.NextResult());
}
return results;
}
Any ideas?

Can you try this? I know, I know. using "using" should mean I don't have to call the dataReader.Close() method...but I still do it. I also slightly altered the dr.Read block.
This guy talks about it.
http://www.joseguay.com/uncategorized/ensure-proper-closure-disposal-of-a-datareader
I know, I know. You shouldn't have to. Even when using Ent library, I do an extra .Close step to try and make sure.
Database db = this.GetDatabase(databaseName);
List<dynamic> results = new List<dynamic>();
// Run the sql query
using (DbCommand dbCommand = db.GetSqlStringCommand(query))
{
foreach (var parameter in inParameters)
{
db.AddInParameter(dbCommand, parameter.Key, parameter.Value.Item1, parameter.Value.Item2);
}
foreach (var parameter in outParameters)
{
db.AddOutParameter(dbCommand, parameter.Key, parameter.Value.Item1, parameter.Value.Item2);
}
using (IDataReader dataReader = db.ExecuteReader(dbCommand))
{
IDictionary<string, object> instance;
while (dataReader.Read())
{
instance = new ExpandoObject() as IDictionary<string, object>;
// Populate the object on the fly with the data
for (int i = 0; i < dataReader.FieldCount; i++)
{
instance.Add(dataReader.GetName(i), dataReader[i]);
}
// Add the object to the results list
results.Add(instance);
}
if (dataReader != null)
{
try
{
dataReader.Close();
}
catch { }
}
}
return results;
}