Develop Reference

Take too long time to write in to Access Data Base using C# WPF application, Visual Studio,

I have a WPF application using C# and VS.
And I am using an Access database.
I have a loop that has to run in a maximum time of 500MS, But its take 570+-
In my program, I have a wait time of ~340MS in total and more ~160MS that I can to optimize
After checking with a Stopwatch I found that when I write my data to my Access Database Its take about ~50MS (I have a 3 writes to there).
And I have no Idea how to optimize my Database write
My Class that connect and using the database is an external DLL file
that look like that (I also give an example of one method that take a 50MS of runtime, named as "AddDataToLocalHeaderResult"):
namespace DataBaseManager
{
public class LocalPulserDBManager
{
private string localConnectionString;
private string databaseName = $#"C:\Pulser\LocalPulserDB.mdb";
private readonly int _30DaysBack = -30;
private static readonly Lazy<LocalPulserDBManager> lazy =new Lazy<LocalPulserDBManager>(() => new LocalPulserDBManager());
public static LocalPulserDBManager LocalPulserDBManagerInstance { get { return lazy.Value; } }
private void CreateConnectionString()
{
localConnectionString = $#"Provider=Microsoft.Jet.OLEDB.4.0;Data Source={databaseName};Persist Security Info=True";
}
private LocalPulserDBManager()
{
CreateConnectionString();
}
public void AddDataToLocalHeaderResult(string reportNumber,string reportDescription,
string catalog,string workerName,int machineNumber, Calibration c,string age)
{
if (IsHeaderLocalDataExist(reportNumber, catalog, machineNumber, c) == false)
{
using (OleDbConnection openCon = new OleDbConnection(localConnectionString))
{
string query = "INSERT into [HeaderResult] ([ReportNumber],[ReportDescription],[CatalogNumber], " +
"[WorkerName], [LastCalibrationDate], [NextCalibrationDate], [MachineNumber], [Age]) " +
"VALUES (#report ,#reportDescription ,#catalog, #workerName," +
" #LastCalibrationDate, #NextCalibrationDate, #machineNumber, #age)";
using (OleDbCommand command = new OleDbCommand(query))
{
command.Parameters.AddWithValue("#report", reportNumber);
command.Parameters.AddWithValue("#reportDescription", reportDescription);
command.Parameters.AddWithValue("#catalog", catalog);
command.Parameters.AddWithValue("#workerName", workerName);
command.Parameters.AddWithValue("#LastCalibrationDate", c.LastCalibrationDate);
command.Parameters.AddWithValue("#NextCalibrationDate", c.NextCalibrationDate);
command.Parameters.AddWithValue("#machineNumber", machineNumber);
command.Parameters.AddWithValue("#age", age);
command.Connection = openCon;
openCon.Open();
int recordsAffected = command.ExecuteNonQuery();
openCon.Close();
}
}
}
}
....
....
METHODS
....
}
}
In my executable program I use that like that :
I have usings as that : using static DataBaseManager.LocalPulserDBManager;
and in my code I exeute the method like that LocalPulserDBManagerInstance.AddDataToLocalHeaderResult(ReportNumber, Date_Description,CatalogNumber, WorkerName, (int)MachineNumber, calibrationForSave, AgeCells);
One of my access database table look like that :
One row in that table look like that:
50MS it is normal runtime in that situation?
If here is missing any information please tell me...
********************* EDITING **************************
I have change my AddDataToLocalHeaderResult method as the first command told me
I got the same result
public void AddDataToLocalHeaderResult(string reportNumber,string reportDescription,
string catalog,string workerName,int machineNumber, Calibration c,string age)
{
if (IsHeaderLocalDataExist(reportNumber, catalog, machineNumber, c) == false)
{
using (OleDbConnection openCon = new OleDbConnection(localConnectionString))
{
string query = "INSERT into [HeaderResult] ([ReportNumber],[ReportDescription],[CatalogNumber], " +
"[WorkerName], [LastCalibrationDate], [NextCalibrationDate], [MachineNumber], [EditTime], [Age]) " +
"VALUES (#report ,#reportDescription ,#catalog, #workerName," +
" #LastCalibrationDate, #NextCalibrationDate, #machineNumber,#edittime, #age)";
DateTime dt = DateTime.Now;
DateTime edittime = new DateTime(dt.Year, dt.Month, dt.Day, dt.Hour, dt.Minute, dt.Second);
using (OleDbCommand command = new OleDbCommand(query))
{
command.Parameters.AddWithValue("#report", reportNumber);
command.Parameters.AddWithValue("#reportDescription", reportDescription);
command.Parameters.AddWithValue("#catalog", catalog);
command.Parameters.AddWithValue("#workerName", workerName);
command.Parameters.AddWithValue("#LastCalibrationDate", c.LastCalibrationDate);
command.Parameters.AddWithValue("#NextCalibrationDate", c.NextCalibrationDate);
command.Parameters.AddWithValue("#machineNumber", machineNumber);
command.Parameters.AddWithValue("#edittime", edittime);
command.Parameters.AddWithValue("#age", age);
command.Connection = openCon;
openCon.Open();
int recordsAffected = command.ExecuteNonQuery();
openCon.Close();
}
}
}
}

Using the method you're showing here, you're adding one row at a time. So the server is opening a connection to the db, the data's being written to memory, then to the physical file (mdb), then the indexes are being updated. To that's a full four steps per row you're trying to execute. Worse than that, the data write to the physical file is time consuming.
I think that if you use a different approach, do these four steps (connection, memory, data write, re-index) for the entire set of data you're trying to insert. So, let's say you're adding 1000 records, rather than 4000 steps (4x1000), you could reduce this processing to 1400 processing steps (1 connection, super-fast 1000 memory writes, 1 data file write, 1 index revision).
The following code gives the rough idea of what I'm talking about:
class Program
{
static void Main(string[] args)
{
//memory-only list for data loading
List<HeaderResult> mylist = new List<HeaderResult>(){ new HeaderResult("report1","desc of report","ete"), new HeaderResult("report2", "desc of report2", "ete2")};
var tableForInsert = new DataTable();
using (SqlDataAdapter dataAdapter = new SqlDataAdapter("SELECT * from HeaderResult", "my conneciton string")) {
dataAdapter.Fill(tableForInsert);
//now I have a live copy of the table into which I want to insert data
//blast in the data
foreach (HeaderResult hr in mylist) {
tableForInsert.Rows.Add(hr);
}
//now all the data is written at once and sql will take care of the indexes after the datat's written
dataAdapter.Update(tableForInsert);
}
}
//class should have same fields as your table
class HeaderResult
{
string report;
string reportDescription;
string etc;
public HeaderResult(string rpt, string desc, string e)
{
report = rpt;
reportDescription = desc;
etc = e;
}
}

In ASP.Net Core, is it possible to start streaming JSON results?

I am using ASP.Net Core WebAPI.
I have a method that retrieves 10000 results from the database at a time, but I notice that it takes 1.17s to "wait" and 0.3s for the actual transfer (based on Chrome's network graph).
With the results from the database (postgres) are iterated through the DataReader and converted into a struct, added to a list, and ultimately returned as a JsonResult.
I do not know what to expect exactly for options, but I would like to be able to start returning as soon as possible to make the total request lower. I am also doing this for the first time on this platform, so I may not be doing things the best way.
[HttpGet("{turbine:int}")]
public IActionResult GetBearingTemperature(int turbine)
{
using (var connection = Database.GetConnection())
{
connection.Open();
int? page = GetPage();
var command = connection.CreateCommand();
if (page.HasValue)
{
command.CommandText = #"select turbine, timestamp, mainbearingtemperature from readings where turbine = :turbine limit 10000 offset :offset;";
command.Parameters.AddWithValue("offset", NpgsqlTypes.NpgsqlDbType.Integer, page.Value * 10000);
} else
{
command.CommandText = #"select turbine, timestamp, mainbearingtemperature from readings where turbine = :turbine limit 10000;";
}
command.Parameters.AddWithValue("turbine", NpgsqlTypes.NpgsqlDbType.Integer, 4, turbine);
var reader = command.ExecuteReader();
var collection = new List<BearingTemperature>();
if (reader.HasRows)
{
var bt = new BearingTemperature();
while (reader.Read())
{
bt.Time = reader.GetDateTime(1);
bt.Turbine = reader.GetInt32(0);
bt.Value = reader.GetDouble(2);
collection.Add(bt);
}
return new JsonResult(collection);
}
else
{
return new EmptyResult();
}
}
}
private int? GetPage()
{
if (Request.Query.ContainsKey("page"))
{
return int.Parse(Request.Query["page"]);
}
else return null;
}
struct BearingTemperature
{
public int Turbine;
public DateTime Time;
public double Value;
}

So I know this question is old, but this is very much possible in Asp.Net Core 2.2 (probably even from earlier versions, ever since IEnumerable<T> was supported as a return result on an action).
While I'm not entirely familiar with postgres and DataReader, the functionality is there to get it streaming the result to the client. Appending to a list, and returning the result in its entirety takes up a lot of memory depending on the size of the result, and streaming helps us avoid that.
Here is an example of an action, that returns an IEnumerable<string> that is streamed to the client (it is sent in chunks until everything has been delivered using the Transfer-Encoding: chunked header).
[HttpGet]
public IEnumerable<string> Get()
{
return GetStringsFor(10000);
}
private static readonly Random random = new Random();
private IEnumerable<string> GetStringsFor(int amount)
{
const string chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
while (amount-- > 0)
{
yield return new string(Enumerable.Repeat(chars, random.Next(1000)).Select(s => s[random.Next(s.Length)]).ToArray());
}
}
This will ensure that not everything is loaded into memory, but sent on demand. You would be able to implement something similar in your case when you're reading to data into memory, because that is one time where the system could just start sending the result instead.
private IEnumerable<BearingTemperature> ReadTemperatures(SqlDataReader reader)
{
if (reader.HasRows)
{
var bt = new BearingTemperature();
while (reader.Read())
{
bt.Time = reader.GetDateTime(1);
bt.Turbine = reader.GetInt32(0);
bt.Value = reader.GetDouble(2);
yield return bt;
}
}
yield break;
}
[HttpGet("{turbine:int}")]
public IEnumerable<BearingTemperature> GetBearingTemperature(int turbine)
{
using (var connection = Database.GetConnection())
{
<snip>
var reader = command.ExecuteReader();
return ReadTemperatures(reader);
}
}

Considering that your database is going to execute the query and return the entire result set, it's not possible for you to stream a partial result set (though you can google streaming database for other offerings). What you could do instead is use a paging technique combined with ajax to retrieve slices of the total result set and compose them together on the client to keep the responsiveness high and create the illusion of streaming query results.
You'll want to look at OFFSET and LIMIT clauses
On your api, you'd include parameters for offset and limit to allow the client to step through and retrieve the result set in whatever size chunks it wants, you can play with it to determine what seems responsive enough. Then on your client, you'll need a loop over an ajax call to your api, probably using jquery, and keep looping page after page adding the results to the bound collection on the client or create ui elements, or whatever, until the results come back empty.
Alternatively, if showing the whole 10k records at once isn't necessary, you could simply page the results and provide an interface to step through the pages. One that I've used for such a purpose is from Sakura on git hub: PagedList

Pooling MySQL Connections with Microsoft Enterprise Library

My setup is MySql.Data.MySqlClient v6.9.8.0 and Microsoft.Practices.EnterpriseLibrary.Data v6.0.0.
The program is a long running program that runs continuously listening for tasks and then performs the job with some form of database action (depending on what the request was.) Sometimes the requests will be one after the other, sometimes there will be several hours between them.
I've tried using Pooling=true in the connection string but it causes me a lot of problems (not all the time - these are intermittent problems.)
Here is an example:
[MySqlException (0x80004005): Authentication to host 'localhost' for user 'root' using method 'mysql_native_password' failed with message: Reading from the stream has failed.]
Turning off pooling fixes the problem but at the same time it makes the queries slower because we can't reuse connections. I've searched online and a lot of people have this same issue and the only fix/workaround I've found is Pooling=false which I'd rather avoid if possible.
Here is an example of my query code:
Database db = this.GetDatabase(databaseName);
List<dynamic> results = new List<dynamic>();
// Run the sql query
using (DbCommand dbCommand = db.GetSqlStringCommand(query))
{
foreach (var parameter in inParameters)
{
db.AddInParameter(dbCommand, parameter.Key, parameter.Value.Item1, parameter.Value.Item2);
}
foreach (var parameter in outParameters)
{
db.AddOutParameter(dbCommand, parameter.Key, parameter.Value.Item1, parameter.Value.Item2);
}
using (IDataReader dataReader = db.ExecuteReader(dbCommand))
{
IDictionary<string, object> instance;
do
{
// Read each row
while (dataReader.Read())
{
instance = new ExpandoObject() as IDictionary<string, object>;
// Populate the object on the fly with the data
for (int i = 0; i < dataReader.FieldCount; i++)
{
instance.Add(dataReader.GetName(i), dataReader[i]);
}
// Add the object to the results list
results.Add(instance);
}
} while (dataReader.NextResult());
}
return results;
}
Any ideas?

Can you try this? I know, I know. using "using" should mean I don't have to call the dataReader.Close() method...but I still do it. I also slightly altered the dr.Read block.
This guy talks about it.
http://www.joseguay.com/uncategorized/ensure-proper-closure-disposal-of-a-datareader
I know, I know. You shouldn't have to. Even when using Ent library, I do an extra .Close step to try and make sure.
Database db = this.GetDatabase(databaseName);
List<dynamic> results = new List<dynamic>();
// Run the sql query
using (DbCommand dbCommand = db.GetSqlStringCommand(query))
{
foreach (var parameter in inParameters)
{
db.AddInParameter(dbCommand, parameter.Key, parameter.Value.Item1, parameter.Value.Item2);
}
foreach (var parameter in outParameters)
{
db.AddOutParameter(dbCommand, parameter.Key, parameter.Value.Item1, parameter.Value.Item2);
}
using (IDataReader dataReader = db.ExecuteReader(dbCommand))
{
IDictionary<string, object> instance;
while (dataReader.Read())
{
instance = new ExpandoObject() as IDictionary<string, object>;
// Populate the object on the fly with the data
for (int i = 0; i < dataReader.FieldCount; i++)
{
instance.Add(dataReader.GetName(i), dataReader[i]);
}
// Add the object to the results list
results.Add(instance);
}
if (dataReader != null)
{
try
{
dataReader.Close();
}
catch { }
}
}
return results;
}

Best way to embed image coming from database in html view

I'm using byte[] symbols to store images data in a database like this:
ITEM_IMAGE VARBINARY(MAX),
And then when I retrieve the image and display it, I proceed like this:
<img src="data:image/png;base64, #(Convert.ToBase64String(Model.mChildCard.NormalImage))" alt="#Model.mChildCard.mCardName" title="#Model.mChildCard.mCardName" class="nullify"/>
I do this because I cannot guarantee that our application will have write access on the server it will be deployed and, instead of storing the images in normal files (and there are a LOT of images, talking about 70k and more), we choose to store them in database and retrieve them as such.
Now I want to make sure this is the best way of handling those files in razor views as there may be a lot of images displayed at once. Will it have an impact on the speed it is rendered? What "weight" will have the database? Is there a better way to do things?

public FileStreamResult GetDBImage(string imageId)
{
using (var conn = GetConnection())
{
conn.Open();
using (var cmd = conn.CreateCommand)
{
cmd.CommandText = "SELECT ITEM_IMAGE FROM ... WHERE id=#id";
cmd.Parameters.Add("#id", imageId);
using (var rdr = cmd.ExecuteReader())
return File(rdr.GetStream(0), "image/png")
}
}
}
Also, consider using async.

To serve images:
Your new controller action:
public ActionResult GetImage(string imageID)
{
byte[] imgArray;
//call your db and get your byte array.
if(imgArray != null)
{
return File(imageArray, "image/png");
}
else
{
throw new HttpException(404);
}
}
Add a route:
routes.MapRoute("Images", "images/{imageId}", New With {.controller = "yourImageController", .action = "GetImage")
And from your HTML:
<img src="#Url.Action("GetImage", "YourImageController", new{ #imageId=Model.mChildCard.imageId})" alt="#Model.mChildCard.mCardName" title="#Model.mChildCard.mCardName" class="nullify"/>

Reading and writing data into sql server simultaneously

I have a service which continuously writes data in a separate thread into SQL database.Now from the same service if i am trying to read from the same table, since i already am writing into it,I get this exception : There is already an open DataReader associated with this Command which must be closed first.
So can anyone help me how to do this simultaneously?
Here s my code for reading data:
public Collection ReadData(string query)
{
{
_result = new Collection<string[]>();
string[] tempResult;
SqlDataReader _readerRead;
using (_command = new SqlCommand(query, _readConnection))
{
_readerRead = _command.ExecuteReader();
while (_readerRead.Read())
{
tempResult = new string[4];
tempResult[0] = _reader[0].ToString();
tempResult[1] = _reader[1].ToString();
tempResult[2] = _reader[2].ToString();
tempResult[3] = _reader[3].ToString();
_result.Add(tempResult);
//Console.WriteLine("Name : {0} Type : {1} Value : {2} timestamp : {3}", _reader[0], _reader[1], _reader[2], _reader[3]);
}
if (_readerRead != null)
{
_readerRead.Close();
}
_readConnection.Close();
return _result;
}
}
}
and here it is for writing to it :
public void WriteData(Collection<TagInfo> tagInfoList)
{
int i = 0;
for (i = 0; i < tagInfoList.Count; i++)
{
using( _command = new SqlCommand(insert statement here)
{
_command.Parameters.AddWithValue("Name", tagInfoList[i].Name);
_command.Parameters.AddWithValue("Type", tagInfoList[i].TagType);
_command.Parameters.AddWithValue("Value", tagInfoList[i].Value);
_reader = _command.ExecuteReader();
if (_reader != null)
{
_reader.Close();
}
}
}
}

You need a different SQLConnection to the database for your writer. You cannot use the same db connection for both.

Although its possible to do, using a separate connection I would question why you need to do this.
If you are reading and writing data to one table in the same service you will be placing unnecessary load on one SQL table, and depending on the number of queries you intend to make this could cause you problems. If you already have this data (in a different thread) why not Marshall the data from the background thread to where you need it as you write it into the database, and you don't need to read the data anymore.
However.... it is difficult to give an fair answer without seeing the code/what you are looking to achieve.