Intro
We are building a c# data collector that collects tagged data from an OPC source (data being sampled every 1/2 second. The code runs well except for the fact that it uses around 25% of the CPU (XEON 2.8GHz 8GB). Using the Visual Studio Performance Profiler, we got the following :
Overall CPU usage
We identified 2 functions with high CPU consumption : Hot path leading to the 2 high consumption functions
We are noticing that the function, itself, doesn't consume much CPU ressources (Self CPU around 1.5%) but around 50% of the whole app consumption (Total CPU around 50%).
Digging further into the "PrepareTags() function, we found that SemaphoreSlim.Release() and Wait() methods are mostly responsible for the CPU usage : PrepareTags() function consumption
Global code structure
When the program begins, the 3 main threads are started. The one I'm focussing on here is the REST API thread. These threads are "permanent" and kept active as long as the application runs. Also, each main thread will start new threads, some are permanent while others will execute and stop.
Based on my observations, around 40 threads are simultaneously active with small variations caused by threads being started and stopped. Thread number is kept constant so it seems there is no deadlocks.
3 Main threads are starting
public void StartThreads()
{
WriteLog.WriteFollowLog("CollecterService : is Started");
//set up default culture
CultureInfo culture = CultureInfo.CreateSpecificCulture(ConfigurationManager.AppSettings["Culturesetting"]);
CultureInfo.DefaultThreadCurrentCulture = culture;
CultureInfo.DefaultThreadCurrentUICulture = culture;
listOfTagUpdated = new SemaphoreSlim(0);
listOfTag = new Dictionary<string, TagSourceAddressObject>();
tcpMemory = new TagMemoryFinFout();
tagAdressMemoryForStat = new TagAdressMemoryFinFout();
tagMemory = new TagMemoryLastValue(false);
//configuration
try
{
// Creation of the Stat thread
this.statThread = new ThreadForCalculation(listOfTag, tagAdressMemoryForStat, tagMemory, tcpMemory);
// Creation of the REST API client thread
this.restApiThread = new ThreadRestApiClient(listOfTag, tcpMemory, listOfTagUpdated, tagMemory);
// Creation of the OPCDA Thread
this.opcDaThread = new ThreadOpcDaClient(listOfTag, tcpMemory, tagAdressMemoryForStat, tagMemory, listOfTagUpdated);
if (configuration != null)
{
this.configuration.StartTracking(this.restApiThread);
}
}
catch (Exception ex)
{
WriteLog.WriteErrorLog(ex);
}
}
The RestAPIThreadClient will also start 3 threads including the permanent "PrepareTags" thread.
RestAPIThread permanent thread starts 3 more permanent threads
...
// Start the authentication and the websocket handlers for this connection
this.AuthenticationConnectionThread = new Thread(CheckConnectionAuthentication);
this.AuthenticationConnectionThread.Name = "AuthentificationConnectionThread";
this.AuthenticationConnectionThread.Start();
this.WebsocketConnectionThread = new Thread(CheckWebsocketConnection);
this.WebsocketConnectionThread.Name = "WebsocketConnectionThread";
this.WebsocketConnectionThread.Start();
this.TcpPrepareTagThread = new Thread(PrepareTags);
this.TcpPrepareTagThread.Name = "TcpPrepareThread";
this.TcpPrepareTagThread.Start();
...
For better comprehension here is what a "TagValueObject" looks like :
TagValueObject newTag = new TagValueObject(null);
newTag.TagID = tag.TagID;
newTag.TagQuality = tag.TagQuality;
newTag.TagTimeStamp = tag.TagTimeStamp;
newTag.TagValue = tag.TagValue;
newTag.TagName = tag.TagName;
The overall process is :
OPC server sends tag data to our collector (every 1/2 second)
The collector stores these in memory
After some processing they are redirected to an API on AWS
The high CPU usage is observed in step 3
What happens in step 3
1. PrepareTags() gets called.
Function PrepareTags()
private void PrepareTags()
{
WriteLog.WriteFollowLog("TCPRestAPIClient : Is Started");
while (!stopWorking)
{
if (token != null)
{
TagValueObject tag = tcpMemory.GetFirstTag();
if (tag != null)
{
tagsToSend.Add(tag);
int numOfTagsToReturn = 50;
if ((readyToSend == true && tagsToSend.Count > 0) || tagsToSend.Count >= numOfTagsToReturn)
{
readyToSend = false;
//WriteLog.WriteFollowLog("Start count:" + tagsToSend.Count.ToString());
List<TagValueObject> tags = tagsToSend.GetRange(0, tagsToSend.Count > numOfTagsToReturn ? numOfTagsToReturn : tagsToSend.Count);
tagsToSend.RemoveRange(0, tagsToSend.Count > numOfTagsToReturn ? numOfTagsToReturn : tagsToSend.Count);
//WriteLog.WriteFollowLog("End count:" + tagsToSend.Count.ToString() + " / Remaining: " + tcpMemory.TagMemorySlot.Count);
Thread thread = new Thread(() =>
{
ProcessTagRequest(tags);
});
thread.Name = "PrepareTags";
thread.Start();
}
}
}
}
}
2. We get a range of tags stored in memory through "GetFirstTag()" and start the ProcessTagRequest Thread
Function GetFirsTag()
public TagValueObject GetFirstTag()
{
TagValueObject tag;
if (valueUpdated.Wait(10))
{
//WriteLog.WriteFollowLog(String.Format("TagMemoryFinFout-GetFirstTag Semaphore(valueUpdated) : Entered. Available Slots = {0}", valueUpdated.CurrentCount));
lock (thislock)
{
tag = tagMemorySlot.FirstOrDefault();
tagMemorySlot.Remove(tag);
}
valueUpdated.Release();
//WriteLog.WriteFollowLog(String.Format("TagMemoryFinFout-GetFirstTag Semaphore(valueUpdated) : Released. Available Slots = {0}", valueUpdated.CurrentCount));
}
else
{
tag = null;
}
return tag;
}
3. "ProcessTagRequest()" thread is started
Function ProcessTagRequest()
private void ProcessTagRequest(List<TagValueObject> tags)
{
if (tags.Count == 0) return;
try
{
sendData(tags, false);
}
catch (Exception ex)
{
WriteLog.WriteErrorLog("TCPRestAPIClient", ex);
}
//WriteLog.WriteErrorLog("Closed Thread");
}
4. We finally send the data to the API through sendData() function.
Function sendData()
private void sendData(List<TagValueObject> valueArray, Boolean externalMemory)
{
if (valueArray.Count > 0)
{
List<object> valueArrayAdjusted = new List<object>();
foreach (var valueEntry in valueArray){
dynamic data = new ExpandoObject();
data.value = valueEntry.TagValue;
data.date = valueEntry.TagTimeStamp;
data.quality = valueEntry.TagQuality;
data._id = valueEntry.TagID;
valueArrayAdjusted.Add(data);
}
var body = new
{
tags = valueArrayAdjusted
};
Boolean sendFail = true;
var tries = 0;
do
{
tries++;
IRestResponse response = APICall(Method.POST, "/senddata", body, token);
if (response.StatusCode == HttpStatusCode.OK)
{
sendFail = false;
// WriteLog.WriteErrorLog("Senddata connected successfully : " + response.StatusCode + " " + response.Content);
}
else
{
WriteLog.WriteFollowLog("Senddata failed to connect : " + response.StatusCode + " " + response.Content);
}
} while (tries < 1 && sendFail);
if (sendFail)
{
try
{
if (ConfigurationManager.AppSettings["UseExternalStorage"] == "true")
{
foreach (TagValueObject value in valueArray)
{
StoreInExternalMemory(value);
}
WriteLog.WriteFollowLog(String.Format("Senddata : Stored {0} tags in external memory", valueArray.Count));
}
} catch (Exception e)
{
WriteLog.WriteErrorLog("ThreatRestApiClient : " + e.ToString());
}
}
}
}
My question is : How to reduce the CPU usage? What is wrong in this code, architecture?
Related
I have code that basically opens a webpage + .ts file from a link and repeats it, but the problem is it increases memory usage each time and never removes the old data. After 2 Hours it uses more than 2GB.
Any ideas on how I can fix this issue?
I'm using "Leaf.Xnet" Library for requests and this is how I create my threads:
new Thread(new ThreadStart(WebHelper.Check)).Start();
Main code:
public static void Check()
{
HttpRequest request = null;
while (Form1.isRuning)
{
Application.DoEvents();
try
{
request = new HttpRequest();
if (!ProxyManager.updating)
{
switch (ProxyManager.curProxyType)
{
case ProxyManager.proxyType.http:
request.Proxy = HttpProxyClient.Parse(ProxyManager.NextProxy(ProxyManager.proxyType.http));
break;
case ProxyManager.proxyType.socks4:
request.Proxy = Socks4ProxyClient.Parse(ProxyManager.NextProxy(ProxyManager.proxyType.socks4));
break;
case ProxyManager.proxyType.socks5:
request.Proxy = Socks5ProxyClient.Parse(ProxyManager.NextProxy(ProxyManager.proxyType.socks5));
break;
}
}
else
{
Thread.Sleep(2000);
Check();
}
request.UserAgentRandomize();
request.AddHeader(HttpHeader.Referer, "https://somesite.com");
request.KeepAlive = true;
request.ConnectTimeout = Form1.timeOut;
request.Reconnect = true;
string html = request.Get(Form1.link, null).ToString();
string auth = html.Substring(",[{\"src\":\"", "\"");
string sign = html.Substring("144p.apt?wmsAuthSign=", "\"");
if (auth != null && sign != null)
{
string auth2 = "";
foreach (char item in auth)
{
if (item != '\\')
auth2 += item;
}
auth = auth2;
string cdn = auth.Substring("https://", ".");
string id = auth.Substring("video/", "-");
if (cdn != null && id != null)
{
Random rnd = new Random();
request.Get(auth);
Form1.sended++;
WriteStat();
}
html = null;
auth = null;
auth2 = null;
sign = null;
}
}
catch (HttpException)
{
Check();
}
catch (ProxyException)
{
Check();
}
}
}
I am not entirely sure if this will fix your problem but for each thread that you start, you pretty much call an infinite number of executions of Check(). Since Check contains a while loop, the thread will run whatever is in side forever anyway, and now you're calling the method again on top of it. This means that everything that was created in the scope of the Check method will not be garbage collected and will increase your memory.
Replace all calls to Check() with continue which will stop the execution in the while loop and start over.
Also, consider not using Threads, but instead use Tasks.
Also you do not dispose your HttpRequest.
So I am trying to build a program to control a machine. Communications with said machine is via a serial port for which I have written a driver. Continuous polling to the machine is necessary for status feedback etc. In my program I have a dedicated ExecutionEngine() class to handle serial send and receive. I also need to have two separate control sequences running, which I have put into methods RunSequenceA() and RunSequenceB() respectively. During normal operation, all three methods need to run until both control sequences finish, at which point the StopSequence() method is called. My issue is that sometimes, for whatever reason, the StopSequence() method is never called, leaving my ExecutionEngine() method in an infinite loop!
Code for ExecutionEngine():
private static void ExecutionEngine()
{
// Clear both lists in case they have old data
_commandList.Clear();
_pollingList.Clear();
// Poll while user has not yet clicked "STOP"
while (!_cTokenSource.Token.IsCancellationRequested)
{
// If there are commands to be sent, send them first
if (_commandList.Count > 0)
{
Command[] tempCommandArray;
lock (_commandList)
tempCommandArray = _commandList.ToArray();
foreach (var c in tempCommandArray)
{
if (_cTokenSource.Token.IsCancellationRequested)
break;
var response = SerialDriver.ComCycle(c.CommandBytes, _serialPort);
var success = CheckErrorReturn(response, false);
if (success)
{
AddPolling(c);
RemoveCommand(c);
}
}
}
// Do polling operation on applicable controllers
if (_pollingList.Count > 0)
{
Command[] tempPollingArray;
lock (_pollingList)
tempPollingArray = _pollingList.ToArray();
foreach (var c in tempPollingArray)
{
if (_cTokenSource.Token.IsCancellationRequested)
break;
var response = SerialDriver.ComCycle(c.PollBytes, _serialPort);
var success = ProcessPollReturn(response);
if (success)
{
c.FlagDone();
RemovePolling(c);
}
}
}
if (_commandList.Count + _pollingList.Count == 0)
{
// Will get stuck here if neither list gets new items added
Console.WriteLine("Bad place");
Thread.Sleep(500);
}
}
// Cancellation has been requested
lock (_commandList)
_commandList.Clear();
lock (_pollingList)
_pollingList.Clear();
ResetTriggers();
var endCommand = new Command("GL_SYSCMD", 0);
SerialDriver.ComCycle(endCommand.CommandBytes, _serialPort);
_serialPort.Close();
_vm.SequenceRunning = false;
return;
}
Code for running sequences:
private static async Task RunSequencesAsync()
{
var taskArray = new Task[2];
var a = new Action(RunSequenceA);
var b = new Action(RunSequenceB);
taskArray[0] = Task.Run(a);
taskArray[1] = Task.Run(b);
await Task.WhenAll(taskArray).ConfigureAwait(continueOnCapturedContext: false);
// Sometimes this never fires, WHY?
UpdateStatus("All done!");
StopSequence();
}
// Run A sequence
internal static void RunSequenceA()
{
if (_sequenceA1 != null && _sequenceA1.Count > 0)
{
foreach (var s in _sequenceA1)
{
if (_cTokenSource.Token.IsCancellationRequested)
return;
s.Execute();
if (s.Reference != null && TriggerStepCompleted != null)
TriggerStepCompleted(s, EventArgs.Empty);
}
// This part always fires
Console.WriteLine("Sequence A finished");
return;
}
else
return;
}
And finally, the methods to start and stop everything:
private static async Task StartSequenceAsync()
{
_serialPort.PortName = _vm.SelectedComPort;
_serialPort.Open();
_serialPort.DiscardInBuffer();
_serialPort.DiscardOutBuffer();
// Start
_cTokenSource = new CancellationTokenSource();
_vm.SequenceRunning = true;
var taskArray = new Task[2];
taskArray[0] = Task.Run(() => ExecutionEngine());
Thread.Sleep(50);
taskArray[1] = Task.Run(() => RunSequencesAsync());
await Task.WhenAll(taskArray).ConfigureAwait(continueOnCapturedContext: false);
}
private static void StopSequence()
{
_cTokenSource.Cancel();
}
To reiterate, the problem doesn't happen every time. In fact, most times the program runs fine. It seems that problems only arise if I manually call the StopSequence() method half way through execution. Then it's 50/50 as to whether the problem shows up. I'm pretty sure my issue is threading related, but not sure exactly what is going wrong. Any help pointing me in the right direction will be greatly appreciated!
I am making a Bloomberg web service GetData call for the "DEBT_TO_EQUITY_FUNDAMENTALS_TKR" field. I am setting secmaster = true and asking for a single instrument with a CUSIP identifier (with yellowkey = MarketSector.Corp).
This strikes me as a fairly lightweight call having seen people asking for thousands of instruments and dozens of fields at once.
I have played around with setting lots of different settings but I just can't get this request to return in a few seconds. It gives me the correct return value but it takes longer than 60 seconds.
Any idea if it is possible to get such a request to execute and return in a few seconds?
Thanks
EDIT - Here is the code I am running:
public string GetFundamentalTicker(string identifier, InstrumentType identifierType = InstrumentType.CUSIP)
{
PerSecurityWS ps = new PerSecurityWS();
try
{
log.DebugFormat("Cert path is: {0}", CertPath);
X509Certificate2 clientCert = new X509Certificate2(CertPath, "<password_redacted>");
ps.ClientCertificates.Add(clientCert);
}
catch (Exception e)
{
log.ErrorFormat("Error in cert setup - {0} - {1}", e.Message, e.InnerException == null ? "" : e.InnerException.Message);
return null;
}
//Set request header
GetDataHeaders getDataHeaders = new GetDataHeaders();
getDataHeaders.secmaster = true;
getDataHeaders.secmasterSpecified = true;
//getDataHeaders.fundamentals = true;
//getDataHeaders.fundamentalsSpecified = true;
//getDataHeaders.programflag = ProgramFlag.oneshot;//unnecessary - defaults to this anyway
//getDataHeaders.programflagSpecified = true;
//getDataHeaders.pricing = true;
getDataHeaders.secid = identifierType;
getDataHeaders.secidSpecified = true;
SubmitGetDataRequest sbmtGtDtreq = new SubmitGetDataRequest();
sbmtGtDtreq.headers = getDataHeaders;
sbmtGtDtreq.fields = new string[] {
"DEBT_TO_EQUITY_FUNDAMENTALS_TKR"
};
int currentFundYear = DateTime.Now.Year;
//var fundYears = new List<int>();
List<Instrument> fundYearInstruments = new List<Instrument>();
Instrument fundYearInstrument = null;
fundYearInstrument = new Instrument();
fundYearInstrument.id = identifier;
fundYearInstrument.typeSpecified = true;
fundYearInstrument.type = identifierType;
fundYearInstrument.yellowkey = MarketSector.Corp;
fundYearInstrument.yellowkeySpecified = true;
//fundYearInstrument.overrides = new Override[] {};//{ new Override() { field = "EQY_FUND_YEAR", value = currentFundYear.ToString() } };
fundYearInstruments.Add(fundYearInstrument);
//fundYears.Add(-1);
Instrument[] instr = fundYearInstruments.ToArray();
Instruments instrs = new Instruments();
instrs.instrument = instr;
sbmtGtDtreq.instruments = instrs;
try
{
SubmitGetDataResponse sbmtGtDtResp = ps.submitGetDataRequest(sbmtGtDtreq);
RetrieveGetDataRequest rtrvGtDrReq = new RetrieveGetDataRequest();
rtrvGtDrReq.responseId = sbmtGtDtResp.responseId;
RetrieveGetDataResponse rtrvGtDrResp;
do
{
System.Threading.Thread.Sleep(POLL_INTERVAL);
rtrvGtDrResp = ps.retrieveGetDataResponse(rtrvGtDrReq);
}
while (rtrvGtDrResp.statusCode.code == DATA_NOT_AVAILABLE);
if (rtrvGtDrResp.statusCode.code == SUCCESS)
{
for (int i = 0; i < rtrvGtDrResp.instrumentDatas.Length; i++)
{
for (int j = 0; j < rtrvGtDrResp.instrumentDatas[i].data.Length; j++)
{
if (rtrvGtDrResp.instrumentDatas[i].data[j].value == "N.A." || rtrvGtDrResp.instrumentDatas[i].data[j].value == "N.S." || rtrvGtDrResp.instrumentDatas[i].data[j].value == "N.D.")
rtrvGtDrResp.instrumentDatas[i].data[j].value = null;
return rtrvGtDrResp.instrumentDatas[i].data[j].value;
}
}
return null;
}
else if (rtrvGtDrResp.statusCode.code == REQUEST_ERROR)
{
log.ErrorFormat("Error in the submitted request: {0}", rtrvGtDrResp.statusCode.description);
return null;
}
}
catch (Exception e)
{
log.ErrorFormat("Error in GetData - {0} - {1}", e.Message, e.InnerException == null ? "" : e.InnerException.Message);
return null;
}
return null;
}
Poll interval is 5 seconds and the SOAP web service url is:
https://software.bloomberg.com/datalicensewp/dlws.wsdl
I am having the same issue. I found out that there is a difference between making the same call to Bloomberg API from, for example, console app (works very fast) and web service (takes a lot of time to start session). And the difference is that console app runs under the same user as bbcomm process, whereas web service (or actually iis process) runs under System account. You can try to log out all users on the PC where web service is hosted and then try to make the call. In this case, I guess, bbcomm goes under System account as no one else is logged in and works fast. It worked for me and the call was answered instantly.
I have a thread that returns a site's http response status, but sometimes my program returns false results. and after a while it gives good results.
False result:
it takes a big a mount of time to check, and then it says that (for example) Google is down, which is quite not reasonable, but after a few seconds it returns good results
Can you take a look and tell me whats wrong? or how I can I improve it?
Checks all sites in datagrid:
private void CheckSites()
{
if (CheckSelected())
{
int rowCount = dataGrid.BindingContext[dataGrid.DataSource, dataGrid.DataMember].Count;
string url;
for (int i = 0; i < rowCount; i++)
{
url = dataGrid.Rows[i].Cells[2].Value.ToString();
if (url != null)
{
Task<string[]> task = Task.Factory.StartNew<string[]>
(() => checkSite(url));
// We can do other work here and it will execute in parallel:
//Loading...
// When we need the task's return value, we query its Result property:
// If it's still executing, the current thread will now block (wait)
// until the task finishes:
string[] result = task.Result;
selectRows();
if (result[0] != System.Net.HttpStatusCode.OK.ToString() && result[0] != System.Net.HttpStatusCode.Found.ToString() && result[0] != System.Net.HttpStatusCode.MovedPermanently.ToString())
{
//bad
notifyIcon1.ShowBalloonTip(5000, "Site Down", dataGrid.Rows[i].Cells[2].Value.ToString() + ", has a status code of:" + result, ToolTipIcon.Error);
dataGrid.Rows[i].DefaultCellStyle.BackColor = System.Drawing.Color.Wheat;
TimeSpan ts;
TimeSpan timeTaken = TimeSpan.Parse(result[1]);
dataGrid.Rows[i].Cells[3].Value = result[0];
dataGrid.Rows[i].Cells[3].Style.BackColor = System.Drawing.Color.Red;
dataGrid.Rows[i].Cells[4].Value = timeTaken.Seconds.ToString() + "." + String.Format("{0:0.00000}", timeTaken.Milliseconds.ToString()) + " seconds.";
string sec = (DateTime.Now.Second < 10) ? "0" + DateTime.Now.Second.ToString() : DateTime.Now.Second.ToString();
string min = (DateTime.Now.Minute < 10) ? "0" + DateTime.Now.Minute.ToString() : DateTime.Now.Minute.ToString();
string hour = (DateTime.Now.Hour < 10) ? "0" + DateTime.Now.Hour.ToString() : DateTime.Now.Hour.ToString();
dataGrid.Rows[i].Cells[5].Value = hour + ":" + min + ":" + sec;
//loadbar
}
else if (result[0] == "catch")//catch
{
notifyIcon1.ShowBalloonTip(10000, "SITE DOWN", dataGrid.Rows[i].Cells[1].Value.ToString() + ", Error:" +result[1], ToolTipIcon.Error);
dataGrid.Rows[i].Cells[3].Value = result[1];
dataGrid.Rows[i].Cells[3].Style.BackColor = System.Drawing.Color.Red;
//loadbar
}
else
{
//good
TimeSpan timeTaken = TimeSpan.Parse(result[1]);
dataGrid.Rows[i].Cells[3].Value = result[0];
dataGrid.Rows[i].Cells[3].Style.BackColor = System.Drawing.Color.LightGreen;
dataGrid.Rows[i].Cells[4].Value = timeTaken.Seconds.ToString() + "." + String.Format("{0:0.00000}", timeTaken.Milliseconds.ToString()) + " seconds.";
string sec = (DateTime.Now.Second < 10) ? "0" + DateTime.Now.Second.ToString() : DateTime.Now.Second.ToString();
string min = (DateTime.Now.Minute < 10) ? "0" + DateTime.Now.Minute.ToString() : DateTime.Now.Minute.ToString();
string hour = (DateTime.Now.Hour < 10) ? "0" + DateTime.Now.Hour.ToString() : DateTime.Now.Hour.ToString();
dataGrid.Rows[i].Cells[5].Value = hour + ":" + min + ":" + sec;
//loadbar
}
selectRows();
}
}
}
}
Checks a site:
/////////////////////////////////
////Check datagrid websites-button - returns response
/////////////////////////////////
private string[] checkSite(string url)
{
string[] response = new string[2];
url = dataGrid.Rows[0].Cells[2].Value.ToString();
if (url != null)
{
try
{
HttpWebRequest httpReq;
httpReq.Timeout = 10000;
//loadbar
dataGrid.Rows[0].DefaultCellStyle.BackColor = System.Drawing.Color.Wheat;
System.Diagnostics.Stopwatch timer = new System.Diagnostics.Stopwatch();
timer.Start();
HttpWebResponse httpRes = (HttpWebResponse)httpReq.GetResponse(); //httpRes.Close();
timer.Stop();
//loadbar
HttpStatusCode httpStatus = httpRes.StatusCode;
response[0] = httpStatus.ToString();
response[1] = timer.Elapsed.ToString();//*
httpRes.Close();
return response;
}
catch (Exception he)
{
response[0] = "catch";
response[1] = he.Message;
return response;
}
}
response[0] = "catch";
response[1] = "No URL entered";
return response;
//dataGrid.Rows[i].DefaultCellStyle.BackColor = System.Drawing.Color.Blue;
}
Thanks in advance.
Assuming the code provided is the actual code used:
First of all, your definition of 'False result' and 'Good result' is wrong. If you expect A but get B, that doesn't mean B is invalid. If your wife is giving birth and you expect a boy but it turns out the be a girl, its not a false result. Just unexpected.
That said: lets analyze your work: If it takes a long long time to check a site only to finally get a ??? result which isn't a 200 response code. We can almost savely assume you are dealing with a timeout. If your router, google or any fundamental network device in between is having problems, its expected to get an unexpected answer. "Timeout", "Bad Request", "Server not available" etc. Why would this happen? Its impossible to say for certain without having direct access to your environment.
Looking at your code however, i see that you're using the default TaskScheduler for making each check run as a task in the background (assuming you havent changed the default task scheduler which would be a vey bad practice to begin with). The default task scheduler, schedules each task on the threadpool which results in many many tasks running simultanious. Here we have a good candidate for overloading your network. Many sites (esspecially google) are kinda sensitive for handling many requests from the same source (esspecially if the frequency is high) so maybe google is blocking you temporarily or holding you back. Again, at this point it's pure speculation but the fact that you're running all checks simultaniously (unless the thread pool is on his max) is very likely the cause of your problem.
UPDATE
I would recommend working with a LimitedConcurrencyTaskScheduler ( see here: http://blogs.msdn.com/b/pfxteam/archive/2010/04/09/9990424.aspx ). Here you can limit the amount of tasks that can be run asynchronously. You have to do some testing for what number works ideally in your situation. Also make sure that the frequency is not 'too' high. Its hard to define what is too high, only testing can proof that.
In order to simulate your scenario, I have created a Winform with data grid and a button. On load of the form, I programmatically creates list of url’s (in a table) and bind to data grid. And on button click, we start the download process. In concise, the you have to write more defensive code and the following code only a skeleton of how you can fix the issue.
using System;
using System.Data;
using System.Net;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace app
{
public partial class Form1 : Form
{
DataTable urls = new DataTable();
public Form1()
{
InitializeComponent();
}
//Fill your uri's and bind to a data grid.
void InitTable()
{
//Silly logic to simulate your scenario.
urls = new DataTable();
urls.Columns.Add(new DataColumn("Srl", typeof(string)));
urls.Columns.Add(new DataColumn("Urls", typeof(Uri)));
urls.Columns.Add(new DataColumn("Result", typeof(string)));
DataRow dr = urls.NewRow();
dr["Srl"] = "1";
dr["Urls"] = new Uri("http://www.microsoft.com");
dr["Result"] = string.Empty;
urls.Rows.Add(dr);
dr = urls.NewRow();
dr["Srl"] = "2";
dr["Urls"] = new Uri("http://www.google.com");
dr["Result"] = string.Empty;
urls.Rows.Add(dr);
dr = urls.NewRow();
dr["Srl"] = "3";
dr["Urls"] = new Uri("http://www.stackoverflow.com");
dr["Result"] = string.Empty;
urls.Rows.Add(dr);
urls.AcceptChanges();
}
void UpdateResult()
{
dataGridView1.DataSource = urls;
}
//Important
// This example will freeze UI. You can avoid this while implementing
//background worker or pool with some event synchronization. I haven't covered those area since
//we are addressing different issue. Let me know if you would like to address UI freeze
//issue. Or can do it your self.
private void button1_Click(object sender, EventArgs e)
{
//Create array for Task to parallelize multiple download.
var tasks = new Task<string[]>[urls.Rows.Count];
//Initialize those task based on number of Uri's
for(int i=0;i<urls.Rows.Count;i++)
{
int index = i;//Do not change this. This is to avoid data race
//Assign responsibility and start task.
tasks[index] = new Task<string[]>(
() => checkSite(
new TaskInput(urls.Rows[index]["Urls"].ToString(), urls.Rows[index]["Srl"].ToString())));
tasks[index].Start();
}
//Wait for all task to complete. Check other overloaded if interested.
Task.WaitAll(tasks);
//block shows how to access result from task
foreach (var item in tasks)
{
DataRow[] rows=urls.Select("Srl='"+item.Result[2]+"'");
foreach (var row in rows)
row["Result"]=item.Result[0]+"|"+item.Result[1];
}
UpdateResult();
}
//This is dummy method which in your case 'Check Site'. You can have your own
string[] checkSite(TaskInput input)
{
string[] response = new string[3];
if (input != null)
{
try
{
WebResponse wResponse = WebRequest.Create(input.Url).GetResponse();
response[0] = wResponse.ContentLength.ToString();
response[1] = wResponse.ContentType;
response[2] = input.Srl;
return response;
}
catch (Exception he)
{
response[0] = "catch";
response[1] = he.Message;
response[2] = input.Srl;
return response;
}
}
response[0] = "catch";
response[1] = "No URL entered";
response[2] = input.Srl;
return response;
}
private void Form1_Load(object sender, EventArgs e)
{
InitTable();
UpdateResult();
}
}
//Supply custom object for simplicity
public class TaskInput
{
public TaskInput(){}
public TaskInput(string url, string srl)
{
Url = url;
Srl = srl;
}
public string Srl { get; set; }
public string Url { get; set; }
}
}
I have a netbook with 1.20Ghz Processor & 1GB Ram.
I'm running a C# WinForms app on it which, at 5 minute intervals, reads every line of a text file and depending on what the content of that line is, either skips it or writes it to an xml file. Sometimes it may be processing about 2000 lines.
When it begins this task, the processor gets maxed out, 100% use. However on my desktop with 2.40Ghz Processor and 3GB Ram it's untouched (for obvious reasons)... is there any way I can actually reduce this processor issue dramatically? The code isn't complex, I'm not bad at coding either and I'm not constantly opening the file, reading and writing... it's all done in one fell swoop.
Any help greatly appreciated!?
Sample Code
***Timer.....
#region Timers Setup
aTimer.Tick += new EventHandler(OnTimedEvent);
aTimer.Interval = 60000;
aTimer.Enabled = true;
aTimer.Start();
radioButton60Mins.Checked = true;
#endregion Timers Setup
private void OnTimedEvent(object source, EventArgs e)
{
string msgLoggerMessage = "Checking For New Messages " + DateTime.Now;
listBoxActivityLog.Items.Add(msgLoggerMessage);
MessageLogger messageLogger = new MessageLogger();
messageLogger.LogMessage(msgLoggerMessage);
if (radioButton1Min.Checked)
{
aTimer.Interval = 60000;
}
if (radioButton60Mins.Checked)
{
aTimer.Interval = 3600000;
}
if (radioButton5Mins.Checked)
{
aTimer.Interval = 300000;
}
// split the file into a list of sms messages
List<SmsMessage> messages = smsPar.ParseFile(smsPar.CopyFile());
// sanitize the list to get rid of stuff we don't want
smsPar.SanitizeSmsMessageList(messages);
ApplyAppropriateColoursToRecSMSListinDGV();
}
public List<SmsMessage> ParseFile(string filePath)
{
List<SmsMessage> list = new List<SmsMessage>();
using (StreamReader file = new StreamReader(filePath))
{
string line;
while ((line = file.ReadLine()) != null)
{
var sms = ParseLine(line);
list.Add(sms);
}
}
return list;
}
public SmsMessage ParseLine(string line)
{
string[] words = line.Split(',');
for (int i = 0; i < words.Length; i++)
{
words[i] = words[i].Trim('"');
}
SmsMessage msg = new SmsMessage();
msg.Number = int.Parse(words[0]);
msg.MobNumber = words[1];
msg.Message = words[4];
msg.FollowedUp = "Unassigned";
msg.Outcome = string.Empty;
try
{
//DateTime Conversion!!!
string[] splitWords = words[2].Split('/');
string year = splitWords[0].Replace("09", "20" + splitWords[0]);
string dateString = splitWords[2] + "/" + splitWords[1] + "/" + year;
string timeString = words[3];
string wholeDT = dateString + " " + timeString;
DateTime dateTime = DateTime.Parse(wholeDT);
msg.Date = dateTime;
}
catch (Exception e)
{
MessageBox.Show(e.ToString());
Application.Exit();
}
return msg;
}
public void SanitizeSmsMessageList(List<SmsMessage> list)
{
// strip out unwanted messages
// list.Remove(some_message); etc...
List<SmsMessage> remove = new List<SmsMessage>();
foreach (SmsMessage message in list)
{
if (message.Number > 1)
{
remove.Add(message);
}
}
foreach (SmsMessage msg in remove)
{
list.Remove(msg);
}
//Fire Received messages to xml doc
ParseSmsToXMLDB(list);
}
public void ParseSmsToXMLDB(List<SmsMessage> list)
{
try
{
if (File.Exists(WriteDirectory + SaveName))
{
xmlE.AddXMLElement(list, WriteDirectory + SaveName);
}
else
{
xmlE.CreateNewXML(WriteDirectory + SaveName);
xmlE.AddXMLElement(list, WriteDirectory + SaveName);
}
}
catch (Exception e)
{
MessageBox.Show(e.ToString());
Application.Exit();
}
}
public void CreateNewXML(string writeDir)
{
try
{
XElement Database = new XElement("Database");
Database.Save(writeDir);
}
catch (Exception e)
{
MessageBox.Show(e.ToString());
}
}
public void AddXMLElement(List<SmsMessage> messages, string writeDir)
{
try
{
XElement Database = XElement.Load(writeDir);
foreach (SmsMessage msg in messages)
{
if (!DoesExist(msg.MobNumber, writeDir))
{
Database.Add(new XElement("SMS",
new XElement("Number", msg.MobNumber),
new XElement("DateTime", msg.Date),
new XElement("Message", msg.Message),
new XElement("FollowedUpBy", msg.FollowedUp),
new XElement("Outcome", msg.Outcome),
new XElement("Quantity", msg.Quantity),
new XElement("Points", msg.Points)));
EventNotify.SendNotification("A New Message Has Arrived!", msg.MobNumber);
}
}
Database.Save(writeDir);
EventNotify.UpdateDataGridView();
EventNotify.UpdateStatisticsDB();
}
catch (Exception e)
{
MessageBox.Show(e.ToString());
}
}
public bool DoesExist(string number, string writeDir)
{
XElement main = XElement.Load(writeDir);
return main.Descendants("Number")
.Any(element => element.Value == number);
}
Use a profiler and/or Performance Monitor and/or \\live.sysinternals.com\tools\procmon.exe and/or ResourceMonitor to determine what's going on
If the 5 minute process is a background task, you can make use of Thread Priority.
MSDN here.
If you do the processing on a separate thread, change your timer to be a System.Threading.Timer and use callback events, you should be able to set a lower priority on that thread than the rest of your application.
Inside your ParseFile loop, you could try adding a Thread.Sleep and/or an Application.DoEvents() call to see if that helps. Its better to do this in the parsing is on a seperate thread, but at least you can try this simple test to see if it helps.
Might be that the MessageBoxes in your catches are running into cross-thread problems. Try swapping them out for writing to the trace output.
In any case, you've posted an entire (little) program, which will not help you get specific advice. Try deleting method bodies -- one at a time, scientifically -- and try to get the problem to occur/stop occurring. This will help you to locate the problem and eliminate the irrelevant parts of your question (both for yourself and for SO).
Your current processing model is batch based - do the parsing, then process the messages, and so on.
You'll likely reduce the memory overhead if you switched to a Linq style "pull" approach.
For example, you could convert your ParseFile() method in this way:
public IEnmerable<SmsMessage> ParseFile(string filePath)
{
using (StreamReader file = new StreamReader(filePath))
{
string line;
while ((line = file.ReadLine()) != null)
{
var sms = ParseLine(line);
yield return sms;
}
}
}
The advantage is that each SmsMessage can be handled as it is generated, instead of parsing all of the messages at once and then handling all of them.
This lowers your memory overhead, which is one of the most likely causes for the performance difference between your netbook and your desktop.