Long story short. ;
I have a class named Scope. And this class contains all logic for scope operations etc. It also starts backround thread that constantly read serial port data (in my case events was unreliable):
Thread BackgroundReader = new Thread(ReadBuffer);
BackgroundReader.IsBackground = true;
BackgroundReader.Start();
private void ReadBuffer()
{
SerialPort.DiscardInBuffer();
while (!_stopCapture)
{
int bufferSize = SerialPort.BytesToRead;
byte[] buffer = new byte[bufferSize];
if(bufferSize > 5)
{
SerialPort.Read(buffer, 0, bufferSize);
Port_DataReceivedEvent(buffer, null);
}
Thread.Sleep(_readDelay);
}
CurrentBuffer = null;
}
In Scope class there is a public field named Buffer
public byte[] Buffer
{
get
{
return CurrentBuffer;
}
}
And here is event fired while there is new data readed
private void Port_DataReceivedEvent(object sender, EventArgs e)
{
//populate buffer
Info(sender, null);
CurrentBuffer = ((byte[])sender);
foreach(byte data in CurrentBuffer)
{
DataBuffer.Enqueue(data);
}
if (DataBuffer.Count() > _recordLength)
{
GenerateFrame(DataBuffer.ToArray());
DataBuffer.Clear(); ;
}
}
To make code more manageable, I splitted it in several classes. One of this classes is for searching specific data pattern in current stream and create specific object from this data. This code works in way that send to serial port specific command and expect return frame. If reponse is not received or not ok, send is performed again and again until correct response arrives or there will be timeout. Response is expected to be in current buffer. Those strange string manipulation is for debug purposes.
public class GetAcknowledgedFrame
{
byte[] WritedData;
string lastEx;
string stringData;
public DataFrame WriteAcknowledged(Type SendType, Type ReturnType, JyeScope scope)
{
var stopwatch = new Stopwatch();
stopwatch.Restart();
while (stopwatch.ElapsedMilliseconds < scope.TimeoutTime)
{
try
{
if (SendType == typeof(GetParameters))
{
WriteFrame(new ScopeControlFrames.GetParameters(), scope.SerialPort);
}
else if(SendType == typeof(GetConfig))
{
WriteFrame(new ScopeControlFrames.GetConfig(), scope.SerialPort);
}
else if (SendType == typeof(EnterUSBScopeMode))
{
WriteFrame(new ScopeControlFrames.EnterUSBScopeMode(), scope.SerialPort);
}
return ReturnFrame(ReturnType, scope.Buffer, scope.TimeoutTime);
}
catch (InvalidDataFrameException ex)
{
lastEx = ex.Message;
System.Threading.Thread.Sleep(10);
}
}
stringData = "";
foreach (var data in scope.Buffer)
{
stringData += data + ",";
}
stringData.Remove(stringData.Length - 1);
throw new TimeoutException($"Timeout while waiting for frame acknowledge: " + SendType.ToString() + ", " + ReturnType.ToString() + Environment.NewLine+ "Add. err: "+lastEx);
}
private DataFrame ReturnFrame(Type FrameType, byte[] buffer, int timeoutTime)
{
if (FrameType == typeof(DataFrames.DSO068.CurrConfigDataFrame))
{
DataFrames.DSO068.CurrConfigDataFrame CurrConfig = new DataFrames.DSO068.CurrConfigDataFrame(buffer);
return CurrConfig;
}
else if (FrameType == typeof(DataFrames.DSO112.CurrConfigDataFrame))
{
DataFrames.DSO112.CurrConfigDataFrame CurrParam = new DataFrames.DSO112.CurrConfigDataFrame(buffer);
return CurrParam;
}
else if (FrameType == typeof(CurrParamDataFrame))
{
CurrParamDataFrame CurrParam = new CurrParamDataFrame(buffer);
return CurrParam;
}
else if (FrameType == typeof(DataBlockDataFrame))
{
DataBlockDataFrame CurrData = new DataBlockDataFrame(buffer);
return CurrData;
}
else if (FrameType == typeof(DataSampleDataFrame))
{
DataSampleDataFrame CurrData = new DataSampleDataFrame(buffer);
return CurrData;
}
else if (FrameType == typeof(ScopeControlFrames.ScopeReady))
{
ScopeControlFrames.ScopeReady ready = new ScopeControlFrames.ScopeReady(buffer);
return ready;
}
else
{
throw new InvalidOperationException("Wrong object type");
}
}
private bool WriteFrame(DataFrame frame, IStreamResource port)
{
WritedData = frame.Data;
port.Write(frame.Data, 0, frame.Data.Count());
return true;
}
}
From main class (and main thread) I call method in this class, for example:
var Ready = (ScopeControlFrames.ScopeReady)new GetAcknowledgedFrame().WriteAcknowledged
(typeof(ScopeControlFrames.EnterUSBScopeMode), typeof(ScopeControlFrames.ScopeReady), this);
The problem is when I pass "this" object (that has thread working in background) to my helper class. It seems like helper class not see changing data in this object. The problem started when I separate code of my helper class from main class.
My questions:
- I know that object are passed by reference, that means I think that when object is dynamically changing its state (in this case data buffer should changing while new data is received) all classes that has reference to this object are also seeing this changes. Maybe I'm missing something?
- I tried passing array (by ref), arrays are also reference types. But this not help me at all. Maybe I'm missing something?
I tried changing this class to static, it not helped.
Many thanks for help.
The code below;
Info(sender, null);
CurrentBuffer = ((byte[])sender);
is creating a new reference variable called CurrentBuffer. Any other code holding a reference 'pointer' to the CurrentBuffer value prior to this line of code will not get the new value of CurrentBuffer when its reset.
Related
I have written an app that goes through our own properties and scraps the data. To make sure I don't run through the same URLs, I am using a MySQL database to store the URL, flag it once its processed. All this was being done in a single thread and it's fine if I had only few thousand entries. But I have few hundred thousand entries that I need to parse so I need to make changes in the code (I am newbie in multithreading in general). I found an example and was trying to copy the style but doesn't seem to work. Anyone know what the issue is with the following code?
EDIT: Sorry didn't mean to make people guess the issue but was stupid of me to include the exception. Here is the exception
"System.InValidCastException: 'Specified cast is not valid.'"
When I start the process it collects the URLs from the database and then never hits DoWork method
//This will get the entries from the database
List<Mappings> items = bot.GetUrlsToProcess(100);
if (items != null)
{
var tokenSource = new CancellationTokenSource();
var token = tokenSource.Token;
Worker.Done = new Worker.DoneDelegate(WorkerDone);
foreach (var item in items)
{
urls.Add(item.Url);
WaitingTasks.Enqueue(new Task(id => new Worker().DoWork((int)id, item.Url, token), item.Url, token));
}
LaunchTasks();
}
static async void LaunchTasks()
{
// keep checking until we're done
while ((WaitingTasks.Count > 0) || (RunningTasks.Count > 0))
{
// launch tasks when there's room
while ((WaitingTasks.Count > 0) && (RunningTasks.Count < MaxRunningTasks))
{
Task task = WaitingTasks.Dequeue();
lock (RunningTasks) RunningTasks.Add((int)task.AsyncState, task);
task.Start();
}
UpdateConsole();
await Task.Delay(300); // wait before checking again
}
UpdateConsole(); // all done
}
static void UpdateConsole()
{
Console.Write(string.Format("\rwaiting: {0,3:##0} running: {1,3:##0} ", WaitingTasks.Count, RunningTasks.Count));
}
static void WorkerDone(int id)
{
lock (RunningTasks) RunningTasks.Remove(id);
}
public class Worker
{
public delegate void DoneDelegate(int taskId);
public static DoneDelegate Done { private get; set; }
public async void DoWork(object id, string url, CancellationToken token)
{
if (token.IsCancellationRequested) return;
Content obj;
try
{
int tries = 0;
bool IsUrlProcessed = true;
DateTime dtStart = DateTime.Now;
string articleDate = string.Empty;
try
{
ScrapeWeb bot = new ScrapeWeb();
SearchApi searchApi = new SearchApi();
SearchHits searchHits = searchApi.Url(url, 5, 0);
if (searchHits.Hits.Count() == 0)
{
obj = await bot.ReturnArticleObject(url);
if (obj.Code != HttpStatusCode.OK)
{
Console.WriteLine(string.Format("\r Status is {0}", obj.Code));
tries = itemfound.UrlMaxTries + 1;
IsUrlProcessed = false;
itemfound.HttpCode = obj.Code;
}
else
{
string title = obj.Title;
string content = obj.Contents;
string description = obj.Description;
Articles article = new Articles();
article.Site = url.GetSite();
article.Content = content;
article.Title = title;
article.Url = url.ToLower();
article.Description = description;
string strThumbNail = HtmlHelper.GetImageUrl(url, obj.RawResponse);
article.Author = HtmlHelper.GetAuthor(url, obj.RawResponse);
if (!string.IsNullOrEmpty(strThumbNail))
{
//This condition needs to be added to remove ?n=<number> from EP thumbnails
if (strThumbNail.Contains("?"))
{
article.ImageUrl = strThumbNail.Substring(0, strThumbNail.IndexOf("?")).Replace("http:", "https:");
}
else
article.ImageUrl = strThumbNail.Replace("http:", "https:");
}
else
{
article.ImageUrl = string.IsNullOrEmpty(strThumbNail) ? article.Url.GetDefaultImageUrls() : strThumbNail.Replace("http:", "https:");
}
articleDate = HtmlHelper.GetPublishDate(url, obj.RawResponse);
if (string.IsNullOrEmpty(articleDate))
article.Pubdate = DateTime.Now;
else
article.Pubdate = DateTime.Parse(articleDate);
var client = new Index(searchApi);
var result = client.Upsert(article);
itemfound.HttpCode = obj.Code;
if (result)
{
itemfound.DateCreated = DateTime.Parse(articleDate);
itemfound.DateModified = DateTime.Parse(articleDate);
UpdateItem(itemfound);
}
else
{
tries = itemfound.UrlMaxTries + 1;
IsUrlProcessed = false;
itemfound.DateCreated = DateTime.Parse(articleDate);
itemfound.DateModified = DateTime.Parse(articleDate) == null ? DateTime.Now : DateTime.Parse(articleDate);
UpdateItem(itemfound, tries, IsUrlProcessed);
}
}
}
else
{
tries = itemfound.UrlMaxTries + 1;
IsUrlProcessed = true;
itemfound.HttpCode = HttpStatusCode.OK;
itemfound.DateCreated = DateTime.Parse(articleDate);
itemfound.DateModified = DateTime.Parse(articleDate) == null ? DateTime.Now : DateTime.Parse(articleDate);
}
}
catch (Exception e)
{
tries = itemfound.UrlMaxTries + 1;
IsUrlProcessed = false;
itemfound.DateCreated = DateTime.Parse(articleDate);
itemfound.DateModified = DateTime.Parse(articleDate) == null ? DateTime.Now : DateTime.Parse(articleDate);
}
finally
{
DateTime dtEnd = DateTime.Now;
Console.WriteLine(string.Format("\r Total time taken to process items is {0}", (dtEnd - dtStart).TotalSeconds));
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
Done((int)id);
}
}
All this code is based from Best multi-thread approach for multiple web requests this link. Can someone tell me how to get this approach running?
I think the problem is in the way you're creating your tasks:
new Task(id => new Worker().DoWork((int)id, item.Url, token), item.Url, token)
This Task constructor overload expected Action<object> delegate. That means id will be typed as object and you need to cast it back to something useful first.
Parameters
action
Type: System.Action<Object>
The delegate that represents the code to execute in the task.
state
Type: System.Object
An object representing data to be used by the action.
cancellationToken
Type: System.Threading.CancellationToken
-The CancellationToken that that the new task will observe.
You decided to cast it to int by calling (int)id, but you're passing item.Url as the object itself. I can't tell you 100% what the type of Url is but I don't expect Url-named property to be of type int.
Based on what #MarcinJuraszek said I just went back to my code and added an int as I couldn't find another way to resolve it. Here is the change I made
int i=0
foreach (var item in items)
{
urls.Add(item.Url);
WaitingTasks.Enqueue(new Task(id => new Worker().DoWork((string)id, item.Url, token), item.Url, token));
i++;
}
I have two reliable queues and they are being accessed by two guest executables and each of them access their own. Sometimes the function I use to access them doesn't update the reliable queue object in the function and the wrong request is sent to the wrong guest executable.
What happens is that the clientId is passed by the guest executable to this function in the Get request. Let us say that there are two clientId(s) called T1 and T2.
What happens is that the guest executable (client) T2 at times gets the request that was meant for T1. Even though I tried line by line debugging the parameters passed to this function are correct.
Here is my API's POST that is passed a json to be added to the queue for the clients to receive from the GET
[HttpPost("MarketInfo")]
public JObject GetMarketInfo([FromBody] JObject jObject)
{
List<JToken> clients = jObject.GetValue("clients").ToList();
string json;
JObject response = new JObject();
JArray jsonArray = new JArray();
try
{
foreach (JToken client in clients)
{
var id = Guid.NewGuid();
json = "{'name':'MarketInfo','id':'" + id.ToString() + "','mtClientId':'" + terminal["name"].ToString() + "','parameters':{'symbol':'" + terminal["symbol"].ToString() + "','property':24}}";
bool result = _requestsCollectionHandler.CreateRequestForClient(JObject.Parse(json));
JObject clientResponse = new JObject();
if (result==true)
{
clientResponse["name"] = client["name"].ToString();
clientResponse["guid"] = id.ToString();
jsonArray.Add(clientResponse);
}
else
{
clientResponse["name"] = terminal.Children()["name"].ToString();
clientResponse["guid"] = "ERROR";
jsonArray.Add(terminalResponse);
}
}
response["clients"] = jsonArray;
return response;
}
catch (Exception e)
{
Debug.Write(e.Message);
return null;
}
}
This is the json that we pass to this API
{"clients":[{"name":"T1","symbol":"SomeInfo"},{"name":"T2","symbol":"SomeInfo"}]}
The problem is always with the clients object that is passed first.
Before I explain further let me also share the code for the client's HttpGet
[HttpGet("{clientId}")]
public string Get([FromRoute] string clientId)
{
try
{
string request = _requestsCollectionHandler.GetRequestJsonFromQueue(clientId);
return request;
}
catch(Exception e)
{
return e.Message;
}
}
This is the function that creates an object that is to be added by another function in the reliable queue
public bool CreateRequestForClient(JObject jObject)
{
try
{
this._jObject = new JObject(jObject);
CreateKey();
AddToRequestToQueueAsync();
return true;
}
catch (Exception e)
{
Debug.Write(e.Message);
_exceptionMessage = e.Message;
return false;
}
}
private void CreateKey()
{
dynamic data = JObject.Parse(_jObject.ToString(Newtonsoft.Json.Formatting.None));
string name = data.name;
string id = data.id;
string clientId = data.clientId;
_key.id = id;
_key.name = name;
_key.clientId = clientId;
//key.timestamp = GetTimestamp();
_key.timestamp = GetTimestamp();
_key.requestJson = _jObject.ToString(Newtonsoft.Json.Formatting.None);
}
_key is a private variable in class a custom class
This is the function in my class of request handler that adds the requests to the queue
private void AddToRequestToQueueAsync()
{
var transaction = this._stateManager.CreateTransaction();
CancellationToken cancellationToken
= new CancellationToken(false);
try
{
string queue = _key.clientId;
IReliableConcurrentQueue<TerminalResponseKey> reliableQueue =
_stateManager.GetOrAddAsync<IReliableConcurrentQueue<TerminalResponseKey>>(queue).Result;
transaction = this._stateManager.CreateTransaction();
if (reliableQueue!=null)
{
long count = reliableQueue.Count;
reliableQueue.EnqueueAsync(transaction, _key);
count = reliableQueue.Count;
transaction.CommitAsync().Wait();
}
else
{
transaction.Abort();
}
}
catch
{
transaction.Abort();
throw;
}
}
This is function that is used by the client
public string GetRequestJsonFromQueue(string clientId)
{
string queue = clientId;
try
{
IReliableConcurrentQueue<TerminalResponseKey> reliableQueue =
this._stateManager.GetOrAddAsync<IReliableConcurrentQueue<TerminalResponseKey>>(queue).Result;
if(reliableQueue != null)
{
ConditionalValue<TerminalResponseKey> key =
reliableQueue.TryDequeueAsync(transaction).Result;
if(key.HasValue)
{
string request = key.Value.requestJson;
transaction.CommitAsync().Wait();
return request;
}
}
else
{
transaction.Abort();
}
return "NO QUEUE";
}
catch (Exception e)
{
Debug.WriteLine(e);
transaction.Abort();
return e.InnerException.Message;
}
}
As far as I have found out I think my problem is in this function above. Because I don't know how the client T2 or client T1 gets another client's queue because the parameters determining the queue are their IDs and are totally unique.
These Ids are also passed correctly to this:
IReliableConcurrentQueue<TerminalResponseKey> reliableQueue =
this._stateManager.GetOrAddAsync<IReliableConcurrentQueue<TerminalResponseKey>>(queue).Result;
As you can see that we have queue=clientId
I have tried adding proper timespans but it was of no use as there is no exception thrown for OperationTimedOut. Furthermore since I am new to ServiceFabric I maybe totally doing anything wrong.
PS: Sorry for maybe a lot of jumbled up and confused code and question AND SOME OF THE INFORMATION IS OBFUSCATED DUE TO CONFIDENTIALITY BUT NOTHING OBSTRUCTING THE UNDERSTANDING OF THIS IS HIDDEN (I Hope not an issue)
I hope this is not an issue maybe an error I am overlooking at my side
When you put the request in the queue, in AddToRequestToQueueAsync(), the name of the queue is set from _key.terminalId (and I don't see where you assign it), but when you read from it, in GetRequestJsonFromQueue(), the clientId
is used as the queue name.
I have to implement a tcp connection where raw xml data is passed.
Unfortunately there is no message framing, I now this is realy bad, but I have to deal with this...
The Message would look like this:
<?xml version="1.0" encoding="utf-8"?>
<DATA></DATA>
or this
<?xml version="1.0" encoding="utf-8"?>
<DATA />
Now I have to receive messages that could have self closed tags. The message is always the same, it is always like xml description and a data tag with inner xml that is the message content.
So if it would be without self closed tags, this would be easy, but how can I read both?
By the way I am using the TcpListener.
Edit :
Everything is fine if there is no self closed tag.
if (_clientSocket != null)
{
NetworkStream networkStream = _clientSocket.GetStream();
_clientSocket.ReceiveTimeout = 100; // 1000 miliseconds
while (_continueProcess)
{
if (networkStream.DataAvailable)
{
bool isMessageComplete = false;
String messageString = String.Empty;
while (!isMessageComplete)
{
var bytes = new byte[_clientSocket.ReceiveBufferSize];
try
{
int bytesReaded = networkStream.Read(bytes, 0, (int) _clientSocket.ReceiveBufferSize);
if (bytesReaded > 0)
{
var data = Encoding.UTF8.GetString(bytes, 0, bytesReaded);
messageString += data;
if (messageString.IndexOf("<DATA", StringComparison.OrdinalIgnoreCase) > 0 &&
messageString.IndexOf("</DATA", StringComparison.OrdinalIgnoreCase) > 0)
{
isMessageComplete = true;
}
}
}
catch (IOException)
{
// Timeout
}
catch (SocketException)
{
Console.WriteLine("Conection is broken!");
break;
}
}
}
Thread.Sleep(200);
} // while ( _continueProcess )
networkStream.Close();
_clientSocket.Close();
}
Edit 2 (30.03.2015 12:00)
Unfortunately it is not possible to use some kind of message frame.
So I ended up to use this part of code (DATA is my root node):
if (_clientSocket != null)
{
NetworkStream networkStream = _clientSocket.GetStream();
_clientSocket.ReceiveTimeout = 100;
string data = string.Empty;
while (_continueProcess)
{
try
{
if (networkStream.DataAvailable)
{
Stopwatch sw = new Stopwatch();
sw.Start();
var bytes = new byte[_clientSocket.ReceiveBufferSize];
int completeXmlLength = 0;
int bytesReaded = networkStream.Read(bytes, 0, (int) _clientSocket.ReceiveBufferSize);
if (bytesReaded > 0)
{
message.AddRange(bytes);
data += Encoding.UTF8.GetString(bytes, 0, bytesReaded);
if (data.IndexOf("<?", StringComparison.Ordinal) == 0)
{
if (data.IndexOf("<DATA", StringComparison.Ordinal) > 0)
{
Int32 rootStartPos = data.IndexOf("<DATA", StringComparison.Ordinal);
completeXmlLength += rootStartPos;
var root = data.Substring(rootStartPos);
int rootCloseTagPos = root.IndexOf(">", StringComparison.Ordinal);
Int32 rootSelfClosedTagPos = root.IndexOf("/>", StringComparison.Ordinal);
// If there is an empty tag that is self closed.
if (rootSelfClosedTagPos > 0)
{
string rootTag = root.Substring(0, rootSelfClosedTagPos +1);
// If there is no '>' between the self closed tag and the start of '<DATA'
// the root element is empty.
if (rootTag.IndexOf(">", StringComparison.Ordinal) <= 0)
{
completeXmlLength += rootSelfClosedTagPos;
string messageXmlString = data.Substring(0, completeXmlLength + 1);
data = data.Substring(messageXmlString.Length);
try
{
// parse complete xml.
XDocument xmlDocument = XDocument.Parse(messageXmlString);
}
catch(Exception)
{
// Invalid Xml.
}
continue;
}
}
if (rootCloseTagPos > 0)
{
Int32 rootEndTagStartPos = root.IndexOf("</DATA", StringComparison.Ordinal);
if (rootEndTagStartPos > 0)
{
var endTagString = root.Substring(rootEndTagStartPos);
completeXmlLength += rootEndTagStartPos;
Int32 completeEndPos = endTagString.IndexOf(">", StringComparison.Ordinal);
if (completeEndPos > 0)
{
completeXmlLength += completeEndPos;
string messageXmlString = data.Substring(0, completeXmlLength + 1);
data = data.Substring(messageXmlString.Length);
try
{
// parse complete xml.
XDocument xmlDocument = XDocument.Parse(messageXmlString);
}
catch(Exception)
{
// Invalid Xml.
}
}
}
}
}
}
}
sw.Stop();
string timeElapsed = sw.Elapsed.ToString();
}
}
catch (IOException)
{
data = String.Empty;
}
catch (SocketException)
{
Console.WriteLine("Conection is broken!");
break;
}
}
This code I had use if ther were some kind of message framing, in this case 4 bytes of message length:
if (_clientSocket != null)
{
NetworkStream networkStream = _clientSocket.GetStream();
_clientSocket.ReceiveTimeout = 100;
string data = string.Empty;
while (_continueProcess)
{
try
{
if (networkStream.DataAvailable)
{
Stopwatch sw = new Stopwatch();
sw.Start();
var lengthBytes = new byte[sizeof (Int32)];
int bytesReaded = networkStream.Read(lengthBytes, 0, sizeof (Int32) - offset);
if (bytesReaded > 0)
{
offset += bytesReaded;
message.AddRange(lengthBytes.Take(bytesReaded));
}
if (offset < sizeof (Int32))
{
continue;
}
Int32 length = BitConverter.ToInt32(message.Take(sizeof(Int32)).ToArray(), 0);
message.Clear();
while (length > 0)
{
Int32 bytesToRead = length < _clientSocket.ReceiveBufferSize ? length : _clientSocket.ReceiveBufferSize;
byte[] messageBytes = new byte[bytesToRead];
bytesReaded = networkStream.Read(messageBytes, 0, bytesToRead);
length = length - bytesReaded;
message.AddRange(messageBytes);
}
try
{
string xml = Encoding.UTF8.GetString(message.ToArray());
XDocument xDocument = XDocument.Parse(xml);
}
catch (Exception ex)
{
// Invalid Xml.
}
sw.Stop();
string timeElapsed = sw.Elapsed.ToString();
}
}
catch (IOException)
{
data = String.Empty;
}
catch (SocketException)
{
Console.WriteLine("Conection is broken!");
break;
}
}
Like you can see I wanted to measure the elapsed time, to see witch methode has a better performance. The strange thing is that the methode whith no message framing has an average time of 0,2290 ms, the other methode has an average time of 1,2253 ms.
Can someone explain me why? I thought the one without message framing would be slower...
Hand the NetworkStream to the .NET XML infrastructure. For example create an XmlReader from the NetworkStream.
Unfortunately I did not find a built-in way to easily create an XmlDocument from an XmlReader that has multiple documents in it. It complains about multiple root elements (which is correct). You would need to wrap the XmlReader and make it stop returning nodes when the first document is done. You can do that by keeping track of some state and by looking at the nesting level. When the nesting level is zero again the first document is done.
This is just a raw sketch. I'm pretty sure this will work and it handles all possible XML documents.
No need for this horrible string processing code that you have there. The existing code looks quite slow as well but since this approach is much better it serves no purpose to comment on the perf issues. You need to throw this away.
I had the same problem - 3rd party system sends messages in XML format via TCP but my TCP client application may receive message partially or several messages at once. One of my colleagues proposed very simple and quite generic solution.
The idea is to have a string buffer which should be populated char by char from TCP stream, after each char try to parse buffer content with regular .Net XML parser. If parser throws an exception - continue adding chars to the buffer. Otherwise - message is ready and can be processed by application.
Here is the code:
private object _dataReceiverLock = new object();
private string _messageBuffer;
private Stopwatch _timeSinceLastMessage = new Stopwatch();
private List<string> NormalizeMessage(string rawMsg)
{
lock (_dataReceiverLock)
{
List<string> result = new List<string>();
//following code prevents buffer to store too old information
if (_timeSinceLastMessage.ElapsedMilliseconds > _settings.ResponseTimeout)
{
_messageBuffer = string.Empty;
}
_timeSinceLastMessage.Restart();
foreach (var ch in rawMsg)
{
_messageBuffer += ch;
if (ch == '>')//to avoid extra checks
{
if (IsValidXml(_messageBuffer))
{
result.Add(_messageBuffer);
_messageBuffer = string.Empty;
}
}
}
return result;
}
}
private bool IsValidXml(string xml)
{
try
{
//fastest way to validate XML format correctness
using (XmlTextReader reader = new XmlTextReader(new StringReader(xml)))
{
while (reader.Read()) { }
}
return true;
}
catch
{
return false;
}
}
Few comments:
Need to control lifetime of string buffer, otherwise in case network disconnection old information may stay in the buffer forever
There major problem here is the performance - parsing after every new character is quite slow. So need to add some optimizations, such as parse only after '>' character.
Make sure this method is thread safe, otherwise several threads may flood string buffer with different XML pieces.
The usage is simple:
private void _tcpClient_DataReceived(byte[] data)
{
var rawMsg = Encoding.Unicode.GetString(data);
var normalizedMessages = NormalizeMessage(rawMsg);
foreach (var normalizedMessage in normalizedMessages)
{
//TODO: your logic
}
}
Scenario
One windows service polls a url every two minutes to retrieve certain data.
If any data has been added since the previous call, the data is retrieved and stored otherwise the loop carries on.
Issue
Sometimes a request takes more than two minutes to return a response.
When this happens, the next request is still made and finds new data, since the previous request hasn't return a response yet
This results in duplicate entries when the data is stored.
What I've tried
I tried to handle that by using a boolean like so:
Boolean InProgress = true;
foreach (var item in Lists)
{
\\Make a request and return new data (if any)
InProgress = false;
if (InProgress = false)
{
\\Store new data
}
}
This doesn't solve the issue. I believe I'm using the boolean in wrong place, but I'm not sure where it should.
This is the loop that makes the request and store the data
void serviceTimer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
Data getCredentials = new Data();
DataTable credentials = getCredentials.loadCredentials();
Boolean InProgress = true;
for (int i = 0; i < credentials.Rows.Count; i++)
{
if (credentials != null)
{
var PBranchID = (int)credentials.Rows[i]["PortalBranchID"];
var negRef = (int)credentials.Rows[i]["NegotiatorRef"];
var Username = credentials.Rows[i]["Username"].ToString();
var Password = credentials.Rows[i]["Password"].ToString();
var Domain = credentials.Rows[i]["Domain"].ToString();
var FooCompanyBaseUrl = "https://" + Domain + ".FooCompany.com/";
Data getCalls = new Data();
DataTable calls = getCalls.loadCalls(PBranchID);
//If it's not the first call
if (calls != null && calls.Rows.Count > 0)
{
//Makes a call
DateTime CreatedSince = DateTime.SpecifyKind((DateTime)calls.Rows[0]["LastSuccessOn"], DateTimeKind.Local);
string IssueListUrl = FooCompany.WebApi.V2.URLs.Issues(BaseUrl, null, CreatedSince.ToUniversalTime(), null);
FooCompany.WebApi.V2.DTO.PrevNextPagedList resultIssueList;
resultIssueList = FooCompany.WebApi.Client.Helper.Utils.Getter<Foocompany.WebApi.V2.DTO.PrevNextPagedList>(IssueListUrl, Username, Password);
InProgress = false;
if (InProgress == false)
{
if (resultIssueList.Items.Count > 0)
{
//If call returns new issues, save call
Data saveCalls = new Data();
saveCalls.saveCalls(PBranchID);
foreach (var item in resultIssueList.Items)
{
var Issue = FooCompany.WebApi.Client.Helper.Utils.Getter<FooCompany.WebApi.V2.DTO.Issue>(item, Username, Password);
string TenantSurname = Issue.Surname;
string TenantEmail = Issue.EmailAddress;
Data tenants = new Data();
int tenantPropRef = Convert.ToInt32(tenants.loadTenantPropRef(PBranchID, TenantSurname, TenantEmail));
Data Properties = new Data();
DataTable propAddress = Properties.loadPropAddress(PBranchID, tenantPropRef);
var Address1 = propAddress.Rows[0]["Address1"];
var Address2 = propAddress.Rows[0]["Address2"];
var AddressFolder = Address1 + "," + Address2;
if (!Directory.Exists("path"))
{
Directory.CreateDirectory("path");
}
string ReportPDFDestination = "path";
if (File.Exists(ReportPDFDestination))
{
File.Delete(ReportPDFDestination);
}
FooCompany.WebApi.Client.Helper.Utils.DownloadFileAuthenticated(FooCompany.WebApi.V2.URLs.IssueReport(BaseUrl, Issue.Id), Username, Password, ReportPDFDestination);
//Store data
}
IssueListUrl = resultIssueList.NextURL;
}
}
}
else
{
continue;
}
}
}
catch (Exception ex)
{
//write to log
}
}
Question
I'm sure there is a better way than a boolean.
Could anyone advice a different method to handle the issue properly?
Thanks.
Solution
I ended up using a combination of both Thomas and Mason suggestions. I wrapped a lock statement around the main function of my windows service and used a boolean inside the function section that makes the call to the remote server.
Tested many times and it's error free.
You seems to have a problem of synchronisation, just surround the code that iterate though the List with a lock, and you will be fine.
public class MyClass{
private readonly object internalLock= new object();
private bool AlreadyRunning { get; set; }
void serviceTimer_Elapsed(object sender, ElapsedEventArgs e)
{
if(AlreadyRunning){
return;
}
try{
lock(internalLock){
Thread.MemoryBarrier();
if(AlreadyRunning){
return;
}
AlreadyRunning = true;
...Do all the things...
}
}
catch(Exception e){
..Exception handling
}
finally
{
AlreadyRunning = false;
}
}
bool InProgress=false;
void serviceTimer_Elapsed(object sender, ElapsedEventArgs e)
{
if(!InProgress)
{
InProgress=true;
//retrieve data
InProgress=false;
}
}
Your InProgress variable needs to be declared outside the event handler. When you enter the method, check to see if it's already running. If it is, then we do nothing. If it's not running, then we say it's running, retrieve our data, then reset our flag to say we've finished running.
You'll probably need to add appropriate locks for thread safety, similar to Thomas's answer.
Im trying to send some object from a server to the client.
My problem is that when im sending only 1 object, everything works correctly. But at the moment i add another object an exception is thrown - "binary stream does not contain a valid binaryheader" or "No map for object (random number)".
My thoughts are that the deserialization does not understand where the stream starts / ends and i hoped that you guys can help me out here.
heres my deserialization code:
public void Listen()
{
try
{
bool offline = true;
Dispatcher.Invoke(System.Windows.Threading.DispatcherPriority.Normal,
new Action(() => offline = Offline));
while (!offline)
{
TcpObject tcpObject = new TcpObject();
IFormatter formatter = new BinaryFormatter();
tcpObject = (TcpObject)formatter.Deserialize(serverStream);
if (tcpObject.Command == Command.Transfer)
{
SentAntenna sentAntenna = (SentAntenna)tcpObject.Object;
int idx = 0;
foreach (string name in SharedProperties.AntennaNames)
{
if (name == sentAntenna.Name)
break;
idx++;
}
if (idx < 9)
{
PointCollection pointCollection = new PointCollection();
foreach (Frequency f in sentAntenna.Frequencies)
pointCollection.Add(new Point(f.Channel, f.Intensity));
SharedProperties.AntennaPoints[idx] = pointCollection;
}
}
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message); // raise an event
}
}
serialization code:
case Command.Transfer:
Console.WriteLine("Transfering");
Thread transfer = new Thread(new ThreadStart(delegate
{
try
{
string aName = tcpObject.Object.ToString();
int indx = 0;
foreach (string name in names)
{
if (name == aName)
break;
indx++;
}
if (indx < 9)
{
while (true) // need to kill when the father thread terminates
{
if (antennas[indx].Frequencies != null)
{
lock (antennas[indx].Frequencies)
{
TcpObject sendTcpObject = new TcpObject();
sendTcpObject.Command = Command.Transfer;
SentAntenna sa = new SentAntenna(antennas[indx].Frequencies, aName);
sendTcpObject.Object = sa;
formatter.Serialize(networkStream, sendTcpObject);
}
}
}
}
}
catch (Exception ex) { Console.WriteLine(ex); }
}));
transfer.Start();
break;
Interesting. There's nothing particularly odd in your serialization code, and I've seen people use vanilla concatenation for multiple objects in the past, although I've actually always advised against it as BinaryFormatter does not explicitly claim this scenario is OK. But: if it isn't, the only thing I can suggest is to implement your own framing; so your write code becomes:
serialize to an empty MemoryStream
note the length and write the length to the NetworkStream, for example as a simple fixed-width 32-bit network-byte-order integer
write the payload from the MemoryStream to the NetworkStream
rinse, repeat
And the read code becomes:
read exactly 4 bytes and compute the length
buffer that many bytes into a MemoryStream
deserialize from the NetworkStream
(Noting in both cases to set the MemoryStream's position back to 0 between write and read)
You can also implement a Stream-subclass that caps the length if you want to avoid a buffer when reading, bit that is more complex.
apperantly i came up with a really simple solution. I just made sure only 1 thread is allowed to transfer data at the same time so i changed this line of code:
formatter.Serialize(networkStream, sendTcpObject);
to these lines of code:
if (!transfering) // making sure only 1 thread is transfering data
{
transfering = true;
formatter.Serialize(networkStream, sendTcpObject);
transfering = false;
}