Develop Reference

Problem with passing constantly reading serial buffer data to another class

Long story short. ;
I have a class named Scope. And this class contains all logic for scope operations etc. It also starts backround thread that constantly read serial port data (in my case events was unreliable):
Thread BackgroundReader = new Thread(ReadBuffer);
BackgroundReader.IsBackground = true;
BackgroundReader.Start();
private void ReadBuffer()
{
SerialPort.DiscardInBuffer();
while (!_stopCapture)
{
int bufferSize = SerialPort.BytesToRead;
byte[] buffer = new byte[bufferSize];
if(bufferSize > 5)
{
SerialPort.Read(buffer, 0, bufferSize);
Port_DataReceivedEvent(buffer, null);
}
Thread.Sleep(_readDelay);
}
CurrentBuffer = null;
}
In Scope class there is a public field named Buffer
public byte[] Buffer
{
get
{
return CurrentBuffer;
}
}
And here is event fired while there is new data readed
private void Port_DataReceivedEvent(object sender, EventArgs e)
{
//populate buffer
Info(sender, null);
CurrentBuffer = ((byte[])sender);
foreach(byte data in CurrentBuffer)
{
DataBuffer.Enqueue(data);
}
if (DataBuffer.Count() > _recordLength)
{
GenerateFrame(DataBuffer.ToArray());
DataBuffer.Clear(); ;
}
}
To make code more manageable, I splitted it in several classes. One of this classes is for searching specific data pattern in current stream and create specific object from this data. This code works in way that send to serial port specific command and expect return frame. If reponse is not received or not ok, send is performed again and again until correct response arrives or there will be timeout. Response is expected to be in current buffer. Those strange string manipulation is for debug purposes.
public class GetAcknowledgedFrame
{
byte[] WritedData;
string lastEx;
string stringData;
public DataFrame WriteAcknowledged(Type SendType, Type ReturnType, JyeScope scope)
{
var stopwatch = new Stopwatch();
stopwatch.Restart();
while (stopwatch.ElapsedMilliseconds < scope.TimeoutTime)
{
try
{
if (SendType == typeof(GetParameters))
{
WriteFrame(new ScopeControlFrames.GetParameters(), scope.SerialPort);
}
else if(SendType == typeof(GetConfig))
{
WriteFrame(new ScopeControlFrames.GetConfig(), scope.SerialPort);
}
else if (SendType == typeof(EnterUSBScopeMode))
{
WriteFrame(new ScopeControlFrames.EnterUSBScopeMode(), scope.SerialPort);
}
return ReturnFrame(ReturnType, scope.Buffer, scope.TimeoutTime);
}
catch (InvalidDataFrameException ex)
{
lastEx = ex.Message;
System.Threading.Thread.Sleep(10);
}
}
stringData = "";
foreach (var data in scope.Buffer)
{
stringData += data + ",";
}
stringData.Remove(stringData.Length - 1);
throw new TimeoutException($"Timeout while waiting for frame acknowledge: " + SendType.ToString() + ", " + ReturnType.ToString() + Environment.NewLine+ "Add. err: "+lastEx);
}
private DataFrame ReturnFrame(Type FrameType, byte[] buffer, int timeoutTime)
{
if (FrameType == typeof(DataFrames.DSO068.CurrConfigDataFrame))
{
DataFrames.DSO068.CurrConfigDataFrame CurrConfig = new DataFrames.DSO068.CurrConfigDataFrame(buffer);
return CurrConfig;
}
else if (FrameType == typeof(DataFrames.DSO112.CurrConfigDataFrame))
{
DataFrames.DSO112.CurrConfigDataFrame CurrParam = new DataFrames.DSO112.CurrConfigDataFrame(buffer);
return CurrParam;
}
else if (FrameType == typeof(CurrParamDataFrame))
{
CurrParamDataFrame CurrParam = new CurrParamDataFrame(buffer);
return CurrParam;
}
else if (FrameType == typeof(DataBlockDataFrame))
{
DataBlockDataFrame CurrData = new DataBlockDataFrame(buffer);
return CurrData;
}
else if (FrameType == typeof(DataSampleDataFrame))
{
DataSampleDataFrame CurrData = new DataSampleDataFrame(buffer);
return CurrData;
}
else if (FrameType == typeof(ScopeControlFrames.ScopeReady))
{
ScopeControlFrames.ScopeReady ready = new ScopeControlFrames.ScopeReady(buffer);
return ready;
}
else
{
throw new InvalidOperationException("Wrong object type");
}
}
private bool WriteFrame(DataFrame frame, IStreamResource port)
{
WritedData = frame.Data;
port.Write(frame.Data, 0, frame.Data.Count());
return true;
}
}
From main class (and main thread) I call method in this class, for example:
var Ready = (ScopeControlFrames.ScopeReady)new GetAcknowledgedFrame().WriteAcknowledged
(typeof(ScopeControlFrames.EnterUSBScopeMode), typeof(ScopeControlFrames.ScopeReady), this);
The problem is when I pass "this" object (that has thread working in background) to my helper class. It seems like helper class not see changing data in this object. The problem started when I separate code of my helper class from main class.
My questions:
- I know that object are passed by reference, that means I think that when object is dynamically changing its state (in this case data buffer should changing while new data is received) all classes that has reference to this object are also seeing this changes. Maybe I'm missing something?
- I tried passing array (by ref), arrays are also reference types. But this not help me at all. Maybe I'm missing something?
I tried changing this class to static, it not helped.
Many thanks for help.

The code below;
Info(sender, null);
CurrentBuffer = ((byte[])sender);
is creating a new reference variable called CurrentBuffer. Any other code holding a reference 'pointer' to the CurrentBuffer value prior to this line of code will not get the new value of CurrentBuffer when its reset.

C# Socket: Client Mishandle 'a' as the Client's id

There are two programs that I made that didn't work. There are the server and the client. The server accepts many client by giving a user an ID (starting from 0). The server sends out the command to the specific client based up the server's id. (Example: 200 client are connected to 1 server. The server's selected id is '5', so the server will send the command to all of the client, and the client will ask the server what ID he wants to execute his command on, and if it's '5', that client will execute and send data to the server). The client has many commands, but to create the smallest code with the same error, I only use 1 command (dir). Basically, the server sends the command to the client and if it matches with the client current id and the server current id, it will process the command. By default, the server's current ID is 10. Here are the list of the commands to help the people who wants to answer:
Server Command:
list (Shows all of the users ID connected and the server's current ID) --> Happens on server
dir (request the client to send its dir listing) --> Sent by the client, read by the Server
set (set the server's current id to any number) (example: 'set 4')
Client:
using System;
using System.Speech.Synthesis;
using System.Windows.Forms;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading;
using System.Net.Sockets;
using System.Net;
namespace clientControl
{
class Program
{
public static string directory = #"C:\";
public static int id = -10;
public static Socket sck = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
static void Main(string[] args)
{
Connect();
getSession();
ReadResponse(sck);
}
static byte[] readResponseFunc()
{
long fileSize = 0;
string fileSizeInString = null;
byte[] fileSizeInByteArray = new byte[1024];
int fileSizeLength = sck.Receive(fileSizeInByteArray);
for (int i = 0; i < fileSizeLength; i++)
{
fileSizeInString = fileSizeInString + (char)fileSizeInByteArray[i];
}
try
{
fileSize = Convert.ToInt64(fileSizeInString);
}
catch { Console.WriteLine(fileSizeInString); }
sck.Send(Encoding.ASCII.GetBytes("a"));
byte[] responseUnknown = new byte[1];
sck.Receive(responseUnknown);
if (Encoding.ASCII.GetString(responseUnknown) == "b")
{
byte[] dataInByteArray = new byte[fileSize];
int dataLength = sck.Receive(dataInByteArray);
return dataInByteArray;
}
return Encoding.ASCII.GetBytes("ERROR RESPONSE FUNC");
}
static void getSession()
{
byte[] message_1 = Encoding.ASCII.GetBytes("make_id");
sck.Send(Encoding.ASCII.GetBytes(message_1.Length.ToString()));
byte[] responseUnknown = new byte[1];
if (Encoding.ASCII.GetString(responseUnknown) == "a")
{
sck.Send(Encoding.ASCII.GetBytes("b"));
sck.Send(message_1);
}
byte[] receivedID = readResponseFunc();
id = Convert.ToInt32(Encoding.ASCII.GetString(receivedID));
}
static bool SocketConnected(Socket s)
{
bool part1 = s.Poll(1000, SelectMode.SelectRead);
bool part2 = (s.Available == 0);
if (part1 && part2)
return false;
else
return true;
}
static void ReadResponse(Socket sck)
{
while (true)
{
if (SocketConnected(sck) == true)
{
try
{
string response = Encoding.ASCII.GetString(readResponseFunc());
byte[] message_1 = Encoding.ASCII.GetBytes("get_id");
sck.Send(Encoding.ASCII.GetBytes(message_1.Length.ToString()));
byte[] responseUnknown = new byte[1];
if (Encoding.ASCII.GetString(responseUnknown) == "a")
{
sck.Send(Encoding.ASCII.GetBytes("b"));
sck.Send(message_1);
}
byte[] response_2InByteArray = readResponseFunc();
string response_2 = Encoding.ASCII.GetString(response_2InByteArray);
if (Convert.ToInt32(response_2) == id)
{
if (response == "dir")
{
string resultOfDirring = "Current Directory: " + directory + "\n\n";
string[] folderListingArray = Directory.GetDirectories(directory);
foreach (string dir in folderListingArray)
{
string formed = "DIRECTORY: " + Path.GetFileName(dir);
resultOfDirring = resultOfDirring + formed + Environment.NewLine;
}
string[] fileListingArray = Directory.GetFiles(directory);
foreach (var file in fileListingArray)
{
FileInfo fileInfo = new FileInfo(file);
string formed = "FILE: " + Path.GetFileName(file) + " - FILE SIZE: " + fileInfo.Length + " BYTES";
resultOfDirring = resultOfDirring + formed + Environment.NewLine;
}
byte[] message_11 = Encoding.ASCII.GetBytes(resultOfDirring);
sck.Send(Encoding.ASCII.GetBytes(message_11.Length.ToString()));
byte[] responseUnknown_11 = new byte[1];
if (Encoding.ASCII.GetString(responseUnknown_11) == "a")
{
sck.Send(Encoding.ASCII.GetBytes("b"));
sck.Send(message_11);
}
}
}
else { }
}
catch { if (SocketConnected(sck) == false) { Console.WriteLine("Client Disconnected: " + sck.RemoteEndPoint); break; }; }
}
else if (SocketConnected(sck) == false) { Console.WriteLine("Client Disconnected: " + sck.RemoteEndPoint); break; }
}
}
static void Connect()
{
while (true)
{
try
{
sck.Connect(IPAddress.Parse("127.0.0.1"), 80);
break;
}
catch { }
}
}
}
}
Server:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Sockets;
using System.IO;
using System.Text;
using System.Threading;
namespace serverControl
{
class Program
{
public static int ftpNum = 1;
public static List<int> listOfClient = new List<int>();
public static TcpListener server = new TcpListener(IPAddress.Parse("127.0.0.1"), 80);
public static string message = null;
public static int id = 0;
public static int selected_id = 10;
static void Main(string[] args)
{
server.Start();
Thread startHandlingClientThread = new Thread(startHandlingClient);
startHandlingClientThread.Start();
while (true)
{
Console.Write(":> ");
string rawmessage = Console.ReadLine();
if (rawmessage == "list")
{
Console.WriteLine("SELECTED ID: " + selected_id);
Console.WriteLine("List of Clients ID:");
for (int i = 0; i < listOfClient.Count; i++)
{
Console.WriteLine(listOfClient[i]);
}
message = rawmessage+"PREVENT_REPETITION_IN_COMMAND";
}
else if (rawmessage.Contains("set ")) { int wantedChangeId = Convert.ToInt32(rawmessage.Replace("set ", "")); selected_id = wantedChangeId; message = rawmessage+ "PREVENT_REPETITION_IN_COMMAND"; }
else
{
message = rawmessage;
}
}
}
static byte[] readResponseFunc(Socket sck)
{
long fileSize = 0;
string fileSizeInString = null;
byte[] fileSizeInByteArray = new byte[1024];
int fileSizeLength = sck.Receive(fileSizeInByteArray);
for (int i = 0; i < fileSizeLength; i++)
{
fileSizeInString = fileSizeInString + (char)fileSizeInByteArray[i];
}
fileSize = Convert.ToInt64(fileSizeInString);
sck.Send(Encoding.ASCII.GetBytes("a"));
byte[] responseUnknown = new byte[1];
sck.Receive(responseUnknown);
if (Encoding.ASCII.GetString(responseUnknown) == "b")
{
byte[] dataInByteArray = new byte[fileSize];
int dataLength = sck.Receive(dataInByteArray);
return dataInByteArray;
}
return Encoding.ASCII.GetBytes("ERROR RESPONSE FUNC");
}
static void startHandlingClient()
{
while (true)
{
handleClient(server);
}
}
static void handleClient(TcpListener clientToAccept)
{
Socket sck = clientToAccept.AcceptSocket();
Thread myNewThread = new Thread(() => ReadResponse(sck));
myNewThread.Start();
}
static bool SocketConnected(Socket s)
{
bool part1 = s.Poll(1000, SelectMode.SelectRead);
bool part2 = (s.Available == 0);
if (part1 && part2)
return false;
else
return true;
}
static void ReadResponse(Socket sck)
{
Thread myNewThread = new Thread(() => SendtoClient(sck));
myNewThread.Start();
Thread.Sleep(2000);
while (true)
{
if (SocketConnected(sck) == true)
{
try
{
byte[] dataInByteArray = readResponseFunc(sck);
string response = Encoding.ASCII.GetString(dataInByteArray);
Console.WriteLine("res: " + response);
if (response != "make_id" && response != "get_id") { Console.WriteLine(response); }
if (response == "make_id")
{
Console.WriteLine("Someone wants an ID");
byte[] message_1 = Encoding.ASCII.GetBytes(id.ToString());
listOfClient.Add(id);
// START
sck.Send(Encoding.ASCII.GetBytes(message_1.Length.ToString()));
byte[] responseUnknown = new byte[1];
if (Encoding.ASCII.GetString(responseUnknown) == "a")
{
sck.Send(Encoding.ASCII.GetBytes("b"));
sck.Send(message_1);
}
id++;
}
if (response == "get_id")
{
byte[] message_1 = Encoding.ASCII.GetBytes(selected_id.ToString());
sck.Send(Encoding.ASCII.GetBytes(message_1.Length.ToString()));
byte[] responseUnknown = new byte[1];
if (Encoding.ASCII.GetString(responseUnknown) == "a")
{
sck.Send(Encoding.ASCII.GetBytes("b"));
sck.Send(message_1);
}
}
}
catch { if (SocketConnected(sck) == false) { Console.WriteLine("Client Disconnected: " + sck.RemoteEndPoint); break; }; }
}
else if (SocketConnected(sck) == false) { Console.WriteLine("Client Disconnected: " + sck.RemoteEndPoint); break; }
}
}
static void SendtoClient(Socket sck)
{
string tempmessage = null;
while (true)
{
if (SocketConnected(sck) == true)
{
if (tempmessage != message)
{
if (!message.Contains("PREVENT_REPETITION_IN_COMMAND"))
{
byte[] message_1 = Encoding.ASCII.GetBytes(message);
sck.Send(Encoding.ASCII.GetBytes(message_1.Length.ToString()));
byte[] responseUnknown = new byte[1];
if (Encoding.ASCII.GetString(responseUnknown) == "a")
{
sck.Send(Encoding.ASCII.GetBytes("b"));
sck.Send(message_1);
}
}
tempmessage = message;
}
}
else if (SocketConnected(sck) == false)
{ Console.WriteLine("Client Disconnected: " + sck.RemoteEndPoint); break; }
}
}
}
}
Problem:
The problem is within the GetSession or the ReadResponseFunc function. The client thinks that his ID received by the server is 'a' (it's suppose to be an integer). I don't want a solution that suggest me to use other libs or
the TcpClient class
Bounty:
I'll put up a bounty with no expiry time to those who solve the problem.

The logic in your code is very confusing. My question to you: Why are you sending 'a' and 'b' back and forth between the server and client? Is it some sort of confirmation that the message has been received?
Anyways, throughout the quick tests I've done just now, it seems that the problem is Line 59 of your server:
sck.Send(Encoding.ASCII.GetBytes("a"));
Here's what I figured out during testing:
Server executes.
Client executes.
Client sends server the length of "make_id" (Line 51 of client)
Client waits for a response to return.
Server reads the value and sends back 'a' (Line 59 of server)
You may want to spend some time to straighten out your protocol so it's less confusing and more organized. That would help people like me and you spot bugs much more easily.

The user 'Bobby' has already found your problem. Credits go to him.
I further suggest that you use less threads, as thread synchronisation needs some effort when doing it right: all data that is accessed from different threads must be secured by locks, so that no outdated values remain in the CPU caches. Use .net Monitor from the "threading synchronisation primitives" for that job.
About the threads themselves:
You should have only one thread in the server. This thread takes all clients from a list (secured by Monitor), in which they were added when connection attempts were received. On each client it checks for incoming messages, evaluates them and replies with own messages if needed.
The client also has just one thread, that will loop (dont forget a sleep in the loop or you will have 100% usage of the used CPU core), send messages when desired and wait for replies when messages were sent.
About the messages:
I already gave some proposals in a comment to Bobby's answer. I suggest you create a CMessage class that has a Serialize() and Deserialze() to create a byte array to send from the CMessage members (serializing the content) or vice versa filling the members from the received bytes. You then may use this class in both programs and you'll have common solution.

Xml over tcp without message frame

I have to implement a tcp connection where raw xml data is passed.
Unfortunately there is no message framing, I now this is realy bad, but I have to deal with this...
The Message would look like this:
<?xml version="1.0" encoding="utf-8"?>
<DATA></DATA>
or this
<?xml version="1.0" encoding="utf-8"?>
<DATA />
Now I have to receive messages that could have self closed tags. The message is always the same, it is always like xml description and a data tag with inner xml that is the message content.
So if it would be without self closed tags, this would be easy, but how can I read both?
By the way I am using the TcpListener.
Edit :
Everything is fine if there is no self closed tag.
if (_clientSocket != null)
{
NetworkStream networkStream = _clientSocket.GetStream();
_clientSocket.ReceiveTimeout = 100; // 1000 miliseconds
while (_continueProcess)
{
if (networkStream.DataAvailable)
{
bool isMessageComplete = false;
String messageString = String.Empty;
while (!isMessageComplete)
{
var bytes = new byte[_clientSocket.ReceiveBufferSize];
try
{
int bytesReaded = networkStream.Read(bytes, 0, (int) _clientSocket.ReceiveBufferSize);
if (bytesReaded > 0)
{
var data = Encoding.UTF8.GetString(bytes, 0, bytesReaded);
messageString += data;
if (messageString.IndexOf("<DATA", StringComparison.OrdinalIgnoreCase) > 0 &&
messageString.IndexOf("</DATA", StringComparison.OrdinalIgnoreCase) > 0)
{
isMessageComplete = true;
}
}
}
catch (IOException)
{
// Timeout
}
catch (SocketException)
{
Console.WriteLine("Conection is broken!");
break;
}
}
}
Thread.Sleep(200);
} // while ( _continueProcess )
networkStream.Close();
_clientSocket.Close();
}
Edit 2 (30.03.2015 12:00)
Unfortunately it is not possible to use some kind of message frame.
So I ended up to use this part of code (DATA is my root node):
if (_clientSocket != null)
{
NetworkStream networkStream = _clientSocket.GetStream();
_clientSocket.ReceiveTimeout = 100;
string data = string.Empty;
while (_continueProcess)
{
try
{
if (networkStream.DataAvailable)
{
Stopwatch sw = new Stopwatch();
sw.Start();
var bytes = new byte[_clientSocket.ReceiveBufferSize];
int completeXmlLength = 0;
int bytesReaded = networkStream.Read(bytes, 0, (int) _clientSocket.ReceiveBufferSize);
if (bytesReaded > 0)
{
message.AddRange(bytes);
data += Encoding.UTF8.GetString(bytes, 0, bytesReaded);
if (data.IndexOf("<?", StringComparison.Ordinal) == 0)
{
if (data.IndexOf("<DATA", StringComparison.Ordinal) > 0)
{
Int32 rootStartPos = data.IndexOf("<DATA", StringComparison.Ordinal);
completeXmlLength += rootStartPos;
var root = data.Substring(rootStartPos);
int rootCloseTagPos = root.IndexOf(">", StringComparison.Ordinal);
Int32 rootSelfClosedTagPos = root.IndexOf("/>", StringComparison.Ordinal);
// If there is an empty tag that is self closed.
if (rootSelfClosedTagPos > 0)
{
string rootTag = root.Substring(0, rootSelfClosedTagPos +1);
// If there is no '>' between the self closed tag and the start of '<DATA'
// the root element is empty.
if (rootTag.IndexOf(">", StringComparison.Ordinal) <= 0)
{
completeXmlLength += rootSelfClosedTagPos;
string messageXmlString = data.Substring(0, completeXmlLength + 1);
data = data.Substring(messageXmlString.Length);
try
{
// parse complete xml.
XDocument xmlDocument = XDocument.Parse(messageXmlString);
}
catch(Exception)
{
// Invalid Xml.
}
continue;
}
}
if (rootCloseTagPos > 0)
{
Int32 rootEndTagStartPos = root.IndexOf("</DATA", StringComparison.Ordinal);
if (rootEndTagStartPos > 0)
{
var endTagString = root.Substring(rootEndTagStartPos);
completeXmlLength += rootEndTagStartPos;
Int32 completeEndPos = endTagString.IndexOf(">", StringComparison.Ordinal);
if (completeEndPos > 0)
{
completeXmlLength += completeEndPos;
string messageXmlString = data.Substring(0, completeXmlLength + 1);
data = data.Substring(messageXmlString.Length);
try
{
// parse complete xml.
XDocument xmlDocument = XDocument.Parse(messageXmlString);
}
catch(Exception)
{
// Invalid Xml.
}
}
}
}
}
}
}
sw.Stop();
string timeElapsed = sw.Elapsed.ToString();
}
}
catch (IOException)
{
data = String.Empty;
}
catch (SocketException)
{
Console.WriteLine("Conection is broken!");
break;
}
}
This code I had use if ther were some kind of message framing, in this case 4 bytes of message length:
if (_clientSocket != null)
{
NetworkStream networkStream = _clientSocket.GetStream();
_clientSocket.ReceiveTimeout = 100;
string data = string.Empty;
while (_continueProcess)
{
try
{
if (networkStream.DataAvailable)
{
Stopwatch sw = new Stopwatch();
sw.Start();
var lengthBytes = new byte[sizeof (Int32)];
int bytesReaded = networkStream.Read(lengthBytes, 0, sizeof (Int32) - offset);
if (bytesReaded > 0)
{
offset += bytesReaded;
message.AddRange(lengthBytes.Take(bytesReaded));
}
if (offset < sizeof (Int32))
{
continue;
}
Int32 length = BitConverter.ToInt32(message.Take(sizeof(Int32)).ToArray(), 0);
message.Clear();
while (length > 0)
{
Int32 bytesToRead = length < _clientSocket.ReceiveBufferSize ? length : _clientSocket.ReceiveBufferSize;
byte[] messageBytes = new byte[bytesToRead];
bytesReaded = networkStream.Read(messageBytes, 0, bytesToRead);
length = length - bytesReaded;
message.AddRange(messageBytes);
}
try
{
string xml = Encoding.UTF8.GetString(message.ToArray());
XDocument xDocument = XDocument.Parse(xml);
}
catch (Exception ex)
{
// Invalid Xml.
}
sw.Stop();
string timeElapsed = sw.Elapsed.ToString();
}
}
catch (IOException)
{
data = String.Empty;
}
catch (SocketException)
{
Console.WriteLine("Conection is broken!");
break;
}
}
Like you can see I wanted to measure the elapsed time, to see witch methode has a better performance. The strange thing is that the methode whith no message framing has an average time of 0,2290 ms, the other methode has an average time of 1,2253 ms.
Can someone explain me why? I thought the one without message framing would be slower...

Hand the NetworkStream to the .NET XML infrastructure. For example create an XmlReader from the NetworkStream.
Unfortunately I did not find a built-in way to easily create an XmlDocument from an XmlReader that has multiple documents in it. It complains about multiple root elements (which is correct). You would need to wrap the XmlReader and make it stop returning nodes when the first document is done. You can do that by keeping track of some state and by looking at the nesting level. When the nesting level is zero again the first document is done.
This is just a raw sketch. I'm pretty sure this will work and it handles all possible XML documents.
No need for this horrible string processing code that you have there. The existing code looks quite slow as well but since this approach is much better it serves no purpose to comment on the perf issues. You need to throw this away.

I had the same problem - 3rd party system sends messages in XML format via TCP but my TCP client application may receive message partially or several messages at once. One of my colleagues proposed very simple and quite generic solution.
The idea is to have a string buffer which should be populated char by char from TCP stream, after each char try to parse buffer content with regular .Net XML parser. If parser throws an exception - continue adding chars to the buffer. Otherwise - message is ready and can be processed by application.
Here is the code:
private object _dataReceiverLock = new object();
private string _messageBuffer;
private Stopwatch _timeSinceLastMessage = new Stopwatch();
private List<string> NormalizeMessage(string rawMsg)
{
lock (_dataReceiverLock)
{
List<string> result = new List<string>();
//following code prevents buffer to store too old information
if (_timeSinceLastMessage.ElapsedMilliseconds > _settings.ResponseTimeout)
{
_messageBuffer = string.Empty;
}
_timeSinceLastMessage.Restart();
foreach (var ch in rawMsg)
{
_messageBuffer += ch;
if (ch == '>')//to avoid extra checks
{
if (IsValidXml(_messageBuffer))
{
result.Add(_messageBuffer);
_messageBuffer = string.Empty;
}
}
}
return result;
}
}
private bool IsValidXml(string xml)
{
try
{
//fastest way to validate XML format correctness
using (XmlTextReader reader = new XmlTextReader(new StringReader(xml)))
{
while (reader.Read()) { }
}
return true;
}
catch
{
return false;
}
}
Few comments:
Need to control lifetime of string buffer, otherwise in case network disconnection old information may stay in the buffer forever
There major problem here is the performance - parsing after every new character is quite slow. So need to add some optimizations, such as parse only after '>' character.
Make sure this method is thread safe, otherwise several threads may flood string buffer with different XML pieces.
The usage is simple:
private void _tcpClient_DataReceived(byte[] data)
{
var rawMsg = Encoding.Unicode.GetString(data);
var normalizedMessages = NormalizeMessage(rawMsg);
foreach (var normalizedMessage in normalizedMessages)
{
//TODO: your logic
}
}

how to parse emails faster than OpenPop.dll

it is possible using OpenPop.dll.
Pop3Client objPOP3Client = new Pop3Client();
int intTotalEmail = 0;
DataTable dtEmail = new DataTable();
object[] objMessageParts;
try
{
dtEmail = GetAllEmailStructure();
if (objPOP3Client.Connected)
objPOP3Client.Disconnect();
objPOP3Client.Connect(strHostName, intPort, bulUseSSL);
try
{
objPOP3Client.Authenticate(strUserName, new Common()._Decode(strPassword));
intTotalEmail = objPOP3Client.GetMessageCount();
AddMapping();
for (int i = 1; i <= intTotalEmail; i++)
{
objMessageParts = GetMessageContent(i, ref objPOP3Client, dtExistMailList);
if (objMessageParts != null && objMessageParts[0].ToString() == "0")
{
AddToDtEmail(objMessageParts, i, dtEmail, dtUserList, dtTicketIDList, dtBlacklistEmails, dtBlacklistSubject, dtBlacklistDomains);
}
}
}
catch (Exception ex)
{
}
}
catch (Exception ex)
{
ParserLogError(ex, "GetAllEmail()");
}
finally
{
if (objPOP3Client.Connected)
objPOP3Client.Disconnect();
}
// function
public object[] GetMessageContent(int intMessageNumber, ref Pop3Client objPOP3Client, DataTable dtExistingMails)
{
object[] strArrMessage = new object[10];
Message objMessage;
MessagePart plainTextPart = null, HTMLTextPart = null;
string strMessageId = "";
try
{
strArrMessage[0] = "";
strArrMessage[1] = "";
strArrMessage[2] = "";
strArrMessage[3] = "";
strArrMessage[4] = "";
strArrMessage[5] = "";
strArrMessage[6] = "";
strArrMessage[7] = null;
strArrMessage[8] = null;
strArrMessage[7] = "";
strArrMessage[8] = "";
objMessage = objPOP3Client.GetMessage(intMessageNumber);
strMessageId = (objMessage.Headers.MessageId == null ? "" : objMessage.Headers.MessageId.Trim());
if (!IsExistMessageID(dtExistingMails, strMessageId)) //check in data base message id is exists or not
{
strArrMessage[0] = "0";
strArrMessage[1] = objMessage.Headers.From.Address.Trim(); // From EMail Address
strArrMessage[2] = objMessage.Headers.From.DisplayName.Trim(); // From EMail Name
strArrMessage[3] = objMessage.Headers.Subject.Trim();// Mail Subject
plainTextPart = objMessage.FindFirstPlainTextVersion();
strArrMessage[4] = (plainTextPart == null ? "" : plainTextPart.GetBodyAsText().Trim());
HTMLTextPart = objMessage.FindFirstHtmlVersion();
strArrMessage[5] = (HTMLTextPart == null ? "" : HTMLTextPart.GetBodyAsText().Trim());
strArrMessage[6] = strMessageId;
List<MessagePart> attachment = objMessage.FindAllAttachments();
strArrMessage[7] = null;
strArrMessage[8] = null;
if (attachment.Count > 0)
{
if (attachment[0] != null && attachment[0].IsAttachment)
{
strArrMessage[7] = attachment[0].FileName.Trim();
strArrMessage[8] = attachment[0];
}
}
}
else
{
strArrMessage[0] = "1";
}
}
catch (Exception ex)
{
ParserLogError(ex, "GetMessageContent()");
}
return strArrMessage;
}
but, i want to make it faster than above OpenPop.dll. so please let me know if any other technique are there for parsing mails.
please check code and then tell me.
Thanks in advance

but, i want to make it faster than above OpenPop.dll. so please let me
know if any other technique are there for parsing mails.
In your GetMessageContent() method, the 1 place that consumes the vast amount of time is:
objMessage = objPOP3Client.GetMessage(intMessageNumber);
The network I/O part of downloading a message cannot really be optimized, but OpenPop.NET's parser is slow (based on my own performance tests).
MimeKit is 25x faster than OpenPop.NET at parsing email messages.
One of the main performance problems in OpenPop.NET's MIME parser is the fact that it uses a StreamReader for parsing (which is slow due to unnecessary charset conversion, reading 1 line at a time, etc - I have an analysis of another email library that uses StreamReader for parsing here: https://stackoverflow.com/a/18787176/87117).
Then there's the problem that OpenPop.NET's parser also uses Regex to remove CFWS (Comments and Folding White Space) from a header string before parsing/decoding it. This is expensive. It's far better to write a good tokenizer that can deal with CFWS.
If you are interested in some of the other techniques I used to optimize MimeKit to be so fast (as fast or faster than highly optimized C implementations), I wrote some blog posts about this:
Optimization Tricks used by MimeKit: Part 1
The summary of the optimization I talk about in part 1 is replacing loops like this that scan for the end of a line:
while (*inptr != (byte) '\n')
inptr++;
with a faster loop, like this:
int* dword = (int*) inptr;
do {
mask = *dword++ ^ 0x0A0A0A0A;
mask = ((mask - 0x01010101) & (~mask & 0x80808080));
} while (mask == 0);
inptr = (byte*) (dword - 1);
while (*inptr != (byte) '\n')
inptr++;
which improved performance by 20% (although on non-x86 architectures, it requires 'dword' to be 4-byte aligned).
Optimization Tricks used by MimeKit: Part 2
In part 2, I talk about writing a more optimized version of System.IO.MemoryStream. The problem with MemoryStream is that it has to keep 1 contiguous block of memory with the content, which means that as you write more data to it and it has to resize its internal byte array, it has to copy the content to the new array (which is expensive, especially once the amount of data in the stream is large).
To work around this performance bottleneck, I wrote a MemoryBlockStream which does not need to use a contiguous block of memory - it uses a linked list of byte arrays. Instead of having to resize the byte array when you overflow the current buffer, it simply allocates another 2048-byte array that the data will overflow into and appends it to the linked list.
Note: MimeKit itself only does email parsing, it doesn't do POP3 or SMTP or IMAP. If you want that kind of functionality, I've also written a library built on MimeKit that does that as well: MailKit
Update:
Sample code using MailKit (as requested) to download/parse all messages:
using System;
using System.Net;
using MailKit.Net.Pop3;
using MailKit;
using MimeKit;
namespace TestClient {
class Program
{
public static void Main (string[] args)
{
using (var client = new Pop3Client ()) {
client.Connect ("pop.gmail.com", 995, true);
// Note: since we don't have an OAuth2 token, disable
// the XOAUTH2 authentication mechanism.
client.AuthenticationMechanisms.Remove ("XOAUTH2");
client.Authenticate ("joey#gmail.com", "password");
int count = client.GetMessageCount ();
for (int i = 0; i < count; i++) {
var message = client.GetMessage (i);
Console.WriteLine ("Subject: {0}", message.Subject);
}
client.Disconnect (true);
}
}
}
}

MsiGetSummaryInformation always returns invalid parameter error

I am trying to set the Template property in the Summary Information Stram but whatever I do, it fails. I can read the property from the handle but can't write it back.
I want to generate multilingual copies of the MSI which is built (candled and light) in English. I am able to replace all the respective translated data in all the tables; the only thing I can not change is the Template property above.
I have tried all the ways I can use to pass the new String value, but it always says invalid parameter.
Here's the function I am using to do the same (C#):
public Boolean ChangeTemplateSummaryProperty(String strLangID) {
IntPtr hSIHandle;
if (MsiError.Success == MsiInterop.MsiGetSummaryInformation(IntPtr.Zero, m_strMSIPath, 1, out hSIHandle))
{
VariantType vtType = VariantType.LPStr;
int iVal = 0;
FILETIME oFileTime;
oFileTime.HighDateTime = 0;
oFileTime.LowDateTime = 0;
int iValSz = 0;
MsiError err = MsiInterop.MsiSummaryInfoGetProperty(hSIHandle, (uint)(SummaryInformationStreamProperty.Template),
out vtType, out iVal, out oFileTime, String.Empty, ref iValSz);
String strValue = new String('l', ++iValSz);
if (err == MsiError.MoreData)
{
err = MsiInterop.MsiSummaryInfoGetProperty(hSIHandle, (uint)(SummaryInformationStreamProperty.Template),
out vtType, out iVal, out oFileTime, strValue, ref iValSz);
}
else
{
Logger.LogError("Failed to get SummaryInformationStreamProperty.Template... err = " + err);
}
//I get the correct value here. as ";1033\0"
Logger.LogInfo("SummaryInformationStreamProperty.Template: " + strValue);
char[] arrNV = new char[strLangID.Length+2];
arrNV[0] = ';';
for (int i = 1; i < strLangID.Length + 1; i++) {
arrNV[i] = strLangID[i-1];
}
arrNV[strLangID.Length+1] = '\0';
String strNewVal = new String(arrNV);
//tried this, but fails.
//string strNV = ";";
//string strNV2 = strNV.Insert(1, strLangID);
//strNV2 = strNV2.Insert(strLangID.Length + 1, "\0");
err = MsiInterop.MsiSummaryInfoSetProperty(hSIHandle, (uint)(SummaryInformationStreamProperty.Template),
vtType, iVal, oFileTime, strNewVal);
if (err != MsiError.NoError)
{
Logger.LogError("Failed to set SummaryInformationStreamProperty.Template... err = " + err);
MsiInterop.MsiSummaryInfoPersist(hSIHandle);
MsiInterop.MsiCloseHandle(hSIHandle);
return false;
}
MsiInterop.MsiSummaryInfoPersist(hSIHandle);
MsiInterop.MsiCloseHandle(hSIHandle);
}
else
{
Logger.LogError("Failed to MsiGetSummaryInformation...");
return false;
}
return true;
}

Get rid of the MsiInterop that you are using and use the interop found in WiX's DTF. The Microsoft.Deploymnet.WindowsInstaller namespace has a SummaryInformation Class that exposes a read/write string Template property. Way better object model and interop without worrying about all the P/Invoke details that your current interop makes you deal with.
I'm home now so here's a code snippet:
using Microsoft.Deployment.WindowsInstaller;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
using( var database = new Database(#"C:\orca.msi", DatabaseOpenMode.Direct ))
{
database.SummaryInfo.Template = "Intel;666";
}
}
}
}
Notice the use of the using() clause. The Database class implements the IDisposable interface and automatically handles ( pun intended ) cleaning up all those pesky unmanaged handles for you.

Database msidb = objInstaller.OpenDatabase(MSIFileNameWithPath,MsiOpenDatabaseMode.msiOpenDatabaseModeTransact);
SummaryInfo info = msidb.get_SummaryInformation(1);
info.set_Property(2, (object)("sample title"));
info.Persist();
msidb.Commit();
The best reference to answer this is in detail is
https://learn.microsoft.com/en-us/archive/blogs/mwade/sail-away-with-me-to-another-world