Develop Reference

Changes to Buffer While BeginWrite is Called

I'm curious as to whether making changes to the byte[] before BeginWrite actually finishes writing, will influence what is finally written by the FileStream.
I've this code below, with currentPage being a byte[] with the data I want to write.
try
{
FileStream.BeginWrite(currentPage, 0, currentPage.Length, new AsyncCallback(EndWriteCallback),new State(logFile.fs, currentPage, BUFFER_SIZE, manualEvent));
manualEvent.WaitOne();
}
catch (Exception e)
{
//handle exception here
}
I have this within a loop that will replace the data in currentPage. What will happen if I make changes to currentPage (like assign a new byte[] with all 0's in it)? Does FileStream buffer the byte[] to be written somewhere or does it actually just references the byte[] I passed in when I call it?
I tried looking at the MSDN article but all I could find was
Multiple simultaneous asynchronous requests render the request completion order uncertain.
Could someone please explain this to me?

This code should answer your questions. Firstly I create a long byte array where every cell is equal to 255. Then I start 2 threads. The first one is responsible for writing the prepared byte array to file. At the same time the second thread modifies this array, starting from the last cell, by setting every cell to 0.
The exact results of executing this code will depend on the machine, current CPU usage etc. On my computer one time I observed that about 77% of the created file contained 255s and the rest 0s. The next time it was about 70%. It confirms that the input array is not blocked for writing by BeginWrite method.
In order to observe this effect try to run this program a few times. It might be also necessary to use the longer array.
var path = #"C:\Temp\temp.txt";
var list = new List<byte>();
for(var i = 0; i < 1000000; ++i)
list.Add(255);
var buffer = list.ToArray();
var t1 = Task.Factory.StartNew(() =>
{
using (var fs = File.OpenWrite(path))
{
var res = fs.BeginWrite(buffer, 0, buffer.Length, null, null);
res.AsyncWaitHandle.WaitOne();
}
});
var t2 = Task.Factory.StartNew(() =>
{
for (var i = buffer.Length - 1; i > 0; --i)
buffer[i] = 0;
});
Task.WaitAll(t1, t2);

How do I achieve better granularity when performing streaming operations?

Okay so I'm working on my file transfer service, and I can transfer the files fine with WCF streaming. I get good speeds, and I'll eventually be able to have good resume support because I chunk my files into small bits before streaming.
However, I'm running into issues with both the server side transfer and the client side receiving when it comes to measuring a detailed transfer speed as the messages are streamed and written.
Here's the code where the file is chunked, which is called by the service every time it needs to send another chunk to the client.
public byte[] NextChunk()
{
if (MoreChunks) // If there are more chunks, procede with the next chunking operation, otherwise throw an exception.
{
byte[] buffer;
using (BinaryReader reader = new BinaryReader(File.OpenRead(FilePath)))
{
reader.BaseStream.Position = currentPosition;
buffer = reader.ReadBytes((int)MaximumChunkSize);
}
currentPosition += buffer.LongLength; // Sets the stream position to be used for the next call.
return buffer;
}
else
throw new InvalidOperationException("The last chunk of the file has already been returned.");
In the above, I basically write to the buffer based on the chunk size I am using(in this case it's 2mb which I found to have the best transfer speeds compared to larger or smaller chunk sizes). I then do a little work to remember where I left off, and return the buffer.
The following code is the server side work.
public FileMessage ReceiveFile()
{
if (!transferSpeedTimer.Enabled)
transferSpeedTimer.Start();
byte[] buffer = chunkedFile.NextChunk();
FileMessage message = new FileMessage();
message.FileMetaData = new FileMetaData(chunkedFile.MoreChunks, buffer.LongLength);
message.ChunkData = new MemoryStream(buffer);
if (!chunkedFile.MoreChunks)
{
OnTransferComplete(this, EventArgs.Empty);
Timer timer = new Timer(20000f);
timer.Elapsed += (sender, e) =>
{
StopSession();
timer.Stop();
};
timer.Start();
}
//This needs to be more granular. This method is called too infrequently for a fast and accurate enough progress of the file transfer to be determined.
TotalBytesTransferred += buffer.LongLength;
return message;
}
In this method, which is called by the client in a WCF call, I get information for the next chunk, create my message, do a little bit with timers to stop my session once the transfer is complete and update the transfer speeds. Shortly before I return the message I increment my TotalBytesTransferred with the length of the buffer, which is used to help me calculate transfer speed.
The problem with this, is it takes a while to stream the file to the client, and so the speeds I'm getting are false. What I'm trying to aim for here is a more granular modification to the TotalBytesTransferred variable so I have a better representation of how much data is being sent to the client at any given time.
Now, for the client side code, which uses an entirely different way of calculating transfer speed.
if (Role == FileTransferItem.FileTransferRole.Receiver)
{
hostChannel = channelFactory.CreateChannel();
((IContextChannel)hostChannel).OperationTimeout = new TimeSpan(3, 0, 0);
bool moreChunks = true;
long bytesPreviousPosition = 0;
using (BinaryWriter writer = new BinaryWriter(File.OpenWrite(fileWritePath)))
{
writer.BaseStream.SetLength(0);
transferSpeedTimer.Elapsed += ((sender, e) =>
{
transferSpeed = writer.BaseStream.Position - bytesPreviousPosition;
bytesPreviousPosition = writer.BaseStream.Position;
});
transferSpeedTimer.Start();
while (moreChunks)
{
FileMessage message = hostChannel.ReceiveFile();
moreChunks = message.FileMetaData.MoreChunks;
writer.BaseStream.Position = filePosition;
// This is golden, but I need to extrapolate it out and do the stream copy myself so I can do calculations on a per byte basis.
message.ChunkData.CopyTo(writer.BaseStream);
filePosition += message.FileMetaData.ChunkLength;
// TODO This needs to be more granular
TotalBytesTransferred += message.FileMetaData.ChunkLength;
}
OnTransferComplete(this, EventArgs.Empty);
}
}
else
{
transferSpeedTimer.Elapsed += ((sender, e) =>
{
totalElapsedSeconds += (int)transferSpeedTimer.Interval;
transferSpeed = TotalBytesTransferred / totalElapsedSeconds;
});
transferSpeedTimer.Start();
host.Open();
}
Here, my TotalBytesTransferred is also based on the length of the chunk coming in. I know I can get a more granular calculation if I do the stream writing myself instead of using the CopyTo for the stream, but I'm not exactly sure how to best go about this.
Can anybody help me out here? Outside of this class I have another class polling the property of TransferSpeed as it's updated internally.
I apologize if I posted too much code, but I wasn't sure what to post and what not.
EDIT: I realize at least with the Server side implementation, the way I can get a more granular reading on how many bytes have been transferred, is by reading the position of the return message value of the stream. However, I don't know a way to do this to ensure absolute integrity on my count. I thought about maybe using a timer and polling the position as the stream was being transferred, but then the next call might be made and I would quickly become out of sync.
How can I poll data from the returning stream and know immediately when the stream finishes so I can quickly add up the remainder of what was left of the stream into my byte count?

Okay I have found what seems to be ideal for me. I don't know if it's perfect, but it's pretty darn good for my needs.
On the Server side, we have this code that does the work of transferring the file. The chunkedFile class obviously does the chunking, but this is the code that sends the information to the Client.
public FileMessage ReceiveFile()
{
byte[] buffer = chunkedFile.NextChunk();
FileMessage message = new FileMessage();
message.FileMetaData = new FileMetaData(chunkedFile.MoreChunks, buffer.LongLength, chunkedFile.CurrentPosition);
message.ChunkData = new MemoryStream(buffer);
TotalBytesTransferred = chunkedFile.CurrentPosition;
UpdateTotalBytesTransferred(message);
if (!chunkedFile.MoreChunks)
{
OnTransferComplete(this, EventArgs.Empty);
Timer timer = new Timer(20000f);
timer.Elapsed += (sender, e) =>
{
StopSession();
timer.Stop();
};
timer.Start();
}
return message;
}
The client basically calls this code, and the server proceeds to get a new chunk, put it in a stream, update the TotalBytesTransferred based on the position of the chunkedFile(which keeps track of the underlying file system file that is used to draw the data from). I'll show the method UpdateTotalBytesTransferred(message) in a moment, as that is where all the code for the server and client reside to achieve the more granular polling of the TotalBytesTransferred.
Next up is the client side work.
hostChannel = channelFactory.CreateChannel();
((IContextChannel)hostChannel).OperationTimeout = new TimeSpan(3, 0, 0);
bool moreChunks = true;
using (BinaryWriter writer = new BinaryWriter(File.OpenWrite(fileWritePath)))
{
writer.BaseStream.SetLength(0);
while (moreChunks)
{
FileMessage message = hostChannel.ReceiveFile();
moreChunks = message.FileMetaData.MoreChunks;
UpdateTotalBytesTransferred(message);
writer.BaseStream.Position = filePosition;
message.ChunkData.CopyTo(writer.BaseStream);
TotalBytesTransferred = message.FileMetaData.FilePosition;
filePosition += message.FileMetaData.ChunkLength;
}
OnTransferComplete(this, EventArgs.Empty);
}
This code is very simple. It calls the host to get the file stream, and also utilizes the UpdateTotalBytesTransferred(message) method. It does a little bit of work to remember the position of the underlying file that is being written, and copies the stream to that file while also updating the TotalBytesTransferred after finishing.
The way I achieved the granularity I was looking for was with the UpdateTotalBytesTransferred method as follows. It works exactly the same for both the Server and the Client.
private void UpdateTotalBytesTransferred(FileMessage message)
{
long previousStreamPosition = 0;
long totalBytesTransferredShouldBe = TotalBytesTransferred + message.FileMetaData.ChunkLength;
Timer timer = new Timer(500f);
timer.Elapsed += (sender, e) =>
{
if (TotalBytesTransferred + (message.ChunkData.Position - previousStreamPosition) < totalBytesTransferredShouldBe)
{
TotalBytesTransferred += message.ChunkData.Position - previousStreamPosition;
previousStreamPosition = message.ChunkData.Position;
}
else
{
timer.Stop();
timer.Dispose();
}
};
timer.Start();
}
What this does is take in the FileMessage which is basically just a stream and some information about the file itself. It has a variable previousStreamPosition to remember the last position it was when it was polling the underlying stream. It also does a simple calculation with totalBytesTransferredShouldBe based on how many bytes are already transferred plus the total length of the stream.
Finally, a timer is created and executed, which upon every tick checks to see if it needs to be incrementing the TotalBytesTransferred. If it's not supposed to update it anymore(reached the end of the stream basically), it stops and disposes of the timer.
This all allows me to get very small reads of how many bytes have been transferred, which lets me better calculate the total progress in a more fluid way, as more accurately measure the file transfer speeds achieved.

Consuming a HTTP stream without reading one byte at a time

I have been trying to read data from the Twitter stream API using C#, and since sometimes the API will return no data, and I am looking for a near-realtime response, I have been hesitant to use a buffer length of more than 1 byte on the reader in case the stream doesn't return any more data for the next day or two.
I have been using the following line:
input.BeginRead(buffer, 0, buffer.Length, InputReadComplete, null);
//buffer = new byte[1]
Now that I plan to scale the application up, I think a size of 1 will result in a lot of CPU usage, and want to increase that number, but I still don't want the stream to just block. Is it possible to get the stream to return if no more bytes are read in the next 5 seconds or something similar?

Async Option
You can use a timer in the async callback method to complete the operation if no bytes are received for e.g. 5 seconds. Reset the timer every time bytes are received. Start it before BeginRead.
Sync Option
Alternatively, you can use the ReceiveTimeout property of the underlying socket to establish a maximum time to wait before completing the read. You can use a larger buffer and set the timeout to e.g. 5 seconds.
From the MSDN documentation that property only applies to a synchronous read. You could perform a synchronous read on a separate thread.
UPDATE
Here's rough, untested code pieced together from a similar problem. It will probably not run (or be bug-free) as-is, but should give you the idea:
private EventWaitHandle asyncWait = new ManualResetEvent(false);
private Timer abortTimer = null;
private bool success = false;
public void ReadFromTwitter()
{
abortTimer = new Timer(AbortTwitter, null, 50000, System.Threading.Timeout.Infinite);
asyncWait.Reset();
input.BeginRead(buffer, 0, buffer.Length, InputReadComplete, null);
asyncWait.WaitOne();
}
void AbortTwitter(object state)
{
success = false; // Redundant but explicit for clarity
asyncWait.Set();
}
void InputReadComplete()
{
// Disable the timer:
abortTimer.Change(System.Threading.Timeout.Infinite, System.Threading.Timeout.Infinite);
success = true;
asyncWait.Set();
}

When NetworkStream.Read(byte[], int, int) needs to return -1, it abort()s the thread it is running on instead?

NetworkStream stream = socket.GetStream();
if (stream.CanRead)
{
while (true)
{
int i = stream.Read(buf, 0, 1024);
result += Encoding.ASCII.GetString(buf, 0, i);
}
}
Above code was designed to retrieve message from a TcpClient while running on a separate thread. Read Method works fine until it is supposed to return -1 to indicate there is nothing to read anymore; instead, it just terminates the thread it is running on without any apparent reason - tracing each step using the debugger shows that it just stops running right after that line.
Also I tried encapsulating it with a try ... catch without much success.
What could be causing this?
EDIT: I tried
NetworkStream stream = socket.GetStream();
if (stream.CanRead)
{
while (true)
{
int i = stream.Read(buf, 0, 1024);
if (i == 0)
{
break;
}
result += Encoding.ASCII.GetString(buf, 0, i);
}
}
thanks to #JonSkeet, but the problem is still there. The thread terminates at that read line.
EDIT2: I fixed the code like this and it worked.
while (stream.DataAvailable)
{
int i = stream.Read(buf, 0, 1024);
result += Encoding.ASCII.GetString(buf, 0, i);
}
I think the problem was simple, I just didn't think thoroughly enough. Thanks everyone for taking a look at this!

No, Stream.Read returns 0 when there's nothing to read, not -1:
Return value
The total number of bytes read into the buffer. This can be less than the number of bytes requested if that many bytes are not currently available, or zero (0) if the end of the stream has been reached.
My guess is that actually, no exception is being thrown and the thread isn't being aborted - but it's just looping forever. You should be able to see this if you step through in the debugger. Whatever's happening, your "happy" termination condition will never be hit...

Since you're trying to read ASCII characters, from a stream, take a look at the following as a potentially simpler way to do it:
public IEnumerable<string> ReadLines(Stream stream)
{
using (StreamReader reader = new StreamReader(stream, Encoding.ASCII))
{
while (!reader.EndOfStream)
yield return reader.ReadLine();
}
}
While this may not be exactly what you want, the salient points are:
Use a StreamReader to do all the hard work for you
Use a while loop with !reader.EndOfStream to loop through the stream
You can still use reader.Read(buffer, 0, 1024) if you'd prefer to read chunks into a buffer, and append to result. Just note that these will be char[] chunks not byte[] chunks, which is likely what you want.

It looks to me like it is simply blocking - i.e. waiting on the end of the stream. For it to return a non-positive number, it is necessary that the stream be closed, i.e. the the caller has not only sent data, but has closed their outbound socket. Otherwise, the system cannot distinguish between "waiting for a packet to arrive" and "the end of the stream".
If the caller is sending one message only, they should close their outbound socket after sending (they can keep their inbound socket open for a reply).
If the caller is sending multiple messages, then you must use a framing approach to read individual sub-messages. In the case of a text-based protocol this usually means "hunt the newline".

How can I improve this underperforming, terrible Serial Port code?

I have an ugly piece of Serial Port code which is very unstable.
void port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
Thread.Sleep(100);
while (port.BytesToRead > 0)
{
var count = port.BytesToRead;
byte[] buffer = new byte[count];
var read = port.Read(buffer, 0, count);
if (DataEncapsulator != null)
buffer = DataEncapsulator.UnWrap(buffer);
var response = dataCollector.Collect(buffer);
if (response != null)
{
this.OnDataReceived(response);
}
Thread.Sleep(100);
}
}
If I remove either Thread.Sleep(100) calls the code stops working.
Of course this really slows things down and if lots of data streams in,
it stops working as well unless I make the sleep even bigger.
(Stops working as in pure deadlock)
Please note the DataEncapsulator and DataCollector are components
provided by MEF, but their performance is quite good.
The class has a Listen() method which starts a background worker to
receive data.
public void Listen(IDataCollector dataCollector)
{
this.dataCollector = dataCollector;
BackgroundWorker worker = new BackgroundWorker();
worker.DoWork += new DoWorkEventHandler(worker_DoWork);
worker.RunWorkerAsync();
}
void worker_DoWork(object sender, DoWorkEventArgs e)
{
port = new SerialPort();
//Event handlers
port.ReceivedBytesThreshold = 15;
port.DataReceived += new SerialDataReceivedEventHandler(port_DataReceived);
..... remainder of code ...
Suggestions are welcome!
Update:
*Just a quick note about what the IDataCollector classes do.
There is no way to know if all bytes of the data that has been sent
are read in a single read operation. So everytime data is read it is
passed to the DataColllector which returns true when a complete and
valid protocol message has been received. In this case here it just
checks for a sync byte, length , crc and tail byte. The real work
is done later by other classes.
*
Update 2:
I replaced the code now as suggested, but still there is something wrong:
void port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
var count = port.BytesToRead;
byte[] buffer = new byte[count];
var read = port.Read(buffer, 0, count);
if (DataEncapsulator != null)
buffer = DataEncapsulator.UnWrap(buffer);
var response = dataCollector.Collect(buffer);
if (response != null)
{
this.OnDataReceived(response);
}
}
You see this works fine with a fast and stable connection.
But OnDataReceived is NOT called every time data is received.
(See the MSDN docs for more). So if the data gets fragmented
and you only read once within the event data gets lost.
And now I remember why I had the loop in the first place, because
it actually does have to read multiple times if the connection is slow or unstable.
Obviously I can't go back to the while loop solution, so what can I do?

My first concern with the original while-based code fragment is the constant allocation of memory for the byte buffer. Putting a "new" statement here specifically going to the .NET memory manager to allocate memory for the buffer, while taking the memory allocated in the last iteration and sending it back into the unused pool for eventual garbage collection. That seems like an awful lot of work to do in a relatively tight loop.
I am curious as to the performance improvement you would gain by creating this buffer at design-time with a reasonable size, say 8K, so you don't have all of this memory allocation and deallocation and fragmentation. Would that help?
private byte[] buffer = new byte[8192];
void port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
Thread.Sleep(100);
while (port.BytesToRead > 0)
{
var count = port.BytesToRead;
var read = port.Read(buffer, 0, count);
// ... more code
}
}
My other concern with re-allocating this buffer on every iteration of the loop is that the reallocation may be unnecessary if the buffer is already large enough. Consider the following:
Loop Iteration 1: 100 bytes received; allocate buffer of 100 bytes
Loop Iteration 2: 75 bytes received; allocate buffer of 75 bytes
In this scenario, you don't really need to re-allocate the buffer, because the buffer of 100 bytes allocated in Loop Iteration 1 is more than enough to handle the 75 bytes received in Loop Iteration 2. There is no need to destroy the 100 byte buffer and create a 75 byte buffer. (This is moot, of course, if you just statically create the buffer and move it out of the loop altogether.)
On another tangent, I might suggest that the DataReceived loop concern itself only with the reception of the data. I am not sure what those MEF components are doing, but I question if their work has to be done in the data reception loop. Is it possible for the received data to be put on some sort of queue and the MEF components can pick them up there? I am interested in keeping the DataReceived loop as speedy as possible. Perhaps the received data can be put on a queue so that it can go right back to work receiving more data. You can set up another thread, perhaps, to watch for data arriving on the queue and have the MEF components pick up the data from there and do their work from there. That may be more coding, but it may help the data reception loop be as responsive as possible.

And it can be so simple...
Either you use DataReceived handler but without a loop and certainly without Sleep(), read what data is ready and push it somewhere (to a Queue or MemoryStream),
or
Start a Thread (BgWorker) and do a (blocking) serialPort1.Read(...), and again, push or assemble the data you get.
Edit:
From what you posted I would say: drop the eventhandler and just Read the bytes inside Dowork(). That has the benefit you can specify how much data you want, as long as it is (a lot) smaller than the ReadBufferSize.
Edit2, regarding Update2:
You will still be much better of with a while loop inside a BgWorker, not using the event at all. The simple way:
byte[] buffer = new byte[128]; // 128 = (average) size of a record
while(port.IsOpen && ! worker.CancelationPending)
{
int count = port.Read(buffer, 0, 128);
// proccess count bytes
}
Now maybe your records are variable-sized and you don't don't want to wait for the next 126 bytes to come in to complete one. You can tune this by reducing the buffer size or set a ReadTimeOut. To get very fine-grained you could use port.ReadByte(). Since that reads from the ReadBuffer it's not really any slower.

If you want to write the data to a file and the serial port stops every so often this is a simple way to do it. If possible make your buffer large enough to hold all the bytes that you plan to put in a single file. Then write the code in your datareceived event handler as shown below. Then when you get an oportunity write the whole buffer to a file as shown below that. If you must read FROM your buffer while the serial port is reading TO your buffer then try using a buffered stream object to avoid deadlocks and race conditions.
private byte[] buffer = new byte[8192];
var index = 0;
void port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
index += port.Read(buffer, index, port.BytesToRead);
}
void WriteDataToFile()
{
binaryWriter.Write(buffer, 0, index);
index = 0;
}