I am writing an application that is interested in the status information from certain network devices. One of the devices provides the status information through Http and uses a multipart message; The first time you query the information it sends down the whole status and from then on whenever the status of the device changes a new multipart message is sent down the stream with just the changes.
I am using C# and am interested in using HttpClient or equivalent to open the stream, read all the information currently in the stream and then monitor this stream for when there is new information so that I can update the status information of the device accordingly in the application.
In essence the code I have looks something like this
using (var handler = new HttpClientHandler { Credentials = new NetworkCredential(username, password) })
{
using (var client = new HttpClient(handler))
{
var task = client.GetStreamAsync(uri);
task.Wait();
var stream = task.Result;
while(true)
{
byte[] bytes = ReadBytesFromStream(stream);
DoSomethingWithBytes(bytes);
}
}
The code in real life runs in a thread however and would need terminated correctly when told to.
The issue I am having is that when there is nothing in the stream the Read call on stream.ReadByte() blocks. If I put a ReadTimeout on the stream then when the Read call fails(i.e. when no new information is ready) then the CanRead property is set to false and I have to restart the process however in doing so recieve all the original status information again instead of only the elements that have changed.
Is there something that can be done to keep the stream alive until I tell it to terminate while being able to unblock on the read if no information is available? The reason I need to do this is since the application is multithreaded I need to terminate this code safely and the read is stopping the application from closing down.
Instead of using HttpClient I used an HttpWebRequest and set KeepAlive to true and AllowReadStreamBuffering properties to true. This keeps the stream alive and allows you to read bytes as of when they become available.
By keeping a reference to the network stream returned from GetResponseStream we can call Dispose on the NetworkStream which interrupts any reads that are currently taking place otherwise the read can block for as long as it needs to i.e. until it recieves data which solves the thread lifetime issues.
The right way to deal with the "I/O operation blocks my thread" problem is to use asynchronous I/O. .NET networking components offer a number of options here, but in your case you seem to be reading from a stream and even using (incorrectly) the GetStreamAsync() method, so the code can be cleaned up to handle both correctly and cleanly.
For example:
async Task ExecuteUriAsync(string username, string password, Uri uri)
{
using (var handler = new HttpClientHandler { Credentials = new NetworkCredential(username, password) })
{
using (var client = new HttpClient(handler))
{
Stream stream = await client.GetStreamAsync(uri);
byte[] buffer = new byte[10240];
while(true)
{
int byteCount = await stream.ReadAsync(buffer, 0, buffer.Length);
if (byteCount == 0)
{
// end-of-stream...must be done with the connection
return;
}
DoSomethingWithBytes(bytes, byteCount);
}
}
}
}
Your post is vague on what your previous ReadBytesFromStream() method did, and what DoSomethingWithBytes() does, but presumably you can figure out how to integrate that logic in the above.
Related
Currently, I'm trying to send many raw requests over a single connection in the faster way. I achieve that using HTTPClient lib but due the need of send raw bytes, I'm using TCPClient.
Some of requisites of this application:
1 connection
Submit all of requets in a pipeline, , continuing to send without waiting for each read
Send raw requests
What I have of code:
Method to send all requests
private static async Task SendAllRequests(Dictionary<int, string> requestsDictionary)
{
var client = new TcpClient();
var ipAddress = Dns.GetHostEntry("localhost").AddressList[0];
await client.ConnectAsync("localhost", 44392);
SslStream sslStream = new SslStream(
client.GetStream(),
false,
null
);
await sslStream.AuthenticateAsClientAsync("localhost");
var tasks = new List<Task<KeyValuePair<int, SslStream>>>();
foreach (var k in requestsDictionary.Keys)
{
tasks.Add(SendStreamAsync(requestString, sslStream));
}
var requestsInfo = await Task.WhenAll(tasks);
}
Method to send and read bytes of a single request
byte[] buffer = new byte[2048];
int bytes;
byte[] request = Encoding.UTF8.GetBytes(requestString);
await sslStream.WriteAsync(request, 0, request.Length);
await sslStream.FlushAsync();
do
{
bytes = await sslStream.ReadAsync(buffer, 0, buffer.Length);
} while (bytes == 2048);
Currently behaviour:
If you want to send multiple requests, you literally just: send multiple requests. It isn't clear what the question is here, or what unexpected behaviour you're seeing. However! A few considerations:
while you don't need to await each response, you do need to await each write - you can't write concurrently to the same socket from two different contexts (which you do currently)
TCP is a stream; there is no inherent framing - all that is guaranteed is that the same bytes will arrive in the right order (or failure, eventually); so: when sending multiple messages, you need to add the framing (so you know where each message starts and ends); this could be, for example, by CR/LF terminators in a text-based protocol, or by a length-prefix in a binary protocol
when dealing with this kind of pipeline, you can't read as part of the write (as that would defy the point), so: you'll need a separate read loop that just reads replies, and correlates them back to requests if needed; this might mean a queue if replies are always ordered with regards to requests, or a dictionary if replies can be out of order, keyed by some correlation identifier
honestly, this kind of network code is non-trivial
I need to stream audio data from the microphone to a REST server.
I am working with a propriatery ASR engine and need to collect the data then stream it in real time in a single call to PostAsync
Looking online, I found articles on PushStreamContent but either I am not using it correctly I don't understand what I'm doing (or both).
I have a MemoryStream called stream_memory to which I write data constantly from the main thread and I want to read it, while data is streaming, and post it in real time in a single post. In the example below, I also use an event stream_data_event and an object lock to prevent multiple threads writing to the MemoryStream at the same time. I clear the memory stream every time I read from it as I don't need the data afterwards.
Here is a snip of my code that is running in a thread of its own:
http_client = new HttpClient();
http_client.http_client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("*/*"));
http_client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Language", "en-us");
http_client.DefaultRequestHeaders.TransferEncodingChunked = true;
HttpContent content = new System.Net.Http.PushStreamContent(async (stream, httpContent, transportContext) =>
{
while (stream_data_event.WaitOne())
{
lock (main_window.stream_memory_lock)
{
stream_data_event.Reset();
long write_position = main_window.stream_memory.Position;
main_window.stream_memory.Seek(0, SeekOrigin.Begin);
main_window.stream_memory.CopyTo(stream, (int)write_position);
main_window.stream_memory.Position = 0;
}
}
});
content.Headers.TryAddWithoutValidation("Content-Type", "audio/L16;rate=8000");
string request_uri = String.Format("/v1/speech:recognize");
HttpResponseMessage response = await http_client.PostAsync(request_uri, content);
string http_result = await response.Content.ReadAsStringAsync();
The call to PostAsync calls the code for the PushStreamContent as expected.
However, as long as I am in the while loop, nothing is sent to the server (checked on wireshark).
If I exit the loop manually in debugger and call close on the stream, PostAsync exists but nothing is sent to the server.
I need to have a way to continue streaming information while in the PostAsync and have the data go out as the audio arrives.
Any ideas?
It turns out that the reason nothing was sent to the server has to do with the way wireshark displays HTTP results.
My expectations were that once the request was sent I would see it immediately in wireshark. However, wireshark only shows the request once it is complete which means it is shown far after the request started streaming.
After I realized that, I could see by data being sent over to the server.
I'm using the following code to send a file over tcp.
If i send many times the same file consecutively to test if it is robust, i receive the first file correctly and the other messed up.
All messed up files have the same incorrect bytes and if i Sleep(a while) all files are transfered correctly. I noticed I must instantiate a new buffer while reading my file to get everything done right. But i don't get why.
I fear my solution to reinstantiate a buffer could be just hiding another major problem. Any suggestion?
using(var fileStream = new FileStream(file, FileMode.Open, FileAccess.Read))
{
using(var binaryReader = new BinaryReader(fileStream))
{
var _sendingBuffer = new byte[BUFFER_SIZE];
int length = (int)fileStream.Length;
int bytesRead = 0;
//Ensure we reached the end of the stream regardless of encoding
while (binaryReader.BaseStream.Position != binaryReader.BaseStream.Length)
{
bytesRead = binaryReader.Read( _sendingBuffer, 0, _sendingBuffer.Length);
_socket.BeginSend(_sendingBuffer, 0, bytesRead, SocketFlags.None, SendFileCallback, null);
//without this i received some messed up data
_sendingBuffer = new byte[BUFFER_SIZE];
}
}
}
BeginSend is an asynchronous operation. It will only be guaranteed to be started after you call it, it won't be finished immediatly. As long as the socket is sending the passed data, that data must not be mutated.
The end of the operation will be signaled through the AsyncCallback callback parameter.
Your problem is exactly that you mutated the transmit buffer while the transmit was still in progress. Creating a new array for each transmit call fixes this.
Other ways to fix the problem:
Use the blocking Socket.Send function which will block until the whole data was sent and the buffer can be reused. This will also make your error handling much easier, because the error will not show up through the AsyncCallback.
Make your complete program acting asynchronously, e.g. using C#5's async Task and async/await functionalities
Therefore:
First read one part of the file asynchronously.
When the async read finishes send it asynchronously through the socket
When this completes and there is more data to read go back to 1.
Basically, I've written a windows phone 7 client, which is supposed to receive a very long string from the server. However, there seem to be some limitations on the phone networking code, and am having trouble figuring out how to do this. The following is the code I am using:
public string Receive()
{
string response = "Operation Timeout";
StringBuilder content = new StringBuilder();
// We are receiving over an established socket connection
if (_socket != null)
{
// Create SocketAsyncEventArgs context object
SocketAsyncEventArgs socketEventArg = new SocketAsyncEventArgs();
socketEventArg.RemoteEndPoint = _socket.RemoteEndPoint;
// Setup the buffer to receive the data
socketEventArg.SetBuffer(new Byte[MAX_BUFFER_SIZE], 0, MAX_BUFFER_SIZE);
// Inline event handler for the Completed event.
// Note: This even handler was implemented inline in order to make this method self-contained.
socketEventArg.Completed += new EventHandler<SocketAsyncEventArgs>(delegate(object s, SocketAsyncEventArgs e)
{
if (e.SocketError == SocketError.Success)
{
response = Encoding.UTF8.GetString(e.Buffer, e.Offset, e.BytesTransferred);
response = response.Trim('\0');
}
else
{
response = e.SocketError.ToString();
}
_clientDone.Set();
});
// Sets the state of the event to nonsignaled, causing threads to block
_clientDone.Reset();
// Make an asynchronous Receive request over the socket
_socket.ReceiveAsync(socketEventArg);
// Block the UI thread for a maximum of TIMEOUT_MILLISECONDS milliseconds.
// If no response comes back within this time then proceed
_clientDone.WaitOne(TIMEOUT_MILLISECONDS);
}
else
{
response = "Socket is not initialized";
}
return response;
}
As it stands, this will only accept the first N bytes of the message and return that... any help on how to do this would be greatly appreciated! Thanks!
Firstly, I would try to redesign your code to be appropriately async - working round the fact that WP7 doesn't offer synchronous IO by building your own isn't nearly as clean as embracing the asynchrony.
Secondly, you're assuming that you only need a single ReceiveAsync call to get the whole data. Assuming your using TCP, that's a stream-based protocol. You'll need to either add some sort of delimiter or length prefix, or keep reading until the other side closes the connection if that's the way your protocol works.
(Any reason you're not doing this with HTTP and WebClient or WebRequest, by the way?)
Firstly, I would try to redesign your code to be appropriately async - working round the fact that WP7 doesn't offer synchronous IO by building your own isn't nearly as clean as embracing the asynchrony.
Secondly, you're assuming that you only need a single ReceiveAsync call to get the whole data. Assuming your using TCP, that's a stream-based protocol. You'll need to either add some sort of delimiter or length prefix, or keep reading until the other side closes the connection if that's the way your protocol works.
(Any reason you're not doing this with HTTP and WebClient or WebRequest, by the way?)
I am trying to implement a tcp client listening function. This means that after connection established with server, this tcp client just sit there and waiting for new data to arrive.
Here is my code but when it runs, it complain about not been able to read anything from the
network stream. But the server hasn't started sending data yet. I guess the complains is because
of the timeout in tcp client.
Is this the right way to do it?
public void listen(dataHandler processDataFuc)
{
NetworkStream stream;
Byte[] data_buffer = new Byte[MAX_PACKET_SIZE];
if(!this.Connected)
{
this.Connect();
}
while (!this.terminate_listening)
{
stream = main_client.GetStream();
while (stream.Read(data_buffer, 0, data_buffer.Length) > 0)
{
processDataFuc(data_buffer);
}
}
}
Thanks
The short answer is yes, it'll do what you want, but it's not ideal. I'd first suggest moving stream = main_client.GetStream(); out of the while loop, as you're just getting the same stream over and over. Also, using NetworkStream.Read isn't the best way to perform a continuous read if you're expecting intermittent data over a long period of time, as it's holding up a thread just for that one task; better to use BeginRead and pass in a callback, which will return immediately but later alert you when data is available (via the callback).
Checkout the methods EstablishConnection() and IncomingDataSyncWorker() (synchronous) or IncomingPacketHandler() (asynchronous) for examples of what you want to do. All these methods are part of networkComms.net an opensource network communication library.
What version of .Net are you using?
If you are using .Net 4.0 or 4.5, then you can use ReadAsync() instead of Read().
Consider this:
public async void listen(dataHandler processDataFuc)
{
NetworkStream stream;
Byte[] data_buffer = new Byte[MAX_PACKET_SIZE];
if(!this.Connected)
this.Connect();
stream = main_client.GetStream();
while (!this.terminate_listening)
{
await stream.ReadAsync(data_buffer, 0, data_buffer.Length)
processDataFuc(data_buffer);
}
}
In such way, ReadAsync() will waits in a separate Thread until server sends some Data. Then the rest of your code will execute.