I'm running into a bit of problem when trying to readToEnd from an asynchronous webrequest. Here's the code:
public void GetHTTP(string http)
{
request = (HttpWebRequest)WebRequest.Create(http);
RequestState rs = new RequestState(); // Class to hold state. Can ignore...
Setup(); // contain statements such as request.Accept =...;
rs.Request = request;
IAsyncResult r = (IAsyncResult)request.BeginGetResponse(new AsyncCallback ResponseSetup), rs);
allDone.WaitOne();
Referer = http; //Can ignore this...
}
private void ResponseSetup(IAsyncResult ar)
{
RequestState rs = (RequestState)ar.AsyncState;
WebRequest request = rs.Request;
WebResponse response = request.EndGetResponse(ar);
Stream ResponseStream = response.GetResponseStream();
rs.ResponseStream = ResponseStream;
IAsyncResult iarRead = ResponseStream.BeginRead(rs.BufferRead, 0, BUFFER_SIZE, new AsyncCallback(ReadCallBack), rs);
StreamReader reader = new StreamReader(ResponseStream);
information = reader.ReadToEnd();
//The problem is right here; ReadToEnd.
}
When trying to invoke the readToEnd method, I get this error message: The stream does not support concurrent I/O read or write operations.
I've searched, but I could not find a solution to this problem. How can it be fixed?
That's because you're trying to do two reads. The call to BeginRead initiates a read operation. Then ReadToEnd initiates another read operation on the same stream.
I think what you want is just the ReadToEnd. Remove the call to ResponseStream.BeginRead.
Related
I'm trying to write a simple OWIN Middleware, in order to intercept the response stream. What I'm trying to do is replace the original stream with custom Stream-based class, where I will be able to intercept writes to the response stream.
However, I'm facing some issues because I cannot know when the response has been completely written to by inner middleware components in the chain. The Dispose override of the Stream is never called. So I don't know when it's time to perform my processing, which should happen at the end of the response Stream.
Here is a sample code:
public sealed class CustomMiddleware: OwinMiddleware
{
public CustomMiddleware(OwinMiddleware next)
: base(next)
{
}
public override async Task Invoke(IOwinContext context)
{
var request = context.Request;
var response = context.Response;
// capture response stream
var vr = new MemoryStream();
var responseStream = new ResponseStream(vr, response.Body);
response.OnSendingHeaders(state =>
{
var resp = (state as IOwinContext).Response;
var contentLength = resp.Headers.ContentLength;
// contentLength == null for Chunked responses
}, context);
// invoke the next middleware in the pipeline
await Next.Invoke(context);
}
}
public sealed class ResponseStream : Stream
{
private readonly Stream stream_; // MemoryStream
private readonly Stream output_; // Owin response
private long writtenBytes_ = 0L;
public ResponseStream(Stream stream, Stream output)
{
stream_ = stream;
output_ = output;
}
... // System.IO.Stream implementation
public override void Write(byte[] buffer, int offset, int count)
{
// capture writes to the response stream in our local stream
stream_.Write(buffer, offset, count);
// write to the real output stream
output_.Write(buffer, offset, count);
// update the number of bytes written
writtenBytes_ += count;
// how do we know the response is complete ?
// we could check that the number of bytes written
// is equal to the content length, but content length
// is not available for Chunked responses.
}
protected override void Dispose(bool disposing)
{
// we could perform our processing
// when the stream is disposed of.
// however, this method is never called by
// the OWIN/Katana infrastructure.
}
}
As I've alluded to in the comments from the code above, there are two strategies that I can think of in order to detect whether the response is complete.
a) I can record the number of bytes written to the response stream and correlate that to the expected response length. However, in the case of responses which use the Chunked Transfer Encoding, the length is not known.
b) I can decide that the response stream is complete when Dispose is called on the response stream. However, the OWIN/Katana infrastructure never calls Dispose on the replaced stream.
I have been investigating Opaque Streaming in order to see whether manipulating the underlying HTTP protocol would be a feasible approach, but I don't seem to find whether Katana supports Opaque Streaming or not.
Is there a way to achieve what I want ?
I do not think you will need a sub-classed stream but then here is how you can read the response. Just ensure this middleware is the first one in the OWIN pipeline so that it will be the last one to inspect the response.
using AppFunc = Func<IDictionary<string, object>, Task>;
public class CustomMiddleware
{
private readonly AppFunc next;
public CustomMiddleware(AppFunc next)
{
this.next = next;
}
public async Task Invoke(IDictionary<string, object> env)
{
IOwinContext context = new OwinContext(env);
// Buffer the response
var stream = context.Response.Body;
var buffer = new MemoryStream();
context.Response.Body = buffer;
await this.next(env);
buffer.Seek(0, SeekOrigin.Begin);
var reader = new StreamReader(buffer);
string responseBody = await reader.ReadToEndAsync();
// Now, you can access response body.
Debug.WriteLine(responseBody);
// You need to do this so that the response we buffered
// is flushed out to the client application.
buffer.Seek(0, SeekOrigin.Begin);
await buffer.CopyToAsync(stream);
}
}
BTW, as far as I know, deriving from OwinMiddleware is not considered a good practice because OwinMiddleware is specific to Katana. It is however nothing to do with your problem though.
Assume I have the following code:
private string PostData(string functionName, string parsedContent)
{
string url = // some url;
var http = (HttpWebRequest)WebRequest.Create(new Uri(url));
http.Accept = "application/json";
http.ContentType = "application/json";
http.Method = "POST";
http.Timeout = 15000; // 15 seconds
Byte[] bytes = Encoding.UTF8.GetBytes(parsedContent);
using (Stream newStream = http.GetRequestStream())
{
newStream.Write(bytes, 0, bytes.Length);
}
using (WebResponse response = http.GetResponse())
{
using (var stream = response.GetResponseStream())
{
var sr = new StreamReader(stream);
var content = sr.ReadToEnd();
return content;
}
}
}
I set up a breakpoint over this line of code:
using (Stream newStream = http.GetRequestStream())
before http.GetRequestStream() gets executed. Here is a screenshot of my active threads:
This whole method is running in background thread with ThreadId = 3 as you can see.
After pressing F10 we get http.GetRequestStream() method executed. And here is an updated screenshot of active threads:
As you can see, now we have one extra active thread that is in state of waiting. Probably the method http.GetRequestStream() spawns it. Everything is fine, but.. this thread keeps hanging like that for the whole app lifecycle, which seems not to be the intended behaviour.
Am I misusing GetRequestStream somehow?
If I use ilspy it looks like the request is send asynchronously. That would explain the extra thread.
Looking a little bit deeper the HttpWebRequest creates a static TimerQueue with one thread and a never ending loop, that has a Monitor.WaitAny in it. Every webrequest in the appdomain will register a timer callback for timeout handling and all those callbacks are handled by that thread. Due to it being static that instance will never get garbage collected and therefore it will keep hold of the thread.
It did register for the AppDomain.Unload event so if that fires it will clean up it's resources including any threads.
Do notice that these are all internal classes and those implementation details might change at any time.
I'm coding a multithreaded web-crawler that performs a lot of concurrent httpwebrequests every second using hundreds of threads, the application works great but sometimes(randomly) one of the webrequests hangs on the getResponseStream() completely ignoring the timeout(this happen when I perform hundreds of requests concurrently) making the crawling process never end, the strange thing is that with fiddler this never happen and the application never hang, it is really hard to debug because it happens randomly.
I've tried to set
Keep-Alive = false
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
but I still get the strange behavior, any ideas?
Thanks
HttpWebRequest code:
public static string RequestHttp(string url, string referer, ref CookieContainer cookieContainer_0, IWebProxy proxy)
{
string str = string.Empty;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
request.UserAgent = randomuseragent();
request.ContentType = "application/x-www-form-urlencoded";
request.Accept = "*/*";
request.CookieContainer = cookieContainer_0;
request.Proxy = proxy;
request.Timeout = 15000;
request.Referer = referer;
//request.ServicePoint.MaxIdleTime = 15000;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
List<byte> list = new List<byte>();
byte[] buffer = new byte[0x400];
int count = responseStream.Read(buffer, 0, buffer.Length);
while (count != 0)
{
list.AddRange(buffer.ToList<byte>().GetRange(0, count));
if (list.Count >= 0x100000)
{
break;
}
count = 0;
try
{
HERE IT HANGS SOMETIMES ---> count = responseStream.Read(buffer, 0, buffer.Length);
continue;
}
catch
{
continue;
}
}
//responseStream.Close();
int num2 = 0x200 * 0x400;
if (list.Count >= num2)
{
list.RemoveRange((num2 * 3) / 10, list.Count - num2);
}
byte[] bytes = list.ToArray();
str = Encoding.Default.GetString(bytes);
Encoding encoding = Encoding.Default;
if (str.ToLower().IndexOf("charset=") > 0)
{
encoding = GetEncoding(str);
}
else
{
try
{
encoding = Encoding.GetEncoding(response.CharacterSet);
}
catch
{
}
}
str = encoding.GetString(bytes);
// response.Close();
}
}
return str.Trim();
}
The Timeout property "Gets or sets the time-out value in milliseconds for the GetResponse and GetRequestStream methods." The default value is 100,000 milliseonds (100 seconds).
The ReadWriteTimeout property, "Gets or sets a time-out in milliseconds when writing to or reading from a stream." The default is 300,000 milliseconds (5 minutes).
You're setting Timeout, but leaving ReadWriteTimeout at the default, so your reads can take up to five minutes before timing out. You probably want to set ReadWriteTimeout to a lower value. You might also consider limiting the size of data that you download. With my crawler, I'd sometimes stumble upon an unending stream that would eventually result in an out of memory exception.
Something else I noticed when crawling is that sometimes closing the response stream will hang. I found that I had to call request.Abort to reliably terminate a request if I wanted to quit before reading the entire stream.
There is nothing apparent in the code you provided.
Why did you comment response.Close() out?
Documentation hints that connections may run out if not explicitly closed. The response getting disposed may close the connection but just releasing all the resources is not optimal I think. Closing the response will also close the stream so that is covered.
The system hanging without timeout can be just a network issue making the response object a dead duck or the problem is due the high number of threads resulting in memory fragmentation.
Looking at anything that may produce a pattern may help find the source:
How many threads are typically running (can you bundle request sets in less threads)
How is the network performance at the time the thread stopped
Is there a specific count or range when it happens
What data was processed last when it happened (are there any specific control characters or sequences of data that can upset the stream)
Want to ask more questions but not enough reputation so can only reply.
Good luck!
Below is some code that does something similar, it's also used to access multiple web sites, each call is in a different task. The difference is that I only read the stream once and then parse the results. That might be a way to get around the stream reader locking up randomly or at least make it easier to debug.
try
{
_webResponse = (HttpWebResponse)_request.GetResponse();
if(_request.HaveResponse)
{
if (_webResponse.StatusCode == HttpStatusCode.OK)
{
var _stream = _webResponse.GetResponseStream();
using (var _streamReader = new StreamReader(_stream))
{
string str = _streamReader.ReadToEnd();
I have a simple C# Windows Service that should respond "OK" to GET requests on port 8080.
Everything works fine, with one simple itch...
Every single request made to the service stays in TIME_WAIT.
Am I missing something, I've looked around and most samples I've seen on HTTPListeners and Windows Services are doing it in a similar fashion?!
private HttpListener _listener;
protected override void OnStart(string[] args)
{
_listener = new HttpListener();
_listener.Prefixes.Add("http://*:8080/");
_listener.Start();
_listener.BeginGetContext(new AsyncCallback(OnRequestReceive), _listener);
}
protected override void OnStop()
{
_listener.Stop();
}
private void OnRequestReceive(IAsyncResult result)
{
if (!_listener.IsListening)
return;
//Get context for a request.
HttpListenerContext context = _listener.EndGetContext(result);
HttpListenerRequest request = context.Request;
//Obtain a response object.
HttpListenerResponse response = context.Response;
response.ContentType = "application/json";
response.KeepAlive = false;
//Our return message...
string responseString = "OK";
//Construct the response.
byte[] buffer = Encoding.UTF8.GetBytes(responseString);
Stream output = response.OutputStream;
response.ContentLength64 = buffer.Length;
output.Write(buffer, 0, buffer.Length);
//Close and send response
try
{
output.Flush();
output.Close();
response.Close();
}
finally
{
//Wait for another request
_listener.BeginGetContext(new AsyncCallback(OnRequestReceive), _listener);
}
}
Edit: Fixed Local declaration of _listener.
Thanks to rene for pointing out the correct direction...
TIME-WAIT
(either server or client) represents waiting for enough time to pass to be sure the remote TCP received the acknowledgment of its connection termination request. [According to RFC 793 a connection can stay in TIME-WAIT for a maximum of four minutes known as a MSL (maximum segment lifetime).]
For anyone else who wants to change this behavior:
The TIME_WAIT period is configurable by modifying the following DWORD registry setting that represents the TIME_WAIT period in seconds.
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\TCPIP\Parameters\TcpTimedWaitDelay
I created simple HttpListener that listens to port 9090 and depending on a request's URL writes some info to the console.
But I'm stuck :( I thought of multithreading, event based system, but I dind't manage to do anything with it.
Here is the code of my listener that I launched as a separate console app:
string urlTemplate = String.Format("/prefix/{0}/suffix", id);
string prefix = String.Format("http://localhost:9090/");
HttpListener listener = new HttpListener();
listener.Prefixes.Add(prefix);
listener.Start();
Console.WriteLine("Listening to {0}...", prefix);
while (true)
{
HttpListenerContext context = listener.GetContext();
HttpListenerRequest request = context.Request;
//Response object
HttpListenerResponse response = context.Response;
//Construct response
if (request.RawUrl.Contains(urlTemplate) && request.HttpMethod == "POST")
{
string requestBody;
Stream iStream = request.InputStream;
Encoding encoding = request.ContentEncoding;
StreamReader reader = new StreamReader(iStream, encoding);
requestBody = reader.ReadToEnd();
Console.WriteLine("POST request on {0} with body = [{1}]", request.RawUrl, requestBody);
response.StatusCode = (int)HttpStatusCode.OK;
//Return a response
using (Stream stream = response.OutputStream) { }
}
else
{
response.StatusCode = (int)HttpStatusCode.BadRequest;
Console.WriteLine("Invalid HTTP request: [{0}] {1}", request.HttpMethod, request.Url);
using (Stream stream = response.OutputStream) { }
}
}
I decided to use it as an utility for unit tests (maybe somewhere else). So when test starts I need to configure the listener, run it, then make some requests and receive the info (which listener wrote earlier to Console), and at the end of the test stop the listener.
My main idea was to incapsulate this listener to separate class MyHttpListener which has methods: StartListener(), StopListener().
But when I call StartListener() my test freezes because of infinite while loop. I tried to create separate background thread or event based system, but my lack of experience with them, prevents me from doing it. I've already spent a lot of time trying to find the solution, but all for nothing.
Hope you can help me finding the solution for such trivial task.
Thanks in advance.
One of the responder's variant (it seems he deleted his post) looked good, but it didn't work for me. I tried to fix things in order it started working, but at the end that variant gave me an idea how to solve the problem - with multithreading and events :)
Here's what I have now and it works:
public delegate void HttpListenerRequestHandler(object sender, HttpListenerEventArgs e);
public event HttpListenerRequestHandler OnCorrectRequest;
...
if(OnCorrectRequest != null)
OnCorrectRequest(this, new HttpListenerEventArgs(response));
lock (threadLock)
{
Console.WriteLine("POST request on {0} with body = [{1}]", request.RawUrl, requestBody);
}
...
public class HttpListenerEventArgs : EventArgs
{
public readonly HttpListenerResponse response;
public HttpListenerEventArgs(HttpListenerResponse httpResponse)
{
response = httpResponse;
}
}
In order to receive detailed responses from HttpListener in main thread, I use the following:
private HttpListenerResponse response;
public void HttpCallbackRequestCorrect(object sender, HttpListenerEventArgs e)
{
response = e.response;
Console.WriteLine("{0} sent: {1}", sender, e.response.StatusCode);
}
Unfortunately I have the following exception which I don't know how to handle:
System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request
at System.Net.HttpListener.GetContext()