Develop Reference

Random 408 error on POST request

I'm trying to send a POST request to a website in C#, and then parse the html in the response to get certain item names from it. However, I am getting a 408 error about 50% of the time I run the program. Here is my code (most is taken from here: https://msdn.microsoft.com/en-us/library/debx8sh9.aspx):
using System;
using System.IO;
using System.Net;
using System.Text;
using System.Collections.Generic;
using HtmlAgilityPack;
using System.Net;
using System.Diagnostics;
using Newtonsoft.Json.Linq;
using System.Linq;
using Fizzler.Systems.HtmlAgilityPack;
class FinderClass
{
//some irrelevant code here
public int getItemIndex(string itemName)
{
itemName = itemName.Replace(" ", "+"); //formatting for request
itemName = itemName.Replace("|", "%7C");
//taken from https://msdn.microsoft.com/en-us/library/debx8sh9(v=vs.110).aspx :
WebRequest request = WebRequest.Create("http://csgolounge.com/ajax/tradeCsRightTmp.php"); //address to send request
request.Method = "POST";
string postData = "type=Type+-+All&quality=0&exterior=0&fraze=" + itemName + "&search=1&page=1"; //request parameters
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = byteArray.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
WebResponse response = request.GetResponse();
Debug.WriteLine(((HttpWebResponse)response).StatusDescription);
dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
Debug.WriteLine(responseFromServer); //print response to debug console (temporary)
reader.Close();
dataStream.Close();
response.Close();
return -1; //placeholder for when item index is parsed from html and returned
}
}
The exception that is thrown:
An unhandled exception of type 'System.Net.WebException' occurred in System.dll
Additional information: The remote server returned an error: (408) Request Timeout.
Normally I would think there is a straightforward solution to this, but since it only happens half the time I run the program, it might be something more complicated.

Any time that I hear "so and such network problem happens after a while", I tend to look at the code and see if using blocks are being used correctly.
using blocks should be used whenever you
Create some resource, and
Use that resource, and
Finish with that resource in the same method
(there are some other times, but this is the most frequent case)
In your code, you are creating a WebResponse, a Stream, and a StreamReader, you're using them, then you're finishing with them, and you're not cleaning up.
Manually using a Dispose or Close call isn't always enough. These will get skipped over if an exception is thrown in the code and unhandled. A using block will ensure that the cleanup happens, even in the presence of unhandled exceptions.

This error is because of a timeout. On the face of it, it looks as though the server is timing out. So, I'd investigate the server side to see how long the call is taking. However, there's a chance that it may just be a client side problem because I sometimes get this error when I pause the debugger on the client side. I.e. the Request is sent to the server, the server sends the Response back, but the client doesn't accept it. I know this sounds funny, and I haven't been able to figure out why this happens myself. Anyway, you can try setting the timeout to something much larger in the WebRequest on the client side:
request.Timeout = 60000;

Fiddler stops StreamWriter timeout

I'm stumped - looking for some help if possible....
I use Fiddler a lot to help me write code against remote APIs, have done for ages without any issues. Happy Days!
But, I have hit a problem I just cannot get my head around...
I am making a call to a remote API. When I make the call WITH fiddler running it works perfectly. When I make the very same call WITHOUT fiddler running, the call times out?
This is not just happening on my development machine. The client reported the problem first. And sure enough, when I asked him to install fiddler it works at his end as well!
I just cannot work out why making a call to this API fails only when Fiddler is not running and works when fiddler is running??
Can anyone suggest anything please?
This is my code:
public static SalesForceModel.ClipIdResponse getClipId(string instance_url, string access_token, string clipTitle)
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create(instance_url + "/services/data/v20.0/sobjects/vClip__c");
httpWebRequest.Headers.Add("Authorization", "Bearer " + access_token);
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = "POST";
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
string json = "{\"Type__c\":\"PowerPoint\"," +
"\"Title__c\":\"" + clipTitle + "\"}";
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
return JsonConvert.DeserializeObject<SalesForceModel.ClipIdResponse>(result);
}
}
}

Typically this behavior indicates that you've failed to properly close the stream returned by the GetResponseStream() function, as described in the HTTPWebRequest section of this blog post.
However, your using statement should be properly disposing of the used StreamReader and that should be closing the underlying stream properly.
Is there any other use of HTTPWebRequest's GetResponseStream() function elsewhere in your app that might not be closing a stream? Is there a change in behavior if you explicitly call httpResponse.Close() after calling the ReadToEnd() method?

HttpWebRequest, BeginGetResponse not called until endOfStream

I am working on a client that uses a webservice to get some events pushed its way - the webservice is designed so, that upon the client POST'ing a subscribe command, it will send back some events of interest and keep doing so as long as the client stay connected.
When POSTing the command, the service responds (immediately) with an initial answer with these headers
Keep-Alive: timeout=5, max=98
Connection: Keep-Alive
Transfer-Encoding: chunked
and then keeps the connection open until it times out (after 30s, if the client does not send some keep-alive data)
Since it is a mix of POST + having to read the response + keeping the connection open until endOFStream, it appears I have to use HttpWebRequest with BeginGetRequestStream (to POST) and BeginGetResponse to read and act on the response.
My problem is that the BeginGetResponse callback is not called until the input stream is actually closed by the server/service (after 30s), despite AllowReadStreamBuffering being set to false.
The doc have this to say on AllowReadStreamBuffering:
The AllowReadStreamBuffering property affects when the callback from BeginGetResponse method is called. When the AllowReadStreamBuffering property is true, the callback is raised once the entire stream has been downloaded into memory. When the AllowReadStreamBuffering property is false, the callback is raised as soon as the stream is available for reading which may be before all data has arrived.
I've seen a few suggestions that no matter what AllowReadStreamBuffering is set to, HttpWebRequest will not call BeginGetResponse until it's buffer is filled up - but I have not been able to find anything on that in the docs.
Does any one have an idea on how to control this buffering behaviour or maybe suggestion to another approach I should try when dealing with this kind of webservice?
The relevant snippets of the code I currently use, look like this:
public void open()
{
string url = "http://funplaceontheinternet/webservice";
HttpWebRequest request = WebRequest.CreateHttp(url);
request.Method = "POST";
request.Credentials = new NetworkCredential("username", "password");
request.CookieContainer = new CookieContainer();
request.AllowReadStreamBuffering = false;
request.BeginGetRequestStream(new AsyncCallback(GetRequestStreamCallback), request);
}
void GetRequestStreamCallback(IAsyncResult result)
{
Debug.WriteLine("open.GetRequestStreamCallback");
HttpWebRequest webRequest = (HttpWebRequest)result.AsyncState;
// End the stream request operation
Stream postStream = webRequest.EndGetRequestStream(result);
// Create the post data
byte[] byteArray = Encoding.UTF8.GetBytes(_xmlEncodedSubscribeCommand);
// Add the post data to the web request
postStream.Write(byteArray, 0, byteArray.Length);
postStream.Close();
// Start the web request
webRequest.BeginGetResponse(new AsyncCallback(BeginGetResponseCallback), webRequest);
}
void BeginGetResponseCallback(IAsyncResult result)
{
HttpWebRequest request = (HttpWebRequest)result.AsyncState;
HttpWebResponse response = null;
if (request != null)
response = (HttpWebResponse)request.EndGetResponse(result);
else
Debug.WriteLine("request==null :-(");
if (response != null)
{
using (var reader = new StreamReader(response.GetResponseStream()))
{
while (!reader.EndOfStream)
{
string line = reader.ReadLine();
Debug.WriteLine("BeginGetResponseCallback - received: " + line);
}
Debug.WriteLine("BeginGetResponseCallback - reader.EndOfStream");
}
}
else
Debug.WriteLine("response==null :-(");
}

You've mentioned that the service is a web service, but not which platform.
If this is a "normal" web service, then I assume that XML is the transport format.
If so, I suspect the problem may be that this style of communication does not really lend itself to streaming. The web service infrastructure at the server end might not be creating the SOAP envelope and payload until all the data is available. If you wanted to stream like this, you might be better using some custom service at the server end, rather than a web service.
Do you know for sure that the server is really streaming the response? (e.g confirmed with something like wireshark?)
If you really want to use a web service, then I would suggest you complete the request when the first event(s) are available, and don't wait for the timeout. This will still achieve the latency reduction that I assume you are trying to get.

System.Net.WebException: The operation has timed out

I have a big problem: I need to send 200 objects at once and avoid timeouts.
while (true)
{
NameValueCollection data = new NameValueCollection();
data.Add("mode", nat);
using (var client = new WebClient())
{
byte[] response = client.UploadValues(serverA, data);
responseData = Encoding.ASCII.GetString(response);
string[] split = Javab.Split(new[] { '!' }, StringSplitOptions.RemoveEmptyEntries);
string command = split[0];
string server = split[1];
string requestCountStr = split[2];
switch (command)
{
case "check":
int requestCount = Convert.ToInt32(requestCountStr);
for (int i = 0; i < requestCount; i++)
{
Uri myUri = new Uri(server);
WebRequest request = WebRequest.Create(myUri);
request.Timeout = 200000;
WebResponse myWebResponse = request.GetResponse();
}
break;
}
}
}
This produces the error:
Unhandled Exception: System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at vir_fu.Program.Main(String[] args)
The requestCount loop works fine outside my base code but when I add it to my project I get this error. I have tried setting request.Timeout = 200; but it didn't help.

It means what it says. The operation took too long to complete.
BTW, look at WebRequest.Timeout and you'll see that you've set your timeout for 1/5 second.

Close/dispose your WebResponse object.

I'm not sure about your first code sample where you use WebClient.UploadValues, it's not really enough to go on, could you paste more of your surrounding code? Regarding your WebRequest code, there are two things at play here:
You're only requesting the headers of the response**, you never read the body of the response by opening and reading (to its end) the ResponseStream. Because of this, the WebRequest client helpfully leaves the connection open, expecting you to request the body at any moment. Until you either read the response body to completion (which will automatically close the stream for you), clean up and close the stream (or the WebRequest instance) or wait for the GC to do its thing, your connection will remain open.
You have a default maximum amount of active connections to the same host of 2. This means you use up your first two connections and then never dispose of them so your client isn't given the chance to complete the next request before it reaches its timeout (which is milliseconds, btw, so you've set it to 0.2 seconds - the default should be fine).
If you don't want the body of the response (or you've just uploaded or POSTed something and aren't expecting a response), simply close the stream, or the client, which will close the stream for you.
The easiest way to fix this is to make sure you use using blocks on disposable objects:
for (int i = 0; i < ops1; i++)
{
Uri myUri = new Uri(site);
WebRequest myWebRequest = WebRequest.Create(myUri);
//myWebRequest.Timeout = 200;
using (WebResponse myWebResponse = myWebRequest.GetResponse())
{
// Do what you want with myWebResponse.Headers.
} // Your response will be disposed of here
}
Another solution is to allow 200 concurrent connections to the same host. However, unless you're planning to multi-thread this operation so you'd need multiple, concurrent connections, this won't really help you:
ServicePointManager.DefaultConnectionLimit = 200;
When you're getting timeouts within code, the best thing to do is try to recreate that timeout outside of your code. If you can't, the problem probably lies with your code. I usually use cURL for that, or just a web browser if it's a simple GET request.
** In reality, you're actually requesting the first chunk of data from the response, which contains the HTTP headers, and also the start of the body. This is why it's possible to read HTTP header info (such as Content-Encoding, Set-Cookie etc) before reading from the output stream. As you read the stream, further data is retrieved from the server. WebRequest's connection to the server is kept open until you reach the end of this stream (effectively closing it as it's not seekable), manually close it yourself or it is disposed of. There's more about this here.

proxy issue can cause this. IIS webconfig put this in
<defaultProxy useDefaultCredentials="true" enabled="true">
<proxy usesystemdefault="True" />
</defaultProxy>

I remember I had the same problem a while back using WCF due the quantity of the data I was passing. I remember I changed timeouts everywhere but the problem persisted. What I finally did was open the connection as stream request, I needed to change the client and the server side, but it work that way. Since it was a stream connection, the server kept reading until the stream ended.

I encountered the same error than adding
Task.Delay(2000);
in each request solved the problem

Error using HttpWebRequest to upload files with PUT

We've got a .NET 2.0 WinForms app that needs to upload files to an IIS6 Server via WebDav. From time to time we get complaints from a remote office that they get one of the following error messages
The underlying connection was closed:
an unexpected error occurred on send.
The underlying connection was closed:
an unexpected error occurred on
receive.
This only seems to occur with large files (~20Mb plus). I've tested it with a 40Mb file from my home computer and tried putting 'Sleep's in the loop to simulate a slow connection so I suspect that it's down to network issues at their end... but
The IT at the remote office are no help
I'd like to rule out the posibility my code is at fault.
So - can anybody spot any misakes or suggest any workarounds that might 'bulletproof' the code against this problem. Thanks for any help. Chopped down version of code follows:
public bool UploadFile(string localFile, string uploadUrl)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uploadUrl);
try
{
req.Method = "PUT";
req.AllowWriteStreamBuffering = true;
req.UseDefaultCredentials = Program.WebService.UseDefaultCredentials;
req.Credentials = Program.WebService.Credentials;
req.SendChunked = false;
req.KeepAlive = true;
Stream reqStream = req.GetRequestStream();
FileStream rdr = new FileStream(localFile, FileMode.Open, FileAccess.Read);
byte[] inData = new byte[4096];
int bytesRead = rdr.Read(inData, 0, inData.Length);
while (bytesRead > 0)
{
reqStream.Write(inData, 0, bytesRead);
bytesRead = rdr.Read(inData, 0, inData.Length);
}
reqStream.Close();
rdr.Close();
System.Net.HttpWebResponse response = (HttpWebResponse)req.GetResponse();
if (response.StatusCode != HttpStatusCode.OK && response.StatusCode!=HttpStatusCode.Created)
{
MessageBox.Show("Couldn't upload file");
return false;
}
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
return false;
}
return true;
}

Try setting KeepAlive to false:
req.KeepAlive = false;
This will allow the connection to be closed and opened again. It will not allow to use a persistent connection. I found a lot of references in the Web that suggested this in order to solve a similar to yours error. This is a relevant link.
Anyway, it is not a good idea to use HTTP PUT (or HTTP POST) to upload large files. It will be better to use FTP or a download/upload manager. These will handle retries, connection problems, timeouts automatically for you. The upload will be faster too and you could also resume a stopped uploading. If you decide to stay with HTTP, you should at least try to add a retry mechanism. If an upload is taking too long, then there is a high probability that it will fail due to proxy, server timeout, firewall or what ever reason not to have with your code.

To remove the risk of a bug in your code, try using WebClient:
using (WebClient client = new WebClient())
{
client.UseDefaultCredentials = Program.WebService.UseDefaultCredentials;
client.Credentials = Program.WebService.Credentials;
client.UploadFile(uploadUrl, "PUT", localFile);
}

Maybe try using POST, but the real culprit is probably the content type.
Try setting
req.ContentType = "application/octet-stream";
req.ContentLength = inData.Length;
or look at the code in the accepted answer here: Upload files with HTTPWebrequest (multipart/form-data)
Both my example and the link I provided involve modifying the ContentType - my example is simpler but might not work, as most applications receiving files expect multipart

Please you check whether [Enable Http Keep-Alives] is set [on] at [Web Site] tab in IIS manager.

The size of the uploads might be limited.
See here for one discussion:
http://www.codeproject.com/KB/aspnet/uploadlargefilesaspnet.aspx

Start by checking some basic configuration. The default values of either of the following may cause problems in file upload - including termination of the connection. I believe IIS 6 would never allow file upload > 2GB (even if it could complete, regardless of config). Msdn describes these nicely.
<httpRuntime executionTimeout = "30" maxRequestLength="200"/>
EDIT: This is ASP.NET config, of course, which assumes you are running your own webdav server or a 3rd party server within ASP.NET. If it's a different webdav server, you'll want to look for the equivalent.