Develop Reference

Random 408 error on POST request

I'm trying to send a POST request to a website in C#, and then parse the html in the response to get certain item names from it. However, I am getting a 408 error about 50% of the time I run the program. Here is my code (most is taken from here: https://msdn.microsoft.com/en-us/library/debx8sh9.aspx):
using System;
using System.IO;
using System.Net;
using System.Text;
using System.Collections.Generic;
using HtmlAgilityPack;
using System.Net;
using System.Diagnostics;
using Newtonsoft.Json.Linq;
using System.Linq;
using Fizzler.Systems.HtmlAgilityPack;
class FinderClass
{
//some irrelevant code here
public int getItemIndex(string itemName)
{
itemName = itemName.Replace(" ", "+"); //formatting for request
itemName = itemName.Replace("|", "%7C");
//taken from https://msdn.microsoft.com/en-us/library/debx8sh9(v=vs.110).aspx :
WebRequest request = WebRequest.Create("http://csgolounge.com/ajax/tradeCsRightTmp.php"); //address to send request
request.Method = "POST";
string postData = "type=Type+-+All&quality=0&exterior=0&fraze=" + itemName + "&search=1&page=1"; //request parameters
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = byteArray.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
WebResponse response = request.GetResponse();
Debug.WriteLine(((HttpWebResponse)response).StatusDescription);
dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
Debug.WriteLine(responseFromServer); //print response to debug console (temporary)
reader.Close();
dataStream.Close();
response.Close();
return -1; //placeholder for when item index is parsed from html and returned
}
}
The exception that is thrown:
An unhandled exception of type 'System.Net.WebException' occurred in System.dll
Additional information: The remote server returned an error: (408) Request Timeout.
Normally I would think there is a straightforward solution to this, but since it only happens half the time I run the program, it might be something more complicated.

Any time that I hear "so and such network problem happens after a while", I tend to look at the code and see if using blocks are being used correctly.
using blocks should be used whenever you
Create some resource, and
Use that resource, and
Finish with that resource in the same method
(there are some other times, but this is the most frequent case)
In your code, you are creating a WebResponse, a Stream, and a StreamReader, you're using them, then you're finishing with them, and you're not cleaning up.
Manually using a Dispose or Close call isn't always enough. These will get skipped over if an exception is thrown in the code and unhandled. A using block will ensure that the cleanup happens, even in the presence of unhandled exceptions.

This error is because of a timeout. On the face of it, it looks as though the server is timing out. So, I'd investigate the server side to see how long the call is taking. However, there's a chance that it may just be a client side problem because I sometimes get this error when I pause the debugger on the client side. I.e. the Request is sent to the server, the server sends the Response back, but the client doesn't accept it. I know this sounds funny, and I haven't been able to figure out why this happens myself. Anyway, you can try setting the timeout to something much larger in the WebRequest on the client side:
request.Timeout = 60000;

HttpWebRequest WebException ProtocolError

I'm trying to understand something about exception handling with a HttpWebRequest.
I have a client library and it's making a request to a WebAPI controller;
HttpWebRequest r = (HttpWebRequest)WebRequest.Create(url);
r.Method = "POST";
r.ContentType = "application/json";
foreach (var header in request.Headers)
{
r.Headers.Add(header.Key, header.Value.ToString());
}
r.ContentLength = request.RequestBody.Length;
using (StreamWriter writer = new StreamWriter(r.GetRequestStream()))
writer.Write(request.RequestBody);
I know the request will throw an exception, and contain the message entity already exists - 1234.
When I get the response;
using (HttpWebResponse response = (HttpWebResponse)r.GetResponse())
{
if (response.StatusCode == HttpStatusCode.OK)
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
return reader.ReadToEnd();
return "Invalid";
}
I get a WebException thrown. So, the caller of the request has a try..catch in it. And I get the WebException. What I get is a protocol error, not the 500 internal server error that was thrown (using correct status codes to represent the message comes later). Now if I read the Response of the WebException, it does contain my message and the stacktrace.
Questions
Why do I not get a status code of 500 in my response, why does it throw a protocol error?
Is there a more correct way of handling the request?
I have searched around and found some people getting this issue when not using the correct headers etc. But as far as I can tell, I have added all the headers that I can and still get the same behavior.

An 500 internal server error usually means that the API received the request but threw an unhandled exception while processing it, thus the "Internal Server Error".
You may log to a database or file all your API's unhandled exceptions to help your debugging process. Good luck.

500 Internal Server Error at HttpWebResponse

I have a few web services up and running fine on my localhost. The problem is when I try requesting a response from it, the following error comes up when debugging:
Web Exception was unhandled by user code
The remote server returned an error: (500) Internal Server Error
The piece of my code where it throws this error is at the HttpWebResponse header:
using (HttpWebResponse httpResponse = httpRequest.GetResponse() as HttpWebResponse)
{
Encoding enc = Encoding.GetEncoding(1252);
using (StreamReader responseStream = new StreamReader(httpResponse.GetResponseStream(), enc))
{
bl = responseStream.ReadToEnd();
}
}
Any tips? I'm sort of new to this...much appreciated.

The 500 error means something went wrong in the server side.
If you also wrote the web services, that's the code you have to inspect and debug.

Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host in C#

I have a code where I am sending the URL request and receiving the response and storing it as a String as
public String GenerateXML(String q)// Here 'q' is the URL
{
// Generating the XML file for reference
// Getting the response in XML format from the URL
Debug.WriteLine("The Http URL after URL encoding :" + q);
try
{
Uri signs1 = new Uri(q);
//Debug.WriteLine("The Requested URL for getting the XML data :" + re);
WebRequest request1 = WebRequest.Create(signs1);
HttpWebResponse response1 = (HttpWebResponse)request1.GetResponse();
//HttpWebResponse response1 = (HttpWebResponse)request1.GetResponse();
Stream receiveStream = response1.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
String ab = readStream.ReadToEnd();// The mentioned error is showing up here.
// Debug.WriteLine("The data :"+a);
//XmlDocument content2 = new XmlDocument();
// content2.LoadXml(ab);
// content2.Save("C:/Users/Administrator/Downloads/direct.xml");
return ab;
}
catch (System.Net.WebException ex)
{
Debug.WriteLine("Exception caught :" + ex);
return null;
}
}
Why is the connection closed by the remote host?
What are the possibilities of getting rid of the error or at least ignore the error and continue with other URL requests? I have included try and catch so as to escape any error and continue functioning with out any stop. Scoured the internet for solution but solutions to this particular problem is pretty much specific. Please any help is appreciated.
Thanks in advance.

I had similar problems with the connection being forcibly closed with different hosts. Seems the issue can be resolved by altering various properties of the WebRequest object.
The following findings were outlined in a blog post by briancaos: An existing connection was forcibly closed by the remote host
The steps mentioned in the the above post include:
Set WebRequest.KeepAlive to False.
Set WebRequest.ProtocolVersion to HttpVersion.Version10.
Set WebRequest.ServicePoint.ConnectionLimit to 1
It indeed work for me, but I haven't tested it on multiple hosts as of yet. However, I seriously suggest reading the post as it goes into way more detail.
In case the link get's broken, here's the Archive.org cached version.

The actual exception is probably an IOException - you would need to catch that exception type as well as WebException. The actual problem may be that your URL is out of date and the system is no longer running a web server, or perhaps, the request needs to be authenticated/needs a header as #L.B suggests.
Also, you are potentially leaking all sorts of resources. You should be wrapping your WebResponse and streams in using statements.
using (var response = (HttpWebResponse)request.GetResponse())
using (var receiveStream = response.GetResponseStream())
using (var reader = new StreamReader(receiveStream))
{
var content = reader.ReadToEnd();
// parse your content, etc.
}

Had the same problem today, So I also wrapped the request in a try/catch with WebException, in my case, adding:
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
Before the webRequest did the trick. Also you should be wrapping your WebResponse and streams in using statements as tvanfosson mentioned.
I hope this helps.

HttpWebResponse.ReadTimeout - Timeouts not supported?

We have an issue where on a single instance of our product we receive an InvalidOperationException exception when we attempt to set the ReadTimeout property of an System.Net.HttpWebResponse object.
This issue is occurring only on a single instance, where we have many multiple live sites without this problem. We've tried to recreate the issue locally, to no avail.
The following code illustrates the issue.
Any ideas are greatly welcome.
Thanks
private static XmlReader GenerateReaderFromResponse(HttpWebResponse response, HttpWebRequest request)
{
Stream responseStream = response.GetResponseStream();
responseStream.ReadTimeout = request.Timeout; //This is where the exception is generated - System.InvalidOperationException: Timeouts are not supported on this stream.
using (StreamReader responseReader = new StreamReader(responseStream, System.Text.Encoding.UTF8))
{
XmlReaderSettings readerSettings = new XmlReaderSettings();
readerSettings.ProhibitDtd = false;
string responseContent = responseReader.ReadToEnd();
return XmlReader.Create(new StringReader(responseContent), readerSettings);
}
}

What you need is the HttpWebRequest.ReadWriteTimeout property.
It specifies the number of milliseconds before the reading (or writing) operation on the response Stream times out, throwing a WebException with Status set to WebExceptionStatus.RequestCanceled.
From the msdn documentation:
The ReadWriteTimeout property is used when writing to the stream
returned by the GetRequestStream method or reading from the stream
returned by the GetResponseStream method.
Specifically, the ReadWriteTimeout property controls the time-out for
the Read method, which is used to read the stream returned by the
GetResponseStream method, and for the Write method, which is used to
write to the stream returned by the GetRequestStream method.
To specify the amount of time to wait for the request to complete, use
the Timeout property.

First, make sure that responseStream and request are not null.
Them, you should make sure that the server did respond to your request after trying to read the response from it.
If you can, please also provide the code that sends the request.