Develop Reference

Capture response information from (3rd party) web service on error

I have the following code, which submits XML to a 3rd party web service, which errors (intentionally, at the moment) on "req.GetResponse()" with the error, detailed below.
byte[] bytes = Encoding.ASCII.GetBytes(myXMLData);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(myWebsite);
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse response = req.GetResponse();
string responseStream = StringFromResponseStream(response);
Error from GetResponse()
Exception thrown: 'System.New.WebExtension' in System.dll
Additional information: The remote server returned an error: (400) Bad Request
When I trace this call in Fiddler, I can see that the response from the service, also includes a far more useful error (below; RAW view), which I am trying to get to:
HTTP/1.1 400 Bad Request
Date: Thu, 16 Jun 2016 10:42:26 GMT
Cache-Control: private
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Content-Type: text/plain; charset=UTF-8
Content-Length: 54
Caseprovider-Credentials: <snip>
Caseprovider-Credentialshash: <snip>
Caseprovider-Apiversion: 15
Connection: close
No supported action 'SomeName' available for 'SomeValue'
Having a 'watch' on the variables doesn't seem to show where I might obtain this from (and quite possibly something simple that I have overlooked)

Have eventually found a resolution for the; where 'result' contains the content from the process; there's more to do, but this is the basics of what I needed.
using (var httpClient = new HttpClient())
{
httpClient.BaseAddress = new Uri(myWebSite);
httpClient.DefaultRequestHeaders.Accept.Clear();
httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("text/xml"));
StringContent stringContent = new StringContent(myXMLData);
stringContent.Headers.ContentType = new MediaTypeHeaderValue("text/xml");
HttpResponseMessage httpResponseMessage = httpClient.PostAsync(httpClient.BaseAddress, stringContent).Result;
string result = httpResponseMessage.Content.ReadAsStringAsync().Result;
}

Http CONNECT request return empty respone

I want to read Https page using TcpClient. I use below code
var client = new TcpClient(url, 443);//"127.0.0.1", 8888);// Fiddler port
client.SendTimeout = 30000;
Stream responseStream = client.GetStream();
// send CONNECT request to server
byte[] tunnelRequest = Encoding.ASCII.GetBytes("CONNECT www.google.com:443 HTTP/1.1\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:44.0) Gecko/20100101 Firefox/35.0\r\nProxy-Connection: keep-alive\r\nConnection: keep-alive\r\nHost: www.google.com:443\r\n\r\n");
responseStream.Write(tunnelRequest, 0, tunnelRequest.Length);
responseStream.Flush();
// read CONNECT response
string connectResponse = ReadResponse(responseStream);
Console.WriteLine("server connect response : " + connectResponse);
that send CONNECT request to host (google.com)
CONNECT www.google.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0
Proxy-Connection: keep-alive
Connection: keep-alive
Host: www.google.com:443
respone must be somthing like this
HTTP/1.1 200 Connection Established
StartTime: 22:42:38.774
Connection: close
but responseStream return nothing. when I use Fiddler as a proxy
var client = new TcpClient("127.0.0.1", 8888);
it works fine and return 200 response. There is something wrong that Fiddler fixed it?
I use windows 8.1 and test with .Net 2 and 4.5.1.

Base on rfc CONNECT method used when proxy exists.
Since TLS, in particular, requires end-to-end connectivity to provide
authentication and prevent man-in-the-middle attacks, this memo
specifies the CONNECT method to establish a tunnel across proxies.
In case of direct communication :
var client = new TcpClient("127.0.0.1", 8888);//url, 443);//
client.SendTimeout = 30000;
Stream responseStream = client.GetStream();
// Wrap in SSL stream
SslStream sslStream = new SslStream(responseStream);
sslStream.AuthenticateAsClient(url);
byte[] byts = Encoding.ASCII.GetBytes("GET / HTTP/1.1\r\nHost: www.google.com\r\nConnection: keep-alive\r\n\r\n");
sslStream.Write(byts, 0, byts.Length);
var str = ReadResponse(sslStream);

Content length is always negative

I don't know but content length is always negative even if repsonse has corrent Content-Length header. For example:
class Program
{
static void Main()
{
try
{
string result = Get("http://stackoverflow.com/");
Console.WriteLine("Response length = {0}", result.Length);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
static string Get(string adr)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(adr);
req.UserAgent = "Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0";
req.Proxy = null;
req.KeepAlive = false;
req.Headers.Add("Accept-Language", "ru-RU,ru;q=0.9,en;q=0.8");
req.AllowAutoRedirect = true;
req.Timeout = 10000;
req.ReadWriteTimeout = 10000;
req.MaximumAutomaticRedirections = 10;
req.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
req.Method = WebRequestMethods.Http.Get;
using (var response = (HttpWebResponse) req.GetResponse())
{
using (var stream = response.GetResponseStream())
{
if (stream == null)
throw new NullReferenceException("Response stream is nulL!");
using (var reader = new StreamReader(stream, Encoding.Default))
{
Console.WriteLine("Content length = {0}", response.ContentLength);
return WebUtility.HtmlDecode(reader.ReadToEnd());
}
}
}
}
}
fiddler shows following output:
HTTP/1.1 200 OK
Date: Wed, 02 Sep 2015 20:36:42 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Cache-Control: private, max-age=0
Cf-Railgun: fe8c0e42fd 44.42 0.042796 0030 3350
X-Frame-Options: SAMEORIGIN
X-Request-Guid: bc3ccb1f-1de5-4375-b30d-f3c89134cf86
Server: cloudflare-nginx
CF-RAY: 21fc0233f6652bca-AMS
Content-Length: 251540
But in program I get this:
how can it be fixed?

A couple of things:
When I try, stackoverflow.com does not set a Content-Length header, so it will come through as -1.
But, changing the URL to use a server which definitely does set a Content-Length header, e.g., www.theguardian.com, still produces the same result: -1.
I think it's your use of AutomaticDecompression on the HttpWebRequest object.
If you don't set that property the ContentLength property comes through correctly.

This means that it is Fiddler that is returning the content-length. The reason that Fiddler would do that is that you have the 'Stream' button pressed (or some otherway to indicate to fiddler you want to return 'Stream' or 'Chunked' data.
When a server returns 'Chunked' data, it sets the content-length to -1 -- you don't know the length of the content since you can have unlimited chunks.
If you unclick (or turn off) streaming, then fiddler returns its default 'buffered' response, which is an exact copy of the response from the server. Which of course will include that Content-Length header.
Literally just went through this -- HTH!

Problems with POST request parameters in C#

This website uses POST to send data whenever the user clicks on a calendar to change the date. I used Firebug to inspect it. The target URL is this. The post data (space-separated) for a particular example is LeagueID=9 GameDate=4-29-2011 Season=2010-2011 Refresh=false LastUpdateTime=01-01-1900 type=Matchups RefreshStartTime=15-5-2011-1308094688683 Week= conferenceID=.
And here are the headers:
Host scores.covers.com
User-Agent Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-gb,en;q=0.5
Accept-Encoding gzip, deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 115
Connection keep-alive
Referer http://scores.covers.com/basketball-scores-matchups.aspx
Content-Length 169
Content-Type text/plain; charset=UTF-8
Cookie __loooooooooongCookieString
I'd like to make that POST request using WebRequest (or whetever else does the trick). Here's my attempt:
string parameters = "LeagueID=\"9\"&GameDate=\"4-29-2011\"&Season=\"2010-2011\"&Refresh=\"false\"&LastUpdateTime=\"01-01-1900\"&type=\"Matchups\"&RefreshStartTime=\"15-5-2011-1308094688683\"&Week=&conferenceID=";
byte[] bytes = Encoding.ASCII.GetBytes(parameters);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://scores.covers.com/ajax/SportsDirect.Controls.LiveScoresControls.Scoreboard,SportsDirect.Controls.LiveScoresControls.ashx?_method=UpdateScoreboard&_session=no");
req.Method = "POST";
req.ContentLength = bytes.Length;
req.ContentType = "text/plain; charset=UTF-8";
Console.WriteLine(req.ContentLength); // 175
Stream reqStream = req.GetRequestStream();
reqStream.Write(bytes, 0, bytes.Length);
reqStream.Close();
WebResponse resp = req.GetResponse();
Console.WriteLine(((HttpWebResponse)resp).StatusDescription); // OK
Stream respStream = resp.GetResponseStream();
StreamReader reader = new StreamReader(respStream);
Console.WriteLine(reader.ReadToEnd());
resp.Close();
But it doesn't work. The response code is OK, but the response itself is this:
new Object();r.error = new ajax_error('System.FormatException','Input string was
not in a correct format.\r\nCould not retreive parameters from HTTP request.',0
)new Object();r.error = new ajax_error('System.ArgumentException','Object of typ
e \'System.DBNull\' cannot be converted to type \'System.Int32\'.',0)
What's the deal? I can see that something's wrong with the params since the content length of the request is 175 (as opposed to the 169 from the request made by Firefox).

Why not use NameValueCollection to POST your parameters using a WebClient? It does the tricky stuff for you. The code at the bottom of the linked page is about as simple as it comes. Unlike the sample, you should probably deal thoughtfully with disposal of the WebClient.

Don't ASCII encode when you specify UTF-8 later. Make sure to url encode parameters. Try changing the content-type to 'x-www-form-urlencoded'.

HttpWebRequest POST gives 500 server error

I need to make from my app an authentificated httpwebrequest. the response to my request should be in json format. for this i'm using the code below:
// Create the web request
Uri address = new Uri("http://www.mysite.com/remote/user/login/format/json");
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.UseDefaultCredentials = false;
request.Credentials = new NetworkCredential(UserName, Password);
request.PreAuthenticate = true;
request.ContentType = "application/x-www-form-urlencoded";
request.Accept = "application/json";
string data = string.Format("username={0}&password={1}", otherusername, otherpassword);
// Create a byte array of the data we want to send
byte[] byteData = UTF8Encoding.UTF8.GetBytes(data);
// Set the content length in the request headers
request.ContentLength = byteData.Length;
//Write data
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(byteData, 0, byteData.Length);
}
// Get response
try
{
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream
StreamReader reader = new StreamReader(response.GetResponseStream());
// Console application output
jsonResponse = reader.ReadToEnd();
reader.Close();
}
user = new User();
JObject o = JObject.Parse(jsonResponse);
user.Unguessable_id = (string)o["unguessable_id"];
user.Print_id = (string)o["print_id"];
user.Rrid = (string)o["rrid"];
user.Raid = (string)o["raid"];
}
catch (WebException ex) {
errorMessage = ex.Message;
}
the problem is that the very first call it always gives a 500 error on the server. and the request fails. if i redo the call(by making an refresh in my browser) the request is successful.
the request should look like this in normal conditions:
POST /remote/user/login/format/json HTTP/1.1
Host: <yourhost>
username=user&password=pass
but when the server sends out the 500 error he received something like this:
username=user&password=passwordPOST /remote/user/login/format/json HTTP/1.1
any idea why this is happening? in my test app if i refresh the page that makes the httpwebrequest the call is successful.
EDIT:
after installing Fiddler the requests made look like this:
=> the one that generates 500
POST http://www.mysite.com/remote/user/login/format/json HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Accept: application/json
Host: www.mysite.com
Content-Length: 30
Expect: 100-continue
Connection: Keep-Alive
username=user&password=pass
=> the one made on refresh
POST http://www.mysite.com/remote/user/login/format/json HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Accept: application/json
Authorization: Basic ZGNpOkFpR2g3YWVj
Host: www.mysite.com
Content-Length: 30
Expect: 100-continue
Connection: Keep-Alive
username=user&password=pass
it seems that Authorization: Basic ZGNpOkFpR2g3YWVj is not included in the first request...why is that happening?(i'm using the same code for both requests)

I would advice you to install Fiddler to see what's really happening

I needed to add:
request.Headers.Add("Authorization: Basic ZGNpOkFpR2g3YWVj");
weird though that for the second request it was added automatically..