I don't know but content length is always negative even if repsonse has corrent Content-Length header. For example:
class Program
{
static void Main()
{
try
{
string result = Get("http://stackoverflow.com/");
Console.WriteLine("Response length = {0}", result.Length);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
static string Get(string adr)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(adr);
req.UserAgent = "Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0";
req.Proxy = null;
req.KeepAlive = false;
req.Headers.Add("Accept-Language", "ru-RU,ru;q=0.9,en;q=0.8");
req.AllowAutoRedirect = true;
req.Timeout = 10000;
req.ReadWriteTimeout = 10000;
req.MaximumAutomaticRedirections = 10;
req.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
req.Method = WebRequestMethods.Http.Get;
using (var response = (HttpWebResponse) req.GetResponse())
{
using (var stream = response.GetResponseStream())
{
if (stream == null)
throw new NullReferenceException("Response stream is nulL!");
using (var reader = new StreamReader(stream, Encoding.Default))
{
Console.WriteLine("Content length = {0}", response.ContentLength);
return WebUtility.HtmlDecode(reader.ReadToEnd());
}
}
}
}
}
fiddler shows following output:
HTTP/1.1 200 OK
Date: Wed, 02 Sep 2015 20:36:42 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Cache-Control: private, max-age=0
Cf-Railgun: fe8c0e42fd 44.42 0.042796 0030 3350
X-Frame-Options: SAMEORIGIN
X-Request-Guid: bc3ccb1f-1de5-4375-b30d-f3c89134cf86
Server: cloudflare-nginx
CF-RAY: 21fc0233f6652bca-AMS
Content-Length: 251540
But in program I get this:
how can it be fixed?
A couple of things:
When I try, stackoverflow.com does not set a Content-Length header, so it will come through as -1.
But, changing the URL to use a server which definitely does set a Content-Length header, e.g., www.theguardian.com, still produces the same result: -1.
I think it's your use of AutomaticDecompression on the HttpWebRequest object.
If you don't set that property the ContentLength property comes through correctly.
This means that it is Fiddler that is returning the content-length. The reason that Fiddler would do that is that you have the 'Stream' button pressed (or some otherway to indicate to fiddler you want to return 'Stream' or 'Chunked' data.
When a server returns 'Chunked' data, it sets the content-length to -1 -- you don't know the length of the content since you can have unlimited chunks.
If you unclick (or turn off) streaming, then fiddler returns its default 'buffered' response, which is an exact copy of the response from the server. Which of course will include that Content-Length header.
Literally just went through this -- HTH!
Related
I have a worker role on Azure. Every two hours, i download products from 3 different affiliate company.
First time, when the job runs, it's works perfect. The next time the job should run, I get from one affiliate, "The request was aborted: The request was canceled." The 2 others works perfect.
It's not just once, every time. I have ask this affiliate company, but there is no problem. So it must be my code. I have this to download json doc:
using (var Client = new WebClient())
{
Client.Headers.Add("X-API-KEY", Key);
Data = Client.DownloadString(URL);
}
What have I missed?
UPDATE 1:
i have try this:
HttpWebRequest Req = (HttpWebRequest)WebRequest.Create(URL);
Req.KeepAlive = false;
Req.Headers.Add("X-API-KEY", Key);
Req.Method = "GET";
using (var Resp = Req.GetResponse())
{
using (var Reader = new StreamReader(Resp.GetResponseStream()))
{
Data = Reader.ReadToEnd();
}
}
Same problem.
UPDATE 2
Request 1
GET https://se.#####.com/1/stores.json HTTP/1.1
X-API-KEY: x.............N
Host: se.#####.com
Connection: Close
Response 1
HTTP/1.1 200
Server: nginx
Date: Mon, 08 Jun 2015 14:56:39 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: close
Set-Cookie: ci_session=..............; expires=Mon, 08-Jun-2015 16:56:39 GMT; Max-Age=7200; path=/
e30
{"status":true,"data":[................]}
0
Request 2
GET https://se.#####.com/1/stores.json HTTP/1.1
X-API-KEY: x.............N
Host: se.#####.com
Connection: Close
Response 2
HTTP/1.1 200
Server: nginx
Date: Mon, 08 Jun 2015 15:06:29 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: close
Set-Cookie: ci_session=..................; expires=Mon, 08-Jun-2015 17:06:29 GMT; Max-Age=7200; path=/
e30
{"status":true,"data":[.....................]}
0
UPDATE 3
[TestMethod]
public void DownloadTest()
{
Test();
Test();
Test();
}
private static void Test()
{
const string merchantsUrl = "https://se.#####.com/1/stores.json";
string Data;
var Req = (HttpWebRequest)WebRequest.Create(merchantsUrl);
Req.KeepAlive = false;
Req.Headers.Add("X-API-KEY", ".....");
Req.Method = "GET";
using (var Resp = Req.GetResponse())
{
using (var Reader = new StreamReader(Resp.GetResponseStream()))
{
Data = Reader.ReadToEnd();
}
}
}
Change the order of requests and see if the request to same company is failing?
By default WebClient opens a connection as KeepAlive which might be causing this issue. So try explicitly mentioning that KeepAlive mode is false.
Capture the request details for the first time and second time in fiddler or similar tools which could give you more details in tracking down the issue.
Hi I'm trying to send the following JSON string to AppEngine server. The string looks as following:
{"param2":50.0,"param1":50.0,"additionalParams":{"param3":"123","userID":"1234561"}}
And the code that I use for sending it is below:
public async Task<string> SendJSONData(string urlToCall, string JSONData)
{
// server to POST to
string url = urlToCall;
// HTTP web request
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.ContentType = "action";
httpWebRequest.Method = "POST";
// Write the request Asynchronously
using (var stream = await Task.Factory.FromAsync<Stream>(httpWebRequest.BeginGetRequestStream,
httpWebRequest.EndGetRequestStream, null))
{
//create some json string
string json = JSONData;
// convert json to byte array
byte[] jsonAsBytes = Encoding.UTF8.GetBytes(json);
// Write the bytes to the stream
await stream.WriteAsync(jsonAsBytes, 0, jsonAsBytes.Length);
}
WebResponse response = await httpWebRequest.GetResponseAsync();
StreamReader requestReader = new StreamReader(response.GetResponseStream());
String webResponse = requestReader.ReadToEnd();
return webResponse;
}
I've sniffed what is being sent to the server, using Fiddler:
POST http://server.appspot.com/method HTTP/1.1
Accept: */*
Content-Length: 85
Accept-Encoding: identity
Content-Type: action
User-Agent: NativeHost
Host: server.appspot.com
Connection: Keep-Alive
Cache-Control: no-cache
Pragma: no-cache
{"param2":50.0,"param1":50.0,"additionalParams":{"param3":"123","userID":"1234561"}}
Please mind that I've expeimented with "Content-Type" parameter, setting it to both "text/plain" and "application/json".
Still the answer from the server looks like this:
HTTP/1.1 500 Internal Server Error
Date: Wed, 20 Feb 2013 18:54:34 GMT
Content-Type: text/html; charset=UTF-8
Server: Google Frontend
Content-Length: 466
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>500 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered an error and could not complete your request.<p>If the problem persists, please report your problem and mention this error message and the query that caused it.</h2>
What should I do, to receive the desired "OK" response?
Ok so the problem was the lack of "action" parameter in my POST.
Workaround looks like this:
// Write the request Asynchronously
using (var stream = await Task.Factory.FromAsync<Stream>
(httpWebRequest.BeginGetRequestStream,httpWebRequest.EndGetRequestStream, null))
{
//create some json string
string json = "action="+JSONData;
// convert json to byte array
byte[] jsonAsBytes = Encoding.UTF8.GetBytes(json);
// Write the bytes to the stream
await stream.WriteAsync(jsonAsBytes, 0, jsonAsBytes.Length);
}
Recently I've been experiencing a nightmare with .NET (C#) and SOAP Transmissions.
I've got to consume a webservice (which was supposed to be an easy task) but it become terrible and nothing seem to works.
HttpWebRequest webRequest = (HttpWebRequest)System.Net.WebRequest.Create("http://api.myapi.com/apis/services/theapi");
webRequest.AllowAutoRedirect = true;
webRequest.Timeout = 1000 * 30;
webRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
webRequest.PreAuthenticate = true;
webRequest.Method = "POST";
webRequest.Headers.Add("SOAPAction: \"\"");
webRequest.Accept = "text/xml";
WebResponse webResponse = null;
try
{
webResponse = webRequest.GetResponse();
Stream Stream = webRequest.GetRequestStream();
string SoapEnvelope = "<soap:Envelope>...SOAP CODE ...</soap:Envelope>";
StreamReader streamReader = new StreamReader(webResponse.GetResponseStream(), System.Text.Encoding.UTF8);
XmlDocument SoapEnvelopeXML = new XmlDocument();
SoapEnvelopeXML.LoadXml(SoapEnvelope);
SoapEnvelopeXML.Save(Stream);
string result = streamReader.ReadToEnd();
return result;
}
When I try to sniff the packages by using Wireshark, that's what I get:
---- CLIENT INPUT ------
POST /apis/services/theapi HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
SOAPAction: ""
Accept: text/xml
Host: api.myapi.com
Connection: Keep-Alive
---- SERVER ANSWER ------
HTTP/1.1 500 Internal Server Error
Date: Sat, 14 May 2011 15:35:32 GMT
X-Powered-By: Servlet 2.4; JBoss-4.0.5.GA (build: CVSTag=Branch_4_0 date=200610162339)/Tomcat-5.5
Content-Type: text/xml;charset=ISO-8859-1
Content-Length: 225
Connection: close
X-Pad: avoid browser bug
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><soap:Fault><faultcode>soap:Client</faultcode><faultstring>Error reading XMLStreamReader.</faultstring></soap:Fault></soap:Body></soap:Envelope>
As expected, since I haven't posted a Soap Request (no XML in the request), I receive a SOAP Fault and ERROR 500.
Any ideas?
I've got to somehow do this manually. I've tried to use even TCPClient (to deal with it in a lower level), but all my attempts were frustrated.
You should use the VS Add Service Reference wizard to load the service into your project. The Add Service Reference generates classes to work with the api automatically at a much higher level, from the url of the service API endpoint. It will look something like this:
MyApiClient client = new MyApiClient();
MyApiResult result;
try {
client.Open();
result = client.CallMethod(param1, param2, ...);
client.Close();
} catch (Exception ex) {
// do something with FaultException or API error
}
// do something with the result returned, if needed
If you've done it correct, you shouldn't have to deal with HttpWebRequest, explicit URL's, or hand-typing out any SOAP XML at all!!
I have problems trying to use custom basic authentication module similar to this. The client uses HttpWebRequest class.
The client runs the following code:
void uploadFile( string serverUrl, string filePath )
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.
Create( serverUrl );
CredentialCache cache = new CredentialCache();
cache.Add( new Uri( serverUrl ), "Basic", new NetworkCredential( "User", "pass" ) );
request.Credentials = cache;
request.Method = "POST";
request.ContentType = "application/octet-stream";
request.Timeout = 60000;
request.KeepAlive = true;
using( BinaryReader reader = new BinaryReader(
File.OpenRead( filePath ) ) ) {
request.ContentLength = reader.BaseStream.Length;
using( Stream stream = request.GetRequestStream() ) {
byte[] buffer = new byte[1024];
while( true ) {
int bytesRead = reader.Read( buffer, 0, buffer.Length );
if( bytesRead == 0 ) {
break;
}
stream.Write( buffer, 0, bytesRead );
}
}
}
HttpWebResponse result = (HttpWebResponse)request.GetResponse();
//handle result - not relevant
}
If the request is created for a URI starting with http:// it works okay - a request reaches the server, the authentication module is passed the request, it replies with WWW-Authenticate, the request is repeated now with authentication parameters, the module validates it and it passes further.
If the request is created for a URI starting with https:// it doesn't work. The initial request gets to the module and the module replies with WWW-Authenticate
void ReplyWithAuthHeader()
{
HttpContext currentContext = HttpContext.Current;
context.Response.StatusCode = 401;
context.Response.AddHeader( "WWW-Authenticate",
String.Format("Basic realm=\"{0}\"", "myname.mycompany.com"));
}
an an exception is thrown at the client with "Unable to write data to the transport connection: An established connection was aborted by the software in your host machine." text.
I tried System.Net tracing and discovered that after sending the initial request the client gets back the following header:
Date: Fri, 04 Feb 2011 12:15:04 GMT
Server: Microsoft-IIS/5.1
X-Powered-By: ASP.NET
while when the URI started with http:// the client received the following:
Content-Length: 1894
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Fri, 04 Feb 2011 12:12:11 GMT
Server: Microsoft-IIS/5.1
WWW-Authenticate: Basic realm="myname.mycompany.com"
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
so clearly the WWW-Authenticate response is swallowed somewhere and doesn't reach the client.
Also if I exclude the code that writes the file data into the request it also authenticates okay.
How do I fix this? How do I make the WWW-Authenticate response get to the client?
What are the http status codes returned from the server?
Is the certificate of the server valid?
Can you post the stack trace?
Try to disable your firewall and other networking tools like fiddler etc on your client.
Also try with the KeepAlive property to false.
I need to make from my app an authentificated httpwebrequest. the response to my request should be in json format. for this i'm using the code below:
// Create the web request
Uri address = new Uri("http://www.mysite.com/remote/user/login/format/json");
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.UseDefaultCredentials = false;
request.Credentials = new NetworkCredential(UserName, Password);
request.PreAuthenticate = true;
request.ContentType = "application/x-www-form-urlencoded";
request.Accept = "application/json";
string data = string.Format("username={0}&password={1}", otherusername, otherpassword);
// Create a byte array of the data we want to send
byte[] byteData = UTF8Encoding.UTF8.GetBytes(data);
// Set the content length in the request headers
request.ContentLength = byteData.Length;
//Write data
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(byteData, 0, byteData.Length);
}
// Get response
try
{
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream
StreamReader reader = new StreamReader(response.GetResponseStream());
// Console application output
jsonResponse = reader.ReadToEnd();
reader.Close();
}
user = new User();
JObject o = JObject.Parse(jsonResponse);
user.Unguessable_id = (string)o["unguessable_id"];
user.Print_id = (string)o["print_id"];
user.Rrid = (string)o["rrid"];
user.Raid = (string)o["raid"];
}
catch (WebException ex) {
errorMessage = ex.Message;
}
the problem is that the very first call it always gives a 500 error on the server. and the request fails. if i redo the call(by making an refresh in my browser) the request is successful.
the request should look like this in normal conditions:
POST /remote/user/login/format/json HTTP/1.1
Host: <yourhost>
username=user&password=pass
but when the server sends out the 500 error he received something like this:
username=user&password=passwordPOST /remote/user/login/format/json HTTP/1.1
any idea why this is happening? in my test app if i refresh the page that makes the httpwebrequest the call is successful.
EDIT:
after installing Fiddler the requests made look like this:
=> the one that generates 500
POST http://www.mysite.com/remote/user/login/format/json HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Accept: application/json
Host: www.mysite.com
Content-Length: 30
Expect: 100-continue
Connection: Keep-Alive
username=user&password=pass
=> the one made on refresh
POST http://www.mysite.com/remote/user/login/format/json HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Accept: application/json
Authorization: Basic ZGNpOkFpR2g3YWVj
Host: www.mysite.com
Content-Length: 30
Expect: 100-continue
Connection: Keep-Alive
username=user&password=pass
it seems that Authorization: Basic ZGNpOkFpR2g3YWVj is not included in the first request...why is that happening?(i'm using the same code for both requests)
I would advice you to install Fiddler to see what's really happening
I needed to add:
request.Headers.Add("Authorization: Basic ZGNpOkFpR2g3YWVj");
weird though that for the second request it was added automatically..