Develop Reference

Passing request headers to HttpWebRequest in C#

I am trying to make a HttpWebRequest and get json data from it. Using Postman, when I set url, parameters & headers, I am able to get json response (please see this). But, using C# when I try, I do not get anything in response.
I searched a couple of posts on Stackoverflow and followed the steps, but cannot find what is wrong or if anything else is required.
string requestUrl = Constants.FLIPKART_INSTALLS_URL;
requestUrl = requestUrl.Replace("##STARTDATE##", DateTime.Now.ToString("yyyy-MM-dd"));
requestUrl = requestUrl.Replace("##ENDDATE##", DateTime.Now.ToString("yyyy-MM-dd"));
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
//request.Credentials = CredentialCache.DefaultCredentials;
request.Method = "GET";
//request.ContentType = "application/json";
request.Headers["Fk-Affiliate-id"] = Constants.FLIPKART_AFFILIATE_ID;
request.Headers["Fk-Affiliate-token"] = Constants.FLIPKART_AFFILIATE_TOKEN;
WebResponse response = request.GetResponse();
This is the response that I get. I am not sure if this is really a silly question, but since I don't have much knowledge of C#, therefore I posted it.
Thanks in advance.

By calling GetResponseStream() on the response object, I was able to read the contents, refer below code if needed:
using System;
using System.Net;
using System.Text;
using System.IO;
public class Test
{
// Specify the URL to receive the request.
public static void Main (string[] args)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create (args[0]);
// Set some reasonable limits on resources used by this request
request.MaximumAutomaticRedirections = 4;
request.MaximumResponseHeadersLength = 4;
// Set credentials to use for this request.
request.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse ();
Console.WriteLine ("Content length is {0}", response.ContentLength);
Console.WriteLine ("Content type is {0}", response.ContentType);
// Get the stream associated with the response.
Stream receiveStream = response.GetResponseStream ();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader (receiveStream, Encoding.UTF8);
Console.WriteLine ("Response stream received.");
Console.WriteLine (readStream.ReadToEnd ());
response.Close ();
readStream.Close ();
}
}

WindowsPhone HttpWebRequest to ashx WebService

I have a problem with HttpWebRequest:
I have an ashx code in a visual studio ws that simply does:
public void ProcessRequest(HttpContext context)
{
var a = context.Request.Form["a"];
var b = context.Request.Form["b"];
context.Response.Write(a + " " + b);
}
I tried to call it with advancedRestClient and it worked, but if i call it with my windows phone device I get a not found exception and the request doesn't reach any breakpoint.
this is my WP code:
public void PostIt2()
{
string url = "http://localhost/blablacode/ecc.ashx";
// Create a new HttpWebRequest object.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.ContentType = "application/x-www-form-urlencoded";
// Set the Method property to 'POST' to post data to the URI.
request.Method = "POST";
// start the asynchronous operation
request.BeginGetRequestStream(new AsyncCallback(GetRequestStreamCallback), request);
// Keep the main thread from continuing while the asynchronous
// operation completes. A real world application
// could do something useful such as updating its user interface.
allDone.WaitOne();
}
private static void GetRequestStreamCallback(IAsyncResult asynchronousResult)
{
HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
// End the operation
Stream postStream = request.EndGetRequestStream(asynchronousResult);
string postData = "{'a': 'avar','b':'bvar'}";
// Convert the string into a byte array.
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
// Write to the request stream.
postStream.Write(byteArray, 0, postData.Length);
postStream.Close();
// Start the asynchronous operation to get the response
request.BeginGetResponse(new AsyncCallback(GetResponseCallback), request);
}
private static void GetResponseCallback(IAsyncResult asynchronousResult)
{
HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
// End the operation
HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(asynchronousResult);
Stream streamResponse = response.GetResponseStream();
StreamReader streamRead = new StreamReader(streamResponse);
string responseString = streamRead.ReadToEnd();
Console.WriteLine(responseString);
// Close the stream object
streamResponse.Close();
streamRead.Close();
// Release the HttpWebResponse
response.Close();
allDone.Set();
}
I took the code from MSDN so i don't know what i'm doing wrong, i also tried a lot of methods found on the internet, the only thing important for me is that i have to use HttpWebRequest and not HttpClient.
Can someone help me please?

Its possible that the problem is not in the code.
Are your services accesible from the phone??
Are your proyect configure with internet access??
If all its ok, look at this post:
httpclient postasync windows phone 8.1 universal app second call 404 error

Ok, really thanks god is friday.
i was using as url "localhost/eccecc.ashx"
device's localhost is not same localhost of computer.

Getting the Redirected URLs from the Original URL [duplicate]

Using the WebClient class I can get the title of a website easily enough:
WebClient x = new WebClient();
string source = x.DownloadString(s);
string title = Regex.Match(source,
#"\<title\b[^>]*\>\s*(?<Title>[\s\S]*?)\</title\>",
RegexOptions.IgnoreCase).Groups["Title"].Value;
I want to store the URL and the page title. However when following a link such as:
http://tinyurl.com/dbysxp
I'm clearly going to want to get the Url I'm redirected to.
QUESTIONS
Is there a way to do this using the WebClient class?
How would I do it using HttpResponse and HttpRequest?

If I understand the question, it's much easier than people are saying - if you want to let WebClient do all the nuts and bolts of the request (including the redirection), but then get the actual response URI at the end, you can subclass WebClient like this:
class MyWebClient : WebClient
{
Uri _responseUri;
public Uri ResponseUri
{
get { return _responseUri; }
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse response = base.GetWebResponse(request);
_responseUri = response.ResponseUri;
return response;
}
}
Just use MyWebClient everywhere you would have used WebClient. After you've made whatever WebClient call you needed to do, then you can just use ResponseUri to get the actual redirected URI. You'd need to add a similar override for GetWebResponse(WebRequest request, IAsyncResult result) too, if you were using the async stuff.

I know this is already an answered question, but this works pretty to me:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://tinyurl.com/dbysxp");
request.AllowAutoRedirect = false;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
string redirUrl = response.Headers["Location"];
response.Close();
//Show the redirected url
MessageBox.Show("You're being redirected to: "+redirUrl);
Cheers.! ;)

With an HttpWebRequest, you would set the AllowAutoRedirect property to false. When this happens, any response with a status code between 300-399 will not be automatically redirected.
You can then get the new url from the response headers and then create a new HttpWebRequest instance to the new url.
With the WebClient class, I doubt you can change it out-of-the-box so that it does not allow redirects. What you could do is derive a class from the WebClient class and then override the GetWebRequest and the GetWebResponse methods to alter the WebRequest/WebResponse instances that the base implementation returns; if it is an HttpWebRequest, then set the AllowAutoRedirect property to false. On the response, if the status code is in the range of 300-399, then issue a new request.
However, I don't know that you can issue a new request from within the GetWebRequest/GetWebResponse methods, so it might be better to just have a loop that executes with HttpWebRequest/HttpWebResponse until all the redirects are followed.

I got the Uri for the redirected page and the page contents.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(strUrl);
request.AllowAutoRedirect = true;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
strLastRedirect = response.ResponseUri.ToString();
StreamReader reader = new StreamReader(dataStream);
string strResponse = reader.ReadToEnd();
response.Close();

In case you are only interested in the redirect URI you can use this code:
public static string GetRedirectUrl(string url)
{
HttpWebRequest request = (HttpWebRequest) HttpWebRequest.Create(url);
request.AllowAutoRedirect = false;
using (HttpWebResponse response = HttpWebResponse)request.GetResponse())
{
return response.Headers["Location"];
}
}
The method will return
null - in case of no redirect
a relative url - in case of a redirect
Please note: The using statement (or a final response.close()) is essential. See MSDN Library for details. Otherwise you may run out of connections or get a timeout when executing this code multiple times.

HttpWebRequest.AllowAutoRedirect can be set to false. Then you'd have to manually http status codes in the 300 range.
// Create a new HttpWebRequest Object to the mentioned URL.
HttpWebRequest myHttpWebRequest=(HttpWebRequest)WebRequest.Create("http://www.contoso.com");
myHttpWebRequest.MaximumAutomaticRedirections=1;
myHttpWebRequest.AllowAutoRedirect=true;
HttpWebResponse myHttpWebResponse=(HttpWebResponse)myHttpWebRequest.GetResponse();

The WebClient class has an option to follow redirects. Set that option and you should be fine.

Ok this is really hackish, but the key is to use the HttpWebRequest and then set the AllowAutoRedirect property to true.
Here's a VERY hacked together example
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://tinyurl.com/dbysxp");
req.Method = "GET";
req.AllowAutoRedirect = true;
WebResponse response = req.GetResponse();
response.GetResponseStream();
Stream responseStream = response.GetResponseStream();
// Content-Length header is not trustable, but makes a good hint.
// Responses longer than int size will throw an exception here!
int length = (int)response.ContentLength;
const int bufSizeMax = 65536; // max read buffer size conserves memory
const int bufSizeMin = 8192; // min size prevents numerous small reads
// Use Content-Length if between bufSizeMax and bufSizeMin
int bufSize = bufSizeMin;
if (length > bufSize)
bufSize = length > bufSizeMax ? bufSizeMax : length;
StringBuilder sb;
// Allocate buffer and StringBuilder for reading response
byte[] buf = new byte[bufSize];
sb = new StringBuilder(bufSize);
// Read response stream until end
while ((length = responseStream.Read(buf, 0, buf.Length)) != 0)
sb.Append(Encoding.UTF8.GetString(buf, 0, length));
string source = sb.ToString();string title = Regex.Match(source,
#"\<title\b[^>]*\>\s*(?<Title>[\s\S]*?)\</title\>",RegexOptions.IgnoreCase).Groups["Title"].Value;
enter code here

GetWebResponse very slow on windows7

I have tried all the settings, i found on internet to make the C# webclient fast on a Windows7 machine
to no avail. The same exe on Windows XP machine responds in less 100ms for every request.
I have overridden the GetWebRequest function in the derived class of
System.Web.Services.Protocols.SoapHttpClientProtocol
protected override WebRequest GetWebRequest(Uri address)
{
//ServicePointManager.UseNagleAlgorithm = false;
HttpWebRequest.DefaultWebProxy = null;
ServicePointManager.DefaultConnectionLimit = 4096;
ServicePointManager.Expect100Continue = false;
System.Net.ServicePointManager.CheckCertificateRevocationList = false;
HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);
request.Proxy = HttpWebRequest.DefaultWebProxy;
return (WebRequest)request;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
HttpWebResponse response = null;
//HttpWebRequest.DefaultWebProxy = null;
//ServicePointManager.Expect100Continue = false;
response = (HttpWebResponse)base.GetWebResponse(request);
return response;
}
I have also added the socket trace settings in the machine.config .Net4.0 folder
and it always waits on the below statement for 7-15 seconds in the output window
System.Net.Sockets Verbose: 0 : [6088] Socket#54042743::Receive()
Hope some champion has resolved this

Try using the GetResponse class and check if you find some difference:
HttpWebRequest wrequest = (HttpWebRequest) WebRequest.Create(objeto.Url);
//Get Response
HttpWebResponse wresp = (HttpWebResponse) wrequest.GetResponse();
//get Response Stream
StreamReader sr = new StreamReader(wresp.GetResponseStream());
//Get string content
string respuestastring = sr.ReadToEnd();

I've had the same issue and it turned out that I didn't close the web request, so when I fired a few in a row, I hit the maximum number of connections allowed by the host and it wouldn't continue until the first requests timed out. So try manually closing your requests or, better, wrap the variable declaration for your request into a using directive.

C# Console/Server access to web site

I am working on a C# project where I need to get data from a secured web site that does not have an API or web services. My plan is to login, get to the page I need, and parse out the HTML to get to the data bits I need to log to a database. Right now I'm testing with a console app, but eventually this will be converted to an Azure Service bus application.
In order to get to anything, you have to login at their login.cfm page, which means I need to load the username and password input controls on the page and click the submit button. Then navigate to the page I need to parse.
Since I don't have a 'browser' to parse for controls, I am trying to use various C# .NET classes to get to the page, set the username and password, and click submit, but nothing seems to work.
Any examples I can look at, or .NET classes I should be reviewing that were designed for this sort of project?
Thanks!

Use the WebClient class in System.Net
For persistence of session cookie you'll have to make a custom WebClient class.
#region webclient with cookies
public class WebClientX : WebClient
{
public CookieContainer cookies = new CookieContainer();
protected override WebRequest GetWebRequest(Uri location)
{
WebRequest req = base.GetWebRequest(location);
if (req is HttpWebRequest)
(req as HttpWebRequest).CookieContainer = cookies;
return req;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse res = base.GetWebResponse(request);
if (res is HttpWebResponse)
cookies.Add((res as HttpWebResponse).Cookies);
return res;
}
}
#endregion
Use a browser add-on like FireBug or the development tools built into Chrome to get the HTTP POST data being sent when you submit a form. Send those POSTs using the WebClientX class and parse the response HTML.
The fastest way to parse HTML when you already know the format is using a simple Regex.Match. So you'd go through the actions in your browser using the development tools to record your POSTs, URLs and HTML content then you'll perform the same tasks using the WebClientX.

Ok, so here is the complete Code to login to one page, then read from a 2nd page after the login.
class Program
{
static void Main(string[] args)
{
string uriString = "http://www.remotesite.com/login.cfm";
// Create a new WebClient instance.
WebClientX myWebClient = new WebClientX();
// Create a new NameValueCollection instance to hold some custom parameters to be posted to the URL.
NameValueCollection myNameValueCollection = new NameValueCollection();
// Add necessary parameter/value pairs to the name/value container.
myNameValueCollection.Add("userid", "myname");
myNameValueCollection.Add("mypassword", "mypassword");
Console.WriteLine("\nUploading to {0} ...", uriString);
// 'The Upload(String,NameValueCollection)' implicitly method sets HTTP POST as the request method.
byte[] responseArray = myWebClient.UploadValues(uriString, myNameValueCollection);
// Decode and display the response.
Console.WriteLine("\nResponse received was :\n{0}", Encoding.ASCII.GetString(responseArray));
Console.WriteLine("\n\n\n pausing...");
Console.ReadKey();
// Go to 2nd page on the site to get additional data
Stream myStream = myWebClient.OpenRead("https://www.remotesite.com/status_results.cfm?t=8&prog=d");
Console.WriteLine("\nDisplaying Data :\n");
StreamReader sr = new StreamReader(myStream);
StringBuilder sb = new StringBuilder();
using (StreamReader reader = new StreamReader(myStream, System.Text.Encoding.UTF8))
{
string line;
while ((line = reader.ReadLine()) != null)
{
sb.Append(line + "\r\n");
}
}
using (StreamWriter outfile = new StreamWriter(#"Logfile1.txt"))
{
outfile.Write(sb.ToString());
}
Console.WriteLine(sb.ToString());
Console.WriteLine("\n\n\n pausing...");
Console.ReadKey();
}
}
public class WebClientX : WebClient
{
public CookieContainer cookies = new CookieContainer();
protected override WebRequest GetWebRequest(Uri location)
// public override WebRequest GetWebRequest(Uri location)
{
WebRequest req = base.GetWebRequest(location);
if (req is HttpWebRequest)
(req as HttpWebRequest).CookieContainer = cookies;
return req;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse res = base.GetWebResponse(request);
if (res is HttpWebResponse)
cookies.Add((res as HttpWebResponse).Cookies);
return res;
}
}