I am trying to get the URL of a page using HttpClient. I've previously only used HttpWebRequest, but I need to make this an async method.
In the code below, myUri always returns null which results in throwing an exception when I try to handle it later on.
Is the location header the wrong thing to be using?
string myUrl = "http://www.example.com/";
Uri myUri= new Uri(myUrl);
using (HttpClient client = new HttpClient())
using (HttpResponseMessage response = await client.GetAsync(myUri))
{
if (response.IsSuccessStatusCode)
{
myUri= response.Headers.Location;
Debug.WriteLine("True "+ myUri);
}
else {
Debug.WriteLine("False " + myUri);
}
}
It's because HttpClient will automatically follows redirects. If you need the URL a page redirects to, you need to stop it from automatically following:
Change your code to the following:
string myUrl = "http://www.example.com/";
Uri myUri= new Uri(myUrl);
HttpClientHandler httpClientHandler = new HttpClientHandler();
httpClientHandler.AllowAutoRedirect = false;
using (HttpClient client = new HttpClient(httpClientHandler))
Here is an async way to resolve the final redirect URL:
public static async Task<string> ResolveFinalRedirectAsync(string url)
{
try
{
var req = WebRequest.CreateHttp(url);
req.AllowAutoRedirect = true;
var res = await req.GetResponseAsync();
return res.ResponseUri.AbsoluteUri;
}
catch
{
Console.WriteLine("Couldn't resolve '{0}'", url);
}
return null;
}
See #Rob's answer about AllowAutoRedirect.
Once you do that, note
The line
if (response.IsSuccessStatusCode)
evaluates to false if you receive a HTTP 301 redirect (anything outside of the 200-299 range)
A value that indicates if the HTTP response was successful. true if HttpStatusCode was in the Successful range (200-299); otherwise false.
(source)
OK thanks I'm trying to get redirected URLs anyway
If you prevent automatically following redirects, you will get an HTTP response in the 3xx range for the redirect. Your check for valid codes will have to be modified accordingly.
Related
I am trying to pass login creds from a WebView into an HttpWebRequest but not having any luck getting an authenticated response. I am able to successfully make the request, but the response is acting like I haven't logged in. My app has 5 WebViews contained within Fragment s and I'm logged in on all of them. I've tried using the CookieSyncManager but it's deprecated and .Sync() didn't work. I've tried a lot of different ways of passing the cookies into the HttpRequest with no success and many hours spent.
One would think this is a simple request; user has logged in within the app; they should be authenticated for all requests. Here's the closest that I've gotten, but the response string is still not the same as through my authenticated WebView :
This attempt parses each Cookie into a string and adds it
public string _cookieString { get; set; }
private class ExtWebViewClient : WebViewClient
{
TheFragment5 _fm5 = new TheFragment5();
public override void OnPageFinished(WebView view, string url)
{
var cookieHeader = Android.Webkit.CookieManager.Instance.GetCookie(url);
var cookiePairs = cookieHeader.Split('&');
_fm5._cookieString = "";
foreach (var cookiePair in cookiePairs)
{
var cookiePieces = cookiePair.Split('=');
if (cookiePieces[0].Contains(":"))
cookiePieces[0] = cookiePieces[0].Substring(0, cookiePieces[0].IndexOf(":"));
cookies.Add(new Cookie
{
Name = cookiePieces[0],
Value = cookiePieces[1]
});
}
foreach (Cookie c in cookies)
{
if (_fm5._cookieString == "")
{
_fm5._cookieString = c.ToString();
}
else
{
_fm5._cookieString += c.ToString();
}
}
}
}
I've also tried just doing:
_fm5._cookieString = cookieHeader.ToString();
but neither of those attempts is working when I add the cookie string into my HttpRequest :
public async void GetNotificationText(string url)
{
//var _cmhc = _cookieMan.HasCookies;
await Task.Run(() =>
{
_notificationHttpRequestInProgress = true;
try
{
var _ctxxx = Android.App.Application.Context;
//URL _url2 = new URL("https://bitchute.com/notifications/");
//HttpURLConnection conn = (HttpURLConnection)_url2.OpenConnection();
//conn.ReadTimeout = 10000 /* milliseconds */;
//conn.ConnectTimeout = 15000 /* milliseconds */;
////conn.SetRequestProperty("Cookie", cookies);
//conn.Connect();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
Uri uri = new Uri(url);
var _req = request;
var _uriii = uri;
var _cookiesss = _fm5._cookieString;
_cookieCon.SetCookies(uri, _cookiesss);
request.CookieContainer = _cookieCon;
//request.CookieContainer.SetCookies(uri, _cookiesss);
request.AutomaticDecompression = DecompressionMethods.GZip;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream))
{
_notificationRawText = reader.ReadToEnd();
Console.WriteLine(_notificationRawText);
_rawNoteText = _notificationRawText;
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
_notificationHttpRequestInProgress = false;
});
}
This returns, but not the authenticated webtext request; I get the same response any user would get on a browser having never logged in. If I were to browse out to this same url on any WebView in my app, I'd get a completely different response.
You will also notice some commented out code that was another failed attempt at adding the cookies into a connection. I had also tried using HttpURLConnection.SetRequestProperty("Cookie", cookies);
where cookies was a CookieCollection and that didn't work either. The code is mostly commented out and layered because I've been trying this for days.
Does anyone know how I can pass WebView cookies into an HttpRequest using Xamarin.Android?
I am putting this code below in Fragment5 of my app; you can see and compile the full context here:
https://github.com/hexag0d/BitChute_Mobile_Android_BottomNav/blob/NotificationAdder/Fragments/TheFragment5.cs
I'm not sure exactly why the above example didn't work; maybe if you're better at .NET than I am, you could figure it out. However, I was able to successfully pass WebView creds into an HttpClient by following these steps, which are returning an authenticated response. This may not be the most elegant way of doing it, but you can always refine my answer, or post a better one.
What I had to do was set the HttpClient.DefaultRequestHeaders using the .Add() method like this: _client.DefaultRequestHeaders.Add("Cookie", TheFragment5._cookieHeader);
I got the CookieHeader (which is just a string btw) like this:
//instantiate a string that will house our cookie header
public static string _cookieHeader;
//you might want to make it private to prevent abuse
//but this example is just for demonstration
//the thing is we need a string to house our headers in scope of both the WebView and the HttpClient
//extend the WebViewClient
private class ExtWebViewClient : WebViewClient
{
public override void OnPageFinished(WebView view, string url)
{
//I get the cookies when the page finishes loading because
//then we know the cookie has our login cred header
//also, most of the previous examples got the cookies OnPageFinished
TheFragment5._cookieHeader = Android.Webkit.CookieManager.Instance.GetCookie(url);
}
}
Then we need another method for the HttpClient and HttpClientHandler ... mine scans a webpage for notification text.
public async void GetNotificationText(string url)
{
await Task.Run(() =>
{
/* this line is pretty important,
we need to instantiate an HttpClientHandler
then set it's UseCookies property to false
so that it doesn't override our cookies
*/
HttpClientHandler handler = new HttpClientHandler() { UseCookies = false };
try
{
Uri _notificationURI = new Uri("https://bitchute.com/notifications/");
//instantiate HttpClient using the handler
using (HttpClient _client = new HttpClient(handler))
{
//this line is where the magic happens;
//we set the DefaultRequestHeaders with the cookieheader we got from WebViewClient.OnPageFinished
_client.DefaultRequestHeaders.Add("Cookie", TheFragment5._cookieHeader);
//do a GetAsync request with our cookied up client
var getRequest = _client.GetAsync("https://bitchute.com/notifications/").Result;
//resultContent is the authenticated html string response from the server, ready to be parsed =]
var resultContent = getRequest.Content.ReadAsStringAsync().Result;
/*
I was writing to console to check the
response.. for me, I am now getting
the authenticated notification html
page
*/
Console.WriteLine(resultContent);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
Hope this helps you, posting for future reference, especially for people using Xamarin.Android.
I need to call two odata URL's of sap from C# Code.
The first URL will set the Filters and the second URL (generic URL) will give the actual response.
First URL:
https://<>/sap/opu/odata/SALM/TM_SERVICE/setFilters?sap-language=EN&filter='2%2Ctime%3D%E2%80%8E4%E2%80%8E%2F%E2%80%8E1%E2%80%8E%2F%E2%80%8E2019%E2%80%8E%20%E2%80%8E10%E2%80%8E%3A%E2%80%8E31%E2%80%8E%3A%E2%80%8E24%E2%80%8E%20%E2%80%8EAMc%3ATest%20Suite%20for%20Focused%20Build%2Cw%3A%2CT%3D<>'
Second URL :
https://<>//sap/opu/odata/SALM/TM_SERVICE/TILECollection(ConfigID='1%20',PageID='2%20',TileID='8%20')?$format=json&sap-language=EN
Second URL is generic and constant URL but after first execution of first URL with different parameter values second URL will give different response.
In a browser I am able to execute these two URL's and see the response difference with different parameter values.
Need C# Code to simulate the browser behavior. In C# Code second URL response is constant and not being changed.
Tried using following : WebRequest,HttpWebRequest, HttpClient,WebClient along with passing cookie ( cookie received from First response is passed to Request while executing the second url )
var httpClientHandler = new HttpClientHandler()
{
Credentials = credential,
CookieContainer = cookieContainer
};
using (HttpClient client = new HttpClient(httpClientHandler))
{
using (HttpResponseMessage response = client.GetAsync("<<first url>>").Result)
{
Uri uri = new Uri(string.Format(ConfigurationManager.AppSettings["TestPlanSetFilters"], s));
IEnumerable<Cookie> responseCookies = cookieContainer.GetCookies(uri).Cast<Cookie>();
foreach (Cookie cookie in responseCookies)
{
cookieContainer.Add(cookie);
}
}
using (HttpResponseMessage response1 = client.GetAsync("<<second url>>").Result)
{
using (HttpContent content1 = response1.Content)
{
var json2 = content1.ReadAsStringAsync().Result;
}
}
}
I am trying to write call a web page that then posts to a web service that outputs a JSON file.
The problem I have is that the GetAsync returns a null value for response. This in turn doesn't provide the proper URL for call back for the GetTestResultAsync method.
Here's my code:
static HttpClient client = new HttpClient();
static async Task RunAsync()
{
// New code:
client.BaseAddress = new Uri("http://10.1.10.10:8080/");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
try
{
Uri url = await CallTestAsync();
string response = await GetTestResultAsync(url.PathAndQuery);
Console.WriteLine(response);
}
catch (Exception e)
{
Console.WriteLine(e);
}
}
static async Task<Uri> CallTestAsync()
{
HttpResponseMessage response = await client.GetAsync("test.html");
response.EnsureSuccessStatusCode();
// return URI of the created resource.
return response.Headers.Location;
}
static async Task<string> GetTestResultAsync(string path)
{
HttpResponseMessage response = await client.GetAsync(path);
string streamResponse = string.Empty;
if (response.IsSuccessStatusCode)
{
streamResponse = await response.Content.ReadAsStringAsync();
}
return streamResponse;
}
static void Main(string[] args)
{
RunAsync().Wait();
}
HttpClient by default will automatically redirect 3xx responses, which means that the response you get when calling GetAsync will not have the Location header as it would have already redirected to the proper location.
To override this behaviour you have to provide an HttpMessageHandler with this feature disabled. For example:
static HttpClientHandler handler = new HttpClientHandler { AllowAutoRedirect = false };
static HttpClient client = new HttpClient(handler);
The important part here being setting the handler's AllowAutoRedirect to false.
Important: By overriding the default behaviour for redirect responses you'll have to handle any 3xx manually, which might add unnecessary work for you as in many cases the default behaviour is sufficient. If you were to leave it as is, it'd already make the 2nd request that you're doing, for you.
Also note that a 3xx response is not a success response, which means that if you don't use the auto-redirect feature when you call response.EnsureSuccessStatusCode(); it'll throw an exception.
Furthermore, although most servers are quite forgiving when it comes to headers such as the Accept header, you are most likely using the wrong one in this case as an HTML page should be a text/html rather than an application/json (which should be used when expecting a JSON object as a response).
Issue: I am trying to use httpclient for fetching data from a site.Now the site requires you to first visit a link then only you can post data to the next link.
Link1 is a simple get request
Link2 is a post request
Now I think the site first store some cookie from the link1 and then only allow you to post data to link2 as whenever I try to open the link2 in incognito the site displays the error message "Session Timed out OR Maximum connections limit reached. Cannot Proceed Further. Please close and restart your browser "
Now I have tried this:
try
{
//Send the GET request
httpResponse = await httpClient.GetAsync(new Uri(link1UriString));
//Send the POSTrequest
httpResponse = await httpClient.PostAsync(new Uri(link2uriString),postContent);
httpResponseBody = await httpResponse.Content.ReadAsStringAsync();
}
But I am getting the session timed out error message. How to maintain cookies for a session in httpClient continuously received from the web.Like in python it can be done by
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookielib.CookieJar()))
urllib2.install_opener(opener)
Link1
Link2
You can use a CookieContainer to handle cookies for you.
Doing so, you'd create the HttpClient like this.
using System.Net;
using System.Net.Http;
CookieContainer cookies = new CookieContainer();
HttpClientHandler handler = new HttpClientHandler();
handler.CookieContainer = cookies;
HttpClient httpClient = new HttpClient(handler);
httpResponse = await httpClient.GetAsync(new Uri(link1UriString));
(Note it uses the version of HttpClient in System.Net.Http)
So, after first response you have Set-Cookie header:
var responseMessage = await httpClient.GetAsync("http://115.248.50.60/registration/Main.jsp?wispId=1&nasId=00:15:17:c8:09:b1");
IEnumerable<string> values;
var coockieHeader = string.Empty;
if (responseMessage.Headers.TryGetValues("set-cookie", out values))
{
coockieHeader = string.Join(string.Empty, values);
}
After that, just setup your cookie into request message:
var httpRequestMessage = new HttpRequestMessage
{
RequestUri = new Uri("http://115.248.50.60/registration/chooseAuth.do"),
Content = postContent,
Method = HttpMethod.Post
};
httpRequestMessage.Headers.Add("Cookie", values);
var httpResponse = await httpClient.SendAsync(httpRequestMessage);
I'm integrating a service that returns a key when I a GET request to a URL that is in the following format:
https://username:password#service.com/refresh.key
When I access the URL in my browser, it returns the new key as expected, by when I do a GET request using HttpClient I get a 401.
HttpClient _client = new HttpClient();
var response = await _client.GetAsync(#"https://username:password#service.com/refresh.key"); // Returns a 401
I think it has something to do with the '#' in the URL, but I'm not sure how to fix it, I tried replacing it with '%40', but when I do that I get a UriFormatException.
Does anyone know how to do this?
You should modify Authorization header of HttpClient, can you try the code below;
HttpClient _client = new HttpClient();
byte[] usernamePasswordBytes = Encoding.ASCII.GetBytes("user:pass");
_client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", Convert.ToBase64String(usernamePasswordBytes));
var response = await _client.GetAsync(#"https://service.com/refresh.key");
PS: Such username:pass#domain.com requests are BasicAuthentication request so in fact you try to make basic authentication request.
Hope this works for you
You don't need to provide credentials in url. Instead you can do:
using (var handler = new HttpClientHandler {Credentials = new NetworkCredential("username", "password")}) {
using (HttpClient _client = new HttpClient(handler)) {
var response = await _client.GetAsync(#"https://service.com/refresh.key");
}
}