There is this online form (https://servizi.ivass.it/RuirPubblica/) where you can make a search (just make a blank search). For each result it gives, I need to click on the result and export the list that is in the 5th table of the details page.
So basically I want to make a software that does that for me:
Submit a search with my own criteria
Access each page of the result items
Access each item detail page
Obtain the rows in the 5th tag so that I can append them to a list
Using Fiddler I checked which parameters where used in the POST request when I clicked the "Search" button, and tried to do the same with .Net.
If I try to access the base address with HttpClient it returns the correct HTML of the search form, but when I submit the following POST request with search parameters I get a web page showing the error "Warning: Session Expired".
This happens also if I make the search POST call alone, without first accessing the home page, so I'm not sure it is related to keeping the session alibe between two requests.
public MainWindow()
{
InitializeComponent();
var cookieJar = new CookieContainer();
var handler = new HttpClientHandler
{
CookieContainer = cookieJar,
UseCookies = true,
UseDefaultCredentials = false
};
client = new HttpClient(handler)
{
BaseAddress = new Uri("https://servizi.ivass.it/RuirPubblica/Search.faces")
};
}
private async Task TryHttp()
{
// Access the search page
var response = await client.GetAsync(client.BaseAddress);
var responseString = await response.Content.ReadAsStringAsync();
// Perform the search
var values = new Dictionary<string, string>
{
{ "FormSearch", "FormSearch" },
{ "FormSearch:j_id_jsp_558348152_13", "PG" },
{ "FormSearch:j_id_jsp_558348152_16", "custom" },
{ "FormSearch:SecE", "on" },
{ "FormSearch:matricola", "" },
{ "FormSearch:ragioneSociale", "" },
{ "FormSearch:provincia", "NA" },
{ "FormSearch:SearchButton", "Ricerca" },
{ "javax.faces.ViewState", "j_id1:j_id5" },
};
var content = new FormUrlEncodedContent(values);
response = await client.PostAsync(client.BaseAddress, content);
// Here I'm getting a web page showing the error "Warning: Session expired"
responseString = await response.Content.ReadAsStringAsync();
}
private void ButtonBase_OnClick(object sender, RoutedEventArgs e)
{
TryHttp();
}
If you can define it, it can be done. As you will understand from the comments StackOverflow is all about programming questions, so I will only help you with that.
In principal if the web page is "parsable" as HTML and communicates using HTTP you can do anything with it that a normal web browser would do. The website you reference does initially appear to do anything out of the ordinary.
HTMLAgilityPack can be very useful for parsing the DOM and navigating and extracting the contents.
To make HTTP requests with C# you should use the HttpClient class.
There are older options like the HttpWebClient, there is good answer here on SO to help you decide between the two.
For quick reference, Fiddler is available here, I too have used it many times and would recommended it, although it can cause problems with HTTPS traffic and debugging.
Related
I've an ASP.NET core 2.2 MVC app which exposes a Login page with a basic form (username/password). The controller action is protected by the AntiForgeryTokenAttribute, and the hidden __RequestVerificationToken is added by MVC.
I'm writing Integration Tests using TestServer and I want to send the form and see if I get 302 status code, but I couldn't find any valid option.
One option I evaluated was to do the GET, extract the __RequestVerificationToken and then submit the tpoken as part of the form. Howerer this won't work as I am missing the cookie (I believe). TestServer.CreateClient doesn't support any handler, so I can't add cookies.
Is there a way to test this ?
Thanks!
so there are two things needed:
during the page GET: get the cookie from the Headers and extract the __RequestVerification
when submitting the form: add the cookie in the Headers and add __RequestVerification to the model
1. GET
You can extract the token using:
headers.FirstOrDefault(x => x.Key == "Set-Cookie").Value.First().Split(" ")[0];
// The cookie we are looking for is .AspNetCore.Antiforgery.<unique guid>=<unique guid>
var tokenCookie = cookieContent.Split("=");
var name = tokenCookie[0];
var value = tokenCookie[1];
You can extract the __RequestVerification using Nuget package HtmlAgilityPack and then doing:
var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(htmlContent);
var tokenValue = htmlDoc.DocumentNode.SelectSingleNode("//input[#name='__RequestVerificationToken']")
.Attributes.Where(x => x.Name == "value").Select(x => x.Value).First();
where htmlContent is HttpResponse.Content.ReadAsStringAsync();
2. POST
When you create the form, add the __RequestVerificationToken:
new FormUrlEncodedContent(new List<KeyValuePair<string, string>>
{
... your stuff here
new KeyValuePair<string, string>("__RequestVerificationToken", token)
});
}
then when sending the request:
var request = new HttpRequestMessage(HttpMethod.Post, endPoint)
{ Content = form };
request.Headers.Add("Cookie", $"{cookie.Name}={cookie.Value}");
await client.SendAsync(request);
where client is HttpClient created using TestServer.CreateClient.
Hope this helps somebody else!
For a nicer way to get the required tokens and field names without resorting to scraping HTML from pages, I've written up a blog post and same sample code about how you can use Application Parts to achieve the same goal:
Blog post
Sample code
In the code I have, I am doing a POST request to a Slack API to create a channel for me. According to Slack's documentation, I could add users into "user_ids." Debugging wise I called another API called channels.invite and adding users is successful, so I don't know what's wrong with adding users through conversations.create
class CreateChannels
{
private static readonly HttpClient client = new HttpClient();
static async Task CreateChannel()
{
var values = new Dictionary<string, string>
{
{ "token", "SLACK TOKEN" },
{ "name", "test" },
{ "is_private", "true" },
{ "user_ids", "USER-ID"}
};
var content = new FormUrlEncodedContent(values);
var response = await client.PostAsync("https://slack.com/api/conversations.create", content);
var responseString = await response.Content.ReadAsStringAsync();
Console.WriteLine(responseString);
}
}
[1]: https://api.slack.com/methods/conversations.create
This does not work, because the parameter user_ids only works for so called workspace apps, but not for normal Slack like yours. You need to use conversations.invite.
From the documentation (emphasis mine):
Required for workspace apps. A list of between 1 and 30 human users
that will be added to the newly-created conversation. This argument
has no effect when used by classic Slack apps.
I am setting up a PHP API and a web-page based on client-side Blazor. But for some reason CORS is triggered and my login process or any requests to my PHP pages result in CORS errors.
I started out testing my PHP API with a C# console app and the Blazor app, I tried using without any database access to test the functionality. The Blazor is right now running with Preview 9. The PHP version is 5.3.8. I could in theory update it, but several other active projects are running on it and I do not have any test environment. MySQL version 5.5.24.
First I figured it might have been because I was running it on my local machine, so I pushed it to the website where the PHP and MySQL is also running. Still I run into this CORS error.
I am still just testing this, so I have tried setting it to allow any origin. I have not had any experience with CORS before this. Pretty sure I ought to be able to add PHP code in each file I access that should allow CORS, but since it should all be on the same website, I figure CORS should not even be relevant?
PHP Code:
function cors() {
// Allow from any origin
if (isset($_SERVER['HTTP_ORIGIN'])) {
// Decide if the origin in $_SERVER['HTTP_ORIGIN'] is one
// you want to allow, and if so:
header("Access-Control-Allow-Origin: {$_SERVER['HTTP_ORIGIN']}");
header('Access-Control-Allow-Credentials: true');
header('Access-Control-Max-Age: 86400'); // cache for 1 day
}
// Access-Control headers are received during OPTIONS requests
if ($_SERVER['REQUEST_METHOD'] == 'OPTIONS') {
if (isset($_SERVER['HTTP_ACCESS_CONTROL_REQUEST_METHOD']))
// may also be using PUT, PATCH, HEAD etc
header("Access-Control-Allow-Methods: GET, POST, OPTIONS");
if (isset($_SERVER['HTTP_ACCESS_CONTROL_REQUEST_HEADERS']))
header("Access-Control-Allow-Headers: {$_SERVER['HTTP_ACCESS_CONTROL_REQUEST_HEADERS']}");
exit(0);
}
echo "You have CORS!";
}
cors();
C# code using the injected HttpClient:
var resp = await Http.GetStringAsync(link);
The error I get is:
Access to fetch at 'https://titsam.dk/ntbusit/busitapi/requestLoginToken.php' from origin 'https://www.titsam.dk' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
The response I hoped to get was that the link I use return a token for the login as it does for my API.
Is it because its running client side maybe and this triggers CORS? But that does not seem to explain why I cannot make it allow all.
Update:
My C# code in OnInitializedAsync:
link = API_RequestLoginTokenEndPoint;
Http.DefaultRequestHeaders.Add("User-Agent", "HttpClientFactory-Sample");
Http.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Authorization", "basic:testuser:testpass");
var requestMessage = new HttpRequestMessage(HttpMethod.Get, link);
requestMessage.Properties[WebAssemblyHttpMessageHandler.FetchArgs] = new
{
credentials = "include"
};
var response = await Http.SendAsync(requestMessage);
var responseStatusCode = response.StatusCode;
var responseBody = await response.Content.ReadAsStringAsync();
output = responseBody + " " + responseStatusCode;
Update 2:
It finally works. The C# code I linked is the solution Agua From Mars suggested and it solved the problem to use SendAsync with a HttpRequestMessage and adding the Fetch property include credentials to it. Another alternative was to add this line to the startup:
WebAssemblyHttpMessageHandler.DefaultCredentials = FetchCredentialsOption.Include;
Then I could keep doing what I did to begin with, using GetStringAsync as it becomes the default.
await Http.GetStringAsync(API_RequestLoginTokenEndPoint);
So all the solutions Agua From Mars suggested worked. But I encountered a browser problem, where it kept the CORS issue in the cache somehow even after it had gotten solved, so it seemed like nothing had changed. Some code changes would show a different result, but I guess the CORS part was kept alive. With Chrome it helped opening a new pane or window. In my Opera browser this was not enough, I had to close all panes with the site open to ensure it would clear the cache and then opening a new window or pane with the site works in Opera as well. I had already in both browsers trying to use ctrl-F5 and Shift-F5 to get them to clear the cache. This did not change anything.
I hope this will help others avoid spending 2-3 days on an issue like this.
update 3.1-preview3
In 3.1-preview3, we cannot use the fetch option per message, the options is global
WebAssemblyHttpMessageHandlerOptions.DefaultCredentials = FetchCredentialsOption.Include;
WebAssemblyHttpMessageHandler has been removed. The HttpMessageHanlder used is WebAssembly.Net.Http.HttpClient.WasmHttpMessageHandler from WebAssembly.Net.Http but don't include WebAssembly.Net.Http in your depencies or the application will failled to launch.
If you want to use the HttpClientFactory you can implement like that :
public class CustomDelegationHandler : DelegatingHandler
{
private readonly IUserStore _userStore;
private readonly HttpMessageHandler _innerHanler;
private readonly MethodInfo _method;
public CustomDelegationHandler(IUserStore userStore, HttpMessageHandler innerHanler)
{
_userStore = userStore ?? throw new ArgumentNullException(nameof(userStore));
_innerHanler = innerHanler ?? throw new ArgumentNullException(nameof(innerHanler));
var type = innerHanler.GetType();
_method = type.GetMethod("SendAsync", BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.InvokeMethod) ?? throw new InvalidOperationException("Cannot get SendAsync method");
WebAssemblyHttpMessageHandlerOptions.DefaultCredentials = FetchCredentialsOption.Include;
}
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
request.Headers.Authorization = new AuthenticationHeaderValue(_userStore.AuthenticationScheme, _userStore.AccessToken);
return _method.Invoke(_innerHanler, new object[] { request, cancellationToken }) as Task<HttpResponseMessage>;
}
}
public void ConfigureServices(IServiceCollection services)
{
services.AddTransient(p =>
{
var wasmHttpMessageHandlerType = Assembly.Load("WebAssembly.Net.Http")
.GetType("WebAssembly.Net.Http.HttpClient.WasmHttpMessageHandler");
var constructor = wasmHttpMessageHandlerType.GetConstructor(Array.Empty<Type>());
return constructor.Invoke(Array.Empty<object>()) as HttpMessageHandler;
})
.AddTransient<CustomDelegationHandler>()
.AddHttpClient("MyApiHttpClientName")
.AddHttpMessageHandler<CustonDelegationHandler>();
}
3.0 -> 3.1-preview2
On Blazor client side your need to tell to the Fetch API to send credentials (cookies and authorization header).
It's describe in the Blazor doc Cross-origin resource sharing (CORS)
requestMessage.Properties[WebAssemblyHttpMessageHandler.FetchArgs] = new
{
credentials = FetchCredentialsOption.Include
};
ex:
#using System.Net.Http
#using System.Net.Http.Headers
#inject HttpClient Http
#code {
private async Task PostRequest()
{
Http.DefaultRequestHeaders.Authorization =
new AuthenticationHeaderValue("Bearer", "{OAUTH TOKEN}");
var requestMessage = new HttpRequestMessage()
{
Method = new HttpMethod("POST"),
RequestUri = new Uri("https://localhost:10000/api/TodoItems"),
Content =
new StringContent(
#"{""name"":""A New Todo Item"",""isComplete"":false}")
};
requestMessage.Content.Headers.ContentType =
new System.Net.Http.Headers.MediaTypeHeaderValue(
"application/json");
requestMessage.Content.Headers.TryAddWithoutValidation(
"x-custom-header", "value");
requestMessage.Properties[WebAssemblyHttpMessageHandler.FetchArgs] = new
{
credentials = FetchCredentialsOption.Include
};
var response = await Http.SendAsync(requestMessage);
var responseStatusCode = response.StatusCode;
var responseBody = await response.Content.ReadAsStringAsync();
}
}
You can set up this option globaly with WebAssemblyHttpMessageHandlerOptions.DefaultCredentials static proprerty.
Or you can implement a DelegatingHandler and set it up in DI with the HttpClientFactory:
public class CustomWebAssemblyHttpMessageHandler : WebAssemblyHttpMessageHandler
{
internal new Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
return base.SendAsync(request, cancellationToken);
}
}
public class CustomDelegationHandler : DelegatingHandler
{
private readonly CustomWebAssemblyHttpMessageHandler _innerHandler;
public CustomDelegationHandler(CustomWebAssemblyHttpMessageHandler innerHandler)
{
_innerHandler = innerHandler ?? throw new ArgumentNullException(nameof(innerHandler));
}
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
request.Properties[WebAssemblyHttpMessageHandler.FetchArgs] = new
{
credentials = "include"
};
return _innerHandler.SendAsync(request, cancellationToken);
}
}
In Setup.ConfigureServices
services.AddTransient<CustomWebAssemblyHttpMessageHandler>()
.AddTransient<WebAssemblyHttpMessageHandler>()
.AddTransient<CustomDelegationHandler>()
.AddHttpClient(httpClientName)
.AddHttpMessageHandler<CustomDelegationHandler>();
Then you can create an HttpClient for your API with IHttpClientFactory.CreateClient(httpClientName)
To use the IHttpClientFactory you need to install Microsoft.Extensions.Http package.
3.0-preview3 => 3.0-preview9
Replace WebAssemblyHttpMessageHandler with BlazorHttpMessageHandler
Background
I have a local backend API at this address:
http://localhost:54641/orders
And a clientside UI at:
http://localhost:3000/
What I need
The UI does a call to the backend, to list all available orders. This includes information on an attachment for each order. It may or not be there. If there is an attachment, you should get this response from the API, for each order:
{
"orderReference": "123456",
"actions": { "download": "http://localhost:54641/orders/123456/download" }
}
Actions will be {} if there's no attachment available.
However
What I do get, is this:
{
"orderReference": "123456",
"actions": { "download": "http://localhost:3000/orders/123456/download" }
}
Which doesn't exist, ofcourse.
What I have right now
Is this code to build the full url, which is going wrong:
var baseUrl = Request.RequestUri.GetLeftPart(UriPartial.Authority);
var uri = Url.Route("DownloadLabel", new {orderReference });
var fullUrl = $"{baseUrl}{uri}";
As in, it returns the requestor's full url path, not that of the current API.
Question
What can I do to get the API url in the response?
So, it should return like this:
http://localhost:54641/orders/123456/download
I guess you want this,
string fullUrl=HttpContext.Current.Request.Url.ToString();
var wantedUrl= fullUrl.SubString(0,fullUrl.IndexOf("/orders"))+"/orders/"+orderReference+"/download";
Found it, I used httpcontext instead:
var baseUrl = HttpContext.Current.Request.Url.GetLeftPart(UriPartial.Authority);
var uri = Url.Route("DownloadLabel", new {orderReference });
var fullUrl = $"{baseUrl}{uri}";
If you start a new Web Project, and create a new MVC4 application (with sub-kind as "WebApi", you can paste the below code in (overwriting HomeController.cs) to get the code to work.
I have a MVC4 application (with WebApi).
I am trying to set a custom-header in a MVC controller method and then do a RedirectToAction. The custom-header is not seen in the second mvc-controller-method.
I am able to set a cookie in the first mvc-controller-method and see it in the second mvc-controller-method (after a RedirectToAction).
Is there a way to see the custom-header I set in the second mvc-controller-method after a RedirectToAction ?
Thanks.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Web;
using System.Web.Mvc;
using System.Web.Security;
namespace MyMvc4WebApiProjectNamespace.Controllers
{
public class HomeController : Controller
{
private const string CustomCookieName = "CustomCookieName";
private const string CustomHeaderName = "X-CustomHeaderName";
private const string IISExpressRootUrl = "http://localhost:55937/"; /* open up the project properties and go to the web tab and find the iis-express area to get the correct value for your environment */
public ActionResult Index()
{
IEnumerable<string> webApiValues = null;
string value1 = null;
string value2 = null;
HttpClientHandler handler = new HttpClientHandler
{
UseDefaultCredentials = true,
PreAuthenticate = true
};
using (var client = new HttpClient(handler))
{
string valuesUri = IISExpressRootUrl + "api/Values";
webApiValues = client
.GetAsync(valuesUri)
.Result
.Content.ReadAsAsync<IEnumerable<string>>().Result;
if (null != webApiValues)
{
value1 = webApiValues.ElementAt(0);
value2 = webApiValues.ElementAt(1);
}
else
{
throw new ArgumentOutOfRangeException("WebApi call failed");
}
}
HttpCookie customCookie = new HttpCookie(CustomCookieName, "CustomCookieValue_ThisShowsUpIn_MyHomeControllerAlternateActionResult_Method");
Response.Cookies.Add(customCookie);
HttpContext.Response.AppendHeader(CustomHeaderName, "CustomHeaderValue_This_Does_Not_Show_Up_In_MyHomeControllerAlternateActionResult_Method");
//Response.AppendHeader(CustomHeaderName, value2);
return RedirectToAction("MyHomeControllerAlternateActionResult");
}
public ActionResult MyHomeControllerAlternateActionResult()
{
IEnumerable<string> webApiReturnValues = null;
CookieContainer cookieContainer = new CookieContainer();
foreach (string cookiename in Request.Cookies)
{
if (cookiename.Equals(CustomCookieName, StringComparison.OrdinalIgnoreCase))
{
var cookie = Request.Cookies[cookiename];
cookieContainer.Add(new Cookie(cookie.Name, cookie.Value, cookie.Path, "localhost"));
}
}
if (cookieContainer.Count < 1)
{
throw new ArgumentOutOfRangeException("CookieContainer did not find the cookie I was looking for");
}
else
{
Console.WriteLine("This is what actually happens. It finds the cookie.");
}
HttpClientHandler handler = new HttpClientHandler
{
UseCookies = true,
UseDefaultCredentials = true,
PreAuthenticate = true,
CookieContainer = cookieContainer
};
using (var client = new HttpClient(handler))
{
bool customHeaderWasFound = false;
if (null != this.Request.Headers)
{
if (null != this.Request.Headers[CustomHeaderName])
{
IEnumerable<string> headerValues = this.Request.Headers.GetValues(CustomHeaderName);
client.DefaultRequestHeaders.Add(CustomHeaderName, headerValues);
customHeaderWasFound = true;
}
}
/*I wouldn't expect it to be in the below, but I looked for it just in case */
if (null != this.Response.Headers)//
{
if (null != this.Response.Headers[CustomHeaderName])
{
IEnumerable<string> headerValues = this.Response.Headers.GetValues(CustomHeaderName);
client.DefaultRequestHeaders.Add(CustomHeaderName, headerValues);
customHeaderWasFound = true;
}
}
if (!customHeaderWasFound)
{
Console.WriteLine("This is what actually happens. No custom-header found. :( ");
}
string valuesUri = IISExpressRootUrl + "api/Values";
webApiReturnValues = client
.GetAsync(valuesUri)
.Result
.Content.ReadAsAsync<IEnumerable<string>>().Result;
if (null == webApiReturnValues)
{
throw new ArgumentOutOfRangeException("WebApi call failed");
}
}
return View(); /* this will throw a "The view 'MyHomeControllerAlternateActionResult' or its master was not found or no view engine supports the searched locations" error, but that's not the point of this demo. */
}
}
}
Response headers are never copied automatically to requests - so setting any custom headers on response will not impact next request issued to handle 302 redirect.
Note that it is the case even with cookies: response comes with "set this cookie" header, and all subsequent request will get "current cookies" header.
If you have your own client you may be able to handle 302 manually (not possible if you are using browser as client).
As another answer stated, response headers are about this response, not the next one. Redirecting is not a server-side action. A redirect instructs the client to perform a completely new request, and of course in a new request, the response headers for the old request are not present. So return RedirectToAction("MyHomeControllerAlternateActionResult"); is guaranteed to not have this response's headers when the browser initiates the new request.
In trying to solve this problem, one might think of trying to persist the data to the next request server-side, such as through a cookie or in an explicit session variable, or implicitly via use of ViewBag/ViewData/TempData. However, I don't recommend this as using session state heavily has performance implications in large/high-usage web sites, plus there are other negative and subtle side-effects that you may run into down the road. For example, if a person has two browser windows open to the same web site, they can't be doing different actions reliably, as the session data for one window can end up being served to the other one. Avoid session usage as much as possible in your web site design—I promise this will benefit you down the road.
A slightly better way, though still with its problems, is to redirect to a URL with querystring parameters containing a payload. And, instead of the whole set of data, you can provide a key that can be pulled from the session (as long as it's also bound to their IP address and is large like a GUID or two together). However, relying on session state is still not ideal as stated before.
Instead, consider using server-side redirection such as child actions. If you find that hard because what you want to call is a main controller you have a few options:
If you're using dependency injection, add a parameter to the current controller (saving it from the constructor and using it in the request method) that is the desired controller you want to "redirect" to. You can then call that controller directly. This may not be ideal (as all calls to this controller also have to new up a copy of that one), but it does work. Trying to new up the other controller manually can also work, but for reasons I don't fully remember, I think this can give some additional problems. In any case, this method can give issues accessing the HttpRequest context and other context objects correctly, though this can be worked around.
Rearchitect your application so that controllers are not the place where full pages are rendered. Instead, use them as "smart routers" that call child actions to perform the real work. Then, you can call the same child actions from any controller. But this still has problems.
Perhaps the best way is to add custom routing logic through action filters or other means (search the web!) so that the correct controller is hit in the first place! This may not always be possible, but sometimes the need to redirect to another controller mid-procedure actually points to a larger design problem. Focusing on how to cause the knowledge of which controller to hit to be available earlier in the pipeline (such as during routing) can reveal architecture problems, and can reveal likely solutions to them.
There may be other options that I haven't thought of, but at least you have a few alternatives to the simple "no way to do that."
I was able to do something similar like what the user is requesting in the following (rudimentary) way:
In the redirect, add a custom query string parameter
Create a custom Module that checks for that parameter and appends the custom header (read http://dotnetlionet.blogspot.com/2015/06/how-to-add-httpmodule-in-mvc5.html on how to do your own module)
In this way I was able to get my custom headers to be picked up