I am trying a pretty simple GET request for a domain in a .NET Core 2.0 console application:
static void Main(string[] args)
{
MainAsync().Wait();
}
static async Task MainAsync()
{
var httpClient = new HttpClient();
var response = await httpClient.GetAsync("https://www.staples.com");
}
This times out every time, throwing a web exception.
I can visit the website https://www.staples.com in my web browser or execute a GET request in postman without a problem, returning in < 1s.
I can even do a simple curl request on the domain and it works fine:
curl https://www.staples.com
Another domain I found the same problem on is https://www.safeco.com/
I have even tried adding some headers to make it seem like this a Chrome browser request, but made no difference:
message.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " +
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36");
message.Headers.Add("Accept-Language", "en-US,en;q=0.8");
message.Headers.Add("Cache-Control", "no-cache");
message.Headers.Add("Pragma", "no-cache");
message.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;" +
"q=0.9,image/webp,image/apng,*/*;q=0.8");
Any other URL that I tried not on those domains seems to work fine. Why are these two domains timing out with HttpClient requests?
It's almost certainly some sort of connection filtering on their end to prevent scraping but only their IT department would ever be able to confirm that. You can get it working by mimicking a browser and sending the correct headers. It seems this site requires a minimum of:
Connection: keep-alive
Accept-Encoding: gzip
Accept-Language: xxx
For example:
static async Task<string> MainAsync()
{
//Added this to decompress the gzip encoded response
HttpClientHandler handler = new HttpClientHandler();
handler.AutomaticDecompression = System.Net.DecompressionMethods.GZip;
var client = new HttpClient(handler);
var request = new HttpRequestMessage()
{
Method = HttpMethod.Get,
RequestUri = new Uri("https://www.staples.com"),
Version = new Version(1, 1)
};
request.Headers.Connection.Add("keep-alive");
request.Headers.AcceptLanguage.Add(new System.Net.Http.Headers.StringWithQualityHeaderValue("en-GB"));
var response = await client.SendAsync(request);
return await response.Content.ReadAsStringAsync();
}
Not an answer, but not appropriate for comment either - Maybe you can glean something from the network trace by adding this to your config. Just change the value of initializeData to a writable location, make a request, then look at the output. It ain't pretty but there may be a clue.
<system.diagnostics>
<sources>
<source name="System.Net" maxdatasize="102400" tracemode="includehex">
<listeners>
<add name="System.Net" />
</listeners>
</source>
</sources>
<switches>
<add name="System.Net" value="Verbose" />
</switches>
<sharedListeners>
<add name="System.Net" type="System.Diagnostics.TextWriterTraceListener" initializeData="c:\somewhere...\networkErr.log" />
</sharedListeners>
</system.diagnostics>
Related
I have an Asp.Net NVC5 web application (running under Visual Studio 2017) on localhost:59569 (SiteApi). I have a second website (also running under Visual Studio 2017) on localhost:61527 (SiteClient) where a page once loaded makes the following api call to SiteApi:
$http({
url: 'http://localhost:59569/api/V2/' + alias,
method: 'POST',
data: pm,
xhrFields: { withCredentials: true },
headers: { 'Content-Type': 'application/json; charset=utf-8' }
})
.then(th, ex);
NOTE: I have tried this with and without the xhrFields + withCredentials information using Microsoft IE, Microsoft Edge and Chrome.
Back on SiteApi the resulting preflight call for OPTIONS is intercepted by the following code in Global.asax which executes exactly as written and I can trace through the if statement when an inbound call for OPTIONS triggers it.
protected void Application_BeginRequest()
{
if (Request.Headers.AllKeys.Contains("Origin") && Request.HttpMethod == "OPTIONS")
{
Response.Clear();
Response.Headers.Add("Access-Control-Allow-Origin", "*");
Response.Headers.Add("Access-Control-Allow-Headers", "Content-Type, Accept, X-Requested-With, Session");
Response.Flush();
Response.End();
}
}
The intention being to send the desired headers back to the client to allow CORS to function properly - however immediately after this code is executed the web page back on SiteClient reports that the request has been blocked due to missing 'Access-Control-Allow-Origin' header is missing and I can see that none of the headers I have specified have made it back to the client.
In an attempt to have CORS work I have the following nuget packages installed on the SiteAPI project.
Microsoft.AspNet.Cors
Microsoft.AspNet.WebApi.Cors
I adjusted the WebApiConfig.Register() method to include:
// Web API configuration and services
config.EnableCors();
I have tried many variations of adding the filter attributes to my controller like so:
[EnableCors("*", "*", "*", SupportsCredentials = true)]
I have tried adding my own custom ActionFilterAttribute from solutions found in other CORS related questions on stackoverflow - for example (among various others):
public override void OnActionExecuting(System.Web.Mvc.ActionExecutingContext filterContext)
{
filterContext.RequestContext.HttpContext.Response.AddHeader("Access-Control-Allow-Origin", "*");
base.OnActionExecuting(filterContext);
I have the following to my web.config file:
<handlers>
<remove name="ExtensionlessUrlHandler-Integrated-4.0" />
<remove name="OPTIONSVerbHandler" />
<remove name="TRACEVerbHandler" />
<add name="ExtensionlessUrlHandler-Integrated-4.0" path="*." verb="*" type="System.Web.Handlers.TransferRequestHandler" preCondition="integratedMode,runtimeVersionv4.0" />
</handlers>
I have ALL these solutions live in my project and in spite of this I still get CORS errors on the client side.
So the caveat here is that I also have a custom filter that looks up security on each API call - which works fine with ALL calls made from pages running on SiteApi. In the case of calls from SiteClient (CORS calls) the security filter never fires at all though I am getting 401 errors reported on the client in addition to the errors due to the missing CORS related headers.
I have cleared the caches of all browsers and the server itself. This is my first time working with CORS and I'm already exhausted from working with what really should be a simple solution. Looking for solutions here and would appreciate some help from those in the know.
Request Headers:
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 2
Content-Type: application/json; charset=UTF-8
Host: localhost:59569
Origin: http://localhost:61527
Referer: http://localhost:61527/Home/Index
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Safari/537.36 Edg/80.0.361.54
Response Headers:
Cache-Control: private
Content-Length: 6115
Content-Type: text/html; charset=utf-8
Date: Wed, 19 Feb 2020 00:46:06 GMT
Server: Microsoft-IIS/10.0
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
X-SourceFiles: =?UTF-8?B?QzpcVXNlcnNcbngyMDA4MjZcRGV2XFRlY2hJVFxFQVNJV2ViQXBwXGFwaVxWMlxHZXRDb21tYW5kcw==?=
This is the flow I use. Sometimes browsers don't like "*". Other times, browsers don't like localhost, either. This is the logic I use (modify the allow headers as you see fit). It could be the fact that you aren't allowing the access control headers in your allowed headers, too:
[Add this to Global.asax]
protected void Application_BeginRequest(object sender, EventArgs e)
{
var originKey =
Request.Headers.AllKeys.FirstOrDefault(
a => a.Equals("origin", StringComparison.InvariantCultureIgnoreCase));
if (originKey != null && Request.HttpMethod == "OPTIONS"))
{
// Optional Whitelist check here can return without the headers below to reject CORS
Response.Headers.Add("Access-Control-Allow-Origin", Request.Headers[originKey]);
Response.Headers.Add("Access-Control-Allow-Credentials", "true");
Response.Headers.Add("Access-Control-Allow-Methods", "GET, POST, PUSH, DELETE, OPTIONS");
Response.Headers.Add("Access-Control-Allow-Headers",
"Authorization, Content-Type, Access-Control-Allow-Headers, X-Requested-With, Access-Control-Allow-Method, Accept");
Response.Flush();
Response.End();
return;
}
}
I have two ASP.NET vNext Web Applications running with CoreCLR on Azure WebSites, published by the lates Visual Studio 2015 CTP.
When I'm trying to make a call from one application to the second with standard HttpClient code:
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(_webUri);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
HttpContent contentPost = new StringContent(request.ToJson(), Encoding.UTF8, "application/json");
var response = await client.PostAsync(uri, contentPost);//.PostAsJsonAsync("api/products", request);
if (response.IsSuccessStatusCode)
{
...
}
}
I get following exception:
WinHttpException: An attempt was made to access a socket in a way forbidden by its access permissions.
System.Net.Http.WinInetProxyHelper.GetProxyForUrl(SafeInternetHandle sessionHandle, Uri uri, WINHTTP_PROXY_INFO& proxyInfo)
HttpRequestException: An error occurred while sending the request.
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
my web.config on the azure websites ftp:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<appSettings>
<add key="kpm-package-path" value="..\approot\packages" />
<add key="bootstrapper-version" value="1.0.0-beta2" />
<add key="kre-package-path" value="..\approot\packages" />
<add key="kre-version" value="1.0.0-beta2" />
<add key="kre-clr" value="CoreCLR" />
<add key="kre-app-base" value="..\approot\src\Ingrid.Web" />
</appSettings>
</configuration>
Solution which I found on: https://github.com/sendgrid/sendgrid-csharp/issues/18
it's better to go with RestSharp than HttpClient, and it is indeed working: http://restsharp.org/
I am trying to get authenticated my Mobile App ( for testing running from visual studio) to authenticate my Web API 2 Authentication Token in Single Page APP
But Ajax always cancelled with status cancelled
I have enabled CORS as below
var cors = new EnableCorsAttribute("*", "*", "*") {SupportsCredentials = true};
config.EnableCors(cors);
//Use only bearer token authentication
config.SuppressDefaultHostAuthentication();
Web Config
<system.web>
<authentication mode="None" />
<compilation debug="true" targetFramework="4.5" />
<httpRuntime targetFramework="4.5" />
<roleManager enabled="true" />
<httpModules>
<add name="ErrorLog" type="Elmah.ErrorLogModule, Elmah" />
<add name="ErrorMail" type="Elmah.ErrorMailModule, Elmah" />
<add name="ErrorFilter" type="Elmah.ErrorFilterModule, Elmah" />
</httpModules>
</system.web>
My Ajax Request
$.ajax('http://retail.leap-tel.com/token', {
type: "POST",
data: data,
xhrFields: {
withCredentials: true
}
Chorme Developer tools Log
Status : Cancelled
Request URL:http://retail.leap-tel.com/token
Request Headersview source
Accept:*/*
Cache-Control:no-cache
Content-Type:application/x-www-form-urlencoded; charset=UTF-8
Origin:http://localhost:1539
Pragma:no-cache
Referer:http://localhost:1539/index.html
User-Agent:Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
Form Dataview sourceview URL encoded
I made some changes in DurandalAuth for enabling CORS and created a sample Javascript client in another repo
Check it out
https://github.com/yagopv/DurandalAuth/commit/4e7c09bc589345c946319b42d320e2b4d8313573
https://github.com/yagopv/DurandalAuthjsclient
Perhaps your main issue is about using the default Web API CORS attribute. You should use Microsoft.Owin.Cors attribute instead for working with the Owin middleware
public static void Register(HttpConfiguration config)
{
var corsAttribute = new EnableCorsAttribute("*", "*", "*");
config.EnableCors(corsAttribute);
// Web API configuration and services
// Web API routes
config.MapHttpAttributeRoutes();
}
Add cors attribute in webapiConfig class register method for application level.
I have a method that does a POST like below
var response = await client.PostAsJsonAsync(url, entity);
if (response.IsSuccessStatusCode)
{
// read the response as strongly typed object
return await response.Content.ReadAsAsync<T>();
}
My question is how can I obtain the actual JSON that got posted from the entity object. I would like to log the JSON that gets POSTED, so it will be nice to have that without me having to do a json serialize myself.
An example of how you could do this:
Some notes:
LoggingHandler intercepts the request before it handles it to HttpClientHandler which finally writes to the wire.
PostAsJsonAsync extension internally creates an ObjectContent and when ReadAsStringAsync() is called in the LoggingHandler, it causes the formatter
inside ObjectContent to serialize the object and that's the reason you are seeing the content in json.
Logging handler:
public class LoggingHandler : DelegatingHandler
{
public LoggingHandler(HttpMessageHandler innerHandler)
: base(innerHandler)
{
}
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
Console.WriteLine("Request:");
Console.WriteLine(request.ToString());
if (request.Content != null)
{
Console.WriteLine(await request.Content.ReadAsStringAsync());
}
Console.WriteLine();
HttpResponseMessage response = await base.SendAsync(request, cancellationToken);
Console.WriteLine("Response:");
Console.WriteLine(response.ToString());
if (response.Content != null)
{
Console.WriteLine(await response.Content.ReadAsStringAsync());
}
Console.WriteLine();
return response;
}
}
Chain the above LoggingHandler with HttpClient:
HttpClient client = new HttpClient(new LoggingHandler(new HttpClientHandler()));
HttpResponseMessage response = client.PostAsJsonAsync(baseAddress + "/api/values", "Hello, World!").Result;
Output:
Request:
Method: POST, RequestUri: 'http://kirandesktop:9095/api/values', Version: 1.1, Content: System.Net.Http.ObjectContent`1[
[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]], Headers:
{
Content-Type: application/json; charset=utf-8
}
"Hello, World!"
Response:
StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Date: Fri, 20 Sep 2013 20:21:26 GMT
Server: Microsoft-HTTPAPI/2.0
Content-Length: 15
Content-Type: application/json; charset=utf-8
}
"Hello, World!"
NOTE: This works with .NET Framework ONLY!
May be working with .NET 7+ (https://github.com/dotnet/runtime/issues/23937)
See http://mikehadlow.blogspot.com/2012/07/tracing-systemnet-to-debug-http-clients.html
To configure a System.Net listener to output to both the console and a log file, add the following to your assembly configuration file:
<system.diagnostics>
<trace autoflush="true" />
<sources>
<source name="System.Net">
<listeners>
<add name="MyTraceFile"/>
<add name="MyConsole"/>
</listeners>
</source>
</sources>
<sharedListeners>
<add
name="MyTraceFile"
type="System.Diagnostics.TextWriterTraceListener"
initializeData="System.Net.trace.log" />
<add name="MyConsole" type="System.Diagnostics.ConsoleTraceListener" />
</sharedListeners>
<switches>
<add name="System.Net" value="Verbose" />
</switches>
</system.diagnostics>
Network tracing also available for next objects (see article on msdn)
System.Net.Sockets
Some public methods of the Socket, TcpListener, TcpClient, and Dns classes
System.Net
Some public methods of the HttpWebRequest, HttpWebResponse, FtpWebRequest, and FtpWebResponse classes, and SSL debug information (invalid certificates, missing issuers list, and client certificate errors.)
System.Net.HttpListener
Some public methods of the HttpListener, HttpListenerRequest, and HttpListenerResponse classes.
System.Net.Cache
Some private and internal methods in System.Net.Cache.
System.Net.Http
Some public methods of the HttpClient, DelegatingHandler, HttpClientHandler, HttpMessageHandler, MessageProcessingHandler, and WebRequestHandler classes.
System.Net.WebSockets.WebSocket
Some public methods of the ClientWebSocket and WebSocket classes.
Put the following lines of code to the configuration file:
<configuration>
<system.diagnostics>
<sources>
<source name="System.Net" tracemode="includehex" maxdatasize="1024">
<listeners>
<add name="System.Net"/>
</listeners>
</source>
<source name="System.Net.Cache">
<listeners>
<add name="System.Net"/>
</listeners>
</source>
<source name="System.Net.Http">
<listeners>
<add name="System.Net"/>
</listeners>
</source>
<source name="System.Net.Sockets">
<listeners>
<add name="System.Net"/>
</listeners>
</source>
<source name="System.Net.WebSockets">
<listeners>
<add name="System.Net"/>
</listeners>
</source>
</sources>
<switches>
<add name="System.Net" value="Verbose"/>
<add name="System.Net.Cache" value="Verbose"/>
<add name="System.Net.Http" value="Verbose"/>
<add name="System.Net.Sockets" value="Verbose"/>
<add name="System.Net.WebSockets" value="Verbose"/>
</switches>
<sharedListeners>
<add name="System.Net"
type="System.Diagnostics.TextWriterTraceListener"
initializeData="network.log"
/>
</sharedListeners>
<trace autoflush="true"/>
</system.diagnostics>
</configuration>
The easiest solution would be to use Wireshark and trace the HTTP tcp flow.
I have an application in C# which uses a service reference to send SMS through a web service. The user's internet connection needs to pass through a proxy server in order to reach the world.
So my question is how to tell .NET to call web service through the proxy? Or how to set proxy settings for internet connection of my application just like what you can do in YMahoo Messenger?
Also I want to let user to choose proxy settings.
I believe what you are looking for is defaultProxy in your config file.
Here's an example from the link:
<configuration>
<system.net>
<defaultProxy>
<proxy
usesystemdefault="true"
proxyaddress="http://192.168.1.10:3128"
bypassonlocal="true"
/>
<bypasslist>
<add address="[a-z]+\.contoso\.com" />
</bypasslist>
</defaultProxy>
</system.net>
</configuration>
Please try below code hope this will help you
tring targetUrl = "http://www.google.com";
string proxyUrlFormat = "http://zend2.com/bro.php?u={0}&b=12&f=norefer";
string actualUrl = string.Format(proxyUrlFormat, HttpUtility.UrlEncode(targetUrl));
// Do something with the proxy-ed url
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(new Uri(actualUrl));
HttpWebResponse resp = req.GetResponse();
string content = null;
using(StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
content = sr.ReadToEnd();
}
Console.WriteLine(content);