I'm using HttpClient but it has problems with DNS resolve (it is using the sync method for this) So I use another lib for doing DNS queries and now I'm tryging to get custom urls by IP but I need to replace Host header. For example I have url http://fb.com but I need to get http://1.1.1.1 with Host set to fb.com I've tryied:
_req = new HttpRequestMessage(HttpMethod.Get, newUri.ToString());
_req.Headers.Host = uri.Host;
_httpClient.DefaultRequestHeaders.Host = uri.Host;
but this doesn't work. Is there any way to set own Host header like in HttpWebRequest?
It's work. The problem was with Fiddler which override Host header based on url. When Fiddler is off everything is going fine.
Related
The company I work for have a limited internet connection and we use the proxy (example: 10.10.10.10:8080) to access at some restricted connections.
I can use the API in Postman (putting the proxy in the Postman settings) but when putting in the C# WebClient code it gives me a 403-Forbidden error.
I only need the var sensorData field but I splitted in var data and var data2 to understand where was the problem. It gives me the error at the var data = ...
Uri uri = new Uri("https://XXXXXXXX/api/DatasourceData/DatasourceDataHistoryBySerialNumber/");
Token token = new Token();
token = GetToken(tokenAPI);
using (WebClient client = new WebClient())
{
try
{
client.Proxy = new WebProxy("10.10.10.10", 8080);
client.Headers.Add("Authorization", "Bearer " + token.AccessToken);
client.QueryString.Add("serialNumbersDatasource", "I2001258");
client.QueryString.Add("startDate", string.Format("{0:s}", "2019-12-01"));
client.QueryString.Add("endDate", string.Format("{0:s}", DateTime.Now));
client.QueryString.Add("isFilterDatesByDataDate", "false");
var data = client.DownloadData(uri);
var data2 = (Encoding.UTF8.GetString(data));
sensorData = (JsonConvert.DeserializeObject<List<Sensor>>(Encoding.UTF8.GetString(client.DownloadData(uri))))[0];
}
}
Seems the problem at this line
client.Headers.Add("Authorization", "Bearer " + "tokenTest");
here you wil add header Authorization with value Bearer tokenTest
so, 403 Forbidden returns by service which you are addressing, but not a proxy
change to
client.Headers.Add("Authorization", "Bearer " + tokenTest);
and check if tokenTest has valid value
Check to see if you need any additional properties on the proxy. You may possibly need to enable:
UseDefaultCredentials (Boolean) true if the default credentials are
used; otherwise, false. The default value is false
Also, check your full url and query string that you are producing - look at the outgoing request fabricated (in the debugger) or through Fiddler and make sure it all lines up, url, query string, headers, etc.
From the docs:
Address
Gets or sets the address of the proxy server.
BypassArrayList
Gets a list of addresses that do not use the proxy server.
BypassList
Gets or sets an array of addresses that do not use the proxy server.
BypassProxyOnLocal
Gets or sets a value that indicates whether to bypass the proxy server for local addresses.
Credentials
Gets or sets the credentials to submit to the proxy server for authentication.
UseDefaultCredentials
Gets or sets a Boolean value that controls whether the DefaultCredentials are sent with requests.
Probably a problem with authorization header.
Is the token valid? Does it work with the same token in Postman?
I bet the api can't validate the token and and gives you no authorization to the resources. This is what a 403 would mean (but don't know what the api programmer actually intended by giving you 403).
Do you have access to the api's source code?
The token is really a string "tokentest" and that works with Postman?
I would suggest you to go for xNet.dll instead of webclient Because xNet library are considered best for proxy and webrequest.
var request = new HttpRequest()
request.UserAgent = Http.ChromeUserAgent();
request.Proxy = Socks5ProxyClient.Parse("10.10.10.10:8080");//can use socks4/5 http
Based on this
Try adding User-Agent in the header
client.Headers.Add("User-Agent", "PostmanRuntime/7.26.1");
In my case i did not specify security protocol. Paste this line of code before running any WebClient requests.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
original answer: How to specify SSL protocol to use for WebClient class
I have a large collection of DNS names that have already been resolved to IP addresses. With this collection I need to download HTML from them. It's a very large list and I need to do it as efficiently as possible.
I'm using System.Net.HttpWebRequest to download HTML from the each domain. HttpWebRequest is repeating the DNS lookup, and this is adding to the connection time. I've run tests to see if sockets for those IPs on port 80 would connect faster and they do.
So I'd like to use HttpWebRequest with a known IP address, but I don't know how. All WebRequest factory methods require a URL.
Now I thought I could do something like this (where 1.2.3.4 is the IP)
var req = WebRequest.Create("http://1.2.3.4/");
req.Headers.Add(....); <-- add something here
I need to somehow add to the HTTP header what the target domain is, but I'm not sure how to do it.
Pretty simple:
var ip = "93.184.216.119";
var host = "example.com";
var ipUri = new UriBuilder(Uri.UriSchemeHttp, ip).Uri;
var request = WebRequest.CreateHttp(ipUri);
request.Host = host;
using (var response = request.GetResponse())
{
// do something with response
}
I'm trying to proxy my requests, but it seems that the proxy setting is ignored.
I'm using the following code:
var req = (HttpWebRequest)WebRequest.Create("http://whatsmyip.net/");
req.Proxy = new WebProxy("195.128.253.243", 8080) { BypassProxyOnLocal = false };
req.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);
var html = new StreamReader(req.GetResponse().GetResponseStream()).ReadToEnd();
The proxy is just a random free proxy from here.
The result always contains my real ip instead of the proxy ip.
When I'm surfing to that website using hidemyass or other alternatives, the ip changes as expected.
Anyone has an idea for what am I doing wrong?
I just tried your code (without the HttpWebRequest cast) bouncing off my local tor server and it works as expected. Have you tried the proxy directly within IE?
This is me publicly documenting my mistake so that if I or anyone does it again, they don't have to spend 3 hours tearing their hair out trying to fix such a simple thing.
Context
I was sending an HttpRequest from one C# MVC ASP.NET application to another.
The applications require an HTTPS connection, and we are using URLRewrite to redirect an HTTP request to an HTTPS url.
One application was sending a POST request with some JSON data in the body, pretty standard stuff. The other application was set up to receive this data with an MVC controller class (CollectionAction and Insert methods for GET and POST respectively).
Symptoms of the problem
The receiving application was running the GET method (CollectionAction) instead of the POST action (ItemAction). The reason for this was that the request coming in to the application was in fact a GET request, and to top it off the JSON data was missing too.
I sent the header "x-http-method" to override the request method from GET to POST (I was already setting the request httpmethod to POST but this was being ignored). This worked but still I had no data being sent.
So now I am stuck pulling my hair out, because I can see a POST request with content-length and data being sent out and I have a GET request with no data or content-length coming in (but the headers were preserved)
Turns out I was using UriBuilder to take a base URL and apply a resource path to it. For example I would have "google.com" in my web.config and then the UriBuilder would take a resource like Pages and construct the url "google.com/Pages". Unfortunately, I was not initializing the UriBuilder with the base URL, and instead was using a second UriBuilder to extract the host and add that to the path like so:
public Uri GetResourceUri(string resourceName)
{
var domain = new UriBuilder(GetBaseUrl());
var uribuilder = new UriBuilder()
{
Path = domain.Path.TrimEnd('/') + "/" + resourceName.TrimStart('/'),
Host = domain.Host
};
var resourceUri = uribuilder.Uri;
return resourceUri;
}
The problem with this code is that the scheme is ignored (HTTP:// vs HTTPS://) and it defaults to HTTP. So my client was sending out the request to an HTTP url instead of the required HTTPS url. This is the interesting part, URLRewrite was kicking in and saying that we needed to go to an HTTPS url instead so it redirected us there. But in doing so, it ignored the Http-Method and the POST data, which just got set to defaults GET and null. This is what the 2nd application could see at the receiving end.
So the function had to be rewritten to this which fixed the problem:
public Uri GetResourceUri(string resourceName)
{
var baseUrl = GetBaseUrl();
var domain = new UriBuilder(baseUrl);
var uribuilder = new UriBuilder(baseUrl)
{
Path = domain.Path.TrimEnd('/') + "/" + resourceName.TrimStart('/'),
};
var resourceUri = uribuilder.Uri;
return resourceUri;
}
Looking for a way to issue an HTTPwebrequest, or use the browser control, or winhttp to make a request to a URL, but override the IP address it connects to from the DNS lookup to a specific one.
Trying to do something similar to the HOSTS file, but programatically without having to modify this file. It can be C# or C+
Why I need it, the host i am sending the request has multiple IPs, and their Domain servers are doing load balancing accross the different IPs. Trying to force the request to a particular IP, but I need the host in the http request to be still the original host. I need this programatically because changing the host file every time i need to run this test is too time consuming.
All you had to do was this:
var request = (HttpWebRequest) WebRequest.Create("http://192.168.1.1");
request.Host = "news.bbc.co.uk";
If I understand correctly you have to make an http request to a web server using virtualhosts but the DNS isn't setup yet so you have to specify the ip address in the url but send something else in the Host: header.
If that's the case you may be able to do so..
In C# using WebProxy:
See Kayode Leonard's answer for .NET 4 and up.
Here's the code I would use if I have my server running on 67.223.227.171:8888 but I need to have www.example.com in the Host: header.
System.Net.WebRequest r = System.Net.WebRequest.Create("http://www.example.com");
r.Proxy = new WebProxy("http://67.223.227.171:8888");
See this link
In C++ using WinHttp:
Using WinHttp you can simply set the Host: header with WinHttpAddRequestHeaders.
So once again if I have my server running on 67.223.227.171:8888 but I need to have www.example.com in the Host: header:
#include <windows.h>
#include <winhttp.h>
#include <assert.h>
int main() {
HINTERNET hSession = WinHttpOpen(L"A WinHTTP Example Program/1.0",
WINHTTP_ACCESS_TYPE_DEFAULT_PROXY,
WINHTTP_NO_PROXY_NAME,
WINHTTP_NO_PROXY_BYPASS, 0);
assert(hSession != NULL);
// Use WinHttpConnect to specify an HTTP server.
HINTERNET hConnect = WinHttpConnect( hSession,
L"67.223.227.171",
8888,
0 );
assert(hConnect != NULL);
// Open and Send a Request Header.
HINTERNET hRequest = WinHttpOpenRequest( hConnect,
L"GET",
L"/downloads/samples/internet/winhttp/retoptions/redirect.asp",
NULL,
WINHTTP_NO_REFERER,
WINHTTP_DEFAULT_ACCEPT_TYPES,
0 );
assert(hRequest != NULL);
BOOL httpResult = WinHttpAddRequestHeaders(
hRequest,
L"Host: www.example.com",
-1L,
0);
assert(httpResult);
httpResult = WinHttpSendRequest( hRequest,
WINHTTP_NO_ADDITIONAL_HEADERS,
0,
WINHTTP_NO_REQUEST_DATA,
0,
0,
0 );
assert(httpResult);
httpResult = WinHttpReceiveResponse( hRequest, NULL );
assert(httpResult);
}
Edited: The class name is WebProxy. Added C# sample code. Added C++ sample code.
[Note, further to Kayode Leonard's answer: A Host property was added to the request in .Net 4.0, making this answer obsolete]
I think you are saying that you want to be able to override the ip address for a given host, without changing the host header.
For example, news.bbc.co.uk maps to IP address 212.58.226.139, but you want to be able to map this to another ip address, while still presenting the same news.bbc.co.uk "Host" http header to the overriden address. This is what you'd acheive by overriding the HOSTS file as you say, which is slightly different to Jason's answer as his won't present the original "Host" http header.
I don't believe you can do this easily (although I'm about to experiment to find out!). Certainly you can't do the following:
var request = (HttpWebRequest) WebRequest.Create("http://192.168.1.1");
request.Headers["Host"] = "news.bbc.co.uk";
as this will fail with an error saying you can't modify the "Host" header.
You probably can do it if your are willing to go down a level below the HttpWebRequest and deal at a more TCP level, but I'm not sure how you'd approach it without going down to that level.
[Edit]: Having played around with various approaches of overriding HttpWebRequest and WebHeaderCollection, I'm pretty sure it can't be done this way. However, Alexandre Jasmin's answer seems to be the solution.
I'm adding an answer that relies on curl and the --resolve command line option.
https://curl.haxx.se/docs/manpage.html#--resolve
From higher level languages, it's usually possible to invoke curl.