I have a large collection of DNS names that have already been resolved to IP addresses. With this collection I need to download HTML from them. It's a very large list and I need to do it as efficiently as possible.
I'm using System.Net.HttpWebRequest to download HTML from the each domain. HttpWebRequest is repeating the DNS lookup, and this is adding to the connection time. I've run tests to see if sockets for those IPs on port 80 would connect faster and they do.
So I'd like to use HttpWebRequest with a known IP address, but I don't know how. All WebRequest factory methods require a URL.
Now I thought I could do something like this (where 1.2.3.4 is the IP)
var req = WebRequest.Create("http://1.2.3.4/");
req.Headers.Add(....); <-- add something here
I need to somehow add to the HTTP header what the target domain is, but I'm not sure how to do it.
Pretty simple:
var ip = "93.184.216.119";
var host = "example.com";
var ipUri = new UriBuilder(Uri.UriSchemeHttp, ip).Uri;
var request = WebRequest.CreateHttp(ipUri);
request.Host = host;
using (var response = request.GetResponse())
{
// do something with response
}
Related
So the goal of this service is to determine if the machine is on a known network such as different office locations or not based on the public IP. I've created a few lists of known IP addresses to compare against. If the public IP is not known, I trigger a process to start an OpenVPN connection to my server.
I have been utilizing http://ipv4.icanhazip.com for retrieving the public IP.
The current behavior is, during the first Timer interval the web request returns the correct public IP. The VPN connection is then called and shortly establishes. In CMD I am able to run curl http://ipv4.icanhazip.com and confirm the VPN server IP as well the IPv4 route table has in fact been updated. All my web browser traffic is being routed correctly.
However the service requests never return the new IP during the 15 second intervals.
Here is my WebRequest function
public static string ReturnPublicIP()
{
string url = "http://ipv4.icanhazip.com";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.GZip;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream))
{
return reader.ReadToEnd().Trim((char)0x0a, (char)0x0d);
}
}
Any idea's on what I can do to remedy this? I would imagine the networking\process thread would be aware of the updated route table and flow the traffic accordingly..
Thanks in advance for any help!
I'm using HttpClient but it has problems with DNS resolve (it is using the sync method for this) So I use another lib for doing DNS queries and now I'm tryging to get custom urls by IP but I need to replace Host header. For example I have url http://fb.com but I need to get http://1.1.1.1 with Host set to fb.com I've tryied:
_req = new HttpRequestMessage(HttpMethod.Get, newUri.ToString());
_req.Headers.Host = uri.Host;
_httpClient.DefaultRequestHeaders.Host = uri.Host;
but this doesn't work. Is there any way to set own Host header like in HttpWebRequest?
It's work. The problem was with Fiddler which override Host header based on url. When Fiddler is off everything is going fine.
I am using this code to perform a simple REST request. (The code mostly comes from this q: How to post JSON to the server?).
Why is it so slow? I'm using VS 2013 and it takes about 15 secs on first try and then about 4 secs. on subsequent tries, yet in another language (Delphi) I can make a http request and it takes about 1 sec consistently.
var request = (HttpWebRequest)WebRequest.Create("http://jsonplaceholder.typicode.com/posts");
request.ContentType = "application/json";
request.Method = "POST";
request.ServicePoint.Expect100Continue = false;
using (var streamWriter = new StreamWriter(request.GetRequestStream()))
{
string json = new JavaScriptSerializer().Serialize(new
{
title = "foo",
body = "bar",
userId = "1"
});
streamWriter.Write(json);
}
var response = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(response.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
textBox1.Text = result;
}
P.S. You can test this code for yourself, it is simply using a test REST server from the internet at above url.
What do you mean by first try? It means the first try after I leave the computer for a while
Before reaching the server, there is a process of finding the IP address of the server. This process is called Dns Resolution.
First time, your application has to go through the process of Dns Resolution in order to find the IP address. Once you resolved the IP address, the IP address will be cached in the local machine.
So, further calls doesn't go through the process of Dns Resolution; it can use the cached IP. After a while, the cache will be dropped and again you'll hit the DNS server for resolving the IP address.
This is the only explanation I can come up for the delay you're noticing. Whenever you're noticing a delay, that probably means that you're hitting the Dns Server just because it is either the first time or cache is expired.
Why it is faster in other environment(Delphi)?
I'm sorry I can't come up with a good reason for this.
I'm trying to proxy my requests, but it seems that the proxy setting is ignored.
I'm using the following code:
var req = (HttpWebRequest)WebRequest.Create("http://whatsmyip.net/");
req.Proxy = new WebProxy("195.128.253.243", 8080) { BypassProxyOnLocal = false };
req.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);
var html = new StreamReader(req.GetResponse().GetResponseStream()).ReadToEnd();
The proxy is just a random free proxy from here.
The result always contains my real ip instead of the proxy ip.
When I'm surfing to that website using hidemyass or other alternatives, the ip changes as expected.
Anyone has an idea for what am I doing wrong?
I just tried your code (without the HttpWebRequest cast) bouncing off my local tor server and it works as expected. Have you tried the proxy directly within IE?
I have asp.net website hosted and I am making WebRequest to post data and get response. The website is having IP filtering. I want to spoof sender IP address for testing purpose. Is it possible to do it programmatically or I have to use any tool.
public string GetResponse(string request)
{
lock (Obj)
{
request = request + _dataControlInfo.SendEndingWith;
Logger.Info(request);
var req = (HttpWebRequest)WebRequest.Create(_serviceUrl);
req.Headers.Add("SOAPAction", "\"\"");
req.ContentType = "text/xml;charset=\"utf-8\"";
req.Accept = "text/xml";
req.Method = "POST";
var stm = req.GetRequestStream();
var bytes = UtfEncoding.StringToUtf8ByteArray(request);
stm.Write(bytes, 0, bytes.Length);
stm.Close();
var resp = req.GetResponse();
var stmr = new StreamReader(resp.GetResponseStream());
var strResponseXml = stmr.ReadToEnd();
Logger.Info(strResponseXml);
return strResponseXml;
}
}
Please specify any possibilities.
What your looking for is SharpPCap which is a .NET port of WinPCap.. it allows you to do IP Spoofing, which is what your talking about. The only problem with your idea is that you wont be able to get a response back. You can send requests out, but if you dont have a proper return address then the request will be lost in the interwebs.
Edit
To do this yoruself w/out the help of a library you will need to construct the raw packets yourself. This has been answered here.
If you're expecting to get a response, then no. Without the correct IP address, the server won't send the response to the correct destination.
If you insist on trying anyway, see this article for programmatically setting the client's IP address.
Or you can use Web Performance Tests and a load test with IP Switching enabled
You can try to use a proxy, as documented here.
( http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.proxy.aspx ).
Setting up a proxy on a different computer, then configuring that computer as your requests proxy server should make the request appear as if it came from the proxy's IP, not yours.
Some servers can also consider X-Forwarded-For and X-Real-IP headers.
So if server checks for these headers you can add them to your Web request.
But it depends on server implementation.
Use the Spoof class found in the System.Security namespace...