Develop Reference

Using System.Net.WebRequest with a known IP end point

I have a large collection of DNS names that have already been resolved to IP addresses. With this collection I need to download HTML from them. It's a very large list and I need to do it as efficiently as possible.
I'm using System.Net.HttpWebRequest to download HTML from the each domain. HttpWebRequest is repeating the DNS lookup, and this is adding to the connection time. I've run tests to see if sockets for those IPs on port 80 would connect faster and they do.
So I'd like to use HttpWebRequest with a known IP address, but I don't know how. All WebRequest factory methods require a URL.
Now I thought I could do something like this (where 1.2.3.4 is the IP)
var req = WebRequest.Create("http://1.2.3.4/");
req.Headers.Add(....); <-- add something here
I need to somehow add to the HTTP header what the target domain is, but I'm not sure how to do it.

Pretty simple:
var ip = "93.184.216.119";
var host = "example.com";
var ipUri = new UriBuilder(Uri.UriSchemeHttp, ip).Uri;
var request = WebRequest.CreateHttp(ipUri);
request.Host = host;
using (var response = request.GetResponse())
{
// do something with response
}

Can't execute GET request from Win8 app

I tried to do GET request, and i have an InnerExeption:
{"Unable to connect to the remote server"}, {"An attempt was made to access a socket in a way forbidden by its access permissions myServerAddress:port"} this is my code :
string url = "url_to_my_server";
WebRequest request = WebRequest.Create(url);
request.Credentials = CredentialCache.DefaultCredentials;
request.Method = "GET";
WebResponse response = request.GetResponseAsync().Result;
How can I do GET request in win8 app?
For example of GET request, I use this link. I also associate my app with winstoreapp for getting Identity Name and Publisher.

To use internet connection in your application, you need (a) set the required capability Internet (Client), (b) use HttpClient class, which is new and much easy way to send requests. You will find the set of GetAsync methods, which allow you to send GET request.

resourceUri="url_to_my_server";
HttpClient httpClient=new HttpClient();
HttpResponseMessage response = await httpClient.GetAsync(resourceUri);

WebRequest fails with "414 Request URI too long" in ASP.NET application

We have an ASP.NET application that requests an SSRS 2005 report in HTML format after passing the parameters for the report as a WebRequest. The application only fails when a report with a large number of multi-select parameters is requested, throwing a "414: Request URI too long" error at the webRequest.GetResponse() line.
The code used to make the request is:
HttpWebRequest webRequest = null;
HttpWebResponse webResponse = null;
string webRequestURL = _ReportManager.GetRSUrl(reportID); //this passes the report link on the SSRS server
//make the request
Byte[] bytes = Encoding.UTF8.GetBytes("xml_doc=" + HttpUtility.UrlEncode(webRequestURL));
webRequest = (HttpWebRequest)WebRequest.Create(webRequestURL);
webRequest.Method = "POST";
webRequest.ContentLength = bytes.Length;
webRequest.Timeout = Configuration.WebRequestTimeOut;
RSExecution2005.ReportExecutionService rsE = new RSExecution2005.ReportExecutionService();
rsE.Url = Configuration.ReportExecutionServiceUrl2005;
rsE.Credentials = System.Net.CredentialCache.DefaultCredentials;
webRequest.Credentials = rsE.Credentials;
Stream reqStream = null;
reqStream = webRequest.GetRequestStream();
reqStream.Write(bytes, 0, bytes.Length);
reqStream.Close();
webResponse = (HttpWebResponse)webRequest.GetResponse();
As the report fails on the server side, I have looked into IIS and ReportServer properties to increase the maxUrl, maxRequestLength, MaxQueryString, etc. in terms of bytes (as per this article) but the application still throws an error. I have tried this in the web.config files and directly on the IIS manager.
The reporting server version in 2005 and it is hosted on Windows Server 2008, which is running IIS 7.
On David Lively's advise I tried requesting the URI by putting the parameters in the body. This works for smaller requests, but still fails for large multi-select parameters. The amended code is as follows:
HttpWebRequest webRequest = null;
HttpWebResponse webResponse = null;
string webRequestURL = _ReportManager.GetRSUrl(reportID); //this passes the report link on the SSRS server
string postData = string.Empty;
string URIrequest = string.Empty;
URIrequest = webRequestURL.Substring(0, webRequestURL.IndexOf("&"));
int requestLen = webRequestURL.Length;
int postDataStart = webRequestURL.IndexOf("&") + 1;
postData = webRequestURL.Substring(postDataStart, (requestLen - postDataStart));
Byte[] bytes1 = Encoding.UTF8.GetBytes(postData);
webRequest = (HttpWebRequest)WebRequest.Create(URIrequest);
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.ContentLength = bytes1.Length;
webRequest.Timeout = Configuration.WebRequestTimeOut;
RSExecution2005.ReportExecutionService rsE = new RSExecution2005.ReportExecutionService();
rsE.Url = Configuration.ReportExecutionServiceUrl2005;
rsE.Credentials = System.Net.CredentialCache.DefaultCredentials;
webRequest.Credentials = rsE.Credentials;
Stream reqStream = webRequest.GetRequestStream();
reqStream.Write(bytes1, 0, bytes1.Length);
reqStream.Close();
webResponse = (HttpWebResponse)webRequest.GetResponse();
Even though the requestURI of the webRequest does not store parameters, it seems that the GetReponse() function adds the parameters to the 'address' property of the webRequest. could this be the problem? if so, how can it be fixed.

Is it possible for you to use POST variables instead of GET? That way, there are no limits that I'm aware of, as all of your data will be sent in packets instead of HTTP headers.
Actually it looks like you might be using POST from what's in your code. Can you look in the server logs to verify the URI that is causing this to fail? If you're sending POST data, the request uri shouldn't be an issue unless it's unrelated to the data you're POSTing.

Check your service's binding settings. I guess the service will allow the string upto 8192 length. Set te readerQuotas to a larger size. Might help.
...
<basicHttpBinding>
<binding name="largeBuffer">
<readerQuotas
maxDepth="2147483647"
maxStringContentLength="2147483647"
maxArrayLength="2147483647"
maxBytesPerRead="2147483647"
maxNameTableCharCount="2147483647" />
<security mode="None"></security></binding>
</basicHttpBinding>
.....

Since you're already using POST to fetch the report, I'd suggest putting the parameters that you're currently passing in the query string in the request body, instead. Querystring parameters work fine for a limited number of parameters, but aren't appropriate for a large number of items.

Can you show the value of webRequestURL?
It's going to be "too big".
If you are passing parameters to this URL, can they be in the POST body instead?

webRequestURL.IndexOf("&") ... Is this meant to be "?" instead of "&"? I'm guessing you construct a valid URL for querying the page and then reverse engineer it to be a POST request by looking for the URL before the first '&'...
However, it's possible the GetResponse is appending the body to the URL because it sees the Question Mark in the URL and assumes that the parameters must go in the URL? Try doing a more exact URL match with zero parameters and no '?'.

I got this at work on my IIS7 site. Got it fixed with a registry hack, i can search it up but won't work before 3/1. Meanwhile, try if you get the error when you use the ip-address in stead of the normal URL, when you don't, chances are high it is the same problem.

Had a similar issue, except that POST was working, but second POST with exactly same parameters returned 414.
Setting req.KeepAlive = false; solved the problem, God knows why.

C# How to spoof IP address for WebRequest

I have asp.net website hosted and I am making WebRequest to post data and get response. The website is having IP filtering. I want to spoof sender IP address for testing purpose. Is it possible to do it programmatically or I have to use any tool.
public string GetResponse(string request)
{
lock (Obj)
{
request = request + _dataControlInfo.SendEndingWith;
Logger.Info(request);
var req = (HttpWebRequest)WebRequest.Create(_serviceUrl);
req.Headers.Add("SOAPAction", "\"\"");
req.ContentType = "text/xml;charset=\"utf-8\"";
req.Accept = "text/xml";
req.Method = "POST";
var stm = req.GetRequestStream();
var bytes = UtfEncoding.StringToUtf8ByteArray(request);
stm.Write(bytes, 0, bytes.Length);
stm.Close();
var resp = req.GetResponse();
var stmr = new StreamReader(resp.GetResponseStream());
var strResponseXml = stmr.ReadToEnd();
Logger.Info(strResponseXml);
return strResponseXml;
}
}
Please specify any possibilities.

What your looking for is SharpPCap which is a .NET port of WinPCap.. it allows you to do IP Spoofing, which is what your talking about. The only problem with your idea is that you wont be able to get a response back. You can send requests out, but if you dont have a proper return address then the request will be lost in the interwebs.
Edit
To do this yoruself w/out the help of a library you will need to construct the raw packets yourself. This has been answered here.

If you're expecting to get a response, then no. Without the correct IP address, the server won't send the response to the correct destination.
If you insist on trying anyway, see this article for programmatically setting the client's IP address.

Or you can use Web Performance Tests and a load test with IP Switching enabled

You can try to use a proxy, as documented here.
( http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.proxy.aspx ).
Setting up a proxy on a different computer, then configuring that computer as your requests proxy server should make the request appear as if it came from the proxy's IP, not yours.

Some servers can also consider X-Forwarded-For and X-Real-IP headers.
So if server checks for these headers you can add them to your Web request.
But it depends on server implementation.

Use the Spoof class found in the System.Security namespace...

C# REST client sending data using POST

I'm trying to send a simple POST request to a REST web service and print the response (code is below, mostly taken from Yahoo! developer documentation and the MSDN code snippets provided with some of the documentation). I would expect the client to send:
Request Method: POST (i.e. I expect $_SERVER['REQUEST_METHOD'] == 'POST' in PHP)
Data: foo=bar (i.e. $_POST['foo'] == 'bar' in PHP)
However, it seems to be sending:
Request Method: FOO=BARPOST
Data: (blank)
I know the API works as I've tested it with clients written in Python and PHP, so I'm pretty sure it must be a problem with my C#. I'm not a .NET programmer by trade so would appreciate any comments/pointers on how to figure out what the problem is - I'm sure it's something trivial but I can't spot it myself.
uri, user and password variables are set earlier in the code - they work fine with GET requests.
request = (HttpWebRequest) WebRequest.Create(uri);
request.Credentials = new NetworkCredential(user, password);
request.Method = WebRequestMethods.Http.Post;
request.ContentType = "application/x-www-form-urlencoded";
string postData = "foo=bar";
request.ContentLength = postData.Length;
StreamWriter postStream = new StreamWriter(request.GetRequestStream(), System.Text.Encoding.ASCII);
postStream.Write(postData);
postStream.Close();
response = (HttpWebResponse) request.GetResponse();
The REST API is written in PHP, and the $_POST array is empty on the server when using the C# client.

Eventually found the HttpWebRequest.PreAuthenticate property which seems to solve the problem if the code is edited like so:
request = (HttpWebRequest) WebRequest.Create(uri);
request.PreAuthenticate = true;
request.Credentials = new NetworkCredential(user, password);
request.Method = WebRequestMethods.Http.Post;
From the documentation I presume this forces authentication before the actual POST request is sent. I'm not sure why the class doesn't do this automatically (libraries for other languages make this process transparent, unless you explicitly turn it off), but it has solved the problem for me and may save someone else another 2 days of searching and hair-pulling.
For what it's worth, PreAuthenticate doesn't need to be set for GET requests, only POST, although if you do set it for a GET request everything will still work, but take slightly longer.