I must individually test each server of a cluster. Each server have one IP address but I must use same URL. I am using a console project.
static string RequestGet(string requestUrl, string ipspecify)
{
HttpWebRequest request = HttpWebRequest.CreateHttp(requestUrl);
// Submit the request, and get the response body.
string responseBodyFromRemoteServer;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
responseBodyFromRemoteServer = reader.ReadToEnd();
}
}
return responseBodyFromRemoteServer;
}
Locally, I can modify my host file, but I want to do this automatically through my program.
RequestGet("https://toto.org/myservice", "172.2.240.16")
RequestGet("https://toto.org/myservice", "172.2.240.17")
IP address is server address.
What is the solution?
Turns out, as of .NET 5, this is possible using HttpClient. SocketsHttpHandler, the default handler used by HttpClient, gained a ConnectCallback property which lets you override how a connection to the remote machine is established.
Here's where it's called, which you can use as inspiration for writing your own. The following seems to work fine:
public static async Task Main()
{
using var client = new HttpClient(new SocketsHttpHandler() { ConnectCallback = ConnectCallback });
await client.GetStringAsync("http://example.com");
}
private static async ValueTask<Stream> ConnectCallback(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
{
var endPoint = new DnsEndPoint("1.2.3.4", context.DnsEndPoint.Port);
var socket = new Socket(SocketType.Stream, ProtocolType.Tcp) { NoDelay = true };
await socket.ConnectAsync(endPoint, cancellationToken);
return new NetworkStream(socket, ownsSocket: true);
}
The request is made to 1.2.3.4 (or whatever IP you specify), but everything above the level of the TCP socket itself, including TLS, the Host header, etc, is still set to example.com, meaning that you shouldn't get certificate errors.
Related
I want to download a file using Tor. Most solutions I found require that additional software (e.g. privoxy) is installed and running, but I don't want to have an additional software running all the time even when I don't use my program.
So I tried the Tor.NET library, but I can't get it using Tor. This example shouldn't return my IP-address, but it does:
ClientCreateParams createParams = new ClientCreateParams(#"D:\tor.exe", 9051);
Client client = Client.Create(createParams);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.icanhazip.com/");
request.Proxy = client.Proxy.WebProxy;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
var reader = new StreamReader(response.GetResponseStream());
Console.WriteLine(reader.ReadToEnd());
}
Console.WriteLine("Press enter to exit...");
Console.ReadLine();
There are already multiple comments about this but unfortunally the author of the library isn't active anymore.
Maybe you know what I'm doing wrong (is more configuration neccessary?) or have an idea for an alternative way to download a file using tor.
You follow the Tor project manual, command line HTTPTunnelPort, that you find here: first you must start an HTTP tunnel with
Tor.exe --HTTPTunnelPort 4711
It supplies you with a HTTP tunnel at 127.0.0.1:4711 (see also here). Now you can connect to this proxy:
WebProxy oWebProxy = new WebProxy (IPAddress.Loopback.ToString (), 4711);
WebClient oWebClient = new WebClient ();
oWebClient.Proxy = oWebProxy;
oWebClient.DownloadFile ("https://myUri", "myFilename");
By now, Microsoft recommends the use of the HttpClient for new developments. Here is the code:
// we must configure the HttpClient
HttpClientHandler oHttpClientHandler = new HttpClientHandler ();
oHttpClientHandler.UseProxy = true;
oHttpClientHandler.Proxy =
new WebProxy (IPAddress.Loopback.ToString (), 4711);
// we start an HttpClient with the handler; Microsoft recommends to start
// one HttpClient per application
HttpClient oHttpClient = new HttpClient (oHttpClientHandler);
// we request the resource by the GET method
HttpRequestMessage oHttpRequestMessage =
new HttpRequestMessage (HttpMethod.Get, "https://myUri");
// we make the request and we do only wait for the headers and not for
// content
Task<HttpResponseMessage>
oTaskSendAsync =
oHttpClient.SendAsync
(oHttpRequestMessage, HttpCompletionOption.ResponseHeadersRead);
// we wait for the arrival of the headers
oTaskSendAsync.Wait ();
// the function throws an exception if something went wrong
oTaskSendAsync.Result.EnsureSuccessStatusCode ();
// we can act on the returned headers
HttpResponseHeaders oResponseHeaders = oTaskSendAsync.Result.Headers;
HttpContentHeaders oContentHeaders = oTaskSendAsync.Result.Content.Headers;
// we fetch the content stream
Task<Stream> oTaskResponseStream =
oTaskSendAsync.Result.Content.ReadAsStreamAsync ();
// we open a file for the download data
FileStream oFileStream = File.OpenWrite ("myFilename");
// we delegate the copying of the content to the file
Task oTaskCopyTo = oTaskResponseStream.Result.CopyToAsync (oFileStream);
// and wait for its completion
oTaskCopyTo.Wait ();
// now we can close the file
oFileStream.Close ();
Please heed the following in using Tor.exe:
If the port is already in use Tor.exe will not be able to supply a proxy. It does not even necessarily inform you about this failure.
Make sure that nobody spoofs your program Tor.exe so that it is Tor that supplies you with this proxy. Hence, Tor.exe should be at a secure place in your file system.
Inform yourself about other precautions with respect to using Tor.
At least, you might want to check that your proxy has a different IP address from your local internet connection.
In the end I used https://github.com/Ogglas/SocksWebProxy by #Ogglas to download a file using Tor.
The project has an example which is not working (the first time you start it waits infinitely for Tor to exit, but when you start the program again it can use the Tor process startet by your first try), so I changed it.
I made a Start() method to start Tor:
public async void Start(IProgress<int> progress)
{
torProcess = new Process();
torProcess.StartInfo.FileName = #"D:\...\tor.exe";
torProcess.StartInfo.Arguments = #"-f ""D:\...\torrc""";
torProcess.StartInfo.UseShellExecute = false;
torProcess.StartInfo.RedirectStandardOutput = true;
torProcess.StartInfo.CreateNoWindow = true;
torProcess.Start();
var reader = torProcess.StandardOutput;
while (true)
{
var line = await reader.ReadLineAsync();
if (line == null)
{
// EOF
Environment.Exit(0);
}
// Get loading status
foreach (Match m in Regex.Matches(line, #"Bootstrapped (\d+)%"))
{
progress.Report(Convert.ToInt32(m.Groups[1].Value));
}
if (line.Contains("100%: Done"))
{
// Tor loaded
break;
}
if (line.Contains("Is Tor already running?"))
{
// Tor already running
break;
}
}
proxy = new SocksWebProxy(new ProxyConfig(
//This is an internal http->socks proxy that runs in process
IPAddress.Parse("127.0.0.1"),
//This is the port your in process http->socks proxy will run on
12345,
//This could be an address to a local socks proxy (ex: Tor / Tor Browser, If Tor is running it will be on 127.0.0.1)
IPAddress.Parse("127.0.0.1"),
//This is the port that the socks proxy lives on (ex: Tor / Tor Browser, Tor is 9150)
9150,
//This Can be Socks4 or Socks5
ProxyConfig.SocksVersion.Five
));
progress.Report(100);
}
Afterwards you can use something like this to download something:
public static string DownloadString(string url)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Proxy = proxy;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
return new StreamReader(response.GetResponseStream()).ReadToEnd();
}
}
And when you exit the program you should also kill the Tor process.
I'm using hybrid connections to request data from a listener. If I can write and read to the connection, how can I know that the response I've read from the connection matches the request I've given it? For example:
private HybridConnectionClient _client = new HybridConnectionClient(***);
public override async Task<RelayResponse> SendAsync(RelayRequest request)
{
var stream = await _client.CreateConnectionAsync();
var writer = new StreamWriter(stream) { AutoFlush = true };
var reader = new StreamReader(stream);
var reqestSerialized = JsonConvert.SerializeObject(request);
await writer.WriteLineAsync(reqestSerialized);
string responseSerialized = await reader.ReadLineAsync();
var response = JsonConvert.DeserializeObject<RelayResponse>(responseSerialized);
return response;
}
If the listener on this connection is reading and responding to many requests at the same time, is there anyway to know that the next Readline() we do on the client side to get the response is the one that is associated with the request? Or is that something that has to be managed?
Understanding a bit more about azure relay hybrid connections, I understand this now.
There isn't really any concept of a synchronous request/response in the framework, but if you use a new connection for each request, and respond on that same connection in the listener, you can be sure the response is for the request you sent.
Spawn a new connection for each request, then make sure the response is written to that connection. So looking at Microsoft's listener example code, whenever listener.AcceptConnectionAsync() fires do all the message response on relayConnection, then go back to waiting at await listener.AcceptConnectionAsync();
while (true)
{
var relayConnection = await listener.AcceptConnectionAsync();
if (relayConnection == null)
{
break;
}
ProcessMessagesOnConnection(relayConnection, cts);
}
I want to download a file using Tor. Most solutions I found require that additional software (e.g. privoxy) is installed and running, but I don't want to have an additional software running all the time even when I don't use my program.
So I tried the Tor.NET library, but I can't get it using Tor. This example shouldn't return my IP-address, but it does:
ClientCreateParams createParams = new ClientCreateParams(#"D:\tor.exe", 9051);
Client client = Client.Create(createParams);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.icanhazip.com/");
request.Proxy = client.Proxy.WebProxy;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
var reader = new StreamReader(response.GetResponseStream());
Console.WriteLine(reader.ReadToEnd());
}
Console.WriteLine("Press enter to exit...");
Console.ReadLine();
There are already multiple comments about this but unfortunally the author of the library isn't active anymore.
Maybe you know what I'm doing wrong (is more configuration neccessary?) or have an idea for an alternative way to download a file using tor.
You follow the Tor project manual, command line HTTPTunnelPort, that you find here: first you must start an HTTP tunnel with
Tor.exe --HTTPTunnelPort 4711
It supplies you with a HTTP tunnel at 127.0.0.1:4711 (see also here). Now you can connect to this proxy:
WebProxy oWebProxy = new WebProxy (IPAddress.Loopback.ToString (), 4711);
WebClient oWebClient = new WebClient ();
oWebClient.Proxy = oWebProxy;
oWebClient.DownloadFile ("https://myUri", "myFilename");
By now, Microsoft recommends the use of the HttpClient for new developments. Here is the code:
// we must configure the HttpClient
HttpClientHandler oHttpClientHandler = new HttpClientHandler ();
oHttpClientHandler.UseProxy = true;
oHttpClientHandler.Proxy =
new WebProxy (IPAddress.Loopback.ToString (), 4711);
// we start an HttpClient with the handler; Microsoft recommends to start
// one HttpClient per application
HttpClient oHttpClient = new HttpClient (oHttpClientHandler);
// we request the resource by the GET method
HttpRequestMessage oHttpRequestMessage =
new HttpRequestMessage (HttpMethod.Get, "https://myUri");
// we make the request and we do only wait for the headers and not for
// content
Task<HttpResponseMessage>
oTaskSendAsync =
oHttpClient.SendAsync
(oHttpRequestMessage, HttpCompletionOption.ResponseHeadersRead);
// we wait for the arrival of the headers
oTaskSendAsync.Wait ();
// the function throws an exception if something went wrong
oTaskSendAsync.Result.EnsureSuccessStatusCode ();
// we can act on the returned headers
HttpResponseHeaders oResponseHeaders = oTaskSendAsync.Result.Headers;
HttpContentHeaders oContentHeaders = oTaskSendAsync.Result.Content.Headers;
// we fetch the content stream
Task<Stream> oTaskResponseStream =
oTaskSendAsync.Result.Content.ReadAsStreamAsync ();
// we open a file for the download data
FileStream oFileStream = File.OpenWrite ("myFilename");
// we delegate the copying of the content to the file
Task oTaskCopyTo = oTaskResponseStream.Result.CopyToAsync (oFileStream);
// and wait for its completion
oTaskCopyTo.Wait ();
// now we can close the file
oFileStream.Close ();
Please heed the following in using Tor.exe:
If the port is already in use Tor.exe will not be able to supply a proxy. It does not even necessarily inform you about this failure.
Make sure that nobody spoofs your program Tor.exe so that it is Tor that supplies you with this proxy. Hence, Tor.exe should be at a secure place in your file system.
Inform yourself about other precautions with respect to using Tor.
At least, you might want to check that your proxy has a different IP address from your local internet connection.
In the end I used https://github.com/Ogglas/SocksWebProxy by #Ogglas to download a file using Tor.
The project has an example which is not working (the first time you start it waits infinitely for Tor to exit, but when you start the program again it can use the Tor process startet by your first try), so I changed it.
I made a Start() method to start Tor:
public async void Start(IProgress<int> progress)
{
torProcess = new Process();
torProcess.StartInfo.FileName = #"D:\...\tor.exe";
torProcess.StartInfo.Arguments = #"-f ""D:\...\torrc""";
torProcess.StartInfo.UseShellExecute = false;
torProcess.StartInfo.RedirectStandardOutput = true;
torProcess.StartInfo.CreateNoWindow = true;
torProcess.Start();
var reader = torProcess.StandardOutput;
while (true)
{
var line = await reader.ReadLineAsync();
if (line == null)
{
// EOF
Environment.Exit(0);
}
// Get loading status
foreach (Match m in Regex.Matches(line, #"Bootstrapped (\d+)%"))
{
progress.Report(Convert.ToInt32(m.Groups[1].Value));
}
if (line.Contains("100%: Done"))
{
// Tor loaded
break;
}
if (line.Contains("Is Tor already running?"))
{
// Tor already running
break;
}
}
proxy = new SocksWebProxy(new ProxyConfig(
//This is an internal http->socks proxy that runs in process
IPAddress.Parse("127.0.0.1"),
//This is the port your in process http->socks proxy will run on
12345,
//This could be an address to a local socks proxy (ex: Tor / Tor Browser, If Tor is running it will be on 127.0.0.1)
IPAddress.Parse("127.0.0.1"),
//This is the port that the socks proxy lives on (ex: Tor / Tor Browser, Tor is 9150)
9150,
//This Can be Socks4 or Socks5
ProxyConfig.SocksVersion.Five
));
progress.Report(100);
}
Afterwards you can use something like this to download something:
public static string DownloadString(string url)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Proxy = proxy;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
return new StreamReader(response.GetResponseStream()).ReadToEnd();
}
}
And when you exit the program you should also kill the Tor process.
I have some code that is making a Server2Server call using an HttpClient. Here is some quick code
Code to make a request
private HttpRequestMessage GetRequest(Uri uri, NameValueCollection headers)
{
var method = HttpMethod.Get;
var request = new HttpRequestMessage(method, uri);
foreach (string v in headers)
{
var success = request.Headers.TryAddWithoutValidation(v, headers[v]);
if (!success)
{
// log something ( no logs are appearing )
}
}
return request;
}
Code to make the request
private void AsyncCallUrl(Uri url, NameValueCollection headers)
{
object result1 = null;
var handler = new HttpClientHandler() { AllowAutoRedirect = false };
using (HttpClient client = new HttpClient(handler))
{
var request = GetRequest(url, headers);
using (HttpResponseMessage response = client.SendAsync(request).Result) // when this line is executed the request object's domain is changed to something else
{
using (HttpContent content = response.Content)
{
result1 = content.ReadAsStringAsync().Result;
}
}
}
I've verified that the request object is created appropriately with the proper domain. I've also verified that the network traffic is going to the wrong domain, even the request object shows the new bad domain. What I don't is why this is happening. I've even set the AllowAutoRedirect to false
NOTE: As a note I notice that if I use GetAsync instead of SendAsync the domain change doesn't happen. However this is not an ideal solution as in order to add headers I would need to add them to the HttpClient itself and this code lives on an IIS server and I don't want to make a new client for every request
So, with SendAsync the value of the Host header of the request is determined by the uri parameter... However it is possible to override the Host header through the Headers property of the request.
It's highly likely that the NameValueCollection headers that you are blindly injecting into the request's headers contains an entry for Host which is different to that which you supplied in the Uri.
As an aside, this behaviour can be useful, if (for instance) you were to discover that the DNS performance of HttpWebRequest (the business end of HttpClient on Windows) is sub-standard. I've bypassed .Net/Windows DNS by using a third party library to look up the IP of the host, rewriting the Uri to use the IP address in place of the host name, then setting the Host header on the outgoing request back to the original host name.
I started down the path of using HttpClient as I thought the service I was accessing is a REST service. Turns out it's a JSON service running on port 80 but is a socket application.
The HttpClient opens the remote port but when it sends the JSON request it never gets a response. I was having the hardest time getting fiddler to get a response back as well. But I was able to get wget and curl to send/receiving a response. That's when I talked to the original developer and he mentioned that it wasn't a true "REST" service, but just a socket application that sends/receives JSON.
Is there something I can do to tweak HttpClient to access a socket application or am I going to have to take a step back and use WebSockets?
This is the test code that sends/receives the JSON packet.
private async Task ProcessZone(string szIPAddress)
{
string responseData = string.Empty;
Uri baseAddress = new Uri(#"http://" + szIPAddress + "/player");
try
{
using (var httpClient = new HttpClient { BaseAddress = baseAddress })
{
var _req = new SendRequest();
_req.id = "rec-100";
_req.url = "/stable/av/";
_req.method = "browse";
var _json = JsonConvert.SerializeObject(_req);
using (var content = new StringContent(_json,Encoding.UTF8, "application/json"))
{
using (var response = await httpClient.PostAsync(baseAddress, content))
{
responseData = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
}
}
}
var _Response = JsonConvert.DeserializeObject<Response>(responseData);
var item = new ZoneInfo();
item.szIPAddress = szIPAddress;
item.szZoneName = _Response.result.item.title;
lstZones.Add(item);
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
private async Task ProcessZones()
{
foreach (var item in ZoneSearch)
{
await ProcessZone(item.IPAddress);
}
}
The connection hangs on this line:
using (var response = await httpClient.PostAsync(baseAddress, content))
I should also mention that the code above does work fine on a true rest service...
That's when I talked to the original developer and he mentioned that it wasn't a true "REST" service, but just a socket application that sends/receives JSON.
Knowing the protocol is the first step towards making a working client.
Is there something I can do to tweak HttpClient to access a socket application or am I going to have to take a step back and use WebSockets?
Neither, unfortunately. HttpClient - as the name implies - only works with HTTP services. Since the server is not an HTTP server, it won't work with HttpClient. WebSockets have a rather confusing name, since they are not raw sockets but instead use the WebSocket protocol, which require an HTTP handshake to set up. Since the server is not an HTTP/WebSocket server, it won't work with WebSockets.
Your only choices are to either pressure the developer to write a real REST service (which makes your job orders of magnitude easier), or use raw sockets (e.g., Socket). Correctly using raw sockets is extremely difficult, so I recommend you pressure the developer to write a REST service like the entire rest of the world does today.