This should really be a simple HttpRequestMessage question for most. I'm trying to send a subscription request to a UPnP device as described in section 4.1.2 of the UPnP Device Architecture doc and the message I need to send is supposed to look like the following:
SUBSCRIBE publisher path HTTP/1.1
HOST: publisher host:publisher port
USER-AGENT: OS/version UPnP/1.1 product/version
CALLBACK: <delivery URL>
NT: upnp:event
TIMEOUT: Second-requested subscription duration
obviously most of this is pretty straight forward. There are a few items I have questions on because my current method does not work. I get a NotFound returned from the server.
Is the request line SUBSCRIBE publisher path HTTP/1.1 the same as when I create the WebRequest (WebRequest.Create(...)) or is there property somewhere I can set?
Is SUBSCRIBE the HttpMethod in this case or is this a Get, POST, PUT etc?
Here's the current request code:
var request = WebRequest.Create(new Uri(eventUri)) as HttpWebRequest;
request.Method = "SUBSCRIBE";
request.UserAgent = "MyTab/1.0 UPnP/1.1 TestApp/1.0";
request.Headers["CALLBACK"] = "<" + hostname.DisplayName + ":8088>";
request.Headers["NT"] = "upnp:event";
request.Headers["TIMEOUT"] = "Second-300";
looks like I had it all correct, except I forgot to append the http:// to the callback url.
For others who land here and wonder how UPNP Subscriptions work. This is a very simplified usage example.
I'll use netcat to open a port and respond to incoming events with an HTTP 200. To initialize the connection I'll use curl. This works on an Ubuntu 19.10 with preinstalled curl and nc.
curl -v http://192.168.1.47:58424/RenderingControl/evt -H "CALLBACK: <http://192.168.1.32:1234>" -H "NT: upnp:event" -H "TIMEOUT: Second-1800" -X SUBSCRIBE
Here http://192.168.1.47:58424/RenderingControl/evt is the device specific URL to register to the event. The CALLBACK header specifies the port and IP address on which our netcat server is running. It is important here to add http:// as mentioned before.
The server that listens and sends the HTTP OK is run with the following:
while true; do echo -e 'HTTP/1.1 200 OK\r\nContent-length: 0\r\nConnection: close\r\n\r\n' | nc -l 1234; done
Related
What am I essencially doing is: I input URL (stackoverflow.com) in browser, catch HTTP request with proxy, proceed it to website, then get response from website and relay it back to browser.
In theory it should work just fine, but in practice I get response with "Moved Permanently" and "Location: https://stackoverflow.com/". If I understand it correctly, I need to get this location address and replace old address in HTTP request ("GET http://stackoverflow.com/ HTTP/1.1" -> "GET https://stackoverflow.com/ HTTP/1.1") and then make request again.
However, that's where I stuck. When I make new request, website responses with same status code - 301, no matter what I do.
That's what I am doing at proxy side:
Get list of IP addresses by using Dns.GetHostAddresses() - it returns 4 addresses:
151.101.65.69
151.101.129.69
151.101.193.69
151.101.1.69
Then I connect to each address and send this request (which I got from browser):
"GET http://stackoverflow.com/ HTTP/1.1\r\nHost: stackoverflow.com\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-US,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nCookie: prov=0014938e-7640-15fd-486c-9e6cb05cd5b1\r\nConnection: keep-alive\r\nUpgrade-Insecure-Requests: 1\r\n\r\n"
I then get response:
"HTTP/1.1 301 Moved Permanently\r\nLocation: https://stackoverflow.com/\r\nX-Request-Guid: 080d3919-7d02-4e48-9348-b9dc8b5b08f6\r\nContent-Length: 143\r\nAccept-Ranges: bytes\r\nDate: Mon, 26 Feb 2018 17:56:59 GMT\r\nVia: 1.1 varnish\r\nConnection: keep-alive\r\nX-Served-By: cache-ams4441-AMS\r\nX-Cache: MISS\r\nX-Cache-Hits: 0\r\nX-Timer: S1519667819.162626,VS0,VE76\r\nVary: Fastly-SSL\r\nX-DNS-Prefetch-Control: off\r\n\r\n<html><head><title>Object moved</title></head><body>\r\n<h2>Object moved to here.</h2>\r\n</body></html>\r\n"
But then even if I change "GET http://stackoverflow.com/ HTTP/1.1" to "GET https://stackoverflow.com/ HTTP/1.1", nothing changes, every of this four IPs return same response.
Am I mising something important? Maybe I should include another tag or change something else in new request header?
I am trying to get a few cURL commands translated to a C# implementation, but I am running into an Exception which I can't solve.
I tried to gather up as much information as I can in the hopes that somebody can help me further. Here it goes...
The cURL statement:
curl -i -X PUT "http://[ipaddress]:[port]/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=[username]&op=CREATE"
The C# version:
var response = await client.PutAsync(
"http://[ipaddress]:[port]/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=[username]&op=CREATE",
null);
The C# version results in a WebException:
HResult = -2146233088
Message = The remote name could not be resolved: 'ip-172-31-9-79.eu-central-1.compute.internal'
The server I am connecting to, is a Hadoop server. There are multiple instances running on that server. So when I store a file, the server replies with information on which instance this file is stored (so that I can reference that instance when I want to write to this file).
Based on the error message, it seems to be that it receives some kind of reference to an IP address it can't access (which makes sense, as that ip address is an internal IP address in that Hadoop server.
I used Wireshark to find the difference in the requests which are sent.
Using cURL:
Frame 57: 204 bytes on wire (1632 bits), 204 bytes captured (1632 bits) on interface 0
Ethernet II, Src: IntelCor_da:f4:44 (fc:f8:ae:da:f4:44), Dst: AsustekC_32:7d:b0 (ac:22:0b:32:7d:b0)
Internet Protocol Version 4, Src: 192.168.1.107, Dst: [ipaddress]
Transmission Control Protocol, Src Port: 60454, Dst Port: 50070, Seq: 1, Ack: 1, Len: 150
Hypertext Transfer Protocol
PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n
[Expert Info (Chat/Sequence): PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: PUT
Request URI: /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE
Request URI Path: /webhdfs/v1/[appname]/staging/sensors/aap.txt
Request URI Query: user.name=hdfs&op=CREATE
Request URI Query Parameter: user.name=hdfs
Request URI Query Parameter: op=CREATE
Request Version: HTTP/1.1
Host: [ipaddress]:50070\r\n
User-Agent: curl/7.50.0\r\n
Accept: */*\r\n
\r\n
[Full request URI: http://[ipaddress]:50070/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE]
[HTTP request 1/1]
[Response in frame: 59]
Using the HttpClient:
Frame 381: 209 bytes on wire (1672 bits), 209 bytes captured (1672 bits) on interface 0
Ethernet II, Src: IntelCor_da:f4:44 (fc:f8:ae:da:f4:44), Dst: AsustekC_32:7d:b0 (ac:22:0b:32:7d:b0)
Internet Protocol Version 4, Src: 192.168.1.107, Dst: [ipaddress]
Transmission Control Protocol, Src Port: 60541, Dst Port: 50070, Seq: 1, Ack: 1, Len: 155
Hypertext Transfer Protocol
PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n
[Expert Info (Chat/Sequence): PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: PUT
Request URI: /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE
Request URI Path: /webhdfs/v1/[appname]/staging/sensors/aap.txt
Request URI Query: user.name=hdfs&op=CREATE
Request URI Query Parameter: user.name=hdfs
Request URI Query Parameter: op=CREATE
Request Version: HTTP/1.1
Host: [ipaddress]:50070\r\n
Content-Length: 0\r\n
Connection: Keep-Alive\r\n
\r\n
[Full request URI: http://[ipaddress]:50070/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE]
[HTTP request 1/1]
[Response in frame: 383]
To me, the only notable (but for me meaningless) differences are:
User-Agent: curl/7.50.0\r\n
Accept: /\r\n
vs
Content-Length: 0\r\n
Connection: Keep-Alive\r\n
I have very little experience with REST programming, and clearly not enough knowledge on analyzing web requests. Some help / guidance / explanation would be greatly appreciated.
Just for completeness:
Yes I am aware that there are Windows executables for cURL, but I run on Raspberry PI / Windows IoT. As far as I know there is nothing for that specific platform (yet)
In the above statements I left out ipaddress/appname/etc intentionally just to be safe side wrt security issues
I found out that the only thing which goes wrong is reading the response in C#. The HTTP responses monitored over Wireshark are identical (using cURL.exe and using PutAsync in C#).
I was able to ignore the redirect and be given the opportunity to handle the response myself.
So disabling auto redirect (and handle the response myself) solved my problem:
var httpClientHandler = new HttpClientHandler {AllowAutoRedirect = false};
var client = new HttpClient(httpClientHandler);
I'm using the HttpClient class to send some data to specific host. I just want to send a pure header without any additional lines in it like ("Host: http"). So this line is the last to be removed from the header, but I don't know how.
The code:
HttpRequestMessage msg = new HttpRequestMessage(HttpMethod.Post, aUrl);
msg.Headers.Clear();
msg.Headers.Remove("Host");
msg.Headers.ExpectContinue = false;
Encoding encoding = ConfiguratorASUST.Instance.Encoding ?? Encoding.GetEncoding(ConfiguratorASUST.ENCODING_DEFAULT);
msg.Content = new StringContent(aStr, encoding);
_client.SendAsync(msg);
The result header in Fiddler:
POST http://http//localhost.fiddler:60001/POS/POSTELESPIS HTTP/1.1
Content-Type: text/plain; charset=windows-1251
Host: http
This line Host: http needs to be removed from the message's header. But how on earth can I do that?! I tried the following:
msg.Headers.Clear();
msg.Headers.Remove("Host");
To no avail. Actually I also see the header Proxy-Connection: Keep-Alive being added.
If you carefully inspect your URL, it looks like your it is wrong anyway: http://http// - is your host really named http, and do you really need two slashes after it? Anyway if you fix that, the Host header will carry localhost.fiddler:60001.
By removing the Host header, you're essentially downgrading your request to HTTP/1.0.
You can set the HTTP version in the HttpRequestMessage as explained in Set HTTP protocol version in HttpClient:
msg.Version = HttpVersion.Version10;
But when using Fiddler, it acts as a proxy, and forwards your request as an HTTP/1.1 request - including the host header again. You can also alter the request in Fiddler. This is explained in How do I prevent fiddler from insering "Host" HTTP header?, but note the bold text, emphasis mine:
Per the RFC, as a HTTP/1.1 proxy, Fiddler is required to add a Host header.
It's not clear why this is problematic-- any server that has a problem with this is, by definition, buggy and should be fixed.
You can remove the header if you'd like (although doing so can cause problems elsewhere). Click Rules > Customize Rules. Scroll to OnBeforeRequest and add the following:
if (oSession.oRequest.headers.HTTPVersion == "HTTP/1.0")
{
oSession["x-overridehost"] = oSession.host;
oSession.oRequest.headers.Remove("Host");
}
My app communicates with an internal web API that requires authentication.
When I send the request I get the 401 challenge as expected, the handshake occurs, the authenticated request is re-sent and everything continues fine.
However, I know that the auth is required. Why do I have to wait for the challenge? Can I force the request to send the credentials in the first request?
My request generation is like this:
private static HttpWebRequest BuildRequest(string url, string methodType)
{
var request = HttpWebRequest.CreateHttp(url);
request.PreAuthenticate = true;
request.AuthenticationLevel = AuthenticationLevel.MutualAuthRequested;
request.Credentials = CredentialCache.DefaultNetworkCredentials;
request.Proxy.Credentials = CredentialCache.DefaultNetworkCredentials;
request.ContentType = CONTENT_TYPE;
request.Method = methodType;
request.UserAgent = BuildUserAgent();
return request;
}
Even with this code, the auth header isn't included in the initial request.
I know how to include the auth info with basic.... what I want to do is to use Windows Auth of the user executing the app (so I can't store the password in a config file).
UPDATE I also tried using a HttpClient and its own .Credentials property with the same result: no auth header is added to the initial request.
The only way I could get the auth header in was to hack it in manually using basic authentication (which won't fly for this use-case)
Ntlm is a challenge/response based authentication protocol. You need to make the first request so that the server can issue the challenge then in the subsequent request the client sends the response to the challenge. The server then verifies this response with the domain controller by giving it the challenge and the response that the client sent.
Without knowing the challenge you can't send the response which is why 2 requests are needed.
Basic authentication is password based so you can short circuit this by sending the credentials with the first request but in my experience this can be a problem for some servers to handle.
More details available here:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa378749(v=vs.85).aspx
I'm not 100% sure, but I suspect that there is no way around this; it's simply the way HttpWebRequest works.
In the online .NET source, function DoSubmitRequestProcessing which is here, you can see this comment just after the start of the function, line 1731:
// We have a response of some sort, see if we need to resubmit
// it do to authentication, redirection or something
// else, then handle clearing out state and draining out old response.
A little further down (line 1795) (some lines removed for brevity)
if (resubmit)
{
if (CacheProtocol != null && _HttpResponse != null) CacheProtocol.Reset();
ClearRequestForResubmit(ntlmFollowupRequest);
...
}
And in ClearRequestForResubmit line 5891:
// We're uploading and need to resubmit for Authentication or Redirect.
and then (Line 5923):
// The second NTLM request is required to use the same connection, don't close it
if (ntlmFollowupRequest) {....}
To my (admittedly n00bish) eyes these comments seem to indicate that the developers decided to follow the "standard" challenge-response protocol for NTML/Kerberos and not include any way of sending authentication headers up-front.
Setting PreAuthenticate is what you want, which you are doing. The very first request will still do the handshake but for subsequent requests it will automatically send the credentials (based on the URL being used). You can read up on it here: http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.preauthenticate(v=vs.110).aspx.
Looking for a way to issue an HTTPwebrequest, or use the browser control, or winhttp to make a request to a URL, but override the IP address it connects to from the DNS lookup to a specific one.
Trying to do something similar to the HOSTS file, but programatically without having to modify this file. It can be C# or C+
Why I need it, the host i am sending the request has multiple IPs, and their Domain servers are doing load balancing accross the different IPs. Trying to force the request to a particular IP, but I need the host in the http request to be still the original host. I need this programatically because changing the host file every time i need to run this test is too time consuming.
All you had to do was this:
var request = (HttpWebRequest) WebRequest.Create("http://192.168.1.1");
request.Host = "news.bbc.co.uk";
If I understand correctly you have to make an http request to a web server using virtualhosts but the DNS isn't setup yet so you have to specify the ip address in the url but send something else in the Host: header.
If that's the case you may be able to do so..
In C# using WebProxy:
See Kayode Leonard's answer for .NET 4 and up.
Here's the code I would use if I have my server running on 67.223.227.171:8888 but I need to have www.example.com in the Host: header.
System.Net.WebRequest r = System.Net.WebRequest.Create("http://www.example.com");
r.Proxy = new WebProxy("http://67.223.227.171:8888");
See this link
In C++ using WinHttp:
Using WinHttp you can simply set the Host: header with WinHttpAddRequestHeaders.
So once again if I have my server running on 67.223.227.171:8888 but I need to have www.example.com in the Host: header:
#include <windows.h>
#include <winhttp.h>
#include <assert.h>
int main() {
HINTERNET hSession = WinHttpOpen(L"A WinHTTP Example Program/1.0",
WINHTTP_ACCESS_TYPE_DEFAULT_PROXY,
WINHTTP_NO_PROXY_NAME,
WINHTTP_NO_PROXY_BYPASS, 0);
assert(hSession != NULL);
// Use WinHttpConnect to specify an HTTP server.
HINTERNET hConnect = WinHttpConnect( hSession,
L"67.223.227.171",
8888,
0 );
assert(hConnect != NULL);
// Open and Send a Request Header.
HINTERNET hRequest = WinHttpOpenRequest( hConnect,
L"GET",
L"/downloads/samples/internet/winhttp/retoptions/redirect.asp",
NULL,
WINHTTP_NO_REFERER,
WINHTTP_DEFAULT_ACCEPT_TYPES,
0 );
assert(hRequest != NULL);
BOOL httpResult = WinHttpAddRequestHeaders(
hRequest,
L"Host: www.example.com",
-1L,
0);
assert(httpResult);
httpResult = WinHttpSendRequest( hRequest,
WINHTTP_NO_ADDITIONAL_HEADERS,
0,
WINHTTP_NO_REQUEST_DATA,
0,
0,
0 );
assert(httpResult);
httpResult = WinHttpReceiveResponse( hRequest, NULL );
assert(httpResult);
}
Edited: The class name is WebProxy. Added C# sample code. Added C++ sample code.
[Note, further to Kayode Leonard's answer: A Host property was added to the request in .Net 4.0, making this answer obsolete]
I think you are saying that you want to be able to override the ip address for a given host, without changing the host header.
For example, news.bbc.co.uk maps to IP address 212.58.226.139, but you want to be able to map this to another ip address, while still presenting the same news.bbc.co.uk "Host" http header to the overriden address. This is what you'd acheive by overriding the HOSTS file as you say, which is slightly different to Jason's answer as his won't present the original "Host" http header.
I don't believe you can do this easily (although I'm about to experiment to find out!). Certainly you can't do the following:
var request = (HttpWebRequest) WebRequest.Create("http://192.168.1.1");
request.Headers["Host"] = "news.bbc.co.uk";
as this will fail with an error saying you can't modify the "Host" header.
You probably can do it if your are willing to go down a level below the HttpWebRequest and deal at a more TCP level, but I'm not sure how you'd approach it without going down to that level.
[Edit]: Having played around with various approaches of overriding HttpWebRequest and WebHeaderCollection, I'm pretty sure it can't be done this way. However, Alexandre Jasmin's answer seems to be the solution.
I'm adding an answer that relies on curl and the --resolve command line option.
https://curl.haxx.se/docs/manpage.html#--resolve
From higher level languages, it's usually possible to invoke curl.