I am trying to get a few cURL commands translated to a C# implementation, but I am running into an Exception which I can't solve.
I tried to gather up as much information as I can in the hopes that somebody can help me further. Here it goes...
The cURL statement:
curl -i -X PUT "http://[ipaddress]:[port]/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=[username]&op=CREATE"
The C# version:
var response = await client.PutAsync(
"http://[ipaddress]:[port]/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=[username]&op=CREATE",
null);
The C# version results in a WebException:
HResult = -2146233088
Message = The remote name could not be resolved: 'ip-172-31-9-79.eu-central-1.compute.internal'
The server I am connecting to, is a Hadoop server. There are multiple instances running on that server. So when I store a file, the server replies with information on which instance this file is stored (so that I can reference that instance when I want to write to this file).
Based on the error message, it seems to be that it receives some kind of reference to an IP address it can't access (which makes sense, as that ip address is an internal IP address in that Hadoop server.
I used Wireshark to find the difference in the requests which are sent.
Using cURL:
Frame 57: 204 bytes on wire (1632 bits), 204 bytes captured (1632 bits) on interface 0
Ethernet II, Src: IntelCor_da:f4:44 (fc:f8:ae:da:f4:44), Dst: AsustekC_32:7d:b0 (ac:22:0b:32:7d:b0)
Internet Protocol Version 4, Src: 192.168.1.107, Dst: [ipaddress]
Transmission Control Protocol, Src Port: 60454, Dst Port: 50070, Seq: 1, Ack: 1, Len: 150
Hypertext Transfer Protocol
PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n
[Expert Info (Chat/Sequence): PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: PUT
Request URI: /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE
Request URI Path: /webhdfs/v1/[appname]/staging/sensors/aap.txt
Request URI Query: user.name=hdfs&op=CREATE
Request URI Query Parameter: user.name=hdfs
Request URI Query Parameter: op=CREATE
Request Version: HTTP/1.1
Host: [ipaddress]:50070\r\n
User-Agent: curl/7.50.0\r\n
Accept: */*\r\n
\r\n
[Full request URI: http://[ipaddress]:50070/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE]
[HTTP request 1/1]
[Response in frame: 59]
Using the HttpClient:
Frame 381: 209 bytes on wire (1672 bits), 209 bytes captured (1672 bits) on interface 0
Ethernet II, Src: IntelCor_da:f4:44 (fc:f8:ae:da:f4:44), Dst: AsustekC_32:7d:b0 (ac:22:0b:32:7d:b0)
Internet Protocol Version 4, Src: 192.168.1.107, Dst: [ipaddress]
Transmission Control Protocol, Src Port: 60541, Dst Port: 50070, Seq: 1, Ack: 1, Len: 155
Hypertext Transfer Protocol
PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n
[Expert Info (Chat/Sequence): PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: PUT
Request URI: /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE
Request URI Path: /webhdfs/v1/[appname]/staging/sensors/aap.txt
Request URI Query: user.name=hdfs&op=CREATE
Request URI Query Parameter: user.name=hdfs
Request URI Query Parameter: op=CREATE
Request Version: HTTP/1.1
Host: [ipaddress]:50070\r\n
Content-Length: 0\r\n
Connection: Keep-Alive\r\n
\r\n
[Full request URI: http://[ipaddress]:50070/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE]
[HTTP request 1/1]
[Response in frame: 383]
To me, the only notable (but for me meaningless) differences are:
User-Agent: curl/7.50.0\r\n
Accept: /\r\n
vs
Content-Length: 0\r\n
Connection: Keep-Alive\r\n
I have very little experience with REST programming, and clearly not enough knowledge on analyzing web requests. Some help / guidance / explanation would be greatly appreciated.
Just for completeness:
Yes I am aware that there are Windows executables for cURL, but I run on Raspberry PI / Windows IoT. As far as I know there is nothing for that specific platform (yet)
In the above statements I left out ipaddress/appname/etc intentionally just to be safe side wrt security issues
I found out that the only thing which goes wrong is reading the response in C#. The HTTP responses monitored over Wireshark are identical (using cURL.exe and using PutAsync in C#).
I was able to ignore the redirect and be given the opportunity to handle the response myself.
So disabling auto redirect (and handle the response myself) solved my problem:
var httpClientHandler = new HttpClientHandler {AllowAutoRedirect = false};
var client = new HttpClient(httpClientHandler);
Related
how do I do HTTP tunneling without using the HTTP Connect method but using HTTP Get method instead. https://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_tunneling I tried IHttpClientFactory and RestSharp both are establishing HTTP Connect. In the node.js request module, there is a param called tunnel which when false establish the connection using HTTP Get, how do I do something similar in .NET. Would really appreciate the help, have been looking for this for a while
Capture1 - C# when using a proxy server, http client creates a tunnel using HTTP Connect
Frame 331: 159 bytes on wire (1272 bits), 159 bytes captured (1272 bits) on interface \Device\NPF_{262229C1-C486-4F85-BCC8-BFC96981C755}, id 0
Ethernet II, Src: IntelCor_37:28:37 (dc:41:a9:37:28:37), Dst: TaicangT_80:60:30 (18:45:93:80:60:30)
Internet Protocol Version 4, Src: {myLocalIp}, Dst: {proxyServerIp}
Transmission Control Protocol, Src Port: 51694, Dst Port: 8888, Seq: 1, Ack: 1, Len: 105
Hypertext Transfer Protocol
CONNECT {host}:443 HTTP/1.1\r\n
Host: {host}:443\r\n
User-Agent: RestSharp/106.11.8.0\r\n
\r\n
[Full request URI: {url}:443]
[HTTP request 1/1]
[Response in frame: 341]
Capture2 - node.js with tunnel:false, this request also goes through the same proxy server but via a normal HTTP GET method and no tunnel
Frame 92: 393 bytes on wire (3144 bits), 393 bytes captured (3144 bits) on interface \Device\NPF_{262229C1-C486-4F85-BCC8-BFC96981C755}, id 0
Ethernet II, Src: IntelCor_37:28:37 (dc:41:a9:37:28:37), Dst: TaicangT_80:60:30 (18:45:93:80:60:30)
Internet Protocol Version 4, Src: {myLocalIp}, Dst: {proxyServerIp}
Transmission Control Protocol, Src Port: 51842, Dst Port: 8888, Seq: 1, Ack: 1, Len: 339
Hypertext Transfer Protocol
GET {url} HTTP/1.1\r\n
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15\r\n
x-cache-proxyname: i-05ad6154426f07671\r\n
host: {host}\r\n
Connection: close\r\n
\r\n
[Full request URI: {url}]
[HTTP request 1/1]
[Response in frame: 96]
I have create an httplistener. So i need when client will send me data to read them. The problem is that i dont know how client should send the data
HttpListener listener = new HttpListener();
listener.Prefixes.Add("http://192.168.1.26:8282/");
listener.Prefixes.Add("http://localhost:8282/");
listener.Prefixes.Add("http://127.0.0.1:8282/");
listener.Start();
new Thread(() =>
{
Thread.CurrentThread.IsBackground = true;
for (;;)
{
Console.WriteLine("Listening...");
// Note: The GetContext method blocks while waiting for a request.
HttpListenerContext context = listener.GetContext();
HttpListenerRequest request = context.Request;
string text;
using (var reader = new StreamReader(request.InputStream,
request.ContentEncoding))
{
text = reader.ReadToEnd();
MessageBox.Show(text);
}
// Obtain a response object.
HttpListenerResponse response = context.Response;
// Construct a response.
string responseString = "HelloWorld";
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
// Get a response stream and write the response to it.
response.ContentLength64 = buffer.Length;
System.IO.Stream output = response.OutputStream;
output.Write(buffer, 0, buffer.Length);
// You must close the output stream.
output.Close();
}
}).Start();
}
So from client i send this command:
GET / 192.168.1.26:8282 HTTP/1.0
But i'm getting this message
Recv 34 bytes
SEND OK
+IPD,1,518:HTTP/1.1 400 Bad Request
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Wed, 13 Jun 2018 13:16:03 GMT
Connection: close
Content-Length: 339
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Bad Request</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Bad Request - Invalid Header</h2>
<hr><p>HTTP Error 400. The request has an invalid header name.</p>
</BODY></HTML>
1,CLOSED
I cant understand what is wrong. Also in my code i set to get a message box every time a request will happen. But it never runs
This s what mozilla is sending
You are not attempting to invoke the service correctly. Here is your client request:
GET / 192.168.1.26:8282 HTTP/1.0
What you should be doing is first establishing a socket connection to host 192.168.1.26 over port 8282. Then you must issue a HTTP request in a valid format:
GET / HTTP/1.0
Don't forget to add some newlines after the request (ie: \r\n\r\n). Then your web server should respond to the HTTP request.
Quick example in Telnet:
telnet 192.168.1.26 8282
GET / HTTP/1.0
Quick example with netcat:
nc 192.168.1.26 8282
GET / HTTP/1.0
Note that these quick examples are provided just to help you ensure that your web service is accessible and functioning correctly. Ideally, you would likely use a more robust HTTP client that is customized for whatever your particular needs are. The process is still the same:
Establish a connection to your host IP address over the listening port
Issue a HTTP request in a valid format: (HTTP_VERB PATH HTTP_VERSION)
*) Maybe check out the developer tools in your browser of choice (F12 -> Network) to see how HTTP headers are sent as well.
Parse the response in some meaningful way.
"Also in my code i set to get a message box every time a request will happen." - You should try putting in a manual message to the message box, instead of reading from the input stream. This is a good debugging technique. In a HTTP GET request you generally are not sending data except in the form of optional query string parameters. I have a feeling that you are not getting the results you are expecting because you are reading from input that isn't there. Before reading from the stream input, first make sure that the connection is successful.
What am I essencially doing is: I input URL (stackoverflow.com) in browser, catch HTTP request with proxy, proceed it to website, then get response from website and relay it back to browser.
In theory it should work just fine, but in practice I get response with "Moved Permanently" and "Location: https://stackoverflow.com/". If I understand it correctly, I need to get this location address and replace old address in HTTP request ("GET http://stackoverflow.com/ HTTP/1.1" -> "GET https://stackoverflow.com/ HTTP/1.1") and then make request again.
However, that's where I stuck. When I make new request, website responses with same status code - 301, no matter what I do.
That's what I am doing at proxy side:
Get list of IP addresses by using Dns.GetHostAddresses() - it returns 4 addresses:
151.101.65.69
151.101.129.69
151.101.193.69
151.101.1.69
Then I connect to each address and send this request (which I got from browser):
"GET http://stackoverflow.com/ HTTP/1.1\r\nHost: stackoverflow.com\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-US,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nCookie: prov=0014938e-7640-15fd-486c-9e6cb05cd5b1\r\nConnection: keep-alive\r\nUpgrade-Insecure-Requests: 1\r\n\r\n"
I then get response:
"HTTP/1.1 301 Moved Permanently\r\nLocation: https://stackoverflow.com/\r\nX-Request-Guid: 080d3919-7d02-4e48-9348-b9dc8b5b08f6\r\nContent-Length: 143\r\nAccept-Ranges: bytes\r\nDate: Mon, 26 Feb 2018 17:56:59 GMT\r\nVia: 1.1 varnish\r\nConnection: keep-alive\r\nX-Served-By: cache-ams4441-AMS\r\nX-Cache: MISS\r\nX-Cache-Hits: 0\r\nX-Timer: S1519667819.162626,VS0,VE76\r\nVary: Fastly-SSL\r\nX-DNS-Prefetch-Control: off\r\n\r\n<html><head><title>Object moved</title></head><body>\r\n<h2>Object moved to here.</h2>\r\n</body></html>\r\n"
But then even if I change "GET http://stackoverflow.com/ HTTP/1.1" to "GET https://stackoverflow.com/ HTTP/1.1", nothing changes, every of this four IPs return same response.
Am I mising something important? Maybe I should include another tag or change something else in new request header?
This should really be a simple HttpRequestMessage question for most. I'm trying to send a subscription request to a UPnP device as described in section 4.1.2 of the UPnP Device Architecture doc and the message I need to send is supposed to look like the following:
SUBSCRIBE publisher path HTTP/1.1
HOST: publisher host:publisher port
USER-AGENT: OS/version UPnP/1.1 product/version
CALLBACK: <delivery URL>
NT: upnp:event
TIMEOUT: Second-requested subscription duration
obviously most of this is pretty straight forward. There are a few items I have questions on because my current method does not work. I get a NotFound returned from the server.
Is the request line SUBSCRIBE publisher path HTTP/1.1 the same as when I create the WebRequest (WebRequest.Create(...)) or is there property somewhere I can set?
Is SUBSCRIBE the HttpMethod in this case or is this a Get, POST, PUT etc?
Here's the current request code:
var request = WebRequest.Create(new Uri(eventUri)) as HttpWebRequest;
request.Method = "SUBSCRIBE";
request.UserAgent = "MyTab/1.0 UPnP/1.1 TestApp/1.0";
request.Headers["CALLBACK"] = "<" + hostname.DisplayName + ":8088>";
request.Headers["NT"] = "upnp:event";
request.Headers["TIMEOUT"] = "Second-300";
looks like I had it all correct, except I forgot to append the http:// to the callback url.
For others who land here and wonder how UPNP Subscriptions work. This is a very simplified usage example.
I'll use netcat to open a port and respond to incoming events with an HTTP 200. To initialize the connection I'll use curl. This works on an Ubuntu 19.10 with preinstalled curl and nc.
curl -v http://192.168.1.47:58424/RenderingControl/evt -H "CALLBACK: <http://192.168.1.32:1234>" -H "NT: upnp:event" -H "TIMEOUT: Second-1800" -X SUBSCRIBE
Here http://192.168.1.47:58424/RenderingControl/evt is the device specific URL to register to the event. The CALLBACK header specifies the port and IP address on which our netcat server is running. It is important here to add http:// as mentioned before.
The server that listens and sends the HTTP OK is run with the following:
while true; do echo -e 'HTTP/1.1 200 OK\r\nContent-length: 0\r\nConnection: close\r\n\r\n' | nc -l 1234; done
HttpWebRequest is used to download files from ASP.NET site with basic authentication. Everything works fine in many cases, but some proxies make answer chunked and HttpWebRequest.GetResponse() throws an exception if answer with 401 status code is chunked. The exception is:
System.Net.WebException: The server committed a protocol violation.
Section=ResponseStatusLine
at System.Net.HttpWebRequest.GetResponse()
Trace of answer is:<pre>
"HTTP/1.1 401 Authorization Required\r\nDate: Fri, 26 Jun 2009 04:45:18 GMT\r\nServer: Microsoft-IIS/6.0\r\nX-Powered-By: ASP.NET\r\nX-AspNet-Version: 2.0.50727\r\nWWW-Authenticate: Basic realm=\"iis-server\"\r\nCache-Control: private\r\nContent-Type: text/html; charset=iso-8859-1\r\nVia: 1.1 server\r\nKeep-Alive: timeout=15, max=100\r\nConnection: Keep-Alive\r\nTransfer-Encoding: chunked\r\nContent-Language: en\r\n\r\n0\r\n\r\n0\r\n\r\n
I made test and found out that Transfer-Encoding: chunked is the only one reason of exception. Is this bug of .NET Framework 2.0 or there any RFC says that 401 answer shouldn't be chunked?
I actually think the error isn't that Chunked Transfer Encoding (CTE) isn't allowed, I think it's that the server is doing the CTE incorrectly. If you look at the response body, it's looking like this:
Language: en \r\n
\r\n
0 \r\n
\r\n
0 \r\n
\r\n
If you notice, there are two 0 length chunks in the response body. The CTE RFC requires that the last chunk of a message be a 0 length chunk, and, by extension, no previous chunk can be of zero length. In this message, you have 0 zero length chunks which is a protocol violation of CTE.
This is akin to setting a content length header for a message and then transmitting more data than is allowed by that content length header.