Kerberos, impersonate causes 500 error - c#

Trying to get kerberos to work.
Looking at wireshark output, the windows username is passed to my test script, but when I turn impersonate on in IIS, I get a 500 internal server error.
The script is as follows:
<%# Page Language="C#" Debug="true" %>
<%# Import Namespace="System.Net" %>
<%
WebClient client = new WebClient();
string downloadString = client.DownloadString("http://10.6.2.117/DEV/api/1.5.12077.001/en-GB/8/56/Incident/GetList?%24id=1&StartIndex=0&PageLength=10");
Response.Write(downloadString);
%>
I can access the url directly via a browser with no problems.
With impersonate turned off I get the following wireshark output (line 8 shows username):
"1","0.000000","10.21.4.3","10.6.2.105","TCP","66","59546 → 7001 [SYN] Seq=0 Win=65535 Len=0 MSS=1260 WS=256 SACK_PERM=1"
"2","0.000092","10.6.2.105","10.21.4.3","TCP","66","7001 → 59546 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1"
"3","0.017328","10.21.4.3","10.6.2.105","TCP","60","59546 → 7001 [ACK] Seq=1 Ack=1 Win=262144 Len=0"
"4","0.019120","10.21.4.3","10.6.2.105","HTTP","404","GET /test.aspx HTTP/1.1 , NTLMSSP_NEGOTIATE"
"5","0.104296","10.6.2.105","10.21.4.3","HTTP","1940","HTTP/1.1 401 Unauthorized , NTLMSSP_CHALLENGE (text/html)"
"6","0.123311","10.21.4.3","10.6.2.105","TCP","60","59546 → 7001 [ACK] Seq=351 Ack=1261 Win=262144 Len=0"
"7","0.123314","10.21.4.3","10.6.2.105","TCP","60","59546 → 7001 [ACK] Seq=351 Ack=1887 Win=261376 Len=0"
"8","0.125557","10.21.4.3","10.6.2.105","HTTP","624","GET /test.aspx HTTP/1.1 , NTLMSSP_AUTH, User: EMEA\xxxxxx"
"9","0.183273","10.6.2.105","10.21.4.3","TCP","3834","[TCP segment of a reassembled PDU]"
"10","0.203950","10.21.4.3","10.6.2.105","TCP","60","59546 → 7001 [ACK] Seq=921 Ack=3147 Win=262144 Len=0"
"11","0.203953","10.21.4.3","10.6.2.105","TCP","60","59546 → 7001 [ACK] Seq=921 Ack=4407 Win=262144 Len=0"
"12","0.203955","10.21.4.3","10.6.2.105","TCP","60","59546 → 7001 [ACK] Seq=921 Ack=5667 Win=262144 Len=0"
"13","0.204018","10.6.2.105","10.21.4.3","HTTP","2389","HTTP/1.1 500 Internal Server Error (text/html)"
...
With impersonate turned on, I get (line 4 - no username):
"1","0.000000","10.21.4.3","10.6.2.105","TCP","66","59648 → 7001 [SYN] Seq=0 Win=65535 Len=0 MSS=1260 WS=256 SACK_PERM=1"
"2","0.000111","10.6.2.105","10.21.4.3","TCP","66","7001 → 59648 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1"
"3","0.018178","10.21.4.3","10.6.2.105","TCP","60","59648 → 7001 [ACK] Seq=1 Ack=1 Win=262144 Len=0"
"4","0.019833","10.21.4.3","10.6.2.105","HTTP","404","GET /test.aspx HTTP/1.1 , NTLMSSP_NEGOTIATE"
"5","0.111015","10.6.2.105","10.21.4.3","HTTP","1466","HTTP/1.1 500 Internal Server Error (text/html)"
...
Any help would be appreciated


Well we got there in the end after much fiddling and with the involvement of half a dozen staff members.
So, in case it is of use to anybody else, here are some useful findings:
Ensure the FQDN is used for the all URLs i.e. both in the client browser, as well as the URL being called by the first web server to the second.
IIS Application Pool Settings:
Managed Pipeline = Integrated
Identity = ApplicationPoolIdentity (other options may also work)
IIS Authentication Settings:
Anonymous Authentication = Disabled
ASP .NET Impersonation = Enabled
Basic Authentication = Disabled
Forms Authentication = Disabled
Windows Authentication = Enabled
Advanced Settings:
Extended Protection = Off
Enable Kernel-mode authentication = Off
Providers: Must be in this order. Do not enable just the ‘Negotiate’ option.
Negotiate:Kerberos
NTLM
And finally, a different test script:
<%# Page Language="C#" Debug="true" %>
<%# Import Namespace="System.Net" %>
<%# Import Namespace="System.IO" %>
<%
// The service we wish to consume
string uri = "http://eu9992k8dvweb01.emea.world.net/DEV/api/1.5.12077.001/en-GB/8/56/Incident/GetList?StartIndex=0&PageLength=10";
//Create web request
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri);
//Create credential cache
CredentialCache myCredCache = new CredentialCache();
myCredCache.Add(new Uri(uri), "Negotiate", (NetworkCredential)CredentialCache.DefaultCredentials);
//Add credentials to web request
req.Credentials = myCredCache;
req.Proxy = null;
// create somewhere for the response to go
HttpWebResponse httpResponse = null;
// now use the request
try
{
// get the requested page
httpResponse = (HttpWebResponse)req.GetResponse();
// output what was returned
using (StreamReader streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
Response.Write(streamReader.ReadToEnd());
}
}
finally
{
// close the response object
if (httpResponse != null)
httpResponse.Close();
}
%>

Related

FluentFTP EPSV Error 425 Can't open data connection for transfer of "/test.csv"

I'm trying to upload a file with the .NET Library FluentFTP using EPSV connection type because I'm behind a HTTP/1.1 proxy and data and control FTP IP addresses are different.
Unfortunately I get the following error when calling the UploadFile method:
Response: 425 Can't open data connection for transfer of "/test.csv"
The same operation works in FileZilla Client with the same proxy settings, so it can't be a network problem.
This is my code:
using (FtpClientProxy client = new FtpClientHttp11Proxy(new ProxyInfo() {
Host = "prox.corp.company.com",
Port = 80,
Credentials = new NetworkCredential("proxyuser", "password")})) {
client.Host = "1.123.123.123";
client.Port = 990;
client.Credentials = new NetworkCredential("ftpuser", "password");
client.SslProtocols = System.Security.Authentication.SslProtocols.Tls12;
client.DataConnectionType = FtpDataConnectionType.EPSV;
client.EncryptionMode = FtpEncryptionMode.Implicit;
client.Connect();
client.UploadFile(#"C:\test.csv", "/test.csv");
}
The comparison between FluentFTP log files and FileZilla Client log files shows the same operations.
FluentFTP log:
Command: SIZE /test.csv
Response: 550 File not found
OpenWrite("/test.csv", Binary)
Command: TYPE I
Response: 200 Type set to I
OpenPassiveDataStream(EPSV, "STOR /test.csv", 0)
Command: EPSV
Response: 229 Entering Extended Passive Mode (|||40160|)
Status: Connecting to 1.123.1.123:80 // HTTP-Proxy
HTTP/1.1 200 Connection established
Command: STOR /test.csv
Response: 425 Can't open data connection for transfer of "/test.csv"
Status: Disposing FtpSocketStream...
FileZilla client log:
Status: Starting upload of C:\test.csv
Command: CWD /
Response: 250 CWD successful. "/" is current directory.
Command: TYPE I
Response: 200 Type set to I
Command: EPSV
Response: 229 Entering Extended Passive Mode (|||40132|)
Command: STOR test.csv
Status: Connection with proxy established, performing handshake...
Response: Proxy reply: HTTP/1.1 200 Connection established
Response: 150 Opening data channel for file upload to server of "/test.csv"
Response: 226 Successfully transferred "/test.csv"
Status: File transfer successful, transferred 24 bytes in 1 second

I had the same problem and managed to upload the file with those settings:
ftpClient.SslProtocols = SslProtocols.None | SslProtocols.Tls12;
ftpClient.ValidateAnyCertificate = true;
ftpClient.DataConnectionEncryption = false;
I think the key here is DataConnectionEncryption.

HTTP GET instead of HTTP Connect in .NET / C#

how do I do HTTP tunneling without using the HTTP Connect method but using HTTP Get method instead. https://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_tunneling I tried IHttpClientFactory and RestSharp both are establishing HTTP Connect. In the node.js request module, there is a param called tunnel which when false establish the connection using HTTP Get, how do I do something similar in .NET. Would really appreciate the help, have been looking for this for a while
Capture1 - C# when using a proxy server, http client creates a tunnel using HTTP Connect
Frame 331: 159 bytes on wire (1272 bits), 159 bytes captured (1272 bits) on interface \Device\NPF_{262229C1-C486-4F85-BCC8-BFC96981C755}, id 0
Ethernet II, Src: IntelCor_37:28:37 (dc:41:a9:37:28:37), Dst: TaicangT_80:60:30 (18:45:93:80:60:30)
Internet Protocol Version 4, Src: {myLocalIp}, Dst: {proxyServerIp}
Transmission Control Protocol, Src Port: 51694, Dst Port: 8888, Seq: 1, Ack: 1, Len: 105
Hypertext Transfer Protocol
CONNECT {host}:443 HTTP/1.1\r\n
Host: {host}:443\r\n
User-Agent: RestSharp/106.11.8.0\r\n
\r\n
[Full request URI: {url}:443]
[HTTP request 1/1]
[Response in frame: 341]
Capture2 - node.js with tunnel:false, this request also goes through the same proxy server but via a normal HTTP GET method and no tunnel
Frame 92: 393 bytes on wire (3144 bits), 393 bytes captured (3144 bits) on interface \Device\NPF_{262229C1-C486-4F85-BCC8-BFC96981C755}, id 0
Ethernet II, Src: IntelCor_37:28:37 (dc:41:a9:37:28:37), Dst: TaicangT_80:60:30 (18:45:93:80:60:30)
Internet Protocol Version 4, Src: {myLocalIp}, Dst: {proxyServerIp}
Transmission Control Protocol, Src Port: 51842, Dst Port: 8888, Seq: 1, Ack: 1, Len: 339
Hypertext Transfer Protocol
GET {url} HTTP/1.1\r\n
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15\r\n
x-cache-proxyname: i-05ad6154426f07671\r\n
host: {host}\r\n
Connection: close\r\n
\r\n
[Full request URI: {url}]
[HTTP request 1/1]
[Response in frame: 96]

How to diagnose why async TCP System.Net.Socket send (Begin/End Send) not actually sending anything?

I have an HTTP server and client (they are my code both, C#, HttpListener on the server, HttpWebrequest on a client).
Every client makes approx 6 HTTP requests to the server per minute (if all is OK). There can be up to 10000 clients per server. The server should respond less than 1 second. Server logic is simple (there are no CPU issues). The most code is async.
All works fine in several environments (Intranet).
But in some Intranet environment, approximately 1 of 3 POST HTTP web requests hangs.
The issue is manifested on single client computer under VMware (Win 8.1 Professional, 6.3.9600 build 9600 .net 4.7.1)
I cannot reproduce it on another computer and cannot move this VM to another network environment. So I need to get some more detailed info.
The issue is detailed below. It is very like as described at
Async TCP System.Net.Socket send (Begin/End Send) not actually sending anything several years ago.
I have got WireShark and detailed System.Net logs from client and server as recommended by Microsoft.
<source name="LowLevelDesign" switchValue="Verbose">
<source name="System.Net.Http" switchValue="Verbose">
<source name="System.Net.HttpListener" switchValue="Verbose">
<source name="System.Net" switchValue="Verbose">
<source name="System.Net.Sockets" switchValue="Verbose">
Several POST HttpWebRequests from client on the same socket work fine, i.e. log shows the same client port and socket ID between requests.
I see all TCP packets in Wireshark OK both on server and client (The only warning on the server is Checksum: 0xd28c incorrect, should be 0xb54c(maybe caused by "TCP checksum offload"?).
The server responds greatly less than 1 sec for all requests.
Then after 3 or 4 successful requests, the client tries to send the next request.
But there are no TCP packets in Wireshark (nor at server nor at the client). So there is no answer from the server due to it does not see the request.
My code detects timeout (10 seconds) without answer and aborts the HttpWebRequest.
Then the same (new) requests works well on other new sockets (i.e. new client port). But after 3-6 requests the issue repeats.
There are no messages with length > 1500. The hanged request is short message too (<100 bytes for headers and <200 bytes for POST request stream data, <400 bytes for response). I see no differences between OK and Hanged requests except OK is first and Hanged is second with 100-continue. All other requests has no request data (i.e. request content-length==0). I see that the second request with 100-continue is hanged always in that environment.
The old connection is not disposed properly due to I see that server sends [RST, ACK] to the old socket (old client port) 2 minutes later. And I do not know how to close it gracefully.
...
System.Net.Sockets Verbose: 0 : [6420] Data from Socket#11429296::PostCompletion
DateTime=2019-06-24T06:43:08.7370581Z
...
System.Net.Sockets Verbose: 0 : [6420] 000000A0 : 35 0D 0A 45 78 70 65 63-74 3A 20 31 30 30 2D 63 : 5..Expect: 100-c
DateTime=2019-06-24T06:43:08.7410625Z
System.Net.Sockets Verbose: 0 : [6420] Socket#11429296::EndSend(OverlappedAsyncResult#13804354)
DateTime=2019-06-24T06:43:08.7420738Z
System.Net.Sockets Verbose: 0 : [6420] Exiting Socket#11429296::EndSend() -> Int32#187
DateTime=2019-06-24T06:43:08.7420738Z
System.Net.Sockets Verbose: 0 : [6420] Socket#11429296::UnsafeBeginReceive()
DateTime=2019-06-24T06:43:08.7420738Z
System.Net.Sockets Verbose: 0 : [6420] Exiting Socket#11429296::UnsafeBeginReceive() -> OverlappedAsyncResult#47962264
DateTime=2019-06-24T06:43:08.7431561Z
System.Net Verbose: 0 : [5896] HttpWebRequest#52253787::EndGetRequestStream()
DateTime=2019-06-24T06:43:09.1071980Z
System.Net Verbose: 0 : [5896] Exiting HttpWebRequest#52253787::EndGetRequestStream() -> ConnectStream#17987329
DateTime=2019-06-24T06:43:09.1081881Z
System.Net Verbose: 0 : [5896] ConnectStream#17987329::BeginWrite()
DateTime=2019-06-24T06:43:09.1081881Z
System.Net Verbose: 0 : [5896] Data from ConnectStream#17987329::BeginWrite
DateTime=2019-06-24T06:43:09.1091858Z
System.Net Verbose: 0 : [5896] 00000000 : 7B 22 43 6F 6E 6E 65 63-74 69 6F 6E 73 22 3A 7B : {"Connections":{
DateTime=2019-06-24T06:43:09.1091858Z
...
System.Net.Sockets Verbose: 0 : [5896] Socket#11429296::BeginSend()
DateTime=2019-06-24T06:43:09.1140970Z
System.Net.Sockets Verbose: 0 : [5896] Exiting Socket#11429296::BeginSend() -> OverlappedAsyncResult#60375305
DateTime=2019-06-24T06:43:09.1151910Z
System.Net Verbose: 0 : [5896] Exiting ConnectStream#17987329::BeginWrite() -> NestedSingleAsyncResult#46228029
DateTime=2019-06-24T06:43:09.1160992Z
System.Net.Sockets Verbose: 0 : [6420] Data from Socket#11429296::PostCompletion
DateTime=2019-06-24T06:43:09.1180991Z
System.Net.Sockets Verbose: 0 : [6420] 00000000 : 7B 22 43 6F 6E 6E 65 63-74 69 6F 6E 73 22 3A 7B : {"Connections":{
DateTime=2019-06-24T06:43:09.1180991Z
...
System.Net.Sockets Verbose: 0 : [6420] Socket#11429296::EndSend(OverlappedAsyncResult#60375305)
DateTime=2019-06-24T06:43:09.1212220Z
System.Net.Sockets Verbose: 0 : [6420] Exiting Socket#11429296::EndSend() -> Int32#125
DateTime=2019-06-24T06:43:09.1212220Z
System.Net Verbose: 0 : [6420] ConnectStream#17987329::EndWrite()
DateTime=2019-06-24T06:43:09.1212220Z
System.Net Verbose: 0 : [6420] Exiting ConnectStream#17987329::EndWrite()
DateTime=2019-06-24T06:43:09.1222229Z
System.Net Verbose: 0 : [6420] ConnectStream#17987329::Close()
DateTime=2019-06-24T06:43:09.1222229Z
System.Net Verbose: 0 : [6420] Exiting ConnectStream#17987329::Close()
DateTime=2019-06-24T06:43:09.1232215Z
System.Net Verbose: 0 : [7368] HttpWebRequest#52253787::BeginGetResponse()
DateTime=2019-06-24T06:43:09.1232215Z
System.Net Verbose: 0 : [7368] Exiting HttpWebRequest#52253787::BeginGetResponse() -> ContextAwareResult#13009416
DateTime=2019-06-24T06:43:09.1241884Z
System.Net Verbose: 0 : [7368] HttpWebRequest#52253787::Abort()
DateTime=2019-06-24T06:43:18.7330583Z
Wireshark:
Protocol Length Info
TCP 66 56916 → 33444 [SYN] Seq=0 Win=8192 Len=0 MSS=1432 WS=256 SACK_PERM=1
TCP 66 33444 → 56916 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1404 WS=256 SACK_PERM=1
TCP 54 56916 → 33444 [ACK] Seq=1 Ack=1 Win=65792 Len=0
HTTP 248 POST /Service_v2/GetServerInformation/ HTTP/1.1
TCP 60 33444 → 56916 [ACK] Seq=1 Ack=195 Win=65792 Len=0
HTTP 438 HTTP/1.1 200 OK (text/plain)
TCP 54 56916 → 33444 [ACK] Seq=195 Ack=385 Win=65536 Len=0
HTTP 220 POST /Service_v2/GetServerLicense/ HTTP/1.1
TCP 60 33444 → 56916 [ACK] Seq=385 Ack=361 Win=65792 Len=0
HTTP 1252 HTTP/1.1 200 OK (text/plain)
TCP 54 56916 → 33444 [ACK] Seq=361 Ack=1583 Win=65792 Len=0
HTTP 220 POST /Service_v2/GetSecondMessage/ HTTP/1.1
TCP 60 33444 → 56916 [ACK] Seq=1583 Ack=527 Win=65536 Len=0
HTTP 858 HTTP/1.1 200 OK (text/plain)
HTTP 265 POST /Service_v2/GetMainMessage/?messageid=10d5d406-634e-4b65-9d99-f614e8b8398b HTTP/1.1
TCP 60 33444 → 56916 [ACK] Seq=2387 Ack=738 Win=65280 Len=0
HTTP 836 HTTP/1.1 200 OK (text/plain)
TCP 241 56916 → 33444 [PSH, ACK] Seq=738 Ack=3169 Win=65792 Len=187 [TCP segment of a reassembled PDU]
TCP 60 33444 → 56916 [ACK] Seq=3169 Ack=925 Win=65024 Len=0
HTTP 79 HTTP/1.1 100 Continue
HTTP 181 POST /Service_v2/ProcessMessage/ HTTP/1.1 (application/raw)
TCP 60 33444 → 56916 [ACK] Seq=3194 Ack=1052 Win=65024 Len=0
HTTP 372 HTTP/1.1 200 OK (text/plain)
TCP 54 56916 → 33444 [ACK] Seq=1052 Ack=3512 Win=65536 Len=0
HTTP 220 POST /Service_v2/GetSecondMessage/ HTTP/1.1
TCP 60 33444 → 56916 [ACK] Seq=3512 Ack=1218 Win=64768 Len=0
HTTP 882 HTTP/1.1 200 OK (text/plain)
TCP 54 56916 → 33444 [ACK] Seq=1218 Ack=4340 Win=64768 Len=0
Here I try to make new POST /Service_v2/ProcessMessage/ HTTP/1.1 (application/raw) but there are NO data in WireShark
I see only next line ([RST, ACK]) after 2 minutes, no [FIN], no [ASK], no [PSH]
TCP 60 33444 → 56916 [RST, ACK] Seq=4340 Ack=1218 Win=0 Len=0
Source code:
public ServerHttpRequest(string methodSuffix, NameValueCollection requestParams = null, string requestBody = null)
{
if (methodSuffix == null)
throw new AggregateException(nameof(methodSuffix));
_methodSuffix = methodSuffix;
_requestParams = requestParams;
_requestBody = requestBody;
}
public async Task<string> SendRequest()
{
var sendRequestTask = Send();
if (await Task.WhenAny(sendRequestTask, Task.Delay(TimeSpan.FromSeconds(Configuration.ServerTimeout))) == sendRequestTask)
return await sendRequestTask.ConfigureAwait(false);
ThreadPool.QueueUserWorkItem(this.Abort);
throw new TimeoutException("HttpRequest timeout exceeded. Probably having problems connecting to the server.");
}
private void Abort(object obj)
{
try
{
_httpRequest?.Abort();
}
catch (Exception ex)
{
Log(ex);
}
}
private HttpWebRequest CreateRequest(string methodSuffix, NameValueCollection requestParams = null)
{
string url = GetUrl(methodSuffix)
if (requestParams != null && requestParams.Count > 0)
serviceAdress = AddParamsToUrl(url, requestParams);
var webRequest = WebRequest.CreateHttp(licServiceAdress);
webRequest.Method = "POST";
//The Timeout property indicates the length of time, in milliseconds,
//until the request times out and throws a WebException.
//The Timeout property affects only synchronous requests made with the GetResponse method.
//https://learn.microsoft.com/en-us/dotnet/api/system.net.webrequest.timeout?view=netframework-4.8#remarks
webRequest.Timeout = (int)TimeSpan.FromSeconds(Configuration.ServerTimeout).TotalMilliseconds;
webRequest.Proxy = null;
webRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
webRequest.Headers.Add("Accept-Encoding", "gzip");
return webRequest;
}
private async Task<string> Send()
{
try
{
_httpRequest = await Task.Run(() => CreateRequest(_methodSuffix, _requestParams)).ConfigureAwait(false);
UTF8Encoding encoding = new UTF8Encoding();
byte[] requestBodyBytes = null;
if (_requestBody == null)
requestBodyBytes = encoding.GetBytes("");
else
requestBodyBytes = encoding.GetBytes(_requestBody);
_httpRequest.ContentType = "application/raw";
_httpRequest.ContentLength = requestBodyBytes.Length;
using (Stream newStream = await _httpRequest.GetRequestStreamAsync().ConfigureAwait(false))
{
await newStream.WriteAsync(requestBodyBytes, 0, requestBodyBytes.Length).ConfigureAwait(false);
}
using (var webResponse = (HttpWebResponse)await _httpRequest.GetResponseAsync().ConfigureAwait(false))
{
using (var responseStream = webResponse.GetResponseStream())
if (responseStream != null)
using (var reader = new StreamReader(responseStream))
return await reader.ReadToEndAsync().ConfigureAwait(false);
return "";
}
}
catch (Exception ex)
{
Log($"Request failed. Uri: '{_httpRequest.RequestUri.AbsoluteUri}'. {ex}");
throw;
}
}
I need to know why socket data are not shown in Wireshark. How to diagnose this?

Translating cURL to HttpClient.PutAsync

I am trying to get a few cURL commands translated to a C# implementation, but I am running into an Exception which I can't solve.
I tried to gather up as much information as I can in the hopes that somebody can help me further. Here it goes...
The cURL statement:
curl -i -X PUT "http://[ipaddress]:[port]/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=[username]&op=CREATE"
The C# version:
var response = await client.PutAsync(
"http://[ipaddress]:[port]/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=[username]&op=CREATE",
null);
The C# version results in a WebException:
HResult = -2146233088
Message = The remote name could not be resolved: 'ip-172-31-9-79.eu-central-1.compute.internal'
The server I am connecting to, is a Hadoop server. There are multiple instances running on that server. So when I store a file, the server replies with information on which instance this file is stored (so that I can reference that instance when I want to write to this file).
Based on the error message, it seems to be that it receives some kind of reference to an IP address it can't access (which makes sense, as that ip address is an internal IP address in that Hadoop server.
I used Wireshark to find the difference in the requests which are sent.
Using cURL:
Frame 57: 204 bytes on wire (1632 bits), 204 bytes captured (1632 bits) on interface 0
Ethernet II, Src: IntelCor_da:f4:44 (fc:f8:ae:da:f4:44), Dst: AsustekC_32:7d:b0 (ac:22:0b:32:7d:b0)
Internet Protocol Version 4, Src: 192.168.1.107, Dst: [ipaddress]
Transmission Control Protocol, Src Port: 60454, Dst Port: 50070, Seq: 1, Ack: 1, Len: 150
Hypertext Transfer Protocol
PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n
[Expert Info (Chat/Sequence): PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: PUT
Request URI: /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE
Request URI Path: /webhdfs/v1/[appname]/staging/sensors/aap.txt
Request URI Query: user.name=hdfs&op=CREATE
Request URI Query Parameter: user.name=hdfs
Request URI Query Parameter: op=CREATE
Request Version: HTTP/1.1
Host: [ipaddress]:50070\r\n
User-Agent: curl/7.50.0\r\n
Accept: */*\r\n
\r\n
[Full request URI: http://[ipaddress]:50070/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE]
[HTTP request 1/1]
[Response in frame: 59]
Using the HttpClient:
Frame 381: 209 bytes on wire (1672 bits), 209 bytes captured (1672 bits) on interface 0
Ethernet II, Src: IntelCor_da:f4:44 (fc:f8:ae:da:f4:44), Dst: AsustekC_32:7d:b0 (ac:22:0b:32:7d:b0)
Internet Protocol Version 4, Src: 192.168.1.107, Dst: [ipaddress]
Transmission Control Protocol, Src Port: 60541, Dst Port: 50070, Seq: 1, Ack: 1, Len: 155
Hypertext Transfer Protocol
PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n
[Expert Info (Chat/Sequence): PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: PUT
Request URI: /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE
Request URI Path: /webhdfs/v1/[appname]/staging/sensors/aap.txt
Request URI Query: user.name=hdfs&op=CREATE
Request URI Query Parameter: user.name=hdfs
Request URI Query Parameter: op=CREATE
Request Version: HTTP/1.1
Host: [ipaddress]:50070\r\n
Content-Length: 0\r\n
Connection: Keep-Alive\r\n
\r\n
[Full request URI: http://[ipaddress]:50070/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE]
[HTTP request 1/1]
[Response in frame: 383]
To me, the only notable (but for me meaningless) differences are:
User-Agent: curl/7.50.0\r\n
Accept: /\r\n
vs
Content-Length: 0\r\n
Connection: Keep-Alive\r\n
I have very little experience with REST programming, and clearly not enough knowledge on analyzing web requests. Some help / guidance / explanation would be greatly appreciated.
Just for completeness:
Yes I am aware that there are Windows executables for cURL, but I run on Raspberry PI / Windows IoT. As far as I know there is nothing for that specific platform (yet)
In the above statements I left out ipaddress/appname/etc intentionally just to be safe side wrt security issues

I found out that the only thing which goes wrong is reading the response in C#. The HTTP responses monitored over Wireshark are identical (using cURL.exe and using PutAsync in C#).
I was able to ignore the redirect and be given the opportunity to handle the response myself.
So disabling auto redirect (and handle the response myself) solved my problem:
var httpClientHandler = new HttpClientHandler {AllowAutoRedirect = false};
var client = new HttpClient(httpClientHandler);

SSL Handshake Timeout

I have 2 client authentication certificates issued by the same certificate authority. One of them enables me to connect to a HTTPS webservice, but the other does not when I use code similar to the following:
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create( endPointUrl );
X509Store store = new X509Store( StoreName.My, StoreLocation.LocalMachine );
store.Open( OpenFlags.MaxAllowed );
X509CertificateCollection col = (X509CertificateCollection)store.Certificates.Find( X509FindType.FindBySerialNumber, certificateSerialNumber, true );
httpWebRequest.ClientCertificates.Add( col[0] );
httpWebRequest.Method = "POST";
httpWebRequest.ContentType = contentType;
httpWebRequest.KeepAlive = false;
httpWebRequest.Timeout = 3000;
httpWebRequest.ContentLength = message.Length;
httpRequestStream = httpWebRequest.GetRequestStream();
When attempting to get the request stream, I get an InvalidOperationException with the message "The operation has timed out".
I've used System.Net.trace when attempting to connect with the certificate that fails and the log shows a connection timeout before "Attempting to restart the session using the user-provided certificate" and just after the first InitializeSecurityContext.
Wireshark show the following:
"TCP","j-link > https [SYN] Seq=0 Win=65535 Len=0 MSS=1260 SACK_PERM=1"
"TCP","https > j-link [SYN, ACK] Seq=0 Ack=1 Win=32768 Len=0 MSS=1380"
"TCP","j-link > https [ACK] Seq=1 Ack=1 Win=65535 Len=0"
"TLSv1","Client Hello"
"TLSv1","Server Hello"
"TCP","[TCP segment of a reassembled PDU]"
"TCP","j-link > https [ACK] Seq=78 Ack=2521 Win=65535 Len=0"
"TLSv1","Certificate, Certificate Request, Server Hello Done"
"TCP","j-link > https [ACK] Seq=78 Ack=3187 Win=64869 Len=0"
"TCP","j-link > https [FIN, ACK] Seq=78 Ack=3187 Win=64869 Len=0"
"TCP","https > j-link [ACK] Seq=3187 Ack=79 Win=32768 Len=0"
"TLSv1","Alert (Level: Warning, Description: Close Notify)"
"TCP","j-link > https [RST, ACK] Seq=79 Ack=3194 Win=0 Len=0"
I can connect using OpenSSL from the command line using both certificates after exporting them and converting them to the PEM format.
Any suggestions would be greatly appreciated.

Thank you to Shawn's question which helped me fix the timeout problem, which was due to the connection taking over 60 seconds to fail.
I then got a "The request was aborted: Could not create SSL/TLS secure channel." error, which was solved by using the Windows HTTP Services Certificate Configuration Tool and information I obtained here.

Categories

Resources