I'm trying to upload a file with the .NET Library FluentFTP using EPSV connection type because I'm behind a HTTP/1.1 proxy and data and control FTP IP addresses are different.
Unfortunately I get the following error when calling the UploadFile method:
Response: 425 Can't open data connection for transfer of "/test.csv"
The same operation works in FileZilla Client with the same proxy settings, so it can't be a network problem.
This is my code:
using (FtpClientProxy client = new FtpClientHttp11Proxy(new ProxyInfo() {
Host = "prox.corp.company.com",
Port = 80,
Credentials = new NetworkCredential("proxyuser", "password")})) {
client.Host = "1.123.123.123";
client.Port = 990;
client.Credentials = new NetworkCredential("ftpuser", "password");
client.SslProtocols = System.Security.Authentication.SslProtocols.Tls12;
client.DataConnectionType = FtpDataConnectionType.EPSV;
client.EncryptionMode = FtpEncryptionMode.Implicit;
client.Connect();
client.UploadFile(#"C:\test.csv", "/test.csv");
}
The comparison between FluentFTP log files and FileZilla Client log files shows the same operations.
FluentFTP log:
Command: SIZE /test.csv
Response: 550 File not found
OpenWrite("/test.csv", Binary)
Command: TYPE I
Response: 200 Type set to I
OpenPassiveDataStream(EPSV, "STOR /test.csv", 0)
Command: EPSV
Response: 229 Entering Extended Passive Mode (|||40160|)
Status: Connecting to 1.123.1.123:80 // HTTP-Proxy
HTTP/1.1 200 Connection established
Command: STOR /test.csv
Response: 425 Can't open data connection for transfer of "/test.csv"
Status: Disposing FtpSocketStream...
FileZilla client log:
Status: Starting upload of C:\test.csv
Command: CWD /
Response: 250 CWD successful. "/" is current directory.
Command: TYPE I
Response: 200 Type set to I
Command: EPSV
Response: 229 Entering Extended Passive Mode (|||40132|)
Command: STOR test.csv
Status: Connection with proxy established, performing handshake...
Response: Proxy reply: HTTP/1.1 200 Connection established
Response: 150 Opening data channel for file upload to server of "/test.csv"
Response: 226 Successfully transferred "/test.csv"
Status: File transfer successful, transferred 24 bytes in 1 second
I had the same problem and managed to upload the file with those settings:
ftpClient.SslProtocols = SslProtocols.None | SslProtocols.Tls12;
ftpClient.ValidateAnyCertificate = true;
ftpClient.DataConnectionEncryption = false;
I think the key here is DataConnectionEncryption.
how do I do HTTP tunneling without using the HTTP Connect method but using HTTP Get method instead. https://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_tunneling I tried IHttpClientFactory and RestSharp both are establishing HTTP Connect. In the node.js request module, there is a param called tunnel which when false establish the connection using HTTP Get, how do I do something similar in .NET. Would really appreciate the help, have been looking for this for a while
Capture1 - C# when using a proxy server, http client creates a tunnel using HTTP Connect
Frame 331: 159 bytes on wire (1272 bits), 159 bytes captured (1272 bits) on interface \Device\NPF_{262229C1-C486-4F85-BCC8-BFC96981C755}, id 0
Ethernet II, Src: IntelCor_37:28:37 (dc:41:a9:37:28:37), Dst: TaicangT_80:60:30 (18:45:93:80:60:30)
Internet Protocol Version 4, Src: {myLocalIp}, Dst: {proxyServerIp}
Transmission Control Protocol, Src Port: 51694, Dst Port: 8888, Seq: 1, Ack: 1, Len: 105
Hypertext Transfer Protocol
CONNECT {host}:443 HTTP/1.1\r\n
Host: {host}:443\r\n
User-Agent: RestSharp/106.11.8.0\r\n
\r\n
[Full request URI: {url}:443]
[HTTP request 1/1]
[Response in frame: 341]
Capture2 - node.js with tunnel:false, this request also goes through the same proxy server but via a normal HTTP GET method and no tunnel
Frame 92: 393 bytes on wire (3144 bits), 393 bytes captured (3144 bits) on interface \Device\NPF_{262229C1-C486-4F85-BCC8-BFC96981C755}, id 0
Ethernet II, Src: IntelCor_37:28:37 (dc:41:a9:37:28:37), Dst: TaicangT_80:60:30 (18:45:93:80:60:30)
Internet Protocol Version 4, Src: {myLocalIp}, Dst: {proxyServerIp}
Transmission Control Protocol, Src Port: 51842, Dst Port: 8888, Seq: 1, Ack: 1, Len: 339
Hypertext Transfer Protocol
GET {url} HTTP/1.1\r\n
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15\r\n
x-cache-proxyname: i-05ad6154426f07671\r\n
host: {host}\r\n
Connection: close\r\n
\r\n
[Full request URI: {url}]
[HTTP request 1/1]
[Response in frame: 96]
I am trying to get a few cURL commands translated to a C# implementation, but I am running into an Exception which I can't solve.
I tried to gather up as much information as I can in the hopes that somebody can help me further. Here it goes...
The cURL statement:
curl -i -X PUT "http://[ipaddress]:[port]/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=[username]&op=CREATE"
The C# version:
var response = await client.PutAsync(
"http://[ipaddress]:[port]/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=[username]&op=CREATE",
null);
The C# version results in a WebException:
HResult = -2146233088
Message = The remote name could not be resolved: 'ip-172-31-9-79.eu-central-1.compute.internal'
The server I am connecting to, is a Hadoop server. There are multiple instances running on that server. So when I store a file, the server replies with information on which instance this file is stored (so that I can reference that instance when I want to write to this file).
Based on the error message, it seems to be that it receives some kind of reference to an IP address it can't access (which makes sense, as that ip address is an internal IP address in that Hadoop server.
I used Wireshark to find the difference in the requests which are sent.
Using cURL:
Frame 57: 204 bytes on wire (1632 bits), 204 bytes captured (1632 bits) on interface 0
Ethernet II, Src: IntelCor_da:f4:44 (fc:f8:ae:da:f4:44), Dst: AsustekC_32:7d:b0 (ac:22:0b:32:7d:b0)
Internet Protocol Version 4, Src: 192.168.1.107, Dst: [ipaddress]
Transmission Control Protocol, Src Port: 60454, Dst Port: 50070, Seq: 1, Ack: 1, Len: 150
Hypertext Transfer Protocol
PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n
[Expert Info (Chat/Sequence): PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: PUT
Request URI: /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE
Request URI Path: /webhdfs/v1/[appname]/staging/sensors/aap.txt
Request URI Query: user.name=hdfs&op=CREATE
Request URI Query Parameter: user.name=hdfs
Request URI Query Parameter: op=CREATE
Request Version: HTTP/1.1
Host: [ipaddress]:50070\r\n
User-Agent: curl/7.50.0\r\n
Accept: */*\r\n
\r\n
[Full request URI: http://[ipaddress]:50070/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE]
[HTTP request 1/1]
[Response in frame: 59]
Using the HttpClient:
Frame 381: 209 bytes on wire (1672 bits), 209 bytes captured (1672 bits) on interface 0
Ethernet II, Src: IntelCor_da:f4:44 (fc:f8:ae:da:f4:44), Dst: AsustekC_32:7d:b0 (ac:22:0b:32:7d:b0)
Internet Protocol Version 4, Src: 192.168.1.107, Dst: [ipaddress]
Transmission Control Protocol, Src Port: 60541, Dst Port: 50070, Seq: 1, Ack: 1, Len: 155
Hypertext Transfer Protocol
PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n
[Expert Info (Chat/Sequence): PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: PUT
Request URI: /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE
Request URI Path: /webhdfs/v1/[appname]/staging/sensors/aap.txt
Request URI Query: user.name=hdfs&op=CREATE
Request URI Query Parameter: user.name=hdfs
Request URI Query Parameter: op=CREATE
Request Version: HTTP/1.1
Host: [ipaddress]:50070\r\n
Content-Length: 0\r\n
Connection: Keep-Alive\r\n
\r\n
[Full request URI: http://[ipaddress]:50070/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE]
[HTTP request 1/1]
[Response in frame: 383]
To me, the only notable (but for me meaningless) differences are:
User-Agent: curl/7.50.0\r\n
Accept: /\r\n
vs
Content-Length: 0\r\n
Connection: Keep-Alive\r\n
I have very little experience with REST programming, and clearly not enough knowledge on analyzing web requests. Some help / guidance / explanation would be greatly appreciated.
Just for completeness:
Yes I am aware that there are Windows executables for cURL, but I run on Raspberry PI / Windows IoT. As far as I know there is nothing for that specific platform (yet)
In the above statements I left out ipaddress/appname/etc intentionally just to be safe side wrt security issues
I found out that the only thing which goes wrong is reading the response in C#. The HTTP responses monitored over Wireshark are identical (using cURL.exe and using PutAsync in C#).
I was able to ignore the redirect and be given the opportunity to handle the response myself.
So disabling auto redirect (and handle the response myself) solved my problem:
var httpClientHandler = new HttpClientHandler {AllowAutoRedirect = false};
var client = new HttpClient(httpClientHandler);
I run an application on the Azure application Standard: 1 Small plan. Framework is 4.6.1
This application is calling a SSL secured API. The SSL is published by StartCom Class 1 DV Server CA, my local browser tells me that the certificate is valid.
If I run the application on my local machine everything works. However, when deployed to azure it fails with follwing error:
System.Net.Http.HttpRequestException: An error occurred while sending
the request. ---> System.Net.WebException: The request was aborted:
Could not create SSL/TLS secure channel.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult
ar)
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
The code:
public async Task<List<QutationOverview>> GetAll(string url, DateTime lastActionDate)
{
var result = string.Empty;
try
{
var userName = await _settingManager.GetSettingValueAsync("API.UserName");
var password = await _settingManager.GetSettingValueAsync("API.Password");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls |
SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
ServicePointManager
.ServerCertificateValidationCallback +=
(sender, cert, chain, sslPolicyErrors) => true;
//Add date filter
//Always request qutations where the last action took place >= Yesterday
var requestUrl =
$"GetALL/?last_action_date={lastActionDate.AddDays(-1).ToString("yyyy-MM-dd")}&format=json";
var baseAddress = new Uri(url);
var credentials = Convert.ToBase64String(Encoding.ASCII.GetBytes($"{userName}:{password}"));
Logger.InfoFormat("GetAllQuotationsAsync for url {0}{1}", url, requestUrl);
using (var httpClient = new HttpClient {BaseAddress = baseAddress})
{
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", credentials);
using (var response = await httpClient.GetAsync(requestUrl))
{
result = await response.Content.ReadAsStringAsync();
Logger.Info(result);
}
}
}
catch (Exception ex)
{
Logger.ErrorFormat("GetAllQuotationsAsync {0}: {1}", url, ex);
}
var data = JsonConvert.DeserializeObject<List<QutationOverview>>(result);
return data;
}
As you can see I skip the validation of the certificate and added the security protocols.
However, the request is still failing.
Here is the caputred response
http://textuploader.com/5ers0
Do you have any idea how to get this one working on Azure?
Capture the TLS handshake. If your ServerHello is missing you most probably don't have a common cipher suite with the remote.
Run both through https://www.ssllabs.com/ssltest/ to check supported cipher suites at both ends. For Windows Server, cipher suites can only be enabled or disabled globally (as in no distinction between client/server component) so that's why this makes for a good test.
UPDATE: Found a glaring problem in my reasoning, App Service has a frontend layer and that's where TLS terminates, so comparing ciphers that way goes nowhere.
Instead, run
Get-TlsCipherSuite
from Kudu's PowerShell and compare ciphers against your remote API's (the ciphers of which you can check over at https://ssllabs.com/ssltest). You should have at least one match.
If none match, you'll need to switch to Cloud Services or VMs and enable at least one of the cipher suites your remote speaks. Having to go this direction usually means one thing -- your remote is using weak cryptography (SSL 3.0 or TLS 1.0 with RC4) and you should have a chat with those citizens, or find new citizens that are rocking TLS 1.2.
From your System.Net trace:
[8356] 00000000 : 15 03 03 00 02
That's byte sequence for Fatal Handshake Error, which builds on my no common cipher theory.
Note the first byte (0x15):
Record Type Values dec hex
-------------------------------------
CHANGE_CIPHER_SPEC 20 0x14
ALERT 21 0x15
HANDSHAKE 22 0x16
APPLICATION_DATA 23 0x17
I ran across this error when hosting a client's certificate on Azure's App Services. The fix was to set WEBSITE_LOAD_CERTIFICATES in the App Settings.
You can set it as " * " to allow all certificates, or you can define specific certificate thumbprints to allow. See more info here.
Trying to get kerberos to work.
Looking at wireshark output, the windows username is passed to my test script, but when I turn impersonate on in IIS, I get a 500 internal server error.
The script is as follows:
<%# Page Language="C#" Debug="true" %>
<%# Import Namespace="System.Net" %>
<%
WebClient client = new WebClient();
string downloadString = client.DownloadString("http://10.6.2.117/DEV/api/1.5.12077.001/en-GB/8/56/Incident/GetList?%24id=1&StartIndex=0&PageLength=10");
Response.Write(downloadString);
%>
I can access the url directly via a browser with no problems.
With impersonate turned off I get the following wireshark output (line 8 shows username):
"1","0.000000","10.21.4.3","10.6.2.105","TCP","66","59546 → 7001 [SYN] Seq=0 Win=65535 Len=0 MSS=1260 WS=256 SACK_PERM=1"
"2","0.000092","10.6.2.105","10.21.4.3","TCP","66","7001 → 59546 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1"
"3","0.017328","10.21.4.3","10.6.2.105","TCP","60","59546 → 7001 [ACK] Seq=1 Ack=1 Win=262144 Len=0"
"4","0.019120","10.21.4.3","10.6.2.105","HTTP","404","GET /test.aspx HTTP/1.1 , NTLMSSP_NEGOTIATE"
"5","0.104296","10.6.2.105","10.21.4.3","HTTP","1940","HTTP/1.1 401 Unauthorized , NTLMSSP_CHALLENGE (text/html)"
"6","0.123311","10.21.4.3","10.6.2.105","TCP","60","59546 → 7001 [ACK] Seq=351 Ack=1261 Win=262144 Len=0"
"7","0.123314","10.21.4.3","10.6.2.105","TCP","60","59546 → 7001 [ACK] Seq=351 Ack=1887 Win=261376 Len=0"
"8","0.125557","10.21.4.3","10.6.2.105","HTTP","624","GET /test.aspx HTTP/1.1 , NTLMSSP_AUTH, User: EMEA\xxxxxx"
"9","0.183273","10.6.2.105","10.21.4.3","TCP","3834","[TCP segment of a reassembled PDU]"
"10","0.203950","10.21.4.3","10.6.2.105","TCP","60","59546 → 7001 [ACK] Seq=921 Ack=3147 Win=262144 Len=0"
"11","0.203953","10.21.4.3","10.6.2.105","TCP","60","59546 → 7001 [ACK] Seq=921 Ack=4407 Win=262144 Len=0"
"12","0.203955","10.21.4.3","10.6.2.105","TCP","60","59546 → 7001 [ACK] Seq=921 Ack=5667 Win=262144 Len=0"
"13","0.204018","10.6.2.105","10.21.4.3","HTTP","2389","HTTP/1.1 500 Internal Server Error (text/html)"
...
With impersonate turned on, I get (line 4 - no username):
"1","0.000000","10.21.4.3","10.6.2.105","TCP","66","59648 → 7001 [SYN] Seq=0 Win=65535 Len=0 MSS=1260 WS=256 SACK_PERM=1"
"2","0.000111","10.6.2.105","10.21.4.3","TCP","66","7001 → 59648 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1"
"3","0.018178","10.21.4.3","10.6.2.105","TCP","60","59648 → 7001 [ACK] Seq=1 Ack=1 Win=262144 Len=0"
"4","0.019833","10.21.4.3","10.6.2.105","HTTP","404","GET /test.aspx HTTP/1.1 , NTLMSSP_NEGOTIATE"
"5","0.111015","10.6.2.105","10.21.4.3","HTTP","1466","HTTP/1.1 500 Internal Server Error (text/html)"
...
Any help would be appreciated
Well we got there in the end after much fiddling and with the involvement of half a dozen staff members.
So, in case it is of use to anybody else, here are some useful findings:
Ensure the FQDN is used for the all URLs i.e. both in the client browser, as well as the URL being called by the first web server to the second.
IIS Application Pool Settings:
Managed Pipeline = Integrated
Identity = ApplicationPoolIdentity (other options may also work)
IIS Authentication Settings:
Anonymous Authentication = Disabled
ASP .NET Impersonation = Enabled
Basic Authentication = Disabled
Forms Authentication = Disabled
Windows Authentication = Enabled
Advanced Settings:
Extended Protection = Off
Enable Kernel-mode authentication = Off
Providers: Must be in this order. Do not enable just the ‘Negotiate’ option.
Negotiate:Kerberos
NTLM
And finally, a different test script:
<%# Page Language="C#" Debug="true" %>
<%# Import Namespace="System.Net" %>
<%# Import Namespace="System.IO" %>
<%
// The service we wish to consume
string uri = "http://eu9992k8dvweb01.emea.world.net/DEV/api/1.5.12077.001/en-GB/8/56/Incident/GetList?StartIndex=0&PageLength=10";
//Create web request
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri);
//Create credential cache
CredentialCache myCredCache = new CredentialCache();
myCredCache.Add(new Uri(uri), "Negotiate", (NetworkCredential)CredentialCache.DefaultCredentials);
//Add credentials to web request
req.Credentials = myCredCache;
req.Proxy = null;
// create somewhere for the response to go
HttpWebResponse httpResponse = null;
// now use the request
try
{
// get the requested page
httpResponse = (HttpWebResponse)req.GetResponse();
// output what was returned
using (StreamReader streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
Response.Write(streamReader.ReadToEnd());
}
}
finally
{
// close the response object
if (httpResponse != null)
httpResponse.Close();
}
%>