Develop Reference

FluentFTP EPSV Error 425 Can't open data connection for transfer of "/test.csv"

I'm trying to upload a file with the .NET Library FluentFTP using EPSV connection type because I'm behind a HTTP/1.1 proxy and data and control FTP IP addresses are different.
Unfortunately I get the following error when calling the UploadFile method:
Response: 425 Can't open data connection for transfer of "/test.csv"
The same operation works in FileZilla Client with the same proxy settings, so it can't be a network problem.
This is my code:
using (FtpClientProxy client = new FtpClientHttp11Proxy(new ProxyInfo() {
Host = "prox.corp.company.com",
Port = 80,
Credentials = new NetworkCredential("proxyuser", "password")})) {
client.Host = "1.123.123.123";
client.Port = 990;
client.Credentials = new NetworkCredential("ftpuser", "password");
client.SslProtocols = System.Security.Authentication.SslProtocols.Tls12;
client.DataConnectionType = FtpDataConnectionType.EPSV;
client.EncryptionMode = FtpEncryptionMode.Implicit;
client.Connect();
client.UploadFile(#"C:\test.csv", "/test.csv");
}
The comparison between FluentFTP log files and FileZilla Client log files shows the same operations.
FluentFTP log:
Command: SIZE /test.csv
Response: 550 File not found
OpenWrite("/test.csv", Binary)
Command: TYPE I
Response: 200 Type set to I
OpenPassiveDataStream(EPSV, "STOR /test.csv", 0)
Command: EPSV
Response: 229 Entering Extended Passive Mode (|||40160|)
Status: Connecting to 1.123.1.123:80 // HTTP-Proxy
HTTP/1.1 200 Connection established
Command: STOR /test.csv
Response: 425 Can't open data connection for transfer of "/test.csv"
Status: Disposing FtpSocketStream...
FileZilla client log:
Status: Starting upload of C:\test.csv
Command: CWD /
Response: 250 CWD successful. "/" is current directory.
Command: TYPE I
Response: 200 Type set to I
Command: EPSV
Response: 229 Entering Extended Passive Mode (|||40132|)
Command: STOR test.csv
Status: Connection with proxy established, performing handshake...
Response: Proxy reply: HTTP/1.1 200 Connection established
Response: 150 Opening data channel for file upload to server of "/test.csv"
Response: 226 Successfully transferred "/test.csv"
Status: File transfer successful, transferred 24 bytes in 1 second

I had the same problem and managed to upload the file with those settings:
ftpClient.SslProtocols = SslProtocols.None | SslProtocols.Tls12;
ftpClient.ValidateAnyCertificate = true;
ftpClient.DataConnectionEncryption = false;
I think the key here is DataConnectionEncryption.

HTTP GET request to stackoverflow.com returns status code 301

What am I essencially doing is: I input URL (stackoverflow.com) in browser, catch HTTP request with proxy, proceed it to website, then get response from website and relay it back to browser.
In theory it should work just fine, but in practice I get response with "Moved Permanently" and "Location: https://stackoverflow.com/". If I understand it correctly, I need to get this location address and replace old address in HTTP request ("GET http://stackoverflow.com/ HTTP/1.1" -> "GET https://stackoverflow.com/ HTTP/1.1") and then make request again.
However, that's where I stuck. When I make new request, website responses with same status code - 301, no matter what I do.
That's what I am doing at proxy side:
Get list of IP addresses by using Dns.GetHostAddresses() - it returns 4 addresses:
151.101.65.69
151.101.129.69
151.101.193.69
151.101.1.69
Then I connect to each address and send this request (which I got from browser):
"GET http://stackoverflow.com/ HTTP/1.1\r\nHost: stackoverflow.com\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-US,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nCookie: prov=0014938e-7640-15fd-486c-9e6cb05cd5b1\r\nConnection: keep-alive\r\nUpgrade-Insecure-Requests: 1\r\n\r\n"
I then get response:
"HTTP/1.1 301 Moved Permanently\r\nLocation: https://stackoverflow.com/\r\nX-Request-Guid: 080d3919-7d02-4e48-9348-b9dc8b5b08f6\r\nContent-Length: 143\r\nAccept-Ranges: bytes\r\nDate: Mon, 26 Feb 2018 17:56:59 GMT\r\nVia: 1.1 varnish\r\nConnection: keep-alive\r\nX-Served-By: cache-ams4441-AMS\r\nX-Cache: MISS\r\nX-Cache-Hits: 0\r\nX-Timer: S1519667819.162626,VS0,VE76\r\nVary: Fastly-SSL\r\nX-DNS-Prefetch-Control: off\r\n\r\n<html><head><title>Object moved</title></head><body>\r\n<h2>Object moved to here.</h2>\r\n</body></html>\r\n"
But then even if I change "GET http://stackoverflow.com/ HTTP/1.1" to "GET https://stackoverflow.com/ HTTP/1.1", nothing changes, every of this four IPs return same response.
Am I mising something important? Maybe I should include another tag or change something else in new request header?

Translating cURL to HttpClient.PutAsync

I am trying to get a few cURL commands translated to a C# implementation, but I am running into an Exception which I can't solve.
I tried to gather up as much information as I can in the hopes that somebody can help me further. Here it goes...
The cURL statement:
curl -i -X PUT "http://[ipaddress]:[port]/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=[username]&op=CREATE"
The C# version:
var response = await client.PutAsync(
"http://[ipaddress]:[port]/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=[username]&op=CREATE",
null);
The C# version results in a WebException:
HResult = -2146233088
Message = The remote name could not be resolved: 'ip-172-31-9-79.eu-central-1.compute.internal'
The server I am connecting to, is a Hadoop server. There are multiple instances running on that server. So when I store a file, the server replies with information on which instance this file is stored (so that I can reference that instance when I want to write to this file).
Based on the error message, it seems to be that it receives some kind of reference to an IP address it can't access (which makes sense, as that ip address is an internal IP address in that Hadoop server.
I used Wireshark to find the difference in the requests which are sent.
Using cURL:
Frame 57: 204 bytes on wire (1632 bits), 204 bytes captured (1632 bits) on interface 0
Ethernet II, Src: IntelCor_da:f4:44 (fc:f8:ae:da:f4:44), Dst: AsustekC_32:7d:b0 (ac:22:0b:32:7d:b0)
Internet Protocol Version 4, Src: 192.168.1.107, Dst: [ipaddress]
Transmission Control Protocol, Src Port: 60454, Dst Port: 50070, Seq: 1, Ack: 1, Len: 150
Hypertext Transfer Protocol
PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n
[Expert Info (Chat/Sequence): PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: PUT
Request URI: /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE
Request URI Path: /webhdfs/v1/[appname]/staging/sensors/aap.txt
Request URI Query: user.name=hdfs&op=CREATE
Request URI Query Parameter: user.name=hdfs
Request URI Query Parameter: op=CREATE
Request Version: HTTP/1.1
Host: [ipaddress]:50070\r\n
User-Agent: curl/7.50.0\r\n
Accept: */*\r\n
\r\n
[Full request URI: http://[ipaddress]:50070/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE]
[HTTP request 1/1]
[Response in frame: 59]
Using the HttpClient:
Frame 381: 209 bytes on wire (1672 bits), 209 bytes captured (1672 bits) on interface 0
Ethernet II, Src: IntelCor_da:f4:44 (fc:f8:ae:da:f4:44), Dst: AsustekC_32:7d:b0 (ac:22:0b:32:7d:b0)
Internet Protocol Version 4, Src: 192.168.1.107, Dst: [ipaddress]
Transmission Control Protocol, Src Port: 60541, Dst Port: 50070, Seq: 1, Ack: 1, Len: 155
Hypertext Transfer Protocol
PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n
[Expert Info (Chat/Sequence): PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[PUT /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: PUT
Request URI: /webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE
Request URI Path: /webhdfs/v1/[appname]/staging/sensors/aap.txt
Request URI Query: user.name=hdfs&op=CREATE
Request URI Query Parameter: user.name=hdfs
Request URI Query Parameter: op=CREATE
Request Version: HTTP/1.1
Host: [ipaddress]:50070\r\n
Content-Length: 0\r\n
Connection: Keep-Alive\r\n
\r\n
[Full request URI: http://[ipaddress]:50070/webhdfs/v1/[appname]/staging/sensors/aap.txt?user.name=hdfs&op=CREATE]
[HTTP request 1/1]
[Response in frame: 383]
To me, the only notable (but for me meaningless) differences are:
User-Agent: curl/7.50.0\r\n
Accept: /\r\n
vs
Content-Length: 0\r\n
Connection: Keep-Alive\r\n
I have very little experience with REST programming, and clearly not enough knowledge on analyzing web requests. Some help / guidance / explanation would be greatly appreciated.
Just for completeness:
Yes I am aware that there are Windows executables for cURL, but I run on Raspberry PI / Windows IoT. As far as I know there is nothing for that specific platform (yet)
In the above statements I left out ipaddress/appname/etc intentionally just to be safe side wrt security issues

I found out that the only thing which goes wrong is reading the response in C#. The HTTP responses monitored over Wireshark are identical (using cURL.exe and using PutAsync in C#).
I was able to ignore the redirect and be given the opportunity to handle the response myself.
So disabling auto redirect (and handle the response myself) solved my problem:
var httpClientHandler = new HttpClientHandler {AllowAutoRedirect = false};
var client = new HttpClient(httpClientHandler);

What is the exact HTTP content used to call a WCF web service?

I have a WCF service, which was created using the Visual Studio 2013 New Project wizard. I can call this service using the WCF Test Client which works fine. However, I'd like to call the service from the command line using curl.
What I've done is copied the Request content from the WCF Test Client, which looks like:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Header>
<Action s:mustUnderstand="1" xmlns="http://schemas.microsoft.com/ws/2005/05/addressing/none">http://tempuri.org/IService1/TestStr</Action>
</s:Header>
<s:Body>
<TestStr xmlns="http://tempuri.org/" />
</s:Body>
</s:Envelope>
And stuck it in a file called Test.txt. Then, from the command line I did:
curl --verbose --header "Content-Type:text/xml; charset=utf-8" -X POST -d #Test.txt http://localhost:64777/Service1.svc
What I get back is:
* Adding handle: conn: 0x50e5a0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x50e5a0) send_pipe: 1, recv_pipe: 0
* About to connect() to localhost port 64777 (#0)
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 64777 (#0)
> POST /Service1.svc HTTP/1.1
> User-Agent: curl/7.33.0
> Host: localhost:64777
> Accept: */*
> Content-Type:text/xml; charset=utf-8
> Content-Length: 343
>
* upload completely sent off: 343 out of 343 bytes
< HTTP/1.1 400 Bad Request
< Cache-Control: private
* Server Microsoft-IIS/8.0 is not blacklisted
< Server: Microsoft-IIS/8.0
< X-AspNet-Version: 4.0.30319
< X-SourceFiles: =?UTF-8?B?YzpcdXNlcnNcbWNocmlzdGVuc2VuXGRvY3VtZW50c1x2aXN1YWwgc3R1ZGlvIDIwMTNcUHJvamVjdHNcV2NmR2VuZXJpY3NcV2NmR2VuZXJpY3NcU2VydmljZTEuc3Zj?=
< X-Powered-By: ASP.NET
< Date: Tue, 06 May 2014 18:47:29 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
What's the difference between how the WCF Test Client calls my web service and how Curl calls my web service? Do I have the URL wrong? Am I missing some HTTP headers? Thanks!
Update:
I've tried the URL http://localhost:64777/Service1.svc/TestStr as well. I've also tried encoding the file:
%3Cs%3AEnvelope%20xmlns%3As%3D%22http%3A%2F%2Fschemas.xmlsoap.org%2Fsoap%2Fenvelope%2F%22%3E%0A%20%20%3Cs%3AHeader%3E%0A%20%20%20%20%3CAction%20s%3AmustUnderstand%3D%221%22%20xmlns%3D%22http%3A%2F%2Fschemas.microsoft.com%2Fws%2F2005%2F05%2Faddressing%2Fnone%22%3Ehttp%3A%2F%2Ftempuri.org%2FIService1%2FTestStr%3C%2FAction%3E%0A%20%20%3C%2Fs%3AHeader%3E%0A%20%20%3Cs%3ABody%3E%0A%20%20%20%20%3CTestStr%20xmlns%3D%22http%3A%2F%2Ftempuri.org%2F%22%20%2F%3E%0A%20%20%3C%2Fs%3ABody%3E%0A%3C%2Fs%3AEnvelope%3E
Both just result in Bad Request.

You can use Fiddler to capture the exact HTTP request. WCF tracing unfortunately does not capture the exact bytes sent over the wire. It reports a modified copy of the actual XML sent. Fiddler is not aware of web services so it does none of that.

It looks like you created a SOAP web service but I don't see any of the SOAP headers included in your example of the request data. In your web service code - add a line to capture the actual request being received by the service through the OperationContext object: OperationContext.Current.RequestContext.RequestMessage.ToString()
You should be able to capture the data from the RequestMessage property, save that to a file, and then submit using CURL.
(You might have to strip out the <Action> elements as I think those are added by WCF and might cause problems if you try to submit a request with them included)

SSL Handshake Timeout

I have 2 client authentication certificates issued by the same certificate authority. One of them enables me to connect to a HTTPS webservice, but the other does not when I use code similar to the following:
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create( endPointUrl );
X509Store store = new X509Store( StoreName.My, StoreLocation.LocalMachine );
store.Open( OpenFlags.MaxAllowed );
X509CertificateCollection col = (X509CertificateCollection)store.Certificates.Find( X509FindType.FindBySerialNumber, certificateSerialNumber, true );
httpWebRequest.ClientCertificates.Add( col[0] );
httpWebRequest.Method = "POST";
httpWebRequest.ContentType = contentType;
httpWebRequest.KeepAlive = false;
httpWebRequest.Timeout = 3000;
httpWebRequest.ContentLength = message.Length;
httpRequestStream = httpWebRequest.GetRequestStream();
When attempting to get the request stream, I get an InvalidOperationException with the message "The operation has timed out".
I've used System.Net.trace when attempting to connect with the certificate that fails and the log shows a connection timeout before "Attempting to restart the session using the user-provided certificate" and just after the first InitializeSecurityContext.
Wireshark show the following:
"TCP","j-link > https [SYN] Seq=0 Win=65535 Len=0 MSS=1260 SACK_PERM=1"
"TCP","https > j-link [SYN, ACK] Seq=0 Ack=1 Win=32768 Len=0 MSS=1380"
"TCP","j-link > https [ACK] Seq=1 Ack=1 Win=65535 Len=0"
"TLSv1","Client Hello"
"TLSv1","Server Hello"
"TCP","[TCP segment of a reassembled PDU]"
"TCP","j-link > https [ACK] Seq=78 Ack=2521 Win=65535 Len=0"
"TLSv1","Certificate, Certificate Request, Server Hello Done"
"TCP","j-link > https [ACK] Seq=78 Ack=3187 Win=64869 Len=0"
"TCP","j-link > https [FIN, ACK] Seq=78 Ack=3187 Win=64869 Len=0"
"TCP","https > j-link [ACK] Seq=3187 Ack=79 Win=32768 Len=0"
"TLSv1","Alert (Level: Warning, Description: Close Notify)"
"TCP","j-link > https [RST, ACK] Seq=79 Ack=3194 Win=0 Len=0"
I can connect using OpenSSL from the command line using both certificates after exporting them and converting them to the PEM format.
Any suggestions would be greatly appreciated.

Thank you to Shawn's question which helped me fix the timeout problem, which was due to the connection taking over 60 seconds to fail.
I then got a "The request was aborted: Could not create SSL/TLS secure channel." error, which was solved by using the Windows HTTP Services Certificate Configuration Tool and information I obtained here.