Develop Reference

Reusing FtpWebRequest

I'm trying to make a simple method to download a file from an FTP using FtpWebRequest with the method WebRequestMethods.Ftp.DownloadFile. The problem is that I wan't to display the progress of downloading and thus need to know the file size ahead to be able to calculate the percentage transfered. But when I call GetResponse in FtpWebRequest the ContentLength member is -1.
OK - so I get the size of the file in advance using the method WebRequestMethods.Ftp.GetFileSize. No problem. Then after getting the size I download the file.
This is where the problem in question appears...
After getting the size I try to reuse the FtpWebRequest and resets the method to WebRequestMethods.Ftp.DownloadFile. This causes an System.InvalidOperationException saying something like "Can't perform this action after sending the request." (may not be the exact formulation - translated from the one I get in Swedish).
I've found elsewhere that as long as I set the KeepAlive property to true, it doesn't matter, the connection is kept active. This is what I don't understand... The only object I've created is my FtpWebRequest object. And if I create another one, how can it know what connection to use? And what credentials?
Pseudo code:
Create FtpWebRequest
Set Method property to GetFileSize
Set KeepAlive property to true
Set Credentials property to new NetworkCredential(...)
Get FtpWebResponse from the request
Read and store ContentLength
Now I got the file size. So it's time to download the file. Setting Method now causes the exception mentioned above. So do I create a new FtpWebRequest? Or is there anyway to reset the request to be reused? (Closing the response made no difference.)
I don't understand how to move forward without re-creating the object. I could do that, but it just doesn't feel right. So i'm posting here in hope to find the correct way of doing this.
Here's the (non working) code (Inputs are sURI, sDiskName, sUser and sPwd.) :
FtpWebRequest request = (FtpWebRequest)FtpWebRequest.Create(sURI);
request.Method = WebRequestMethods.Ftp.GetFileSize;
request.Credentials = new NetworkCredential(sUser, sPwd);
request.UseBinary = true;
request.UsePassive = true;
request.KeepAlive = true;
FtpWebResponse resp = (FtpWebResponse)request.GetResponse();
int contLen = (int)resp.ContentLength;
resp.Close();
request.Method = WebRequestMethods.Ftp.DownloadFile;
resp = (FtpWebResponse)request.GetResponse();
Stream inStr = resp.GetResponseStream();
byte[] buff = new byte[16384];
sDiskName = Environment.ExpandEnvironmentVariables(sDiskName);
FileStream file = File.Create(sDiskName);
int readBytesCount;
int readTotal=0;
while ((readBytesCount = inStr.Read(buff, 0, buff.Length)) > 0)
{
readTotal += readBytesCount;
toolStripProgressBar1.Value = 100*readTotal/contLen;
Application.DoEvents();
file.Write(buff, 0, readBytesCount);
}
file.Close();
I hope someone can explain how this is supposed to work. Thanks in advance.

I don't think this will be answered so I'm "closing it" by telling you how I solved it.
Well, I didn't really solve it. I did however test the download by recreating the FtpWebRequest and noticed that on the FTP server it behaved as I wanted i.e. only one log on and then sequentially executing my requests.
This is how the code getting the file size and starting the download ended up:
// Start by fetching the file size
FtpWebRequest request = (FtpWebRequest)FtpWebRequest.Create(sURI);
request.Method = WebRequestMethods.Ftp.GetFileSize;
NetworkCredential nc = new NetworkCredential(sUser, sPwd);
request.Credentials = nc;
request.UseBinary = true;
request.UsePassive = true;
request.KeepAlive = true;
// Get the result (size)
FtpWebResponse resp = (FtpWebResponse)request.GetResponse();
Int64 contLen = resp.ContentLength;
// and now download the file
request = (FtpWebRequest)FtpWebRequest.Create(sURI);
request.Method = WebRequestMethods.Ftp.DownloadFile;
request.Credentials = nc;
request.UseBinary = true;
request.UsePassive = true;
request.KeepAlive = true;
resp = (FtpWebResponse)request.GetResponse();
So no answer on if it's possible to reset the FtpWebRequest for re-use. But at least I know there's no redundant information being transferred.
Thanks to everybody who took an interest and spent time thinking of an answer.

FtpWebRequest can be used for only 1 request, like getting the file size or to download the file, but not both. You have to create 2 FtpWebRequests. Behind the scene, FtpWebRequest notices that it is the same URL and credentials and will reuse the same ftp connection without closing it, as long IsKeepAlieve is true, which is the default setting.
This is a sad example of bad design by Microsoft. Instead of letting us explicitly open and close a connection, they want to do it automatically for us and confuse everyone.

You're going to probably want to use the Async method. Here's the link to the MSDN doc.
http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx
GetResponseAsync()
That will keep your application from locking up too, so you won't have to use
Application.DoEvents();
You could also look at possibly using an alternative ftp library. imo the FtpWebRequest is not exactly the best ftp class. A quick search turned up this library. Ftp isn't stateless like HTTP. I prefer libraries that let you create a client, open a connect, and keep the connection alive.
http://sanity-free.org/dist/NullFX.Net-binary.zip
Here's the code exacmple I found
FtpClient client =
new FtpClient(
new IPEndPoint( IPAddress.Loopback, 21 ),
new NetworkCredential( "test", "testing#localdomain" )
);
client.Connect();
client.Download("testfile.zip", #"C:\downloads\testfile.zip");
The source is there too, so you would be able to possibly attach some events to the read process for your download progress tracking.

FTP through FTP proxy

I am trying to download a file using FTP through a FTP proxy (on my side).
This is script I am trying to implement in C#:
On Commandline:
ftp -i -s:get.ini CORPORATE_PROXY.com
-----------get.ini------------
CORPORATE_PROXY_USER#CLIENT_FTP.com abc/user_name
CORPORATE_PROXY_PASSWORD
user_name_password
cd pub/linux/knoppix
get packages.txt
bye
-----------get.ini------------
abc/user_name is my user name who was granted by permissions to FTP through my corporate proxy.
I want to implement above script in C#, but after playing with many types of code found on Internet I cannot do that.
FtpWebRequest request = FtpWebRequest.Create(new Uri(#"ftp://" + CORPORATE_PROXY.com + #"/" + Path.GetFileName(fileToUpload))) as FtpWebRequest;
request.UseBinary = true;
request.KeepAlive = false;
request.Method = WebRequestMethods.Ftp.UploadFile;
if (!string.IsNullOrEmpty(CORPORATE_PROXY_USER) && !string.IsNullOrEmpty(CORPORATE_PROXY_PASSWORD ))
request.Credentials = new NetworkCredential(CORPORATE_PROXY_USER, CORPORATE_PROXY_PASSWORD );
//Get physical file
FileInfo fi = new FileInfo(fileToUpload);
Byte[] contents = new Byte[fi.Length];
//Read file
FileStream fs = fi.OpenRead();
fs.Read(contents, 0, Convert.ToInt32(fi.Length));
fs.Close();
request.Proxy = new WebProxy("CLIENT_FTP.com");
request.Proxy.Credentials = new NetworkCredential(abc/user_name, user_name_password);
//Write file contents to FTP server
Stream rs = request.GetRequestStream();
rs.Write(contents, 0, Convert.ToInt32(fi.Length));
rs.Close();
FtpWebResponse response = request.GetResponse() as FtpWebResponse;
string statusDescription = response.StatusDescription;
response.Close();
return statusDescription;
The main problem is that for the proxy I am using WebProxy, while I suspect I should use FTPProxy - which I cannot find anythere? Any ideas which direction should I go, or maybe WebProxy is fine?

In the past I have used Indy Project to get through FTP proxies.

Try using WebRequest instead of FtpWebRequest in your code example.
By doing so, the connection from client to proxy can be HTTP whereas the connection from proxy to destination server is FTP. The proxy will handle the protocol translation, this technique is referred to as FTP over HTTP.
It is also possible to use a native FTP Proxy where client to proxy and proxy to server connections are FTP. Make sure your proxy supports this.
The proxy offers a separate proxy port to serve native FTP proxy connections.

C# 403 error because the file contains an inaccessible image? or what?

I'm trying to get a stream from a url:http://actueel.nl.pwc.com/site/syndicate.jsp but i get the 403 error. It doest requier login. I used fiddler to check why IE can open it while my code doesn't. What i got was that there were 2 connections done when opening the link in IE. 1 succeeded while the other got a 403. The 403 was a sublink to a giff image. Seems like the xml is a public file, but the image it contains is located in a inaccesible folder.
I need to know how to ignore the image so i can still get the rest of stream. this is my code to test it(by the way..i tryed with WeClient too and headers) :
try
{
WebRequest request = WebRequest.Create("http://actueel.nl.pwc.com/site/syndicate.jsp");
request.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
MessageBox.Show(reader.ReadToEnd());
}
catch(Exception ex){
MessageBox.Show(ex.Message);
}
Thanks for your reactions ;)

I agree with Dmytro. The WebRequest is NOT attempting to download the gif image referenced in the jsp file, only the contents of the jsp itself is being downloaded. Try looking carefully (in Fiddler) at the IE request compared to yours - only the url but also all the request/response headers - and see if anything else is missing, such as cookies or ACCEPT headers.

Using Wireshark and wget, the differences were in the headers only.
The remote server requires User Agent and an Accept headers.
eg:
WebRequest request = WebRequest.Create("http://actueel.nl.pwc.com/site/syndicate.jsp");
((HttpWebRequest)request).UserAgent = "stackoverflow.com/q/4233673/111013";
((HttpWebRequest) request).Accept = "*/*";

System.Net.WebException: The operation has timed out

I have a big problem: I need to send 200 objects at once and avoid timeouts.
while (true)
{
NameValueCollection data = new NameValueCollection();
data.Add("mode", nat);
using (var client = new WebClient())
{
byte[] response = client.UploadValues(serverA, data);
responseData = Encoding.ASCII.GetString(response);
string[] split = Javab.Split(new[] { '!' }, StringSplitOptions.RemoveEmptyEntries);
string command = split[0];
string server = split[1];
string requestCountStr = split[2];
switch (command)
{
case "check":
int requestCount = Convert.ToInt32(requestCountStr);
for (int i = 0; i < requestCount; i++)
{
Uri myUri = new Uri(server);
WebRequest request = WebRequest.Create(myUri);
request.Timeout = 200000;
WebResponse myWebResponse = request.GetResponse();
}
break;
}
}
}
This produces the error:
Unhandled Exception: System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at vir_fu.Program.Main(String[] args)
The requestCount loop works fine outside my base code but when I add it to my project I get this error. I have tried setting request.Timeout = 200; but it didn't help.

It means what it says. The operation took too long to complete.
BTW, look at WebRequest.Timeout and you'll see that you've set your timeout for 1/5 second.

Close/dispose your WebResponse object.

I'm not sure about your first code sample where you use WebClient.UploadValues, it's not really enough to go on, could you paste more of your surrounding code? Regarding your WebRequest code, there are two things at play here:
You're only requesting the headers of the response**, you never read the body of the response by opening and reading (to its end) the ResponseStream. Because of this, the WebRequest client helpfully leaves the connection open, expecting you to request the body at any moment. Until you either read the response body to completion (which will automatically close the stream for you), clean up and close the stream (or the WebRequest instance) or wait for the GC to do its thing, your connection will remain open.
You have a default maximum amount of active connections to the same host of 2. This means you use up your first two connections and then never dispose of them so your client isn't given the chance to complete the next request before it reaches its timeout (which is milliseconds, btw, so you've set it to 0.2 seconds - the default should be fine).
If you don't want the body of the response (or you've just uploaded or POSTed something and aren't expecting a response), simply close the stream, or the client, which will close the stream for you.
The easiest way to fix this is to make sure you use using blocks on disposable objects:
for (int i = 0; i < ops1; i++)
{
Uri myUri = new Uri(site);
WebRequest myWebRequest = WebRequest.Create(myUri);
//myWebRequest.Timeout = 200;
using (WebResponse myWebResponse = myWebRequest.GetResponse())
{
// Do what you want with myWebResponse.Headers.
} // Your response will be disposed of here
}
Another solution is to allow 200 concurrent connections to the same host. However, unless you're planning to multi-thread this operation so you'd need multiple, concurrent connections, this won't really help you:
ServicePointManager.DefaultConnectionLimit = 200;
When you're getting timeouts within code, the best thing to do is try to recreate that timeout outside of your code. If you can't, the problem probably lies with your code. I usually use cURL for that, or just a web browser if it's a simple GET request.
** In reality, you're actually requesting the first chunk of data from the response, which contains the HTTP headers, and also the start of the body. This is why it's possible to read HTTP header info (such as Content-Encoding, Set-Cookie etc) before reading from the output stream. As you read the stream, further data is retrieved from the server. WebRequest's connection to the server is kept open until you reach the end of this stream (effectively closing it as it's not seekable), manually close it yourself or it is disposed of. There's more about this here.

proxy issue can cause this. IIS webconfig put this in
<defaultProxy useDefaultCredentials="true" enabled="true">
<proxy usesystemdefault="True" />
</defaultProxy>

I remember I had the same problem a while back using WCF due the quantity of the data I was passing. I remember I changed timeouts everywhere but the problem persisted. What I finally did was open the connection as stream request, I needed to change the client and the server side, but it work that way. Since it was a stream connection, the server kept reading until the stream ended.

I encountered the same error than adding
Task.Delay(2000);
in each request solved the problem

Error using HttpWebRequest to upload files with PUT

We've got a .NET 2.0 WinForms app that needs to upload files to an IIS6 Server via WebDav. From time to time we get complaints from a remote office that they get one of the following error messages
The underlying connection was closed:
an unexpected error occurred on send.
The underlying connection was closed:
an unexpected error occurred on
receive.
This only seems to occur with large files (~20Mb plus). I've tested it with a 40Mb file from my home computer and tried putting 'Sleep's in the loop to simulate a slow connection so I suspect that it's down to network issues at their end... but
The IT at the remote office are no help
I'd like to rule out the posibility my code is at fault.
So - can anybody spot any misakes or suggest any workarounds that might 'bulletproof' the code against this problem. Thanks for any help. Chopped down version of code follows:
public bool UploadFile(string localFile, string uploadUrl)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uploadUrl);
try
{
req.Method = "PUT";
req.AllowWriteStreamBuffering = true;
req.UseDefaultCredentials = Program.WebService.UseDefaultCredentials;
req.Credentials = Program.WebService.Credentials;
req.SendChunked = false;
req.KeepAlive = true;
Stream reqStream = req.GetRequestStream();
FileStream rdr = new FileStream(localFile, FileMode.Open, FileAccess.Read);
byte[] inData = new byte[4096];
int bytesRead = rdr.Read(inData, 0, inData.Length);
while (bytesRead > 0)
{
reqStream.Write(inData, 0, bytesRead);
bytesRead = rdr.Read(inData, 0, inData.Length);
}
reqStream.Close();
rdr.Close();
System.Net.HttpWebResponse response = (HttpWebResponse)req.GetResponse();
if (response.StatusCode != HttpStatusCode.OK && response.StatusCode!=HttpStatusCode.Created)
{
MessageBox.Show("Couldn't upload file");
return false;
}
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
return false;
}
return true;
}

Try setting KeepAlive to false:
req.KeepAlive = false;
This will allow the connection to be closed and opened again. It will not allow to use a persistent connection. I found a lot of references in the Web that suggested this in order to solve a similar to yours error. This is a relevant link.
Anyway, it is not a good idea to use HTTP PUT (or HTTP POST) to upload large files. It will be better to use FTP or a download/upload manager. These will handle retries, connection problems, timeouts automatically for you. The upload will be faster too and you could also resume a stopped uploading. If you decide to stay with HTTP, you should at least try to add a retry mechanism. If an upload is taking too long, then there is a high probability that it will fail due to proxy, server timeout, firewall or what ever reason not to have with your code.

To remove the risk of a bug in your code, try using WebClient:
using (WebClient client = new WebClient())
{
client.UseDefaultCredentials = Program.WebService.UseDefaultCredentials;
client.Credentials = Program.WebService.Credentials;
client.UploadFile(uploadUrl, "PUT", localFile);
}

Maybe try using POST, but the real culprit is probably the content type.
Try setting
req.ContentType = "application/octet-stream";
req.ContentLength = inData.Length;
or look at the code in the accepted answer here: Upload files with HTTPWebrequest (multipart/form-data)
Both my example and the link I provided involve modifying the ContentType - my example is simpler but might not work, as most applications receiving files expect multipart

Please you check whether [Enable Http Keep-Alives] is set [on] at [Web Site] tab in IIS manager.

The size of the uploads might be limited.
See here for one discussion:
http://www.codeproject.com/KB/aspnet/uploadlargefilesaspnet.aspx

Start by checking some basic configuration. The default values of either of the following may cause problems in file upload - including termination of the connection. I believe IIS 6 would never allow file upload > 2GB (even if it could complete, regardless of config). Msdn describes these nicely.
<httpRuntime executionTimeout = "30" maxRequestLength="200"/>
EDIT: This is ASP.NET config, of course, which assumes you are running your own webdav server or a 3rd party server within ASP.NET. If it's a different webdav server, you'll want to look for the equivalent.