Develop Reference

How to determine the end of a file when using RETR on FTP server (PASV)

I've decided to roll my own FTP Socket client in C# in order to have more control but mostly for the learning experience. But I cannot figure out how much file data I need to read after issuing the RETR command. I am able to establish the initial FTP communication connection, and then connect a second socket on a new port by issuing the PASV command and all that good stuff.
My problem is, when I issue the RETR command on the Communication port, the FTP server begins sending the file over the Passive port and then issues the 226 command when it finishes sending on the Communication port. I can receive the data fine but sometimes the 226 command arrives on the Communication port before all of the file data arrives on the Passive port. So, if I immediately signal my Passive socket task to stop reading after I get the 226 command, it will sometimes miss out on the data that comprises the end of the file. I've tried checking for Socket.Available before exiting out of my Socket.Receive loop but more often than not Socket.Available will report 0 bytes available on the socket but with sockets that doesn't mean there isn't more data on the way. Reading the FTP spec it seems as though the FTP server is supposed to shut down the data connection after it finishes sending all the data but I don't think that is happening. I can keep looping reading 0 bytes of data all day long. Am I looking at a broken FTP server or am I doing something wrong in my methodology?
Now for some additional info that might be the answer to my question. Using Filezilla, connected to the same FTP server, I see that when requesting a file the FTP server appends the file length to the 150 command in parenthesis. For example, "150 Opening BINARY mode data connection for test.zip (123421 bytes)." Is this text part of the FTP standard? If all FTP servers use this same behavior and formatting then I can just keep reading until I hit 123,421 bytes. But if this isn't common behavior then I'm back at square one. Appreciate any ideas.

I cannot figure out how much file data I need to read after issuing the RETR command.
It depends on the transfer mode (see RFC 959 Section 3.4).
In STREAM mode (the usual case), the end of file is denoted by the sender closing the data socket. This means the data connection cannot be reused for multiple transfers.
In BLOCK mode, the data is sent in blocks, and the sender will send an EOF block at the end of the transfer. This allows the data connection to be reused for multiple transfers.
My problem is, when I issue the RETR command on the Communication port, the FTP server begins sending the file over the Passive port and then issues the 226 command when it finishes sending on the Communication port. I can receive the data fine but sometimes the 226 command arrives on the Communication port before all of the file data arrives on the Passive port.
Simply don't read the response on the communication port until after you have finished reading the data on the transfer port.
So, if I immediately signal my Passive socket task to stop reading after I get the 226 command, it will sometimes miss out on the data that comprises the end of the file.
So don't signal your socket to stop reading. Let it stop reading on its own.
I've tried checking for Socket.Available before exiting out of my Socket.Receive loop but more often than not Socket.Available will report 0 bytes available on the socket but with sockets that doesn't mean there isn't more data on the way.
Correct. Available is simply reporting how many bytes are waiting to be read at that moment. Just keep reading until Receive() tells you that a disconnect has actually occurred. If you want to poll the socket status before calling Receive(), use Poll(), a disconnect will be reported as a readable state. Either way, Receive() will return 0 bytes on a graceful disconnect, and throw an exception on an abnormal disconnect.
Reading the FTP spec it seems as though the FTP server is supposed to shut down the data connection after it finishes sending all the data
In STREAM mode, yes.
but I don't think that is happening.
Yes, it is.
I can keep looping reading 0 bytes of data all day long. Am I looking at a broken FTP server or am I doing something wrong in my methodology?
You are doing something wrong in your methodology.
Using Filezilla, connected to the same FTP server, I see that when requesting a file the FTP server appends the file length to the 150 command in parenthesis. For example, "150 Opening BINARY mode data connection for test.zip (123421 bytes)." Is this text part of the FTP standard?
No, it is not. The text is arbitrary, it can be anything the server wants. What is important is the response code (150) to indicate the transfer is proceeding.
If all FTP servers use this same behavior and formatting
They don't use the same text formatting.
then I can just keep reading until I hit 123,421 bytes.
You are supposed to keep reading until the server closes the data connection. And then read the final response on the communication port to make sure the server considers the transfer to be successful, and not aborted prematurely.

It seems I did not read the spec well enough as there is a SIZE command in RFC 3659 that will return the size of the file. I tested it and it will do exactly what I need.

How to implement file downloader to detect communication breakage?

I need to implement a file downloader in C#. This downloader will be running on the client computer, and will download several files according to several conditions.
The main restriction I have is that the client will probably go offline during downloading (sometime more than once), so I need the following things to happen:
1) Downloader should notice there isn’t any network communication anymore and pause downloading.
2) Downloader should resume downloading once communication is back, and continue collecting the packages, adding them to those that were already downloaded to the local disk.
I have checked StackOverflow previous posts and saw that there are two options – WebClient and WebRequest (using one of the inheritance classes). I was wondering if someone can advise which one to use based on the requirements I have specified. How can I detect communication breakage?

You will need System.Net.HttpWebRequest to send HTTP requests and System.IO.FileStream to access files. Two methods needed are HttpWebRequest.AddRange and Filestream.seek
HttpWebRequest.AddRange method adds a byte range header to the request, and the range parameter specifies the starting point of the range. The server will start sending data from the range parameter specified to the end of the data in the HTTP entity. While Filestream.seek method is used to access the current position within a stream.
Source and example

You need download resume (if supported by server you are downloading from) which means you should go with WebRequest since with WebClient you cannot do that (probably next versions will support RANGE requests).
As soon as connection is dropped your code which is reading network stream throws an exception. That tells you there is a problem with download (i.e. network problem) then you can try in some periods to make a new connection and if succeeded, resume from last successful byte (using RANGE in HTTP HEADER).

check network connection while downloading file (c#)

I would like to check my internet connection while using WebClient in order to download a file. (WebClient.DownloadFile()).
I want to check if the connection still available in the middle of the downloading process.
How do I do that ? (in C#)
Thanks a lot.

You cannot generally detect that the internet is available or not. But heuristically, you can start a second thread which tries to GET google.com with a read timeout of 5s every 15s or so. That way your connection check can have different, harder timeouts than your main download.

You can't. There is no way in TCP to check the status of a connection other than by trying to use it. Set a read timeout, and respond accordingly to the resulting SocketTimeoutException; and respond to any other IOException by closing the connection and maybe retrying, depending on your specific requirements.

Resume download of file in Silverlight 4 OOB

I have a Silverlight 4 out-of-browser application that needs to be able to resume the download of an external file if the download is interrupted for any reason. I would like to be able resume instead of restart from the beginning because the file will be rather large and we have the potential to have users on slower connections.
I found some code at,
http://www.codeproject.com/Tips/157532/Silverlight-4-OOB-HTTP-Download-Component-with-Pau.aspx
but there seems to be numerous errors in it, so I'm not exactly confident that I'll be able to get it to work.
So, if anyone has any other original suggestions or alternatives, I'd like to hear them.
Thanks,

One approach you might consider is managing the download using the HTTP/1.1 Acccept-Ranges response header and the Range request header.
Make sure the resource you are downloading will include the header:-
Accept-Ranges: bytes
when it is requested (a static file sent by IIS will do this by default).
Now using the ClientHTTP stack you make an initial "HEAD" request to determine the server will accept a Range: bytes= header in the request and find the total size of the content to be sent.
You then make a "GET" request for the resource including the header:-
Range: bytes=0-65535
This limits the downloaded content to just the first 64K chunk. You then repeat the same request with:-
Range: bytes=65536-131071
Each time you can save the content of the response stream to your destination file. You keep track of how many bytes you have received. When you determine the final chunk which is likely to less than full just use a header like:-
Range: bytes=131072-
That will read to the end of file.
If the requests to the server are fail you can resume at an appropriate point in this sequence.
You need to be degrade gracefully, if the server does not include the Accept-Ranges header in the initial "HEAD" request then you'll just have to download the whole file.

WebClient.UploadFile() throws exception on some binary files

I am using the WebClient.UploadFile() method to post files to a service for processing. The file contains an XML document with a compressed, b64-encoded content element in it. For some files (currently 1), the UploadFile throws an exception, indicating that the underlying connection was closed. The innermost exception on socket level gives the message 'An existing connection was forcibly closed by the remote host'.
Questions:
Has anyone necountered the same problem?
Why does it throw an exception for some files, and not for all?
Should I set some additional parameter for files with binary content?
Is there a workaround?
This functionality does work fine in a VPN situation, but obviously we want to use it to work in standard Internet situations.
Thanks, Rine

Sounds like a firewall or other security software sitting in between you and the server may be rejecting the request as a potential attack. I've run into this before where firewalls were rejecting requests that contained a specific file extension-- even if that file extension was encoded in a query string parameter!
If I were you, I'd take the problematic file and start trimming XML out of it. You may be able to find a specific chunk of XML which triggers the issue. Once you've identified the culprit, you can figure out how to get around the issue (e.g. by encoding those characters using their Unicode values instead of as text before sending the files). If, however, any change to the file causes the problem to go away (it's not caused by a specific piece of worrisome text), then I'm stumped.

Any chance it's a size issue and the problematic file is above a certain size and all the working files are below it? The server closing the connection when it hits a max accepted request size matches your symptom. You mentioned it worked in VPN so it's admittedly a stretch, but maybe the VPN case was a different server that's configured differently (or the max request is different for some other reason).
Are there non-WebClient methods for uploading the file to the same service from the same machine and if so, do they work?