Develop Reference

Streaming from SFTP using WinSCP Session.GetFile fails – ((WinSCP.PipeStream)stream).Length' threw an exception of type 'System.NotSupportedException'

My app requires copying file using SFTP from a location directly to Azure storage.
Our app is using C# with .NET 4.6 and our WinSCP version is 5.21.1.
My old code works using Session.GetFileToDirectory() method, but the problem is it need to store the file on temp folder inside our hosting.
using (Session session = new Session())
{
session.Open(sessionOptions);
TransferOptions transferOptions = new TransferOptions();
transferOptions.TransferMode = TransferMode.Binary;
var transfer = session.GetFileToDirectory(FilePath, fullPath);
using (Stream stream = File.OpenRead(transfer.Destination))
{
UploadToAzure(stream, Filename, Foldername);
}
}
As we planned to entirely use Azure storage, I change my code like this
using (Session session = new Session())
{
session.Open(sessionOptions);
TransferOptions transferOptions = new TransferOptions();
transferOptions.TransferMode = TransferMode.Binary;
using (Stream stream = session.GetFile(FilePath, transferOptions))
{
UploadToAzure(stream, Filename, Foldername);
}
}
Here my library that uploads the file using Stream to Azure.
This code is working fine using my old code that still save to temp folder before send to Azure.
public static string UploadToAzure(Stream attachment, string Filename, string Foldername)
{
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
var connectionString = $"{ConfigurationManager.AppSettings["AzureFileShareConnectionString"]}";
string shareName = $"{ConfigurationManager.AppSettings["AzureFileShareFolderName"]}";
string dirName = $"files\\{Foldername}";
string fileName = Filename;
try
{
ShareClient share = new ShareClient(connectionString, shareName);
share.CreateIfNotExists();
ShareDirectoryClient directory = share.GetDirectoryClient(dirName);
directory.CreateIfNotExists();
// Get a reference to a file and upload it
ShareFileClient file = directory.GetFileClient(fileName);
file.Create(attachment.Length);
file.UploadRange(
new HttpRange(0, attachment.Length), attachment);
}
catch (Exception e)
{
return $"Uploaded {Filename} failed : {e.ToString()}";
}
return $"{Filename} Uploaded";
}
But currently my new code not working with error message
'((WinSCP.PipeStream)stream).Length' threw an exception of type 'System.NotSupportedException'.
This is the object description on creating stream using Session.GetFile method
This is 'exception stacktrace' on sending the empty-stream to Azure

The Stream returned by WinSCP Session.GetFile does not implement the Stream.Length property, because WinSCP cannot guarantee that the size of the file is fixed. The remote file might be changing while you are downloading the file. Not to mention ASCII transfer mode, when the file is converted while being transferred, with unpredictable impact on the final size.
You use the size (Stream.Length) in two places:
When creating the file:
file.Create(attachment.Length);
The parameter of ShareFileClient.Create is maxSize. So it does not look like it's a real size. You can possibly just put an arbitrary large number here.
Or if you prefer (and know that the file is not changing), read the current size of the remote file using Session.GetFileInfo and RemoteFileInfo.Length:
file.Create(session.GetFileInfo(FilePath).Length);
When uploading the contents:
file.UploadRange(new HttpRange(0, attachment.Length), attachment);
The above can be replaced with simple ShareFileClient.Upload:
file.Upload(attachment);

Read folder paths from FTP directory to IEnumerable

I'm currently working on a .NET 4.6 console application. I need to parse a couple of XML files from different directories on my FTP server. I thought the best approach would be, to read all file paths and store them into an IEnumerable, to process them further (Serialize XML Files to objects).
The root FTP path looks like this:
string urlFtpServer = #"ftp://128.0.1.70";
File paths look like this:
string file1 = #"ftp://128.0.1.70/MyFolder1/Mainfile.xml";
string file2 = #"ftp://128.0.1.70/MyFolder1/Subfile.xml";
string file3 = #"ftp://128.0.1.70/MyFolder2/Mainfile.xml";
string file4 = #"ftp://128.0.1.70/MyFolder2/Subfile.xml";
string file5 = #"ftp://128.0.1.70/MyFolder3/Mainfile.xml";
My question is, do you know how I can get those specific file paths?
I currently can read the folders of my FTP directory with this coding:
static void Main(string[] args)
{
string url = #"ftp://128.0.1.70";
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(url);
request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
request.Credentials = new NetworkCredential("My-User", "mypassword");
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
Console.WriteLine(reader.ReadToEnd());
Console.WriteLine("Directory List Complete, status {0}", response.StatusDescription);
reader.Close();
response.Close();
Console.ReadKey();
}
Do you know how I can read all file paths from the FTP main directory and possibly store them into a List<string>?
Thank you very much!!

Using FtpWebRequest
The FtpWebRequest does not have any explicit support for recursive file operations (including listing). You have to implement the recursion yourself:
List the remote directory
Iterate the entries, recursing into subdirectories (listing them again, etc.)
Tricky part is to identify files from subdirectories. There's no way to do that in a portable way with the FtpWebRequest. The FtpWebRequest unfortunately does not support the MLSD command, which is the only portable way to retrieve directory listing with file attributes in FTP protocol. See also Checking if object on FTP server is file or directory.
Your options are:
Do an operation on a file name that is certain to fail for file and succeeds for directories (or vice versa). I.e. you can try to download the "name". If that succeeds, it's a file, if that fails, it's a directory.
You may be lucky and in your specific case, you can tell a file from a directory by a file name (i.e. all your files have an extension, while subdirectories do not)
You use a long directory listing (LIST command = ListDirectoryDetails method) and try to parse a server-specific listing. Many FTP servers use *nix-style listing, where you identify a directory by the d at the very beginning of the entry. But many servers use a different format. The following example uses this approach (assuming the *nix format)
void ListFtpDirectory(
string url, string rootPath, NetworkCredential credentials, List<string> list)
{
FtpWebRequest listRequest = (FtpWebRequest)WebRequest.Create(url + rootPath);
listRequest.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
listRequest.Credentials = credentials;
List<string> lines = new List<string>();
using (FtpWebResponse listResponse = (FtpWebResponse)listRequest.GetResponse())
using (Stream listStream = listResponse.GetResponseStream())
using (StreamReader listReader = new StreamReader(listStream))
{
while (!listReader.EndOfStream)
{
lines.Add(listReader.ReadLine());
}
}
foreach (string line in lines)
{
string[] tokens =
line.Split(new[] { ' ' }, 9, StringSplitOptions.RemoveEmptyEntries);
string name = tokens[8];
string permissions = tokens[0];
string filePath = rootPath + name;
if (permissions[0] == 'd')
{
ListFtpDirectory(url, filePath + "/", credentials, list);
}
else
{
list.Add(filePath);
}
}
}
Use the function like:
List<string> list = new List<string>();
NetworkCredential credentials = new NetworkCredential("user", "mypassword");
string url = "ftp://ftp.example.com/";
ListFtpDirectory(url, "", credentials, list);
Using 3rd party library
If you want to avoid troubles with parsing the server-specific directory listing formats, use a 3rd party library that supports the MLSD command and/or parsing various LIST listing formats; and recursive downloads.
For example with WinSCP .NET assembly you can list whole directory with a single call to the Session.EnumerateRemoteFiles:
// Setup session options
SessionOptions sessionOptions = new SessionOptions
{
Protocol = Protocol.Ftp,
HostName = "ftp.example.com",
UserName = "user",
Password = "mypassword",
};
using (Session session = new Session())
{
// Connect
session.Open(sessionOptions);
// List files
IEnumerable<string> list =
session.EnumerateRemoteFiles("/", null, EnumerationOptions.AllDirectories).
Select(fileInfo => fileInfo.FullName);
}
Internally, WinSCP uses the MLSD command, if supported by the server. If not, it uses the LIST command and supports dozens of different listing formats.
(I'm the author of WinSCP)

Upload file to SFTP server with different extension

I went through many posts on Stack Overflow, but I have not found what I needed.
I have a file suppose temp.csv
I want to upload this file to an SFTP server which is working fine.
I need to save this file with different extension like temp.csv.ready
Can you please suggest something on this.
Here is the code, which I tried and working fine. But I'm not not able to change file extension.
SessionOptions sessionOptions = new SessionOptions
{
Protocol = Protocol.Sftp,
HostName = System.Configuration.ConfigurationManager.AppSettings["puthost"].ToString(),
UserName = System.Configuration.ConfigurationManager.AppSettings["putusername"].ToString(),
Password = System.Configuration.ConfigurationManager.AppSettings["putpassword"].ToString(),
PortNumber = Convert.ToInt32(System.Configuration.ConfigurationManager.AppSettings["putport"].ToString()),
SshHostKeyFingerprint = ... //followed by your 16 bit key
};
using (Session session = new Session())
{
session.SessionLogPath = "log.txt";
session.Open(sessionOptions); //Attempts to connect to your sFtp site
//Get Ftp File
TransferOptions transferOptions = new TransferOptions();
transferOptions.TransferMode = TransferMode.Binary; //The Transfer Mode -
//<em style="font-size: 9pt;">Automatic, Binary, or Ascii
transferOptions.FilePermissions = null; //Permissions applied to remote files;
//null for default permissions. Can set user,
//Group, or other Read/Write/Execute permissions.
transferOptions.PreserveTimestamp = false; //Set last write time of
//destination file to that of source file - basically change the timestamp
//to match destination and source files.
transferOptions.ResumeSupport.State = TransferResumeSupportState.Off;
WinSCP.TransferOperationResult transferResult;
//the parameter list is: local Path, Remote Path, Delete source file?, transfer Options
transferResult = session.PutFiles(#System.Configuration.ConfigurationManager.AppSettings["sendfilesource"].ToString(), System.Configuration.ConfigurationManager.AppSettings["sendfiletarget"].ToString(), false, transferOptions);
}

The remotePath argument of the Session.PutFiles method is:
Full path to upload the file to.
So, all you need to do, is to specify the full path like:
/remote/path/temp.csv.ready

If you are willing to use a library, you may want to use a 3rd library like componenentpro sftp library. You can accomplish the job with client.UploadFile(#"xxx\temp.csv", "/temp.new.csv"). That line of code is excerpt from the following code example:
// Create a new class instance.
Sftp client = new Sftp();
// Connect to the SFTP server.
client.Connect("localhost");
// Authenticate.
client.Authenticate("test", "test");
// ...
// Upload local file 'c:\test.dat' to '/test.dat'.
client.UploadFile(#"xxx\temp.csv", "/temp.new.csv");
// ...
// Disconnect.
client.Disconnect();

Use "DirectoryInfo" with FTP server

I would use this instruction:
System.IO.DirectoryInfo dir = new System.IO.DirectoryInfo("ftp://192.168.47.1/DocXML");
But I can´t.
How can I use ("ftp://192.168.47.1/DocXML"); with new System.IO.DirectoryInfo("");?
This is the code
System.IO.DirectoryInfo dir = new System.IO.DirectoryInfo(#"\\192.168.47.1\DocXML");`
IEnumerable<System.IO.FileInfo> fileList = dir.GetFiles("*.*", System.IO.SearchOption.AllDirectories);

I'm afraid you can't.
Try this instead:
FtpWebRequest req = (FtpWebRequest)WebRequest.Create("ftp://192.168.47.1/DocXML");
req.Credentials = new NetworkCredential("foo", "foo#foo.com");
req.Method = WebRequestMethods.Ftp.ListDirectory;
FtpWebResponse res = (FtpWebResponse)req.GetResponse();
using (StreamReader streamReader = new StreamReader(res.GetResponseStream()))
{
...
}

If you a need structured information about files in an FTP directory, you have to use a 3rd party library. The .NET framework does not offer such functionality.
Particularly because it does not support an MLSD FTP command, what is the only reliable way to retrieve a machine-readable listing of remote files with their attributes.
There are many 3rd party libraries that allow this.
For example with WinSCP .NET assembly:
// Setup session options
SessionOptions sessionOptions = new SessionOptions
{
Protocol = Protocol.Ftp,
HostName = "example.com",
UserName = "username",
Password = "password",
};
using (Session session = new Session())
{
// Connect
session.Open(sessionOptions);
// Get list of files in the directory
string remotePath = "/remote/path/";
RemoteDirectoryInfo directoryInfo = session.ListDirectory(remotePath);
foreach (RemoteFileInfo fileInfo in directoryInfo.Files)
{
Console.WriteLine("{0} with size {1}, permissions {2} and last modification at {3}",
fileInfo.Name, fileInfo.Length, fileInfo.FilePermissions,
fileInfo.LastWriteTime);
}
}
References:
https://winscp.net/eng/docs/library_session_listdirectory
https://winscp.net/eng/docs/library_remotefileinfo
From your comment and your other question, you seem to actually need to retrieve the oldest file in FTP directory. For that see:
Download the latest file from an FTP server (C#)
Downloading the most recent file (PowerShell, but translates easily to C#)
Both are for the newest, not oldest, file. Just replace the .OrderByDescending with the .Order in the C# code to get the oldest file.
(I'm the author of WinSCP)

Not working in this way.
I recommend using SFTP instead of FTP. For this I'm using the 3rd party library "SharpSSH".
The following example seems to work:
using System.IO;
using Tamir.SharpSsh;
using Tamir.SharpSsh.jsch;
string ip = "DestinationIp";
string user = "JohnDoe";
string password = "YourPassword";
Sftp sftp = new Tamir.SharpSsh.Sftp(ip, user, password);
sftp.Connect();
FileInfo yourFileInfo = new FileInfo("path");
There's also the possibility to add a primary key with
sftp.AddIdentityFile();

C# Multiple Download from FTP using parallel task - Duplicate Download issue

I am facing a strange issue, I want to download a list of files from FTP. I preferred to go with Parallel Task. Below is my code. The issue is, all the list of files are getting downloaded, but duplicate files with different name are being generated. I am very new to Parallel task concept. Please help me to find out the issue.
Note: I am using SSH.Net for sftp connection and download.
private void ConcurrentDownload()
{
// Declaring Connection Information
PasswordAuthenticationMethod pm = new PasswordAuthenticationMethod("FTPUserName", "Password");
ConnectionInfo connectionInfo = new ConnectionInfo("FTPHost", 22, "FTPUserName", ProxyTypes.Socks5, "127.0.0.1", 8080, string.Empty, string.Empty, pm);
using (SftpClient sfc = new SftpClient(connectionInfo))
{
// Establish the remote connection
sfc.Connect();
// Getting Remote Directory Contents
IEnumerable<SftpFile> sFiles = new List<SftpFile>();
sFiles = sfc.ListDirectory(".\\");
// Building the File List
List<string> remotefiles = new List<string>();
foreach (SftpFile sfile in sFiles)
{
if (!sfile.IsDirectory)
{
string ss = sfile.Name;
remotefiles.Add(ss);
}
}
// Parallel Download
Parallel.ForEach(remotefiles.Distinct(), file => DownloadFile(sfc, file));
sfc.Disconnect();
}
}
private void DownloadFile(SftpClient sf, string RemoteFileName)
{
using (Stream ms = File.OpenWrite(RemoteFileName))
{
sf.DownloadFile(RemoteFileName, ms);
}
}

You better use Distinct like below
Parallel.ForEach(remotefiles.Distinct(), file => DownloadFile(sfc, file));
if you have duplicate file names and when parallel processing start on same file you will get exception on those duplicate files.
And also you are not downloading to another location, what you are doing is download to same ftp source location. is that correct?
I would give diferent download directory and get file name from source file and then download to that location as below
private void DownloadFile(SftpClient sf, string RemoteFileName)
{
string downloadTo = Path.Combine(DownloadDirectoryPath, Path.GetFileName(RemoteFileName));
using (Stream ms = File.OpenWrite(downloadTo))
{
sf.DownloadFile(RemoteFileName, ms);
}
}
Related Reference : SFTP Async Upload in Parallel