Develop Reference

Streaming from SFTP using WinSCP Session.GetFile fails – ((WinSCP.PipeStream)stream).Length' threw an exception of type 'System.NotSupportedException'

My app requires copying file using SFTP from a location directly to Azure storage.
Our app is using C# with .NET 4.6 and our WinSCP version is 5.21.1.
My old code works using Session.GetFileToDirectory() method, but the problem is it need to store the file on temp folder inside our hosting.
using (Session session = new Session())
{
session.Open(sessionOptions);
TransferOptions transferOptions = new TransferOptions();
transferOptions.TransferMode = TransferMode.Binary;
var transfer = session.GetFileToDirectory(FilePath, fullPath);
using (Stream stream = File.OpenRead(transfer.Destination))
{
UploadToAzure(stream, Filename, Foldername);
}
}
As we planned to entirely use Azure storage, I change my code like this
using (Session session = new Session())
{
session.Open(sessionOptions);
TransferOptions transferOptions = new TransferOptions();
transferOptions.TransferMode = TransferMode.Binary;
using (Stream stream = session.GetFile(FilePath, transferOptions))
{
UploadToAzure(stream, Filename, Foldername);
}
}
Here my library that uploads the file using Stream to Azure.
This code is working fine using my old code that still save to temp folder before send to Azure.
public static string UploadToAzure(Stream attachment, string Filename, string Foldername)
{
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
var connectionString = $"{ConfigurationManager.AppSettings["AzureFileShareConnectionString"]}";
string shareName = $"{ConfigurationManager.AppSettings["AzureFileShareFolderName"]}";
string dirName = $"files\\{Foldername}";
string fileName = Filename;
try
{
ShareClient share = new ShareClient(connectionString, shareName);
share.CreateIfNotExists();
ShareDirectoryClient directory = share.GetDirectoryClient(dirName);
directory.CreateIfNotExists();
// Get a reference to a file and upload it
ShareFileClient file = directory.GetFileClient(fileName);
file.Create(attachment.Length);
file.UploadRange(
new HttpRange(0, attachment.Length), attachment);
}
catch (Exception e)
{
return $"Uploaded {Filename} failed : {e.ToString()}";
}
return $"{Filename} Uploaded";
}
But currently my new code not working with error message
'((WinSCP.PipeStream)stream).Length' threw an exception of type 'System.NotSupportedException'.
This is the object description on creating stream using Session.GetFile method
This is 'exception stacktrace' on sending the empty-stream to Azure

The Stream returned by WinSCP Session.GetFile does not implement the Stream.Length property, because WinSCP cannot guarantee that the size of the file is fixed. The remote file might be changing while you are downloading the file. Not to mention ASCII transfer mode, when the file is converted while being transferred, with unpredictable impact on the final size.
You use the size (Stream.Length) in two places:
When creating the file:
file.Create(attachment.Length);
The parameter of ShareFileClient.Create is maxSize. So it does not look like it's a real size. You can possibly just put an arbitrary large number here.
Or if you prefer (and know that the file is not changing), read the current size of the remote file using Session.GetFileInfo and RemoteFileInfo.Length:
file.Create(session.GetFileInfo(FilePath).Length);
When uploading the contents:
file.UploadRange(new HttpRange(0, attachment.Length), attachment);
The above can be replaced with simple ShareFileClient.Upload:
file.Upload(attachment);

Number of entries expected in End Of Central Directory does not correspond to number of entries in Central Directory

We have code to upload a zip file, at the start of the process if the zip file contains __MACOSX folder in the zip it will strip these folders/files from the zip and pass the file bytes on.
The issue we're having is that further down the chain if we create a new ZipArchive using fileBytes we get the error:
Number of entries expected in End Of Central Directory does not correspond to number of entries in Central Directory.
This only happens when the zip doesn't contain any __MAXOSX folders to be removed. Commenting out the below code prevents this error ever occuring.
// Strip MacOSX folder if zip
var fileBytes = uploadController.FileBytes;
if (uploadController.FileName.EndsWith(".zip", StringComparison.CurrentCultureIgnoreCase))
{
try
{
using (var data = new MemoryStream(fileBytes))
{
var changed = false;
using (var archive = new ZipArchive(data, ZipArchiveMode.Update))
{
while (true)
{
var osx = archive.Entries.FirstOrDefault(
c => c.FullName.StartsWith("__MACOSX/", StringComparison.CurrentCultureIgnoreCase)
);
if (osx == null) break;
osx.Delete();
changed = true;
}
}
// Archive is updated once disposed
if (changed)
{
fileBytes = data.ToArray();
}
}
}
catch (Exception e)
{
return new ObjectReturnMethodResult<UploadedFileV2>("Uploaded zip appears to be invalid." + e.Message);
}
}
When running the above code on a zip that doesn't contain __MACOSX why might it create this error?

Changing:
using (var data = new MemoryStream(fileBytes))
To:
using (var data = new MemoryStream(fileBytes.ToArray()))
Seems to fix this, although not sure why!

Getting "it is being used by another process" after trying to extract a ZIP file downloaded from SFTP server using SSH.NET

I got a problem when I want to unzip a file.
After I download a file from SFTP, I want to unzip that, but it always tells me it's being used by another process.
I want to find the question by google but it seems nobody has this question.
Can somebody teach me how to solve it?
Thank you a lot.
bool result = false;
string zipFile = "";
using (SftpClient sftp = new SftpClient(ip, user, pw))
{
sftp.Connect();
var sftpFiles = GetFileList("/", sftp);
zipFile = GetZipFile(sftpFiles);
if (zipFile != null)
{
var file = File.OpenWrite(fileName);
sftp.DownloadFile(fileName,file);
result = true;
sftp.Disconnect();
}
}
if (result)
{
using (ZipFile zipList = ZipFile.Read(fileName))
{
ZipEntry zipFile = zipList[fileName];
zipFile.Extract("/", ExtractExistingFileAction.OverwriteSilently);
}
}

You do not close the file after you download it.
The download code should be like:
using (var file = File.OpenWrite(fileName))
{
sftp.DownloadFile(fileName, file);
}

Get FTP file details based on datetime in C#

Question: I want to get file Details from FTP server based on some specific datetime without using any 3rd party.
Problem : My FTP server contains 1000s of files so getting all files and after that filtering it takes time.
Is there any Quicker way to do this ?
string ftpPath = "ftp://directory/";
// Some expression to match against the files...do they have a consistent
// name? This example would find XML files that had 'some_string' in the file
Regex matchExpression = new Regex("^test.+\.xml$", RegexOptions.IgnoreCase);
// DateFilter
DateTime cutOff = DateTime.Now.AddDays(-10);
List<ftplineresult> results = FTPHelper.GetFilesListSortedByDate(ftpPath, matchExpression, cutOff);
public static List<FTPLineResult> GetFilesListSortedByDate(string ftpPath, Regex nameRegex, DateTime cutoff)
{
List<FTPLineResult> output = new List<FTPLineResult>();
FtpWebRequest request = FtpWebRequest.Create(ftpPath) as FtpWebRequest;
ConfigureProxy(request);
request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
FtpWebResponse response = request.GetResponse() as FtpWebResponse;
StreamReader directoryReader = new StreamReader(response.GetResponseStream(), System.Text.Encoding.ASCII);
var parser = new FTPLineParser();
while (!directoryReader.EndOfStream)
{
var result = parser.Parse(directoryReader.ReadLine());
if (!result.IsDirectory && result.DateTime > cutoff && nameRegex.IsMatch(result.Name))
{
output.Add(result);
}
}
// need to ensure the files are sorted in ascending date order
output.Sort(
new Comparison<FTPLineResult>(
delegate(FTPLineResult res1, FTPLineResult res2)
{
return res1.DateTime.CompareTo(res2.DateTime);
}
)
);
return output;
}

Problem : My FTP server contains 1000s of files so geting all files and after that filtering it takes time.
Is there any Quicker way to do this ?
No.
The only standard FTP API, is the LIST command and its companions. All these will give you list of all files in a folder. There's no FTP API to give you files filtered by a timestamp.
Some servers support non-standard file masks in the LIST command.
So they will allow you to return only the *.xml files.
See How to get list of files based on pattern matching using FTP?
Similar questions:
Download files from FTP if they are created within the last hour
C# - Download files from FTP which have higher last-modified date

I have got an alternative solution to do my functionality using FluentFTP.
Explanation:
I am downloading the files from FTP (Read permission reqd.) with same folder structure.
So everytime the job/service runs I can check into the physical path same file(Full Path) exists or not If not exists then it can be consider as a new file. And Ii can do some action for the same and download as well.
Its just an alternative solution.
Code Changes:
private static void GetFiles()
{
using (FtpClient conn = new FtpClient())
{
string ftpPath = "ftp://myftp/";
string downloadFileName = #"C:\temp\FTPTest\";
downloadFileName += "\\";
conn.Host = ftpPath;
//conn.Credentials = new NetworkCredential("ftptest", "ftptest");
conn.Connect();
//Get all directories
foreach (FtpListItem item in conn.GetListing(conn.GetWorkingDirectory(),
FtpListOption.Modify | FtpListOption.Recursive))
{
// if this is a file
if (item.Type == FtpFileSystemObjectType.File)
{
string localFilePath = downloadFileName + item.FullName;
//Only newly created files will be downloaded.
if (!File.Exists(localFilePath))
{
conn.DownloadFile(localFilePath, item.FullName);
//Do any action here.
Console.WriteLine(item.FullName);
}
}
}
}
}

Upload file to skydrive through SkyDrive API

I try to upload a text file to my skydrive or at least create new text file in SD and edit it's content, through SkyDrive API in my Windows 8 application.
How can I do that?
I tried to do something like that:
LiveConnectClient client = new LiveConnectClient(session);
var fileData = new Dictionary<string, object>();
fileData.Add("name", "new_file.txt");
try
{
LiveOperationResult fileOperationResult = await client.PutAsync("me/skydrive", fileData);
this.infoTextBlock.Text = fileOperationResult.ToString();
}
catch (LiveConnectException exception)
{
this.infoTextBlock.Text = exception.Message;
}
but I get error
"The provided request is not valid. The root SkyDrive folder cannot be updated."
If I write something like "me/skydrive/" I get
"The provided URL is not valid. The requested path '' is not supported".
Method LiveConnectClient.PutAsync allows me only to update existing properties (but not it's content).
How it should be done properly?
Btw - Is content on LCDC(http://msdn.microsoft.com/en-us/library/live/hh826531.aspx) updated? I'm asking because some methods, which are in documentation, doesn't exist in dlls (f.e. LiveConnectClient.Upload. There's only BackgroundUploadAsync).
Thanks for help in advance,
Micheal

Close but as I wrote: I can't use client.upload method because LiveConnectClient class doesn't contain it. That's why I asked about site content update.
Anyway - I've got an answer:
//create a StorageFile (here is one way to do that if it is stored in your ApplicationData)
StorageFile file = awaitApplicationData.Current.LocalFolder.GetFileAsync("yourfilename.txt");
try {
client = new LiveConnectClient(session);
LiveOperationResult operationResult = await client.BackgroundUploadAsync("me/skydrive", file.Name, file, OverwriteOption.Overwrite);
}
catch (LiveConnectException exception) {
//handle exception
}

You should use the Upload method on LiveConnectionClient. For example, see the Uploading Files example in the Live SDK. Something like ...
LiveOperationResult fileOperationResult =
await client.Upload("me/skydrive", /*file name here*/, /*file stream here*/);

Here's another way to upload a file from a console application using a SkyDriveApiClient downloaded from http://skydriveapiclient.codeplex.com/releases/view/103081
static void Main(string[] args)
{
var client = new SkyDriveServiceClient();
client.LogOn("YourEmail#hotmail.com", "password");
WebFolderInfo wfInfo = new WebFolderInfo();
WebFolderInfo[] wfInfoArray = client.ListRootWebFolders();
wfInfo = wfInfoArray[0];
client.Timeout = 1000000000;
string fn = #"test.txt";
if (File.Exists(fn))
{
client.UploadWebFile(fn, wfInfo);
}
}