Transfer of large number of files using SFTP is slow in C# - c#

I am uploading 4000 zip files of size 85 KB of each using over Linux Server using SFTP in C# WPF application.
This whole process takes 30 minutes.
Is there any way to speed up the uploading using SFTP?
I'm using WinSCP .NET assembly:
https://winscp.net/eng/docs/library
I had also used Chilkat previously.
Here is my code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using WinSCP;
namespace SFTP_Demo
{
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
string line;
SessionOptions sessionoptions = new SessionOptions()
{
Protocol = WinSCP.Protocol.Sftp,
HostName = "172.168.1.7",
PortNumber = 22,
UserName = "lduser",
Password = "lduser",
GiveUpSecurityAndAcceptAnySshHostKey = true
};
using (Session session = new Session())
{
session.Open(sessionoptions);
TransferOptions transferOptions = new TransferOptions();
transferOptions.TransferMode = TransferMode.Binary;
TransferOperationResult transferResult;
System.IO.StreamReader file = new System.IO.StreamReader(txtFile.Text);
while ((line = file.ReadLine()) != null)
{
transferResult = session.PutFiles(#"D:\Test\signature\ldoutput\"+line, "/SFTP/", false, transferOptions);
transferResult.Check();
counter++;
strbldr = strbldr.AppendLine(string.Format("{0} Upload of {1} succeeded", counter + 1.ToString(), line));
}
}
}
}
}

There's quite an overhead with each file (opening, closing, updating timestamps). So the transfer of a large number of small files is pretty inefficient.
What you can do, is to parallelize the transfer.
Collect the list of files using Session.ListDirectory (or Session.EnumerateRemoteFiles if you need recursion) and split the list to batches, transferring each in a separate thread.
See this example:
Automating download in parallel connections over SFTP/FTP protocol

Related

Close Excel in C# after the saving the file

I am using C# to write some information to an Excel file:
using System;
using System.IO;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
using System.Diagnostics;
using Microsoft.Office.Interop.Excel;
using Excel = Microsoft.Office.Interop.Excel;
using System.Globalization;
public void OpenExcel(string folder, string filename)
{
this.folder = folder;
this.filename = filename;
path = folder + #"\" + filename;
if (File.Exists(path) != true) //if the output file does not exists, create it
{
var app = new Microsoft.Office.Interop.Excel.Application();
var temp = app.Workbooks.Add();
temp.SaveAs(path);
temp.Close();
}
wb = excel.Workbooks.Open(path);
}
After I am done and have saved the file, I want to close it:
public void Close()
{
wb.Close(false);
excel.Quit();
excel.Application.Quit();
System.Runtime.InteropServices.Marshal.ReleaseComObject(workSheet);
System.Runtime.InteropServices.Marshal.ReleaseComObject(wb);
System.Runtime.InteropServices.Marshal.ReleaseComObject(excel);
workSheet = null;
wb = null;
excel = null;
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
}
Both of the voids above are part of a class object named ExcelFile. However, when I use the code above and exit the program I have made, Excel still is left as a background process like in this question: C# closing Excel after using
I have seen that there are a few threads on stackoverflow on this topic, but no solution seems to work for me. I want to avoid killing all Excel processes if possible since this is a program that will make calculations using thread pool that will take hours to make.

Custom log4net azure blob appender stops logging when a file reaches 5.4MB

I am using the appender below to pipe our log messages to Azure with Log4Net.
Everything works fine, until the blob file reaches the size of 5.4MB. From than, no logs are appended to the log file anymore.
What is causing this? And how can I solve this?
namespace Digicreate.Core.Infrastructure.Logging
{
using System;
using System.Configuration;
using System.Diagnostics;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Text;
using System.Threading.Tasks;
using log4net.Appender;
using log4net.Config;
using log4net.Core;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
// ReSharper disable once UnusedMember.Global
// justification: used in log4net configuration file
public class AzureBlobTxtFileAppender : BufferingAppenderSkeleton
{
private CloudStorageAccount _account;
...removed unimportant code...
protected override void SendBuffer(LoggingEvent[] events)
{
var appendBlobReference = _cloudBlobContainer.GetAppendBlobReference(Filename(DirectoryName));
if (!appendBlobReference.Exists())
{
appendBlobReference.CreateOrReplace();
}
Parallel.ForEach(events, ProcessEvent);
}
private static string Filename(string directoryName)
{
return $"{directoryName}/{DateTime.Today.ToString("yyyy_MM_dd", DateTimeFormatInfo.InvariantInfo)}.log.txt";
}
private void ProcessEvent(LoggingEvent loggingEvent)
{
using (var memoryStream = new MemoryStream())
{
using (var streamWriter = new StreamWriter(memoryStream, Encoding.UTF8, 1024, true))
{
Layout.Format(streamWriter, loggingEvent);
streamWriter.Flush();
}
memoryStream.Position = 0;
_cloudBlobContainer
.GetAppendBlobReference(Filename(DirectoryName))
.AppendBlock(memoryStream);
}
}
}
}
According to Append Block documentation you might be hittin 50,000 block limit for a blob. Make sure to check for number of committed blocks before committing to the Blob using:
var blockCount = _cloudBlobContainer.GetAppendBlobReference(Filename(DirectoryName))
.Properties
.AppendBlobCommittedBlockCount;
and make sure it is not more than 50000.

How to read text from word file? C#

I am trying to read a text from document file in C#
I can't create new Document in C#.
I try to using "using Microsoft.Office.Interop.Word"
or "using System.Windows.Documents" but it does'nt recognize the code "Document doc=new Document".
In addition how can I read a text from .docx file?
The code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using Microsoft.Office.Interop.Word; //didnt recognize "Office"
namespace DocumentTest1
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
}
Application word = new Application();
Document doc = new Document();
}
}
What could be the problem? tnx
Hi Please see below code
List<string> data = new List<string>();
Application app = new Application();
Document doc = app.Documents.Open(ref readFromPath);
foreach (Paragraph objParagraph in doc.Paragraphs)
data.Add(objParagraph.Range.Text.Trim());

write to memoerystream using epplus

I have this assembly that basically creates an empty excel file using EPPLUS library. The file is created directly to desk, but i want first to write it in a memoerystream and then save the memoerystream to desk. I have this so far.
using OfficeOpenXml;
using System;
using System.Collections.Generic;
using System.Data;
using System.Data.SqlClient;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleApplicationXLSX
{
class Program
{
static void Main(string[] args)
{
Createxlsx("XLSX");
}
public static void Createxlsx(string filename)
{
FileInfo newFile = new FileInfo("C:\\ConsoleApplicationXLSX\\" + filename + ".xlsx");
MemoryStream ms = new MemoryStream();
//create a package
using (ExcelPackage package = new ExcelPackage(ms))
{
package.Workbook.Worksheets.Add("worksheet");
package.Save();
}
}
}
}
Try
package.SaveAs(ms)
That should fill your stream with the excelfile-content.

Obtaining Errors on a code for loading and displaying a DICOM image in C#

There was individual, whom post a code to load and display a DICOM image in C# with the ClearCanvas Library. However, I tried t run the code and i'm receiving an error and have a missing library dll component. I would like to know where the individual obtained the library dll file ClearCanvas.Dicom.ImageViewwer.StudyManagement. I've been unable to locate that file on the internet. The code is displayed after the errors. I am grateful thanks.
Usings:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
using ClearCanvas.Common;
using ClearCanvas.Dicom;
using System.Windows.Media.Imaging;
using ClearCanvas.ImageViewer;
using ClearCanvas.ImageViewer.StudyManagement;
using System.Windows.Interop;
using System.Windows.Media;
This is the body:
OpenFileDialog ofd = new OpenFileDialog();
ofd.Filter = "DICOM Files(*.*)|*.*";
if (ofd.ShowDialog() == DialogResult.OK)
{
if (ofd.FileName.Length > 0)
{
var imagen = new DicomFile(ofd.FileName);
LocalSopDataSource DatosImagen = new LocalSopDataSource(ofd.FileName);
ImageSop imageSop = new ImageSop(DatosImagen);
IPresentationImage imagen_a_mostrar = PresentationImageFactory.Create(imageSop.Frames[1]);
int width = imageSop.Frames[1].Columns;
int height = imageSop.Frames[1].Rows;
Bitmap bmp = imagen_a_mostrar.DrawToBitmap(pictureBox1.Width, pictureBox1.Height);
pictureBox1.Image = bmp;
imageOpened = true;
}
ofd.Dispose();
}
}
In addition to referencing the proper assemblies (here... it's linked from the main page of clearcanvas.ca) I mentioned in a comment, consider employing the using keyword since OpenFileDialog implements IDisposable like so:
using(OpenFileDialog ofd = new OpenFileDialog())
{
ofd.Filter = "DICOM Files(*.*)|*.*";
if (ofd.ShowDialog() == DialogResult.OK)
{
// rest of your code goes here
}
}
This has the benefit of calling ofd.Dispose() automatically, even if the code inside the using block throws an exception. This works for anything that implements IDisposable (so bitmaps, fonts, streams, etc.)
I posted the code actually..
These are the DLLs I used in my project:
Clearcanvas:
ClearCanvas.Common.dll
ClearCanvas.Desktop.dll
ClearCanvas.Dicom.dll
ClearCanvas.ImageViewer.Common.dll
ClearCanvas.ImageViewer.dll
ClearCanvas.Utilities.Manifest.dll
Others I also added since I was getting an error when trying to show the image..
log4net.dll
nunit.framework.dll
WindowsBase.dll
BilinearInterpolation.dll --> For this one make sure you add either the 32 or 64 bits dll depending on what's your project defaults.

Categories

Resources