Using ThreadPool to Upload Files on S3 via AWS SDK - c#

I am trying to upload files to S3, to improve performance I am trying to spawn a threadpool of 100 threads to execute this task however when I trigger my console application it just stops executing once I run it and breaks.
This is what I tried so far,
string bucketName = <<bucket-name>>;
var threadCount = Environment.ProcessorCount;
ThreadPool.SetMaxThreads(100, 100);
ThreadPool.QueueUserWorkItem(new WaitCallback((obj) =>
{
UploadFile(bucketName);
}));
public void UploadFile(string bucketName)
{
TransferUtilityUploadDirectoryRequest uploadDirectoryRequest;
string directoryName= #"D:\Files\100KB";
try
{
uploadDirectoryRequest = new TransferUtilityUploadDirectoryRequest
{
BucketName = bucketName,
Directory = directoryName,
KeyPrefix = "upload_dir",
UploadFilesConcurrently = true
};
var stopwatch = new Stopwatch();
stopwatch.Start();
transferUtility.UploadDirectory(uploadDirectoryRequest);
stopwatch.Stop();
Console.WriteLine("Time Elapsed:"+stopwatch.Elapsed);
}
catch (AmazonS3Exception exception)
{
Console.WriteLine(exception.Message);
}
catch (Exception exception)
{
Console.WriteLine(exception.Message);
}
}
Any idea what I could be doing wrong with respect to using the threadpool? What would be the correct way to write this logic?

Related

HttpClient GetAsync taking ~2 seconds

My wpf program reads commands from a text file, and then sends those commands out over HttpClient.
Each GetAsync response takes about 2 seconds to complete. This seems excessively long, especially when I have 50 commands to send.
Is this a normal amount of time for HttpClient to send/receive a GetAsync Message? Is there a faster way to do this?
static readonly HttpClient client = new HttpClient();
public MainWindow()
{
InitializeComponent();
}
private async void Register_ClickAsync(object sender, RoutedEventArgs e)
{
int counter = 0;
string line;
System.IO.StreamReader file = new System.IO.StreamReader(#"C\path.txt");
while ((line = file.ReadLine()) != null)
{
try
{
var watch = Stopwatch.StartNew();
HttpResponseMessage response = await client.GetAsync(line);
response.EnsureSuccessStatusCode();
string responseBody = await response.Content.ReadAsStringAsync();
watch.Stop();
var elapsedMS = watch.ElapsedMilliseconds;
RequestTextBox.Text += "\n" + elapsedMS;
}
catch (HttpRequestException ex)
{
Console.WriteLine("\nException Caught!");
Console.WriteLine("Message :{0} ", ex.Message);
this.RequestTextBox.Text += ("\n" + "Message: {0} ", ex.Message);
}
}
file.Close();
}
}
UPDATE:
This original program is a .net Core WPF app. I created a .net framework WPF app with the exact same code. The .net framework app takes 2000 ms to send the first HttpClient GetAsync command, and then <10 ms for subsequent commands. This is a massive performance increase. Does anyone have an explanation? I have been unable to find any articles explaining this.
Your code is awaiting for each request.
You can create many tasks by calling GetAsync and then wait them all with Task.WhenAll(). It will be more faster.
You are not using the async paradigm correctly. You are awaiting an async call immediately, and as a result it acts the same as a synchronous method. The idea of an async method is that it starts work and runs in the background, while not blocking other threads from executing. This allows you to run many threads at once and then return them when you are ready to use them. I would highly recommend the msft docs in this case. They use the example of making breakfast, in this example they are doing it incorrectly, as you are in your question:
Coffee cup = PourCoffee();
Console.WriteLine("coffee is ready");
Task<Egg> eggsTask = FryEggs(2);
Egg eggs = await eggsTask;
Console.WriteLine("eggs are ready");
Task<Bacon> baconTask = FryBacon(3);
Bacon bacon = await baconTask;
Console.WriteLine("bacon is ready");
Task<Toast> toastTask = ToastBread(2);
Toast toast = await toastTask;
ApplyButter(toast);
ApplyJam(toast);
Console.WriteLine("toast is ready");
Juice oj = PourOJ();
Console.WriteLine("oj is ready");
Console.WriteLine("Breakfast is ready!");
To fix it you must call Task.WhenAny() or Task.WhenAll(). This allows you to run all of the tasks at once and return them after they have finished (or as they finish). This will greatly improve your performance.
static async Task Main(string[] args)
{
Coffee cup = PourCoffee();
Console.WriteLine("coffee is ready");
var eggsTask = FryEggsAsync(2);
var baconTask = FryBaconAsync(3);
var toastTask = MakeToastWithButterAndJamAsync(2);
var allTasks = new List<Task>{eggsTask, baconTask, toastTask};
while (allTasks.Any())
{
Task finished = await Task.WhenAny(allTasks);
if (finished == eggsTask)
{
Console.WriteLine("eggs are ready");
}
else if (finished == baconTask)
{
Console.WriteLine("bacon is ready");
}
else if (finished == toastTask)
{
Console.WriteLine("toast is ready");
}
allTasks.Remove(finished);
}
Juice oj = PourOJ();
Console.WriteLine("oj is ready");
Console.WriteLine("Breakfast is ready!");
async Task<Toast> MakeToastWithButterAndJamAsync(int number)
{
var toast = await ToastBreadAsync(number);
ApplyButter(toast);
ApplyJam(toast);
return toast;
}
}

create multiple files using multiple threads c#

I am creating console app to simulate server. I create multiple virus files together using multiple threads to see whether all files get quarantined, if yes, how long it takes to quarantined. The problem with multithreading application is one thread starts writing another thread so I get exception - The process can not access the file X because the file is being used by another process. This is the reason that all files don't get quarantined. I use framework 4.5.2
I have created app using thread and task. I don't get the desire result. What is the best practice to write this app? Thank you for helping me in advance.
Using Thread:
class Program
{
static string folderPath;
static readonly string fileContent = #"X5O!P%#AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*";
static void Main(string[] args)
{
folderPath = "F:\VirusScan";
int counter = 1000;
for (int i = 0; i < counter; i++)
{
var thread = new Thread(() => GenerateVirusFile(i));
thread.Start();
}
Console.ReadKey();
}
static void GenerateVirusFile(int i)
{
string filePath = $#"{folderPath}\TestForVirusScan_{i}_{DateTime.Now.ToString("yyyyMMddHHmmssffff")}.txt";
try
{
using (StreamWriter writer = new StreamWriter(filePath))
{
writer.WriteLine(fileContent);
}
var timer = Stopwatch.StartNew();
while (true)
{
if (!File.Exists(filePath))
{
Console.WriteLine($"{i}: File was removed in {timer.ElapsedMilliseconds}ms");
break;
}
else
{
Thread.Sleep(1);
}
}
}
catch (Exception ex)
{
Console.WriteLine($"{i}: Exception {ex.GetType().Name} occurred: {ex.Message}");
}
}
}
Using Task:
class Program
{
static string folderPath;
static readonly string fileContent = #"X5O!P%#AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*";
static void Main(string[] args)
{
folderPath = "F:\VirusScan";
int counter = 1000;
List<Task> tasks = new List<Task>();
for (int i = 1; i <= counter; i++)
{
Task newTask = new Task((x) => GenerateVirusFile(x), i);
tasks.Add(newTask);
}
foreach (var task in tasks)
{
task.Start();
}
Task.WaitAll(tasks.ToArray());
Console.ReadKey();
}
public static void GenerateVirusFile(object i)
{
string filePath = $#"{folderPath}\TestForVirusScan_{i}_{DateTime.Now.ToString("yyyyMMddHHmmssffff")}.txt";
try
{
using (StreamWriter writer = new StreamWriter(filePath))
{
writer.WriteLine(fileContent);
}
var timer = Stopwatch.StartNew();
while (true)
{
if (!File.Exists(filePath))
{
Console.WriteLine($"{i}: File was removed in {timer.ElapsedMilliseconds}ms");
break;
}
else
{
Thread.Sleep(1);
}
}
}
catch (Exception ex)
{
Console.WriteLine($"{i}: Exception {ex.GetType().Name} occurred: {ex.Message}");
}
}
}
The problem is in the following code:
for (int i = 0; i < counter; i++)
{
var thread = new Thread(() => GenerateVirusFile(i));
thread.Start();
}
The closure () => GenerateVirusFile(i) is referencing changing variable
Rewrite it in the following way:
Parallel.For(0, counter, GenerateVirusFile);
Have you tried something like this in your loop:
int x = i;
var thread = new Thread(() => GenerateVirusFile(x));
this prevents that the same i is used for more threads/file names.

C# Download Speed Asynchronously

I'm trying to get the current user's network download speed. After hitting a dead end with NetworkInterfaces and all I tried a solution I found online. I edited it a bit and it works great but it's not asynchronous.
public static void GetDownloadSpeed(this Label lbl)
{
double[] speeds = new double[5];
for (int i = 0; i < 5; i++)
{
int fileSize = 407; //Size of File in KB.
WebClient client = new WebClient();
DateTime startTime = DateTime.Now;
if (!Directory.Exists($"{CurrentDir}/tmp/speedtest"))
Directory.CreateDirectory($"{CurrentDir}/tmp/speedtest");
client.DownloadFile(new Uri("https://ajax.googleapis.com/ajax/libs/threejs/r69/three.min.js"), "/tmp/speedtest/three.min.js");
DateTime endTime = DateTime.Now;
speeds[i] = Math.Round((fileSize / (endTime - startTime).TotalSeconds));
}
lbl.Text = string.Format("{0}KB/s", speeds.Average());
}
That function is called within a timer at an interval of 2 minutes.
MyLbl.GetDownloadSpeed()
I've tried using WebClient.DownloadFileAsync but that just shows the unlimited symbol.My next try would be to use HttpClient but before I go on does anyone have a recommended way of getting the current users download speed asynchronously (without lagging the main GUI thread)?
As it was suggested you could make an async version of GetDownloadSpeed():
async void GetDownloadSpeedAsync(this Label lbl, Uri address, int numberOfTests)
{
string directoryName = #"C:\Work\Test\speedTest";
string fileName = "tmp.dat";
if (!Directory.Exists(directoryName))
Directory.CreateDirectory(directoryName);
Stopwatch timer = new Stopwatch();
timer.Start();
for (int i = 0; i < numberOfTests; ++i)
{
using (WebClient client = new WebClient())
{
await client.DownloadFileTaskAsync(address, Path.Combine(directoryName, fileName), CancellationToken.None);
}
}
lbl.Text == Convert.ToString(timer.Elapsed.TotalSeconds / numberOfTests);
}
WebClient class being relatively old does not have awaitable DownloadFileAsync().
EDITED
As it was correctly pointed out WebClient in fact has a task-based async method DownloadFileTaskAsync(), which i advise to use. The code below can still help addressing the case when async method returning Task is not provided.
We can fix it with the help of TaskCompletionSource<T>:
public static class WebClientExtensions
{
public static Task DownloadFileAwaitableAsync(this WebClient instance, Uri address,
string fileName, CancellationToken cancellationToken)
{
TaskCompletionSource<object> tcs = new TaskCompletionSource<object>();
// Subscribe for completion event
instance.DownloadFileCompleted += instance_DownloadFileCompleted;
// Setup cancellation
var cancellationRegistration = cancellationToken.CanBeCanceled ? (IDisposable)cancellationToken.Register(() => { instance.CancelAsync(); }) : null;
// Initiate asyncronous download
instance.DownloadFileAsync(address, fileName, Tuple.Create(tcs, cancellationRegistration));
return tcs.Task;
}
static void instance_DownloadFileCompleted(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
{
((WebClient)sender).DownloadDataCompleted -= instance_DownloadFileCompleted;
var data = (Tuple<TaskCompletionSource<object>, IDisposable>)e.UserState;
if (data.Item2 != null) data.Item2.Dispose();
var tcs = data.Item1;
if (e.Cancelled)
{
tcs.TrySetCanceled();
}
else if (e.Error != null)
{
tcs.TrySetException(e.Error);
}
else
{
tcs.TrySetResult(null);
}
}
}
Try `await Task.Run(()=> { //your code });
Edit: #JustDevInc I still think you should use DownloadAsync. Task.Run(delegate) creates a new thread and you might want to avoid that. If you want, post some of your old code so we can try to fix it.
Edit: The first solution turned out to be the only one of the two working. DownloadFileAsync doesn't return task, so can't it awaited.

c# BackgroundWorker pause

I have a small C# application that communicate with a server and get some data via API request, using POST method. It is an Apache server by the way.
My problem is that my C# app sends a tons of requests continuously, and the server creates a tons of log files.
I use a BackgroundWorker and I want to pause it for a few seconds, but Thread.Sleep(5000) doesn't working.
This app is running in the system tray it doesn't have a GUI, just get some content, and print them out.
Code:
private void _bgWorker_DoWork(object sender, DoWorkEventArgs e)
{
JSONParser parser = new JSONParser(_config.Prop.server + "content/api.php", "print", "getAll");
try
{
while (!_bgWorker.CancellationPending)
{
try
{
JSONPrintNeeds needs = parser.DownloadAll();
List<JSONPrintNeed> temp = new List<JSONPrintNeed>();
foreach (JSONPrintNeed need in needs.data)
{
temp.Add(need);
}
foreach (JSONPrintNeed need in temp)
{
Printer printer = new Printer(need.megrendeles);
printer.PrintController = new StandardPrintController();
List<String> installed = new List<String>();
for (int i = 0; i < PrinterSettings.InstalledPrinters.Count; i++)
{
installed.Add(PrinterSettings.InstalledPrinters[i]);
}
if (installed.Contains(need.nyomtato))
{
printer.PrinterSettings.PrinterName = need.nyomtato;
}
int format = int.Parse(need.format);
switch (format)
{
case 0:
default: // txt
printer.Print();
break;
case 1: // html
SetDefaultPrinter(need.nyomtato);
browser.DocumentText = need.megrendeles;
browser.Print();
break;
}
JSONResult result = parser.DeleteOne(int.Parse(need.ny_id));
}
parser.DeleteAll();
Thread.Sleep(5000);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
you can use EventWaitHandle for sync threads.
EventWaitHandle flag = new EventWaitHandle(false, EventResetMode.AutoReset);
if (stop)
{
flag.WaitOne(5000);
}

how to run asynchronously

I have to load two large files in parallels
so far I have this code
The code below is click button method
private async void MILoadLogFile_Click(object sender, RoutedEventArgs e)
{
...
if (oFD.ShowDialog() == true)
{
await myLogSession.LoadCompassLogAsync(oFD.FileName);
await myLogSession.LoadCoreServiceLogAsync(oFD.FileName);
}
}
loading method:
public async Task LoadCompassLogAsync(String fileName)
{
StreamReader streamReader = new StreamReader(fileName);
if (fileName.Contains("Compass"))
{
...
try
{
using (streamReader)
{
//Console.Out.WriteLine("lineCount: " + lineCount);
while (((line = await streamReader.ReadLineAsync()) != null)
&& !CompassLogLoadCompleted)
{
...
loggingLvl = new LoggingLvl(eLoggingLvl);
CompassLogData cLD = new CompassLogData(id, dateTime, loggingLvl, threadId, loggingMessage);
await addRoCompassLogCollectionAsync(cLD);
}
}
}
catch (Exception e)
{
Console.WriteLine("The file could not be read:");
Console.WriteLine(e.Message);
}
}
}
the LoadCoreServiceLogAsync is almost identical to LoadCompassLogAsync.
The two loading methods runs sequentially. I want them to run in parallel.
Your code will run one task after the other. To run the two tasks in parallel you can use the Task.WaitAll method:
var loadCompassLogTask = myLogSession.LoadCompassLogAsync(oFD.FileName);
var loadCoreServiceLogTask = myLogSession.LoadCoreServiceLogAsync(oFD.FileName);
Task.WaitAll(loadCompassLogTask, loadCoreServiceLogTask);
Or if you want to use await you can use Task.WhenAll:
var loadCompassLogTask = myLogSession.LoadCompassLogAsync(oFD.FileName);
var loadCoreServiceLogTask = myLogSession.LoadCoreServiceLogAsync(oFD.FileName);
await Task.WhenAll(loadCompassLogTask, loadCoreServiceLogTask);

Categories

Resources