It seems like putting Task.WhenAll() in a try-catch block doesn't work. The code should upload all the images to ftp, but it seems like if one upload fails (for example, the image file is open in another process), the whole uploadTasks will stop, and counts is empty.
private async Task Upload(Ftp ftpHost)
{
var images = GetFileInfo() // will get a List<FileInfo>
var uploadTasks = images.Where(fi => fi.Exists).Select(async fi =>
{
var ret = await ftp.Upload(fi,_cancellationTokenSource.Token);
fi.Delete();
return ret;
});
IEnumerable<int> counts = new List<int>();
try
{
counts = await Task.WhenAll(uploadTasks);
}
catch
{
//ignore to allow all upload complete
}
var uploadedOrderFileByteCount = counts.Sum();
var uploadedOrderFileCount = counts.Count();
}
Apart from the fact that an empty catch block is often a bad idea (try to catch the exception which the ftp upload can throw especifically), if that's what you want to do, then the easiest way is to catch inside the operation itself, something similar to the code below.
var uploadTasks = images.Where(fi => fi.Exists).Select(async fi =>
{
try
{
var ret = await ftp.Upload(fi,_cancellationTokenSource.Token);
fi.Delete();
return ret;
} catch {
// again, please don't do this...
return 0;
}
});
IEnumerable<int> counts = new List<int>();
try
{
counts = await Task.WhenAll(uploadTasks);
}
catch
{
// Something bad happened, don't just ignore it
}
Related
New to this parallel coding.
I am trying to fire off a list of tasks (in this case, emails to send).
The below code does work, however I am unsure, if ONE email was to fail sending, or that task not finish for whatever reason etc.
Would me code just hang on the await line?
What the solution here, or is it a non problem?
I need to await OR intercept each one when it finishes as I need to (mark the email as sent correctly or retry etc).
However is awaiting all safe, or should I have an event to intercept the response?
(I do not really care if the odd one takes a while, My key is not to block up the rest of the program)
Over all this will be in a 5 mins loop, and I wouldn't want the email in the future to be blocked up is a timeout per task, as option.
Any help would be great, thank You.
// Create a list of emails to send
List<Task<string>> tasks = new List<Task<string>>();
foreach (var item in lstEmailList) // loop the list of emails to send
{
tasks.Add(EmailLib.SendEmailOldSmtpAuth(item)); // add each email as a task
}
// now run the list as parallel tasks
await Task.Run(() => Parallel.ForEach(tasks, s =>
{
//EmailLib.SendEmailOldSmtpAuth(item);
}));
foreach (var task in tasks)
{
var result = ((Task<string>)task).Result;
if (result.Contains("PH_OK") == true)
{
string strFindId = result.Replace("PH_OK=", "").ToString();
int EmailID = Int32.Parse(strFindId);
DbFunc.MarkEmailAsSent(ref SQLConnX, EmailID);
}
}
In case it is helpful:
public async Task<string> SendEmailOldSmtpAuth(Data_PendingEmails objEmail)
{
try
{
var emailMessage = new MimeMessage();
string SendToName = "";
if (objEmail.CarerID > 0)
{
SendToName = objEmail.carForename + " " + objEmail.carSurname;
}
else
{
SendToName = objEmail.cliForename + " " + objEmail.cliSurname;
}
if (SendToName == "")
{
SendToName = "User";
}
emailMessage.From.Add(new MailboxAddress(objEmail.EMailYourName, objEmail.EMailAddress));
emailMessage.To.Add(new MailboxAddress(SendToName, objEmail.ToEmailAddress));
emailMessage.Subject = objEmail.EmailSubject;
emailMessage.Body = new TextPart("html") { Text = objEmail.EmailMessage };
try
{
var client = new SmtpClient();
await client.ConnectAsync(objEmail.SMTPServer, Int32.Parse(objEmail.SMTPPort), SecureSocketOptions.SslOnConnect);
await client.AuthenticateAsync(objEmail.SMTPUserName, objEmail.SMTPPassword);
await client.SendAsync(emailMessage);
await client.DisconnectAsync(true);
return "PH_OK=" + objEmail.EmailID.ToString();
}
catch (Exception ex)
{
var e = ex;
return e.Message;
}
}
catch (Exception SendEmailOldSmtpAuthOverAll)
{
return SendEmailOldSmtpAuthOverAll.Message.ToString();
}
}
I will try to simulate your scenario with a much simpler example but the main idea will be the same. I do it this way to simulate the potential exception.
First af all you need concurrent calls to a web service, so the best way is to use Task.WhenAll because this is an I/O operation (as #Charlieface already mentioned)
Lets say that we have a list of emails:
var emailList = new List<string>() { "1", "2", "3", "4", "5", "6" };
then we need to create an ienumerable of tasks:
var tasks = emailList.Select(async email =>
{
var response = await SendEmailAsync(email);
Console.WriteLine(response);
});
Then we "mock" an exception and append it to task list in order to simulate an exception:
var problematicTask = ThrowExceptionAsync("Error from initial task");
var allTasks = tasks.Append(problematicTask);
Now, in order to fire the tasks and to catch the exceptions we need to do this:
var aggregateTasks = Task.WhenAll(allTasks);
try
{
await aggregateTasks;
}
catch
{
AggregateException aggregateException = aggregateTasks.Exception!;
foreach (var ex in aggregateException.InnerExceptions)
{
Console.WriteLine(ex.Message);
}
}
and as helper methods for this example i created these two arbitrary methods:
async Task<string> SendEmailAsync(string email)
{
await Task.Delay(1500);
if (email.Equals("7") || email.Equals("10"))
{
await ThrowExceptionAsync("Error from sendEmail");
}
return $"Email: {email} sent";
}
async Task ThrowExceptionAsync(string msg)
{
throw new Exception(msg);
}
If you run this simple example you would inspect all exceptions may thrown in each call..
Now in regards to your particular example i think you need to remove all try/catch blocks in order to catch the exceptions as aggregate exception and not blocking your app.
My telegram bot is necessary so that the user can answer questions in order and save these answers in the same order for a specific user in parallel.
static readonly ConcurrentDictionary<int, string[]> Answers = new ConcurrentDictionary<int, string[]>();
static void Main(string[] args)
{
try
{
Task t1 = CreateHostBuilder(args).Build().RunAsync();
Task t2 = BotOnMessage();
await Task.WhenAll(t1, t2);
}
catch (Exception ex)
{
Console.WriteLine("Error" + ex);
}
}
here is my BotOnMessage() method to receive and process messages from users
async static Task BotOnMessage()
{
int offset = 0;
int timeout = 0;
try
{
await bot.SetWebhookAsync("");
while (true)
{
var updates = await bot.GetUpdatesAsync(offset, timeout);
foreach (var update in updates)
{
var message = update.Message;
if (message.Text == "/start")
{
Registration(message.Chat.Id.ToString(), message.Chat.FirstName.ToString(), createdDateNoTime.ToString("yyyy-MM-dd HH:mm:ss.fff", CultureInfo.InvariantCulture));
var replyKeyboard = new ReplyKeyboardMarkup
{
Keyboard = new[]
{
new[]
{
new KeyboardButton("eng"),
new KeyboardButton("ger")
},
}
};
replyKeyboard.OneTimeKeyboard = true;
await bot.SendTextMessageAsync(message.Chat.Id, "choose language", replyMarkup: replyKeyboard);
}
switch (message.Text)
{
case "eng":
var replyKeyboardEN = new ReplyKeyboardMarkup
{
Keyboard = new[]
{
new[]
{
new KeyboardButton("choice1"),
new KeyboardButton("choice2")
},
}
};
replyKeyboardEN.OneTimeKeyboard = true;
await bot.SendTextMessageAsync(message.Chat.Id, "Enter choice", replyMarkup: replyKeyboardEN);
await AnonymEN();
break;
case "ger":
var replyKeyboardGR = new ReplyKeyboardMarkup
{
Keyboard = new[]
{
new[]
{
new KeyboardButton("choice1.1"),
new KeyboardButton("choice2.2")
},
}
};
replyKeyboardGR.OneTimeKeyboard = true;
await bot.SendTextMessageAsync(message.Chat.Id, "Enter choice", replyMarkup: replyKeyboardGR);
await AnonymGR();
break;
}
offset = update.Id + 1;
}
}
}
catch (Exception ex)
{
Console.WriteLine("Error" + ex);
}
}
and AnonymEN() method for eng case in switch. The problem appears here when I call this method from switch case in BotOnMessage(). Until switch (message.Text) multiple users can asynchronously send messages and get response. When first user enters AnonymEN() second user can't get response from this method until first user will finish it till the end. Also I call BotOnMessage() in the end of AnonymEN() to get back for initial point with possibility to start bot again. For the ordered structure of questions and answers I used ConcurrentDictionary way from here Save user messages sent to bot and send finished form to other user. Any suggestion and solution how to edit code to make this bot available for multiple users at one time?
async static Task AnonymEN()
{
int offset = 0;
int timeout = 0;
try
{
await bot.SetWebhookAsync("");
while (true)
{
var updates = await bot.GetUpdatesAsync(offset, timeout);
foreach (var update in updates)
{
var message = update.Message;
int userId = (int)message.From.Id;
if (message.Type == MessageType.Text)
{
if (Answers.TryGetValue(userId, out string[] answers))
{
var title = message.Text;
if (answers[0] == null)
{
answers[0] = message.Text;
await bot.SendTextMessageAsync(message.Chat, "Enter age");
}
else
{
SaveMessage(message.Chat.Id.ToString(), "anonym", "anonym", "anonym", answers[0].ToString(), title.ToString(), createdDateNoTime.ToString("yyyy-MM-dd HH:mm:ss.fff", CultureInfo.InvariantCulture));
Answers.TryRemove(userId, out string[] _);
await bot.SendTextMessageAsync(message.Chat.Id, "ty for request click /start");
await BotOnMessage();
}
}
else if (message.Text == "choice1")
{
Answers.TryAdd(userId, new string[1]);
await bot.SendTextMessageAsync(message.Chat.Id, "Enter name");
}
}
offset = update.Id + 1;
}
}
}
catch (Exception ex)
{
Console.WriteLine("Error" + ex);
}
}
I can see multiple issues with your code:
It is hard to read. While this is a personal preference I strongly advise to write short concise methods that have 1 responsibility. This will make it easier to understand and maintain your code. https://en.wikipedia.org/wiki/Single-responsibility_principle
Everything is static. This makes it very hard to keep track of any state such as language that should be tracked per user.
Using infinite loops and recursion with no escape. I highly doubt this was intended but you could get an infinite chain of calls like this BotOnMessage -> AnonymEN -> BotOnMessage -> AnonymEN. I think you want to exit the AnonymEN function using either a return, break or while(someVar) approach instead of calling the BotOnMessage function.
If two users are sending messages you get mixed responses. Example message flow user1: /start, user1: eng, user2: hello. The bot will now give an english response to user2. I'm sure this is not intended
The code below is a minimal example that addresses the issues I mentioned. It is not perfect code but should help you get started.
private Dictionaty<string, UserSession> userSessions = new ();
async Task BotOnMessage()
{
try
{
while(true)
{
var message = await GetMessage(timeout);
var userSession = GetUserSession(message.user);
userSession.ProcessMessage(message);
}
}
catch(){}
}
async void GetUserSession(string user)
{
if(!userSessions.HasKey(user))
{
userSessions[user](new Session());
}
return userSessions[user];
}
public class UserSession
{
public async Task ProcessMessage(message)
{
// Existing message processing code goes here.
// Do not use a loop or recursion.
// Instead track the state (e.g. languge) using fields.
}
}
Trying to make up a function that takes a list of redis keys, and outputs a list of timespans for the time to live (TTL) of the keys. I'm not familiar with Lua Scripting though. I've tried following other guides online for this, but am unable to get something to work for this situation. Any insight would be super helpful and very reusable.
public async Task<TimeSpan?[]> GetTTLManyAsync(string[] keySet)
{
try
{
var db = await this.GetDatabaseAsync();
string script = #"...";
...
var keysCasted = keySet.Select(k => new RedisKey(k)).ToArray();
var result = await db.ScriptEvaluateAsync(script, keysCasted);
...
return ...[array of timespans];
}
catch (Exception e)
{
WriteLogException(e);
return null;
}
}`
Here is a rough solution I worked out.
public async Task<List<TimeSpan?>> GetTTLManyAsync(IEnumerable<string> keys)
{
List<string> returnList = new List<string>();
try
{
var db = await this.GetDatabaseAsync();
string script = #"local result = {}
for i,key in ipairs(KEYS) do
local ttl = tonumber(redis.call('ttl', key))
result[i] = ttl
end
return result";
var result = await db.ScriptEvaluateAsync(script, keys.Select(k => new RedisKey(k)).ToArray());
var arrResult = ((double[])result).Select(ttl =>
{
if (ttl == -2) return default(TimeSpan?); // Key does not exist
else if (ttl == -1) return default(TimeSpan?); // No expiration set.
else return TimeSpan.FromSeconds(ttl);
}).ToList();
return arrResult;
}
catch (Exception e)
{
WriteLogException(e);
}
// Fill w/ empty values
return new List<TimeSpan?>(new TimeSpan?[keys.Count()]);
}
I would like to download multiple download files recursively from a FTP Directory, to do this I'm using FluentFTP library and my code is this one:
private async Task downloadRecursively(string src, string dest, FtpClient ftp)
{
foreach(var item in ftp.GetListing(src))
{
if (item.Type == FtpFileSystemObjectType.Directory)
{
if (item.Size != 0)
{
System.IO.Directory.CreateDirectory(Path.Combine(dest, item.Name));
downloadRecursively(Path.Combine(src, item.Name), Path.Combine(dest, item.Name), ftp);
}
}
else if (item.Type == FtpFileSystemObjectType.File)
{
await ftp.DownloadFileAsync(Path.Combine(dest, item.Name), Path.Combine(src, item.Name));
}
}
}
I know you need one FtpClient per download you want, but how can I make to use a certain number of connections as maximum, I guess that the idea is to create, connect, download and close per every file I find but just having a X number of downloading files at the same time. Also I'm not sure if I should create Task with async, Threads and my biggest problem, how to implement all of this.
Answer from #Bradley here seems pretty good, but the question does read every file thas has to download from an external file and it doesn't have a maximum concurrent download value so I'm not sure how to apply these both requirements.
Use:
ConcurrentBag class to implement a connection pool;
Parallel class to parallelize the operation;
ParallelOptions.MaxDegreeOfParallelism to limit number of the concurrent threads.
var clients = new ConcurrentBag<FtpClient>();
var opts = new ParallelOptions { MaxDegreeOfParallelism = maxConnections };
Parallel.ForEach(files, opts, file =>
{
file = Path.GetFileName(file);
string thread = $"Thread {Thread.CurrentThread.ManagedThreadId}";
if (!clients.TryTake(out var client))
{
Console.WriteLine($"{thread} Opening connection...");
client = new FtpClient(host, user, pass);
client.Connect();
Console.WriteLine($"{thread} Opened connection {client.GetHashCode()}.");
}
string remotePath = sourcePath + "/" + file;
string localPath = Path.Combine(destPath, file);
string desc =
$"{thread}, Connection {client.GetHashCode()}, " +
$"File {remotePath} => {localPath}";
Console.WriteLine($"{desc} - Starting...");
client.DownloadFile(localPath, remotePath);
Console.WriteLine($"{desc} - Done.");
clients.Add(client);
});
Console.WriteLine($"Closing {clients.Count} connections");
foreach (var client in clients)
{
Console.WriteLine($"Closing connection {client.GetHashCode()}");
client.Dispose();
}
Another approach is to start a fixed number of threads with one connection for each and have them pick files from a queue.
For an example of an implementation, see my article for WinSCP .NET assembly:
Automating transfers in parallel connections over SFTP/FTP protocol
A similar question about SFTP:
Processing SFTP files using C# Parallel.ForEach loop not processing downloads
Here is a TPL Dataflow approach. A BufferBlock<FtpClient> is used as a pool of FtpClient objects. The recursive enumeration takes a parameter of type IEnumerable<string> that holds the segments of one filepath. These segments are combined differently when constructing the local and the remote filepath. As a side effect of invoking the recursive enumeration, the paths of the remote files are sent to an ActionBlock<IEnumerable<string>>. This block handles the parallel downloading of the files. Its Completion property contains eventually all the exceptions that may have occurred during the whole operation.
public static Task FtpDownloadDeep(string ftpHost, string ftpRoot,
string targetDirectory, string username = null, string password = null,
int maximumConnections = 1)
{
// Arguments validation omitted
if (!Directory.Exists(targetDirectory))
throw new DirectoryNotFoundException(targetDirectory);
var fsLocker = new object();
var ftpClientPool = new BufferBlock<FtpClient>();
async Task<TResult> UsingFtpAsync<TResult>(Func<FtpClient, Task<TResult>> action)
{
var client = await ftpClientPool.ReceiveAsync();
try { return await action(client); }
finally { ftpClientPool.Post(client); } // Return to the pool
}
var downloader = new ActionBlock<IEnumerable<string>>(async path =>
{
var remotePath = String.Join("/", path);
var localPath = Path.Combine(path.Prepend(targetDirectory).ToArray());
var localDir = Path.GetDirectoryName(localPath);
lock (fsLocker) Directory.CreateDirectory(localDir);
var status = await UsingFtpAsync(client =>
client.DownloadFileAsync(localPath, remotePath));
if (status == FtpStatus.Failed) throw new InvalidOperationException(
$"Download of '{remotePath}' failed.");
}, new ExecutionDataflowBlockOptions()
{
MaxDegreeOfParallelism = maximumConnections,
BoundedCapacity = maximumConnections,
});
async Task Recurse(IEnumerable<string> path)
{
if (downloader.Completion.IsCompleted) return; // The downloader has failed
var listing = await UsingFtpAsync(client =>
client.GetListingAsync(String.Join("/", path)));
foreach (var item in listing)
{
if (item.Type == FtpFileSystemObjectType.Directory)
{
if (item.Size != 0) await Recurse(path.Append(item.Name));
}
else if (item.Type == FtpFileSystemObjectType.File)
{
var accepted = await downloader.SendAsync(path.Append(item.Name));
if (!accepted) break; // The downloader has failed
}
}
}
// Move on to the thread pool, to avoid ConfigureAwait(false) everywhere
return Task.Run(async () =>
{
// Fill the FtpClient pool
for (int i = 0; i < maximumConnections; i++)
{
var client = new FtpClient(ftpHost);
if (username != null && password != null)
client.Credentials = new NetworkCredential(username, password);
ftpClientPool.Post(client);
}
try
{
// Enumerate the files to download
await Recurse(new[] { ftpRoot });
downloader.Complete();
}
catch (Exception ex) { ((IDataflowBlock)downloader).Fault(ex); }
try
{
// Await the downloader to complete
await downloader.Completion;
}
catch (OperationCanceledException)
when (downloader.Completion.IsCanceled) { throw; }
catch { downloader.Completion.Wait(); } // Propagate AggregateException
finally
{
// Clean up
if (ftpClientPool.TryReceiveAll(out var clients))
foreach (var client in clients) client.Dispose();
}
});
}
Usage example:
await FtpDownloadDeep("ftp://ftp.test.com", "", #"C:\FtpTest",
"username", "password", maximumConnections: 10);
Note: The above implementation enumerates the remote directory lazily, following the tempo of the downloading process. If you prefer to enumerate it eagerly, gathering all info available about the remote listings ASAP, just remove the BoundedCapacity = maximumConnections configuration from the ActionBlock that downloads the files. Be aware that doing so could result in high memory consumption, in case the remote directory has a deep hierarchy of subfolders, containing cumulatively a huge number of small files.
I'd split this into three parts.
Recursively build a list of source and destination pairs.
Create the directories required.
Concurrently download the files.
It's the last part that is slow and should be done in parallel.
Here's the code:
private async Task DownloadRecursively(string src, string dest, FtpClient ftp)
{
/* 1 */
IEnumerable<(string source, string destination)> Recurse(string s, string d)
{
foreach (var item in ftp.GetListing(s))
{
if (item.Type == FtpFileSystemObjectType.Directory)
{
if (item.Size != 0)
{
foreach(var pair in Recurse(Path.Combine(s, item.Name), Path.Combine(d, item.Name)))
{
yield return pair;
}
}
}
else if (item.Type == FtpFileSystemObjectType.File)
{
yield return (Path.Combine(s, item.Name), Path.Combine(d, item.Name));
}
}
}
var pairs = Recurse(src, dest).ToArray();
/* 2 */
foreach (var d in pairs.Select(x => x.destination).Distinct())
{
System.IO.Directory.CreateDirectory(d);
}
/* 3 */
var downloads =
pairs
.AsParallel()
.Select(x => ftp.DownloadFileAsync(x.source, x.destination))
.ToArray();
await Task.WhenAll(downloads);
}
It should be clean, neat, and easy to reason about code.
I have the following code:
WebClient wc = new WebClient();
string result;
try
{
result = await wc.DownloadStringTaskAsync( new Uri( "http://badurl" ) );
}
catch
{
result = await wc.DownloadStringTaskAsync( new Uri( "http://fallbackurl" ) );
}
Basically I want to download from a URL and when it fails with an exception I want to download from another URL. Both time async of course. However the code does not compile, because of
error CS1985: Cannot await in the body of a catch clause
OK, it's forbidden for whatever reason but what's the correct code pattern here?
EDIT:
The good news is that C# 6.0 will likely allow await calls both in catch and finally blocks.
Update: C# 6.0 supports await in catch
Old Answer: You can rewrite that code to move the await from the catch block using a flag:
WebClient wc = new WebClient();
string result = null;
bool downloadSucceeded;
try
{
result = await wc.DownloadStringTaskAsync( new Uri( "http://badurl" ) );
downloadSucceeded = true;
}
catch
{
downloadSucceeded = false;
}
if (!downloadSucceeded)
result = await wc.DownloadStringTaskAsync( new Uri( "http://fallbackurl" ) );
Awaiting in a catch block is now possible as of the End User Preview of Roslyn as shown here (Listed under Await in catch/finally) and will be included in C# 6.
The example listed is
try … catch { await … } finally { await … }
Update: Added newer link, and that it will be in C# 6
This seems to work.
WebClient wc = new WebClient();
string result;
Task<string> downloadTask = wc.DownloadStringTaskAsync(new Uri("http://badurl"));
downloadTask = downloadTask.ContinueWith(
t => {
return wc.DownloadStringTaskAsync(new Uri("http://google.com/")).Result;
}, TaskContinuationOptions.OnlyOnFaulted);
result = await downloadTask;
Give this a try:
try
{
await AsyncFunction(...);
}
catch(Exception ex)
{
Utilities.LogExceptionToFile(ex).Wait();
//instead of "await Utilities.LogExceptionToFile(ex);"
}
(See the Wait() ending)
Use C# 6.0. see this Link
public async Task SubmitDataToServer()
{
try
{
// Submit Data
}
catch
{
await LogExceptionAsync();
}
finally
{
await CloseConnectionAsync();
}
}
You could put the await after the catch block followed by a label, and put a goto in the try block.
(No, really! Goto's aren't that bad!)
The pattern I use to rethrow the exception after await on a fallback task:
ExceptionDispatchInfo capturedException = null;
try
{
await SomeWork();
}
catch (Exception e)
{
capturedException = ExceptionDispatchInfo.Capture(e);
}
if (capturedException != null)
{
await FallbackWork();
capturedException.Throw();
}
You can use a lambda expression as follows:
try
{
//.....
}
catch (Exception ex)
{
Action<Exception> lambda;
lambda = async (x) =>
{
// await (...);
};
lambda(ex);
}
In a similar instance, I was unable to await in a catch block. However, I was able to set a flag, and use the flag in an if statement (Code below)
---------------------------------------...
boolean exceptionFlag = false;
try
{
do your thing
}
catch
{
exceptionFlag = true;
}
if(exceptionFlag == true){
do what you wanted to do in the catch block
}