Develop Reference

How can I return error from a task in azure job batch so task may be reactivated?

I want that when a job executes, in azure batch service, in case of an error to be able to correct data and then from portal just reactivate the task. The tasks are scheduled.
Currently I am just throwing Exceptions in case of failure. Those tasks cannot be reactivated.
But looking at other tasks I see this
"Task failed "The task exited with an exit code representing a failure""
For that task I can click on 'Reactivate' button. How can I do the same?
This is my current code:
public class Program
{
private static void Main(string[] args)
{
try
{
ConsoleLogger.Info($"Job.DataTransfer process started! ");
DataTransferSettings dataTransferSettings = DataTransferSettingsReader.GetDataTransferSettings();
if (dataTransferSettings != null)
{
ServicePointManager.DefaultConnectionLimit = int.MaxValue;
CopyData(dataTransferSettings);
}
else
{
throw new Exception($"Process stopped, check data transfer settings.");
}
ConsoleLogger.Info($"Job.DataTransfer process completed.");
}
catch (Exception ex)
{
ConsoleLogger.Error(GetExceptionMessage(ex), ex);
ExceptionDispatchInfo.Capture(ex).Throw();
}
}
}
I just didn't found yet any solution on how to do this.

So here is how you can handle it or the links below will give you good ideas how you can choose to handle.
Have a play around with the sample code here: c# samples reside here: https://github.com/Azure-Samples/azure-batch-samples/tree/master/CSharp/GettingStarted You will see some stuff here as to how you can handle the failing task: https://github.com/Azure-Samples/azure-batch-samples/blob/master/CSharp/GettingStarted/02_PoolsAndResourceFiles/JobSubmitter/JobSubmitter.cs
Pseudo code steps:
Capture failed task.
Get the non-zero exit code and if you think you can do something about it programmatically call foo_funct_correct and then try again.
Task Failed events mentioned here: https://learn.microsoft.com/en-us/azure/batch/batch-task-fail-event
Azure Batch Service Task Exit Code error
Task exit code: https://learn.microsoft.com/en-us/azure/batch/error-handling#task-exit-codes
Extra stuff
How to re-queue failed task: Azure Batch re-queue failed tasks
Hope this helps! Thanks =)

prevent application crashes when sending data over a closed websocket connection

The ASP.NET Core application uses websocket connection on the client side and Microsoft.AspNetCore.WebSockets.Server 0.1.0 (latest stable version on nuget as I know) on the server side. The simple sending code is
await _ws.SendAsync(new ArraySegment<byte>(arrbr), WebSocketMessageType.Text, true, ctk);
the problem is this line throws error when it is a closed connection. I would like that method to return a Boolean if process was successful. I already check if the connection is open like this:
_ws.State == WebSocketState.Open
But this does not work if user has
unplugged the network cable or disconnected his device(almost all situations except closing the browsers).
As an extra, I do not know how to simulate network connection loss for one of two clients and I suppose WebSocketState is readonly, please warn me if I am wrong and I do not think shorter pings will solve this problem.
I have two ideas:
I may use the sender code in a try catch block. (I am not comfortable with using try catch in production code)
I may set interval on the client side, and ask the server like "what is new for me". I feel bad with this because it is away from being a websocket(closer to http).
Is there any way to fix this without using try catch? I know that this is not a qualified question but a qualified problem for me. I can share full code if needed.
Update 1
after using server-side logging:
While messaging is working well in production environment, I disconnect the client by unplugging the network cable and send data to the client. I use try catch and no catch. then i get this error.
This means I cant detect lost connection by using try catch. and i think i can solve this by handling this throw.
How can I handle this error?
update2
I have noticed that "Exceptions from an Async Void Method Can’t Be Caught with Catch" and "it's possible to use await in catch" since c# 6 link however I can not catch the throw. I may try running synchronously await _ws.SendAsync(new ArraySegment<byte>(arrbr), WebSocketMessageType.Text, true, ctk).runsynchronously(); if I cant fix in this way
update3
running synchronously does not help. using try catch does not differ. as a result question, asp.net core websocket how to ping
update4
my log file when using Microsoft.aspnetcore.websockets.server 0.1.0
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HL1940BSEV8O": An unhandled exception was thrown by the application.
System.IO.IOException: Unexpected end of stream
at Microsoft.AspNetCore.WebSockets.Protocol.CommonWebSocket.<EnsureDataAvailableOrReadAsync>d__38.MoveNext()
my log file when using Microsoft.aspnetcore.websockets 1.0.0
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HL19H5R4CS21": An unhandled exception was thrown by the application.
System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake. ---> System.IO.IOException: Error -4077 ECONNRESET connection reset by peer ---> Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -4077 ECONNRESET connection reset by peer

I might be missing something, but why can't wrap the sending operation in a method that returns bool in the following manner:
private async Task<bool> DoSend()
{
bool success = true;
try
{
await _ws.SendAsync(new ArraySegment<byte>(arrbr), WebSocketMessageType.Text, true, ctk);
}
catch (Exception ex)
{
// Do some logging with ex
success = false;
}
return success;
}
I also suggest reading about Async All The Way, it should clear some of the confusion with async void, async Task and async Task<T>

Until C# 6.0 to capture an exceptions from async methods you should use the ExceptionDispatchInfo type. Then the code will look like this:
private async Task<bool> DoSend()
{
bool success = true;
ExceptionDispatchInfo capturedException = null;
try
{
await _ws.SendAsync(new ArraySegment<byte>(arrbr), WebSocketMessageType.Text, true, ctk);
}
catch (Exception ex)
{
capturedException = ExceptionDispatchInfo.Capture(ex);
}
if (capturedException != null)
{
await ExceptionHandler();
if (needsThrow)
{
capturedException.Throw();
}
}
success = capturedException == null;
return success;
}

Task stops working on different environment

I'm having hard time with this one.
So in my asp.net application there is such a method:
public CopyResponse thirdStage(CopyRequest request)
{
CopyCCResponse response = new CopyCCResponse();
Task.Run(() =>
{
performCopying(request);
});
return response;
}
private void performCopying(CopyCCRequest request)
{
using (Repository = new myDbContext())
{
// do some initial action
try
{
// in general it looks like below
foreach(var children in father)
{
var newChildren = chldren.Copy();
Repository.Childrens.Add(newChildren);
foreach (var grandchldren in children.grandchildrens)
{
var newGrandchildren = grandchldren.Copy();
newGrandchildren.Parent = newChildren;
Repository.Grandchildrens.Add(newGrandchildren);
}
Repository.SaveChanges();
}
}
catch (Exception ex)
{
// log that action failed
throw ex;
}
}
}
This method and all other (there are some similar) works as designed on my local computer without any problems.
Unfortunately, on another environment those methods fail:
Copying smaller parts of data works fine. But when there is over 3000 objects to operate on, method fails.
Main application is responding correctly nevertheless.
Most of the operation is done well (most data is copied and saved in database)
Application doesn't enter catch block. Instructions for failed copying are not executed. Exception isn't caught by the error handler (BTW, I know by default the app can't catch exceptions from independent task, I wrote my handler so it will manage to do so).
IIS worker process seems to take over 300MB and 0% of processor power after copying stopped. More than half of RAM on server is still free.
I looked into windows event log, but haven't found anything.
Do you have any suggestions how I can handle this issue?

You can't do reliable "Fire and forget" tasks from inside IIS, if the site is not being served the application pool will get its AppDomain shut down after a while.
Two options to use are:
HostingEnvironment.QueueBackgroundWorkItem to tell IIS you are doing background work. This will let the server know of the work and it will delay the shutdown as long as it can (default up to 90 seconds max) before it kills your process.
public CopyResponse thirdStage(CopyRequest request)
{
CopyCCResponse response = new CopyCCResponse();
HostingEnvironment.QueueBackgroundWorkItem(() =>
{
performCopying(request);
});
return response;
}
Another option is to use a 3rd party library that is designed for doing background work in IIS like Hangfire.io, this will run a service inside of IIS that does the work and attempts to keep the instance alive till the work is done. You can also configure Hangfire to run as a separate process so you don't need to rely on the lifetime of the IIS instance.
public CopyResponse thirdStage(CopyRequest request)
{
CopyCCResponse response = new CopyCCResponse();
BackgroundJob.Enqueue(() =>
{
performCopying(request);
});
return response;
}
Note, using hangfire with a seperate process may require you to do a little redesign of performCopying(CopyCCRequest request) to support being run from a separate process, using it from inside the IIS instance should not require any changes.

HttpWebRequest timing out, but no packets leaving client

I have a c# application with the following requirements.
Periodically(700ms) send data to a cgi using a GET.
The server will respond with only the GET string sent as confirmation.
Values sent are constantly changing and are not stored, ordered, or retried.
Only one connection made to the server at a time.
Timer events that find a previous connection still in progress should simply exit quietly and let the next event carry on.
The client machine is Windows Server 2012. The HTTP server is a black box appliance(believed to be running linux). Environment is low-latency production with extremely tight security(network/policies/etc).
The problem:
Periodically, the client will begin throwing exceptions
"The operation timed out"
and the drip feed stops. Simultaneously, Wireshark shows
no connections being made from the client to the server.
However, VisualStudio shows that
every 700ms the client is attempting to connect and is timing out.
When it's not throwing the above exception, it seems to work fine. At no time does the HTTP server appear to be rejecting connections. Sometimes the client will recover, most of the time it will continue to throw exceptions until restarted. AppDomain thread pool is stable. Memory consumption is fine.
To confuse matters, I have identical hardware/software/network without the security in a lab where everything runs flawlessly for days at a time. My code is being fingered as "malfunctory".
I have recently discovered that the HttpWebRequest.Timeout property bounds the entire life of the transaction, not just the response wait. I am increasing this and it's currently under test.
Anyone see anything glaring(or otherwise) that would cause this problem in the below code?
// timer call back at 700ms interval
void m_postTimer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
if (Interlocked.Read(ref this.m_isLocked) == 0)
{
if (Monitor.TryEnter(this.m_lock, 10))
{
Interlocked.Exchange(ref this.m_isLocked, 1);
try
{
// value1 & value2 set elsewhere
string url = String.Format("http://1.1.1.1/set.cgi?value1={0}&value2={1}", value1, value2);
this.m_post = (HttpWebRequest)WebRequest.Create(url);
this.m_post.Timeout = 500;
this.m_post.Credentials = new NetworkCredential(this.user, this.pass);
WebResponse response = this.m_post.GetResponse();
response.Close();
Monitor.Exit(this.m_lock);
Interlocked.Exchange(ref this.m_isLocked, 0);
}
catch (WebException ex)
{
// handle web exceptions
// if we've locked earlier and hit an exception,
// the unlock has been skipped. unlock
if (Interlocked.Read(ref this.m_isLocked) > 0)
{
Monitor.Exit(this.m_lock);
Interlocked.Exchange(ref this.m_isLocked, 0);
}
}
}
}
else
{
// indicate that a pre-existing connection is still in progress
}
}
catch (Exception ex)
{
// handle generic exception
// if we've locked earlier and hit an exception,
// the unlock has been skipped. unlock
if (Interlocked.Read(ref this.m_isLocked) > 0)
{
Monitor.Exit(this.m_lock);
Interlocked.Exchange(ref this.m_isLocked, 0);
}
}
}

Process.Kill() related crashes

Or not!
I have a fairly simple application timer program. The program will launch a user selected (from file dialog) executable and then terminate the process after the user specified number of minutes. During testing I found that a crash occurs when I call the Process.Kill() method and the application is minimized to the system tray.
The executable in question is Frap.exe which I use frequently and is the reason I wrote the app timer in the first place. I always minimize fraps to the tray, and this is when the crash occurs.
My use of Kill() is straight forward enough...
while (true)
{
//keep checking if timer expired or app closed externally (ie. by user)
if (dtEndTime <= DateTime.Now || p.HasExited)
{
if (!p.HasExited)
p.Kill();
break;
}
System.Threading.Thread.Sleep(500);
}
In searching for alternatives methods to close an external application programmatically, I found only Close() and Kill() (CloseMainWindow is not helpful to me at all). I tried using Close(), which works providing the application is minimized the tray. If the app is minimized, Close() doesn't cause a crash but the app remains open and active.
One thing I noticed in a few posts posts regarding closing external applications was the comment: "Personally I'd try to find a more graceful way of shutting it down though." made in THIS thread found here at stack flow (no offense to John). Thing is, I ran across comments like that on a few sites, with no attempt at what a graceful or elegant (or crash-free!!) method might be.
Any suggestions?
The crash experienced is not consistant and I've little to offer as to details. I am unable to debug using VS2008 as I get message - cant debug crashing application (or something similar), and depending on what other programs I have running at the time, when the Kill() is called some of them also crash (also programs only running in the tray) so I'm thinking this is some sort of problem specifically related to the system tray.

Is it possible that your code is being executed in a way such that the Kill() statement could sometimes be called twice? In the docs for Process.Kill(), it says that the Kill executes asynchronously. So, when you call Kill(), execution continues on your main thread. Further, the docs state that Kill will throw a Win32Exception if you call it on an app that is already in the process of closing. The docs state that you can use WaitForExit() to wait for the process to exit. What happens if you put a call to WaitForExit() immediately following the call to Kill(). The loop looks ok (with the break statement). Is it possible that you have code entering that loop twice?
If that's not the problem, maybe there is another way to catch that exception:
Try hooking the AppDomain.CurrentDomain.UnhandledException event
(currentDomain is a static member)
The problem is that Kill runs asynchronously, so if it's throwing an exception, it's occurring on a different thread. That's why your exception handler doesn't catch it. Further (I think) that an unhandled async exception (which is what I believe you have) will cause an immediate unload of your application (which is what is happening).
Edit: Example code for hooking the UnhandledExceptionEvent
Here is a simple console application that demonstrates the use of AppDomain.UnhandledException:
using System;
public class MyClass
{
public static void Main()
{
System.AppDomain.CurrentDomain.UnhandledException += MyExceptionHandler;
System.Threading.ThreadPool.QueueUserWorkItem(DoWork);
Console.ReadLine();
}
private static void DoWork(object state)
{
throw new ApplicationException("Test");
}
private static void MyExceptionHandler(object sender, System.UnhandledExceptionEventArgs e)
{
// get the message
System.Exception exception = e.ExceptionObject as System.Exception;
Console.WriteLine("Unhandled Exception Detected");
if(exception != null)
Console.WriteLine("Message: {0}", exception.Message);
// for this console app, hold the window open until I press enter
Console.ReadLine();
}
}

My first thought is to put a try/catch block around the Kill() call and log the exception you get, if there is one. It might give you a clue what's wrong. Something like:
try
{
if(!p.HasExited)
{
p.Kill();
}
break;
}
catch(Exception ex)
{
System.Diagnostics.Trace.WriteLine(String.Format("Could not kill process {0}, exception {1}", p.ToString(), ex.ToString()));
}

I dont think I should claim this to be "THE ANSWER" but its a decent 'work around'. Adding the following to lines of code...
p.WaitForInputIdle(10000);
am.hWnd = p.MainWindowHandle;
...stopped the crashing issue. These lines were placed immediately after the Process.Start() statement. Both lines are required and in using them I opened the door to a few other questions that I will be investigating over the next few days. The first line is just an up-to 10 second wait for the started process to go 'idle' (ie. finish starting). am.hWnd is a property in my AppManagement class of type IntPtr and this is the only usage of both sides of the assignment. For lack of better explaination, these two lines are analguous to a debouncing method.
I modified the while loop only slightly to allow for a call to CloseMainWindow() which seems to be the better route to take - though if it fails I then Kill() the app:
while (true)
{
//keep checking if timer expired or app closed externally (ie. by user)
if (dtEndTime <= DateTime.Now || p.HasExited) {
try {
if (!p.HasExited) // if the app hasn't already exitted...
{
if (!p.CloseMainWindow()) // did message get sent?
{
if (!p.HasExited) //has app closed yet?
{
p.Kill(); // force app to exit
p.WaitForExit(2000); // a few moments for app to shut down
}
}
p.Close(); // free resources
}
}
catch { // blah blah }
break;
}
System.Threading.Thread.Sleep(500);
}
My initial intention for getting the MainWindowHandle was to maximize/restore an app if minimized and I might implement that in the near future. I decided to see if other programs that run like Fraps (ie, a UI but mostly run in the system tray (like messanger services such as Yahoo et al.)). I tested with XFire and nothing I could do would return a value for the MainWindowHandle. Anyways, this is a serperate issue but one I found interesting.
PS. A bit of credit to JMarsch as it was his suggestion RE: Win32Exception that actually lead me to finding this work around - as unlikely as it seems it true.