I'm using Pythonnet to embed a Python script launcher into a C# WPF application. I can pass variable to python scripts using Scope and i get the result on a console using MVVM pattern.
Now I want to allow the user to stop a script execution at anytime. I couldn't find how to make that work in order to close the Thread properly.
class PythonRuntime
{
private static NLog.Logger logger = NLog.LogManager.GetCurrentClassLogger();
private MainViewModel viewModel;
private string pythonCode;
private bool runtimeThreadLock = false;
Thread thread;
private PyScope scope;
private dynamic pyThread;
private dynamic pyLock;
ConsoleWriter consoleWriter;
public PythonRuntime(MainViewModel viewModel, ConsoleWriter consoleWriter)
{
this.viewModel = viewModel;
this.consoleWriter = consoleWriter;
SetUpPython();
}
public string PythonCode { get => pythonCode; set => pythonCode = value; }
private void SetUpPython()
{
PythonEngine.Initialize(true);
scope = Py.CreateScope();
// consoleWriter to make python prints into C# UI
scope.Set("Console", consoleWriter);
}
public void LaunchScript()
{
if (!runtimeThreadLock)
{
thread = new Thread(PythonNetTest);
thread.Start();
}
}
public void StopScript()
{
// ???
}
[HandleProcessCorruptedStateExceptions]
private void PythonNetTest()
{
runtimeThreadLock = true;
pyThread = PythonEngine.BeginAllowThreads();
pyLock = PythonEngine.AcquireLock();
using (Py.GIL())
{
try
{
scope.Exec(pythonCode);
}
catch (PythonException exception)
{
consoleWriter.WriteError(exception.ToString());
}
}
PythonEngine.ReleaseLock(pyLock);
PythonEngine.EndAllowThreads(pyThread);
runtimeThreadLock = false;
}
}
Besides my question, I was wondering what is the purpose of wrapping code in using(Py.GIL()). Because with or whithout it my script runs the same way.
Pythonnet : 2.4.0
Python : 2.7.2 32bit
NetFramework : 4.7.1
OK, I'm just beginning work on embedding CPython and may know only a little more than you. What that caveat...
First, you need to get your script to terminate. When it does the call to .Exec() will return, and the thread will exit. If your script runs for a finite amount of time then you just wait for it. Otherwise, you must arrange some signal that it should exit.
Second, mainline will wait for thread to complete using one of several .NET patterns described at: How to wait for thread to finish with .NET?
using(Py.GIL()) is shorthand for PythonEngine.AcquireLock(); and PythonEngine.ReleaseLock(pyLock); It creates an IDisposable object that acquires the lock and then releases it on Dispose(). So, in your sample, it is redundant.
I'm unsure effects of your call to BeginAllowThreads(). Documentation says that it releases the lock to allow other threads. When you call it you don't have the GIL. Next line acquires the GIL. Therefore, it appears to have no function to me.
See https://docs.python.org/3/c-api/init.html for details on threading. This seems more related to python threading and saving thread state so that other non-python things can be done. This is python 3. Python 2 did not seem to support the equivalent.
I've got a problem with making calls to a third-party C++ dll which I've wrapped in a class using DllImport to access its functions.
The dll demands that before use a session is opened, which returns an integer handle used to refer to that session when performing operations. When finished, one must close the session using the same handle. So I did something like this:
public void DoWork(string input)
{
int apiHandle = DllWrapper.StartSession();
try
{
// do work using the apiHandle
}
catch(ApplicationException ex)
{
// log the error
}
finally
{
DllWrapper.CloseSession(apiHandle);
}
}
The problem I have is that CloseSession() sometimes causes the Dll in question to throw an error when running threaded:
System.AggregateException: One or more errors occurred. --->
System.AccessViolationException: Attempted to read or write protected
memory. This is often an indication that other memory is corrupt.
I'm not sure there's much I can do about stopping this error, since it seems to be arising from using the Dll in a threaded manner - it is supposed to be thread safe. But since my CloseSession() function does nothing except call that Dll's close function, there's not much wiggle room for me to "fix" anything.
The end result, however, is that the session doesn't close properly. So when the process tries again, which it's supposed to do, it encounters an open session and just keeps throwing new errors. That session absolutely has to be closed.
I'm at a loss as to how to design an error handling statement that's more robust any will ensure the session always closes?
I would change the wrapper to include disposal of the external resource and to also wrap the handle. I.e. instead of representing a session by a handle, you would represent it by a wrapper object.
Additionally, wrapping the calls to the DLL in lock-statements (as #Serge suggests), could prevent the multithreading issues completely. Note that the lock object is static, so that all DllWrappers are using the same lock object.
public class DllWrapper : IDisposable
{
private static object _lockObject = new object();
private int _apiHandle;
private bool _isOpen;
public void StartSession()
{
lock (_lockObject) {
_apiHandle = ...; // TODO: open the session
}
_isOpen = true;
}
public void CloseSession()
{
const int MaxTries = 10;
for (int i = 0; _isOpen && i < MaxTries; i++) {
try {
lock (_lockObject) {
// TODO: close the session
}
_isOpen = false;
} catch {
}
}
}
public void Dispose()
{
CloseSession();
}
}
Note that the methods are instance methods, now.
Now you can ensure the closing of the session with a using statement:
using (var session = new DllWrapper()) {
try {
session.StartSession();
// TODO: work with the session
} catch(ApplicationException ex) {
// TODO: log the error
// This is for exceptions not related to closing the session. If such exceptions
// cannot occur, you can drop the try-catch completely.
}
} // Closes the session automatically by calling `Dispose()`.
You can improve naming by calling this class Session and the methods Open and Close. The user of this class does not need to know that it is a wrapper. This is just an implementation detail. Also, the naming of the methods is now symmetrical and there is no need to repeat the name Session.
By encapsulating all the session related stuff, including error handling, recovery from error situations and disposal of resources, you can considerably diminish the mess in your code. The Session class is now a high-level abstraction. The old DllWrapper was somewhere at mid distance between low-level and high-level.
I have a problem I cannot seem to figure out, please help. I have created a class to handle an interface to some HW using TcpClient. I want this class to send one last command to the HW before it is destroyed.
To solve this I have implemented IDisposable.Dispose to take care of the sending of the last command and then close the connection. I have also in the destructor made a call to Dispose. This is the Microsoft recommendation as I read it in this article. My code is as follows:
class MyHWInterface : IDisposable
{
~MyHWInterface()
{
Dispose();
}
private bool disposed = false;
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool disposing)
{
if (!disposed)
{
CloseConnection();
disposed = true;
}
}
private System.Net.Sockets.TcpClient Port = new System.Net.Sockets.TcpClient();
public bool OpenConnection()
{
...
}
private bool SendCommand(string command)
{
var strm = Port.GetStream(); // This throws the exception Cannot access disposed object!
var w = new System.IO.StreamWriter(strm, System.Text.Encoding.ASCII);
var r = new System.IO.StreamReader(strm, System.Text.Encoding.ASCII);
w.WriteLine(command);
w.Flush();
string l = r.ReadLine();
return l == "OK";
}
internal void CloseConnection()
{
try
{
SendCommand("power down now");
}
catch
{
}
finally
{
Port.Close();
}
}
}
My problem is: When my program ends, and my object of MyHWInterface therefore falls out of scope and then gets garbage collected. The destructor is called which tries to send the last command, which fails because somehow my TcpClient is already disposed.
Please tell me why an object which is clearly not yet out of scope is being disposed. And please help with a method that makes sure my last command always will be send without an explicit call to Dispose.
Please tell me why an object which is clearly not yet out of scope is being disposed.
Objects don't have a concept of "scope" as such. At the end of your program, both the TcpClient and the instance of your class are eligible for finalization - and there's no guarantee which will be finalized first. It sounds like the TcpClient is being finalized (and the connection closed) first, hence the issue.
The best fix is not to rely on finalization for this in the first place - remove your own finalizer (realizing that the connection will just be closed at some point if the client doesn't call Dispose) and make sure that you do dispose of everything in an orderly fashion when your program terminates, assuming it's terminated cleanly (i.e. through some path you control).
So i am currently disposing many objects when i close my form. Even though it probably disposes it automatically. But still i prefer to follow the "rules" in disposing, hopefully it will stick and help prevent mistakes.
So here is how i currently dispose, which works.
if (connect == true)
{
Waloop.Dispose();
connect = false;
UninitializeCall();
DropCall();
}
if (KeySend.Checked || KeyReceive.Checked)
{
m_mouseListener.Dispose();
k_listener.Dispose();
}
if (NAudio.Wave.AsioOut.isSupported())
{
Aut.Dispose();
}
if (Wasout != null)
{
Wasout.Dispose();
}
if (SendStream != null)
{
SendStream.Dispose();
}
So basically, the first is if a bool is true, meaning if it isn´t those can be ignore, as they haven´t been made i think.
The others are just ways for me to dispose if it´s there. but it´s not a very good way, i would like to have it in 1 big function, meaning.
Dispose if it´s NOT disposed. or something.
I know that many of them has the "isdisposed" bool, so it should be possible if i can check every object, and dispose if it´s false.
How about a helper method which takes objects which implement IDisposable as params?
void DisposeAll(params IDisposable[] disposables)
{
foreach (IDisposable id in disposables)
{
if (id != null) id.Dispose();
}
}
When you want to dispose multiple objects, call the method with whatever objects you want to dispose.
this.DisposeAll(Wasout, SendStream, m_mouseListener, k_listener);
If you want to avoid calling them explicity, then store them all in a List<>:
private List<IDisposable> _disposables;
void DisposeAll() {
foreach(IDisposable id in _disposables) {
if(id != null) id.Dispose();
}
}
You can implement a Disposer class, that will do the work for you, along these lines:
public class Disposer
{
private List<IDisposable> disposables = new List<IDisposable>();
public void Register(IDisposable item)
{
disposables.Add(item);
}
public void Unregister(IDisposable item)
{
disposables.Remove(item);
}
public void DisposeAll()
{
foreach (IDisposable item in disposables)
{
item.Dispose();
}
disposables.Clear();
}
}
Then, instead of the ugly code in your main class, you can have something like:
public class Main
{
//member field
private Disposer m_disposer;
//constructor
public Main()
{
....
m_disposer = new Disposer();
//register any available disposables
disposer.Register(m_mouseListener);
disposer.Register(k_listener);
}
...
public bool Connect()
{
...
if (isConnected)
{
Waloop = ...
Wasout = ...
// register additional disposables as they are created
disposer.Register(Waloop);
disposer.Register(Wasout);
}
}
...
public void Close()
{
//disposal
disposer.DisposeAll();
}
}
I suggest you use the using statement. So with your code, it would look something like this:
using (WaloopClass Waloop = new WaloopClass())
{
// Some other code here I know nothing about.
connect = false; // Testing the current value of connect is redundant.
UninitializeCall();
DropCall();
}
Note there is now no need to explicitly Dispose Waloop, as it happens automatically at the end of the using statement.
This will help to structure your code, and makes the scope of the Waloop much clearer.
I am going to suppose that the only problem you’re trying to solve is how to write the following in a nicer way:
if (Wasout != null)
Wasout.Dispose();
if (SendStream != null)
SendStream.Dispose();
This is a lot of logic already implemented by the using keyword. using checks that the variable is not null before calling Dispose() for you. Also, using guarantees that thrown exceptions (perhap by Wasout.Dispose()) will not interrupt the attempts to call Dispose() on the other listed objects (such as SendStream). It seems that using was intended to allow management of resources based on scoping rules: using using as an alternative way to write o.Dispose() may be considered an abuse of the language. However, the benefits of using’s behavior and the concision it enables are quite valuable. Thus, I recommend to replace such mass statically-written batches of the “if (o != null) o.Dispose()” with an “empty” using:
using (
IDisposable _Wasout = Wasout,
_SendStream = SendStream)
{}
Note that the order that Dispose() is called in is in reverse of how objects are listed in the using block. This follows the pattern of cleaning up objects in reverse of their instantiation order. (The idea is that an object instantiated later may refer to an object instantiated earlier. E.g., if you are using a ladder to climb a house, you might want to keep the ladder around so that you can climb back down before putting it away—the ladder gets instantiated first and cleaned up last. Uhm, analogies… but, basically, the above is shorthand for nested using. And the unlike objects can be smashed into the same using block by writing the using in terms of IDisposable.)
dotnetfiddle of using managing exceptions.
Summary: C#/.NET is supposed to be garbage collected. C# has a destructor, used to clean resources. What happen when an object A is garbage collected the same line I try to clone one of its variable members? Apparently, on multiprocessors, sometimes, the garbage collector wins...
The problem
Today, on a training session on C#, the teacher showed us some code which contained a bug only when run on multiprocessors.
I'll summarize to say that sometimes, the compiler or the JIT screws up by calling the finalizer of a C# class object before returning from its called method.
The full code, given in Visual C++ 2005 documentation, will be posted as an "answer" to avoid making a very very large questions, but the essential are below:
The following class has a "Hash" property which will return a cloned copy of an internal array. At is construction, the first item of the array has a value of 2. In the destructor, its value is set to zero.
The point is: If you try to get the "Hash" property of "Example", you'll get a clean copy of the array, whose first item is still 2, as the object is being used (and as such, not being garbage collected/finalized):
public class Example
{
private int nValue;
public int N { get { return nValue; } }
// The Hash property is slower because it clones an array. When
// KeepAlive is not used, the finalizer sometimes runs before
// the Hash property value is read.
private byte[] hashValue;
public byte[] Hash { get { return (byte[])hashValue.Clone(); } }
public Example()
{
nValue = 2;
hashValue = new byte[20];
hashValue[0] = 2;
}
~Example()
{
nValue = 0;
if (hashValue != null)
{
Array.Clear(hashValue, 0, hashValue.Length);
}
}
}
But nothing is so simple...
The code using this class is wokring inside a thread, and of course, for the test, the app is heavily multithreaded:
public static void Main(string[] args)
{
Thread t = new Thread(new ThreadStart(ThreadProc));
t.Start();
t.Join();
}
private static void ThreadProc()
{
// running is a boolean which is always true until
// the user press ENTER
while (running) DoWork();
}
The DoWork static method is the code where the problem happens:
private static void DoWork()
{
Example ex = new Example();
byte[] res = ex.Hash; // [1]
// If the finalizer runs before the call to the Hash
// property completes, the hashValue array might be
// cleared before the property value is read. The
// following test detects that.
if (res[0] != 2)
{
// Oops... The finalizer of ex was launched before
// the Hash method/property completed
}
}
Once every 1,000,000 excutions of DoWork, apparently, the Garbage Collector does its magic, and tries to reclaim "ex", as it is not anymore referenced in the remaning code of the function, and this time, it is faster than the "Hash" get method. So what we have in the end is a clone of a zero-ed byte array, instead of having the right one (with the 1st item at 2).
My guess is that there is inlining of the code, which essentially replaces the line marked [1] in the DoWork function by something like:
// Supposed inlined processing
byte[] res2 = ex.Hash2;
// note that after this line, "ex" could be garbage collected,
// but not res2
byte[] res = (byte[])res2.Clone();
If we supposed Hash2 is a simple accessor coded like:
// Hash2 code:
public byte[] Hash2 { get { return (byte[])hashValue; } }
So, the question is: Is this supposed to work that way in C#/.NET, or could this be considered as a bug of either the compiler of the JIT?
edit
See Chris Brumme's and Chris Lyons' blogs for an explanation.
http://blogs.msdn.com/cbrumme/archive/2003/04/19/51365.aspx
http://blogs.msdn.com/clyon/archive/2004/09/21/232445.aspx
Everyone's answer was interesting, but I couldn't choose one better than the other. So I gave you all a +1...
Sorry
:-)
Edit 2
I was unable to reproduce the problem on Linux/Ubuntu/Mono, despite using the same code on the same conditions (multiple same executable running simultaneously, release mode, etc.)
It's simply a bug in your code: finalizers should not be accessing managed objects.
The only reason to implement a finalizer is to release unmanaged resources. And in this case, you should carefully implement the standard IDisposable pattern.
With this pattern, you implement a protected method "protected Dispose(bool disposing)". When this method is called from the finalizer, it cleans up unmanaged resources, but does not attempt to clean up managed resources.
In your example, you don't have any unmanaged resources, so should not be implementing a finalizer.
What you're seeing is perfectly natural.
You don't keep a reference to the object that owns the byte array, so that object (not the byte array) is actually free for the garbage collector to collect.
The garbage collector really can be that aggressive.
So if you call a method on your object, which returns a reference to an internal data structure, and the finalizer for your object mess up that data structure, you need to keep a live reference to the object as well.
The garbage collector sees that the ex variable isn't used in that method any more, so it can, and as you notice, will garbage collect it under the right circumstances (ie. timing and need).
The correct way to do this is to call GC.KeepAlive on ex, so add this line of code to the bottom of your method, and all should be well:
GC.KeepAlive(ex);
I learned about this aggressive behavior by reading the book Applied .NET Framework Programming by Jeffrey Richter.
this looks like a race condition between your work thread and the GC thread(s); to avoid it, i think there are two options:
(1) change your if statement to use ex.Hash[0] instead of res, so that ex cannot be GC'd prematurely, or
(2) lock ex for the duration of the call to Hash
that's a pretty spiffy example - was the teacher's point that there may be a bug in the JIT compiler that only manifests on multicore systems, or that this kind of coding can have subtle race conditions with garbage collection?
I think what you are seeing is reasonable behavior due to the fact that things are running on multiple threads. This is the reason for the GC.KeepAlive() method, which should be used in this case to tell the GC that the object is still being used and that it isn't a candidate for cleanup.
Looking at the DoWork function in your "full code" response, the problem is that immediately after this line of code:
byte[] res = ex.Hash;
the function no longer makes any references to the ex object, so it becomes eligible for garbage collection at that point. Adding the call to GC.KeepAlive would prevent this from happening.
Yes, this is an issue that has come up before.
Its even more fun in that you need to run release for this to happen and you end up stratching your head going 'huh, how can that be null?'.
Interesting comment from Chris Brumme's blog
http://blogs.msdn.com/cbrumme/archive/2003/04/19/51365.aspx
class C {<br>
IntPtr _handle;
Static void OperateOnHandle(IntPtr h) { ... }
void m() {
OperateOnHandle(_handle);
...
}
...
}
class Other {
void work() {
if (something) {
C aC = new C();
aC.m();
... // most guess here
} else {
...
}
}
}
So we can’t say how long ‘aC’ might live in the above code. The JIT might report the reference until Other.work() completes. It might inline Other.work() into some other method, and report aC even longer. Even if you add “aC = null;” after your usage of it, the JIT is free to consider this assignment to be dead code and eliminate it. Regardless of when the JIT stops reporting the reference, the GC might not get around to collecting it for some time.
It’s more interesting to worry about the earliest point that aC could be collected. If you are like most people, you’ll guess that the soonest aC becomes eligible for collection is at the closing brace of Other.work()’s “if” clause, where I’ve added the comment. In fact, braces don’t exist in the IL. They are a syntactic contract between you and your language compiler. Other.work() is free to stop reporting aC as soon as it has initiated the call to aC.m().
That's perfectly nornal for the finalizer to be called in your do work method as after the
ex.Hash call, the CLR knows that the ex instance won't be needed anymore...
Now, if you want to keep the instance alive do this:
private static void DoWork()
{
Example ex = new Example();
byte[] res = ex.Hash; // [1]
// If the finalizer runs before the call to the Hash
// property completes, the hashValue array might be
// cleared before the property value is read. The
// following test detects that.
if (res[0] != 2) // NOTE
{
// Oops... The finalizer of ex was launched before
// the Hash method/property completed
}
GC.KeepAlive(ex); // keep our instance alive in case we need it.. uh.. we don't
}
GC.KeepAlive does... nothing :) it's an empty not inlinable /jittable method whose only purpose is to trick the GC into thinking the object will be used after this.
WARNING: Your example is perfectly valid if the DoWork method were a managed C++ method... You DO have to manually keep the managed instances alive manually if you don't want the destructor to be called from within another thread. IE. you pass a reference to a managed object who is going to delete a blob of unmanaged memory when finalized, and the method is using this same blob. If you don't hold the instance alive, you're going to have a race condition between the GC and your method's thread.
And this will end up in tears. And managed heap corruption...
The Full Code
You'll find below the full code, copy/pasted from a Visual C++ 2008 .cs file. As I'm now on Linux, and without any Mono compiler or knowledge about its use, there's no way I can do tests now. Still, a couple of hours ago, I saw this code work and its bug:
using System;
using System.Threading;
public class Example
{
private int nValue;
public int N { get { return nValue; } }
// The Hash property is slower because it clones an array. When
// KeepAlive is not used, the finalizer sometimes runs before
// the Hash property value is read.
private byte[] hashValue;
public byte[] Hash { get { return (byte[])hashValue.Clone(); } }
public byte[] Hash2 { get { return (byte[])hashValue; } }
public int returnNothing() { return 25; }
public Example()
{
nValue = 2;
hashValue = new byte[20];
hashValue[0] = 2;
}
~Example()
{
nValue = 0;
if (hashValue != null)
{
Array.Clear(hashValue, 0, hashValue.Length);
}
}
}
public class Test
{
private static int totalCount = 0;
private static int finalizerFirstCount = 0;
// This variable controls the thread that runs the demo.
private static bool running = true;
// In order to demonstrate the finalizer running first, the
// DoWork method must create an Example object and invoke its
// Hash property. If there are no other calls to members of
// the Example object in DoWork, garbage collection reclaims
// the Example object aggressively. Sometimes this means that
// the finalizer runs before the call to the Hash property
// completes.
private static void DoWork()
{
totalCount++;
// Create an Example object and save the value of the
// Hash property. There are no more calls to members of
// the object in the DoWork method, so it is available
// for aggressive garbage collection.
Example ex = new Example();
// Normal processing
byte[] res = ex.Hash;
// Supposed inlined processing
//byte[] res2 = ex.Hash2;
//byte[] res = (byte[])res2.Clone();
// successful try to keep reference alive
//ex.returnNothing();
// Failed try to keep reference alive
//ex = null;
// If the finalizer runs before the call to the Hash
// property completes, the hashValue array might be
// cleared before the property value is read. The
// following test detects that.
if (res[0] != 2)
{
finalizerFirstCount++;
Console.WriteLine("The finalizer ran first at {0} iterations.", totalCount);
}
//GC.KeepAlive(ex);
}
public static void Main(string[] args)
{
Console.WriteLine("Test:");
// Create a thread to run the test.
Thread t = new Thread(new ThreadStart(ThreadProc));
t.Start();
// The thread runs until Enter is pressed.
Console.WriteLine("Press Enter to stop the program.");
Console.ReadLine();
running = false;
// Wait for the thread to end.
t.Join();
Console.WriteLine("{0} iterations total; the finalizer ran first {1} times.", totalCount, finalizerFirstCount);
}
private static void ThreadProc()
{
while (running) DoWork();
}
}
For those interested, I can send the zipped project through email.