Develop Reference

How To DEALLOCATE PREPARE Statement In C# Using Mysql.Data Client?

hope you guys are fine?
OK.. i am using MySQL.Data client/library to access and use MySQL database. I was using happily it for sometimes on quite a few project. But suddenly facing a new issue that causing me hold on my current project. :(
Because current project makes some (looks like it's a lot) db queries. and i am facing following exception :
Can't create more than max_prepared_stmt_count statements (current value: 16382)
i am closing and disposing the db engine/connection every time i am done with it. But getting damn confused why i am still getting this error.
​here is the sample code just to give you idea.. (trimmed out unnecessary parts)
//this loop call an API with pagination and get API response
while(ContinueSalesOrderPage(apiClient, ref pageNum, days, out string response, window) == true)
{
//this handle the API date for the current page, it's normally 500 entry per page, and it throws the error on 4th page
KeyValueTag error = HandleSalesOrderPageData(response, pageNum, out int numOrders, window);
}
private KeyValueTag HandleSalesOrderPageData(string response, int pageNum, out int numOrders, WaitWindow window)
{
numOrders = json.ArrayOf("List").Size;
//init db
DatabaseWriter dbEngine = new DatabaseWriter()
{
Host = dbHost,
Name = dbName,
User = dbUser,
Password = dbPass,
};
//connecting to database
bool pass = dbEngine.Connect();
//loop through all the entry for the page, generally it's 500 entries
for(int orderLoop = 0; orderLoop < numOrders; orderLoop++)
{
//this actually handle the queries, and per loop there could be 3 to 10+ insert/update query using prepared statements
KeyValueTag error = InsertOrUpdateSalesOrder(dbEngine, item, config, pageNum, orderLoop, numOrders, window);
}
//here as you can see, i disconnect from db engine, and following method also close the db connection before hand
dbEngine.Disconnect();
}
//code from DatabaseWriter class, as you see this method close and dispose the database properly
public void Disconnect()
{
_CMD.Dispose();
_engine.Close();
_engine.Dispose();
}
so, as you can see i close/dispose the database connection on each page processing, but still it shows me that error on 4th page. FYI, 4th page data is not the matter i checked that. If i skip the page and only process the 4th page, it process successfully.
and after some digging more in google, i found prepare statement is saved in database server and that needs to be close/deallocate. But i can't find any way to do that using MySQL.Data Client :(
following page says:
https://dev.mysql.com/doc/refman/8.0/en/sql-prepared-statements.html
A prepared statement is specific to the session in which it was created. If you terminate a session without deallocating a previously prepared statement, the server deallocates it automatically. 
but that seems incorrect, as i facing the error even after closing connection on each loop
so, i am at dead end and looking for some help here? 
thanks in advance
best regards

From the official docs, the role of max_prepared_stmt_count is
This variable limits the total number of prepared statements in the server.
Therefore, you need to increase the value of the above variable, so as to increase the maximum number of allowed prepared statements in your MySQL server's configuration
Open the my.cnf file
Under the mysqld section, there is a variable max_prepared_stmt_count. Edit the value accordingly(remember the upper end of this value is 1048576)
Save and close the file. Restart MySQL service for changes to take place.

You're probably running into bug 77421 in MySql.Data: by default, it doesn't reset connections.
This means that temporary tables, user-declared variables, and prepared statements are never cleared on the server.
You can fix this by adding Connection Reset = True; to your connection string.
Another fix would be to switch to MySqlConnector, an alternative ADO.NET provider for MySQL that fixes this and other bugs. (Disclaimer: I'm the lead author.)

Gaining access to a MemoryMappedFile from low-integrity process

I'm trying to create a MemoryMappedFile on a medium-integrity process, then open the same file on a low-integrity child process and use this shared memory for IPC. There's no real disk file (using MemoryMappedFile.CreateNew).
My problem is that the low-integrity process cannot open the shared memory, throwing this: "System.UnauthorizedAccessException: Access to the path is denied.". I'm not surprised that this is the case, given that I want write access from the low-integrity process, but how do you grant it access?
Here's my code:
Medium integrity process:
MemoryMappedFileSecurity security = new MemoryMappedFileSecurity();
var file = MemoryMappedFile.CreateNew("test", 4096, MemoryMappedFileAccess.ReadWrite, MemoryMappedFileOptions.None, security, HandleInheritability.Inheritable);
var view = file.CreateViewAccessor();
view.Write(0, true);
Low integrity process:
try
{
MemoryMappedFile file = MemoryMappedFile.OpenExisting("test", MemoryMappedFileRights.ReadWrite);
var view = file.CreateViewAccessor();
var v = view.ReadBoolean(0);
Log.Info("MAPPED: " + v);
}
catch (Exception e)
{
Log.Info("Error: " + e);
}
Works fine if both processes work in medium integrity. After reading this, I tried setting the SDDL string on the medium integrity process like this:
security.SetSecurityDescriptorSddlForm("S:(ML;;NW;;;LW)");
But that gives me another exception, this time when the memory mapped file is created: "System.IO.IOException: A required privilege is not held by the client.". Not really sure this is the right way to do it anyway, I'm not really clear on how the Win32/C++ examples translates to C#...
Anyone know anything more about this?

Okay, got a working solution. There were two problems:
Passing an empty MemoryMappedFileSecurity object to MemoryMappedFile.CreateNew() made the mapped memory inaccessible even to the same process. That explained my error in my comment ("System.UnauthorizedAccessException: Access to the path is denied").
I couldn't actually get security.SetSecurityDescriptorSddlForm to work (and even though google reveals several other attempts at this, none of them worked for me). Instead, I used this solution: https://stackoverflow.com/a/14424623/5105846. As far as I can tell, it does the same thing, but using PInvoke instead. So I just called InterProcessSecurity.SetLowIntegrityLevel(file.SafeMemoryMappedFileHandle), and it made it accessible from the low-integrity child process. Success!
Not the perfect solution, but a working one is all I need for now. Thanks Harry for your help!

Adding cabinet file to msi programatically with DTF (wix)

Introduction to the Task at hand: can be skipped if impatient
The company I work for is not a software company, but focus on mechanical and thermodynamic engineering problems.
To help solve their system design challenges, they have developed a software for calculating the system impact of replacing individual components.
The software is quite old, written in FORTRAN and has evolved over a period of 30 years, which means that we cannot quickly re-write it or update it.
As you may imagine the way this software is installed has also evolved, but significantly slower than the rest of the system, meaning that packaging is done by a batch script that gathers files from different places, and puts them in a folder, which is then compiled into an iso, burned to a cd, and shipped with mail.
You young programmers (I am 30), may expect a program to load dll's, but otherwise be fairly self-contained after linking. Even if the code is made up of several classes, from different namespaces etc..
In FORTRAN 70 however.. Not so much. Which means that the software it self consists of an alarming number of calls to prebuilt modules (read: seperate programs)..
We need to be able to distribute via the internet, as any other modern company have been able to for a while. To do this we could just make the *.iso downloadable right?
Well, unfortunately no, the iso contains several files which are user specific.
As you may imagine with thousands of users, that would be thousands of isos, that are nearly identical.
Also we wan't to convert the old FORTRAN based installation software, into a real installation package, and all our other (and more modern) programs are C# programs packaged as MSI's..
But the compile time for a single msi with this old software on our server, is close to 10 seconds, so it is simply not an option for us to build the msi, when requested by the user. (if multiple users requests at the same time, the server won't be able to complete before requests timeout..)
Nor can we prebuild the user specific msi's and cache them, as we would run out of memory on the server.. (total at ~15 giga Byte per released version)
Task Description tl:dr;
Here is what I though I would do: (inspired by comments from Christopher Painter)
Create a base MSI, with dummy files instead of the the user specific files
Create cab file for each user, with the user specific files
At request time inject the userspecific cab file into a temporary copy of the base msi using the "_Stream" table.
Insert a reference into the Media table with a new 'DiskID' and a 'LastSequence' corresponding to the extra files, and the name of the injected cabfile.
Update the Filetable with the name of the user specific file in the new cab file, a new Sequence number (in the range of the new cab files sequence range), and the file size.
Question
My code fails to do the task just described. I can read from the msi just fine, but the cabinet file is never inserted.
Also:
If I open the msi with DIRECT mode, it corrupts the media table, and if I open it in TRANSACTION mode, it fails to change anything at all..
In direct mode the existing line in the Media table is replaced with:
DiskId: 1
LastSequence: -2145157118
Cabinet: "Name of action to invoke, either in the engine or the handler DLL."
What Am I doing wrong ?
Below I have provided the snippets involved with injecting the new cab file.
snippet 1
public string createCabinetFileForMSI(string workdir, List<string> filesToArchive)
{
//create temporary cabinet file at this path:
string GUID = Guid.NewGuid().ToString();
string cabFile = GUID + ".cab";
string cabFilePath = Path.Combine(workdir, cabFile);
//create a instance of Microsoft.Deployment.Compression.Cab.CabInfo
//which provides file-based operations on the cabinet file
CabInfo cab = new CabInfo(cabFilePath);
//create a list with files and add them to a cab file
//now an argument, but previously this was used as test:
//List<string> filesToArchive = new List<string>() { #"C:\file1", #"C:\file2" };
cab.PackFiles(workdir, filesToArchive, filesToArchive);
//we will ned the path for this file, when adding it to an msi..
return cabFile;
}
snippet 2
public int insertCabFileAsNewMediaInMSI(string cabFilePath, string pathToMSIFile, int numberOfFilesInCabinet = -1)
{
//open the MSI package for editing
pkg = new InstallPackage(pathToMSIFile, DatabaseOpenMode.Direct); //have also tried direct, while database was corrupted when writing.
return insertCabFileAsNewMediaInMSI(cabFilePath, numberOfFilesInCabinet);
}
snippet 3
public int insertCabFileAsNewMediaInMSI(string cabFilePath, int numberOfFilesInCabinet = -1)
{
if (pkg == null)
{
throw new Exception("Cannot insert cabinet file into non-existing MSI package. Please Supply a path to the MSI package");
}
int numberOfFilesToAdd = numberOfFilesInCabinet;
if (numberOfFilesInCabinet < 0)
{
CabInfo cab = new CabInfo(cabFilePath);
numberOfFilesToAdd = cab.GetFiles().Count;
}
//create a cab file record as a stream (embeddable into an MSI)
Record cabRec = new Record(1);
cabRec.SetStream(1, cabFilePath);
/*The Media table describes the set of disks that make up the source media for the installation.
we want to add one, after all the others
DiskId - Determines the sort order for the table. This number must be equal to or greater than 1,
for out new cab file, it must be > than the existing ones...
*/
//the baby SQL service in the MSI does not support "ORDER BY `` DESC" but does support order by..
IList<int> mediaIDs = pkg.ExecuteIntegerQuery("SELECT `DiskId` FROM `Media` ORDER BY `DiskId`");
int lastIndex = mediaIDs.Count - 1;
int DiskId = mediaIDs.ElementAt(lastIndex) + 1;
//wix name conventions of embedded cab files is "#cab" + DiskId + ".cab"
string mediaCabinet = "cab" + DiskId.ToString() + ".cab";
//The _Streams table lists embedded OLE data streams.
//This is a temporary table, created only when referenced by a SQL statement.
string query = "INSERT INTO `_Streams` (`Name`, `Data`) VALUES ('" + mediaCabinet + "', ?)";
pkg.Execute(query, cabRec);
Console.WriteLine(query);
/*LastSequence - File sequence number for the last file for this new media.
The numbers in the LastSequence column specify which of the files in the File table
are found on a particular source disk.
Each source disk contains all files with sequence numbers (as shown in the Sequence column of the File table)
less than or equal to the value in the LastSequence column, and greater than the LastSequence value of the previous disk
(or greater than 0, for the first entry in the Media table).
This number must be non-negative; the maximum limit is 32767 files.
/MSDN
*/
IList<int> sequences = pkg.ExecuteIntegerQuery("SELECT `LastSequence` FROM `Media` ORDER BY `LastSequence`");
lastIndex = sequences.Count - 1;
int LastSequence = sequences.ElementAt(lastIndex) + numberOfFilesToAdd;
query = "INSERT INTO `Media` (`DiskId`, `LastSequence`, `Cabinet`) VALUES (" + DiskId.ToString() + "," + LastSequence.ToString() + ",'#" + mediaCabinet + "')";
Console.WriteLine(query);
pkg.Execute(query);
return DiskId;
}
update: stupid me, forgot about "committing" in transaction mode - but now it does the same as in direct mode, so no real changes to the question.

I will answer this my self, since I just learned something about DIRECT mode that I didn't know before, and wan't to keep it here to allow for the eventual re-google..
Apparently we only succesfully updates the MSI, if we closed the database handle before the program eventually chrashed.
for the purpose of answering the question, this destructor should do it.
~className()
{
if (pkg != null)
{
try
{
pkg.Close();
}
catch (Exception ex)
{
//rollback not included as we edit directly?
//do nothing..
//atm. we just don't want to break anything if database was already closed, without dereferencing
}
}
}
after adding the correct close statement, the MSI grew in size
(and a media row was added to the media table :) )
I will post the entire class for solving this task, when its done and tested,
but I'll do it in the related question on SO.
the related question on SO

C#: ImageList in foreach loop causes application to crash on terminal server

i'm part of a (small) development team who took over the development and support of a C#.Net program. It has ~ 300.000 LOC and the software design makes it impossible to change anything big without causing millions of side effects. It's like trying to turn a jungle full of poisenous snakes into a nice little garden. Using only a small scissor.
The application is a big WinForms-Application with Database access.
About the problem: A customer received our software and cannot run it. Unlike other customers, they have multiple Windows Server 2008 R1 terminal servers and installed the software on a network drive. Their users connect to one of the terminal servers and run our application (and others, like windows office etc) from the network drive. Our application however crashes after ~ 5 seconds without any notice. Our loading screen appears and closes again. The application produces a log file which shows this exception:
2014-08-04 11:15:23 [3372] ERROR – An exception occurred:
'OutOfMemoryException': Not enough memory.
System.Drawing
at System.Drawing.Graphics.CheckErrorStatus(Int32 status)
at System.Drawing.Graphics.DrawImage(Image image, Rectangle destRect, Int32 srcX, Int32 srcY, Int32 srcWidth, Int32 srcHeight, GraphicsUnit srcUnit, ImageAttributes imageAttrs, DrawImageAbort callback, IntPtr callbackData)
at System.Drawing.Bitmap.MakeTransparent(Color transparentColor)
at System.Windows.Forms.ImageList.CreateBitmap(Original original, Boolean& ownsBitmap)
at System.Windows.Forms.ImageList.CreateHandle()
at System.Windows.Forms.ImageList.get_Handle()
at System.Windows.Forms.ImageList.GetBitmap(Int32 index)
at System.Windows.Forms.ImageList.ImageCollection.GetEnumerator()
at <our application>.InitializeHtmlResources()
The method that calls ImageCollection.GetEnumerator() is this:
void InitializeHtmlResources()
{
string baseDirPath = ... //It's in AppData/Local/<ourFolder>/Icons;
int index = -1;
foreach (Image image in UIResources.Icons.Images)
{
index += 1;
image.Save(baseDirPath + Path.DirectorySeparatorChar + index.ToString(), ImageFormat.Png);
}
}
Those images are stored inside a Resource.Icons.dll. The icons have their own project in the solution that contains of a few helper classes (like UIResources) and contains a folder with every icon we use + an xml where they are all listed, named and indexed. UIResources is a static class that allows access to the icons and the image list. This is how the property "Images" is initialized:
...
// Code snippet of Images Initialization
var ilist = new ImageList();
ilist.ColorDepth = ColorDepth.Depth32Bit;
ilist.ImageSize = new Size(16, 16);
ilist.TransparentColor = Color.Fuchsia;
UIResources.Icons.Images = ilist;
...
This method is used to extract an Icon from the DLL file.
static IEnumerable<IconInfo> GetIcons()
{
XDocument doc;
using (var stream = GetResourceStream("Our.Namespace.Resources.Icons.Icons.xml"))
{
doc = XDocument.Load(stream);
}
// ReSharper disable once PossibleNullReferenceException
foreach (var elem in doc.Element("Icons").Elements("Icon"))
{
int index = (int)elem.Attribute("Index");
var bmp = ReadIcon("Icons", (string)elem.Attribute("FileName"));
string name = (string)elem.Attribute("Name");
yield return new IconInfo(bmp, index, name);
}
}
static Bitmap ReadIcon(string kind, string fileName)
{
using (var stream = GetResourceStream("Our.Namespace.Resources." + kind + "." + fileName))
{
return new Bitmap(stream);
}
}
static Stream GetResourceStream(string resourceName)
{
return typeof(IconProvider).Assembly.GetManifestResourceStream(resourceName);
}
IconInfo is only a record containing the values.
And finally, the ImageList is filled with values:
foreach (var icon in GetIcons())
UIResources.Icons.Images.Add(icon.Name, icon.Image);
The application works fine. But when that customer runs the software on his terminal server via Remote Desktop, the application crashes and throws an OutOfMemory Exception in InitializeHtmlResources() right at the foreach-loop (when accessing the enumerator of ImageList).
The confusing part: A "OutOfMemory" exception is thrown, though the memory is neither full, nor is the 2 GB limit for 32bit application reached. The app peaks at 120 MB during loading.
I have absolutely no idea how this error is caused and spent the last 2-3 days trying to find a solution. I haven't.
I appreciate every bit of advice you can give me.
EDIT:
I tried disabling the InitializeHtmlResources-Method. This enabled the application to start. However: After working a few seconds with the application, an outofmemory exception appeared anyway. The cause is another ImageList accessor.
It works fine with Server 2012. We created a VM with Windows Server 2008 and the error happens there too.

We found the issue! Instead of
UIResources.Icons.Images.Add(icon.Name, icon.Image);
to fill the ImageList, we now use
UIResources.Icons.Images.Add(icon.Name, new Bitmap(icon.Image));
Now our application works on Windows Server 2008 :-)

Is it possible?
Memory limitations of 32 bit apps on 64 bit Terminal Server 2008 Standard

Cannot access SystemListView contents using Managed Win32 API's

I am using the ManagedWindows API in a C# environment:
http://mwinapi.sourceforge.net/
In the past I have successfully scraped the contents of listbox-like parts of other running programs using the code below, where I iterate through key/value pairs to find the list items. For this particular list of items, however, I can get an accurate number of items, but the value is always null!
Using this:
TargetMidWindow.Content.ComponentType
I have discovered that the list I am having issues with is a 'listview' whereas the other windows I have had success with are 'detailslistview' in case it matters. Below is the code I have for finding the data I want, which is almost identical to my other successful code with the exception of altering the search terms I used. Also, in case its relevant, the program I'm trying to pull data out of is MetaTrader4, and I've been able to scrape data off other parts of the program successfully.
// Find the main window
SystemWindow[] TopLevel = SystemWindow.AllToplevelWindows;
SystemWindow TargetTopWindow = SystemWindow.ForegroundWindow;
foreach (SystemWindow SearchWindow in TopLevel)
{
string Title = SearchWindow.Title;
if (Title.Contains("MetaTrader"))
{
TargetTopWindow = SearchWindow;
break;
}
}
// Find the section where positions are contained
SystemWindow[] MidLevel = TargetTopWindow.AllDescendantWindows;
SystemWindow TargetMidWindow = SystemWindow.ForegroundWindow;
foreach (SystemWindow SearchWindow in MidLevel)
{
string ClassName = SearchWindow.ClassName;
if (ClassName.Contains("SysListView32"))
{
SystemWindow ParentWindow = SearchWindow.Parent;
if ((ParentWindow.Title.Contains("Terminal")))
{
TargetMidWindow = SearchWindow;
}
}
}
// Get the positions
Dictionary<string, string> RawValues = new Dictionary<string, string>();
foreach (KeyValuePair<string, string> KVP in TargetMidWindow.Content.PropertyList)
{
string key = KVP.Key;
string value = KVP.Value;
}
Is there something special I need to do so that I do not get 'null' values for each list item?
Thanks!
Bill

Hmya, wrapping Windows messages with a friendly API isn't that difficult. Windows Forms would be a good example. But that has a knack for running into a very solid wall once you start doing this with another process.
The specific message you need in order to read ListView items is LVM_GETITEM. That's one of those solid wall messages. The LPARAM argument you pass to SendMessage() needs to be a pointer to an LVITEM structure. The control fills in the fields in that structure. Problem is, the pointer you pass is only valid in your process, not the process who owns that window.
Fixing this takes a great deal of hackery. You have to allocate memory that's valid inside that process. That takes VirtualAllocEx() and ReadProcessMemory(). Plus all the glue calls you need to make these work. I assume that this library you are using is not taking care of this. Easy to find out, grep the source code files for these API function names.

If you want to find the correct handle to a particular SysListView32 window, you need to start with the right window hierarchy. From the code snippet, it doesn't appear that you're actually finding the correct handle to retrieve a quote from the SysListView32 window. This is why you're receiving null values back. You would do well to run spy++ and determine the correct windows structure of the Metatrader terminal for your specific broker and build. I've found that the classes are different between builds for some of the windows, and also between some brokers, though to a lesser extent.
You're looking for the specific quote window hierarchy like this:
Metatrader -> Market Watch -> Market Watch -> SysListView32
By contrast, currently you're looking here in your code:
Metatrader -> Terminal -> (many sub-windows with SysListView32 class)
Where each level to the right is a child window of the window to the left.
Find the parent "Metatrader" window then chain down looking for the child window until you get to SysListView32. If you use spy++ you can read the class for the SysListView32 parent window (market watch), and use that to enumerate the windows to find the correct SysListView32 window. FYI the correct Market Watch class name for build 419 is:
Afx:00400000:b:00010003:00000000:00000000
Once you find the correct window, you may be able to extract its contents using your current component. I haven't tried that and am looking to port some code from VB6 from a ListView module that does in fact involve epic hackery. ;) I may take a look at the .NET Managed Windows API to see if this can help make the process simpler.
But in the mean time, if you do have to go low-level, the following VB6 source should help you get an idea of what is involved. This is fairly advanced material so good luck!
Public Function GetListviewItem(ByVal hWindow As Long, ByVal pColumn As Long, ByVal pRow As Long) As String
Dim result As Long
Dim myItem As LV_ITEMA
Dim pHandle As Long
Dim pStrBufferMemory As Long
Dim pMyItemMemory As Long
Dim strBuffer() As Byte
Dim index As Long
Dim tmpString As String
Dim strLength As Long
Dim ProcessID As Long, ThreadID As Long
ThreadID = GetWindowThreadProcessId(hWindow, ProcessID)
'**********************
'init the string buffer
'**********************
ReDim strBuffer(MAX_LVMSTRING)
'***********************************************************
'open a handle to the process and allocate the string buffer
'***********************************************************
pHandle = OpenProcess(PROCESS_VM_OPERATION Or PROCESS_VM_READ Or PROCESS_VM_WRITE, False, ProcessID)
pStrBufferMemory = VirtualAllocEx(pHandle, 0, MAX_LVMSTRING, MEM_COMMIT, PAGE_READWRITE)
'************************************************************************************
'initialize the local LV_ITEM structure
'The myItem.iSubItem member is set to the index of the column that is being retrieved
'************************************************************************************
myItem.mask = LVIF_TEXT
myItem.iSubItem = pColumn
myItem.pszText = pStrBufferMemory
myItem.cchTextMax = MAX_LVMSTRING
'**********************************************************
'write the structure into the remote process's memory space
'**********************************************************
pMyItemMemory = VirtualAllocEx(pHandle, 0, Len(myItem), MEM_COMMIT, PAGE_READWRITE)
result = WriteProcessMemory(pHandle, pMyItemMemory, myItem, Len(myItem), 0)
'*************************************************************
'send the get the item message and write back the memory space
'*************************************************************
result = SendMessage(hWindow, LVM_GETITEMTEXT, pRow, ByVal pMyItemMemory)
result = ReadProcessMemory(pHandle, pStrBufferMemory, strBuffer(0), MAX_LVMSTRING, 0)
result = ReadProcessMemory(pHandle, pMyItemMemory, myItem, Len(myItem), 0)
'**************************************************
'turn the byte array into a string and send it back
'**************************************************
For index = LBound(strBuffer) To UBound(strBuffer)
If Chr(strBuffer(index)) = vbNullChar Then Exit For
tmpString = tmpString & Chr(strBuffer(index))
Next index
tmpString = Trim(tmpString)
'**************************************************
'deallocate the memory and close the process handle
'**************************************************
result = VirtualFreeEx(pHandle, pStrBufferMemory, 0, MEM_RELEASE)
result = VirtualFreeEx(pHandle, pMyItemMemory, 0, MEM_RELEASE)
result = CloseHandle(pHandle)
If Len(tmpString) > 0 Then GetListviewItem = tmpString
End Function