Develop Reference

System.NotSupportedException: Character set 'utf8mb3' is not supported by .Net Framework

I am trying to run a server with a MySQL Database, however I keep getting this huge error and I am not sure why.
[21:15:49,107] Server Properties Lookup: Error While Initialization
DOL.Database.DatabaseException: Table DOL.Database.ServerProperty is not registered for Database Connection...
at DOL.Database.ObjectDatabase.SelectAllObjects[TObject]()
at DOL.GS.ServerProperties.Properties.get_AllDomainProperties()
at DOL.GS.ServerProperties.Properties.InitProperties()
at DOL.GS.GameServer.InitComponent(Action componentInitMethod, String text)```
also this error
[21:15:35,991] ExecuteSelectImpl: UnHandled Exception for Select Query "DESCRIBE `Specialization`"
System.NotSupportedException: Character set 'utf8mb3' is not supported by .Net Framework.
at MySql.Data.MySqlClient.CharSetMap.GetCharacterSet(DBVersion version, String charSetName)
at MySql.Data.MySqlClient.MySqlField.SetFieldEncoding()
at MySql.Data.MySqlClient.NativeDriver.GetColumnData(MySqlField field)
at MySql.Data.MySqlClient.NativeDriver.GetColumnsData(MySqlField[] columns)
at MySql.Data.MySqlClient.Driver.GetColumns(Int32 count)
at MySql.Data.MySqlClient.ResultSet.LoadColumns(Int32 numCols)
at MySql.Data.MySqlClient.ResultSet..ctor(Driver d, Int32 statementId, Int32 numCols)
at MySql.Data.MySqlClient.Driver.NextResult(Int32 statementId, Boolean force)
at MySql.Data.MySqlClient.MySqlDataReader.NextResult()
at MySql.Data.MySqlClient.MySqlDataReader.Close()
at MySql.Data.MySqlClient.MySqlCommand.ResetReader()
at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
at MySql.Data.MySqlClient.MySqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.ExecuteReader()
at DOL.Database.SQLObjectDatabase.ExecuteSelectImpl(String SQLCommand, IEnumerable`1 parameters, Action`1 Reader)```

There is an update available to the MySQL Connector/NET.
After updating to version 8.0.28 (NuGet package MySql.Data) everything is working again.

In my case I had installed 10.6.4-MariaDB that came by default with utf8mb3.
I got the same error when trying to read tables.
I had fixed it by changing all charset settings in MariaDB configuration to utf8mb4.
Then dumping my database and importing it again. This time specifying utf8mb4 when creating the database.
So the normal SELECT,UPDATE queries were working fine, no more error.
But I kept getting an error when my application was calling stored procedures.
I think maybe that was because the stored procedures are saved into the information_schema database, that was still utf8mb3 and I could not find a way to change it to utf8mb4.
After losing a lot of time trying to implement some weird workarounds, I had run into this bug report:
MariaDB 10.6 cannot be used from C# client applications:
https://jira.mariadb.org/browse/MDEV-26105?attachmentViewMode=list
One of the users there says:
The system character set is used only internally by MariaDB and does not need to be changed. In my client program written in C# here is what I did after connecting to make it work with MariaDB 10.6:
MySqlCommand setcmd = new MySqlCommand("SET character_set_results=utf8", conn);
int n = setcmd.ExecuteNonQuery();
setcmd.Dispose();
in this command utf8 was defaulted to utf8mb4 by MariaDB due to the old_mode setting. That is the results charset that was causing problem with my program.
So I ended up adding this in my VB project just before executing my stored procedures:
Dim sqlCommand As New MySqlCommand
sqlCommand.Connection = _MySqlCn
sqlCommand.CommandText = "SET character_set_results=utf8"
sqlCommand.ExecuteNonQuery()
This fixed the error for me.
Also, this is what I had previously changed in MariaDB server configuration:
[client]
default-character-set=utf8mb4
[mysql]
default-character-set=utf8mb4
# this is read by the standalone daemon and embedded servers
[server]
# this is only for the mysqld standalone daemon
[mysqld]
old_mode=
character-set-server = utf8mb4
character-set-client=utf8mb4
collation-server = utf8mb4_unicode_520_ci
init-connect='SET NAMES utf8mb4'

Maybe a solution.
Source :
https://dba.stackexchange.com/questions/8239/how-to-easily-convert-utf8-tables-to-utf8mb4-in-mysql-5-5
Change your CHARACTER SET AND COLLATE to utf8mb4.
For each database:
ALTER DATABASE
database_name
CHARACTER SET = utf8mb4
COLLATE = utf8mb4_unicode_ci;
For each table:
ALTER TABLE
table_name
CONVERT TO CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci;
For each column:
ALTER TABLE
table_name
CHANGE column_name column_name
VARCHAR(191)
CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci;
Worked for me with Powershell and MariaDB 10.6.
Hope this will help ;)

In my case (MariaDB 10.6) helped changing character-set-server in ...\MariaDB 10.6\data\my.ini from utf8 to utf8mb4. Now everything works fine.

For .NET CORE 5.0, I had to upgrade 3 NuGet Packages from v.5.0.0 to v.5.0.13:
MySql.EntityFrameworkCore
Microsoft.EntityFrameworkCore.Design
Microsoft.EntityFrameworkCore.Tools

I "solved" this by abandoning Oracle's MySQL Connector/Net and rewrote the application to use MySqlConnector. It was very little work, but I also had to rewrite everything about the user management since I had it depend on MySql.Web, and this was not as easy.

In C# how to provide parameters to sql statements using StackExchange.Redis Execute method using Redis with Redisql module

There is very limited information regarding the usage of Redisql module with C#.
I am using StackExchange.Redis nuget package v2.2.4 to connect Redis v5.0.7 with Redisql module installed. I am developing .NET 5 C# Application that connects and create a database and a table with predefined values.
Below is the code block that works fine and as expected.
ConnectionMultiplexer muxer = ConnectionMultiplexer.Connect("127.0.0.1:6380");
IDatabase conn = muxer.GetDatabase();
conn.Execute("DEL", "DB");
conn.Execute("REDISQL.CREATE_DB", "DB");
conn.Execute("REDISQL.EXEC", "DB", "CREATE TABLE TABLE1(A INT, B TEXT);");
conn.Execute("REDISQL.EXEC", "DB" ,"INSERT INTO TABLE1 VALUES(1, 'Value1');");
conn.Execute("REDISQL.EXEC", "DB" ,"INSERT INTO TABLE1 VALUES(2, 'Value2');");
var res = conn.Execute("REDISQL.EXEC", "DB", "SELECT * FROM TABLE1");
But what i want to do is to execute insert statements with db parameters instead of providing the values directly in the sql statements. As there is literally no examples or documentations on that I cannot find a way to do that.
I tried to rewrite the insert statement as below but it gives and error
conn.Execute("REDISQL.EXEC", "DB", "INSERT INTO TABLE1 VALUES(?1, ?2);", 1, "Value1");
StackExchange.Redis.RedisServerException: "Wrong number of arguments,
it accepts 3, you provide 5" at
StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message
message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817\n at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in
//src/StackExchange.Redis/RedisBase.cs:line 54\n at
StackExchange.Redis.RedisDatabase.Execute(String command,
ICollection`1 args, CommandFlags flags) in
//src/StackExchange.Redis/RedisDatabase.cs:line 1204\n at
StackExchange.Redis.RedisDatabase.Execute(String command, Object[]
args) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 1200\n at
deneme.Program.Main(String[] args) in
/Users/serhatonal/Projects/deneme/deneme/Program.cs:23
After that I changed the script as follows
ConnectionMultiplexer muxer = ConnectionMultiplexer.Connect("127.0.0.1:6380");
IDatabase conn = muxer.GetDatabase();
conn.Execute("DEL", "DB");
conn.Execute("REDISQL.CREATE_DB", "DB");
conn.Execute("REDISQL.EXEC", "DB", "CREATE TABLE TABLE1(A INT, B TEXT);");
conn.Execute("REDISQL.CREATE_STATEMENT", "DB", "INSERTINTOTABLE1STMT", "INSERT INTO TABLE1 VALUES(?1,?2)");
conn.Execute("REDISQL.EXEC_STATEMENT", "DB", "INSERTINTOTABLE1STMT", 1, "Value1" );
conn.Execute("REDISQL.EXEC_STATEMENT", "DB", "INSERTINTOTABLE1STMT", 2, "Value2");
var res = conn.Execute("REDISQL.EXEC", "DB", "SELECT * FROM TABLE1");
It gives below error while executing REDISQL.CREATE_STATEMENT line as described in the documentation https://redisql.redbeardlab.com/references/#redisqlexec_statement
System.ArgumentOutOfRangeException: "Specified argument was out of the
range of valid values. (Parameter 'Command 'REDISQL.CREATE_STATEMENT'
exceeds library limit of 23 bytes')"
at StackExchange.Redis.CommandBytes..ctor(String value) in
//src/StackExchange.Redis/CommandBytes.cs:line 109\n at
StackExchange.Redis.CommandMap.GetBytes(String command) in >//src/StackExchange.Redis/CommandMap.cs:line 181\n at >StackExchange.Redis.RedisDatabase.ExecuteMessage..ctor(CommandMap map, >Int32 db, CommandFlags flags, String command, ICollection1 args) in >/_/src/StackExchange.Redis/RedisDatabase.cs:line 3720\n at >StackExchange.Redis.RedisDatabase.Execute(String command, ICollection1 >args, CommandFlags flags) in >//src/StackExchange.Redis/RedisDatabase.cs:line 1203\n at >StackExchange.Redis.RedisDatabase.Execute(String command, Object[] args) >in //src/StackExchange.Redis/RedisDatabase.cs:line 1200\n at >deneme.Program.Main(String[] args) in >/Users/serhatonal/Projects/deneme/deneme/Program.cs:23
In our realtime scenario we have many sqls that uses multiple type parameters so it is not elegant to continue with sql including parameters as strings.
Any help is appreciated

From the error message, it sounds like redisql doesn't accept parameters, so: you'll probably need to inline the values yourself, taking care to avoid SQL injection. That's a topic for the module vendor.
Re the 23-byte limit: interesting that this is the first time I've seen this overflown, but: yes, we can increase that. Sorry.

Simone from RediSQL.
So, you are right about pretty much anything.
RediSQL V1 does not support passing arguments to the EXEC, it was my mistake, and it is a bad design. Fortunately we moved on with RediSQL V2 which support passing arguments to the EXECs.
Your solution to create a statement is the correct one. Statements are suppose to be used when you want to repeat, multiple times, the same query, each time with different arguments.
So you are definitely on a good track.
The second problem is due to StackExchange own limit. Here there is no much we do about.
They are storing the command in a 3 ulongs for fast memory allocation: https://github.com/StackExchange/StackExchange.Redis/blob/main/src/StackExchange.Redis/CommandBytes.cs#L28
It turns out to be (3 * 8) - 1 = 23 bytes and REDISQL.CREATE_STATEMENT is 24 bytes. It does not fit.
Solution to this problem would be to rename the REDISQL.CREATE_STATEMENT into something like REDISQL.NEW_STATEMENT, you can rename Redis commands adding something like this in your config:
rename-command REDISQL.CREATE_STATEMENT REDISQL.NEW_STATEMENT
I understand it is a cumbersome process, but it is the only solution I see.
I would really suggest moving to RediSQL V2 or zeeSQL (https://zeesql.com and documentation in https://doc.zeesql.com).
It would have made these two problem disappears.
Sorry for not answering earlier, but I didn't receive the notification. I believe I fix them now.

.NET Dapper/MySql 8.0 issue: Only root user is able to execute stored procedure. Other users get SqlNullValueException [duplicate]

I am trying to setup my .NET 4.7.1 program that is connecting to a MySQL database 8.0 to use the minimum privileges to run.
The .NET program is using MySql.Data to make connection. The minimum right for a user to execute a stored procedure is typically only EXECUTE privilege. This works fine from MySQL workbench or command line.
Upon running the .NET program this does return the following exception:
System.Data.SqlTypes.SqlNullValueException: 'Data is Null. This method or property cannot be called on Null values.'
To make it easy, I have create a very small demo program to demonstrate the issue.
Setup of the database:
CREATE DATABASE Spike;
CREATE PROCEDURE TestAccess()
BEGIN
END;
CREATE USER Spike#localhost IDENTIFIED WITH mysql_native_password BY 'sample';
GRANT EXECUTE ON PROCEDURE `TestAccess` TO Spike#localhost;
Setup program code:
static void Main(string[] args)
{
using (MySqlConnection conn = new MySqlConnection("Server=localhost;Database=Spike;uid=Spike;pwd=sample"))
{
conn.Open();
Console.WriteLine("Connection open");
MySqlCommand cmd = new MySqlCommand();
cmd.Connection = conn;
cmd.CommandText = "TestAccess";
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.ExecuteNonQuery();
Console.WriteLine("Query executed");
}
Console.ReadKey();
}
The crash happens at the line cmd.ExecuteNonQuery();
The stack from the crash is interesting, since it seems to indicate that the information_schema is queried. When logging all statements I can see that the last statement before the exception is:
SELECT * FROM information_schema.routines WHERE 1=1 AND routine_schema LIKE 'Spike' AND routine_name LIKE 'TestAccess'
I cannot grant different rights on information_schema, but I could give more rights on the stored procedure to make more information visible in the routines table, this feels wrong however. Simple tests with granting CREATE and ALTER access also did not work.
Is there something else I can do, without granting too much privileges?

This appears to be a bug in Connector/NET, similar to bug 75301 but a little different. When it's trying to determine parameter metadata for the procedure, it first creates a MySqlSchemaCollection named Procedures with all metadata about the procedure. (This is the SELECT * FROM information_schema.routines WHERE 1=1 AND routine_schema LIKE 'Spike' AND routine_name LIKE 'TestAccess' query you see in your log.)
The Spike user account doesn't have permission to read the ROUTINE_DEFINITION column, so it is NULL. Connector/NET expects this field to be non-NULL and throws a SqlNullValueException exception trying to read it.
There are two workarounds:
1) The first, which you've discovered, is to set CheckParameters=False in your connection string. This will disable retrieval of stored procedure metadata (avoiding the crash), but may lead to harder-to-debug problems calling other stored procedures if you don't get the order and type of parameters exactly right. (Connector/NET can no longer map them for you using the metadata.)
2) Switch to a different ADO.NET MySQL library that doesn't have this bug: MySqlConnector on NuGet. It's highly compatible with Connector/NET, performs faster, and fixes a lot of known issues.

I found an answer with which I am quite pleased. It is changing the connection string by adding CheckParameters=false:
using (MySqlConnection conn = new MySqlConnection("Server=localhost;Database=Spike;uid=Spike;pwd=sample;CheckParameters=false"))
This disables parameter checking, and thereby information_schema queries.

How to check if my SQL Server Express database exceeds the 10 GB size limit?

I am developing a web site, it uses SQL Server 2008 R2 Express for its database. And in testing, there is a lot of data and images stored into this database.
According to wiki, the SQL Server Express edition has a 10 GB size limit. When I insert data and reach the limit, what exception will be thrown? Or, how do I detect the approaching limit problem by codes ?
I use EF 5 with code-first approach to insert large data set.

In tests I have seen that:
sp_spaceused
won't work as expected, it showed 12GB after deleting lots of records. And the other answers regarding query sys.databases were not clear enough to me.
Searching around I found a very good explanation regarding SQL Server 2012 Express Edition 10GB Size Limit on Ramons weblog [EDIT2018 updated link]
SELECT
[name] AS [Filename],
[size]/128.0 AS [Filesize],
CAST(FILEPROPERTY([name],'SpaceUsed') AS int)/128.0 AS [UsedSpaceInMB],
[size]/128.0 - CAST(FILEPROPERTY([name],'SpaceUsed') AS int)/128.0 AS [AvailableSpaceInMB],
[physical_name] AS [Path]
FROM sys.database_files
"... space includes the transaction log and it also includes all unused space within these files. .... SQL Server Express will start complaining when it cannot reserve any more space for the datafile."
So checking
CAST(FILEPROPERTY([name],'SpaceUsed') AS int)/128.0 AS [UsedSpaceInMB]
seems to be the best option.
In combination with EF in c# my request to the DB looks like
string sqlSelect = "SELECT CAST(FILEPROPERTY([name],'SpaceUsed') AS int)/128.0 AS [UsedSpaceInMB] FROM sys.database_files";
var dbResult = dbInstance.Database.SqlQuery<Decimal>(sqlSelect).FirstOrDefault();
double spaceUsedInGb = Convert.ToDouble(dbResult)/1024;

Execute this SQL command, and it will reveal the disk-space usage of current database.
sp_spaceused
It also can be used to query the space usage of specific table. This link provides useful information about this problem.

To check the database size query:
sys.databases
Just query this, perhaps with C# or if you use SSMS (sql server management studio) shell, you can schedule a job that emails you or whatever you want.
Example:
SQL Server 2008: How to query all databases sizes?
Edit: NOT sure if error is thrown, it should log to event log or a sql log...
Side note:
Developer version is only $50 and holds same as Datacenter which hold 524 PB
http://technet.microsoft.com/en-us/library/cc645993%28v=sql.105%29.aspx

To Check the Size of the Database Two Ways:
/* new school way - data plus log and run in the local db that you want to see
here you can see the log and the mdf file.
*/
SELECT size*8.0/1024.0 as size_in_gb, *
FROM sys.database_files
GO
/* old school way, run for all db size*/
sp_helpdb
FYI - the MDF and NDF files are the only ones that attribute to the file size exceeding 10GB.

I am using the following method to calculate database current size crucial for comparing with sql size limitations:
public static int GetDbSizeInMB([NotNull] string connectionString) {
using (SqlConnection sqlConnection = new SqlConnection(connectionString)) {
sqlConnection.Open();
using (var sqlCommand = new SqlCommand()) {
sqlCommand.CommandType = CommandType.Text;
sqlCommand.CommandText = #"
SELECT SUM(CAST(FILEPROPERTY([name],'SpaceUsed') AS int)/128.0) AS [UsedSpaceInMB]
FROM sys.database_files
WHERE type_desc like 'ROWS' or type_desc like 'FULLTEXT'
";
sqlCommand.Connection = sqlConnection;
return Convert.ToInt32(sqlCommand.ExecuteScalar());
}
}
)

Simple SQL Insert

I'm new to databases and am trying to add a new record using SQL. The code runs fine the first time, but the second time throws an error saying that it can't write duplicates to a unique key. The third time runs fine, but the fourth time throws the error. Basically, it seems that every other time, the error is thrown. I understand why the error would be thrown if data was written, but when I examine the database, it remains empty. What am I doing wrong in my code that is causing the query to not bother writing the data?
EDIT If I enter the SQL directly within the database, it works. It doesn't work when I use the C# code below.
using (SqlCeConnection con = new SqlCeConnection(conString))
{
con.Open();
using (SqlCeCommand com = new SqlCeCommand("INSERT INTO User (Name, Age, URL) VALUES ( #name, #age, #url )", con))
{
com.Parameters.AddWithValue("#name", "James Y");
com.Parameters.AddWithValue("#age", 28);
com.Parameters.AddWithValue("#url", "www.example.com" );
com.ExecuteNonQuery();
}
}

I've partially figured it out. Apparently you have to also download SQL CE Tools for Visual Studio. I did this and I had a new option to include SQL CE 4 into my project (I was using the SQLCE option, assuming that it would use 4.0 by default since that's the one I installed). Only problem now is that when I try and add it, it says that it's not supported by the project type (Console project). I saw a post on MSDN that said that SQLCE 4 was for web-only projects but it was a post from a few months back and the current download page says it's for web or desktop applications. Either way, this is proving to be too much of a hassle to bother with and so I'm just going to look for an alternate database if I can't resolve this soon.
FIXED I uninstalled SQLCE4 and the SQLCE Tools, then reinstalled them.

Try and do the following as your SQL:
INSERT INTO [User] (Name, Age, URL) VALUES ( #name, #age, #url )
User is a reserved word in sql server. You should try not to name your table as "user".

Name is a reserved word. Wrap it in []
using (SqlCeCommand com = new SqlCeCommand("INSERT INTO User ([Name], Age, URL) VALUES ( #name, #age, #url )", con))

Since you use DataDirectory in your connection string, the file is copied to your bin/debug folder, so you have several copies of the database, one in your project and one in your debug folder. Make the connection string a full path to avoid any confusion while debugging!