I am having a problem with writing a string to a mysql database that contains a utf-8 character. Everything before the character gets written to the table, but the character and everything after it is not there.
I have checked the character sets of the database and the default collation is utf8_general_ci and the default characterset is utf8. The column being written to is type longtext with the collation utf8_general_ci.
I have also tried adding SET NAMES utf8; to the query but this did not change the result.
Here is an example of the code being run:
using (var cmd = new MySqlCommand("insert into tablename (BodyText) values (#p1)", connection as MySqlConnection) { CommandType = CommandType.Text })
{
cmd.Parameters.Add("#p1", BodyText);
cmd.ExecuteNonQuery();
}
The connection string is:
"SERVER=xx;DATABASE=xx;USER=xx;PASSWORD=xx;Pooling=true;Validate Connection=true;CHARSET=UTF8"
And the text that is attempting to write to the table is "Thank you! I saw it as a 2 😝 more text...", and what is written to the table is "Thank you! I saw it as a 2 ".
Any help on the matter would be appreciated.
Update: After further research the problem appears to be that the base utf8 encoding in MySql does not support 4-byte characters (of which 😝 is). The solutions to this are either to update the MySql database to use utf8mb4 or remove the characters from the string before writing to the table. The problem with the second solution is on a large codebase this check would have to be done everywhere text is written to the database. Any suggestions on how to handle this issue would be welcome.
You can probably to this to change from utf8 to utf8mb4 in a table without losing or mangling any data:
ALTER TABLE tbl MODIFY COLUMN col ... CHARACTER SET utf8mb4 ...;
(Where the '...' is the stuff currently used to define the column, such as LONGTEXT.) You might want to specify the collation at the same time.
If that fails, this will probably work:
ALTER TABLE Tbl MODIFY COLUMN col LONGBLOB ...;
ALTER TABLE Tbl MODIFY COLUMN col LONGTEXT ... CHARACTER SET utf8mb4 ...;
If col is in any indexes, you might want to DROP INDEX in the first ALTER and ADD INDEX in the second. (This is for efficiency and possibly to avoid index limitations.)
Related
We trying to save article (long string) in database column using EF6 database first, the article character is more than 4000 char so we changed the column type from NVARCHAR(4000) to VARCHAR(8000) but after when we tried to save in database, We found the data saved as quationmark '???? ???? ????' , we changed the column collation to Arabic_CI_AI
as following :
ALTER TABLE laws.ArticleContent ALTER COLUMN TextAr
VARCHAR(8000) COLLATE Arabic_CI_AI
but still have the same problem also we update the Edmx after we changed the column properties but still the same, and we change the property in Edmx to Unicode = true and we got the same, so please any help
We use MSSQL2008R2 and EF6 database first
the table after insert :
the Edmx Update :
the table after update column collation :
Change the column data type to nvarchar(max)
by default this will set the field length to 4000 unless you send data exceed the 4000, it will will convert it to a pointer to a data.
Some helpful questions and answers:
What is the maximum characters for the NVARCHAR(MAX)?
Are there any disadvantages to always using nvarchar(MAX)?
I am trying to insert square root symbol √ into my SQL Server database from an ASP.NET page. The radical symbol (√) gets inserted as letter v instead.
Where could I be going wrong?
Thanks for your assistance.
Your database column type should be nVarChar to insert Unicode characters.
Also you need to pass values like below:
cmd.Parameters.Add("#ColumnName", SqlDbType.NVarChar, 1024).Value = txtName.Text;
Column in your table should be able to store unicode characters, try change the type to nvarchar
and while inserting you should use N symbol before the value
let say the column name is square_root and table is test
insert into test(square_root) values(N'√25 = 5')
i want to insert data into my tbl_Transaction_Master table.
here is my database table tbl_Transaction_Master
and my query for database insert command is
SqlCommand cmd2 = new SqlCommand("insert into tbl_Transaction_Master(Supplier_ID,Order_Price,Unique_ID,Supplier_Name,He_Is_a,Transaction_Date) values ('" + WebUserID + "','" + Session["Order_Price"] + "','" + Session["WebUserid"] + "','"+User+"','WebCustomer',getdate()); SELECT SCOPE_IDENTITY()", conn);
int temp = cmd2.ExecuteNonQuery();
Session["order_ID"] = temp;
i am getting the error as mentioned above.
There are many problems in your code, not just the one that raises the current exception.
First: You have text fields with a precise size, you cannot insert more characters than the size of the field. We don't know anything about the value passed for the SupplierName but the string WebCustomer is 11 chars and you try to insert it in a field with space only for 10 chars.
Second: Using nchar means that your fields are always filled with spaces to reach the length required. Of course this is inefficient with a lot of records, not to mention the work required to trim away those spaces when you need to show your data.
Third: ExecuteNonQuery returns the number of rows affected by your command. In this case it is always 1, you don't get the return value of SELECT SCOPE_IDENTITY(). Use ExecuteScalar.
Fourth: Some of your fields are numeric, you insert strings for them. This means that the database engine try to convert them back to the correct datatype. Usually, for integers, you can go away freely, but with floats there is a higher chance that you get a datatype mismatch error caused by difference between the locale settings of your code and the locale settings of your database. Fix for that in the next point.
Fifth: You should use parameters not string concatenation. This avoids the Sql Injection hack or the parsing error caused by automatic conversion of a string back to a numeric value. A mismatch from the locale settings used in your code and the database setting will cause errors. With a parameter you specify the type and do not convert the value. The database engine is happy.
So.....(after changing to nvarchar and after checking the length of the values)
string cmdText = #"insert into tbl_Transaction_Master
(Supplier_ID,Order_Price,Unique_ID,
Supplier_Name,He_Is_a,Transaction_Date) values
(#webid, #price,#sessionid,#user,'WebCust.',getdate());
SELECT SCOPE_IDENTITY()";
SqlCommand cmd2 = new SqlCommand(cmdText, conn);
cmd2.Parameters.Add("#webid", SqlDbType.Int).Value = WebUserID
cmd2.Parameters.Add("#price", SqlDbType.Decimal).Value = Session["Order_Price"];
cmd2.Parameters.Add("#sessionid", SqlDbType.Int).Value = Session["WebUserid"];
cmd2.Parameters.Add("#user", SqlDbType.NVarChar).Value =User;
int temp = Convert.ToInt32(cmd2.ExecuteScalar());
Session["order_ID"] = temp;
You are inserting a value in a table column which exceeds column length. That's why the error is there. Check length of each of the values you are inserting into table against respective column lengths.
You are probably inserting a text longer than the designated size. For instance, for Supplier_Name, you are designating an nchar of 20. If you are trying to insert a value larger than 20 characters. You will get this error.
Try changing the size of the data type to fix this.
Two things.
One: The error is occurring because you trying to place data into the field which is larger than allowed so for nchar(20) it will accept an input of 20 or less and pad it up to 20 charters (Varchar will carry the came cap but not pad the data and save space).
Two: It is safer and advised as a best practice to use parameters when inserting values into the database.
You are inserting more characters but you define less.
for example:
column datatype is varchar(5) but you are inserting 6 characters.
I have a column in table with text datatype and trying to save some string value to this column from C# code. Issue comes when I use some very large string.
I am not able to save more than 43679 character into text field. I know text size can be 2^31.
I also tried saving same value from SSMS and noticed same scenario.
There is nothing special about code, but still SQL query is given below...
update TableName
set ColumnName = 'some text more than 43679 char'
where id=<some int id>
just to mention... column is declare in table as
[columnname] [text] NULL
Can anyone tell me what could be wrong.
You can try to use varchar(max) to store huge amount of data. See MSDN
We recommend that you store large data by using the varchar(max),
nvarchar(max), or varbinary(max) data types. To control in-row and
out-of-row behavior of these data types, use the large value types out
of row option.
You can also check the same issue here: SSMS - Can not paste more than 43679 characters from a column in Grid Mode
I'm using the C# .NET Mysql Connector, and when running this query:
INSERT INTO convos (`userid`,`time`,`from`,`content`,`read`,`deleted`, `ip`, `source`, `charname`, `to`) VALUES ('3', '1347396787', 'Chára', '........', '0', '0', '0.0.0.0:0000', 'C', 'óóóíííí', 'óóóíííí');
I get the following error:
Incorrect string value: '\xE1ra' for column 'from' at row 1
I understand my encoding, everything was configured for utf8, utf8_general_ci. Database, table and columns are configured for utf8. The data is sent from the client in utf8.
If i use a 3rd party tool like, Workbench to insert the query or use the mysql command line it works fine. I don't know if there is a bug with the connector or i need to be doing something else with the values before insert?
Any idea?
Thanks
Is there any in mysql to covert to the correct type?
I believe you need to alter the column's char set:
use below code for those columns which is using UTF-8.
ALTER TABLE database.table MODIFY COLUMN col VARCHAR(255)
CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL;
Unicode string prefix with N
First see your table convos and make sure columns data types is either one of nchar, nvarchar and You must precede all Unicode strings with a prefix N when you deal with Unicode string constants in SQL Server
Tyr:
insertQuery = "INSERT INTO convos (`userid`,`time`,`from`,`content`,`read`,`deleted`, `ip`, `source`, `charname`, `to`) VALUES
(N'3', N'1347396787', N'Chára', N'........', N'0', N'0', N'0.0.0.0:0000', N'C', N'óóóíííí', N'óóóíííí')";
I figured this out, its taken a while but it seems i was setting the charset too often. The database, tables, columns are all in UTF8. When i made a connection i had "CHARSET=UTF8" in the connection string. I was also running "SET NAMES 'utf8' COLLATE 'utf8_general_ci'" everytime i made a connection. I dropped the CHARSET=UTF8 and "SET NAMES 'utf8' COLLATE 'utf8_general_ci'" and its all working now.
Update it
INSERT INTO convos (`userid`,`time`,`from`,`content`,`read`,`deleted`, `ip`, `source`, `charname`, `to`) VALUES ('3', '1347396787', 'Chara', '........', '0', '0', '0.0.0.0:0000', 'C', 'óóóíííí', 'óóóíííí');
I think for "chara"in your 3rd value gives it
For someone who has tried all the suggestions, and nothing has worked (like myself), it is worth checking what MySQL types are your fields mapped to in C#. My text fields were automatically mapped as MySqlDbType.Blob and that was causing the error. I changed the type to MySqlDbType.Text, and I don't see the error any more.
Here is my original response to a similar thread:
https://stackoverflow.com/a/16989466/2199026
config mysql like below. it will solve the unicode error when insert into mysql from java or C#
enter image description here