I have a C# code which does lot of insert statements in a batch. While executing these statements, I got "String or binary data would be truncated" error and transaction roledback.
To find out the which insert statement caused this, I need to insert one by one in the SQLServer until I hit the error.
Is there clever way to findout which statement and which field caused this issue using exception handling? (SqlException)
In general, there isn't a way to determine which particular statement caused the error. If you're running several, you could watch profiler and look at the last completed statement and see what the statement after that might be, though I have no idea if that approach is feasible for you.
In any event, one of your parameter variables (and the data inside it) is too large for the field it's trying to store data in. Check your parameter sizes against column sizes and the field(s) in question should be evident pretty quickly.
This type of error occurs when the datatype of the SQL Server column has a length which is less than the length of the data entered into the entry form.
this type of error generally occurs when you have to put characters or values more than that you have specified in Database table like in that case: you specify
transaction_status varchar(10)
but you actually trying to store
_transaction_status
which contain 19 characters. that's why you faced this type of error in this code
Generally it is that you are inserting a value that is greater than the maximum allowed value. Ex, data column can only hold up to 200 characters, but you are inserting 201-character string
BEGIN TRY
INSERT INTO YourTable (col1, col2) VALUES (#val1, #val2)
END TRY
BEGIN CATCH
--print or insert into error log or return param or etc...
PRINT '#val1='+ISNULL(CONVERT(varchar,#val1),'')
PRINT '#val2='+ISNULL(CONVERT(varchar,#val2),'')
END CATCH
For SQL 2016 SP2 or higher follow this link
For older versions of SQL do this:
Get the query that is causing the problems (you can also use SQL Profiler if you dont have the source)
Remove all WHERE clauses and other unimportant parts until you are basically just left with the SELECT and FROM parts
Add WHERE 0 = 1 (this will select only table structure)
Add INTO [MyTempTable] just before the FROM clause
You should end up with something like
SELECT
Col1, Col2, ..., [ColN]
INTO [MyTempTable]
FROM
[Tables etc.]
WHERE 0 = 1
This will create a table called MyTempTable in your DB that you can compare to your target table structure i.e. you can compare the columns on both tables to see where they differ. It is a bit of a workaround but it is the quickest method I have found.
It depends on how you are making the Insert Calls. All as one call, or as individual calls within a transaction? If individual calls, then yes (as you iterate through the calls, catch the one that fails). If one large call, then no. SQL is processing the whole statement, so it's out of the hands of the code.
I have created a simple way of finding offending fields by:
Getting the column width of all the columns of a table where we're trying to make this insert/ update. (I'm getting this info directly from the database.)
Comparing the column widths to the width of the values we're trying to insert/ update.
Assumptions/ Limitations:
The column names of the table in the database match with the C# entity fields. For eg: If you have a column like this in database:
You need to have your Entity with the same column name:
public class SomeTable
{
// Other fields
public string SourceData { get; set; }
}
You're inserting/ updating 1 entity at a time. It'll be clearer in the demo code below. (If you're doing bulk inserts/ updates, you might want to either modify it or use some other solution.)
Step 1:
Get the column width of all the columns directly from the database:
// For this, I took help from Microsoft docs website:
// https://learn.microsoft.com/en-us/dotnet/api/system.data.sqlclient.sqlconnection.getschema?view=netframework-4.7.2#System_Data_SqlClient_SqlConnection_GetSchema_System_String_System_String___
private static Dictionary<string, int> GetColumnSizesOfTableFromDatabase(string tableName, string connectionString)
{
var columnSizes = new Dictionary<string, int>();
using (var connection = new SqlConnection(connectionString))
{
// Connect to the database then retrieve the schema information.
connection.Open();
// You can specify the Catalog, Schema, Table Name, Column Name to get the specified column(s).
// You can use four restrictions for Column, so you should create a 4 members array.
String[] columnRestrictions = new String[4];
// For the array, 0-member represents Catalog; 1-member represents Schema;
// 2-member represents Table Name; 3-member represents Column Name.
// Now we specify the Table_Name and Column_Name of the columns what we want to get schema information.
columnRestrictions[2] = tableName;
DataTable allColumnsSchemaTable = connection.GetSchema("Columns", columnRestrictions);
foreach (DataRow row in allColumnsSchemaTable.Rows)
{
var columnName = row.Field<string>("COLUMN_NAME");
//var dataType = row.Field<string>("DATA_TYPE");
var characterMaxLength = row.Field<int?>("CHARACTER_MAXIMUM_LENGTH");
// I'm only capturing columns whose Datatype is "varchar" or "char", i.e. their CHARACTER_MAXIMUM_LENGTH won't be null.
if(characterMaxLength != null)
{
columnSizes.Add(columnName, characterMaxLength.Value);
}
}
connection.Close();
}
return columnSizes;
}
Step 2:
Compare the column widths with the width of the values we're trying to insert/ update:
public static Dictionary<string, string> FindLongBinaryOrStringFields<T>(T entity, string connectionString)
{
var tableName = typeof(T).Name;
Dictionary<string, string> longFields = new Dictionary<string, string>();
var objectProperties = GetProperties(entity);
//var fieldNames = objectProperties.Select(p => p.Name).ToList();
var actualDatabaseColumnSizes = GetColumnSizesOfTableFromDatabase(tableName, connectionString);
foreach (var dbColumn in actualDatabaseColumnSizes)
{
var maxLengthOfThisColumn = dbColumn.Value;
var currentValueOfThisField = objectProperties.Where(f => f.Name == dbColumn.Key).First()?.GetValue(entity, null)?.ToString();
if (!string.IsNullOrEmpty(currentValueOfThisField) && currentValueOfThisField.Length > maxLengthOfThisColumn)
{
longFields.Add(dbColumn.Key, $"'{dbColumn.Key}' column cannot take the value of '{currentValueOfThisField}' because the max length it can take is {maxLengthOfThisColumn}.");
}
}
return longFields;
}
public static List<PropertyInfo> GetProperties<T>(T entity)
{
//The DeclaredOnly flag makes sure you only get properties of the object, not from the classes it derives from.
var properties = entity.GetType()
.GetProperties(System.Reflection.BindingFlags.Public
| System.Reflection.BindingFlags.Instance
| System.Reflection.BindingFlags.DeclaredOnly)
.ToList();
return properties;
}
Demo:
Let's say we're trying to insert someTableEntity of SomeTable class that is modeled in our app like so:
public class SomeTable
{
[Key]
public long TicketID { get; set; }
public string SourceData { get; set; }
}
And it's inside our SomeDbContext like so:
public class SomeDbContext : DbContext
{
public DbSet<SomeTable> SomeTables { get; set; }
}
This table in Db has SourceData field as varchar(16) like so:
Now we'll try to insert value that is longer than 16 characters into this field and capture this information:
public void SaveSomeTableEntity()
{
var connectionString = "server=SERVER_NAME;database=DB_NAME;User ID=SOME_ID;Password=SOME_PASSWORD;Connection Timeout=200";
using (var context = new SomeDbContext(connectionString))
{
var someTableEntity = new SomeTable()
{
SourceData = "Blah-Blah-Blah-Blah-Blah-Blah"
};
context.SomeTables.Add(someTableEntity);
try
{
context.SaveChanges();
}
catch (Exception ex)
{
if (ex.GetBaseException().Message == "String or binary data would be truncated.\r\nThe statement has been terminated.")
{
var badFieldsReport = "";
List<string> badFields = new List<string>();
// YOU GOT YOUR FIELDS RIGHT HERE:
var longFields = FindLongBinaryOrStringFields(someTableEntity, connectionString);
foreach (var longField in longFields)
{
badFields.Add(longField.Key);
badFieldsReport += longField.Value + "\n";
}
}
else
throw;
}
}
}
The badFieldsReport will have this value:
'SourceData' column cannot take the value of
'Blah-Blah-Blah-Blah-Blah-Blah' because the max length it can take is
16.
It could also be because you're trying to put in a null value back into the database. So one of your transactions could have nulls in them.
Most of the answers here are to do the obvious check, that the length of the column as defined in the database isn't smaller than the data you are trying to pass into it.
Several times I have been bitten by going to SQL Management Studio, doing a quick:
sp_help 'mytable'
and be confused for a few minutes until I realize the column in question is an nvarchar, which means the length reported by sp_help is really double the real length supported because it's a double byte (unicode) datatype.
i.e. if sp_help reports nvarchar Length 40, you can store 20 characters max.
Checkout this gist.
https://gist.github.com/mrameezraja/9f15ad624e2cba8ac24066cdf271453b.
public Dictionary<string, string> GetEvilFields(string tableName, object instance)
{
Dictionary<string, string> result = new Dictionary<string, string>();
var tableType = this.Model.GetEntityTypes().First(c => c.GetTableName().Contains(tableName));
if (tableType != null)
{
int i = 0;
foreach (var property in tableType.GetProperties())
{
var maxlength = property.GetMaxLength();
var prop = instance.GetType().GetProperties().FirstOrDefault(_ => _.Name == property.Name);
if (prop != null)
{
var length = prop.GetValue(instance)?.ToString()?.Length;
if (length > maxlength)
{
result.Add($"{i}.Evil.Property", prop.Name);
result.Add($"{i}.Evil.Value", prop.GetValue(instance)?.ToString());
result.Add($"{i}.Evil.Value.Length", length?.ToString());
result.Add($"{i}.Evil.Db.MaxLength", maxlength?.ToString());
i++;
}
}
}
}
return result;
}
With Linq To SQL I debugged by logging the context, eg. Context.Log = Console.Out
Then scanned the SQL to check for any obvious errors, there were two:
-- #p46: Input Char (Size = -1; Prec = 0; Scale = 0) [some long text value1]
-- #p8: Input Char (Size = -1; Prec = 0; Scale = 0) [some long text value2]
the last one I found by scanning the table schema against the values, the field was nvarchar(20) but the value was 22 chars
-- #p41: Input NVarChar (Size = 4000; Prec = 0; Scale = 0) [1234567890123456789012]
In our own case I increase the sql table allowable character or field size which is less than the total characters posted from the front end. Hence that resolve the issue.
Simply Used this:
MessageBox.Show(cmd4.CommandText.ToString());
in c#.net and this will show you main query , Copy it and run in database .
Related
I'm using Dapper with C# to get result from a store procedure.
The Table (TPTable) has Pid as Primary key, and Value as name for second column, and some other columns.
I have a store that is: select * from TPTable.
When i call QueryAsync<TPTableModel> method the result comes the value of Value Column is allways the same as Primary Key
Let say in DB
Pid = 1, Value = 2
Pid = 2, Value = 568
The Result will be a list of {{Pid=1,Value=1},{Pid=2,Value=2}}.
Why i will getting Value equals to PrimaryKey, everytime?
I found the problem. Is not with Dapper. I was repro a small project and i found the store has a inner join with an sql user defined type. This type has a Column named Value that has the values of primary key. When i call Dapper with select * on store it will fill the value of user defined type.
This works fine locally:
[Fact]
public async void SO35470588_WrongValuePidValue()
{
// nuke, rebuild, and populate the table
try { connection.Execute("drop table TPTable"); } catch { }
connection.Execute(#"
create table TPTable (Pid int not null primary key identity(1,1), Value int not null);
insert TPTable (Value) values (2), (568)");
// fetch the data using the query in the question, then force to a dictionary
var rows = (await connection.QueryAsync<TPTable>("select * from TPTable"))
.ToDictionary(x => x.Pid);
// check the number of rows
rows.Count.IsEqualTo(2);
// check row 1
var row = rows[1];
row.Pid.IsEqualTo(1);
row.Value.IsEqualTo(2);
// check row 2
row = rows[2];
row.Pid.IsEqualTo(2);
row.Value.IsEqualTo(568);
}
public class TPTable
{
public int Pid { get; set; }
public int Value { get; set; }
}
I'd be happy to investigate, but a repro would be much appreciated.
I'm trying to recieve data from an database table and my sql query also selects a column from another table that i would like to return aswell
while (dr.Read())
{
Inlagg comentsar = new Inlagg
{
names = (int)dr["name"],
dates = (DateTime)dr["Date"]
//get column value from the other table not the Inlagg table here pobbisble?
};
//This gets the column value from the other table but i dont know how to return it
string test = (string)dr["test"];
comen.Add(comentsar);
}
return comen;
How can i return a result that includes columns both columns from different tables? Side note: I have a column that i dont need to recieve here, altough the column is an int and i'm trying to recieve an string, is there any way to cast the column to an string? Any help or input highly appreciated, thanks!
EDIT: "Solved" it by simply creating a new class with the values i wanted which included columnvalues from different tables
Inside of your reader, your missing the following:
if(reader["Column"] != DBNull.Value)
model.Name = reader["Column"].ToString();
Your query for your SQL should correctly join the multiple tables, then in the returned result set you would have a column name to return by. You call the reader["..."] to access the value for said column.
As you notice the model class calls a name of a property.
The model is important as it represents your data, but to access through the reader you simply add another property.
That is because the column is null, so you can't cast a ToString() that is why the error occurs. You could attempt to also do:
int? example = reader["Column"] as int?;
Extend your class:
public class Inlagg{
...
addMyString(String myString){
this.myString = myString;
}
getMyString(){
return this.myString;
}
}
Similar to this question I am running through a datatable, using the data to fill a new dataset for the purposes of data migration.
The migration inserts into a data set then every 5000 records the added rows get saved to the database using EricEJ SqlCeBulkCopy method.
My problem is that for the first amount of records (5000 ish) the average milliseconds taken per record is around 150-200, but it gradually increases. at record 11000 this figure is now at around 475 milliseconds.
I have a typed data set with EnforceConstraints turned off.
The actual database write always takes less than a second so I am pretty sure it is not the database itself, so I am left with the code taken longer to run each iteration, which could be down to the code itself or something I am not realising about datasets.
Could the dataset be increasing the time because it is using indexes or some keys that are not turned off by using the EnforceConstraints = false property?
One other thought is that I am checking to see if a record exists before inserting it, so I have tried both the Linq methods .ANY() and FirstOrDefault() != null
I iterate through a datatable, for each record I read some values then pass them to this method.
private int MigrateItems(string reference, string brand, string captureSite, string captureOperator, DateTime captureDate, DateTime addedDate, DateTime updatedDate, bool retain)
{
//prepare the inputs
reference = reference.Trim();
int brandID = -1, databaseUpdateID = -1, captureID = -1, insertedRowID = -1;
//get the foreign keys
brandID = MigrateBrands(brand);
databaseUpdateID = MigrateDatabaseUpdates(reference);
captureID = MigrateCaptures(captureSite, captureOperator, captureDate);
//if the item doesn't exist then add it
bool exists = dataSet.Item.FirstOrDefault(a => string.Equals(a.Reference, reference, StringComparison.CurrentCultureIgnoreCase)) == null ? false : true; ;
if (exists == false)
{
var insertedRow = dataSet.Item.AddItemRow(brandID, databaseUpdateID, captureID, reference, retain, updatedDate, addedDate);
insertedRowID = insertedRow.ID;
}
else insertedRowID = dataSet.Item.Single(a => string.Equals(a.Reference, reference, StringComparison.CurrentCultureIgnoreCase)).ID;
return insertedRowID;
}
Once 5000 records have been iterated or all records have been done then I call this method:
private void BulkInsertData()
{
using (var bulkCopier = new SqlCeBulkCopy(connectionString))
{
bulkCopier.DestinationTableName = dataSet.Brand.TableName;
bulkCopier.WriteToServer(dataSet.Brand.Where(a => a.RowState == DataRowState.Added).AsEnumerable());
//(same code for all the tables)
//change all row states to unchanged
dataSet.AcceptChanges();
}
}
I'm using the following:
C#
Visual Studio 2012
Sql Server Ce 4.0
I want to bulk upload csv file data to sql server 2005 from c# code but I am encountering the below error -
Received an invalid column length from the bcp client for colid 6.
when bulk copy write to database server
I know this post is old but I ran into this same issue and finally figured out a solution to determine which column was causing the problem and report it back as needed. I determined that colid returned in the SqlException is not zero based so you need to subtract 1 from it to get the value. After that it is used as the index of the _sortedColumnMappings ArrayList of the SqlBulkCopy instance not the index of the column mappings that were added to the SqlBulkCopy instance. One thing to note is that SqlBulkCopy will stop on the first error received so this may not be the only issue but at least helps to figure it out.
try
{
bulkCopy.WriteToServer(importTable);
sqlTran.Commit();
}
catch (SqlException ex)
{
if (ex.Message.Contains("Received an invalid column length from the bcp client for colid"))
{
string pattern = #"\d+";
Match match = Regex.Match(ex.Message.ToString(), pattern);
var index = Convert.ToInt32(match.Value) -1;
FieldInfo fi = typeof(SqlBulkCopy).GetField("_sortedColumnMappings", BindingFlags.NonPublic | BindingFlags.Instance);
var sortedColumns = fi.GetValue(bulkCopy);
var items = (Object[])sortedColumns.GetType().GetField("_items", BindingFlags.NonPublic | BindingFlags.Instance).GetValue(sortedColumns);
FieldInfo itemdata = items[index].GetType().GetField("_metadata", BindingFlags.NonPublic | BindingFlags.Instance);
var metadata = itemdata.GetValue(items[index]);
var column = metadata.GetType().GetField("column", BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance).GetValue(metadata);
var length = metadata.GetType().GetField("length", BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance).GetValue(metadata);
throw new DataFormatException(String.Format("Column: {0} contains data with a length greater than: {1}", column, length));
}
throw;
}
One of the data columns in the excel (Column Id 6) has one or more cell data that exceed the datacolumn datatype length in the database.
Verify the data in excel. Also verify the data in the excel for its format to be in compliance with the database table schema.
To avoid this, try exceeding the data-length of the string datatype in the database table.
Hope this helps.
I faced a similar kind of issue while passing a string to Database table using SQL BulkCopy option. The string i was passing was of 3 characters whereas the destination column length was varchar(20). I tried trimming the string before inserting into DB using Trim() function to check if the issue was due to any space (leading and trailing) in the string. After trimming the string, it worked fine.
You can try text.Trim()
Check the size of the columns in the table you are doing bulk insert/copy. the varchar or other string columns might needs to be extended or the value your are inserting needs to be trim. Column order also should be same as in table.
e.g, Increase size of varchar column 30 to 50 =>
ALTER TABLE [dbo].[TableName]
ALTER COLUMN [ColumnName] Varchar(50)
I have been using SqlBulkCopy to transfer an Access database to SQL. Simply put, I create each table in SQL, load the corresponding Access table into a DataTable, then SqlBulkCopy the DataTable to the corresponding SQL table. Because I used a standardized function to do this, I did not setup any ColumnMappings. When there are no ColumnMappings, SqlBulkCopy seems to simply map column 0 to column 0, column 1 to column 1, etc.
In my case, I had to make sure that the column order in my Access table matched the column order in my new SQL table EXACTLY. Once I did that, it ran as expected.
Great piece of code, thanks for sharing!
I ended up using reflection to get the actual DataMemberName to throw back to a client on an error (I'm using bulk save in a WCF service). Hopefully someone else will find how I did it useful.
static string GetDataMemberName(string colName, object t) {
foreach(PropertyInfo propertyInfo in t.GetType().GetProperties()) {
if (propertyInfo.CanRead) {
if (propertyInfo.Name == colName) {
var attributes = propertyInfo.GetCustomAttributes(typeof(DataMemberAttribute), false).FirstOrDefault() as DataMemberAttribute;
if (attributes != null && !string.IsNullOrEmpty(attributes.Name))
return attributes.Name;
return colName;
}
}
}
return colName;
}
I got this error message with a much more recent ssis version (vs 2015 enterprise, i think it's ssis 2016). I will comment here because this is the first reference that comes up when you google this error message. I think it happens mostly with character columns when the source character size is larger than the target character size. I got this message when I was using an ado.net input to ms sql from a teradata database. Funny because the prior oledb writes to ms sql handled all the character conversion perfectly with no coding overrides. The colid number and the a corresponding Destination Input column # you sometimes get with the colid message are worthless. It's not the column when you count down from the top of the mapping or anything like that. If I were microsoft, I'd be embarrased to give an error message that looks like it's pointing at the problem column when it isn't. I found the problem colid by making an educated guess and then changing the input to the mapping to "Ignore" and then rerun and see if the message went away. In my case and in my environment I fixed it by substr( 'ing the Teradata input to the character size of the ms sql declaration for the output column. Check and make sure your input substr propagates through all you data conversions and mappings. In my case it didn't and I had to delete all my Data Conversion's and Mappings and start over again. Again funny that OLEDB just handled it and ADO.net threw the error and had to have all this intervention to make it work. In general you should use OLEDB when your target is MS Sql.
I just stumbled upon this and using #b_stil's snippet, I was able to figure the culprit column. And on futher investigation, I figured i needed to trim the column just like #Liji Chandran suggested but I was using IExcelDataReader and I couldn't figure out an easy way to validate and trim each of my 160 columns.
Then I stumbled upon this class, (ValidatingDataReader) class from CSVReader.
Interesting thing about this class is that it gives you the source and destination columns data length, the culprit row and even the column value that's causing the error.
All I did was just trim all (nvarchar, varchar, char and nchar) columns.
I just changed my GetValue method to this:
object IDataRecord.GetValue(int i)
{
object columnValue = reader.GetValue(i);
if (i > -1 && i < lookup.Length)
{
DataRow columnDef = lookup[i];
if
(
(
(string)columnDef["DataTypeName"] == "varchar" ||
(string)columnDef["DataTypeName"] == "nvarchar" ||
(string)columnDef["DataTypeName"] == "char" ||
(string)columnDef["DataTypeName"] == "nchar"
) &&
(
columnValue != null &&
columnValue != DBNull.Value
)
)
{
string stringValue = columnValue.ToString().Trim();
columnValue = stringValue;
if (stringValue.Length > (int)columnDef["ColumnSize"])
{
string message =
"Column value \"" + stringValue.Replace("\"", "\\\"") + "\"" +
" with length " + stringValue.Length.ToString("###,##0") +
" from source column " + (this as IDataRecord).GetName(i) +
" in record " + currentRecord.ToString("###,##0") +
" does not fit in destination column " + columnDef["ColumnName"] +
" with length " + ((int)columnDef["ColumnSize"]).ToString("###,##0") +
" in table " + tableName +
" in database " + databaseName +
" on server " + serverName + ".";
if (ColumnException == null)
{
throw new Exception(message);
}
else
{
ColumnExceptionEventArgs args = new ColumnExceptionEventArgs();
args.DataTypeName = (string)columnDef["DataTypeName"];
args.DataType = Type.GetType((string)columnDef["DataType"]);
args.Value = columnValue;
args.SourceIndex = i;
args.SourceColumn = reader.GetName(i);
args.DestIndex = (int)columnDef["ColumnOrdinal"];
args.DestColumn = (string)columnDef["ColumnName"];
args.ColumnSize = (int)columnDef["ColumnSize"];
args.RecordIndex = currentRecord;
args.TableName = tableName;
args.DatabaseName = databaseName;
args.ServerName = serverName;
args.Message = message;
ColumnException(args);
columnValue = args.Value;
}
}
}
}
return columnValue;
}
Hope this helps someone
So I'm doing something like this, and I need to specifically NOT insert the value for one of the columns, a uniqueidentifier that is defaulted to newid() in the database. How do I do this? The following code is what I want, but doesn't work because the tagID column is getting defaulted to new Guid() -- all zeros and SQL server obviously doesn't allow multiple inserts of the same uniqueidentier in a PK column.
String[] tags = this.TextBoxTags.Text.Split(',');
for (int i = 0; i < tags.Length; i++)
{
tags[i] = tags[i].Trim();
}
if (tags.Length > 0)
{
foreach (String tag in tags)
{
tblDATA_BlogTag blogTag = new tblDATA_BlogTag();
blogTag.blogID = blogPost.postID;
blogTag.tagText = tag;
blogPost.tblDATA_BlogTags.Add(blogTag);
}
}
dbTimecard.SubmitChanges();
using-default-values-in-linq-to-sql
set the Auto-Synch property to OnInsert, in code it looks like...
[Column(AutoSync=AutoSync.OnInsert,IsDbGenerated=true)]
public DateTime Created
{
}