I am processing extremely large delimited files. These files have been pre-processed to ensure that field and row delimiters are valid. Occasionally a row is processed that fails TSQL constraints (usually a datatype issue). 'Fixing' the input data is not an option in this case.
We use MAXERRORS to set an acceptable number of input errors and ERRORFILE to log failed rows.
The bulk insert completes in SSMS with severity level 16 error messages logged to the messages window for each failed row. Attempting to execute this code via the C# SqlCommand class causes an exception to be thrown when the first severity level 16 error message is generated, causing the batch to fail.
Is there a way to complete the operation and ignore SQL error messages via C# and something like SqlCommand?
Example Command:
BULK INSERT #some-table FROM 'filename'
WITH(FIELDTERMINATOR ='\0',ROWTERMINATOR ='\n',FIRSTROW = 2, MAXERRORS = 100, ERRORFILE = 'some-file')
Why not use SqlBulkCopy, and then capture the rows copied using the SqlRowsCopied event. This will more closely mimic the BULK INSERT T-SQL command.
EDIT: It looks like the error handling isn't that robust with SqlBulkCopy. However, here is an example that seems to do what you're looking for:
http://www.codeproject.com/Articles/387465/Retrieving-failed-records-after-an-SqlBulkCopy-exc
Since .NET support all the datatype as SQL you should be able to TryParse in .NET to catch any conversion errors. On date you also need to test for within the SQL data range. On text need to test length. I do exactly this on some very large inserts were I parse down some CSV. TryParse is pretty fast. Better than a Try Catch as it does not have the overhead of throwing and error.
And why not insert in .NET C#. There is a class for bulkcopy. I use TVP asynch insert while parsing and do 10,000 at a time.
It appears that an exception is thrown for each row that has an error but is just counted by SQL. However, it is also passed back to C# (SSIS in my case). I found that wrapping the bulk insert with TRY/CATCH logic and using THROW (to re-throw the exceptions) when more than MAXERRORS occur, works for me.
BEGIN TRY
BULK INSERT #some-table FROM 'filename'
WITH(FIELDTERMINATOR ='\0',ROWTERMINATOR ='\n',FIRSTROW = 2, MAXERRORS = 100,
ERRORFILE = 'some-file')
END TRY
BEGIN CATCH
THROW;
END CATCH
Related
I have a button in the UI that reveals all remaining records in a table.
To handle this, in my controller, I have a simple SQL SELECT * statement with a LIMIT and OFFSET. My ExecuteReader is currently returning my data from the SQL command, which I am adding to a List. The list contains instances of my custom Run class.
In SSMS, the SQL query executes without exception no matter how large of a LIMIT I request. If my limit is > the number of rows in the table, it just returns all rows.
In webAPI, though, when my limit is > 200, it returns an exception. Otherwise, when less than 200, it returns a List of Runs without exception. I'm trying to debug the exception that occurs when I try to return all the data, but when it passes to the catch block, the exception is null. Which is weird.
So, I think there is a step I'm missing. Maybe I shouldn't be transforming the data into the Run class while the Reader is streaming. If I verified that the SQL command is accurate, then this seems to be the step that is causing the bug. Maybe transforming the data is making the Reader sorta time out? I don't understand ExecuteReader well enough to be able to figure out how I can pass all the data to List and then transform the data in that list into Runs after closing the connection. And don't even know if that would solve problem anyway.
All misgivings about potential SQL injections and lack of dbContext, etc. aside, how can I return all my records from the database utilizing ExecuteReader()?
Thanks.
Edit to add:
My exception value in the catch block is {"Data is Null. This method or property cannot be called on Null values."}.
In the debugger output, I my exception Exception thrown: 'System.Data.SqlTypes.SqlNullValueException' in Microsoft.Data.SqlClient.dll.
Edit to comment on the solution.
Ann L. figured this out. I had null values coming from the database. I learned from her and PSGuy that I can check for null values by using DbNull. Thank you!
Note - an easy place to get tripped up is that your class has to allow for nulls or else VS won't allow you to check for nulls in the method in the controller.
Here's one approach to the syntax you'll need to use (although there are lots of other approaches: see here for a bunch of alternatives!)
shoeAge = reader.IsDBNull(13) ? null : reader.GetInt64(13)
This assumes shoeAge is a nullable Int64. If it isn't, you'll get another error since you won't be able to assign null to it.
I have a script runner written in C# and I need to improve the logging to include the result of successful arbritary statements...
If the script fails I can log the details from the exception.
If it succeeds I'm having trouble obtaining the detail for my logs.
The input is an arbritary statement - usually ddl generated from Oracle Schema Compare and broken into individual commands by a custom tool.
I thought I could obtain the information by using dbms_output.get_line but that appears to only work when I actually log a line using dbms_output.put_line in a PLSQL block.
What I'm trying to get:
Statement: "ALTER TABLE report_info MODIFY CONSTRAINT ri_fk ENABLE"
Result: "Table altered"
I'm currently using the managed oracle data provider. This seems pretty straightforward but I'm having no luck finding the correct approach.
My Current approach (Psueudo-code) :
Create Connection
Call "begin dbms_output.enable; end;"
Run Query
While (line != null) Call "begin dbms_output.get_line(:line, :status); end;"
This works for dbms_output.put_line but does not return the result of arbritary queries.
I am using the IBM DB2 driver for .NET with command chaining.
I first open a DB2Connection and start a transaction. Then I call DB2Connection.BeginChain on my connection to start a bulk insert. I execute a bunch of prepared statements with 0 as the DB2Command.CommandTimeout. Last, I call DB2Connection.EndChain and commit the transaction.
I expect some of the inserts to fail due to duplicate key errors. I trap this by catching a DB2Exception and inspecting the DB2Exception.Errors collection. I know which row failed because I can look at the DB2Error.RowNumber inside the Errors collection.
The problem is that sometimes I trap a DB2Exception when I call DB2Connection.EndChain and the affected row number is negative.
[IBM][DB2] SQL0952N Processing was cancelled due to an interrupt. SQLSTATE=57014
Searching the DB2 documentation for this error seems to indicate that a query has timed out. I didn't see any information how this relates to chaining. Did the entire chain process time out or was it a problem with an individual query? If the later, then why didn't I get a valid row number? Why am I timing out at all if my DB2Connection.ConnectionTimeout is 0 and my DB2Command.CommandTimeout is 0?
I'm using the following code to do a batch insert using the C# driver. I have a unique index, and I want it to fail silently if I try to insert a record that isn't unique.
Even though I have InsertFlags.ContinueOnError set, I still get an error on the InsertBatch call. If I swallow the error as I have shown below, everything works ok. But this certainly feels wrong.
var mio = new MongoInsertOptions {Flags = InsertFlags.ContinueOnError};
// newImages is a list of POCO objects
try
{
_db.GetCollection("Images").InsertBatch(newImages, mio);
}
catch (WriteConcernException)
{
}
Are you using version 1.8 of the csharp Mongo driver?
If so, try upgrading to version 1.8.1 which contains a fix for the following two issues:
InsertBatch fails when large batch has to be split into smaller sub batches
InsertBatch throws duplicate key exception with too much data...
So your inserts could succeed, but the driver is still throwing an exception on bulk insert operations due to the bug above.
And this exception doesn't originate from the database itself, explaining why the inserts succeed but you still need to catch the exception afterwards - i.e. the db is in fact respecting your ContinueOnError flag but the driver throws an exception anyway afterwards.
I'm doing a project in Asp .Net and C#. I'm also using ODP .NET to connect to my Oracle DB.
I am using a stored procedure to insert a value into the database. Everything is fine, it also raises all the exceptions I have.
I have 3 exceptions:
when the value I try to insert already exists in the database
when i try to insert a null value
Default "Others"
When the first two exceptions are raised I insert the error into a table Errors, everything goes well.
What I really want to know is, how can I show a message to the user when a Exception occurs?
Something like dbms_output.put_line("Error");...but I want it to show in my webpage. Is this possible?
Any tips are welcome, thanks in advance.
Since your .NET program is your client, you should let any unhandled exceptions propagate from your PL/SQL program and back to the client, where you can deal with it like any other exception.
In other words, you should remove the "when others" exception from your PL/SQL code, and instead wrap your database call (using ODP.NET) with a C# exception block. There you can catch the exception and get the Oracle error number and text, and display it to the user if you want.
(Using this approach you can also use RAISE_APPLICATION_ERROR in your PL/SQL code to signal errors back to the C# client.)
You can fix the stored procedure to return an integer, 0 means ok, 1 means exception of type 1, 2 exception of type 2 and so on... in this way the UI can notify the user that saving method did not go well as expected.
Normally I would not have handled the exception in the stored procedure but I understand that you want to log the error from SQL so the above way should allow you to do what you want.
In our sql server stored procedures we have something like this:
IF ##ERROR <> 0
return (1)
ELSE
return (0)
but it depends on what the stored does and what else it returns, if the stored returns a table, you can then use output parameter to send back the integer discussed above.