Basically, I have a some code that takes either an (1) alphanumeric or (2) numeric serial number and increments them. Everything works for the numeric serial numbers, but when I try to insert the alphanumeric serial number, it gives me the "invalid column name" error.
I've looked at a lot of the "invalid column name" posts here and none of them seem to answer my question. I've put breakpoints in and ran the code for both cases (numeric and alphanumeric) and I'm getting the same datatypes. Basically everything seems to line up correctly, so I'm at a loss.
The following code shows how I increment for both cases. Note that for the alphanumeric increment, I am calling a method IncrementAlphaNumeric, which takes the variable 'Output', which is the result of a SQL query that sorts the table and gets the last serial number.
// Increment Numeric Serial Numbers
if (isNum)
{
int lastNumber = Int32.Parse(Output);
int[] ints = Enumerable.Range(lastNumber + 1, printQuantity).Select(i => (int)i / 1).ToArray();
increments = ints.Select(x => x.ToString()).ToArray();
output.AppendText("Serial numbers to print: " + string.Join(", ", increments));
}
// Increment AlphaNumeric Serial Numbers
if (!isNum)
{
for (int i = 0; i < printQuantity; i++)
{
increments[i] = IncrementAlphaNumeric(Output);
snList.Add(increments[i]);
Output = increments[i];
}
output.AppendText("Serial numbers to print: " + string.Join(", ", increments));
}
Finally, I use Stringbuilder in order to insert the data into the database as follows:
// (5) Store new SNs in Database
StringBuilder sb = new StringBuilder();
foreach (string newSns in increments)
{
sb.AppendLine("INSERT INTO [Manufacturing].[dbo].[Device.Devices]([SerialNumber],[DeviceTypeID]) VALUES(" + newSns + "," + dType +")");
}
using (SqlCommand insertCommand = new SqlCommand(sb.ToString(), cnn))
{
var executeNonQuery = insertCommand.ExecuteNonQuery();
}
Again, it works for numeric, but not for alphanumeric. When I put my breakpoints in and step through the code, the datatypes (Strings) are the same for each of the cases, numeric and alphanumeric.
The error message I'm getting is, again, "invalid column name". Basically, the expected results should be that the serial number, regardless of if it's numeric or alphanumeric, should be inserted into the correct table of the database, which is based on the device type (dType).
As explained in comments, you should never concatenate strings to build your sql statement.
In your case I assume that you want to insert multiple records in your database using a single statement. This can be done also using parameters and manually building the VALUES part or your query (This syntax is available from Sql Server 2008)
// Sample values, replace them with your code that builds the increments array
string[] increments = new string[] {"VALUE1", "VALUE2","VALUE3", "VALUE4"};
// Invariant part of your query
string baseQuery = "INSERT INTO [Manufacturing].[dbo].[Device.Devices]([SerialNumber],[DeviceTypeID]) VALUES";
// Fixed value for the type
string dType = "42";
List<SqlParameter> prms = new List<SqlParameter>();
List<string> placeHolders = new List<String>();
// Build a list of parameter placeholders and a list of those parameter and their values
for(int x = 0; x < increments.Length; x++)
{
placeHolders.Add($"(#p{x},{dType})");
prms.Add(new SqlParameter { ParameterName = $"#p{x}", SqlDbType = SqlDbType.NVarChar, Value = increments[x]});
}
// Put the text together
string queryText = baseQuery + string.Join(",", placeHolders);
// This should be the final text
// INSERT INTO [Manufacturing].[dbo].[Device.Devices]([SerialNumber],[DeviceTypeID])
// VALUES(#p0,42),(#p1,42),(#p2,42),(#p3,42)
using (SqlCommand insertCommand = new SqlCommand(queryText, cnn))
{
// Add all parameters to the command...
insertCommand.Parameters.AddRange(prms.ToArray());
var executeNonQuery = insertCommand.ExecuteNonQuery();
}
Related
I have a nice piece of C# code which allows me to import data into a table with less columns than in the SQL table (as the file format is consistently bad).
My problem comes when I have a blank entry in a column. The values statement does not pickup an empty column from the csv. And so I receive the error
You have more insert columns than values
Here is the query printed to a message box...
As you can see there is nothing for Crew members 4 to 11, below is the file...
Please see my code:
SqlConnection ADO_DB_Connection = new SqlConnection();
ADO_DB_Connection = (SqlConnection)
(Dts.Connections["ADO_DB_Connection"].AcquireConnection(Dts.Transaction) as SqlConnection);
// Inserting data of file into table
int counter = 0;
string line;
string ColumnList = "";
// MessageBox.Show(fileName);
System.IO.StreamReader SourceFile =
new System.IO.StreamReader(fileName);
while ((line = SourceFile.ReadLine()) != null)
{
if (counter == 0)
{
ColumnList = "[" + line.Replace(FileDelimiter, "],[") + "]";
}
else
{
string query = "Insert into " + TableName + " (" + ColumnList + ") ";
query += "VALUES('" + line.Replace(FileDelimiter, "','") + "')";
// MessageBox.Show(query.ToString());
SqlCommand myCommand1 = new SqlCommand(query, ADO_DB_Connection);
myCommand1.ExecuteNonQuery();
}
counter++;
}
If you could advise how to include those fields in the insert that would be great.
Here is the same file but opened with a text editor and not given in picture format...
Date,Flight_Number,Origin,Destination,STD_Local,STA_Local,STD_UTC,STA_UTC,BLOC,AC_Reg,AC_Type,AdultsPAX,ChildrenPAX,InfantsPAX,TotalPAX,AOC,Crew 1,Crew 2,Crew 3,Crew 4,Crew 5,Crew 6,Crew 7,Crew 8,Crew 9,Crew 10,Crew 11
05/11/2022,241,BOG,SCL,15:34,22:47,20:34,02:47,06:13,N726AV,"AIRBUS A-319 ",0,0,0,36,AV,100612,161910,323227
Not touching the potential for sql injection as I'm free handing this code. If this a system generated file (Mainframe extract, dump from Dynamics or LoB app) the probability for sql injection is awfully low.
// Char required
char FileDelimiterChar = FileDelimiter.ToChar()[0];
int columnCount = 0;
while ((line = SourceFile.ReadLine()) != null)
{
if (counter == 0)
{
ColumnList = "[" + line.Replace(FileDelimiterChar, "],[") + "]";
// How many columns in line 1. Assumes no embedded commas
// The following assumes FileDelimiter is of type char
// Add 1 as we will have one fewer delimiters than columns
columnCount = line.Count(x => x == FileDelimiterChar) +1;
}
else
{
string query = "Insert into " + TableName + " (" + ColumnList + ") ";
// HACK: this fails if there are embedded delimiters
int foundDelimiters = line.Count(x => x == FileDelimiter) +1;
// at this point, we know how many delimiters we have
// and how many we should have.
string csv = line.Replace(FileDelimiterChar, "','");
// Pad out the current line with empty strings aka ','
// Note: I may be off by one here
// Probably a classier linq way of doing this or string.Concat approach
for (int index = foundDelimiters; index <= columnCount; index++)
{
csv += "','";
}
query += "VALUES('" + csv + "')";
// MessageBox.Show(query.ToString());
SqlCommand myCommand1 = new SqlCommand(query, ADO_DB_Connection);
myCommand1.ExecuteNonQuery();
}
counter++;
}
Something like that should get you a solid shove in the right direction. The concept is that you need to inspect the first line and see how many columns you should have. Then for each line of data, how many columns do you actually have and then stub in the empty string.
If you change this up to use SqlCommand objects and parameters, the approximate logic is still the same. You'll add all the expected parameters by figuring out columns in the first line and then for each line you will add your values and if you have a short row, you just send the empty string (or dbnull or whatever your system expects).
The big take away IMO is that CSV parsing libraries exist for a reason and there are so many cases not addressed in the above psuedocode that you'll likely want to trash the current approach in favor of a standard parsing library and then while you're at it, address the potential security flaws.
I see your updated comment that you'll take the formatting concerns back to the source party. If they can't address them, I would envision your SSIS package being
Script Task -> Data Flow task.
Script Task is going to wrangle the unruly data into a strict CSV dialect that a Data Flow task can handle. Preprocessing the data into a new file instead of trying to modify the existing in place.
The Data Flow then becomes a chip shot of Flat File Source -> OLE DB Destination
Here's how you can process this file... I would still ask for Json or XML though.
You need two outputs set up. Flight Info (the 1st 16 columns) and Flight Crew (a business key [flight number and date maybe] and CrewID).
Seems to me the problem is how the crew is handled in the CSV.
So basic steps are Read the file, use regex to split it, write out first 16 col to output1 and the rest (with key) to flight crew. And skip the header row on your read.
var lines = System.File.IO.ReadAllLines("filepath");
for(int i =1; i<lines.length; i++)
{
var = new System.Text.RegularExpressions.Regex("new Regex("(?:^|,)(?=[^\"]|(\")?)\"?((?(1)(?:[^\"]|\"\")*|[^,\"]*))\"?(?=,|$)"); //Some code I stole to split quoted CSVs
var m = r.Matches(line[i]); //Gives you all matches in a MatchCollection
//first 16 columns are always correct
OutputBuffer0.AddRow();
OutputBuffer0.Date = m[0].Groups[2].Value;
OutputBuffer0.FlightNumber = m[1].Groups[2].Value;
[And so on until m[15]]
for(int j=16; j<m.Length; j++)
{
OutputBuffer1.AddRow(); //This is a new output that you need to set up
OutputBuffer1.FlightNumber = m[1].Groups[2].Value;
[Keep adding to make a business key here]
OutputBuffer1.CrewID = m[j].Groups[2].Value;
}
}
Be careful as I just typed all this out to give you a general plan without any testing. For example m[0] might actually be m[0].Value and all of the data types will be strings that will need to be converted.
To check out how regex processes your rows, please visit https://regex101.com/r/y8Ayag/1 for explanation. You can even paste in your row data.
UPDATE:
I just tested this and it works now. Needed to escape the regex function. And specify that you wanted the value of group 2. Also needed to hit IO in the File.ReadAllLines.
The solution that I implemented in the end avoided the script task completely. Also meaning no SQL Injection possibilities.
I've done a flat file import. Everything into one column then using split_string and a pivot in SQL then inserted into a staging table before tidy up and off into main.
Flat File Import to single column table -> SQL transform -> Load
This also allowed me to iterate through the files better using a foreach loop container.
ELT on this occasion.
Thanks for all the help and guidance.
This is my first foray into C# as a SSIS and Informatica developer living only in SQL. I have a script task that is reading data from a single SQL Server table via Query and simply writing that data to a text file. Everything works except what I think are two small formatting problems I can't figure out.
The following requirements are in place for this build. Thanks in advance I'm here to answer any questions!
SQL query is purposefully set as a Select * to pick up any new columns added(already in code)
First 3 columns excluded from write to file(already in code)
Problems:
" " wrappers need to be added to all values, column and rows.
Date in database is true Date but when writing to file it shows Datetime. Needs to be only date.
Current:
ID
Name
Date
Ratio
12345678
John Wayne
12/31/2018 12:00:00 AM
1/1
Needs to be:
"ID"
"Name"
"Date"
"Ratio"
"12345678"
"John Wayne"
"2018-12-31"
"1/1"
Code:
// Declare Variables
string DestinationFolder = Dts.Variables["User::Target_FilePath"].Value.ToString();
string QueryStage = Dts.Variables["User::Query_Stage"].Value.ToString();
//string TableName = Dts.Variables["User::TableName"].Value.ToString();
string FileName = Dts.Variables["User::OutputFileName"].Value.ToString();
string FileDelimiter = Dts.Variables["User::Target_FileDelim"].Value.ToString();
//string FileExtension = Dts.Variables["User::AC_Prefix"].Value.ToString();
//USE ADO.NET Connection from SSIS Package to get data from table
SqlConnection myADONETConnection = new SqlConnection();
myADONETConnection = (SqlConnection)(Dts.Connections["ADO_TEST_CONN"].AcquireConnection(Dts.Transaction) as SqlConnection);
// Read data from table or view to data table
string query = QueryStage;
SqlCommand cmd = new SqlCommand(query, myADONETConnection);
//myADONETConnection.Open();
DataTable d_table = new DataTable();
d_table.Load(cmd.ExecuteReader());
myADONETConnection.Close();
string FileFullPath = DestinationFolder + "\\" + FileName + ".txt";
StreamWriter sw = null;
sw = new StreamWriter(FileFullPath, false);
// Write the Header Row to File
int ColumnCount = d_table.Columns.Count;
for (int ic = 4; ic < ColumnCount; ic++)
{
sw.Write(d_table.Columns[ic]);
if (ic < ColumnCount - 1)
{
sw.Write(FileDelimiter);
}
}
sw.Write(sw.NewLine);
// Write All Rows to the File
foreach (DataRow dr in d_table.Rows)
{
for (int ir = 4; ir < ColumnCount; ir++)
{
if (!Convert.IsDBNull(dr[ir]))
{
sw.Write(dr[ir].ToString());
}
if (ir < ColumnCount - 1)
{
sw.Write(FileDelimiter);
}
}
sw.Write(sw.NewLine);
}
sw.Close();
Dts.TaskResult = (int)ScriptResults.Success;
Blindly running .ToString() on an object, which is done in the line sw.Write(dr[ir].ToString());, is going to use the default settings of converting that data type into a string. If it's a DateTime (the c# data type, not the SQL Date column type), then it will include the time information.
C# converts SQL column types (such as Date) into C# data types (DateTime). You need to detect this, just as you're detecting if a value is DBNull.
if (!Convert.IsDBNull(dr[ir]))
{
if (dr[ir] is DateTime dt)
{
// use DateTime's specific string rendering
sw.Write(dt.ToString("d"));
}
else
{
// fall back to standard string rendering
sw.Write(dr[ir].ToString());
}
}
You can change out the format ("d" in this case) to be something else if you need a different format. Keep in mind that the Culture of a computer will affect how the string is rendered, unless you explicitly use a named Culture.
The other thing in your problem is adding quotes around printed values. This can be done with string concatination. For example:
string result = "\"" + "my string" + "\"";
// result is "my string", with quotes
Remember to escape the quote mark.
I am haveing strange issue comparing string value. I have a value in SQL with type (nchar) this value equals "text". And I set this value to string variable x, after that I compared the string variable x with the word "text" .
The problem is it shows that x doesn't equal the value "text" even that when I add x value to label, it shows the word "text".
Here is my code:
string x;
using (SqlCommand cmd = new SqlCommand(
"select column from text_table where column = 'text'", sqlCon))
{
using (SqlDataReader reader = cmd.ExecuteReader())
{
if (reader.Read())
{
x = reader["column"].ToString();
if (x.Equals("text"))
{
// run code
}
else
{
label1.Text ="x doesn't equal text";
}
when I use the code it always shos the message "x doesn't equal text" but it should run the code as the value of the string x is "text"
You have used the "nchar" column type, which has a constant length. I am not sure, but trimming before comparison may solve your problem.
Try
x = reader["column"].ToString().Trim();
Also there is a Bug, I think: There is no "x" here in (if (stat.Equals("text")).
Demo:
create table text_table (
col1 nchar(10) ); -- fixed length column
insert text_table(col1) values ('text');
Pay attention to col1+'1' expression
select col1, col1+'1'
from text_table
where col1 = 'text'
What is going on? sql-server really ignores trailing spaces e.g.
select 'OK' where 'text' = 'text ';
returns 'OK'.
But c# doesn't ignore trailing spaces and your if is not true.
What to do?
Use Nvarchar in your db or trim db column with String.Trim() before comparison.
Always use the .Equals() method with StringComparison.OrdinalIgnoreCase
Change your code to
if(x.Equals("text",StringComparison.OrdinalIgnoreCase){
//Do something
}
This is my first project in c#. I have a little experience in Access VBA. I would like to move my apps over to be stand alone programs. I'm querying a table that has training types and dates. I would like to compare some of the types of training against each other based on the dates they were performed. The three training types are RWT010, RWP000, and RWT010BP. If RWT010BP exists and is newer it is the only one I need. Otherwise I need RWT010 and RWP000. I have figured out how to load the values into variables, but I need to be able to work with them. I would like the name of the dateTime value to be the trainType for the same row. That way I can compare them and output the right combination.
My old Access logic looked like this:
LABEL_DATE: IIf(IsNull([RWT010]),"RWT010BP: " & _
Format([RWT010BP],"Short Date"),IIf([RWT010BP]>[RWT010],"RWT010BP: " & _
Format([RWT010BP],"Short Date"),"RWT010: " & _
Format([RWT010],"Short Date") & " & " & "RWP000: " & _
Format([RWP000],"Short Date")))
This is how far I've gotten in c#:
Console.Write("Enter ID: ");
int idnum = Convert.ToInt32(Console.ReadLine());
string sql = "SELECT EXPID, TYPE, DATE_LATEST FROM TRAINING_TABLE where expid =" + idnum;
OracleCommand cmd = new OracleCommand();
cmd.Connection = conn;
cmd.CommandText = sql;
using (DbDataReader reader = cmd.ExecuteReader())
{
if (reader.HasRows)
{
while (reader.Read())
{
int expid = reader.GetInt32(0);
string trainType = reader.GetString(1);
DateTime trainDate = reader.GetDateTime(2);
It looks like the original Access logic has one DB row with three date fields, [RWT010], [RWT010BP], and [RWP000]. But in Oracle that's been normalized so you're getting back multiple rows, each of which has a a datetime field named [DATE_LATEST], and then name field called [TYPE] that's equal to "RWT010", "RWT010BP", or "RWP000".
And you were thinking, you want to handle those RWP000 date values by name, just like in Access. You were right, that's the clearest way to do it, and I'll show you how. I misunderstood what you were asking.
One way to do this would be to write an Oracle stored procedure that duplicates the Access logic. That's not the question you asked, but it's a legitimate way to do it. However, it would be more complicated than the Access version due to the change in the database, and anyway I haven't written Oracle SQL in years and I don't have an Oracle server handy to give me arbitrary, cryptic syntax errors about semicolons and whitespace.
So what I'm going to do is write a loop in C# to grab the datetimes from the DB rows and put them in local variables, and then duplicate the Access logic in C# using those variables instead of fields. It'll be a little verbose compared to the Access version, but sometimes that's how it goes.
int idnum = Convert.ToInt32(Console.ReadLine());
string sql = "SELECT EXPID, TYPE, DATE_LATEST FROM TRAINING_TABLE where expid =" + idnum;
// I don't know how you're using this so I'll just declare it here
// and leave that to you.
String dateLabel = "";
OracleCommand cmd = new OracleCommand();
cmd.Connection = conn;
cmd.CommandText = sql;
using (DbDataReader reader = cmd.ExecuteReader())
{
DateTime? RWT010 = null;
DateTime? RWT010BP = null;
DateTime? RWP000 = null;
// No need to check reader.HasRows. If it has no rows, reader.Read()
// will return false the first time, that's all.
while (reader.Read())
{
// Doesn't look to me like expid is used
//int expid = reader.GetInt32(0);
string trainType = reader.GetString(1);
DateTime trainDate = reader.GetDateTime(2);
switch (trainType) {
case "RWT010":
RWT010 = trainDate;
break;
case "RWT010BP":
RWT010BP = trainDate;
break;
case "RWP000":
RWP000 = trainDate;
break;
}
}
if (RWT010 == null || RWT010BP > RWT010) {
dateLabel = String.Format("RWT010BP: {0:d}", RWT010BP);
} else {
dateLabel = String.Format("RWT010: {0:d} & RWP000: {1:d}", RWT010, RWP000);
}
}
The original logic was this:
If RWT010 isn't null,
Do A
Otherwise, if RWT010BP > RWT010
ALSO do A
But if none of the above,
Do B
The first two branches do the exact same thing, so we can condense them both into one branch.
"Don't Repeat Yourself", as they say. You don't want to return to this code a year from now, wonder if those two lines were required to be the same, and then guess wrong or else not notice that they are the same, and only change one or the other. It's just a mess.
If you're not familiar with String.Format(), there's a lot to it. In the first argument string, {0} means "insert the second argument here"; {1} means "insert the third", and so on. The ":d" inside the curly braces is optional; it means to pass "d" as format information to the value its inserting. DateTime will interpret that "d" to mean "Short Date". You could also do it like this:
dateLabel = String.Format("RWT010BP: {0}", RWT010BP.Value.ToShortDateString());
Or like this:
dateLabel = "RWT010BP: " + RWT010BP.Value.ToShortDateString();
I have to use RWT010BP.Value in that line instead of just RWT010BP because RWT010BP is declared with a ? after it. That makes it a "nullable" value. A regular DateTime can't be null, but we need to accommodate nulls here.
If you're using C#6, you can do it like this, which I prefer. I didn't use it above because I don't know what version of C# you're on. Always prefer the least amount of "noise" cluttering up the code.
dateLabel = $"RWT010BP: {RWT010BP:d}";
That's the same ":d" as in String.Format("{0:d}", ...) above.
One more thing: idnum is an int, but don't ever concatenate a string value into a SQL string. That's a massive security vulnerability and people here will (rightly, I'm afraid) give you a very hard time for even contemplating it.
Use OracleCommand.Parameters instead, as shown in this answer. I would have used that even in this case, personally, just as a conditioned reflex.
Given a text file, how would I go about reading an particular digits in line .
Say, I have a file 123.txt. How would I go about reading line number and store first 5 digits in different variable and next 6 digits to another variable.
All I've seen is stuff involving storing the entire text file as a String array . but there are some complications: The text file is enormously huge and the machine that the application I'm coding isn't exactly a top-notch system. Speed isn't the top priority, but it is definitely a major issue.
// Please Help here
// Want to compare data of input file with database table columns.
// How to split data in to parts
// Access that split data later for comparison.
// Data in input file is like,
//
// 016584824684000000000000000+
// 045787544574000000000000000+
// 014578645447000000000000000+
// 047878741489000000000000000+ and so on..
string[] lines = System.IO.File.ReadAllLines("F:\\123.txt"); // Input file
// How can I divide lines from input file in 2 parts (For ex. 01658 and 4824684) and save it in variable so that I can use it for comparing later.
string conStr = ConfigurationManager.ConnectionStrings["BVI"].ConnectionString;
cnn = new SqlConnection(conStr);
cnn.Open();
// So I want to compare first 5 digits of all lines of input file (ex. 01658)with Transit_ID and next 6 digits with Client_Account and then export matching rows in excel file.
sql = "SELECT Transit_ID AS TransitID, Client_Account AS AccountNo FROM TCA_CLIENT_ACCOUNT WHERE Transit_ID = " //(What should I put here to comapare with first 5 digits of all lines of input file)" AND Client_Account = " ??" );
All I've seen is stuff involving storing the entire text file as a String array
Large text files should be processed by streaming one line at a time so that you don't allocate a large amount of memory needlessly
using (StreamReader sr = File.OpenText(path))
{
string s;
while ((s = sr.ReadLine()) != null)
{
// How would I go about reading line number and store first 5
// digits in different variable and next 6 digits to another variable.
string first = s.Substring(0, 5);
string second = s.Substring(6, 6);
}
}
https://msdn.microsoft.com/en-us/library/system.io.file.opentext(v=vs.110).aspx
Just use Substring(int32, int32) to get the appropriate values like this:
string[] lines = System.IO.File.ReadAllLines("F:\\123.txt");
List<string> first = new List<string>();
List<string> second = new List<string>();
foreach (string line in lines)
{
first.Add(line.Substring(0, 5));
second.Add(line.Substring(6, 6));
}
Though Eric's answer is much cleaner. This was just a quick and dirty proof of concept using your sample data. You should definitely use the using statement and StreamReader as he suggested.
first will contain the first 5 digits from each element in lines, and second will contain the next 6 digits.
Then to build your SQL, you'd do something like this;
sql = "SELECT Transit_ID AS TransitID, Client_Account AS AccountNo FROM TCA_CLIENT_ACCOUNT WHERE Transit_ID = #TransitId AND Client_Account = #ClientAcct");
SqlCommand cmd = new SqlCommand(sql);
for (int i = 0; i < lines.Count; i++)
{
cmd.Parameters.AddWithValue("#TransitId", first[i]);
cmd.Parameters.AddWithValue("#ClientAcct", second[i]);
//execute your command and validate results
}
That will loop N times and run a command for each of the values in lines.