I have a table in SQL Server. One of the columns in this table holds XML data (ntext) like this:
<data>
<id>123</id>
<title>This is the title</title>
</data>
I then have some C# code that gets the innerText of title and checks if this value exists in another table. Something like this:
private void TestTitleExist(string innerTextTitle)
{
SqlConnection conn = new SqlConnection("Data Source=;Initial Catalog=;Persist Security Info=True;User ID=;Password=");
conn.Open();
SqlCommand command = new SqlCommand("select 'x' from titles where lower(title) = :#title", conn);
command.Parameters.AddWithValue("#title",innerTextTitle);
using (SqlDataReader reader = command.ExecuteReader())
{
if (reader.Read())
{
Console.WriteLine(String.Format("{0}",reader["title"]));
}
}
conn.Close();
}
After debugging this method, the innerText title is being passed into this method.
For some reason, this query is returning empty even though this title exists in the titles table.
If I run the query directly in SQL server:
select 'x' from titles where title = 'This is the title'
I get 0 rows. But if I remove the white spaces from the title itself (This is the title) and add the spaces back in manually and run the query I get 1 row as expected. It seems like the actual spaces in the titles are adding some kind of formatting. Any idea?
It is better to use XML data type column for any data in XML format.
Please try the following conceptual example for your exact scenario.
It covers just SQL Server side. All you need to do is to call its SQL statement. The idea is very simple:
Convert NTEXT data type into XML data type.
Query XML data by using powerful XQuery methods available in SQL Server.
Casting as xs:token does the following 'magic':
All invisible TAB, Carriage Return, and Line Feed characters will be
replaced with spaces.
Then leading and trailing spaces are removed from the value.
Further, contiguous occurrences of more than one space will be
replaced with a single space.
SQL #1
-- DDL and sample data population, start
DECLARE #tbl TABLE (ID INT IDENTITY PRIMARY KEY, xmldata NTEXT);
INSERT INTO #tbl (xmldata) VALUES
(N'<data>
<id>123</id>
<title>This is the title</title>
</data>'),
(N'<data>
<id>123</id>
<title>This is NOT the title</title>
</data>');
-- DDL and sample data population, end
;WITH rs AS
(
SELECT ID, TRY_CAST(xmldata AS XML) AS xmldata
FROM #tbl
)
SELECT *
FROM rs
WHERE rs.xmldata.exist('/data[title/text()="This is the title"]') = 1;
Output
+----+-----------------------------------------------------------+
| ID | xmldata |
+----+-----------------------------------------------------------+
| 1 | <data><id>123</id><title>This is the title</title></data> |
+----+-----------------------------------------------------------+
SQL #2
Second scenario covers embedded white spaces in the XML.
-- DDL and sample data population, start
DECLARE #tbl TABLE (ID INT IDENTITY PRIMARY KEY, xmldata NTEXT);
INSERT INTO #tbl (xmldata) VALUES
(N'<data>
<id>123</id>
<title>This is the title </title>
</data>'),
(N'<data>
<id>123</id>
<title>This is NOT the title</title>
</data>');
-- DDL and sample data population, end
;WITH rs AS
(
SELECT ID, TRY_CAST(xmldata AS XML) AS xmldata
FROM #tbl
)
SELECT *
FROM rs
WHERE rs.xmldata.exist('/data[string((title/text())[1]) cast as xs:token? eq "This is the title"]') = 1;
Output
+----+----------------------------------------------------------------+
| ID | xmldata |
+----+----------------------------------------------------------------+
| 1 | <data><id>123</id><title>This is the title </title></data> |
+----+----------------------------------------------------------------+
Related
I have a SQL Server 2008 R2 datatable dbo.Forum_Posts with columns Subject (nvarchar(255)) andBody (nvarchar(max)).
I would like to get all words with length >= 3 from columns Subject and Body and insert them into datatable dbo.Search_Word (column Word, nvarchar(100)) and datatable dbo.SearchItem (column Title (nvarchar(200)).
I also want to get new generated SearchWordsID (primary key, autoincrement, int) from dbo.Search_Word, and SearchItemID (primary key, autoincrement,int) from dbo.SearchItem, and insert them into datatable dbo.SearchItemWord (columns SearchWordsID (foreign key,int, not null) and SearchItemID (foreign key,int,not null).
What is a fastest way to do this in T-SQL? Or I have to use C#? Thank you in advance for any help.
As requested, this will keep the ID's. So you will get a DISTINCT list of works BY id.
Slightly different approach than the first answer, but easily achieved via the Outer Apply
**
You must edit the initial query Select KeyID=[YourKeyID],Words=[YourField1]+' '+[YourField2] from [YourTable]
**
Declare #String varchar(max) = ''
Declare #Delimeter varchar(25) = ' '
-- Generate and Strip special characters
Declare #StripChar table (Chr varchar(10));Insert Into #StripChar values ('.'),(','),('/'),('('),(')'),(':') -- Add/Remove as needed
-- Generate Base Data and Expand via Outer Apply
Declare #XML xml
Set #XML = (
Select A.KeyID
,B.Word
From ( Select KeyID=[YourKeyID],Words=[YourField1]+' '+[YourField2] from [YourTable]) A
Outer Apply (
Select Word=split.a.value('.', 'varchar(150)')
From (Select Cast ('<x>' + Replace(A.Words, #Delimeter, '</x><x>')+ '</x>' AS XML) AS Data) AS A
Cross Apply data.nodes ('/x') AS Split(a)
) B
For XML RAW)
-- Convert XML to varchar(max) for Global Search & Replace (could be promoted to Outer Appy)
Select #String = Replace(Replace(cast(#XML as varchar(max)),Chr,' '),' ',' ') From #StripChar
Select #XML = cast(#String as XML)
Select Distinct
KeyID = t.col.value('#KeyID', 'int')
,Word = t.col.value('#Word', 'varchar(150)')
From #XML.nodes('/row') AS t (col)
Where Len(t.col.value('#Word', 'varchar(150)'))>3
Order By 1
Returns
KetID Word
0 UNDEF
0 Undefined
1 HIER
1 System
2 Control
2 UNDEF
3 JOBCONTROL
3 Market
3 Performance
...
87 Analyitics
87 Market
87 UNDEF
88 Branches
88 FDIC
88 UNDEF
...
You're going to need T-SQL to do the inserting into your tables. Your biggest challenge is going to be splitting the posts into words.
My suggestion would be to read the posts into C#, split each post into words (you can use the Split method to split on spaces or punctuation), filter the collection of words, and then execute your inserts from C#.
You can avoid using T-SQL directly if you use Entity Framework or a similar ORM.
Don't try to use T-SQL to split your posts into words unless you really want a totally SQL solution and are willing to take time to perfect it. And, yes, it will be slow: T-SQL is not fast at string operations.
You can also investigate full text indexing, which I believe has support for search keywords.
Perhaps this will help
Declare #String varchar(max) = ''
Declare #Delimeter varchar(25) = ' '
Select #String = #String + ' '+Words
From (
Select Words=[YourField1]+' '+[YourField2] from [YourTable]
) A
-- Generate and Strip special characters
Declare #StripChar table (Chr varchar(10));Insert Into #StripChar values ('.'),(','),('/'),('('),(')'),(':') -- Add/Remove as needed
Select #String = Replace(Replace(#String,Chr,' '),' ',' ') From #StripChar
-- Convert String into XML and Split Delimited String
Declare #Table Table (RowNr int Identity(1,1), String varchar(100))
Declare #XML xml = Cast('<x>' + Replace(#String,#Delimeter,'</x><x>')+'</x>' as XML)
Insert Into #Table Select String.value('.', 'varchar(max)') From #XML.nodes('x') as T(String)
-- Generate Final Resuls
Select Distinct String
From #Table
Where Len(String)>3
Order By 1
Returns (sample)
String
------------------
Access
Active
Adminstrators
Alternate
Analyitics
Applications
Branches
Cappelletti
City
Class
Code
Comments
Contact
Control
Daily
Data
Date
Definition
Deleted
Down
Email
FDIC
Variables
Weekly
I have about 10,000 XML files where I need to convert them into SQL table.
However, here are the problems, each XML files has some variations between each other thus it is almost impossible for me to specify the element name. For example:
//XML #1
<color>Blue</color>
<height>14.5</height>
<weight>150</weight>
<price>56.78</price>
//XML #2
<color>Red</color>
<distance>98.7</distance>
<height>15.5</height>
<price>56.78</price>
//XML #3: Some of the elements have no value
<color />
<height>14.5</height>
<price>78.11</price>
//XML #4: Elements has parent/child
<color>
<bodyColor>Blue</bodyColor>
<frontColor>Yellow</frontColor>
<backColor>White</backColor>
</color>
<height>14.5</height>
<weight>150</weight>
<price>56.78</price>
With the example above, I should expect a table created with columns name: color, height, weight, price, distance (Because XML #2 has distance), bodyColor, frontColor, backColor.
Expected output:
XML# color height weight price distance bodyColor frontColor backColor
1 Blue 14.5 150 56.78 NULL NULL NULL NULL
2 Red 15.5 NULL 56.78 98.7 NULL NULL NULL
3 NULL 14.5 NULL 78.11 NULL NULL NULL NULL
4 NULL 14.5 150 56.78 NULL Blue Yellow White
In this case, NULL or empty value are acceptable.
These are just examples, there are at least 500 elements in each XML file. Also, even though I mentioned C# here, if anyone can suggest a better way of doing so, please let me know.
One possibility to iterate over all xml files and get all unique tags could use LINQ2XML, the HashSet class and could look like this:
try
{
// add as many elements you want, they will appear only once!
HashSet<String> uniqueTags = new HashSet<String>();
// recursive helper delegate
Action<XElement> addSubElements = null;
addSubElements = (xmlElement) =>
{
// add the element name and
uniqueTags.Add(xmlElement.Name.ToString());
// if the given element has some subelements
foreach (var element in xmlElement.Elements())
{
// add them too
addSubElements(element);
}
};
// load all xml files
var xmls = Directory.GetFiles("d:\\temp\\xml\\", "*.xml");
foreach (var xml in xmls)
{
var xmlDocument = XDocument.Load(xml);
// and take their tags
addSubElements(xmlDocument.Root);
}
// list tags
foreach (var tag in uniqueTags)
{
Console.WriteLine(tag);
}
}
catch (Exception exception)
{
Console.WriteLine(exception.Message);
}
Now you have the columns for the basic SQL table. With little enhancing, you could also mark the parent and the sub nodes. This could help you for the normalization.
You can do this in TSQL using xQuery, a staging table and dynamic pivot.
Staging table:
create table dbo.XMLStage
(
ID uniqueidentifier not null,
Name nvarchar(128) not null,
Value nvarchar(max) not null,
primary key (Name, ID)
);
ID is unique per file, Name hold the node name and Value the node value.
Stored procedure to populate the staging table:
create procedure dbo.LoadXML
#XML xml
as
declare #ID uniqueidentifier;
set #ID = newid();
insert into dbo.XMLStage(ID, Name, Value)
select #ID,
T.X.value('local-name(.)', 'nvarchar(128)'),
T.X.value('text()[1]', 'nvarchar(max)')
from #XML.nodes('//*[text()]') as T(X);
//*[text()] will give you all nodes that have a text value
Dynamic query to unpivot the data in the staging table:
declare #Cols nvarchar(max);
declare #SQL nvarchar(max);
set #Cols = (
select distinct ',' + quotename(X.Name)
from dbo.XMLStage as X
for xml path(''), type
).value('substring(text()[1], 2)', 'nvarchar(max)');
set #SQL = '
select '+#Cols+'
from dbo.XMLStage
pivot (max(Value) for Name in ('+#Cols+')) as P';
exec sp_executesql #SQL;
Try it out in this SQL Fiddle
I having XMl file I am Reading All the Xml using this code . But mean while I want to insert XML records to sql table,Whatever I have Read. I did not Find any Solution for that .XMl consists No of rows and how can i insert in Proper way .
if (File.Exists(xmlpath))
{
try
{
XDocument xmlDoc = XDocument.Load(xmlpath);
var vrresult = from a in xmlDoc.XPathSelectElements("/Parts/Part")
select new
{
S = a.Element("Section").Value,
M = a.Element("Mark").Value,
Q = a.Element("Qty").Value,
W = a.Element("Weight").Value,
RD = a.Element("databaseUpdateMark").Value
};
GridView1.DataSource = vrresult;
GridView1.DataBind();
}
catch (Exception ex)
{
Lbmsg.Text = ex.Message;
}
finally
{
}
}
here is one example how to use OPENXML function
Bulk Insertion of Data Using C# DataTable and SQL server OpenXML function
Make use of function : OPENXML (Transact-SQL) which is avialble in sql server allows you to insert data in server....
also read : Use OPENXML to insert values from a string of XML
Example :
--Use OPENXML to pull records from an XML string and insert them into a database
DECLARE #idoc int
DECLARE #doc varchar (1000)
--XML document
SET #doc ='
<ROOT>
<Employee>
<FirstName>Tom</FirstName>
<LastName>Jones</LastName>
</Employee>
<Employee>
<FirstName>Gus</FirstName>
<LastName>Johnson</LastName>
</Employee>
</ROOT>'
--Create an internal representation of the XML document
--the idoc variable is set as the document handler, so we can refer to it later
EXEC sp_xml_preparedocument #idoc OUTPUT, #doc
-- Use the OPENXML rowset provider with an Insert
-- #idoc lets us know which internal representation of an xml doc to use
-- '/ROOT/Employee' shows us which node in the xml tree to work on
-- the 2 denotes we want to use elemenet centric mapping for the values, as oppsed to attributes or a combination of both
-- in the 'With' we specify how the output rowset will look
INSERT INTO Employees (FirstName, LastName)
SELECT *
FROM OPENXML (#idoc, '/ROOT/Employee',2)
WITH (FirstName varchar(10),
LastName varchar(20)
)
-- Clear the XML document from memory
EXEC sp_xml_removedocument #idoc
I'm attempting to create a small console app in C# to perform inserts on a table of Products (ITEMS) in SQL Server 2008 according to the contents of an XML file in the FASTEST way possible. I already have an .XSD file that contains the proper mappings to the SQL table (which may not be necessary with the approach outlined below).
Here's a high-level of my approach:
Read the XML, using it to create a table.
Perform a MERGE against the ITEMS table using the table created from the XML file.
2a. If the item exists, update it.
2b. If the item does not exist, insert it.
Create a log of only the records inserted in XML.
Consider the following ITEMS table and XML file:
ITEMS
Item_Id Name Price
1 Coke 5.00
2 Pepsi 3.00
3 Sprite 2.00
ITEMS.XML
<?xml version="1.0" encoding="ISO-8859-1"?>
<Item>
<Id>5</Id>
<Name>Mountain Dew</Name>
<Price>4.50</Price>
</Item>
<Item>
<Id>3</Id>
<Name>Sprite Zero</Name>
<Price>1.75</Price>
</Item>
After the import, the ITEMS table should look like:
ITEMS
Item_Id Name Price
1 Coke 5.00
2 Pepsi 3.00
3 Sprite Zero 1.75
5 Mountain Dew 4.50
Once that's done, I also need to generate an XML formatted log file that contains the "new" record that was inserted into the table (ITEMS_LOG.XML):
ITEMS_LOG.XML
<?xml version="1.0" encoding="ISO-8859-1"?>
<Item>
<Id>5</Id>
<Name>Mountain Dew</Name>
<Price>4.50</Price>
</Item>
I have tried implementing this using SQLXMLBulkLoad, but unfortunately it does not provide the logging that I need, nor does it permit me to access any of the messages returned from SQL Server (i.e. what's been inserted/updated). Although I have an intermediate level of SQL expertise, I am fairly new to working with XML, especially in this context. Any help/guidance would be greatly appreciated!
You can use merge with output to a table variable and then query the table variable to build the log XML.
Put it in a stored procedure where you have the item XML as an in parameter and the log XML as an out parameter.
create procedure AddItemXML
#ItemsXML xml,
#ItemsLogXML xml out
as
declare #Changes table
(
Item_Id int,
Name nvarchar(20),
Price money,
Action nvarchar(10)
);
merge Items as T
using
(
select T.N.value('Id[1]', 'int') as Item_Id,
T.N.value('Name[1]', 'varchar(20)') as Name,
T.N.value('Price[1]', 'money') as Price
from #ItemsXML.nodes('/Item') T(N)
) as S
on T.Item_Id = S.Item_Id
when matched then
update set Name = S.Name, Price = S.Price
when not matched then
insert (Item_Id, Name, Price) values (S.Item_Id, S.Name, S.Price)
output inserted.Item_Id,
inserted.Name,
inserted.Price,
$action
into #Changes;
set #ItemsLogXML =
(
select Item_Id as ID,
Name,
Price
from #Changes
where Action = 'INSERT'
for xml path('Item'), type
);
Working sample on SE-Data
Hope this helps you, What I did was to create a stored procedure as below. Basically the stored procedure takes xml values and checks the flags which are passed from code and determines if it is insert or update:
DECLARE #xml xml
SET #xml = #xmlCredentials
SELECT
item.value('#Id', 'int') As ID,
item.value('#AgentID', 'int') As AgentID,
item.value('#Username', 'varchar (50)') As Username,
item.value('#Password', 'varchar (50)') As [Password],
item.value('#IsDirty', 'bit') As IsDirty,
item.value('#IsDeleted', 'bit') As IsDeleted
INTO #tmp
FROM #xml.nodes('Credentials/Credential') x(item)
BEGIN TRY
BEGIN TRAN
INSERT INTO Credentials (AgentID, Username, [Password])
SELECT
AgentID, Username, [Password]
FROM
#tmp
WHERE
ID = 0 AND IsDirty = 1
UPDATE c
SET c.[AgentID] = t.AgentID,
c.[Username] = t.Username,
c.[Password] = t.[Password]
FROM
[dbo].[Credentials] c
JOIN
#tmp t ON t.Id = c.ID
WHERE
t.IsDirty = 1 AND t.IsDeleted = 0
DELETE FROM [dbo].[Credentials]
FROM [dbo].[Credentials] c
JOIN #tmp t ON t.Id = c.ID
WHERE
t.IsDirty = 1 AND t.IsDeleted = 1
COMMIT TRAN
END TRY
BEGIN CATCH
IF ##TRANCOUNT > 0
ROLLBACK TRAN
DECLARE #errorMSG varchar(4000)
DECLARE #errorSeverity int
DECLARE #errorState int
SET #errorMSG = ERROR_MESSAGE()
SET #errorSeverity = ERROR_SEVERITY()
SET #errorState = ERROR_STATE()
RAISERROR (#errorMSG,
#errorSeverity, #errorState);
END CATCH
SELECT [ID], [AgentID], [Username], [Password]
FROM [dbo].[Credentials]
In code behind I have my xml and pass the xml as parameter to the stored procedure:
// read xml and assign it to string variable
string xml = readxml();
try
{
string command = "EXEC SaveCredentails '" + xml + "'";
}
catch(Exception e)
{
}
I would use a staging table to import the xml into a SQL Server table. Add an extra column to indicate the action (insert or update). Then use regular sql to perform upserts as usual. You can then use the staging table to generate the XML logging you need (reading the action column to determine if it was an insert or update).
I have a DataTable which is generated from .xls table.
I would like to store this DataTable into an existing table in SQL Server database.
I use SqlBulkCopy to store rows which have unique PK.
Problem is, I also have other rows which have same PK as SQL Server table but contain cells with different value compared to SQL Server table.
In short:
Let's say in my DataTable I have a row like this:
id(PK) | name | number
005 | abc | 123
006 | lge | 122
For my SQL server I have sth like this;
id(PK) | name | number
004 | cbs | 345
005 | lks | 122
Now you see the row 006 can be uploaded straight away into SQL Server using SqlBulkCopy. On the other hand the row 005 can't be inserted using it since SQL server table contains row with identical PK.
Now I tried to manually extract the row. Extract each single cell into an ArrayList then generate an UPDATE Table statement afterwards. However this method seems to be unfeasible as I have so many rows to process.
I am looking for a better method to achieve this goal.
Any help is appreciated.
Thank's
Use the below code:
C# Side code for reading data from DataTable and preparing the XML data:
DataTable dt = new DataTable();
StringBuilder sb = new StringBuilder();
sb.Append("<R>");
for (int i = 0; i < dt.Rows.Count; i++)
{
sb.Append("<C><ID>" + dt.Rows[0].ToString() + "</ID>");
sb.Append("<N>" + dt.Rows[1].ToString() + "</N>");
sb.Append("<I>" + dt.Rows[2].ToString() + "</I></C>");
}
sb.Append("</R>");
///pass XML string to DB side
///
//sb.ToString(); //here u get all data from data table as xml format
Database side Stored Procedure (you will need to update your table name):
CREATE PROCEDURE dbo.UpdateData
-- Add the parameters for the stored procedure here
#data XML
AS
BEGIN
SET NOCOUNT ON;
-- keep data into temp table
create table #tmp_data (id nchar(2),name varchar(20), number int)
DECLARE #XMLDocPointer INT
EXEC sp_xml_preparedocument #XMLDocPointer OUTPUT, #DATA
INSERT INTO #tmp_data(id,name,number)
SELECT ID,N,I
FROM OPENXML(#XMLDocPointer,'/R/C',2)
WITH(
ID nchar(30),
N VARCHAR(20),
I int
)
EXEC sp_xml_removedocument #XMLDocPointer
begin tran
-------------------INSERT not existing ones
INSERT INTO TABLE (id,name,number)
SELECT id,name,number
FROM #tmp_data
WHERE NOT EXISTS
(
SELECT 1
FROM TABLE
WHERE ID = #tmp_data.ID
)
--- update existing ones
UPDATE TABLE
SET name = #tmp_data.name, number = #tmp_data.number
FROM #tmp_data
WHERE #tmp_data.id = TABLE.id
commit tran
if(##error <> 0)
rollback tran
END