how can i map xml file(s) of different formats to a table in my database say for example i have a table called Person
FirstName nvarchar(50),
LastName nvarchar(50),
Address nvarchar(500)
i have to map xml files submitted by users to the fields of Person, the xml files can be in different formats say for example one user submits in the following format
<Person>
<Names>
<FirstName>john</FirstName>
<LastName>smith</LastName>
</Names>
<Names>
<FirstName>john</FirstName>
<LastName>smith</LastName>
</Names>
</Person>
and another user submits it in the following format
<Person>
<PersonalInfo>
<Names>
<FirstName>john</FirstName>
<LastName>smith</LastName>
</Names>
.
.
.
<PersonalInfo>
</Person>
how can i device a solution that will handle different cases (different xml formats), so that if a user submits a xml file in whatever format i just have to tell the format to my aplication and its values will automatically gets mapped to the said table... im using visual studio 2010 as an IDE and C#.NET, i hope have narrated the problem well enough...
please advice...
Since I don't how dynamic this should be (meaning how often you will change the mappings) or how many different XMLs do u have I will suggest you to map the table(s) to a common class using your preferred ORM and create class instances using LinqToXml (which is very easy). One method or class per XML structure.
There is a sample here for the LinqToXml -> http://www.switchonthecode.com/tutorials/introduction-to-linq-simple-xml-parsing
Also, if you don´t want to use it and are using .NET 4 I created a simple library to produce dynamic objects from a XML string. You can take a look at it here -> https://github.com/tucaz/XmlToObjectParser
you can identify the schema you are acquiring and use appropriate XSLT File to convert it to the latest version.
Note: you should make something generic to identify the schema (e.g. )
(I Like 1. better) create an Software Adapter interface that changes the mappings (Like add-ins work)
Related
I have approximately 100.000 xml files that each contain 1 row of data (but it has 2 parents).
What is the best way to read these XML files, and write the data to a SQL Server table ?
This is my XML:
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<recording>
<dataformat>audio</dataformat>
<starttime>2015-02-09 08:57:08:000 +0200</starttime>
<endtime>2015-02-09 08:58:48:000 +0200</endtime>
<nostart>false</nostart>
<noend>false</noend>
<recordingtype>stnbulk</recordingtype>
<recordingline></recordingline>
<servicename>005379763634</servicename>
<servicenumber></servicenumber>
<deliberatebreak>0</deliberatebreak>
<calldirection>Outgoing</calldirection>
<filename>890001000000003.wav</filename>
<otherinum>0</otherinum>
<callparty>1</callparty>
<recordingowners>
<recordingowner>131</recordingowner>
</recordingowners>
<parties>
<party id="1">
<number>131</number>
<name>Santral131</name>
<pstarttime>2015-02-09 08:58:48:912 +0200</pstarttime>
<pendtime>2015-02-09 08:58:48:912 +0200</pendtime>
</party>
<party id="2">
<number>005379763634</number>
<pstarttime>2015-02-09 08:58:48:912 +0200</pstarttime>
<pendtime>2015-02-09 08:58:48:912 +0200</pendtime>
</party>
</parties>
<guids>
</guids>
</recording>
I tried to read this data by using a XmlReader to a Dataset for insert to SQL Server database, but it didn't work.
Thanks in advance for your support.
Is this a one-time-action? Are the file names easy to list? Does the file's name carry any information? Are all 100k files identically structured?
Please provide the target structure of your tables.
On the first sight, the easiest was to write a stored procedure, accepting a paramter of type XML.
Your C# application uses XmlDocument to read the XML directly from file path and then passes the OuterXml to the SP. This SP will read the XML and insert its values into the target tables. Pass this just as string and do not pass the declaration <?xml ...?>. C# uses unicode implicitly and so does T-SQL-XML...
The following code will give you templates for each type of element how to read it. It should be easy to use something like this within your SP (where #xml is the passed in parameter).
Attention: However this XML was generated, the contained date-time values do not fullfill the appropriate format (ISO8601). This needs some tricksing...
Try this within a T-SQL SP:
DECLARE #xml XML=
'<recording>
<dataformat>audio</dataformat>
<starttime>2015-02-09 08:57:08:000 +0200</starttime>
<endtime>2015-02-09 08:58:48:000 +0200</endtime>
<nostart>false</nostart>
<noend>false</noend>
<recordingtype>stnbulk</recordingtype>
<recordingline></recordingline>
<servicename>005379763634</servicename>
<servicenumber></servicenumber>
<deliberatebreak>0</deliberatebreak>
<calldirection>Outgoing</calldirection>
<filename>890001000000003.wav</filename>
<otherinum>0</otherinum>
<callparty>1</callparty>
<recordingowners>
<recordingowner>131</recordingowner>
</recordingowners>
<parties>
<party id="1">
<number>131</number>
<name>Santral131</name>
<pstarttime>2015-02-09 08:58:48:912 +0200</pstarttime>
<pendtime>2015-02-09 08:58:48:912 +0200</pendtime>
</party>
<party id="2">
<number>005379763634</number>
<pstarttime>2015-02-09 08:58:48:912 +0200</pstarttime>
<pendtime>2015-02-09 08:58:48:912 +0200</pendtime>
</party>
</parties>
<guids>
</guids>
</recording>';
--This is the actual query
--INSERT INTO YourTargetTable(col1,col2,col3...)
SELECT r.value('(dataformat/text())[1]','nvarchar(max)') AS dataformat
,CONVERT(DATETIMEOFFSET,STUFF(STUFF(STUFF(STUFF(r.value('(starttime/text())[1]','nvarchar(max)'),11,1,'T'),20,1,'.'),24,1,''),27,0,':'),127) AS starttime
,r.value('(nostart/text())[1]','bit') AS nostart
--more columns
,r.value('(recordingowners/recordingowner/text())[1]','bit') AS recordingowner --if there might be more than one, this would need one more call to .nodes()
,p.value('#id','int') AS party_id
,p.value('(number/text())[1]','nvarchar(max)') AS party_number
--more columns
FROM #xml.nodes('/recording') AS A(r)
OUTER APPLY A.r.nodes('parties/party') AS B(p)
The (partial) result
audio 2015-02-09 08:57:08.0000000 +02:00 0 1 1 131
audio 2015-02-09 08:57:08.0000000 +02:00 0 1 2 005379763634
Hint: If you don't need the time zone, you might use this trick to implicitly add the timezone to the datetime...
DECLARE #dtZ DATETIMEOFFSET=CONVERT(DATETIMEOFFSET,'2015-02-09T08:58:48.912+02:00',127);
SELECT (SELECT #dtZ FOR XML PATH(''),TYPE).value('.','datetime');
The result has got the hour 6 now...
2015-02-09 06:58:48.913
Some background: The ISO8601 format would look like this
2015-02-09T08:58:48.9120000+02:00
If the strings within the XML were correctly formatted, the reading was done just as easy as any other value. My code uses multiple STUFF() calls to correct the format to make it convertible...
I have an XML file in which I store data about a list of persons and another one in which I store a list of objects like this.
people.xml
<People>
<Person>
<Name>itsName</Name>
<Age> itsAge </Age>
<RecentAcquisitions>
<Acquisition>
<name>Apple</name>
<quantity>5</quantity>
</Acquisition>
</RecentAcquisitions>
</Person>
</People>
objects.xml
<Objects>
<Object>
<Name>Apple</Name>
<Description>Fresh Apple</Description>
<Price>10</Price>
<etc>..lots of attributes..</etc>
</Object>
</Objects>
What is the most efficient way of extracting information from objects.xml based on the person Acquisition List at the runtime? (in example the person should have 5 objects of type "Apple").
Momentarily I use a solution which consists of storing each object from objects.xml in a list and when I'm loading a person I search for the respective object based on Acquisition->Name and add it in the person.AquisitionList;
Is there another way of doing this?
Maybe I misunderstood the XML role but it feels wrong to store the information from an XML file in a list or array at runtime.
to my knowledge, using the runtime memory instead of constant read-write operations is the best way to do it / what you're doing is the right way.
XML can be seen as 2 things:
1 - A way to store information, much like a database, until it needs to be retrieved for processing at runtime
this is what you are doing now... you store the objects list on disk using XML, and then you retrieve it for processing/load it into memory at runtime.
2 - A standardized way of passing information around, regardless of technology.
XML can be read in a multitude of languages and any language that can read a string can technically read and extract the data from an XML document.
I need to generate a report in csv format. Data is retrieved as List<T> and only certain columns needs to be displayed in the report. I am thinking of doing it by storing its schema in an XML document under App_Data folder and use LINQ to XML to retrieve the field names and create the report.
Sample XML:
<report>
<fields>
<field headercaption='Customer Name'>CName</field>
<field headercaption='Address'>CAddress</field>
...
</fields>
</report>
Is it advisable to completely depend on XML file this way or do I need to do it through coding.
Edit
Fields are properties of List<T> which needs to be populated in the CSV files. Header caption is the name of the column for the field in the report.
Sample Report
Customer Name, Address,
ALFKI , 31 Independence Ave., Washington
If you just want to add/remove columns at run time in this one report, it would be even easier to use a plain text file. Store column information in a format you can easily parse ( something like CName|Customer Name). When you need to generate CSV, read all lines, split by | (or whatever separator you choose) and go from there. You will probably also want to add a comment or documentation note somewhere about the format (in case you or someone else need to come back and modify it in a year or more).
XML might become useful if you:
plan to put several reports into one file, and maybe give them names
provide some way for other people to submit their own reports
create some authoring tool to allow people to customize reports
localize column names in CSV. You could define something like
.
<report>
<fields>
<field name="CName">
<header language="en-us">Customer Name</header>
<header language="es-es">Nombre de cliente</header>
...
</field>
...
</fields>
</report>
want to experiment with XML libraries
In those cases XML may be useful because you could define an XSD and easily validate that submitted reports are valid.
I am trying to dump the contents of an xml file in to a sql table (relational format).
In the past, I have used dataset.readxml but have had some issues with it while handling many to many relationships.
So I took an approach as follows, create a unique list of xpath expression for all the elements and attributes in the xml file and then using these expressions create a sql db structure to persist this data.
I wanted to create this feature as a dynamic implementation to handle any xml.
Let's say I have an xml file and the unique Xpath are as follows
/movie/MovieId[1]
/movie/MovieName[1]
/movie/cast[1]
/movie/cast[1]/name[1]
/movie/cast[1]/name[2]
/movie/cast[1]/name[3]/#rolename
/movie/description[1]
/movie/directors[1]
/movie/directors[1]/name[1]
/movie/directors[1]/name[1]
/movie/directors[1]/name[2]
/movie/directors[1]/name[3]
/movie/releasedate[1]
/movie/releasedate[1]/#reldatetestatt
/movie/runtime[1]
I would like the table structure to be created as follows:
TableName: Movie
Columns: MovieId, MovieName, description, releasedate, reldatetestatt, runtime
TableName: Cast
Columns: MovieId, name, rolename
TableName: Directors
Columns: MovieId, name
Sample Xml for your reference:
var xml = #"<movies>
<movie>
<MovieId>277345</MovieId>
<MovieName>The Life and Passion of Jesus Christ</MovieName>
<description>The Gospels of the New Testament.</description>
<runtime>44</runtime>
<releasedate reldatetestatt='testattribute'>3/26/1904 12:00:00 AM</releasedate>
<directors>
<name>Lucien Nonguet</name>
<name>Ferdinand Zecca</name>
<name>Ferdinand Zecca 2</name>
</directors>
<cast>
<name rolename='test'>Madame Moreau</name>
<name>Monsieur Moreau</name>
</cast>
</movie>
</movies>";
You could try Talend Open Studio (aka Jaster ETL). It will happily take xml and fill a database. You can do it from the UI or export the "job" as independent java code.
I have a nested XML document and I am looking at the plausability of using DataSets to parse it.
<?xml version="1.0" standalone="yes"?>
<Workbench>
<Overrides>
<Override name='firstoverride' value='overridevalue'/>
</Overrides>
<DataSets>
<BASIC>
<MEMBNO>1</MEMBNO>
<PERSONNO>0</PERSONNO>
</BASIC>
</DataSets>
</Workbench>
What i want to be able to do is essentially access the contents of the Overrides and DataSets as if there were an actual Dataset.
So to validate I check the root element is workbench.
Then I check to see if there are any overrides, I then want to be able to iterate around the Override Items.
Following that, and this is the hard part I want to support abitary but well form XML that will be inserted into a database but the parsing code can make so assumptions about the data as I want it to be generic.
I can do this if I make the DataSets the root element and iterate around it but it doesn't seem to work if nested?
hlep!