Develop Reference

converting xml of sharepoint list to dataset

its been more than one week and i still cant figure out what is the problem here. hope you could help me. i am successfully retrieving xml from share point server using SOAP web service, then i am converting the xml to dataset object, i am getting the dataset successfully but its "damaged" - there are few columns that has a missing values from the xml. here is the code for importing the xml using SOAP:
private void button2_Click(object sender, EventArgs e)
{
oportal.Lists list = new oportal.Lists();
list.Credentials = System.Net.CredentialCache.DefaultCredentials;
list.Url = "http://xxx/xxx/xxx/xxx/_vti_bin/Lists.asmx";
XmlDocument xmlDoc = new System.Xml.XmlDocument();
XmlNode ndQUery = xmlDoc.CreateNode(XmlNodeType.Element, "Query", "");
XmlNode ndViewFields = xmlDoc.CreateNode(XmlNodeType.Element, "ViewFields", "");
XmlNode ndQueryOptions = xmlDoc.CreateNode(XmlNodeType.Element, "QueryOptions", "");
ndQueryOptions.InnerXml =
"<IncludeMandatoryColumns>TRUE</IncludeMandatoryColumns>" +
"<DateInUtc>FALSE</DateInUtc>";
ndViewFields.InnerXml = #"<FieldRef Name='שם לקוח' />
<FieldRef Name='שם מתל'/>";
try
{
XmlNode ndListItems = list.GetListItems("{DD1CF626-62E1-4E36-BF2B-C7D08EA73674}",null, ndQUery, ndViewFields, "14000", ndQueryOptions, null);
System.Diagnostics.Debug.WriteLine(ndListItems.OuterXml);
dataGridView1.DataSource = ConverttYourXmlNodeToDataSet(ndListItems).Tables[1];
}
catch(System.Web.Services.Protocols.SoapException ex) {
MessageBox.Show(ex.Message + Environment.NewLine + ex.Detail.InnerText + Environment.NewLine + ex.StackTrace);
}
}
the xml i am getting looks ok, the column (field) names are in hebrew language but the xml shows them in HTML Entity (Hexadecimal) - maybe thats the root of the problem?
after i am getting the xml i am converting it into dataset with ConverttYourXmlNodeToDataSet() function here is the code:
public static DataSet ConverttYourXmlNodeToDataSet(XmlNode xmlnodeinput)
{
DataSet dataset = null;
if (xmlnodeinput != null)
{
XmlTextReader xtr = new XmlTextReader(xmlnodeinput.OuterXml, XmlNodeType.Element,null);
dataset = new DataSet();
dataset.ReadXml(xtr);
}
return dataset;
}
i am getting the dataset succefuly but like i mentioned its damaged because of the missing values, they exist in the xml but not in the dataset (the columns exist but not the values).
please have a look at this screen shoot:
iv`e surrounded with red color one of the columns that dont get their value from the XML. here is a screen shoot of the xml and the missing value that should be in the dataset surronded with red color:
also tryed to convert the xml to dataset like that but the results are the same:
public static DataSet read(XmlNode x) {
DataSet ds = new DataSet();
XmlReader r = new XmlNodeReader(x);
ds.ReadXml(r);
return ds;
}
hope someone can help me here. tnx.
UPDATE:
ok i have not solved it yet but i discovered few things that may lead to the solution:
i have noticed that all the columns that appears without values in the dataset are the columns that filled by the user in the website controls, and guess what? all the captions for those columns are in hebrew language , hence the columns that appears with values on the dataset are sharepint default columns, and their captions are in English, and they dont have HTML Entity (Hexadecimal)! name (look at the xml). So it makes me suspect that the problem is related to the HTML Entity (Hexadecimal) column names that related to the Hebrew captions... my assumption is that the dataset cant interpret this HTML Entity (Hexadecimal) encode. another clue is that the column name as its spelled in the dataset (for example look at the screen shoot of the datagridview above - column 4 from the left side (index 3)) is not interpreted right, the column name should be 'שם מתל' and thats all - as you can see (you don`t have to understand Hebrew for that) only half of this Hebrew string is there and concatenated to it part of the encoded HTML Entity (Hexadecimal).
i have noticed that when im sorting columns in share point website the requested url using the hexadecimal html entity of the column and not the hebrew name of the column:
http://xxx/xxx/xxx/xxx/Lists/1/view9.aspx?View={c2538b95-efae-453b-b536-aad6f98265ed}&SortField=_x05e9__x05dd__x0020__x05de__x05&SortDir=Desc
and i expected to see something like:
http://xxx/xxx/xxx/xxx/Lists/1/view9.aspx?View={c2538b95-efae-453b-b536-aad6f98265ed}&SortField=_'שם מתל'=Desc
so i made a change in my code in order to explicitly declare the column names in the encoded HTML Entity (Hexadecimal) and i did this (the original code is above):
ndViewFields.InnerXml = #"<FieldRef Name='_x05d0__x05d9__x05e9__x05d5__x05' />
<FieldRef Name='_x05e9__x05dd__x0020__x05de__x05'/>";
now the result i was getting in the dataset has changed! the change was that the columns i explicitly declared them moved to the first column indexes of the dataset but still there aren`t any values in those columns.
so, to summery all of that digging, here are my assumptions:
*. the problem is the interpreter between the xml and the dataset
*. the interpreter is defective because he cant interpret ecoded HTML Entity (Hexadecimal) properly
*. column captions written in HTML Entity (Hexadecimal) because their captions are in hebrew
*. solution can be or making columns captions to plain hebrew (in xml) or doing something that will make the interpreter between xml and dataset work properly (maybe using XmlParserContext class?? - tryed a little with no success or other class that can manipulate encoded xml text).

finally, after i have accomplish to solve this. the solution i found was super simple.
i have been searching and struggling some time for a solution, never found one and then this simple solution crossed my mind.
only one line of code was needed:
String s = xmlnodeinput.OuterXml.Replace("ows__x05e9__x05dd__x0020__x05de__x05",
"AccountManager");
just replacing the hex value and the dataset loads properly.
also i checked to see that there are no time proccessing issues(replacing string takes less then a second):
start reading 12000 rows: 26/03/2016 17:18:00
start replacing strings: 26/03/2016 17:18:04
load xml string to dataset: 26/03/2016 17:18:04
finish loading dataset: 26/03/2016 17:18:04
the complte converstion from xml to dataset function:
public static DataSet ConverttYourXmlNodeToDataSet(XmlNode xmlnodeinput)
{
//declaring data set object
DataSet dataset = null;
if (xmlnodeinput != null)
{
NameTable nt = new NameTable();
nt.Add("row");
XmlNamespaceManager nsmgr = new XmlNamespaceManager(nt);
XmlParserContext context = new XmlParserContext(nt, null, "heb",null, null, null, null, null, XmlSpace.None,Encoding.Unicode);
String s = xmlnodeinput.OuterXml.Replace("ows__x05e9__x05dd__x0020__x05de__x05", "AccountManager");
XmlTextReader xtr = new XmlTextReader(s, XmlNodeType.Element,context);
dataset = new DataSet();
dataset.ReadXml(xtr);
}
return dataset;
}

Encountered the same problem (Missing values after Xml load via/to DataSet).
Seems there is a problem with some chars (in my case "-").
Solution from jonathana works (replacing chars from attribute names before loading data to the dataset).
I will additionally provide a solution for .NET2 that changes all attribute names in the resulting SharePoint SOAP query XML to make sure the conversation into dataset won't result in an error (can be done more nicly with .NET3+ but im forced to .NET2 in my situation):
using System.Text.RegularExpressions;
using System.Web;
using System.Xml;
XmlDocument doc = new XmlDocument();
doc.LoadXml(spResXml.OuterXml);
System.Xml.XmlNamespaceManager nm = new System.Xml.XmlNamespaceManager(doc.NameTable);
nm.AddNamespace("rs", "urn:schemas-microsoft-com:rowset");
nm.AddNamespace("z", "#RowsetSchema");
nm.AddNamespace("rootNS", "http://schemas.microsoft.com/sharepoint/soap");
var zRows = doc.SelectNodes("//z:row", nm);
for (int i = 0; i < zRows.Count; i++)
{
XmlNode zRow = zRows[i];
List<XmlAttribute> attsList = new List<XmlAttribute>();
for (int j = 0; j < zRow.Attributes.Count; j++)
{ attsList.Add(zRow.Attributes[j]); }
foreach (XmlAttribute att in attsList)
{
string patchedAttName = att.Name;
patchedAttName = patchedAttName.Replace("_x", "%u");
patchedAttName = HttpUtility.UrlDecode(patchedAttName);
patchedAttName = Regex.Replace(patchedAttName,"[^A-Za-z0-9_]", "_", RegexOptions.None);
if (att.Name.Equals(patchedAttName))
{ continue; }
var newAtt = doc.CreateAttribute(att.Prefix, patchedAttName, att.NamespaceURI);
newAtt.Value = att.Value;
zRow.Attributes.Remove(att);
zRow.Attributes.Append(newAtt);
}
}
DataSet ds = new DataSet();
ds.ReadXml(new XmlNodeReader(doc));
DataTable t = ds.Tables[1];

vb.net to c#.net mongodb sort by date

I'm trying to get a small snippet of code from vb.net to c#.net using mongoDB driver working. It's almost working, but I can't seem to get the doc out of the mongoCursor in the same way I do from vb.net.
vb.net code:
Dim latest = updateCollection.FindAll.SetSortOrder(SortBy.Descending("date")).SetLimit(1)
Dim latestDoc As BsonDocument = latest(0)
c#.net code:
var updateCollection = database.GetCollection<BsonDocument>("updateInfo");
var sortby = SortBy.Descending("date");
var latest = (updateCollection.FindAll().SetSortOrder(sortby).SetLimit(1));
var latestDoc = latest.ToBsonDocument();
I've tried something like
var latestDoc = latest[0];
as well, but it hasn't worked. What am I missing?
Thank you!

You're getting a MongoCursor, not an item back from that that method, so you'll want to do something like this:
//add at the top of your code file
using System.Linq;
//...
//get the first result from the cursor like this:
var latestDoc = latest.FirstOrDefault();

XML to Object to Database

I need to store xml values into two different tables of a database.
Right now I'm using LINQ, but I can't get it to work as I want.
This is the XML:
<?xml version="1.0" encoding="utf-8" ?>
<nfeProc xmlns="http://www.tst.com">
<NFe xmlns="http://www.tst.com">
<infNFe Id="409" versao="2.00">
<ide>
<cUF>43</cUF>
<cNF>00011740</cNF>
</ide>
<emit>
<GGT>0483</GGT>
</emit>
<det nItem="1">
<prod>
<cProd>27</cProd>
<NCM>85437099</NCM>
</prod>
</det>
<det nItem="2">
<prod>
<cProd>30</cProd>
<NCM>85457099</NCM>
</prod>
</det>
</infNFe>
</NFe>
</nfeProc>
Right now I'm using the following code:
XDocument xml = XDocument.Load(#file);
XNamespace a = "http://www.portalfiscal.inf.br/nfe";
var NFe = from data in xml.Descendants(a + "nfeProc")
select new
{
cUF = (string)data.Elements(a + "NFe").Elements(a + "infNFe").Elements(a + "ide").Elements(a + "cUF").FirstOrDefault(),
nNF = (string)data.Elements(a + "NFe").Elements(a + "infNFe").Elements(a + "ide").Elements(a + "nNF").FirstOrDefault(),
};
var Prod = from Produto in xml.Descendants(a + "det")
select new
{
cProd = (string)Produto.Elements(a + "prod").Elements(a + "cProd").FirstOrDefault(),
NCM = (string)Produto.Elements(a + "prod").Elements(a + "NCM").FirstOrDefault(),
};
foreach (var nf in NFe)
{
}
foreach (var p in prod)
{
}
Then I use ADO.NET to send to the database. But I would like to do in only one foreach, so I could use the OUTPUT clause from SQL Server to relate the two collections(get ID from table 1(NFe) and INSERT into table 2(Prod)). How can I achieve that. I tried a lot of things with no success. I tried in one:
var NFe from data in xml.Descendants(a + "nfeProc")
select new {}...
But this way I only can get one "cProd" and not all of them.
Other approaches are welcome.
Thanks.
EDIT:
Thanks for the answers, but actually I got it to work simply changing my foreach.
foreach (var nf in NFe)
{
foreach (var p in prod)
{
}
}
This way I can use cmd.ExecuteScalar() to get OUTPUT.ID from my first foreach and use it in the next INSERT in my second foreach.

Another way to word your var Prod is
var Prod = xml.Descendants(a + "cProd")
.Select(x => new
{
cProd = (string)x,
NCM = (string)x.Parent.Element(a + "NCM")
});
Or if you absolutely know the NCM node follows the cProd node
NCM = (string)(XElement)x.NextNode

This has been around since .Net 1.1 I guess and no longer in fashion, yet you can:
Create a schema out of your xml file. Without any manual intervention (no element reorganizations, type or namespace customizations) this looks like:
Generate a strongly typed dataset out of your xsd. Or you can use the /c switch for xsd instead of /d and generate classes instead of a typed dataset. Either way you got yourself a data access layer for free. You can read xml into a dataset with one of the DataSet.ReadXml overloads and use that to persist to database or data binding.

FileHelpers - Column mapping

Quick question regarding filehelper library:
I have used file helper engine to read stream, do my validation and if the CSV file has not got a header we need to match/map it to my model: i.e
id, name, age, phone, sex,
but the CSV might not come in this format/order all the time and we need to match them using a drop down list for each column.
Is there any way I can do this?
Thannks,

The short answer, no. BUT you can create a dependent class dynamically:
Since you have the list of possible fields in your JSON file, I would recommend doing a basic System.IO ReadLine for the first data row, and then parse by your delimiter for the individual headers. i.e.:
string headerString;
var headers = new List<String>();
var file = new System.IO.StreamReader("C:\\myFile.txt");
headerString = file.ReadLine();
file.Close();
headers = headerString.Split(',').ToList();
now you have the list of strings for the first row to match against your JSON file. Then you can create your dependent class using System.Reflection.Emit (referenced link below)
typeBuilder.SetParent(typeof(MyFileHelperBaseClass));
// can place the property definitions in a for loop against your headers
foreach(string h in headers){
typeBuilder.DefineProperty("<header/col#>", ..., typeof(System.Int32), null);
}
stackoverflow article 14724822: How Can I add properties to a class on runtime in C#?
File Helpers gets a little finicky at times, so it will take some tweaking.
Hope this helps

You can use File.ReadLines(#"C:\myfile.txt").First() to read the first line and get the headers.
Then you can just use a FileHelpers CodeBuilder to build your runtime class. From the example for a delimited csv file:
DelimitedClassBuilder cb = new DelimitedClassBuilder("Customers", ",");
cb.IgnoreFirstLines = 1;
cb.IgnoreEmptyLines = true;
cb.AddField("BirthDate", typeof(DateTime));
cb.LastField.TrimMode = TrimMode.Both;
cb.LastField.FieldNullValue = DateTime.Today;
cb.AddField("Name", typeof(string));
cb.LastField.FieldQuoted = true;
cb.LastField.QuoteChar = '"';
cb.AddField("Age", typeof(int));
engine = new FileHelperEngine(cb.CreateRecordClass());
DataTable dt = engine.ReadFileAsDT("testCustomers.txt");
Then you can traverse the resulting data table.

Linq2Sql - Converting Code - Visual Studio 2012 - Unit Tests

I have been trying to convert the code below. I have attempted it and I will post the attempt beneath the code below. I am confused when it comes to
Code That I have to convert:
[TestMethod]
public void SRSTransactiondataCleanTest()
{
using (SqlConnection sqlConn = new
SqlConnection(#"Data Source=localhost;Initial Catalog=CorporateDWTest;Integrated Security=SSPI;"))
{
StringBuilder str = new StringBuilder();
str.Append(#"SELECT * FROM [dbo].[SRS_Ticket_Transaction_Stage_Cleaned]");
sqlConn.Open();
SqlDataAdapter da = new SqlDataAdapter(str.ToString(), sqlConn);
DataTable dResults = new DataTable();
DataTable dAudit = new DataTable();
da.Fill(dResults);
da.SelectCommand.CommandText = #"SELECT [Table_Name],[Package_Name],[Execution_Start_Time],[Execution_End_Time],[Processing_Successful], Audit_Key," +
#"[Table_Initial_Row_Count],[Table_Final_Row_Count] FROM [SRS_Dim_Audit] WHERE Package_Name LIKE 'SAPSRSDataToStageClean'";
da.Fill(dAudit);
var r = dAudit.Rows[0][2].ToString();
Assert.AreEqual(1, dAudit.Rows.Count);
Assert.AreEqual("SRS_Ticket_Transaction_Stage_Cleaned", dAudit.Rows[0][0].ToString());
Assert.AreEqual("SAPSRSDataToStageClean", dAudit.Rows[0][1].ToString());
Assert.AreEqual(bool.TrueString, dAudit.Rows[0][4].ToString());
Assert.IsTrue(int.Parse(dAudit.Rows[0][7].ToString()) > 10);
Assert.IsTrue(dResults.Rows.Count > 50);
Assert.AreEqual(int.Parse(dAudit.Rows[0][7].ToString()), dResults.Rows.Count);
Assert.AreEqual(int.Parse(dAudit.Rows[0][5].ToString()), int.Parse(dResults.Rows[0][2].ToString()))
sqlConn.Close();
}
}
What I have so far:
[TestMethod]
public void SRSTransactiondataCleanTest()
{
using (var context = new CorporateDWTestEntities4())
{
var stageCleaned = context.SRS_Ticket_Transaction_Stage_Cleaned;
var auditRecords = context.SRS_Dim_Audit.Where(s => s.Package_Name == "SAPSRSDataToStageClean");
var auditRecord = auditRecords.FirstOrDefault();
Assert.AreEqual(1, stageCleaned.Count());
Assert.AreEqual(auditRecord.Table_Final_Row_Count, stageCleaned.Count());
Assert.AreEqual(52, auditRecords.Count());
Assert.AreEqual("SRS_Ticket_Transaction_Stage_Cleaned", auditRecord.Table_Name);
Assert.AreEqual("SAPSRSDataToStageClean", auditRecord.Package_Name);
Assert.AreEqual(true, auditRecord.Processing_Successful);
Assert.IsTrue(auditRecord.Table_Final_Row_Count > 10);
}
}
I am fairly new at this code conversion stuff.. Am I on the right track? When I run this I recieve a message that states:
"Assert.AreEqual failed. Expected <1>. Actual <9461>."
When I change it to 9461, the entire test passes. Why is that? Am I using the wrong variable? I would just like to know if the code that I wrote is actually mirroring the code that I was to convert. I know with programming syntax must be meticulous. I was wondering if someone out there could review this and modify IF necessary.
Ps. the line:
Assert.AreEqual(int.Parse(dAudit.Rows[0][5].ToString()), int.Parse(dResults.Rows[0][2].ToString()));
confused the hell out of me, What is that even asking of me?

Your first test should be
Assert.AreEqual(1, auditRecords.Count());
rather than
Assert.AreEqual(1, stageCleaned.Count());
You also have a test
Assert.AreEqual(52, auditRecords.Count());
I can't see where you get 52 free. If this is supposed to be the equivalent of
Assert.IsTrue(dResults.Rows.Count > 50);
then it should be
Assert.IsTrue(stageCleaned.Count() > 50);
Not sure why you are confused by the line
Assert.AreEqual(int.Parse(dAudit.Rows[0][5].ToString()), int.Parse(dResults.Rows[0][2].ToString()));
You must be very close as you have already worked out what all the other dAudit.Rows[0][...] values are, so presumably you know that int.Parse(dAudit.Rows[0][5].ToString()) represents auditRecord.Audit_Key (assuming this is defined as an int). Similarly dResults.Rows[0][2] will be the the third column of the first row from the SRS_Ticket_Transaction_Stage_Cleaned table.
So presumably this represents something like
var firstStageRecord = stageCleaned.First();
Assert.IsTrue(auditRecord.Audit_Key , firstStageRecord.xxx);
where xxx is the name of the third column.
There are a few other issues to point out.
If you have several calls to stageCleaned.Count(), it will issue a sql command each time is is called.
The original code was hard-coding the sqlConnection, whereas your new code is either using the default or picking up a connection from a configuration file, so you need to check to ensure you are talking to the correct database.