Develop Reference

C# hanging code when XML from URL is interrupted

I asked this question yesterday.
Essentially, I'm trying to parse an XML from a URL but my code hangs forever if the connection is lost when attempting to read the XML.
I am still having the same problem, however I changed the code in a way I thought would prevent the program from freezing if the connection to the URL was interrupted. Could someone please explain why my solution didn't work and how I can fix it? Thanks!
Here are the two functions I am using. CanReach just checks the connection to make sure the URL is there, and GetTags gets all the parent tags of the XML file. I want it to break if the connection is interrupted. I tried to do this by loading the xml file instead of parsing it right from the URL and using try and catch to catch the error. xmlLocation is the URL.
public static bool CanReach(string xmlLocation)
{
WebRequest request = WebRequest.Create(xmlLocation);
request.Timeout = 1000;
try
{
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
response.Dispose();
request.Abort();
return true;
}
catch (System.Net.WebException)
{
request.Abort();
return false;
}
}
public static List<string> GetTopTags(string xmlLocation)
{
bool canBeReached = CanReach(xmlLocation);
if (canBeReached)
{
try
{
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.Load(xmlLocation);
XmlReader reader = new XmlNodeReader(xmlDoc);
List<string> dataList = new List<string>();
while (reader.Read())
{
switch (reader.NodeType)
{
case XmlNodeType.Text:
dataList.Add(reader.Name);
break;
}
}
reader.Dispose();
return topTags;
}
catch
{
return null;
}
}
else
{
return null;
}
}

We can start of another thread that is reading the XML file and the current method can continuously check whether connection is alive or not. If the we can not reach the URL anymore, we can cancel the operation and return null. If the read operation is finished, isFinished flag is set and we can return returnValue.
Try this:
public static List<string> GetTopTags(string xmlLocation)
{
bool canBeReached = CanReach(xmlLocation);
if (!canBeReached)
return null;
List<string> returnValue = null;
CancellationTokenSource cts = new CancellationTokenSource();
bool isFinished = false;
Task.Factory.StartNew(() =>
{
try
{
var xmlDoc = new XmlDocument();
xmlDoc.Load(xmlLocation);
using var reader = new XmlNodeReader(xmlDoc);
List<string> dataList = new List<string>();
while (reader.Read())
{
switch (reader.NodeType)
{
case XmlNodeType.Text:
dataList.Add(reader.Name);
break;
}
}
if (reader.ReadState == ReadState.Error)
returnValue = null;
else
returnValue = topTags;
}
catch
{
returnValue = null;
}
isFinished = true;
}, cts.Token, TaskCreationOptions.LongRunning, TaskScheduler.Default);
while (!isFinished)
if (!CanReach(xmlLocation))
cts.Cancel();
return returnValue;
}

I'm not wild about that XmlDocument.Load() and passing it a URL. You're handing over too much control and it's making it hard for you to debug. I would separate out the networking and the XML reading. With the networking separated out, you eliminate the need for your CanReach() function. Anything network related needs to be ran in a separate thread. Based on your source, something like the following is what I might start with.
public static async Task<List<string>> GetTopTags(string xmlLocation)
{
string xmlText = null;
try
{
// You are free to use WebRequest here, I've used WebClient for simplicity.
using (var webClient = new WebClient())
{
xmlText = await webClient.DownloadStringTaskAsync(xmlLocation);
}
}
catch (Exception)
{
// Handle network related issues.
}
if (string.IsNullOrWhiteSpace(xmlText))
{
// We weren't able to download the XML, or the downloaded XML is not valid,
// "CanReach()" is false.
return null;
}
// We downloaded the XML successfully if you get here, now just read it.
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.Load(new StringReader(xmlText));
using (XmlReader reader = new XmlNodeReader(xmlDoc))
{
List<string> dataList = new List<string>();
while (reader.Read())
{
switch (reader.NodeType)
{
case XmlNodeType.Text:
dataList.Add(reader.Name);
break;
}
}
return dataList;
}
}

Get longitude and latitude from API

I have searched on Google and here on Stack Overflow, but due to my limited expertise in C# and SSIS I have some troubles getting through.
My steps in this SSIS package is as follows:
I have a SQL task connected to a for loop which only specifies how many times I should iterate in Forloop container which is =1
Inside Forloop container have a dataflow task
In dataflow task, I use an OLEDB source to get addresses from a SQL table
I connect those OLEDB source to a Script task from which the following script "Should" return my XML answer from which I want to return Latitude and longitude. It does not return the answer that I want.
Example address is - Beihinger Strasse 160 DE-71726 BENNINGEN GERMANY
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
string url = "http://dev.virtualearth.net/REST/v1/Locations?addressLine={0}&output=xml&key={1}";
var Address = Row.Address;
var wGet = WebRequest.Create(String.Format(url, Address, Variables.APIkey));
wGet.Method = WebRequestMethods.Http.Get;
var response = wGet.GetResponse();
var status = ((HttpWebResponse)response).StatusDescription;
Console.WriteLine("Input: " + Address);
if (status == "OK")
{
StreamReader reader = new StreamReader(response.GetResponseStream());
string responseFromServer = reader.ReadToEnd();
reader.Close();
var xmlDoc = new XmlDocument();
xmlDoc.LoadXml(responseFromServer);
var nsmgr = new XmlNamespaceManager(xmlDoc.NameTable);
nsmgr.AddNamespace("Point", "http://schemas.microsoft.com/search/local/ws/rest/v1");
var latNode = xmlDoc.DocumentElement.SelectSingleNode("/ResourceSets/ResourceSet/Resources/Location/Point/Latitude", nsmgr);
var lngNode = xmlDoc.DocumentElement.SelectSingleNode("/ResourceSets/ResourceSet/Resources/Location/Point/Longitude", nsmgr);
if (latNode != null && lngNode != null)
{
var numberFormatInfo = new NumberFormatInfo { NumberDecimalSeparator = "." };
Row.Latitude = decimal.Parse(latNode.InnerText, numberFormatInfo);
Row.Longitude = decimal.Parse(lngNode.InnerText, numberFormatInfo);
Row.FullResponse = responseFromServer.ToString();
}
else
{
//Set to null
Row.Latitude_IsNull = true;
Row.Longitude_IsNull = true;
Row.FullResponse = responseFromServer.ToString();
}
}
else
{
//Set to null
Row.Latitude_IsNull = true;
Row.Longitude_IsNull = true;
Row.FullResponse_IsNull = true;
}
response.Close();
}
}
The response I get is:
<?xml version="1.0" encoding="utf-8"?><Response xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/search/local/ws/rest/v1"><Copyright>Copyright © 2020 Microsoft and its suppliers. All rights reserved. This API cannot be accessed and the content and any results may not be used, reproduced or transmitted in any manner without express written permission from Microsoft Corporation.</Copyright><BrandLogoUri>http://dev.virtualearth.net/Branding/logo_powered_by.png</BrandLogoUri><StatusCode>200</StatusCode><StatusDescription>OK</StatusDescription><AuthenticationResultCode>ValidCredentials</AuthenticationResultCode><TraceId>b9a94b411dc246ea82d161dae5e7a0c3|DU00000D73|0.0.0.1|Ref A: ACDD23537F6649F6B2F66333BB6D6D01 Ref B: DB3EDGE0918 Ref C: 2020-02-12T11:25:05Z</TraceId><ResourceSets><ResourceSet><EstimatedTotal>0</EstimatedTotal><Resources /></ResourceSet></ResourceSets></Response>
I want a correct call as I get when using URL so I can get Longitude and Latitude- http://dev.virtualearth.net/REST/v1/Locations?addressLine=Beihinger%20Strasse%20160%20DE-71726%20BENNINGEN%20GERMANY&output=xml&key={APIkey}
Any help would be greatly appreciated to what I am missing?

How to read specific values from response

This is the page I'm using for documentation https://lichess.org/api#operation/player
I want to get player usernamename, rating, and title.
My code.
public class Player {
public string username;
public double rating;
public string title;
}
HttpClient client = new HttpClient();
client.BaseAddress = new Uri("https://lichess.org/");
HttpResponseMessage response = client.GetAsync("player/top/200/bullet").Result;
Here I'm getting response, But I have no clue how to take only properties that I need and store it in a list of players.

After a discussion with you on this problem, it was found that the response that you are receiving is a HTML string, therefore you need to deal with this case differently. I was playing around with the HTML that you have posted in the comments and I was able to parse the string with HTML Agility Pack which can be found here. You can also download this pack from the Nuget Package Manager in Visual Studio.
I am giving you a very basic example of the parsing process that I tried out:
public class ProcessHtml()
{
List<Player> playersList = new List<Player>();
//Get your HTML loaded from a URL. Giving me SSL exceptions so took a different route
//var url = "https://lichess.org/player/top/200/bullet";
//var web = new HtmlWeb();
//var doc = web.Load(url);
//Get your HTML loaded as a file in my case
var doc = new HtmlDocument();
doc.Load("C:\\Users\\Rahul\\Downloads\\CkBsZtvf.html", Encoding.UTF8);
foreach (HtmlNode table in doc.DocumentNode.SelectNodes("//tbody"))
{
foreach (HtmlNode row in table.SelectNodes("tr"))
{
int i = 0;
Player player = new Player();
//Since there are 4 rounds per tr, hence get only what is required based on loop condition
foreach (HtmlNode cell in row.SelectNodes("th|td"))
{
if(i==1)
{
player.username = cell.InnerText;
}
if(i==2)
{
player.rating = Convert.ToDouble(cell.InnerText);
}
if(i==3)
{
player.title = cell.InnerText;
}
i++;
}
playersList.Add(player);
}
}
var finalplayerListCopy = playersList;
}
public class Player
{
public string username;
public double rating;
public string title;
}
After running this, your finalplayerListCopy has a count of 200 and an example data would look like:
Obviously, you would have to play with the data and tailor it as per your need. I hope this helps you out.
Cheers!

from what Ive read from the documentation
async Task<Player> getPlayerAsync(string path)
{
Player player= null;
HttpResponseMessage response = await client.GetAsync(path);
if (response.IsSuccessStatusCode)
{
player = await response.Content.ReadAsAsync<Player>();
}
return player;
}
getPlayerAsync("https://lichess.org/player/top/200/bullet");

I used SvcUtil to create classes from WSDL. What do I deserialize into?

I used svcUtil.exe to create classes from https://gw.sam.gov/SAMWS/1.0/Entity?wsdl
but I cannot for the life of me find what to deserialize the result into? When I created my own classes the root is envelope but it isn't even in the new classes.
I can paste the classes but it is really long? Is there a general answer to this?
I will paste the classes upon request.
Thanks in advance...
The classes are over 10x too long to post.
Adding code for the pull:
static void Main(string[] args)
{
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
var _url = "https://gw.sam.gov/SAMWS/1.0/Entity";
//Run date is a specific date if provided otherwise use yesterday
DateTime startDateTime = DateTime.Now.AddDays(-1).Date;
for (int hr = 0; hr < 24; hr++)
{
XMLclassRequest xmlSoap = new XMLclassRequest();
string soap = xmlSoap.BuildSOAPrequest(startDateTime.AddHours(hr));
//string soap2 = xmlSoap.BuildSOAPrequest2(startDateTime.AddHours(hr));
string response = null; //This is the original pull with FAR and DFAR Responses
//string response2 = null; //This is FAR and DFAR
using (MyWebClient client = new MyWebClient())
{
client.Headers.Add("Content-Type", "text/xml;charset=utf-8");
client.Headers.Add("SOAPAction", "\"http://tempuri.org/IMyService/MyOperation\"");
try
{
response = client.UploadString(_url, soap);
//response2 = client.UploadString(_url, soap2);
}
catch
{ } //This will skip the hour attempted and move to next. The error I have been receiving is no data which is differently formatted XML that causes the error
}
//File.WriteAllText(#"D:\temp\bigpull.xml", response);
MemoryStream stream = null;
if (response != null)
{
byte[] byteArray = Encoding.Unicode.GetBytes(response);
stream = new MemoryStream(byteArray);
}
getEntities results;
XmlSerializer serializer = new XmlSerializer(typeof(getEntities));
try
{ results = (getEntities)serializer.Deserialize(stream); }
catch
{ } //This will skip the hour attempted and move to next. The error I have been receiving is no data which is differently formatted XML that causes the error
stream.Close();
string str;
}
The response is coming back. I just can't get it into an object to use.

I couldn't get this loaded into an object but ended up loading it to an XDocument and parsed that:
var _url = "https://gw.sam.gov/SAMWS/1.0/Entity";
//Run date is a specific date if provided otherwise use yesterday
DateTime startDateTime = DateTime.Now.AddDays(-1).Date;
for (int hr = 0; hr < 24; hr++)
{
XMLclassRequest xmlSoap = new XMLclassRequest();
string soap = xmlSoap.BuildSOAPrequest(startDateTime.AddHours(hr));
//string soap2 = xmlSoap.BuildSOAPrequest2(startDateTime.AddHours(hr));
string response = null; //This is the original pull with FAR and DFAR Responses
//string response2 = null; //This is FAR and DFAR
using (MyWebClient client = new MyWebClient())
{
client.Headers.Add("Content-Type", "text/xml;charset=utf-8");
client.Headers.Add("SOAPAction", "\"http://tempuri.org/IMyService/MyOperation\"");
try
{
response = client.UploadString(_url, soap);
//response2 = client.UploadString(_url, soap2);
}
catch
{ } //This will skip the hour attempted and move to next. The error I have been receiving is no data which is differently formatted XML that causes the error
}
//File.WriteAllText(#"D:\temp\far20190604.xml", response);
XDocument xdoc = XDocument.Parse(response);
var entities = from e in xdoc.Descendants("entity") select e;
foreach (var e in entities)
{
string DUNS = e.Descendants("DUNS").FirstOrDefault().Value;
var provisions = from p in e.Descendants("provision") select p;
foreach (var p in provisions)
{
string ParentID = p.Descendants("id").FirstOrDefault().Value;
var answers = from a in p.Descendants("answer") select a;
foreach (var a in answers)
{
var section = a.Descendants("section").Count()>0? a.Descendants("section").FirstOrDefault().Value : "";
var answerText = a.Descendants("answerText").Count() > 0 ? a.Descendants("answerText").FirstOrDefault().Value : "";
Console.WriteLine(DUNS + " " + ParentID + " " + section + " " + answerText);
}
}
}

Load text from web during xml parsing in Windows 8 Async

I have a unique issue, I want to get the name of an application from it's AppID while I convert an XML file into objects. This is the code I'm using at present:
if (xdoc.Element("Application") != null)
{
var data = from query in xdoc.Descendants("AppID")
select new APP
{
AppID = query.Value,
AppName = GetName(query.Value).ToString(),
};
itemGridView.DataContext = data;
}
This is the code I'm using to convert the GUID into an app name using Microsoft's Store API. I can confirm that it does return the app name. I'm just unsure how I can get this to display.
private async Task<string> GetName(string guid)
{
try
{
string link = "https://services.apps.microsoft.com/browse/6.2.9200-1/615/en-NZ_en-NZ/c/NZ/cp/10005001/Apps/{0}";
string url = string.Format(link, guid);
var httpClient = new HttpClient();
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, url);
var response = await httpClient.SendAsync(httpRequestMessage);
var xmlString = await response.Content.ReadAsStringAsync();
XmlDocument NameXML = new XmlDocument();
NameXML = await XmlDocument.LoadFromUriAsync(new Uri(url));
string sAppName = NameXML.GetElementsByTagName("T")[0].ChildNodes[0].NodeValue.ToString();
return sAppName;
}
catch(Exception)
{
return guid;
}
}
I think my problem is with the async / await tasks. I've just been exposed to it now... how would I load up the App Name alongside the AppID when I parse the xml file?
The output that's being displayed when I run the app is "System.Threading.Tasks.Task[System.String]" (The objects load and the links and everything works fine, its just that the above is displayed instead of the app name).
I've been debugging using breakpoints, it appears that the GetName method only seems to be triggered later on, I'm not sure however.

Try to change this line :
AppName = GetName(query.Value).ToString(),
To this :
AppName = await GetName(query.Value),
GetName will return Task<string> instead of string when not awaited. And the method where above code resides required to be async because of using await inside that method :
private async void SomeMethod()
{
....
if (xdoc.Element("Application") != null)
{
var data = from query in xdoc.Descendants("AppID")
select new APP
{
AppID = query.Value,
AppName = await GetName(query.Value),
};
itemGridView.DataContext = data;
}
....
}
UPDATE :
As you already noticed, LINQ has very limited support for async/await currently. So to workaround this limitation, we can use normal for loop to avoid calling async function inside LINQ :
private async void SomeMethod()
{
....
if (xdoc.Element("Application") != null)
{
var query = from query in xdoc.Descendants("AppID")
select query.Value;
var data = new List<App>();
foreach (var q in query)
{
data.Add(new App{ AppId = q, AppName = await GetName(q) });
}
itemGridView.DataContext = data;
}
....
}