Develop Reference

My C# WPF webscraper returns error when more than one result is found

I am working on a WPF XAML application that scrapes certain websites for products. I have the search part working and it finds what I'm looking for. But as soon as there is more then 1 result I get a System.InvalidoperationException. I use a ObservableCollection to put the results into a <ListBox>.
Here is the search method:
private static ObservableCollection<EntryModel> _entries = new ObservableCollection<EntryModel>();
public static ObservableCollection<EntryModel> LoadCollectionData
{
get { return _entries; }
set { _entries = value; }
}
public static void PrehmSearchResults(string SearchQuery)
{
HtmlWeb web = new HtmlWeb();
try
{
string ZoekOpdracht = SearchQuery.Replace(" ", "+");
HtmlDocument doc = web.Load("https://www.prehmshop.de/advanced_search_result.php?keywords=" + ZoekOpdracht);
var title = doc.DocumentNode.CssSelect("div.header_cell > a").Single().InnerText;
var links = doc.DocumentNode.CssSelect("a.product_link");
var productLink = new List<string>();
var productTitle = new List<string>();
foreach (var item in links)
{
if (item.Attributes["href"].Value.Contains(".html"))
{
productLink.Add(item.Attributes["href"].Value);
productTitle.Add(title);
}
}
var TitleAndLink = productLink.Zip(productTitle, (l, t) => new { productLink = l, productTitle = t });
foreach (var nw in TitleAndLink)
{
var product = new List<EntryModel>();
var adDetails = new EntryModel
{
Title = nw.productTitle,
Link = nw.productLink
};
Debug.Print(adDetails.ToString());
var ZoekOpdrachtInTitle = adDetails.Title.ToLower().Contains(ZoekOpdracht.ToLower());
if (ZoekOpdrachtInTitle)
{
_entries.Add(adDetails);
}
}
}

So I found the solution without changing too much code. thanks for the help from #PaulSinnema.
The link is part of the title so I only had to change
var title = doc.DocumentNode.CssSelect("div.header_cell > a").ToList();
And I had to change the foreach loop:
foreach (var item in title)
{
if (item.Attributes["href"].Value.Contains(".html"))
{
productLink.Add(item.Attributes["href"].Value);
productTitle.Add(item.InnerText);
}
}

How to use lastEvaluatedKey with .ScanAsync?

I want to paginate through records in a dynamo table using a lastEvaluatedKey. How can I use the scanfilter to use the lastEvaluatedKey to begin the next request at that given location?
public List<Record>? GetRecords(Request request)
{
// Define marker variable
Dictionary<string, AttributeValue> startKey = null!;
var records = new List<Record>();
do
{
// Issue request
Condition cond = new Condition();
cond.ComparisonOperator = "NULL";
cond.AttributeValueList = new List<AttributeValue>() { };
var scanFilter = new Dictionary<string, Condition>() { { "ExecutedTime", cond } };
var allEvents = client.ScanAsync("rRecords", scanFilter).Result; //Dictionary<string, Condition> scanFilter
// View all returned items
List<Dictionary<string, AttributeValue>> items = allEvents.Items;
foreach (Dictionary<string, AttributeValue> item in items)
{
//do stuff
}
// Set marker variable
startKey = allEvents.LastEvaluatedKey;
} while (startKey != null && startKey.Count != 0);
return scheduledEventRecords;
}
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LowLevelDotNetScanning.html I have tried this approach but those APIs (.Scan on the AmazonDynamoDBClient) are not available the version AWSSDK 3.5.1.20 that I am currently using.

XDocument get Element value with Namespace (RSS Feed)

Im working on getting some values from an RSS feed but i am having difficulties getting a value which has the namespace in the element tag. I've tried adding the namespace to the lookup of the value but i always get null
Any idea on how this is achieved?
Feed
https://wegotthiscovered.com/movies/feed/
Element
xmlns:content="http://purl.org/rss/1.0/modules/content/"
Namespace
content:encoded
public async Task<bool> GetNewsFeeds()
{
Database db = new Database();
Dictionary<string, string> dictionary = new Dictionary<string, string>();
dictionary.Add("https://wegotthiscovered.com/movies/feed/", "Movie");
dictionary.Add("https://wegotthiscovered.com/blu-ray/feed/", "Blu-ray");
dictionary.Add("https://wegotthiscovered.com/reviews/feed/", "Reviews");
dictionary.Add("https://wegotthiscovered.com/featured/feed/", "Featured");
dictionary.Add("https://wegotthiscovered.com/galleries/feed/", "Galleries");
db.DeletMovieNews();
foreach (var pair in dictionary.ToList())
{
try
{
if (PhysicalDevice.HasInternetConnection())
{
XDocument doc = XDocument.Load(pair.Key);
XNamespace nsSys = "http://purl.org/rss/1.0/modules/content/";
var entries = (from item in doc.Descendants("item")
select new Movie_News
{
Content = item.Element(nsSys + "encoded").Value, // ISSUE HERE
Link = item.Element("link").Value,
PublishedDate = item.Element("pubDate").Value,
Title = item.Element("title").Value,
Description = item.Element("description").Value,
GroupName = "News",
FeedName = pair.Value
});
List<Movie_News> newsCollection = entries.ToList();
if (newsCollection.Count() != 0)
{
using (var rateGate = new RateGate(40, TimeSpan.FromSeconds(10)))
{
rateGate.WaitToProceed();
foreach (Movie_News item in newsCollection)
{
string regex = #"((http|ftp|https):\/\/)?([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,#?^=%&:\/~+#-]*[\w#?^=%&\/~+#-])?";
Match match = Regex.Match(item.Description, regex);
if (match.Success)
{
item.ImageUrl = match.Value;
item.B64Image = await DownloadImage(item.ImageUrl);
}
item.Description = item.Description.Remove(0, item.Description.IndexOf("</div>"));
item.Description = item.Description.Replace("</div>","");
db.InsertNewsData(item);
}
}
}
return true;
}
}
catch(Exception ex)
{
return false;
}
}
return true;
}
}

Typical , soon as i completed the write up, its working now

How to get value of a specific value of a list in C#

var ctsDB = mooe.Files66.ToList();
Dictionary<string, string> mappedfields = new Dictionary<string, string>();
Dictionary<string, string> ctsfieldsValue = new Dictionary<string, string>();
for (int i = 0; i < ctsDB.Count; i++)
{
foreach(var item in mappedfields) {
// this line returns the item.key string not the value of it.
// item.key is the column name
var rowValue = mooe.Files66.Select(k = >item.Key).ToList();
// ctsfieldsValue.Add(item.Value, rowValue);
ctsfieldsValue.ToList();
}
}
I want to iterate through ctsDB List and get the row value of a specific
column like this:
if ctsDB [i] = fileID Field612 Fiel613
and I have these column names in the value field of ctsfieldsValue.
I want to iterate on ctsDB[i] to get the value of column Field612 only.
Can anyone provide a thought?

var ctsDB = mooe.Files66.ToList();
var mappedfields = new Dictionary<string, string>(); // I assume this is not empty in your real code.
var ctsfieldsValue = new List<Dictionary<string, string>>();
foreach (var row in ctsDB)
{
var d = new Dictionary<string, string>();
foreach (var mapping in mappedfields)
{
var v = row[mapping.Key]; //throws if not found
d[mapping.Value] = v;
}
ctsfieldsValue.Add(d);
}

Anonymous collection initializer for a dictionary

Is it possible to implicitly declare next Dictionary<HyperLink, Anonymous>:
{ urlA, new { Text = "TextA", Url = "UrlA" } },
{ urlB, new { Text = "TextB", Url = "UrlB" } }
so I could use it this way:
foreach (var k in dic)
{
k.Key.Text = k.Value.Text;
k.Key.NavigateUrl = k.Value.Url;
}
?

How about:
var dict = new[] {
new { Text = "TextA", Url = "UrlA" },
new { Text = "TextB", Url = "UrlB" }
}.ToDictionary(x => x.Url);
// or to add separately:
dict.Add("UrlC", new { Text = "TextC", Url = "UrlC" });
However, you could just foreach on a list/array...
var arr = new[] {
new { Text = "TextA", Url = "UrlA" },
new { Text = "TextB", Url = "UrlB" }
};
foreach (var item in arr) {
Console.WriteLine("{0}: {1}", item.Text, item.Url);
}
You only need a dictionary if you need O(1) lookup via the (unique) key.

Yes, but only with great workaround, and only within a method.
This is how you can do it:
static Dictionary<TKey, TValue> NewDictionary<TKey, TValue>(TKey key, TValue value)
{
return new Dictionary<TKey, TValue>();
}
public void DictRun()
{
var myDict = NewDictionary(new { url="a"},
new { Text = "dollar", Url ="urlA"});
myDict.Add(new { url = "b" }, new { Text = "pound", Url = "urlB" });
myDict.Add(new { url = "c" }, new { Text = "rm", Url = "urlc" });
foreach (var k in myDict)
{
var url= k.Key.url;
var txt= k.Value.Text;
Console.WriteLine(url);
Console.WriteLine(txt);
}
}
You can refer to this SO question for more info.