query a website and retrive public data from it - c#

I am really new in c# programming. I would like some help from you guys (if possible). I have a website (it is a shopping website ) with data : products, price, description...etc. What I would like to do is: Since the website has a search capability so I would like to get the data from it by querying the search link and get only the important data (product id, name, price and description). When I perform the search I get many pages, and every time I press next I get new page with extra list of products. How can I simply make automation of these tasks?
I searched a lot over internet I found that I need to use webclient() with regular expression, and I thought that maybe a loop over the page content and over the search result pages would be necessary.
what do you think guys?
Website Example.
I´ll appreciate any effort from your side.

What you're describing is called scraping.
What you'll want is to use something like HtmlAgilityPack to get the website. Then you find the nodes you're interested in by using the DOM, and reading their inner text.
The whole process is rather complicated, but at least I've sent you off in the right direction. For the most part, search urls tend to have the same format.
In your link for instance
http://cdon.se/hemelektronik/advanced-search?manufacturer-id=&title=.&title-matchtype=1&genre-id=&page-size=15&sort-order=142&page=2
You can change 'page' to be smething else and you can go through all the pages that way.
Added:
Also don't TRY to use regex to parse html. It drove one particular person mad...
RegEx match open tags except XHTML self-contained tags

Related

Saving SSRS multi select parameter value with wildcard

I have searched and searched and have not been able to find the answer to this. I'm no stranger to SSRS, .Net (c# and vb.net), SQL, etc...been in it for years. I currently have a multi-select report parameter that is populated by a dataset in my report. There are hundreds of entries, so I built it to be driven by a wildcard character in a preceding parameter. Everything works fine right now. My question is this: is it possible to enter a wildcard value, select one (or more) of the filtered values and then store that/those value(s) on selection so that a user can go back and enter another wildcard value and select from a newly filtered list? (Basically, remember what has been selected in the overall dataset before report execution and create some sort of comma-separated list as the final parameter value to be passed to the report) I realize this may be better served in a web app w/a reportviewer control, but I'm trying to avoid deviating from the native SSRS server if possible. Thanks in advance!
The way I might approach this (not actually done it but the theory sounds ok)
Have 2 parameters for user input, your current one and a hidden one called say #filter (visible) and #filterHistory (this is the hidden one)
Have a textbox (formatted like button) with something like "Refine" as the text. Set the action to call your report again but set the #filterHistory to be something like #filterHistory & ", " & #filter. Basically we keep appending the last user input to the history.
Then your report would filter based on both parameters. You'll have to do some parsing of the delimited parameter now to split it out into the constituent parts but you get the idea.
I've no time to build a test report but hopefully that will point you in the right direction. If it doesn't help or work then comment and I'll see if I can knock up a quick example.

Twilio call usage paging

I am trying to get a paged list of call records from Twilio
var records = Twilio.Rest.Api.V2010.Account.CallResource.Read(pageSize: 10).ToList();
But nowhere is there anywhere to specify what the page number to retrieve is? How would I do this?
Sales Engineer at Twilio here.
There isn't a way to go to a specific page. You can navigate through the pages using the previous_page_uri and next_page_uri fields. I think the philosophy is that pages are a very implicit/arbitrary way of navigating your data. It's better to be explicit about what you're actually looking for, e.g., "Records from this date to this date" by using query parameters like this:
Category=calls&StartDate=2017-10-13&EndDate=2017-10-13"

How would I implement tags within a database?

I have an app and I want users to be able to add tags to articles (similar to stack overflows tags) but I want it to be dynamic. So far I have it as JSON strings (dont squirm too much) but this has a major shortcoming. Firstly I will show you an example, and then explain my problem.
Say I have an article and it's on bees, so users tag: bees, insect, honey, outdoors.
Then in my article class (entity framework) I have
string AssosciatedTags { get; set; }
which would hold: "[\"bees\", \"insect\",\"honey\"]"
then when I render the article I just do in javascript:
JSON.parse(model.AssosciatedTags);
and do whatever I want ... add/remove tags with ease. Ok, so here's where I didn't really think enough, and now I have I still am rather confused. How can I implement a sorting mechanism? Let's say my users are bee crazy. They love bees. Their cute little black and yellow stripes. So they want to click tags they are interested in and my server returns articles that have those tags assosciated.
But how? If I have lots of articles, it seems a bad idea to parse all tags from JSON to an array, look for this tag and return those articles.
My other consideration is to simple to a lookup for ",tagName," which is pretty filthy.
Is there a standard way or one that is hopefully more optimal.
My other consideration is to simple to a lookup for ",tagName," which is pretty filthy.
Yes, that is filthy. This technique has it's place but not here.
Store the tags in another table with the schema int ArticleID, string TagName. That way you can index TagName and query it efficiently.
This is the standard solution.
If you want to you can keep the JSON string but you need to keep it in sync with the table storing the tags. Preferably remove it, or be very careful.
Don't store your tags as JSON. Parsing them will be a huge slowdown on your database. Split them up into their own table. Something like this:
articles <--- table of articles
tags <--- table of tags
articles_tags <--- join table that associates articles with tags
This is called a Many-to-Many relationship.

ASP.NET MVC / C# - String to valid URL characters?

I don't know how to ask this, and I don't know what it is called either so I'll just describe what I want to achieve.
In the database, some articles' title originaly has spaces:
my title with spaces
But in the url, spaces are replaced by other characters such as plus sign (+) or underscore (_)
http://www.mydomain.com/mycontroller/myaction/my_title_with_spaces
or
http://www.mydomain.com/mycontroller/myaction/my+title+with+spaces
Now, how do you do that in C#? Or is there any helper in ASP.NET MVC that can do something like that?
Let say we achieved the said URL, is there any risk that two unique titles become the same in the URL? Please consider these titles:
Title's
Titles
after parsing, they became the same
Titles
Titles
This will be a problem when retrieving the article from the database since I'll get two results, one for "Title" and one for "Title's".
I would implement that functionality like this:
1. When creating a new article, generate the URL representation based on the title.
Use a function that converts the title for a suitable representation.
For example, the title "This is an example" might generate something like "This_is_an_example".
This is up to you. You can create a function that parses the title with rules you define, or use an existing one if it suits better your problem.
2. Ensure the URL representation is unique
If it's going to be an ID, it must be unique. So, when creating new articles you must query your database for the resulting URL representation. If you get a result from the database, it means the newly created article generated the same representation as one of the already created articles. Add something to it so it remains unique.
This could be something like "This_is_an_example_2". In this case, we added the "_2" to the end of the generated representation so it differs from the already existing one. Once more, with each change you have to ensure this representation remains unique.
3. Save the created ID in the database, along with the article data
In the database be sure to save the "This_is_an_example" ID and relate it to the article. Maybe even as the table primary key?
4. Query the database for the correct article
Now, about showing a site visitor the correct article:
When a visitor asks for the following resource, for example:
http://www.mydomain.com/mycontroller/myaction/this_is_an_example_2
Extract the URL part that identifies the article, in this case "this_is_an_example_2".
When you have that, you have the identifier of the article in the database. So, you can query the database for the article with the "this_is_an_example_2" ID and show the article's content to the user.
This might involve some URL rewriting. Unfortunately I'm unable to help you with that in asp.NET. Some search on the subject will surely help you.

Bread crumbs with url rewriting

In database I have a list of hierarchical list of categories so i construct a Custom Sitemap provider (by extending StaticSiteMapProvider ) with this list of categories.
Now, the page that display articles /ShowArticle.aspx?id=" + Eval("ID") is rewrited into:
/id/article-title.aspx.
So it's not physical exists.
I want the following: When the selected page is displayed I want to display the complete category path in the "Bread crumbs" (every article has a "category" property that contains it's 1 lvl category - without parents)
For example categories structure:
Home
Products
CdRom
DvdRom
Toshiba
Support
Hardware
Software
When an article id=xx,category=Toshiba, title="Best Toshiba DVD" is opened, the path in "Bread crumbs" should be:
Home|Products|DvdRom|Toshiba|Best Toshiba DVD
How to do it? Should I add something to my custom Sitemap provider?
I don't get any answers :(
Maybe I should do like this:
Instead constructing the site map for categories, i should construct it both for categories and all available articles in the database.
It's not efficient (even with caching), but it's the only way i think.
What else can I do?
Update:
What is left is just to create dynamically list of links that will act as bread crumbs,
when an article is opened it's containing category will be read and full hierarchical categories path will be calculated (from root to this current node).
I had to look up what the SiteMap provider was, and in doing so accidently came across the answer you may be looking for on the MSDN website: Breadcrumb using SiteMapPath Class
It might not be quite what you are after but it could be a point in the right direction :)

Categories

Resources