Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a book, tool, software library, tutorial or other off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
Just curious: What do you find to be your best tools for creating automated screen scrapes these days? is the .Net Agility pack a good option? What do you do about scraping sites that use a lot of AJAX?
I find that if the page has a pretty static layout, then the HTML Agility Pack is perfect for getting all the data I need. I've not run into a single page that it hasn't been able to handle and not get me the results I wanted.
If you find that the page is rendered with a great deal of dynamic code, you're going to have to do more than just download the page, you'll have to actually execute it.
To do that, you'll need something like the WebKit .NET library (a .NET wrapper around the WebKit rendering engine) which will allow you to download the page and actually execute Javascript as well. Then, once you are sure the document has been rendered completely, you can get the page details.
For the very basics I use:
Asynchronous HTTP Client - notably faster than the standard HttpWeb* (preliminary tests showed that it was about 25% faster).
Majestic 12 HTML Parser - about 50-100% faster than HTML Agility Pack.
I don't have JavaScript enabled yet, but I'm planning on using Google's V8 JavaScript Engine. This requires that you make calls to unmanaged code, but the performance of V8 justifies it.
For automating screen scraping, Selenium is a good tool. There are 2 things- 1) install Selenium IDE (works only in Firefox). 2) Install Selenium RC Server
After starting Selenium IDE, go to the site that you are trying to automate and start recording events that you do on the site. Think it as recording a macro in the browser. Afterwards, you get the code output for the language you want.
Just so you know Browsermob uses Selenium for load testing and for automating tasks on browser.
I've uploaded a ppt that I made a while back. This should save you a good amount of time- http://www.4shared.com/get/tlwT3qb_/SeleniumInstructions.html
In the above link select the option of regular download.
I spent good amount of time in figuring it out, so thought it may save somebody's time.
The best tool "these days" is one that not only gives you the desired features (Javascript, automation), but also the one that you don't have to run yourself... I am, of course, alluding to using a cloud service. This approach will save you network bandwidth, will deliver results faster (because it can scale better than a custom solution you'll likely end up developing) and, most importantly, save you the IT and maintenance headache.
On that note, check out a scraping solution called Bobik (http://usebobik.com). I've written an article about it at http://zscraper.wordpress.com/2012/07/03/a-comparison-shopping-android-app-without-backend/.
Hope this helps.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I'm writing a video on demand solution. I want to take care of the end to end process, from ingestion of content to playback.
I have decided to utilise IIS Smooth Streaming for the method of delivery, which means all the video content must be encoded as H246 Adaptive Streaming Video.
I originally started using Azure for this project, but I as dove deeper I realised this really was too big a platform for what I need. It loaded unnecessary work and complexity for what I'm trying to achieve.
The 'issue' I'm having is with choosing an appropriate method to encode uploaded content. There are many encoding packages available but I can't find one which meets my criteria.
I'm happy to use an off the shelf package or write something with an appropriate SDK.
Must operate on Windows Server 2012
Must operate while the interactive user is logged off (i.e. as a windows service)
Ideally notify when the job is complete (can be an indirect method)
Ideally create a thumbnail
Invoking the encode process can be simple command line, watch folder or API / SDK
Must on on my server, not a cloud service
Must encode H264 Adaptive Streaming for IIS
I've tried:
Expression Encoder - Doesn't work in 2012, and no longer developed by MS
Sorenson Squeeze - Almost works, but leaves itself open when launched from command prompt so each time it encodes so I'd end up with 100's of instances
Azure .. too big & expensive
I know Sorenson has an server product which can be self-hosted, but this is cost prohibitive.
MainConcept have several SDK's and I've emailed them, however they don't list prices so this to me means expensive. (You may have noticed cost is a big factor. I'm one guy and a company)
Can anyone recommend a .NET SDK (c#) SDK or encoder package which will hit my criteria?
Many thanks
Take a look at http://www.ffmpeg.org/
While not a .net solution, it's free, and meets most of your criteria
I've not used it for adaptive streaming video, but apparently it supports it
It's all done from the command line - depending on what you want to do, you may need to write a wrapper for it (we had to do this for monitoring folders/databases and for notifications when complete), but i've successfully used it in the past to encode tens of gigs of video on a daily basis
You can write your own service in c# or directly run it in IIS application with the needed permission.
Using ffmpeg :
http://vbffmpegwrapper.codeplex.com/
https://code.google.com/p/ffmpeg-sharp/
Using vlc:
https://code.google.com/p/libvlc-sharp/
http://libvlcnet.codeplex.com/
Rhozet Promedia Carbon is the tool you're looking for. Supports all your needs. You can request a free demo. They handle all the licensing for the formats.
http://www.harmonicinc.com/product/promedia-carbon
Any tool you might want for this is going to be cost-prohibitive due to licensing.
As others have mentioned, your other option is FFMpeg
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Today I've been bouncing all around the internet, and after reading up on a lot of solutions I've decided that writing a Chromium wrapper would be an interesting learning experience; not to mention it would be worthwhile for several projects that I am currently working on.
I know that there are already some wrappers out there (awesomium, berkelium), not to mention some basic tutorials like this one, but I'm looking at writing my own wrapper.
My main interest in doing this is so that I am able to create and access the DOM structure generated by WebKit for use in C#, both before and after any javascript which the html depends on has been ran. I know I could look at using the HTMLAgilityPack and Majestic12 for HTML parsing, but I'm looking to build something on top of a tried and tested engine that is fast, efficient, and will be actively developed for the foreseeable future. WebKit is an actively developed layout engine embedded in some of the most popular browsers, and V8 is an impressive javascript engine, hence the decided use of Chromium.
Seeing as how I do not care about actually rendering any content and instead I am looking to access the structure of the web page, I feel that embedding WebKit in its entirety could potentially be overkill. Even in the instances where I'd like javascript to alter the DOM, I still have no use for rendering the results.
In summary:
All I truly want from Chromium is the ability to create the DOM of a web page, and (optionally) the ability to run said page's scripts using Chromium's impressive V8 engine to further manipulate the DOM. I know that this will be quite an undertaking, but I'm interested to see what you have to say. In short, has someone worked with WebKit and V8 (possibly via Chromium), who could assess the feasibility of this and lend me their expertise (via recommendations for how to approach this)? I know that I could eventually determine an appropriate way of approaching this, but I'm looking to expedite the process a bit. Links or perhaps a relatively short and sweet high-level "This is how I'd do it" are what I'm looking for (aka I don't need anything too terribly fancy).
I'm hoping that what I'm asking isn't too scatter-brained. If you find it too general and want to close this topic, instead of doing so please leave a comment and I'll try to narrow the focus. As I just stated my mind is bouncing around today since I'm coming down with a cold.
I'm on the same path. I'm looking deep on QtWebKit and the implementation done on PhantomJs which is a Selenium like, and worked like a charm on my tests.
When you use QtWebKit, as your driver to WebKit youll save time for next releases of it, because maintain a WebKit Port its not that easy.
I use Selenium Webdriver with ChromeDriver for automated script, web page and acceptance testing. Easy to set up, easy to use.
The idea is that you can use this existing interface, but change the path to the Chrome executable with ChromeOptions.BinaryLocation. Try pointing it to one of the existing headless builds of WebKit, like WebKitDriver - it might just work.
Our goals might be different, but I'd be interested in a headless version myself. It would be less obtrusive during testing on my dev machine, and I already take automated screenshots when a test fails.
selenium webdriver google-chrome chromedriver
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I have a customer who needs to convert a diagramming application (which was developed in MFC a long time ago) to C#.
The application displays large networks (lots of graphical elements), and lets the user edit/manipulate the data through a graphical ui.
I decided that it would be best to use a library rather than to develop all from scracth (all graphic objects/selection/tools/events/etc)
I am looking for a commercial solution.
I found three that seem to be very mature, and I wonder if
anyone had used them and can write his/her opinon:
Tom Sawyer Visualization
IBM ILog Diagrammer for .Net
yWorks - yFiles for .Net
thanks
Yaron
I am not familiar with this libraries, but one of my old projects we have use GoDiagram library, and can suggest that too
For making good looking diagrams in .Net you should check out Frank Hileman's VG.Net. He's a reputed MVP and I believe his solution is really good:
http://www.vgdotnet.com/
I recommend MindFusion's Flowchart.NET. It's very easy to use and is very affordable considering it comes with a multitude of powerful layouting algorithms. Previously we used GoDiagram, but I recommend against it because their licensing system is a nightmare if you have build machines, and it's much more expensive.
I believe Dundas charts is the most famous one...or at least the most advertised one:
http://www.dundas.com/Microsite/ChartNET/Default.aspx?Campaign=GoogleCSharpChart&gclid=CM-wncOq354CFUmK3godxENfMQ
I know this probably isn't the best solution but I'm going to put it out there anyway.
I've done something similar to this using .Net's System.ComponenetModel.DesignSurface. This is the same design service used in Visual Studio's Windows Forms so all you do is create your controls, add your control designers if you want and you're good to go. You can use the PropertyGrid to display the data for each object as they are selected. Code Project has several articles about this like this one.
That said it's not going to be the best performance wise, I've got several thousand controls on my DesignSurface in some cases and it gets sluggish. You may be able to get around this by using another root designer type (WPF maybe?).
This could be a very good option if you already know how to do custom Windows Forms controls. And best of all it’s free!
Have a look at Orbifold. They have got WPF based solutions (commercial) or libraries supporting GDI+ (free).
It's also a good starting point for information about diagramming algorithms in general.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I'm doing this application in C# using the free Krypton Toolkit but the Krypton Navigator is a paid product which is rather expensive for me and this application is being developed on my free time and it will be available to the public for free.
So, I'm looking for a free control to integrate better into my Krypton application because the default one doesn't quite fit and it will be different depending on the OS version...
Any suggestions?
P.S: I know I could owner-draw it but I'm trying not to have that kind of work... I prefer something already done if it exists for free.
EDIT:
I just found exactly what I wanted:
http://www.angelonline.net/CodeSamples/Lib_3.5.0.zip
If you need an updated lib with Office 2010 Palettes: http://www.angelonline.net/CodeSamples/Lib_4.2.0.zip
My first suggestion would be to talk to Phil at ComponentFactory. I find him to be a very reasonable fellow. Maybe he can give you a special deal or make a design suggestion on how to customize the existing tab control.
But your's is more of a design/subjective question that, I think, would benefit from a screenshot to better communicate the design challenge you need to "integrate better". Saying "the default one doesn't quite fit" is pretty vague.
After that, people will have a better starting point for making suggestions. In the mean time, I would look at the WindowsClient.NET control gallery.
I don't know of any open source or free tab controls, but I wonder why you don't just use the framework's tab control. Is there something you are trying to do that the Forms.TabControl doesn't do?
Download the Flat Tab Control (.NET) from Code Project. Takes about 30 seconds to get this working, and it gets you away from the default Windows tab control look and feel. But it's not Kryptonized.
See this post on a slick custom Kryptonized Tab Control based on the Flat Tab control you built in Step 1. You can download the control assembly on this downloads page (it's a little hard to find). So far as I know, the source code isn't available, however, Reflector can be of use here if you're curious as to how the Krypton-theming was done.
Replace the Flat Tab display logic (Paint, etc.) with Krypton-aware display logic. This is straightforward, because there's not a lot of code to the Flat Tab control.
If anyone is looking for the latest version of the AC.ExtendedRenderer.ToolKit which is mentioned in other posts, then the following links point to the central source of them. However the website is dead....even WaybackMachine can't help us get to the DLLs :(!
http://web.archive.org/web/20130725145918/http://www.advancedcomputing.ch/Downloads/tabid/62/Default.aspx
The latest I have been able to find is 4.1.6b here:
http://web.ticino.com/angelo/CodeSamples/Lib_4.1.6b.zip
If anyone knows where the later versions can be downloaded, please let us know.
EDIT:
The Component Factory Krypton components were migrated to https://github.com/ComponentFactory/Krypton
You could look at the Magic TabControl project over at CodeProject.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I'm a C# developer looking to get into home automation as a hobby. I have done a little research, but was wondering if anyone knows of a good .NET library that supports Insteon hardware. I'd rather use Insteon than X10 due to reliability issues.
My ultimate objective at this point is to have a simple home automation server (maybe lights and climate control) with a secure ASP.NET web application interface. I'm more interested in actually building it and learning about it rather than finding an existing solution.
Thanks for any suggestions or comments.
Edit:
Thanks for the help, everyone.
Does anyone have experience with Z-wave technology? Seems promising - appears to be higher quality hardware, includes a core library, supports .NET, etc. ControlThink appears to have a pretty good controller and SDK.
Here's an interesting application to consider: Stall Status: Know Before You Go
We found there really wasn't much developer support for Insteon unless you wanted to buy their SDK and agree to their rather heavy-handed license agreement. Rather than go that route, we wrote our own .NET library called FluentDwelling and we open-sourced it. You can find a download link, and some get-you-started code samples if you follow that link.
The source code comes with a full suite of unit tests (requires NUnit, also free), so you can add improvements and make changes if you like.
I would avoid X10 like the plague. Between things like modern TV's and power strips, bridged power junction boxes and just plain strange wiring, X10 signals tend to just "disappear" and never get to their destination.
If you really want to give X10 a shot, I've got a box of X10 stuff in the garage that was worth $250+ new and it's all completely useless in my house, so you can have it. Some of it worked in my old house, but it won't so much as turn a light on 2 outlets away where I live now.
X10 is viewed by most modern electronics as "noise" on the line (which, technically, it is) and something to be filtered out rather than passed along or left alone.
I can't remember if it covers the specific technologies you mention but you should definately check out this episode of .NET Rocks. They talk about all the different stuff you can do with home automation and a lot about how to do it. I believe one of the main points was that Microsoft Robotics Studio was a good tool to use, as it uses a lot of the same abstractions as you would otherwise use.
I know it's been a long time since this post was made, however I was wondering if you'd picked a route, and what you've ended up doing with home automation since.
I've been doing this kind of stuff with many of the technologies available, but I've always done it on top of a product called homeseer. .net plug-ins and vb.net and c# scripting can be done on top of this foundation. I've been thinking of rolling my own similar to what you were describing. Any thoughts, tips, decisions you've made etc?
A bit off topic, but listen to the latest episode of dot net rocks, one of the finalists of my .net story built an home automation solution that really made me want to dive into home automation again.
Totally awsome solution.
http://www.dotnetrocks.com/default.aspx?showNum=518
I think this would bee a place to start
insteon sdk
Some research via Google looking for an Insteon SDK only yields stuff from way back in 2005. There's only information on serial port interfaces (no USB) and the only language mentioned is something called DockLight scripting that also looks like it hasn't been updated since 2005.
X10, on the other hand, boasts support for C++, VisualBasic, VB Script, and JavaScript. No .Net listed, but VisualBasic/VB Script likely implies a COM object you can easily import for use in C#. They're also much more USB friendly, and the kit costs 1/4 of the Insteon kit ($50 vs $200).
If you were actually building a product I would understand the reliability concern. But since this is a home project I think you'll have much better luck with X10.
you can try C-bus by Clipsal(schneider) it's free.