So, I want to automate actions on my website (including some Javascript execution and XPath element findings, performing click/type events) but the thing is that I don't want the endpoint users to install any third party software.
Been looking through many solutions and found out that selenium would be a great solution for my needs, however:
It does require endpoint user to have Google Chrome / Firefox browser installed in case of using chrome/firefox driver that support headless browsing.
So I kept searching and found CefSharp which would embed the browser into my application. That however, it's really a boilerplate solution without the ability to find elements by XPath and compared to selenium is less functional.
My question is:
Are there any other solutions (for C#) for having a functional headless browser, without requiring the endpoint user to have the browser installed so that I could deliver my binary as a ready to use?
Related
I am developing a website crawler API to scrape a javascript-generated website. The website that we are crawling requires the Javascript to be enabled to fully-render the HTML. I have tried many solutions such as HtmlAgilityPack and AngleSharp, but they are just HTML parsers and they cannot render the page due to missing Javascript capability.
I tried implementing headless browser using Selenium.WebDriver.ChromeDriver, it worked very well in my local machine. However, our production environment is very limited such that only Internet Explorer browser is available and we are not allowed to install any more browser. So this chromedriver did not work, too. Internet Explorer cannot even fully render the website from the browser itself. So IE is definitely out.
Is there a way to scrape a javascript-generated website without having to install a browser? Like implementing a headless browser on a server without that browser installed?
Or is it a dead-end situation. Thanks!
You can try using a solution that uses a fully-functional built-in Chromium and doesn't require installing Google Chrome in the target environment. All the required Chromium binaries will be shipped with the solution.
There are many such solutions for .NET and C#:
CefSharp
An open source .NET wrapper around the Chromium Embedded Framework (CEF). It allows you to embed Chromium in .NET apps.
Supported by community. If you need help with the library use, read docs or ask community. If you need a feature or a bug fix, you would probably need to do it by yourself.
DotNetBrowser
A commercial library that allows integrating a Chromium-based browser with your .NET app to display and process HTML5, CSS3, JavaScript, etc.
It's a proprietary solution supported by a commercial company. If you need help with the library use, read docs or get help from the engineers of this product. If you need a feature or a bug fix, it will be done by the product team as soon as possible. I know that, because I know the engineers from DotNetBrowser team.
WebView2
This control allows you to embed web technologies (HTML, CSS, and JavaScript) in your native apps. The WebView2 control uses Microsoft Edge (Chromium) as the rendering engine to display the web content in native apps. With WebView2, you can embed web code in different parts of your native app, or build all of the native app within a single WebView instance. Supported by Microsoft.
If you need some help, you should contact WebView2 team.
I'm building a developer-themed browser in a C# .NET Windows Form Application and want to allow users to use Chrome or Firefox DevTools to edit/debug the current page they are viewing. I have found several repositories online but none of them seem to be what I want.
Example Chrome DevTools
Example Firefox DevTools
The project uses Geckofx60.64 to create an embedded Gecko web browser. I have already tried debugger.html on GitHub but that didn't help me. All I need is a simple way to show website developer tools for any link.
If this is not possible, are there some other developer tools that I might be able to use with this project?
I'm developing a software on C# which has to get info from a website which the user opens in chrome, the user has to input some data and then the website returns a list of different items.
What I want is a way to be able to access to the source code of the page in order to get the info, I cant open the web myself as it doesnt show anything because I didnt input any data, so I need to get it directly from chrome.
How can I achieve this ? A chrome extension ? Or can I access to chrome directly from my software ?
Off the top of my head, I don't know any application that gets data directly from an open instance of Chrome. You'd have to write your own Chrome extension.
Alternatively, you can open the web browser from your application initially.
You can look into these libraries for doing so:
Watin (My personal favourite)
Selenium
Awesomium (You'd have to roll out your own UI, it's invisible)
Cef
Essential Objects Web Browser
EDIT: I didn't think about using QA tools as the actual browser hook as #TheAnathema mentions. That would probably work for your needs.
You're going to need to create it as Chrome extension if you must be dependent on the user actually going to a specific web site (i.e. not being able to do the requests yourself with either Selenium or standard web requests in Python).
The reason why a Chrome extension would be required is because think of how bad it could be for any software to easily read the pages you browse. Banking, medical, email, etc. could all be accessed anonymously from any process if Google allowed any outside process to tap into the web page.
Even Chrome extensions have to ask for permission to be able to do what they want, but at least it is software the user knowingly installed and agreed to the permissions.
A quick search yielded this example of modifying a page's HTML with a Chrome extension: https://blog.lateral.io/2016/04/create-chrome-extension-modify-websites-html-css/
It sounds like you want to do web scraping. Here's a good tutorial to get you started: HTML Scraping.
And this answer has a good example of how to scrape data from a website where you need to submit a form to get access to the data.
I am using the Selenium Firefox web driver and attempting to start an existing profile which I have created specifically for Selenium.
Firefox starts up with the correct profile, but the web driver shows as disabled in the extensions and therefore Selenium fails to connect to the web driver. How can I make sure that it starts with the extension enabled?
I would post code, but the code is working properly and the correct profile is used. This is a Firefox configuration issue I believe. If I let Selenium create a temp profile it works fine, but I need access to one of the extensions that I have setup as well as the configuration for that extension. I saw that it is possible to load extensions through code but I don't know how I could load an identical configuration.
I wanted to make an application wherein you specify the name of the websites, your username and password and that application automatically logs in to all your accounts in the specified websites. I have done this using windows form application, using a web browser. But i wanted my application to open all these websites in chrome and log it in there. Plz Help
I'd check out the chrome API failing that
Look into getting a handle to the window through window API calls
But why not just a chrome extension?!? Miles simpler
Have a look at Selenium WebDriver
Doing a quick Google search for "chrome C# api" turned up a number of results I think you may find relevant.
I thought the following were particularly promising, if you're willing to accept a few concessions:
Automating Chrome Browser from C#
ChromeDevTools; a C# Library to interact with Chrome's Developer Tools
Chrome Debugging API