Develop Reference

C# Download a complete weired HTML page

I'm sorry if this question has allready been answered , but I litterally spent more than two weeks searching the Internet for a solution to my issue.
Now , I definitly do not perform the best google searches , and it might seem that my question has several effective answers on the Internet. but I really tried every single solution that I found , without any positive results.
What i'm trying to do is simple , and I did it successfully on many websites :
Navigating to a website using WebBrowser (1).
Waiting for everything to load properly (document completed event).
Download the page using DocumentText property (1).
(1) : I also use WebClient from time to time.
And there it is , I get the html page , and I can exploit it anyway I like. The issue is with a particular website that I cannot obtain the full content inspite of using all the different solutions that I found. I suspected the fact that this page might need to load several scripts before getting the full content. Yet again, I read that WebBrowser does run all the necessarry scripts before triggering the " completed " event, so , apparently , that's not the issue. The page that i'm inquiring about is : http://www.coolmod#com/tarjetas-graficas-nvidia-pci-express
I tried , after that the WebBrowser loads the entire page , looking for random elements using GetElementByID property and checking if I get a null result. It appears that when I try getting an element that does not belong to the products list , i'm successful. But , whenever I try to get an element that belongs to the list it self , I always get a null. Which means , the list it self does not load. and I really don't know why. By the way , I do not prevent the WebBrowser. Navigate () from delivering multiple responses , I allow it to give as many feedbacks as possible , and still , the product list does not load , even when I pass the cookies. I Even tried copying all the content of the document and pasting it through the clipboard. Here is a simple example of what I try to do :
private void catalogueDownload()
{
System.Windows.Forms.WebBrowser wb = new System.Windows.Forms.WebBrowser();
wb.ScriptErrorsSuppressed = true;
wb.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(Catalogue_DocumentCompleted);
wb.Navigate("http://www.coolmod.com/tarjetas-graficas-nvidia-pci-express");
}
public void Catalogue_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
var wb = sender as System.Windows.Forms.WebBrowser;
string output = wb.DocumentText;
File.WriteAllText("testing.html", output);
}
Thanks for giving up your time to read all this.

System.Windows.Forms.WebBrowser is a bit outdated, If I were you, I would consider using an external library for that, Selenium would be my 1st choice, given it has all the necessary integrations with .NET Framework (and a lot of other languages)

Selenium Webdriver not returning Javascript code

Hi I am new to Selenium Webdriver. I can successfully open a webpage and find elements on it.
In one case I have noted that there is a link on a page that becomes clickable after a while. In Firebug on the Script tab, I can see the code for the javascript that does the timer function.
But using Selenium Webdriver if I issue:
driver.PageSource
I cannot see the source code for the Javascript. Delaying for 30 seconds before requesting the source makes no difference. I have tried finding it with various By options using:
driver.FindElement
and so on, but it isnt there.
How does firebug manage to find and show the Javascript source code? Is there a way that I can coerce Selenium Webdriver to return all code referenced by the page?
Or is there a better approach?
Thanks for any advice!
EDIT---------------------
I tried the following in Firefox:
Dim Driver2 As IWebDriver = New Chrome.ChromeDriver
Driver2.Url = "http://mypage"
Dim js As IJavaScriptExecutor = TryCast(Driver2, IJavaScriptExecutor)
Dim title As String = DirectCast(js.ExecuteScript("return JSON.stringify(window)"), String)
and I got
Permission denied to access property 'toJSON'
I read that this wont work in firefox so I tried in Chrome, and got
Blocked a frame with origin "http://mypage" from accessing a
cross-origin frame
and from there no solutions because according to this its a security restriction, apparently you can't access an with Javascript
I'm starting to think Im a bit out of my depth here.

PageSource probably doesn't return an exact snapshot of the DOM & etc.
You can instead inspect javascript using driver.executeScript() but the burden of analyzing the return object may be discouraging.
Regardless - Here's a contrived example:
Object result = driver.executeScript("return JSON.stringify(window)");
System.out.println(result.toString());

An error occurred executing the click atom

Using C# and Selenium, I am building an automated script where I, amongst other things, try to select a certain value from a droplist (value being specified in a .csv-file). I get the error;
"An error occurred executing the click atom (WARNING: The server did not provide any stacktrace information)"
I have no idea what a click atom is, much less how to fix it... Any help is appreciated greatly!
thanks in advance
this is the code for the droplist:
public bool isellHOSelectAdultsDroplist(string adults)
{
writeToLog(String.Format("Selecting adults from drop list"), this.GetType().Name);
String xpathString = HO_ADULT_SELECTION;
if(GpoExplicitWaitXpathElement(xpathString, 3, 5))
{
IWebElement dropListObjects = webDriver.FindElement(By.XPath(xpathString));
writeToLog(String.Format("DEBUG: Trying to click on appropriate number of adults..."), this.GetType().Name);
selectValueFromAdultDropList(dropListObjects, adults);
return true;
}
else
{
return false;
}
}
//...and this is my select-method
private void selectValueFromAdultDropList(IWebElement dropListObjects, string adults)
{
SelectElement manipulateDroplistObject = new SelectElement(dropListObjects);
manipulateDroplistObject.SelectByValue(adults);
String selection = manipulateDroplistObject.SelectedOption.Text;
int numberOfElements = manipulateDroplistObject.Options.Count;
writeToLog("Number of elements in Adult Droplist: " + numberOfElements, this.GetType().Name);
writeToLog("Selection from adult droplist: " + selection, this.GetType().Name);
}

I'll answer the specific question you asked, which is, "What is a click atom?" There is quite a bit of functionality in the IE driver, and the implementation of this functionality rests on three pillars.
First is IE's COM interfaces. These are the objects and methods that have been used to automate various parts of IE for more than a decade.
The second technology is so-called "native events." That is, using OS-level mechanisms to perform user interactions, like key presses and mouse clicks. On Windows, that means using the Windows SendMessage API. Almost anytime you're using the keyboard or the mouse with the IE driver, you're using native events by default.
Finally, a good portion of the IE driver functionality is implemented using JavaScript functions, which are shared by all of the browsers. These functions are known as "automation atoms".
One of the very few exceptions to using native events for mouse operations is in selecting an <option> element from a <select> element. Since IE doesn't give discoverable dimensions to <option> elements, the IE driver is forced to simulate the click action via JavaScript. This means using the automation atom for the click action. In your case, something must've gone wrong executing that JavaScript, which was faithfully reported as a "failure to execute the click atom." Without more detail, including sample HTML pages to reproduce the issue, it will be exceedingly difficult to diagnose the root cause of the issue.
It's at this point I will echo the call to update to the latest IE driver. Some of the code in this area has been overhauled, and at the least, it should be possible to extract more precise errors from failure cases with a more recent driver.

IE 9 BHO addon. Error SCRIPT1014 with loading external JS-file

I'm trying to create simple Internet Explorer 9 addon using .Net and BHO to load external js-file on page and execute it.
I created ieInstance_DownloadComplete (I also tried ieInstance_DocumentComplete, but things were worse) event handler:
InternetExplorer explorer = this.ieInstance;
var document = explorer.Document as IHTMLDocument2;
document.parentWindow.execScript(
#"
if (document.getElementById('KCScript') == null)
{
var fileref=document.createElement('script');
fileref.setAttribute('id', 'KCScript');
fileref.setAttribute('type','text/javascript');
fileref.setAttribute('charset', 'UTF-8')
fileref.setAttribute('src', 'C:/test.js');
fileref.onload = function () { eee(); };
if (typeof fileref!='undefined')
document.getElementsByTagName('head')[0].appendChild(fileref);}","JavaScript");
}
When page is loaded I can see my test.js attached to page in IE Developer Tools. But function eee() does not raise and i have an error:
"SCRIPT1014: Invalid character
test.js, line 1, character 1"
test.js:
function eee()
{
alert('ttt!');
};
test.js is UTF-8, so there is no reading problem.. there is something else
What's wrong? Any ideas?
Thanks in advance!

Try to resave JS in "UTF-8 without BOM" encoding
Try to place JS to remote server as http://example.com/test.js

How can you programmatically detect if javascript is enabled/disabled in a Windows Desktop Application? (WebBrowser Control)

I have an application which writes HTML to a WebBrowser control in a .NET winforms application.
I want to detect somehow programatically if the Internet Settings have Javascript (or Active Scripting rather) disabled.
I'm guessing you need to use something like WMI to query the IE Security Settings.
EDIT #1: It's important I know if javascript is enabled prior to displaying the HTML so solutions which modify the DOM to display a warning or that use tags are not applicable in my particular case. In my case, if javascript isn't available i'll display content in a native RichTextBox instead and I also want to report whether JS is enabled back to the server application so I can tell the admin who sends alerts that 5% or 75% of users have JS enabled or not.

Thanks to #Kickaha's suggestion. Here's a simple method which checks the registry to see if it's set. Probably some cases where this could throw an exception so be sure to handle them.
const string DWORD_FOR_ACTIVE_SCRIPTING = "1400";
const string VALUE_FOR_DISABLED = "3";
const string VALUE_FOR_ENABLED = "0";
public static bool IsJavascriptEnabled( )
{
bool retVal = true;
//get the registry key for Zone 3(Internet Zone)
Microsoft.Win32.RegistryKey key = Registry.CurrentUser.OpenSubKey(#"Software\Microsoft\Windows\CurrentVersion\Internet Settings\Zones\3", true);
if (key != null)
{
Object value = key.GetValue(DWORD_FOR_ACTIVE_SCRIPTING, VALUE_FOR_ENABLED);
if( value.ToString().Equals(VALUE_FOR_DISABLED) )
{
retVal = false;
}
}
return retVal;
}
Note: in the interest of keep this code sample short (and because I only cared about the Internet Zone) - this method only checks the internet zone. You can modify the 3 at end of OpenSubKey line to change the zone.

If you are having troubles with popups popping up, i've included a solution for you, and if you want to disable/enable javascript on th client machine (or even just read/query if it is enabled/disabled) ive included that answer for you as well, here we go:
Which popup message do you want to disable? If it's the alert message, try this, obviously resolving the window or frame object to your particular needs, I’ve just assumed top-level document, but if you need an iframe you can access it using window.frames(0). for the first frame and so on... (re the JavaScript part)... here is some code, assuming WB is your webbrowser control...
WB.Document.parentWindow.execScript "window.alert = function () { };", "JScript"
You must run the above code only after the entire page is done loading, i understand this is very difficult to do (and a full-proof version hasn't been published yet) however I have been doing it (full proof) for some time now, and you can gather hints on how to do this accurately if you read some of my previous answers labelled "webbrowser" and "webbrowser-control", but getting back to the question at hand, if you want to cancel the .confirm JavaScript message, just replace window.alert with window.confirm (of course, qualifying your window. object with the correct object to reach the document hierarchy you are working with). You can also disable the .print method with the above technique and the new IE9 .prompt method as well.
If you want to disable JavaScript entirely, you can use the registry to do this, and you must make the registry change before the webbrowser control loads into memory, and every time you change it (on & off) you must reload the webbrowser control out and into memory (or just restart your application).
The registry key is \HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Zones\ - the keyname is 1400 and the value to disable it is 3, and to enable it is 0.
Of course, because there are 5 zones under the Zones key, you need to either change it for the active zone or for all zones to be sure. However, you really don't need to do this if all you want to do is supress js dialog popup messages.
Let me know how you go, and if I can help further.

Here is a suggestion - Encode the warning into your webpage as default. Create a javascript that runs on page load which removes that element. The warning will be there when ever javascript is not allowed to run.
It's a long while since I coded client side HTML javascript to interact with the DOM so I may be a little out of date. i.e. you will need to fix details, but I hope I get the general idea across.
<script>
document.getElemntByID("noJavascriptWarning").innerHTML="";
</script>
and in your HTML body
<DIV id="noJavascriptWarning" name="noJavaScriptWarning">Please enable Javascript</DIV>