Develop Reference

How can I get the source of an iframe that only works on specified domains?

So I'm trying to read the source of an url, let's say domain.xyz. No problem, I can simply get it work using HttpWebRequest.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlAddress);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
My problem is that it will return the page source, but without the source of the iframe inside this page. I only get something like this:
<iframe src="http://anotherdomain.xyz/frame_that_only_works_on_domain_xyz"></iframe>
I figured out that I can easily get the src of the iframe with WebBrowser, or basic string functions (the results are the same), and create another HttpWebRequest using the address. The problem is that if I view the full page (where the frame was inserted) in a browser (Chrome), i get the expected results. But if I copy the src to another tab, the contents are not the same. It says that the content I want to view is blocked because it's only allowed through domain.xyz.
So my final question is:
How can I simulate the request through a specified domain, or get the full, rendered page source?

That's likely the referer property of the web request: typically a browser tells the web server where it found the link to the page it is requesting.
That means, when you create the web request for the iframe, you set the referer property of that request to the page containing the link.
If that doesn't work, cookies may be another option. I.e. you have to collect the cookies sent for the first request, and send them with the second request.

Grab the contents of a Drupal website that is secured with a login form

I would like to grab some content from a website that is made with Drupal.
The challenge here is that i need to login on this site before i can access the page i want to scrape. Is there a way to automate this login process in my C# code, so i can grab the secure content?

To access the secured content, you'll need to store and send cookies with every request to your server, starting with the request that sends your log in info and then saving the session cookie that the server gives you (which is your proof that you are who you say you are).
You can use the System.Windows.Forms.WebBrowser for a less control but out-of-the-box solution that will handle cookies.
My preferred method is to use System.Net.HttpWebRequest to send and receive all web data and then use the HtmlAgilityPack to parse the returned data into a Document Object Model (DOM) which can be easily read from.
The trick to getting System.Net.HttpWebRequest to work is that you must create a long-lived System.Net.CookieContainer that will keep track of your log in info (and other things the server expects you to keep track of). The good news is that the HttpWebRequest will take care of all of this for you if you provide the container.
You need a new HttpWebRequest for each call you make, so you must sets their .CookieContainer to the same object every time. Here is an example:
UNTESTED
using System.Net;
public void TestConnect()
{
CookieContainer cookieJar = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/login.htm");
request.CookieContainer = cookieJar;
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
// do page parsing and request setting here
request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/submit_login.htm");
// add specific page parameters here
request.CookeContainer = cookieJar;
response = (HttpWebResponse) request.GetResponse();
request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/secured_page.htm");
request.CookeContainer = cookieJar;
// this will now work since you have saved your authentication cookies in 'cookieJar'
response = (HttpWebResponse) request.GetResponse();
}
http://msdn.microsoft.com/en-us/library/system.windows.forms.webbrowser.aspx
HttpWebRequest Class
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.cookiecontainer.aspx

You'll have to use the Services module to do that. Also check out this link for a bit of explanation.

receiving details from POST

im trying to build a program that can "login" to site as user to get the html code,
then ill fix the code to make more options to the user:)
i googled it and as i understand i need to send a cookie to identify myself as the user,
i use firefox edit cookies to see which cookies saves at my cmputer and saw only one 'phpsessid' that saves a string represent the session ,
i use wireshark to see how its real going and as i saw when im getting the response page im getting also this line
PHPSESSID=xxxxxxxxxxxxxxxxxxxxxxx; path=/
how i can read it from the response string to be able surf another pages as the 'user'?
tyvm for your help:)
edit:
i got it:
Req.GetResponse().Headers.Get("Set-Cookie");

Before you do the login, create a CookieContainer and assign it to the request:
var request = (HttpWebRequest)WebRequest.Create(loginUrl);
var cookies = new CookieContainer();
request.CookieContainer = cookies;
After you make the request, cookies will contain the cookies that make you logged-in.
When you want to make another request, now as a logged-in user, use cookies again:
var request = (HttpWebRequest)WebRequest.Create(anotherUrl);
request.CookieContainer = cookies;

What domain should I give this cookie?

I'm trying to call a web service from a c# application, with sessionID.
In order to do this I need to set the "Domain" header in a cookie.
In Fiddler it looks like - "ASP.NET_SessionId=izdtd4tbzczsa3nlt5ujrbf5" (no domain is specified in the cookie).
The web service is at - "http://[some ip goes here]:8989/MyAPI.asmx".
I've tried:
http://[ip] ,
http://[ip]:8989 ,
http://[ip]:8989/MyAPI.asmx
All of these cause runtime error.
I've also tried the ip alone (i.e. 100.10.10.10) , which doesn't cause a runtime error, and sets the cookie, but the cookie is never sent when I invoke a web method.
Here's my code for setting the domain:
if (!string.IsNullOrEmpty(currentSessionID))
{
req.CookieContainer=new CookieContainer();
Cookie cookie = new Cookie("ASP.NET_SessionId", currentSessionID);
cookie.Domain = GetCookieUrl(); //<- What should this be?
req.CookieContainer.Add(cookie);
}
So what should the domain be?
Thanks.

I believe it should simply be [ip]. Drop the http:// part of what you've tried.
According to this page on MSDN, your code should be
cookie.Domain = "100.10.10.10";
Next, exactly what error are you getting? Also, are you confusing a Compile error with a Runtime error? I find it hard to believe you are getting a compilation error as Domain is a String property which means you can put pretty much anything into it.
Finally, why are you sending a cookie to a web service? The normal way is to pass everything in the form post or on the query string.
Update
BTW, if you absolutely must add a cookie to the header in order to pass it to a web service, the way you do this is (taken from here):
byte[] buffer = Encoding.ASCII.GetBytes("fareId=123456"); //the data you want to send to the web service
HttpWebRequest WebReq = (HttpWebRequest)WebRequest.Create(url);
WebReq.Method = "POST";
WebReq.ContentType = "application/x-www-form-urlencoded";
WebReq.ContentLength = buffer.Length;
WebReq.Headers["Cookie"] = "ASP.NET_SessionId=izdtd4tbzczsa3nlt5ujrbf5"
Stream PostData = WebReq.GetRequestStream();
Note that this sets the header inline with the request without instantiating a "cookie" object. The Domain property of a cookie is to help ensure the cookie is only sent to the domain listed. However, if you are initiating the request and trying to append a cookie to it, then the best way is to just add it as a string to the request headers.

The reason the cookie was not sent is that the request's content length should be set after adding the cookie, and not before.
The domain is the ip alone.

// Simple function to get cookie domain
private string GetCookieDomain(string uri)
{
Uri req_uri = new Uri(uri);
return req_uri.GetComponents(UriComponents.Host, UriFormat.Unescaped);
}

HttpRequest: pass through AuthLogin

I would need to make a simple program that logs with given credentials to certain website and then navigate to some element (link).
It is even possible (I mean this Authlogin thing)?
EDIT: SORRY - I am on my company machine and I cannot click on "Vote" or "Add comment" - the page says "Done, but with errors on page" (IE..). I do appreciate your answers and comments, you have helped me a lot!

Main things to do are:
Start using Fiddler to see what needs to be sent and in what way
Assuming we're talking a normal web form you'll probably need to use a CookieContainer with your WebRequests in order to accept the cookies that come from the login request and then re-supply them when sending subsequent requests (such context is not automagically maintained by HttpWebRequest) :-
CookieContainer _cookieContainer = new CookieContainer();
((HttpWebRequest)request).CookieContainer = _cookieContainer;

yes. it is possible.
see following code:
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.AuthenticationLevel = System.Net.Security.AuthenticationLevel.MutualAuthRequested;
req.Credentials = new NetworkCredential("admin", "admin");
req.PreAuthenticate = true;

It will partly depend on how the login process is managed. Is this actually done via a web form? If so, you'll need to post the form, just as a normal browser would.
If it's done over HTTP authentication, you should be able to set the credentials in the web request, tell it to pre-authenticate, and all should be well.