Load webpage with reseting cookies in c# net - c#

I want to make a simple program, that loads a webpage(eg. to webclieant control) and i want to reset cookies in every time, when i load this page. I don't know, how to do it, so maybe can give me a example how to do it?
Thank you for any help :)


If you want to make simple loads using WebClient class of the Framework, it is actually simple. WebClient class can use Cookies by using Headers and ResponseHeaders properties. If you want to clear cookies on every request, just clean the right Header before doing a request. I don't know exactly if this is your question, so I will send an example on how to work with cookies using WebClient. I hope it is what you are looking for.
You can set cookies on WebClient easily by using the Headers property, and get the Cookies that must be sent back using the ResponseCookies property. If you want to manage your cookies, try doing something like this:
class Program
{
static void Main(string[] args)
{
// Put your cookies content here
string myCookies = "Cookie1=Value; Cookie2=Value";
// The URL you want to send a request
string url = "https://www.google.com.br/";
using (var client = new WebClient())
{
// If you want to attach cookies, you can do it
// easily using this code.
client.Headers.Add("Cookie", myCookies);
// Now you get the content of the response the way
// better suits your application.
// Here i'm using DownloadString, a method that returns
// the HTML of the response as a System.String.
// You can use other options, like OpenRead, wich
// creates a Stream to get the Response content.
string responseContent = client.DownloadString(url);
// If you want to save the cookies of the response to
// use them later, just use the ResponseHeaders property.
string responseCookies = client.ResponseHeaders["Set-Cookie"];
// Here is the response of your question (I I'm correct).
// If you need to use the same instance of WebClient to make
// another request, you will need to clear the headers that
// are being used as cookies.
// You can do it easily by using this code.
client.Headers.Remove("Cookie");
}
}
}


HttpCookie aCookie;
string cookieName;
int limit = Request.Cookies.Count;
for (int i=0; i<limit; i++)
{
cookieName = Request.Cookies[i].Name;
aCookie = new HttpCookie(cookieName);
aCookie.Expires = DateTime.Now.AddDays(-1);
Response.Cookies.Add(aCookie);
}

Related

How to get Cookie of Request Header after reponse in C#

I want to get cookies "Request Header" not "Reponse Header" of links "https://www.udemy.com/join/login-popup/"
I have used WebRequest and HttpClient but when I did not see Cookie debug in it
Please help me write it in C # I find trying to find every possible way but I still have not found.

Using System.Net.Http.HttpClient for example, we can get cookies like following:
var myClientHandler = new HttpClientHandler();
myClientHandler.CookieContainer = new CookieContainer();
var client = new HttpClient(myClientHandler);
var response = await client.GetAsync("https://www.udemy.com/join/login-popup/");
var cookieCollection = myClientHandler.CookieContainer.GetCookies(new Uri("https://www.udemy.com/join/login-popup/"));
foreach (var cookie in cookieCollection.Cast<Cookie>())
{
Debug.WriteLine(cookie);
}
HttpClient saves cookies that are sent by a server and automatically add them to subsequent requests to that URI within the same app container. So if we didn't modify cookies manually, the cookies in "Request Header" should be the same as what returned in previous reponse.

How do I get the destination URL of a shortened URL?

I have an API (https://www.readability.com/developers/api/parser#idm386426118064) to extract the contents of the webapges, but on passing a shortened url or an url that redirects to other, it gives error.
I am developing windows phone 8.1 (xaml) app. Is there any way to get the destination url in c# or any work around?
eg url - http://www.bing.com/r/2/BB7Q4J4?a=1&m=EN-IN

You could intercept the Location header value before the HttpClient follows it like this:
using (var handler = new HttpClientHandler())
{
handler.AllowAutoRedirect = false;
using (var client = new HttpClient(handler))
{
var response = await client.GetAsync("shortUrl");
var longUrl = response.Headers.Location.ToString();
}
}
This solution will always be the most efficient because it only issue one request.
It is possible however, that the short url will reference another short url and consequently cause this method to fail.
An alternative solution would be to allow the HttpClient to follow the Location header value and observe the destination:
using (var client = new HttpClient())
{
var response = client.GetAsync("shortUrl").Result;
var longUrl = response.RequestMessage.RequestUri;
}
This method is both terser and more reliable than the first.
The drawback is that this code will issue two requests instead of one.

You can get the ResponseUri from GetResponse():
string redirectedURL = WebRequest.Create("http://www.bing.com/r/2/BB7Q4J4?a=1&m=EN-IN")
.GetResponse()
.ResponseUri
.ToString();
Interesting article, by the way.

You need to inspect the headers returned from the URL.
If you get HTTP return codes 301 or 302, then you are being notified that the page is redirecting you to another URL.
See http://www.w3.org/Protocols/HTTP/HTRESP.html for more details about HTTP return codes.

Windows Phone REST Call issue

I got a Windows Phone project which are getting an list of a object that i call friend from an webservice using REST. The whole connection part works good the first time, the second time it fails. I get the same value from the method even tho i change the value in the database. Here is an example.
Users goes to an page
The right value is fetched from the database using an WCF service
I change the value in the database manually
User goes into the page again and the method for fetching the data is started again but it still gets the same result as the first time it fetched data from the webservice.
Any ideas?
Code:
HttpWebRequest request;
protected override void OnNavigatedTo(System.Windows.Navigation.NavigationEventArgs e)
{
UpdateData();
}
public void UpdateData()
{
((App)App.Current).ShowProgressBar();
string uri = App.SERVICE_URI + "/GetFriends?userid=" + App.CurrentUserId;
request = (HttpWebRequest)HttpWebRequest.Create(new Uri(uri));
request.BeginGetResponse(new AsyncCallback(GetFriendsReadCallback), request);
}
private void GetFriendsReadCallback(IAsyncResult asynchronousResult)
{
List<Friend> friends;
HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(asynchronousResult);
using (StreamReader streamReader1 = new StreamReader(response.GetResponseStream()))
{
string resultString = streamReader1.ReadToEnd();
var ser = new DataContractJsonSerializer(typeof(List<Friend>));
var stream = new MemoryStream(Encoding.Unicode.GetBytes(resultString));
DataContractJsonSerializer jsonSerializer = new DataContractJsonSerializer(typeof(List<Friend>));
friends = (List<Friend>)jsonSerializer.ReadObject(stream);
}

You are running into a caching issue, most likely. Either change your REST service to disable caching:
Response.Cache.SetCacheability(HttpCacheability.NoCache);
If you don't have access to the service code, you can append a changing dummy parameter to the end of your request URL (e.g. a GUID or DateTime.Now.Ticks).
Check out my answer here: WebClient in a WP7.1 app called only once

I'd run this in the emulator with fiddler2 operating - check what id being sent over http to find out if the call is being made second time, and if the correct answer is being sent back. Once you know whether the problem is client or server side then start debugging using breakpoints - where is the wrong value first picked up.

get data from page that a url points to

Given a Url, I'd like to be able to capture the Title of the page this url points to, as well
as other info - eg a snippet of text from the first paragraph on a page? - maybe even an image from the page.
Digg.com does this nicely when you submit a url.
How could something like this be done in .Net c#?

You're looking for the HTML Agility Pack, which can parse malformed HTML documents.
You can use its HTMLWeb class to download a webpage over HTTP.
You can also download text over HTTP using .Net's WebClient class.
However, it won't help you parse the HTML.

You could try something like this:
using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Text;
namespace WebGet
{
class progMain
{
static void Main(string[] args)
{
ASCIIEncoding asc = new ASCIIEncoding();
WebRequest wrq = WebRequest.Create("http://localhost");
WebResponse wrp = wrq.GetResponse();
byte [] responseBuf = new byte[wrp.ContentLength];
int status = wrp.GetResponseStream().Read(responseBuf, 0, responseBuf.Length);
Console.WriteLine(asc.GetString(responseBuf));
}
}
}
Once you have the buffer, you can process it looking for paragraph or image HTML tags to extract portions of the returned data.

You can extract the title of a page with a function like the following. You would need to modify the regular expression to look for, say, the first paragraph of text but since each page is different, that may prove difficult. You could look for a meta description tag and take the value from that, however.
public static string GetWebPageTitle(string url)
{
// Create a request to the url
HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
// If the request wasn't an HTTP request (like a file), ignore it
if (request == null) return null;
// Use the user's credentials
request.UseDefaultCredentials = true;
// Obtain a response from the server, if there was an error, return nothing
HttpWebResponse response = null;
try { response = request.GetResponse() as HttpWebResponse; }
catch (WebException) { return null; }
// Regular expression for an HTML title
string regex = #"(?<=<title.*>)([\s\S]*)(?=</title>)";
// If the correct HTML header exists for HTML text, continue
if (new List<string>(response.Headers.AllKeys).Contains("Content-Type"))
if (response.Headers["Content-Type"].StartsWith("text/html"))
{
// Download the page
WebClient web = new WebClient();
web.UseDefaultCredentials = true;
string page = web.DownloadString(url);
// Extract the title
Regex ex = new Regex(regex, RegexOptions.IgnoreCase);
return ex.Match(page).Value.Trim();
}
// Not a valid HTML page
return null;
}

You could use Selenium RC (Open Source, www.seleniumhq.org) to parse data etc. from the pages. It is a web test automation tool with an C# .Net lib.
Selenium have full API to read out specific items on a html page.

Request Web Page in c# spoofing the Host

I need to create a request for a web page delivered to our web sites, but I need to be able to set the host header information too. I have tried this using HttpWebRequest, but the Header information is read only (Or at least the Host part of it is). I need to do this because we want to perform the initial request for a page before the user can. We have 10 web server which are load balanced, so we need to request the file from each of the web servers.
I have tried the following:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://192.168.1.5/filename.htm");
request.Headers.Set("Host", "www.mywebsite.com");
WebResponse response = request.GetResponse();
Obviously this does not work, as I can't update the header, and I don't know if this is indeed the right way to do it.

Although this is a very late answer, maybe someone can get benefit of it
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(new Uri("http://192.168.1.1"));
request.Headers.GetType().InvokeMember("ChangeInternal", BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.InvokeMethod, null, request.Headers, new object[] {"Host","www.mysite.com"});
Reflection is your friend :)

I have managed to find out a more long winded route by using sockets. I found the answer in the MSDN page for IPEndPoint:
string getString = "GET /path/mypage.htm HTTP/1.1\r\nHost: www.mysite.mobi\r\nConnection: Close\r\n\r\n";
Encoding ASCII = Encoding.ASCII;
Byte[] byteGetString = ASCII.GetBytes(getString);
Byte[] receiveByte = new Byte[256];
Socket socket = null;
String strPage = null;
try
{
IPEndPoint ip = new IPEndPoint(IPAddress.Parse("10.23.1.93"), 80);
socket = new Socket(ip.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
socket.Connect(ip);
}
catch (SocketException ex)
{
Console.WriteLine("Source:" + ex.Source);
Console.WriteLine("Message:" + ex.Message);
}
socket.Send(byteGetString, byteGetString.Length, 0);
Int32 bytes = socket.Receive(receiveByte, receiveByte.Length, 0);
strPage = strPage + ASCII.GetString(receiveByte, 0, bytes);
while (bytes > 0)
{
bytes = socket.Receive(receiveByte, receiveByte.Length, 0);
strPage = strPage + ASCII.GetString(receiveByte, 0, bytes);
}
socket.Close();

I had a problem where the URL dns I used had several different IP addresses, I wanted to call each address separately using the same dns name in the host - the solution is using a proxy:
string retVal = "";
// Can't change the 'Host' header property because .NET protects it
// HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// request.Headers.Set(HttpRequestHeader.Host, DEPLOYER_HOST);
// so we must use a workaround
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Proxy = new WebProxy(ip);
using (WebResponse response = request.GetResponse())
{
using (TextReader reader = new StreamReader(response.GetResponseStream()))
{
string line;
while ((line = reader.ReadLine()) != null)
retVal += line;
}
}
return retVal;
Host header is set from 'url' automatically by .NET, and 'ip' contains the actual address of the web server you want to contact (you can use a dns name here too)

I know this is old, but I came across this same exact problem, and I found a better solution to this then using sockets or reflection...
What I did was create a new class that durives from WebHeaderCollection and bypasses validation of what you stick inside it:
public class MyHeaderCollection:WebHeaderCollection
{
public new void Set(string name, string value)
{
AddWithoutValidate(name, value);
}
//or
public new string this[string name]
{
get { return base[name]; }
set { AddWithoutValidate(name, value); }
}
}
and here is how you use it:
var http = WebRequest.Create("http://example.com/");
var headers = new MyHeaderCollection();
http.Headers = headers;
//Now you can add/override anything you like without validation:
headers.Set("Host", http.RequestUri.Host);
//or
headers["Host"] = http.RequestUri.Host;
Hope this helps anyone looking for this!

I know this is an old question, but these days, you can do.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://192.168.1.5/filename.htm");
request.Host = "www.mywebstite.com";
WebResponse response = request.GetResponse();

The "Host" header is protected and cannot be modified programmatically. I suppose to work around this, you could try and bind via reflection to the private "InnerCollection" property of the WebRequest object and calling the "Set" ar "Add" method on it to modify the Host header. I haven't tried this, but from a quick look at the source code in Reflector, I think it's easily accomplished. But yeah, binding to private properties of framework objects is not the best way to accomplish things. :) Use only if you MUST.
edit: Or like the other guy mentions in the linked question, just open up a socket and do a quick "GET" manually. Should be a no brainer, if you don't need to tinker with other stuff, like cookies or whatever else niceties the HttpWebRequest provides.

Alright, little bit of research turns up this:
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=384456
Seems MS may do something about this at some point.

You can use my solution for this problem, it posted here :
How to set custom "Host" header in HttpWebRequest?
This can help you to edit host header, and avoid to using proxy and direct socket requests.

Necromancing.
For those still on .NET 2.0
It is in fact quite easy, if you know how.
Problem is, you can't set the host header, because the framework won't let you change the value at runtime. (.net framework 4.0+ will let you override host in a httpwebrequest).
Next attempt will be setting the header with reflection - as demonstrated in the top upvoted answer here - to get around it, which will let you change the header value. But at runtime, it will overwrite this value with the host part of the url, which means reflection will bring you nothing, which is why I don't understand why people keep upvoting this.
If the dns-name doesn't exist, which is quite frankly the only case in which you want to do this in the first place, you can't set it, because .NET can't resolve it, and you can't override the .NET DNS resolver.
But what you can do, is setting a webproxy with the exact same IP as the destination server.
So, if your server IP is 28.14.88.71:
public class myweb : System.Net.WebClient
{
protected override System.Net.WebRequest GetWebRequest(System.Uri address)
{
System.Net.WebRequest request = (System.Net.WebRequest)base.GetWebRequest(address);
//string host = "redmine.nonexistantdomain.com";
//request.Headers.GetType().InvokeMember("ChangeInternal",
// System.Reflection.BindingFlags.NonPublic |
// System.Reflection.BindingFlags.Instance |
// System.Reflection.BindingFlags.InvokeMethod, null,
// request.Headers, new object[] { "Host", host }
//);
//server IP and port
request.Proxy = new System.Net.WebProxy("http://28.14.88.71:80");
// .NET 4.0 only
System.Net.HttpWebRequest foo = (System.Net.HttpWebRequest)request;
//foo.Host = host;
// The below reflection-based operation is not necessary,
// if the server speaks HTTP 1.1 correctly
// and the firewall doesn't interfere
// https://yoursunny.com/t/2009/HttpWebRequest-IP/
System.Reflection.FieldInfo horribleProxyServicePoint = (typeof(System.Net.ServicePoint))
.GetField("m_ProxyServicePoint", System.Reflection.BindingFlags.NonPublic |
System.Reflection.BindingFlags.Instance);
horribleProxyServicePoint.SetValue(foo.ServicePoint, false);
return foo; // or return request; if you don't neet this
}
}
and voila, now
myweb wc = new myweb();
string str = wc.DownloadString("http://redmine.netexistantdomain.com");
and you get the correct page back, if 28.14.88.71 is a webserver with virtual name-based hosting (based on http-host-header).
Now you have the correct answer to the original question, for both WebRequest and WebClient. I think using custom sockets to do this would be the wrong approach, particularly when SSL should be used, and when an actual solution is that simple...

Categories

Resources