I'm trying to get all cookies from a website using this code
CookieContainer cookieJar = new CookieContainer();
var request = (HttpWebRequest)HttpWebRequest.Create("http://www.foetex.dk/ugenstilbud/Pages/Zmags.aspx");
request.CookieContainer = cookieJar;
var response = request.GetResponse();
foreach (Cookie c in cookieJar.GetCookies(request.RequestUri))
{
Console.WriteLine("Cookie['" + c.Name + "']: " + c.Value);
}
Console.ReadLine();
The only thing i want is to display with console.writeline, but im not getting a single of them.
//Please use this,
HttpWebRequest request = null;
request = HttpWebRequest.Create(StringURL) as HttpWebRequest;
HttpWebResponse TheRespone = (HttpWebResponse)request.GetResponse();
String setCookieHeader = TheRespone.Headers[HttpResponseHeader.SetCookie];
The following example uses the HttpCookie class and its properties to read a cookie with a specific name.
HttpCookie myCookie = new HttpCookie("MyTestCookie");
myCookie = Request.Cookies["MyTestCookie"];
// Read the cookie information and display it.
if (myCookie != null)
Response.Write("<p>"+ myCookie.Name + "<p>"+ myCookie.Value);
else
Response.Write("not found");
Retrieve the values in the HttpWebResponse.Cookies property of HttpWebResponse. In this example, the cookies are retrieved and saved to isolated storage:
private void ReadCallback(IAsyncResult asynchronousResult)
{
HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
HttpWebResponse response = (HttpWebResponse)
request.EndGetResponse(asynchronousResult);
using (IsolatedStorageFile isf =
IsolatedStorageFile.GetUserStoreForSite())
{
using (IsolatedStorageFileStream isfs = isf.OpenFile("CookieExCookies",
FileMode.OpenOrCreate, FileAccess.Write))
{
using (StreamWriter sw = new StreamWriter(isfs))
{
foreach (Cookie cookieValue in response.Cookies)
{
sw.WriteLine("Cookie: " + cookieValue.ToString());
}
sw.Close();
}
}
}
}
To get the list of cookies, you can use the below method;
private async Task<List<Cookie>> GetCookies(string url)
{
var cookieContainer = new CookieContainer();
var uri = new Uri(url);
using (var httpClientHandler = new HttpClientHandler
{
CookieContainer = cookieContainer
})
{
using (var httpClient = new HttpClient(httpClientHandler))
{
await httpClient.GetAsync(uri);
return cookieContainer.GetCookies(uri).Cast<Cookie>().ToList();
}
}
}
Related
I am trying to unit test some code, and I need to to replace this:
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create( uri );
httpWebRequest.CookieContainer = new CookieContainer();
with
WebRequest webRequest = WebRequest.Create( uri );
webRequest.CookieContainer = new CookieContainer();
Basically, how do I get cookies into the request without using a HttpWebRequest?
Based on your comments, you might consider writing an extension method:
public static bool TryAddCookie(this WebRequest webRequest, Cookie cookie)
{
HttpWebRequest httpRequest = webRequest as HttpWebRequest;
if (httpRequest == null)
{
return false;
}
if (httpRequest.CookieContainer == null)
{
httpRequest.CookieContainer = new CookieContainer();
}
httpRequest.CookieContainer.Add(cookie);
return true;
}
Then you can have code like:
WebRequest webRequest = WebRequest.Create( uri );
webRequest.TryAddCookie(new Cookie("someName","someValue"));
Try with something like this:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.contoso.com/default.html");
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(new Cookie("ConstoCookie", "Chocolate Flavour"));
WebRequest is an abstract class that does not have a CookieContainer property. In addition you can't use the Headers collection (not implemented exception) so any attempt like webRequest.Headers.Add("Cookie", "...") will fail.
Sorry, but you have no chance to use cookies with WebRequest.
Stick on HttpWebRequest and add/edit as many cookies you like using its Headers collection!
dlev's answer ended up working, but I had problems implementing the solution ("The parameter '{0}' cannot be an empty string."), so I decided to write the full code in case anybody else has similar problems.
My goal was to get the html as a string, but I needed to add the cookies to the web request. This is the function that downloads the string using the cookies:
public static string DownloadString(string url, Encoding encoding, IDictionary<string, string> cookieNameValues)
{
using (var webClient = new WebClient())
{
var uri = new Uri(url);
var webRequest = WebRequest.Create(uri);
foreach(var nameValue in cookieNameValues)
{
webRequest.TryAddCookie(new Cookie(nameValue.Key, nameValue.Value, "/", uri.Host));
}
var response = webRequest.GetResponse();
var receiveStream = response.GetResponseStream();
var readStream = new StreamReader(receiveStream, encoding);
var htmlCode = readStream.ReadToEnd();
return htmlCode;
}
}
We are using the code from dlev's answer:
public static bool TryAddCookie(this WebRequest webRequest, Cookie cookie)
{
HttpWebRequest httpRequest = webRequest as HttpWebRequest;
if (httpRequest == null)
{
return false;
}
if (httpRequest.CookieContainer == null)
{
httpRequest.CookieContainer = new CookieContainer();
}
httpRequest.CookieContainer.Add(cookie);
return true;
}
This is how you use the full code:
var cookieNameValues = new Dictionary<string, string>();
cookieNameValues.Add("varName", "varValue");
var htmlResult = DownloadString(url, Encoding.UTF8, cookieNameValues);
How can I use the cookies from a response in a new request?
So basically I have an if statement within my getresponse stream for redirects,
Example of code -
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream stream2 = response.GetResponseStream())
{
if ((int)response.StatusCode >= 300 && (int)response.StatusCode <= 399)
{
string newurl = "https://www.example.com/page2";
request = request = (HttpWebRequest)WebRequest.Create(newurl);
}
using (StreamReader reader = new StreamReader(stream2, Encoding.UTF8))
{
str6 = reader.ReadToEnd();
}
}
return str6;
}
How can I apply the response cookies/header data to my new request -
request = request = (HttpWebRequest)WebRequest.Create(newurl);
I know if I do
response.Headers["Location"];
it'll give me the response location, but what about cookies? & how could i apply those cookies to the request
var myCookie = new HttpCookie("token");
myCookie.Value = Guid.NewGuid().ToString();
myCookie.Expires = DateTime.UtcNow.AddHours(10);
response.Cookies.Add(myCookie);
I try loop cookie container with loop, but its return error. How correct convert CookieContainer to string
foreach (Cookie item in cookieContainer)
{
var data = item.Value + "=" + item.Name;
}
Error 2 The foreach statement can not be used for variables of type
"System.Net.CookieContainer",
If you are only interested in the cookies for a specific domain, then you can iterate using the GetCookies() method.
var cookieContainer = new CookieContainer();
var testCookie = new Cookie("test", "testValue");
var uri = new Uri("https://www.google.com");
cookieContainer.Add(uri, testCookie);
foreach (var cookie in cookieContainer.GetCookies(uri))
{
Console.WriteLine(cookie.ToString()); // test=testValue
}
If your interested in getting all the cookies, then you may need to use reflection as provided by this answer.
Sample :
public static void Main(string[] args)
{
if (args == null || args.Length != 1)
{
Console.WriteLine("Specify the URL to receive the request.");
Environment.Exit(1);
}
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(args[0]);
request.CookieContainer = new CookieContainer();
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
// Print the properties of each cookie.
foreach (Cookie cook in response.Cookies)
{
// Show the string representation of the cookie.
Console.WriteLine ("String: {0}", cook.ToString());
}
}
I'm trying to log into Amazon Seller Central account and then return the HTML from the orders page (which requires a login). My streamreader is returning null, so something must be wrong with my code or my idea of doing this is flawed.
In the first section of this code before the ConsoleReadline(). I'm POST my username and password and then I receive a cookie, which shows up in the console, but I believe, I've got the 2nd part wrong when I try to retrieve the HTML of the next URI string.
CODE
using System;
using System.IO;
using System.Text;
using System.Net;
namespace HandleWebRequest
{
class Program
{
static void Main(string[] args)
{
string loginUri = "https://sellercentral.amazon.com/gp/homepage.html?";
string orderUri =
"https://sellercentral.amazon.com/gp/orders-v2/list/ref=ag_myo_wos3_home?byDate=shipDate&statusFilter=ItemsToShip&searchFulfillers=mfn&ignoreSearchType=1&searchType=OrderStatus&_encoding=UTF8&searchDateOption=preSelected&sortBy=OrderStatusDescending&shipSearchDateOption=noTimeLimit";
string username = "myusernmae";
string password = "mypassword";
string reqString = "username=" + username + "&password=" + password;
byte[] requestData = Encoding.UTF8.GetBytes(reqString);
CookieContainer cc = new CookieContainer();
var request = (HttpWebRequest) WebRequest.Create(loginUri);
request.Proxy = null;
request.CookieContainer = cc;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = requestData.Length;
using (Stream s = request.GetRequestStream())
s.Write(requestData, 0, requestData.Length);
using (var response = (HttpWebResponse)request.GetResponse())
foreach (Cookie c in response.Cookies)
{
Console.WriteLine(c.Name + " = " + c.Value);
Console.WriteLine(c.Domain);
}
Console.ReadLine();
request = (HttpWebRequest) WebRequest.Create(orderUri);
request.CookieContainer = cc;
using (WebResponse response = request.GetResponse())
using (Stream s = response.GetResponseStream())
using (StreamReader sr = new StreamReader(s))
File.WriteAllText("code.html", sr.ReadToEnd());
System.Diagnostics.Process.Start("code.html");
}
}
}
response is a new object; it doesn't just inherit the cookie container you put into request. You need to feed the cookies from response.Cookies into your CookieContainer, otherwise the next request won't have them:
foreach (Cookie c in response.Cookies)
cc.Add(new Uri(this.m_HttpWebResponse.ResponseUri.Scheme + "://" + c.Domain), c);
request = (HttpWebRequest) WebRequest.Create(orderUri);
request.CookieContainer = cc;
I am attempting to load a page I've received from an RSS feed and I receive the following WebException:
Cannot handle redirect from HTTP/HTTPS protocols to other dissimilar ones.
with an inner exception:
Invalid URI: The hostname could not be parsed.
I wrote a code that would attempt loading the url via an HttpWebRequest. Due to some suggestions I received, when the HttpWebRequest fails I then set the AllowAutoRedirect to false and basically manually loop through the iterations of redirect until I find out what ultimately fails. Here's the code I'm using, please forgive the gratuitous Console.Write/Writeline calls:
Uri url = new Uri(val);
bool result = true;
System.Net.HttpWebRequest req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(url);
string source = String.Empty;
Uri responseURI;
try
{
using (System.Net.WebResponse webResponse = req.GetResponse())
{
using (HttpWebResponse httpWebResponse = webResponse as HttpWebResponse)
{
responseURI = httpWebResponse.ResponseUri;
StreamReader reader;
if (httpWebResponse.ContentEncoding.ToLower().Contains("gzip"))
{
reader = new StreamReader(new GZipStream(httpWebResponse.GetResponseStream(), CompressionMode.Decompress));
}
else if (httpWebResponse.ContentEncoding.ToLower().Contains("deflate"))
{
reader = new StreamReader(new DeflateStream(httpWebResponse.GetResponseStream(), CompressionMode.Decompress));
}
else
{
reader = new StreamReader(httpWebResponse.GetResponseStream());
}
source = reader.ReadToEnd();
reader.Close();
}
}
req.Abort();
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(source);
result = true;
}
catch (ArgumentException ae)
{
Console.WriteLine(url + "\n--\n" + ae.Message);
result = false;
}
catch (WebException we)
{
Console.WriteLine(url + "\n--\n" + we.Message);
result = false;
string urlValue = url.ToString();
try
{
bool cont = true;
int count = 0;
do
{
req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(urlValue);
req.Headers.Add("Accept-Language", "en-us,en;q=0.5");
req.AllowAutoRedirect = false;
using (System.Net.WebResponse webResponse = req.GetResponse())
{
using (HttpWebResponse httpWebResponse = webResponse as HttpWebResponse)
{
responseURI = httpWebResponse.ResponseUri;
StreamReader reader;
if (httpWebResponse.ContentEncoding.ToLower().Contains("gzip"))
{
reader = new StreamReader(new GZipStream(httpWebResponse.GetResponseStream(), CompressionMode.Decompress));
}
else if (httpWebResponse.ContentEncoding.ToLower().Contains("deflate"))
{
reader = new StreamReader(new DeflateStream(httpWebResponse.GetResponseStream(), CompressionMode.Decompress));
}
else
{
reader = new StreamReader(httpWebResponse.GetResponseStream());
}
source = reader.ReadToEnd();
if (string.IsNullOrEmpty(source))
{
urlValue = httpWebResponse.Headers["Location"].ToString();
count++;
reader.Close();
}
else
{
cont = false;
}
}
}
} while (cont);
}
catch (UriFormatException uriEx)
{
Console.WriteLine(urlValue + "\n--\n" + uriEx.Message + "\r\n");
result = false;
}
catch (WebException innerWE)
{
Console.WriteLine(urlValue + "\n--\n" + innerWE.Message+"\r\n");
result = false;
}
}
if (result)
Console.WriteLine("testing successful");
else
Console.WriteLine("testing unsuccessful");
Since this is currently just test code I hardcode val as http://rss.nytimes.com/c/34625/f/642557/s/3d072012/sc/38/l/0Lartsbeat0Bblogs0Bnytimes0N0C20A140C0A70C30A0Csarah0Ekane0Eplay0Eamong0Eofferings0Eat0Est0Eanns0Ewarehouse0C0Dpartner0Frss0Gemc0Frss/story01.htm
the ending url that gives the UriFormatException is: http:////www-nc.nytimes.com/2014/07/30/sarah-kane-play-among-offerings-at-st-anns-warehouse/?=_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&_php=true&_type=blogs&partner=rss&emc=rss&_r=6&
Now I'm sure if I'm missing something or if I'm doing the looping wrong, but if I take val and just put that into a browser the page loads fine, and if I take the url that causes the exception and put it in a browser I get taken to an account login for nytimes.
I have a number of these rss feed urls that are resulting in this problem. I also have a large number of these rss feed urls that have no problem loading at all. Let me know if there is any more information needed to help resolve this. Any help with this would be greatly appreciated.
Could it be that I need to have some sort of cookie capability enabled?
You need to keep track of the cookies while doing all your requests. You can use an instance of the CookieContainer class to achieve that.
At the top of your method I made the following changes:
Uri url = new Uri(val);
bool result = true;
// keep all our cookies for the duration of our calls
var cookies = new CookieContainer();
System.Net.HttpWebRequest req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(url);
// assign our CookieContainer to the new request
req.CookieContainer = cookies;
string source = String.Empty;
Uri responseURI;
try
{
And in the exception handler where you create a new HttpWebRequest, you do the assignment from our CookieContainer again:
do
{
req = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(urlValue);
// reuse our cookies!
req.CookieContainer = cookies;
req.Headers.Add("Accept-Language", "en-us,en;q=0.5");
req.AllowAutoRedirect = false;
using (System.Net.WebResponse webResponse = req.GetResponse())
{
This makes sure that on each successive call the already present cookies are resend again in the next request. If you leave this out, no cookies are sent and therefore the site you try to visit assumes you are a fresh/new/unseen user and gives you a kind of authentication path.
If you want to store/keep cookies beyond this method you could move the cookie instance variable to a static public property so you can use all those cookies program-wide like so:
public static class Cookies
{
static readonly CookieContainer _cookies = new CookieContainer();
public static CookieContainer All
{
get
{
return _cookies;
}
}
}
And to use it in a WebRequest:
var req = (System.Net.HttpWebRequest) WebRequest.Create(url);
req.CookieContainer = Cookies.All;