Develop Reference

http get works in browser and postman but get a 401 using c# httpwebrequest

I have a web app hosted in azure. When I use postman to make the request I get a
json result, which is the correcet response. When I try to make the same request via C# using the same token I receive a errpr - The remote server returned an error: (401) Unauthorized.
here is the code I use to make the request.
public string RequestData(string queryString, string token)
{
var request = (HttpWebRequest)WebRequest.Create(queryString);
request.Proxy = GetProxy();
request.Credentials = CredentialCache.DefaultCredentials;
request.PreAuthenticate = true;
request.UseDefaultCredentials = true;
request.Method = "GET";
request.ContentType = "application/json";
request.ContentLength = 0;
request.CookieContainer = new CookieContainer();
request.Headers.Add("authorization", "Bearer " + token);
using (var webresponse = request.GetResponse())
{
if (webresponse.GetResponseStream() == Stream.Null)
{
throw new Exception("Response stream is empty");
}
var response = (HttpWebResponse)webresponse;
if (response.StatusCode != HttpStatusCode.OK)
{
return response.StatusCode.ToString();
}
else
{
return response.StatusCode.ToString();
}
}
}
I have double checked the token to ensure it is correct and it is.
Another point I wanted to mention is that it did not work initially in
Postman without enabling Interceptor. This goes for Advanced Rest Client.
The request did not work until I enabled "XHR" and installed ARC cookie exchange.
I have checked the request headers in Fiddler and noticed there are no additional headers except for the authorization one (which I add as well).
UPDATE:
I got a successfull response in Postman (https://www.getpostman.com/)
and ran the code it generated for c# using RestSharp. In the response
the error thrown was
"You do not have permission to view this directory or page."
Which points to the token not being correct. Which is confusing since it works
in Postman and Advanced Rest Client. Also I must mention I retrieve the token
on each call using the clientid and secret using the following code:
public async static Task<AzureAccessToken> CreateOAuthAuthorizationToken(string clientId, string clientSecret, string resourceId, string tenantId)
{
AzureAccessToken token = null;
var oauthUrl=string.Format("https://login.microsoftonline.com/{0}/oauth2/token", tenantId);
var reqBody = String.Format("grant_type=client_credentials&client_id={0}&client_secret={1}",clientId, clientSecret);
var client = new HttpClient();
HttpContent content = new StringContent(reqBody);
content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/x-www-form-urlencoded");
using (HttpResponseMessage response = await client.PostAsync(oauthUrl, content))
{
if (response.IsSuccessStatusCode)
{
DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(AzureAccessToken));
Stream json = await response.Content.ReadAsStreamAsync();
token = (AzureAccessToken)serializer.ReadObject(json);
return token;
}
return null;
}
}

after checking the log in azure, I saw the following error message:
JWT validation failed: IDX10214: Audience validation failed. Audiences: '00000002-0000-0000-c000-000000000000'. Did not match: validationParameters.ValidAudience: 'f50a9d02-b8f4-408f-aaf8-0046e6cbf7a6' or validationParameters.ValidAudiences: 'null'.
I resolved the issue by adding '00000002-0000-0000-c000-000000000000' to the "Allowed Token Audiences" under Azure Active Directory Settings.

I have called third party API. When I use postman to make the request I get a json result, which is the correct response. When I try to make the same request via C# using the same token I receive a error - The remote server returned an error: (401) Unauthorized. Finally I got the solution.
When I make the login request some cookies will send by the server and that cookie will store in postman. If you see code snippet you will see information about request that is raised by postman.
When I call the Login method I stored the cookies like below:
public ResponseData OnGetResponseFromAPI(string URL, string Method, string PostData = null, Dictionary<string, string> Headers = null, string body = null, string ContentType = "application/json")
{
ResponseData response = new ResponseData();
try
{
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
var webRequest = (HttpWebRequest)WebRequest.Create(URL);
CookieContainer cookieJar = new CookieContainer();
webRequest.CookieContainer = cookieJar;
webRequest.Method = Method;
webRequest.ContentType = ContentType;
if (Method == "GET")
{
var type = webRequest.GetType();
var currentMethod = type.GetProperty("CurrentMethod", BindingFlags.NonPublic | BindingFlags.Instance).GetValue(webRequest);
var methodType = currentMethod.GetType();
methodType.GetField("ContentBodyNotAllowed", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(currentMethod, false);
}
if (Headers == null)
Headers = new Dictionary<string, string>();
foreach (KeyValuePair<string, string> header in Headers)
{
webRequest.Headers.Add(header.Key, header.Value);
}
if (!string.IsNullOrEmpty(PostData))
{
var RequestStream = new StreamWriter(webRequest.GetRequestStream());
RequestStream.Write(PostData);
RequestStream.Close();
}
if (!string.IsNullOrEmpty(body))
{
byte[] byteArray = Encoding.UTF8.GetBytes(body);
webRequest.ContentLength = byteArray.Length;
Stream dataStream = webRequest.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
}
var ResponseStream = new StreamReader(webRequest.GetResponse().GetResponseStream());
string cookie = string.Empty;
CookieCollection allCookies = cookieJar.GetCookies(webRequest.RequestUri);
foreach (Cookie c in allCookies)
{
cookie = cookie + c.Name + "=" + c.Value+";";
}
cookie = cookie.Substring(0, cookie.LastIndexOf(';'));
var ResponseData = ResponseStream.ReadToEnd();
response.response=ResponseData.ToString();
response.cookie=cookie;
return response;
}
catch (WebException webException)
{
if (webException == null || webException.Response == null)
return null;
var responseStream = webException.Response.GetResponseStream() as MemoryStream;
if (responseStream == null)
return null;
var responseBytes = responseStream.ToArray();
var responseString = Encoding.UTF8.GetString(responseBytes);
response.response = responseString;
return response;
}
}
Whenever I am calling any api method I am sending token and cookie in header like below:
public string DownLoadDocument( string FilePath, string FileName, string token,string cookie)
{
try
{
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
HttpWebRequest webRequest;
webRequest = (HttpWebRequest)WebRequest.Create(URL);
webRequest.Method = "GET";
webRequest.ContentType = "application/octet-stream;charset=UTF-8";
webRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1";
webRequest.Headers.Add("Cookie", cookie);
webRequest.Headers.Add("Authentication", "Bearer "+token);
webRequest.Headers.Add("Content-Disposition", "attachment");
Stream responseReader = webRequest.GetResponse().GetResponseStream();
using (var fs = new FileStream(FilePath, FileMode.Create))
{
responseReader.CopyTo(fs);
}
}
catch (Exception ex)
{
throw;
}
return FilePath;
}

Webclient UploadValues encoding

Im making a POST in a URL and getting the response as a string like this:
var responseBytes = client.UploadValues("www.example.com", requestData.Body);
var html = Encoding.UTF8.GetString(responseBytes, 0, responseBytes.Length);
html = HttpUtility.HtmlDecode(html);
Ive created a WebClient child to accept cookies so i made a this class:
public class CookieWebClient : WebClient
{
public CookieContainer CookieContainer { get; }
public CookieWebClient()
{
CookieContainer = new CookieContainer();
Encoding = Encoding.UTF8;
}
public CookieWebClient(CookieContainer cookieContainer)
{
CookieContainer = cookieContainer;
}
protected override WebRequest GetWebRequest(Uri address)
{
var request = base.GetWebRequest(address) as HttpWebRequest;
if (request == null) return base.GetWebRequest(address);
request.CookieContainer = CookieContainer;
return request;
}
}
As you can see i set the encode in the constructor. But when i get the html property some characters are coming like: � necess�rio and it should be É necessário.
Any ideas whats wrong?

You're using "Latin 1" characters (portuguese, maybe?), so you should go with:
var html = Encoding.GetEncoding("ISO-8859-1")
.GetString (responseBytes, 0, responseBytes.Length);

Get cookies from httpwebrequest

I'm trying to get all cookies from a website using this code
CookieContainer cookieJar = new CookieContainer();
var request = (HttpWebRequest)HttpWebRequest.Create("http://www.foetex.dk/ugenstilbud/Pages/Zmags.aspx");
request.CookieContainer = cookieJar;
var response = request.GetResponse();
foreach (Cookie c in cookieJar.GetCookies(request.RequestUri))
{
Console.WriteLine("Cookie['" + c.Name + "']: " + c.Value);
}
Console.ReadLine();
The only thing i want is to display with console.writeline, but im not getting a single of them.

//Please use this,
HttpWebRequest request = null;
request = HttpWebRequest.Create(StringURL) as HttpWebRequest;
HttpWebResponse TheRespone = (HttpWebResponse)request.GetResponse();
String setCookieHeader = TheRespone.Headers[HttpResponseHeader.SetCookie];

The following example uses the HttpCookie class and its properties to read a cookie with a specific name.
HttpCookie myCookie = new HttpCookie("MyTestCookie");
myCookie = Request.Cookies["MyTestCookie"];
// Read the cookie information and display it.
if (myCookie != null)
Response.Write("<p>"+ myCookie.Name + "<p>"+ myCookie.Value);
else
Response.Write("not found");
Retrieve the values in the HttpWebResponse.Cookies property of HttpWebResponse. In this example, the cookies are retrieved and saved to isolated storage:
private void ReadCallback(IAsyncResult asynchronousResult)
{
HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
HttpWebResponse response = (HttpWebResponse)
request.EndGetResponse(asynchronousResult);
using (IsolatedStorageFile isf =
IsolatedStorageFile.GetUserStoreForSite())
{
using (IsolatedStorageFileStream isfs = isf.OpenFile("CookieExCookies",
FileMode.OpenOrCreate, FileAccess.Write))
{
using (StreamWriter sw = new StreamWriter(isfs))
{
foreach (Cookie cookieValue in response.Cookies)
{
sw.WriteLine("Cookie: " + cookieValue.ToString());
}
sw.Close();
}
}
}
}

To get the list of cookies, you can use the below method;
private async Task<List<Cookie>> GetCookies(string url)
{
var cookieContainer = new CookieContainer();
var uri = new Uri(url);
using (var httpClientHandler = new HttpClientHandler
{
CookieContainer = cookieContainer
})
{
using (var httpClient = new HttpClient(httpClientHandler))
{
await httpClient.GetAsync(uri);
return cookieContainer.GetCookies(uri).Cast<Cookie>().ToList();
}
}
}

How to pass cookies to HtmlAgilityPack or WebClient?

I use this code to login:
CookieCollection cookies = new CookieCollection();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("example.com");
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
cookies = response.Cookies;
string getUrl = "example.com";
string postData = String.Format("my parameters");
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies);
getRequest.Method = WebRequestMethods.Http.Post;
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0";
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
byte[] byteArray = Encoding.ASCII.GetBytes(postData);
getRequest.ContentLength = byteArray.Length;
Stream newStream = getRequest.GetRequestStream();
newStream.Write(byteArray, 0, byteArray.Length);
newStream.Close();
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream(), Encoding.GetEncoding("windows-1251")))
{
doc.LoadHtml(sr.ReadToEnd());
webBrowser1.DocumentText = doc.DocumentNode.OuterHtml;
}
then I want to use HtmlWeb (HtmlAgilityPack) or Webclient to parse the HTML to HtmlDocument(HtmlAgilityPack).
My problem is that when I use:
WebClient wc = new WebClient();
webBrowser1.DocumentText = wc.DownloadString(site);
or
doc = web.Load(site);
webBrowser1.DocumentText = doc.DocumentNode.OuterHtml;
The login disappear so i think I must somehow pass the cookies.. Any suggestions?

Check HtmlAgilityPack.HtmlDocument Cookies
Here is an example of what you're looking for (syntax not 100% tested, I just modified some class I usually use):
public class MyWebClient
{
//The cookies will be here.
private CookieContainer _cookies = new CookieContainer();
//In case you need to clear the cookies
public void ClearCookies() {
_cookies = new CookieContainer();
}
public HtmlDocument GetPage(string url) {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
//Set more parameters here...
//...
//This is the important part.
request.CookieContainer = _cookies;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
var stream = response.GetResponseStream();
//When you get the response from the website, the cookies will be stored
//automatically in "_cookies".
using (var reader = new StreamReader(stream)) {
string html = reader.ReadToEnd();
var doc = new HtmlDocument();
doc.LoadHtml(html);
return doc;
}
}
}
Here is how you use it:
var client = new MyWebClient();
HtmlDocument doc = client.GetPage("http://somepage.com");
//This request will be sent with the cookies obtained from the page
doc = client.GetPage("http://somepage.com/another-page");
Note: If you also want to use POST method, just create a method similar to GetPage with the POST logic, refactor the class, etc.

There are some recommendations here: Using CookieContainer with WebClient class
However, it's probably just easier to keep using the HttpWebRequest and set the cookie in the CookieContainer:
HTTPWebRequest and CookieContainer
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.cookiecontainer.aspx
The code looks something like this:
// Create a HttpWebRequest
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(getUrl);
// Create the cookie container and add a cookie
request.CookieContainer = new CookieContainer();
// Add all the cookies
foreach (Cookie cookie in response.Cookies)
{
request.CookieContainer.Add(cookie);
}
The second thing is that you don't need to download the site again, since you already have it from your web response and you're saving it here:
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream(), Encoding.GetEncoding("windows-1251")))
{
webBrowser1.DocumentText = doc.DocumentNode.OuterHtml;
}
You should be able to just take the HTML and parse it with the HTML Agility Pack:
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(webBrowser1.DocumentText);
And that should do it... :)

Try caching cookies from previous response locally and resend them each web request as follows:
private CookieCollection cookieCollection;
...
parserObject = new HtmlWeb
{
AutoDetectEncoding = true,
PreRequest = request =>
{
if (cookieCollection != null)
cookieCollection.Cast<Cookie>()
.ForEach(cookie => request.CookieContainer.Add(cookie));
return true;
},
PostResponse = (request, response) => { cookieCollection = response.Cookies; }
};

Http request to web service in java

I have the following method in a C# project which builds a web request to consume a web service:
private HttpWebRequest BuildRequest(string fullUri, object param, RequestMethod method, Dictionary<string, string> requestHeaders)
{
HttpWebRequest request = WebRequest.Create(fullUri) as HttpWebRequest;
request.Method = method.ToString().ToUpper();
request.Accept = "application/json";
request.ContentType = "application/json";
foreach (var h in requestHeaders)
{
request.Headers.Add(string.Format("{0}: {1}", h.Key, h.Value));
}
if (method == RequestMethod.Post)
{
JavaScriptSerializer ser = new JavaScriptSerializer();
string postData = ser.Serialize(param);
request.ContentLength = postData.Length;
Stream s = request.GetRequestStream();
StreamWriter sw = new StreamWriter(s);
sw.Write(postData);
sw.Flush();
}
return request;
}
Now I am wanting to do the same thing in Java but cant quite figure out how. So far I have...
private HttpRequestBase BuildRequest(String fullUri, Object param,
RequestMethod method, HashMap<String, String> requestHeaders) {
HttpRequestBase request = null;
if(method == RequestMethod.GET) {
request = new HttpGet(fullUri);
} else if (method == RequestMethod.POST) {
request = new HttpPost(fullUri);
}
request.addHeader(new BasicHeader("Accept", "application/json"));
request.addHeader(new BasicHeader("ContentType", "application/json"));
for(Map.Entry<String, String> header : requestHeaders.entrySet()) {
request.addHeader(new BasicHeader(header.getKey(), header.getValue()));
}
return request;
}
As far as I can tell that looks like it will work but I need to get the request stream like I do in the C# version in order to send an object as json but cant see how to get it. So my question is how to get it from a HttpRequestBase or any other object that I can use to build a HttpRequest in java.
Thanks.
FYI.. RequestMethod is an Enum either GET or POST"
And this is in an android application so I am able to use anything in the android sdk.