Hello I'm trying to simulate a browser and this specific URL is not working well:
https://www.btgpactualdigital.com/fundos-de-investimento/spx-nimitz-fic-fim-access
It returns always the same response code... I'm using for test purpose this simple code:
using (var client = new WebClient())
{
client.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0";
client.Headers[HttpRequestHeader.Accept] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
client.Headers[HttpRequestHeader.AcceptLanguage] = "en-us,en;q=0.5";
var responseStream = new GZipStream(client.OpenRead(url), CompressionMode.Decompress);
var reader = new StreamReader(responseStream);
textResponse = reader.ReadToEnd();
}
But it is always returning a different code from I can see on browser... I got all Headers from browser to try simulate same scenario.
Any suggestion ? Thanks !
Related
So I am trying to connect to http://cmyip.org because the content contains information about the proxy you are connecting with. Which is perfect because I am learning about making web-requests with proxies so it will show me if I actually connected with the proxy or not.
However instead of getting the content of the website I am making a request to I am getting the content of the proxy html
Before start thread
<html><head><title>Wowza Streaming Engine 4 Subscription Edition 4.7.2.01 build2
1094</title></head><body>Wowza Streaming Engine 4 Subscription Edition 4.7.2.01
build21094</body></html>
Working.
Why is it showing the contents of the proxy and not cmyip.org? I'm sure I am just too close to the code to see it but could really use some help.
If you try to connect to 192.99.46.182:1935 on your web browser you will see the same thing.
public static bool TestProxy(string proxyIP, int proxyPort)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.cmyip.org/");
WebProxy myproxy = new WebProxy(proxyIP, proxyPort);
myproxy.BypassProxyOnLocal = false;
request.Proxy = myproxy;
request.Method = "GET";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36";
request.Timeout = 2000;
try
{
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
string Content = sr.ReadToEnd();
console.WriteLine(Content);
}
}
catch (Exception)
{
return false;
}
return true;
}
private static void SimpleTest()
{
var aBool = TestProxy("192.99.46.182", 1935);
if (aBool == true)
{
Console.WriteLine("Working.");
}
Console.ReadLine();
}
I have this code that creates a request and reads the data but it is always empty
static string uri = "http://yiimp.ccminer.org/api/wallet?address=DshDF3zmCX9PUhafTAzxyQidwgdfLYJkBrd";
static void Main(string[] args)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
// Get the stream associated with the response.
Stream receiveStream = response.GetResponseStream();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
Console.WriteLine("Response stream received.");
Console.WriteLine(readStream.ReadToEnd());
response.Close();
readStream.Close();
}
When i try to access the link from browser i get this json:
{"currency": "DCR", "unsold": 0.030825917365192, "balance": 0.02007306, "unpaid": 0.05089898, "paid24h": 0.05796425, "total": 0.10886323}
what am I missing?
When you perform the request from a browser there are a lot of headers that get sent to the web service. Apparently this web service validates the UserAgent. This is a decision on the part of the web service implementation, they might not want you to programmatically access it.
var client = (HttpWebRequest)HttpWebRequest.Create(new Uri("http://yiimp.ccminer.org/api/wallet?address=DshDF3zmCX9PUhafTAzxyQidwgdfLYJkBrd"));
client.AutomaticDecompression = DecompressionMethods.GZip;
client.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063";
client.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate";
client.Host = "yiimp.ccminer.org";
client.KeepAlive = true;
using (var s = client.GetResponse().GetResponseStream())
using (var sw = new StreamReader(s))
{
var ss = sw.ReadToEnd();
Console.WriteLine(ss);
}
Sending the headers seems to make it work.
My current job is to consume a RESTful API with OAuth2. Currently I worked out how to get the access token and it is working ok while I use the chrome extension Rest Console, but when I try to do it from my application I always get the error that I am sending an invalid OAuth request. Below you can see three of the ways I tried to consume the API, but to no success. The page always returns error 500. Any help will be appreciated, if I had missed something crucial.
var auth = "Bearer " + item.access_token;
/* First Attempt */
var client = new RestClient("http://<link>");
var request = new RestRequest("sample", Method.GET);
request.AddHeader("Authorization", auth);
request.AddHeader("Content-Type", "application/json;charset=UTF-8");
request.AddHeader("Pragma", "no-cache");
request.AddHeader("User-Agent", "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36");
request.AddHeader("Accept", "application/json");
request.RequestFormat = DataFormat.Json;
var response = client.Execute(request);
var content = response.Content;
/* Second Attempt */
string sURL = "http://<link>/sample";
string result = "";
using (WebClient client = new WebClient())
{
client.Headers["Authorization"] = auth;
client.Headers["Content-Type"] = "application/json;charset=UTF-8";
client.Headers["Pragma"] = "no-cache";
client.Headers["User-Agent"] = "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36";
client.Headers["Accept"] = "application/json";
byte[] byteArray = Encoding.UTF8.GetBytes(parameters);
var result1 = client.DownloadString(sURL);
}
/* Third Attempt */
var request = (HttpWebRequest)WebRequest.Create(sURL);
request.Method = "GET";
request.ContentType = "application/json;charset=UTF-8";
request.Accept = "application/json";
request.Headers["Authorization"] = auth;
request.UserAgent = "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36";
string content;
HttpStatusCode statusCode;
using (var response = request.GetResponse())
using (var stream = response.GetResponseStream())
{
var contentType = response.ContentType;
Encoding encoding = null;
if (contentType != null)
{
var match = Regex.Match(contentType, #"(?<=charset\=).*");
if (match.Success)
encoding = Encoding.GetEncoding(match.ToString());
}
encoding = encoding ?? Encoding.UTF8;
statusCode = ((HttpWebResponse)response).StatusCode;
using (var reader = new StreamReader(stream, encoding))
content = reader.ReadToEnd();
}
--------EDIT--------
For the first attempt I also tried to add the authentication to the client variable client.Authenticator = Authenticate; where OAuth2AuthorizationRequestHeaderAuthenticator Authenticate = new OAuth2AuthorizationRequestHeaderAuthenticator(item.access_token, item.token_type);
The code seems right. The fail attempts you did suggest that the issue is with the token and not the code. Bearer tokens have expiration time. So semms like your token expired between the first time you got it using chrome REST Console extension and when you wrote your code. But the strange situation here is the 500 error code you got. 401 is standard response when you token expired or not exist. 500 error code always mean server error.
Here is my code to get the xml document from a url that is passed in.
var request = WebRequest.Create(url);
request.Method = "GET";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = 0;
var response = request.GetResponse(); // Error is thrown here
When I copy and paste the url into my browser it works just fine.
Here is the complete xml that is returned
<Model>
<Item>
<Id>7908</Id>
</Item>
</Model>
Is the xml in an incorrect format? I have tried changing the content type to be application/xml but I still get this error.
EDIT=======================================================
I am trying to use webclient using this code:
using (var wc = new System.Net.WebClient())
{
wc.Headers["Method"] = "GET";
wc.Headers["ContentType"] = "text/xml;charset=\"utf-8\"";
wc.Headers["Accept"] = "text/xml, */*";
wc.Headers["User-Agent"] = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; .NET CLR 3.5.30729;)";
wc.Headers[HttpRequestHeader.AcceptLanguage] = "en-us";
wc.Headers["KeepAlive"] = "true";
wc.Headers["AutomaticDecompression"] = (DecompressionMethods.Deflate | DecompressionMethods.GZip).ToString();
var response = wc.DownloadString(url);
}
The response string is empty!!! Any ideas why this is not returning any result but pasting the url into the browser returns the xml?
I finally got it working. I had to use this code:
using (var wc = new System.Net.WebClient())
{
wc.Headers["Method"] = "GET";
wc.Headers["Accept"] = "application/xml";
var response = wc.DownloadString(url);
}
The key was using the accept header of "application/xml" otherwise the response would come back empty.
This should hopefully do the trick:
try
{
using(var response = (HttpWebResponse)request.GetResponse())
{
// Do things
}
}
catch(WebException e)
{
// Handled!...
}
Try what Joel Lee suggested if this fails.
Why not use a WebClient instead.
public class MyWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
var request = base.GetWebRequest(address);
if (request.GetType() == typeof(HttpWebRequest)){
((HttpWebRequest)request).UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.17 Safari/537.36";
}
return request;
}
}
using(var wc = new MyWebClient()){
var response = wc.DownloadString(url);
//do stuff with response
}
I'm a little bit stuck here. I am busy making an Windows application that reads data from a website. However de website requires a login first and i don't seem to be able to get passes that. I'm fairly new to programming, so i hope someone know a solution.
This is the code i use to login:
private void btnLogin2_Click(object sender, EventArgs e)
{
HttpWebRequest request = WebRequest.Create(LoginPageURL) as HttpWebRequest;
request.KeepAlive = true;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.2 Safari/537.36";
string postData = "j_username=" + number + "&j_password=" + password;
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
request.ContentLength = dataBytes.Length;
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse httpResponse = request.GetResponse() as HttpWebResponse;
request = WebRequest.Create(Page2URL) as HttpWebRequest;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.2 Safari/537.36";
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(httpResponse.Cookies);
request.Method = "GET";
HttpWebResponse httpResponse2 = request.GetResponse() as HttpWebResponse;
StreamReader stream = new StreamReader(httpResponse2.GetResponseStream(), System.Text.Encoding.UTF8);
string result = stream.ReadToEnd();
stream.Close();
tbOutput2.Text = result;
}
The point is that i get the HTML of the page after the login (Page2URL). But i keep getting the HTML from the Login Page.
You are adding cookies to the request from first response:
request.CookieContainer.Add(httpResponse.Cookies);
Probably the cookies in response are null! To cope up with this issue, read cookie values from response header and add them to the next request like this:
string response_header_cookies = response.Headers.Get("Set-Cookie")
req.Headers.Add("Cookie",response_header_cookies);
In most of the cases this is the more efficient way. Hope this helps!
Source: msdn