I was given a json feed.
It does not need any user name or password to access.
It can be seen from google Chrome.
In the google Chrome developer tool, it also say status 401,
but still able to read data.
Then I try to use C# web request or webclient, i get 401 and thrown , no longer able to continue.
So how does Chrome do it? and what should i do?
My C# code:
WebRequest request = WebRequest.Create(url);
request.UseDefaultCredentials = true;
request.Method = WebRequestMethods.Http.Get;
((HttpWebRequest)request).UserAgent = userAgent;
request.Timeout = timeOut;
((HttpWebRequest)request).Accept = "application/json";
response = request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream, Encoding.UTF8);
You can "catch" the 401 exception that is thrown by HttpWebRequest and read the content off the WebException using code like this:
try {
// your code above
}
catch(Exception oEx)
{
if(oEx.GetType() == typeof(WebException))
{
HttpWebResponse wr = ((WebException)oEx).Response as HttpWebResponse;
Stream dataStream = wr.GetResponseStream();
StreamReader reader = new StreamReader(dataStream, Encoding.UTF8);
string str = reader.ReadToEnd(); // str will contain the body of the HTTP 401 response
reader.Close();
}
}
Related
I'm trying to use .net to put datapoints in OpenTSDB, using the HTTP /api/put API.
I've tried with httpclient, webRequest and HttpWebRequest. The outcome is always 400 - bad request: chunked request not supported.
I've tried my payload with an api tester (DHC) and works well.
I've tried to send a very small payload (even plain wrong, like "x") but the reply is always the same.
Here's one of my code instances:
public async static Task PutAsync(DataPoint dataPoint)
{
try
{
HttpWebRequest http = (HttpWebRequest)WebRequest.Create("http://127.0.0.1:4242/api/put");
http.SendChunked = false;
http.Method = "POST";
http.ContentType = "application/json";
Encoding encoder = Encoding.UTF8;
byte[] data = encoder.GetBytes( dataPoint.ToJson() + Environment.NewLine);
http.Method = "POST";
http.ContentType = "application/json; charset=utf-8";
http.ContentLength = data.Length;
using (Stream stream = http.GetRequestStream())
{
stream.Write(data, 0, data.Length);
stream.Close();
}
WebResponse response = http.GetResponse();
var streamOutput = response.GetResponseStream();
StreamReader sr = new StreamReader(streamOutput);
string content = sr.ReadToEnd();
Console.WriteLine(content);
}
catch (WebException exc)
{
StreamReader reader = new StreamReader(exc.Response.GetResponseStream());
var content = reader.ReadToEnd();
}
return ;
}
where I explicitly set to false the SendChunked property.
note that other requests, like:
public static async Task<bool> Connect(Uri uri)
{
HttpWebRequest http = (HttpWebRequest)WebRequest.Create("http://127.0.0.1:4242/api/version");
http.SendChunked = false;
http.Method = "GET";
// http.Headers.Clear();
//http.Headers.Add("Content-Type", "application/json");
http.ContentType = "application/json";
WebResponse response = http.GetResponse();
var stream = response.GetResponseStream();
StreamReader sr = new StreamReader(stream);
string content = sr.ReadToEnd();
Console.WriteLine(content);
return true;
}
work flawlessly.
I am sure I am doing something really wrong.
I'd like to to reimplement HTTP in Sockets from scratch.
I've found a solution I'd like to share here.
I've used wireshark to sniff my packets, and I've found that this header is added:
Expect: 100-continue\r\n
(see 8.2.3 of https://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html)
This is the culprit. I've read the post http://haacked.com/archive/2004/05/15/http-web-request-expect-100-continue.aspx/ by Phil Haack, and found that HttpWebRequest puts that header by default, unless you tell it to stop. In this article I've found that using ServicePointManager I can do just this.
Putting the following code on top of my method, when declaring the http object, makes it work very well, and solves my issue:
var uri = new Uri("http://127.0.0.1:4242/api/put");
var spm = ServicePointManager.FindServicePoint(uri);
spm.Expect100Continue = false;
HttpWebRequest http = (HttpWebRequest)WebRequest.Create(uri);
http.SendChunked = false;
I'm trying to do a HTTP GET request for a json file from an api in a C# application. I'm having trouble getting the authorization, request headers and the webresponse (.GetResponse not working).
The example on the api's site is in curl.
curl -H "Authorization: Bearer ACCESS_TOKEN" https://erikberg.com/nba/boxscore/20120621-oklahoma-city-thunder-at-miami-heat.json
Here is my request method, which will also include JSON deseralization
public static string HttpGet(string URI)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(URI);
// Not sure if the credentials input is the correct
string cred = $"{"Bearer"} {"ACCESS_TOKEN_IS_A_GUID"}";
req.Headers[HttpRequestHeader.Authorization] = cred;
req.Method = "GET";
// GetResponse() is "red", won't work.
WebResponse response = req.GetResponse();
using (Stream responseStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(responseStream, Encoding.UTF8);
return reader.ReadToEnd().Trim();
}
}
EDIT It was resolved. The problem was that the request was for a GZIP file and that had to be decompressed
var request = (HttpWebRequest)WebRequest.Create(requestUri);
request.UserAgent = userAgent;
request.ContentType = "application/json";
request.Method = WebRequestMethods.Http.Get;
request.Headers[HttpRequestHeader.Authorization] = bearer;
request.Headers[HttpRequestHeader.AcceptEncoding] = "gzip";
var response = (HttpWebResponse) request.GetResponse();
string jsonString;
using (var decompress = new GZipStream(response.GetResponseStream(), CompressionMode.Decompress))
{
using (var sr = new StreamReader(decompress))
jsonString = sr.ReadToEnd().Trim();
}
_Game = JsonConvert.DeserializeObject<Game>(jsonString);
You are not getting it because you don't have access.
The cURL command from API's site(that you mentioned in your question) gives the following JSON
{
"error" : {
"code" : "401",
"description" : "Invalid access token: ACCESS_TOKEN"
}
}
And so does the following code:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("URL");
req.UserAgent = "Bearer";
WebResponse response = req.GetResponse();
So what you need is a valid username/password or userAgent. You might want to contact the site for that.
I'm trying to scrape a website that requires a login. Getting an error that I haven't received before, copied the code from another forum successfully in the past:
Exception Details: System.Net.ProtocolViolationException: Cannot send a content-body with this verb-type.
with the code:
Stream newStream = http.GetRequestStream(); //open connection
Here's the entire code:
#{
var strUserId = "userName";
var strPassword = "password";
var url = "formSubmitLandingSite";
var url2 = "pageToScrape";
HttpWebRequest http = WebRequest.Create(url) as HttpWebRequest;
http.KeepAlive = true;
http.Method = "POST";
http.ContentType = "application/x-www-form-urlencoded";
string postData = "email=" + strUserId + "&password=" + strPassword;
byte[] dataBytes = UTF8Encoding.UTF8.GetBytes(postData);
http.ContentLength = dataBytes.Length;
using (Stream postStream = http.GetRequestStream())
{
postStream.Write(dataBytes, 0, dataBytes.Length);
}
HttpWebResponse httpResponse = http.GetResponse() as HttpWebResponse;
// Probably want to inspect the http.Headers here first
http = WebRequest.Create(url2) as HttpWebRequest;
http.CookieContainer = new CookieContainer();
http.CookieContainer.Add(httpResponse.Cookies);
HttpWebResponse httpResponse2 = http.GetResponse() as HttpWebResponse;
Stream newStream = http.GetRequestStream(); //open connection
newStream.Write(dataBytes, 0, dataBytes.Length); // Send the data.
newStream.Close();
string sourceCode;
HttpWebResponse getResponse = (HttpWebResponse)http.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
sourceCode = sr.ReadToEnd();
}
Response.Write(sourceCode);
}
You're creating a new request object here:
http = WebRequest.Create(url2) as HttpWebRequest;
Keep in mind that the default HTTP verb used is GET. Then you try to open the request stream here:
Stream newStream = http.GetRequestStream();
This method is used to enable writing data to the request's content. However, GET requests don't have content. As you do in the code above the error, you'll need to use a different HTTP verb. POST is most common for this, and is what you're using above:
http.Method = "POST";
So just use a POST request again. (Assuming, of course, that's what the server is expecting. In any event, if the server is expecting content then it's definitely not expecting a GET request.)
I'm trying to solve an issue that I Mostly (70%) have (30% is succesfull).
I trying to do a webrequest (POST) with the following code:
private string HttpWebRequest(string busStopCode)
{
//XML input
string xml = "<?xml version='1.0' encoding='UTF-8' standalone='yes'?><Siri version='1.0' xmlns='http://www.siri.org.uk/'><ServiceRequest> <RequestTimestamp>2011-10-24T15:09:12Z</RequestTimestamp><RequestorRef><username></RequestorRef><StopMonitoringRequest version='1.0'> <RequestTimestamp>2011-10-24T15:09:12Z</RequestTimestamp><MessageIdentifier>12345</MessageIdentifier><MonitoringRef>"+busStopCode+"</MonitoringRef></StopMonitoringRequest></ServiceRequest></Siri>";
string responseFromServer = null;
// Create a request using a URL that can receive a post.
WebRequest request = WebRequest.Create("http://<username>:<username>#nextbus.mxdata.co.uk/nextbuses/1.0/1");
// Set the Method property of the request to POST.
request.Method = "POST";
request.Credentials = CredentialCache.DefaultNetworkCredentials;
// Create POST data and convert it to a byte array.
string postData = xml;
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
Stream dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
// Get the response.
WebResponse response = null;
while (response == null)
{
try
{
response = request.GetResponse();
}
catch (Exception e)
{
MessageBox.Show(e.ToString());
}
}
// Display the status.
MessageBox.Show(((HttpWebResponse)response).StatusDescription + " Completed");
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content.
responseFromServer = reader.ReadToEnd();
// Clean up the streams.
reader.Close();
dataStream.Close();
response.Close();
return responseFromServer;
}
When I call this function I get mostly a messagebox of my exception with:
"System.Net.WebException: The remote Server returned an error(401) not authorized with System.Net.Http.Webrequest.GetResponse() with WindowsFormApplication1.Form1.HttpWebRequest(string BusstopCode) in <my pathfile>...."
What I'm doing wrong?
I already tried several solutions from previous threads but without success...
Thanks!
I am trying to read xml from the url(well its not url its restful service) and below is my code. I dont know what the problem is but Iam unable to read the xml from url. I am getting error and I am not getting any data..
string soap = "";
if (fPortalGuid != string.Empty)
{
HttpWebRequest request = WebRequest.Create("http://cramapp-dt-02s.cable.comcast.com:8158/restfulqueryservice/queryservice/getrequestdetails?api_key=......&request_id=33ebc6e9-9def-4f39-adf9-bba2edef3b54") as HttpWebRequest;
//request.Credentials = new NetworkCredential("rolland", "409cleaner");
//request.ContentType = "application / json; charset = utf - 8";
request.Method = "POST";
using (Stream stm = request.GetRequestStream())
{
using (StreamWriter stmw = new StreamWriter(stm))
{
stmw.Write(soap);
}
}
// Get response
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream
StreamReader reader = new StreamReader(response.GetResponseStream());
// Console application output
Console.WriteLine(reader.ReadToEnd());
}
}
and the xml looks like this
This line may makes error
HttpWebRequest request = WebRequest.Create("http://cramapp-dt-02s.cable.comcast.com:8158/restfulqueryservice/queryservice/getrequestdetails?api_key=......&request_id=33ebc6e9-9def-4f39-adf9-bba2edef3b54") as HttpWebRequest;
Because the url in the create function is not avaiable too access (too long so has ... for shout display), try locate and paste the full link or this link :
http://www.w3schools.com/xml/note.xml