Develop Reference

using OpenTSDB HTTP api in .NET : 400 Bad Request

I'm trying to use .net to put datapoints in OpenTSDB, using the HTTP /api/put API.
I've tried with httpclient, webRequest and HttpWebRequest. The outcome is always 400 - bad request: chunked request not supported.
I've tried my payload with an api tester (DHC) and works well.
I've tried to send a very small payload (even plain wrong, like "x") but the reply is always the same.
Here's one of my code instances:
public async static Task PutAsync(DataPoint dataPoint)
{
try
{
HttpWebRequest http = (HttpWebRequest)WebRequest.Create("http://127.0.0.1:4242/api/put");
http.SendChunked = false;
http.Method = "POST";
http.ContentType = "application/json";
Encoding encoder = Encoding.UTF8;
byte[] data = encoder.GetBytes( dataPoint.ToJson() + Environment.NewLine);
http.Method = "POST";
http.ContentType = "application/json; charset=utf-8";
http.ContentLength = data.Length;
using (Stream stream = http.GetRequestStream())
{
stream.Write(data, 0, data.Length);
stream.Close();
}
WebResponse response = http.GetResponse();
var streamOutput = response.GetResponseStream();
StreamReader sr = new StreamReader(streamOutput);
string content = sr.ReadToEnd();
Console.WriteLine(content);
}
catch (WebException exc)
{
StreamReader reader = new StreamReader(exc.Response.GetResponseStream());
var content = reader.ReadToEnd();
}
return ;
}
where I explicitly set to false the SendChunked property.
note that other requests, like:
public static async Task<bool> Connect(Uri uri)
{
HttpWebRequest http = (HttpWebRequest)WebRequest.Create("http://127.0.0.1:4242/api/version");
http.SendChunked = false;
http.Method = "GET";
// http.Headers.Clear();
//http.Headers.Add("Content-Type", "application/json");
http.ContentType = "application/json";
WebResponse response = http.GetResponse();
var stream = response.GetResponseStream();
StreamReader sr = new StreamReader(stream);
string content = sr.ReadToEnd();
Console.WriteLine(content);
return true;
}
work flawlessly.
I am sure I am doing something really wrong.
I'd like to to reimplement HTTP in Sockets from scratch.

I've found a solution I'd like to share here.
I've used wireshark to sniff my packets, and I've found that this header is added:
Expect: 100-continue\r\n
(see 8.2.3 of https://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html)
This is the culprit. I've read the post http://haacked.com/archive/2004/05/15/http-web-request-expect-100-continue.aspx/ by Phil Haack, and found that HttpWebRequest puts that header by default, unless you tell it to stop. In this article I've found that using ServicePointManager I can do just this.
Putting the following code on top of my method, when declaring the http object, makes it work very well, and solves my issue:
var uri = new Uri("http://127.0.0.1:4242/api/put");
var spm = ServicePointManager.FindServicePoint(uri);
spm.Expect100Continue = false;
HttpWebRequest http = (HttpWebRequest)WebRequest.Create(uri);
http.SendChunked = false;

C# - Request Json File with authorization key (cURL example)

I'm trying to do a HTTP GET request for a json file from an api in a C# application. I'm having trouble getting the authorization, request headers and the webresponse (.GetResponse not working).
The example on the api's site is in curl.
curl -H "Authorization: Bearer ACCESS_TOKEN" https://erikberg.com/nba/boxscore/20120621-oklahoma-city-thunder-at-miami-heat.json
Here is my request method, which will also include JSON deseralization
public static string HttpGet(string URI)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(URI);
// Not sure if the credentials input is the correct
string cred = $"{"Bearer"} {"ACCESS_TOKEN_IS_A_GUID"}";
req.Headers[HttpRequestHeader.Authorization] = cred;
req.Method = "GET";
// GetResponse() is "red", won't work.
WebResponse response = req.GetResponse();
using (Stream responseStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(responseStream, Encoding.UTF8);
return reader.ReadToEnd().Trim();
}
}
EDIT It was resolved. The problem was that the request was for a GZIP file and that had to be decompressed
var request = (HttpWebRequest)WebRequest.Create(requestUri);
request.UserAgent = userAgent;
request.ContentType = "application/json";
request.Method = WebRequestMethods.Http.Get;
request.Headers[HttpRequestHeader.Authorization] = bearer;
request.Headers[HttpRequestHeader.AcceptEncoding] = "gzip";
var response = (HttpWebResponse) request.GetResponse();
string jsonString;
using (var decompress = new GZipStream(response.GetResponseStream(), CompressionMode.Decompress))
{
using (var sr = new StreamReader(decompress))
jsonString = sr.ReadToEnd().Trim();
}
_Game = JsonConvert.DeserializeObject<Game>(jsonString);

You are not getting it because you don't have access.
The cURL command from API's site(that you mentioned in your question) gives the following JSON
{
"error" : {
"code" : "401",
"description" : "Invalid access token: ACCESS_TOKEN"
}
}
And so does the following code:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("URL");
req.UserAgent = "Bearer";
WebResponse response = req.GetResponse();
So what you need is a valid username/password or userAgent. You might want to contact the site for that.

How to ignore 401 and continue to read feeds

I was given a json feed.
It does not need any user name or password to access.
It can be seen from google Chrome.
In the google Chrome developer tool, it also say status 401,
but still able to read data.
Then I try to use C# web request or webclient, i get 401 and thrown , no longer able to continue.
So how does Chrome do it? and what should i do?
My C# code:
WebRequest request = WebRequest.Create(url);
request.UseDefaultCredentials = true;
request.Method = WebRequestMethods.Http.Get;
((HttpWebRequest)request).UserAgent = userAgent;
request.Timeout = timeOut;
((HttpWebRequest)request).Accept = "application/json";
response = request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream, Encoding.UTF8);

You can "catch" the 401 exception that is thrown by HttpWebRequest and read the content off the WebException using code like this:
try {
// your code above
}
catch(Exception oEx)
{
if(oEx.GetType() == typeof(WebException))
{
HttpWebResponse wr = ((WebException)oEx).Response as HttpWebResponse;
Stream dataStream = wr.GetResponseStream();
StreamReader reader = new StreamReader(dataStream, Encoding.UTF8);
string str = reader.ReadToEnd(); // str will contain the body of the HTTP 401 response
reader.Close();
}
}

C# Webrequest to Sockets

I have an application that starts multiple threads, and each thread sends requests to a remote server. After profiling, I noticed that the most time is taken in sending & processing the request. This is the function:
private string GetRequest(string url, string postFields)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Proxy = proxy;
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
request.Host = Constants.URLDomain;
request.KeepAlive = true;
request.Accept = Constants.Accept;
request.ContentType = "application/json";
request.Headers.Add("MyHeader1", "true");
request.Headers.Add("MyHeader2", "header2value");
request.Headers.Add("MyHeader3", "header3value");
request.Method = "POST";
byte[] postBytes = Encoding.UTF8.GetBytes(postFields);
request.ContentLength = postBytes.Length;
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(postBytes, 0, postBytes.Length);
}
string text = string.Empty;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream answer = response.GetResponseStream())
using (StreamReader reader = new StreamReader(answer, System.Text.Encoding.UTF8))
{
text = reader.ReadToEnd();
}
}
return text;
}
I was thinking of changing this code to use sockets so I could improve performance. Will there be actually an improvement if I change to sockets, and more importantly, how do will a sockets implementation of the above code look like?

Programmatically login to google app engine c#

I've tried to login to my google app engine application from ASP.NET for a few days, but no luck. I've read the following articles and got the basic ideas. But nothing works for me.
http://code.activestate.com/recipes/577217-routines-for-programmatically-authenticating-with-/
http://dalelane.co.uk/blog/?p=303
http://dalelane.co.uk/blog/?p=894
http://krasserm.blogspot.com/2010/01/accessing-security-enabled-google-app.html
http://blog.notdot.net/2010/05/Authenticating-against-App-Engine-from-an-Android-app
I know what to do. 1) Get an auth token from ClientLogin. 2) Get a cookie from Google App Engine. 3) Post data to my app with the cookie (Yes, I want to post data, not redirect after the second part). But the third part doesn't work for me at all. It give me 403 error. Here is my code:
void PostToGAE()
{
var auth = GetAuth(); // I can get the authtoken
var cookies = GetCookies(auth); // I can get the ACSID cookie
var url = string.Format("http://test.appspot.com/do/something/");
var content = "testvalue=test";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.KeepAlive = false;
request.CookieContainer = cookies;
byte[] byteArray = Encoding.UTF8.GetBytes(content);
request.ContentLength = byteArray.Length;
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse(); // This gives me 403
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
string result = reader.ReadToEnd();
reader.Close();
}
CookieContainer GetCookies(string auth)
{
CookieContainer cookies = new CookieContainer();
var url = string.Format("http://test.appspot.com/_ah/login?auth={0}",
System.Web.HttpUtility.UrlEncode(auth));
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AllowAutoRedirect = false;
request.CookieContainer = cookies;
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
string result = reader.ReadToEnd();
reader.Close();
return cookies;
}
string GetAuth()
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.google.com/accounts/ClientLogin");
var content = "Email=test#gmail.com&Passwd=testpass&service=ah&accountType=HOSTED_OR_GOOGLE";
byte[] byteArray = Encoding.UTF8.GetBytes(content);
request.ContentLength = byteArray.Length;
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
string loginStuff = reader.ReadToEnd();
reader.Close();
var auth = loginStuff.Substring(loginStuff.IndexOf("Auth")).Replace("Auth=", "").TrimEnd('\n');
return auth;
}
My app.yaml looks like this:
- url: /do/something/
script: something.py
login: admin
If I change the method POST to GET, that works. Could anyone tell me how I can post data?
Thanks.
EDITED:
Still no luck. I've tried several ways such as changing to [login: required] in app.yaml, adding [secure: always] to app.yaml and changing the request protocol to https, appending continue parameter to /_ah/login, but all of them don't work :(
I totally have no idea why POST doesn't work at all but GET. Any ideas?

I made it. I was on the wrong track. That was not the problem of app engine but Django. I am using Django-nonrel on google app engine, and I totally forgot to put #csrf_exempt decorator to my handler. I had the same problem before, but again. Anyway, the code above has been apparently working correctly since at the beginning. What a smart boy :)