Develop Reference

using OpenTSDB HTTP api in .NET : 400 Bad Request

I'm trying to use .net to put datapoints in OpenTSDB, using the HTTP /api/put API.
I've tried with httpclient, webRequest and HttpWebRequest. The outcome is always 400 - bad request: chunked request not supported.
I've tried my payload with an api tester (DHC) and works well.
I've tried to send a very small payload (even plain wrong, like "x") but the reply is always the same.
Here's one of my code instances:
public async static Task PutAsync(DataPoint dataPoint)
{
try
{
HttpWebRequest http = (HttpWebRequest)WebRequest.Create("http://127.0.0.1:4242/api/put");
http.SendChunked = false;
http.Method = "POST";
http.ContentType = "application/json";
Encoding encoder = Encoding.UTF8;
byte[] data = encoder.GetBytes( dataPoint.ToJson() + Environment.NewLine);
http.Method = "POST";
http.ContentType = "application/json; charset=utf-8";
http.ContentLength = data.Length;
using (Stream stream = http.GetRequestStream())
{
stream.Write(data, 0, data.Length);
stream.Close();
}
WebResponse response = http.GetResponse();
var streamOutput = response.GetResponseStream();
StreamReader sr = new StreamReader(streamOutput);
string content = sr.ReadToEnd();
Console.WriteLine(content);
}
catch (WebException exc)
{
StreamReader reader = new StreamReader(exc.Response.GetResponseStream());
var content = reader.ReadToEnd();
}
return ;
}
where I explicitly set to false the SendChunked property.
note that other requests, like:
public static async Task<bool> Connect(Uri uri)
{
HttpWebRequest http = (HttpWebRequest)WebRequest.Create("http://127.0.0.1:4242/api/version");
http.SendChunked = false;
http.Method = "GET";
// http.Headers.Clear();
//http.Headers.Add("Content-Type", "application/json");
http.ContentType = "application/json";
WebResponse response = http.GetResponse();
var stream = response.GetResponseStream();
StreamReader sr = new StreamReader(stream);
string content = sr.ReadToEnd();
Console.WriteLine(content);
return true;
}
work flawlessly.
I am sure I am doing something really wrong.
I'd like to to reimplement HTTP in Sockets from scratch.

I've found a solution I'd like to share here.
I've used wireshark to sniff my packets, and I've found that this header is added:
Expect: 100-continue\r\n
(see 8.2.3 of https://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html)
This is the culprit. I've read the post http://haacked.com/archive/2004/05/15/http-web-request-expect-100-continue.aspx/ by Phil Haack, and found that HttpWebRequest puts that header by default, unless you tell it to stop. In this article I've found that using ServicePointManager I can do just this.
Putting the following code on top of my method, when declaring the http object, makes it work very well, and solves my issue:
var uri = new Uri("http://127.0.0.1:4242/api/put");
var spm = ServicePointManager.FindServicePoint(uri);
spm.Expect100Continue = false;
HttpWebRequest http = (HttpWebRequest)WebRequest.Create(uri);
http.SendChunked = false;

Strange behavior of Stream from HttpWebResponse

I experienced some strange issue while reading stream from HttpWebResponse. In case of successful response (200 status code) I get exception when access to Length property of stream. But in case of WebException when server returns some error (e.g. BadRequest) everything is perfect, Length works ok. I can get ContentLength directly from HttpWebResponse but want to know the reason of such behavior.
Here is debug screenshot of both cases
try
{
var request = (HttpWebRequest) WebRequest.Create(uriString);
request.Method = HttpMethod.Get.Method;
request.ContentType = "application/json";
response = (HttpWebResponse) request.GetResponse();
statusCode = response.StatusCode;
responseString = DecodeResponse(response);
}
catch (WebException ex)
{
response = (HttpWebResponse) ex.Response;
responseString = DecodeResponse(response);
statusCode = response.StatusCode;
}
private static string DecodeResponse(HttpWebResponse response)
{
byte[] data;
using (Stream stream = response.GetResponseStream())
{
data = new byte[stream.Length];
stream.Read(data, 0, data.Length);
}
return Encoding.UTF8.GetString(data);
}

The stream (ConnectStream) which is being created internally as part of the response does not support the Length property. Different streams support different properties depending on their implementation.
You could use a different way to read from the stream as in this post:
C# How do i convert System.Net.ConnectStream to a byte[] (array)

ProtocolViolationException while sending POST request

I have read other similar questions and tried the solutions from those questions, but since that did not work, hence I am posting this here.
When I send POST request below, it fails with the following error message:
System.Net.ProtocolViolationException: You must write ContentLength bytes to the request stream before calling [Begin]GetResponse.
at System.Net.HttpWebRequest.GetResponse()
....
....
My GET requests for other URL end points are working fine, I am only having this issue while issuing a POST request. Also, I have already set the ContentLength in the code appropriately. I am still unable to send the POST request. Thoughts?
public void TestSubmitJobWithParams1()
{
const string RestActionPath = "URL_GOES_HERE";
// if you have multipe parameters seperate them with teh '&' delimeter.
var postData = HttpUtility.UrlEncode("MaxNumberOfRowsPerSFSTask") + "=" + HttpUtility.UrlEncode("3000");
var request = (HttpWebRequest)WebRequest.Create(RestActionPath);
request.Method = "POST";
request.Credentials = CredentialCache.DefaultCredentials;
request.PreAuthenticate = true;
request.ContentLength = 0;
request.Timeout = 150000;
request.CachePolicy = new RequestCachePolicy(RequestCacheLevel.BypassCache);
request.ContentType = "application/x-www-form-urlencoded";
byte[] bytes = Encoding.ASCII.GetBytes(postData);
request.ContentLength = bytes.Length;
Stream newStream = request.GetRequestStream();
newStream.Write(bytes, 0, bytes.Length);
string output = string.Empty;
try
{
using (var response = request.GetResponse())
{
using (var stream = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding(1252)))
{
output = stream.ReadToEnd();
}
}
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError)
{
using (var stream = new StreamReader(ex.Response.GetResponseStream()))
{
output = stream.ReadToEnd();
}
}
else if (ex.Status == WebExceptionStatus.Timeout)
{
output = "Request timeout is expired.";
}
}
catch (ProtocolViolationException e)
{
Console.WriteLine(e);
}
Console.WriteLine(output);
Console.ReadLine();
}

A few things:
Firstly, you don't need to set ContentLength directly - just leave it out (defaults to -1). You're actually calling it twice, so remove both calls.
Also, you need to call Close() on the stream before calling GetResponse()
Stream newStream = request.GetRequestStream();
newStream.Write(bytes, 0, bytes.Length);
newStream.Close();
Alternatively, you could have it within a using statement, which handles the closing and disposal for you):
using (var newStream = request.GetRequestStream())
{
newStream.Write(bytes, 0, bytes.Length);
}
You also don't technically need to use HttpUtility.UrlEncode() for the postData since there's nothing in your string that would violate a Url's integrity. Just do:
string postData = "MaxNumberOfRowsPerSFSTask=3000");
Let me know if that solves it for you.
For a more thorough rundown, check this out: http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse.aspx
Specifically, the section about ProtocolViolationException and also where it says:
When using the POST method, you must get the request stream, write the data to be posted, and close the stream. This method blocks waiting for content to post; if there is no time-out set and you do not provide content, the calling thread blocks indefinitely.

How to get access token for google oauth?

I am using C# (ASP.NET). I want to use Google OAuth for accessing the user profile detail in my app. I successfully got the authorization code but having a problem in getting the access token.
I prefer the Google tutorials. In tutorial, I read that I have to send the request and get the response from google. For that, I use System.Net.HttpWebRequest/HttpWebResponse (am I going in the right way). I have used this code...
byte[] buffer = Encoding.ASCII.GetBytes("?code=" + code + "&client_id=xxx&client_secret=xxx&redirect_uri=xxxx&grant_type=authorization_code");
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://accounts.google.com");
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = buffer.Length;
Stream strm = req.GetRequestStream();
strm.Write(buffer, 0, buffer.Length);
strm.Close();
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
Response.Write(((HttpWebResponse)resp).StatusDescription);
But, I got the error:
The remote server returned an error: (405) Method Not Allowed.
Update: Here variable code is authorization code.

I think you are sending the POST request to the wrong endpoint, the correct one is https://accounts.google.com/o/oauth2/token

As I had similar problems in the process of implementing Google auth, I will post the code that works.. The last mentioned problem: error (400) Bad request could be caused by leading '?' in the above code..
string codeClient = "code="+ t +"&client_id=number.apps.googleusercontent.com&";
string secretUri = "client_secret=yoursecret&" + "redirect_uri=path&"
+ "grant_type=authorization_code";
postString = codeClient + secretUri;
string url = "https://accounts.google.com/o/oauth2/token";
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url.ToString());
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
UTF8Encoding utfenc = new UTF8Encoding();
byte[] bytes = utfenc.GetBytes(postString);
Stream os = null;
try
{
request.ContentLength = bytes.Length;
os = request.GetRequestStream();
os.Write(bytes, 0, bytes.Length);
}
catch
{ }
try
{
HttpWebResponse webResponse = (HttpWebResponse) request.GetResponse();
Stream responseStream = webResponse.GetResponseStream();
StreamReader responseStreamReader = new StreamReader(responseStream);
result = responseStreamReader.ReadToEnd();//parse token from result

My code is working, I have done mistakes in above two lines. It should be like this
byte[] buffer = Encoding.ASCII.GetBytes("code=" + code + "&client_id=xxx&client_secret=xxx&redirect_uri=xxxx&grant_type=authorization_code");
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://accounts.google.com/o/oauth2/token");
Remaining code is correct.

The original request seems to be somewhat outdated. But I found that the Google's code examples contain lots of "Best Practices" housekeeping code that's hard to separate from the essential operations.
I recently published a document that represents all the REST operations as curl commands. It's hard to be conversant in every language, but curl seems universal. Most people know it- otherwise, it's pretty easy to grasp. In my curl examples, the -d flag indicates a POST operation. Otherwise, the parameters are appended to the URL.
http://www.tqis.com/eloquency/googlecalendar.htm

public string ReceiveTokenGmail(string code, string GoogleWebAppClientID, string GoogleWebAppClientSecret, string RedirectUrl)
{
string postString = "code=" + code + "&client_id=" + GoogleWebAppClientID + #"&client_secret=" + GoogleWebAppClientSecret + "&redirect_uri=" + RedirectUrl;
string url = "https://accounts.google.com/o/oauth2/token";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url.ToString());
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
UTF8Encoding utfenc = new UTF8Encoding();
byte[] bytes = utfenc.GetBytes(postString);
Stream os = null;
try
{
request.ContentLength = bytes.Length;
os = request.GetRequestStream();
os.Write(bytes, 0, bytes.Length);
}
catch
{ }
string result = "";
HttpWebResponse webResponse = (HttpWebResponse)request.GetResponse();
Stream responseStream = webResponse.GetResponseStream();
StreamReader responseStreamReader = new StreamReader(responseStream);
result = responseStreamReader.ReadToEnd();
return result;
}

It was surprisingly difficult to find the correct and simple way of getting access token by auth code. (Especially because it has taken some time for me and then even with the correct code I got “invalid_grant” error because my auth code expired while searching :) )
So here is the code:
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow(
new GoogleAuthorizationCodeFlow.Initializer
{
ClientSecrets = new ClientSecrets()
{
// Use ones from "Web SDK configuration" section if you created your app in Firebase.
ClientId = "…",
ClientSecret = "…"
},
Scopes = new[] { "email" },
}
);
TokenResponse token = await flow.ExchangeCodeForTokenAsync(string.Empty, "4/…", string.Empty, CancellationToken.None);
As you can see, userId can be just empty, as well as redirectUri.
Don’t forget to add the Google.Apis.Auth Nuget package reference.

Performance of HttpWebRequest using POST

I have a small tool I use for testing a webservice.
It can either call the webservice using POST or using GET.
The code for using POST is
public void PerformRequest()
{
WebRequest webRequest = WebRequest.Create(_uri);
webRequest.ContentType = "application/ocsp-request";
webRequest.Method = "POST";
webRequest.Credentials = _credentials;
webRequest.ContentLength = _request.Length;
((HttpWebRequest)webRequest).KeepAlive = false;
using (Stream st = webRequest.GetRequestStream())
st.Write(_request, 0, _request.Length);
using (HttpWebResponse httpWebResponse = (HttpWebResponse)webRequest.GetResponse())
using (Stream responseStream = httpWebResponse.GetResponseStream())
using (BufferedStream bufferedStream = new BufferedStream(responseStream))
using (BinaryReader reader = new BinaryReader(bufferedStream))
{
if (httpWebResponse.StatusCode != HttpStatusCode.OK)
throw new WebException("Got response status code: " + httpWebResponse.StatusCode);
byte[] response = reader.ReadBytes((int)httpWebResponse.ContentLength);
httpWebResponse.Close();
}
}
The code for using GET is:
protected override void PerformRequest()
{
WebRequest webRequest = WebRequest.Create(_uri + "/" + Convert.ToBase64String(_request));
webRequest.Method = "GET";
webRequest.Credentials = _credentials;
((HttpWebRequest)webRequest).KeepAlive = false;
using (HttpWebResponse httpWebResponse = (HttpWebResponse)webRequest.GetResponse())
using (Stream responseStream = httpWebResponse.GetResponseStream())
using (BufferedStream bufferedStream = new BufferedStream(responseStream))
using (BinaryReader reader = new BinaryReader(bufferedStream))
{
if (httpWebResponse.StatusCode != HttpStatusCode.OK)
throw new WebException("Got response status code: " + httpWebResponse.StatusCode);
byte[] response = reader.ReadBytes((int)httpWebResponse.ContentLength);
httpWebResponse.Close();
}
}
As you can see the code is quite similar. If anything, I would expect the GET-method to be slightly slower, as it has to encode and transmit the data in Base64.
But when I run it, I see that the POST-method uses far more processing power than the GET-method. On my machine, I can run 80 threads of the GET-method using approximately 5% CPU, while 80 threads of the POST-method uses 95% CPU.
Is there something inherently more expensive about using POST? Is there anything I can do to optimize the POST-method? I cannot reuse the connection, as I want to simulate requests from different clients.
dotTrace reports that 65% of the processing-time is spent in webRequest.GetResponse() when using POST.
The underlying webservice uses Digest-Authentication if that makes any difference.

Well, depending on the complexity of the final uri, it might be that the "GET" requests are being cached? "POST" is not cached by default, but "GET" often is (as it should be idempotent). Have you tried sniffing to see if there is any difference here?
Also - you might find WebClient easier to use - something like:
using (WebClient wc = new WebClient())
{
byte[] fromGet = wc.DownloadData(uriWithData);
byte[] fromPost = wc.UploadData(uri, data);
}