Develop Reference

Occasional response headers in C# HTTP Request

Request:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
String responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
Console.WriteLine(responseString);
Response:
{"code":"SUCCESS","details":
{"created_time":"","id":"xxxx"},
"message":"uploaded",
"status":"success"}
HTTP/1.1 200 OK
Date: Wed, 18 Dec 2019 11:42:26 IST
Last-Modified: Wed, 18 Dec 2019 11:42:25 IST
Content-Type: application/json
Connection: Keep-Alive
Server: AWServer
Pragma: no-cache
Cache-Control: no-cache
Expires: 1
Whenever the above-mentioned C# request is executed, the response occasionally contains headers(HTTP/1.1 200 OK...), When I'm only trying to get the body part({"code"....} alone(response.GetResponseStream()). Is this the intended behavior?

Take a look at the basic article on http headers
HTTP headers let the client and the server pass additional information with an HTTP request or response. An HTTP header consists of its case-insensitive name followed by a colon (:), then by its value. Whitespace before the value is ignored.
Headers are additional information. I guess that since you left out the url and the whole creation of the Request and the url, this means that some responses have Headers and some not. That depends on the additional non-body information the api or web server wants to respond with.
It's in the control of the responder and not the receiver.
Don't ignore them: Some times interesting metadata come from Headers. It should not be data but information about it, like encoding, CORS info etc.
last modified header link
date header link

Return Unauthorized (401) only when calling from code (c#)

UPDATE
As #Alexandru Clonțea suggested, I checked the fiddler log and found:
In both success or fail cases, there are actually 2 requests being sent. The first request are mostly the same for both cases, it's something like:
GET http://myservice.com/handler?param1=something&param2=somethingelse HTTP/1.1
Authorization: Basic xxxxxx
Accept: application/json, application/xml, text/json, text/x-json,
text/javascript, text/xml
User-Agent: RestSharp/100.0.0.0
Host: myservice.com
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
The response for them are the same, which is:
HTTP/1.1 301 Moved Permanently
Content-Type: text/html; charset=utf-8
Location: /handler/?param1=something&param2=somethingelse
Date: Sat, 08 Sep 2018 01:50:16 GMT
Content-Length: 115
Moved Permanently.
I have noticed that it always try to redirect the call to /handler/?param1=something&param2=somethingelse, and that's because of the setup of the server code. it's actually working as expected. The difference is in the second request. The second request of the failure case (which is the c# code) doesn't have the authorization header and that's why it failed. Now, my question will be, why does the second request miss the authorization header? How can I fix it? Below is an example of the failed request:
GET http://myservice.com/handler/?param1=something&param2=somethingelse HTTP/1.1
Accept: application/json, application/xml, text/json, text/x-json,
text/javascript, text/xml
User-Agent: RestSharp/100.0.0.0
Accept-Encoding: gzip, deflate
Host: myservice.com
Backgroud:
I have a service written in GO deployed on a server. It requires a basic authentication. For example, I can call it successfully with the following request:
GET /handler/?
param1=something&param2=somethingelse HTTP/1.1
> Host: myservice.com
> Authorization: Basic xxxxxx
> User-Agent: RestClient/5.16.6
> Accept: */*
The request above is made by a rest api client tool (like postman), and it's working fine. It's also working fine if I call it from a browser.
Problem:
Now, I try to make the same call to the same service using c# code, and I have it as:
// pass cert validation
ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
HttpClient client = new HttpClient();
var byteArray = Encoding.ASCII.GetBytes(username + ":" + password);
var auth = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", Convert.ToBase64String(byteArray));
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.Authorization = auth;
var response = client.SendAsync(request).Result; // don't need async
But in this case, I am getting Unauthorized (401) back. I have checked into the actually request that was sent by the code, it had exactly the same authorization header as the one shows above (Authorization: Basic xxxxxx, and the xxxxxx is the same as above) and same uri as well. Actually, everything it sent looks the same as when I used the rest api client tool, but it was just failed in code.
when I check the log on the server side, I see the log below when it returns 401:
[GIN-debug] redirecting request 301: /handler --> /hanlder/?param1=something&param2=somethingelse
but I don't see this log when the call is from the rest api client tool (or browser)
As you may know from the log, the server-side code is using the go gin framework. But since it works fine in other cases, I don't think it's a problem with the server-side code.
Back to the C# code, I have tried to use the HttpWebRequest with NetworkCredential instead of the HttpClient, and I also try to use client.DefaultRequestHeaders.Authorization = auth, but I was still getting the same error.
I am wondering if someone has seen this before or could help? It will be really appreciated.

As a workaround, I can modify the request to be http://myservice.com/handler/?param1=something&param2=somethingelse so that no redirection is needed. Thus, it will be correctly authorized.
But still, haven't figure out how to make the second request to be sent with the authorize header yet.

Microsoft Emotions API Get Recognition in Video Operation Result

How do I return the status and results from a video I posted to the emotions api using Get Recognition in Video Operation Result?
I can successfully run the Get Recognition in Video Operation Result using an operation id of a video I uploaded, but the response does not show the status of the video and the results from the emotions API but only shows the following:
StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent,
Headers:{
Pragma: no-cache
apim-request-id: 010962c0-f907-4ba3-a7fd-564ddff7f97d
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Cache-Control: no-cache
Date: Wed, 02 Nov 2016 02:33:16 GMT
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Content-Length: 16048
Content-Type: application/json; charset=utf-8 Expires: -1 }
I've tried using the same oid in the open api testing console
(https://dev.projectoxford.ai/docs/services/5639d931ca73072154c1ce89/operations/56f8d4471984551ec0a0984f/console)
and the response returns the status and the results from the emotions api.
I'm using the following code from the API reference:
var client = new HttpClient();
var queryString = HttpUtility.ParseQueryString(string.Empty);
// Request headers
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "{subscription key}");
var uri = "https://api.projectoxford.ai/emotion/v1.0/operations/{oid}?" + queryString;
var response = await client.GetAsync(uri);

Based on what you've pasted, it looks like you do have a response, 16048 bytes of JSON.
Since you're using C#, I think you'll save a lot of effort by using the client library, authored by members of the Cognitive Services team. If you use Visual Studio, you can simply pull down the NuGet package.
If you don't want to use the prebuilt library, you'll need to parse out the JSON yourself. The JSON will be in response.Content.

(Instagram API) Trouble Reaching Tag Endpoints

I'm trying to gather a list of recent posts that contain a certain hashtag. The API Documentation states that I should be using the following GET call:
https://api.instagram.com/v1/tags/{tag-name}/media/recent?access_token=ACCESS-TOKEN
When I load the page where I want this information displayed, I perform the following:
using(HttpClient Client = new HttpClient())
{
var uri = "https://api.instagram.com/v1/tags/" + tagToLookFor + "/media/recent?access_token=" + Session["instagramaccesstoken"].ToString();
var results = Client.GetAsync(uri).Result;
// Result handling below here.
}
For reference, tagToLookFor is a constant string defined at the top of the class (eg. foo), and I store the Access Token returned from the OAuth process in the Session object with a key of 'instagramaccesstoken'.
While debugging this, I checked to make sure the URI was being formed correctly, and it does contain both the tag name and the just-created access_token. Using Apigee with the same URI (Save for a different access_token) returns the valid results I would expect. However, attempting to GET using the URI on my webstie returns:
{
StatusCode: 400,
ReasonPhrase: 'BAD REQUEST',
Version: 1.1,
Content: System.Net.Http.StreamContent,
Headers:{
X-Ratelimit-Remaining: 499
Vary: Cookie
Vary: Accept-Language
X-Ratelimit-Limit: 500
Pragma: no-cache
Connection: keep-alive
Cache-Control: no-store, must-revalidate, no-cache, private
Date: Fri, 27 Nov 2015 21:39:56 GMT
Set-Cookie: csrftoken=97cc443e4aaf11dbc44b6c1fb9113378; expires=Fri, 25-Nov-2016 21:39:56 GMT; Max-Age=31449600; Path=/
Content-Length: 283
Content-Language: en
Content-Type: application/json; charset=utf-8
Expires: Sat, 01 Jan 2000 00:00:00 GMT
}
}
I'm trying to determine what the difference between the two could be; the only thing that I can think of is that access_token is somehow being invalidated when I switch between pages. The last thing I do on the Login/Auth page is store the access_token using Session.Add, then call Server.Transfer to move to the page that I'm calling this on.
Any Ideas on what the issue could be? Thanks.

Attach the token to the header when making the request.
Client.DefaultRequestHeaders.Add("access_token", "Bearer " + token);

The problem ended up being one regarding Sandbox Mode. I had registered an app after the switch, and I was the only user in my sandbox. As a result, it had no problem finding my posts/info, but Sandbox Mode acts as if the Sandbox users are the only users on Instagram, so naturally it would not find anything else.
It turns out there was an existing registered application in my organization (made before the switch date) that does not have any such limitations, so I have been testing using that AppID/secret.
tl;dr: If you're the only user in your app's sandbox, work on getting users into your sandbox. See their article about it for more info.

HttpWebRequest with caching enabled throws exceptions

I'm working on a small C#/WPF application that interfaces with a web service implemented in Ruby on Rails, using handcrafted HttpWebRequest calls and JSON serialization. Without caching, everything works as it's supposed to, and I've got HTTP authentication and compression working as well.
Once I enable caching, by setting request.CachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.CacheIfAvailable);, things go awry - in the production environment. When connecting to a simple WEBrick instance, things work fine, I get HTTP/1.1 304 Not Modified as expected and HttpWebRequest delivers the cached content.
When I try the same against the production server, running nginx/0.8.53 + Phusion Passenger 3.0.0, the application breaks. First request (uncached) is served properly, but on the second request which results in the 304 response, I get a WebException stating that "The request was aborted: The request was canceled." as soon as I invoke request.GetResponse().
I've run the connections through fiddler, which hasn't helped a whole lot; both WEBrick and nginx return an empty entity body, albeit different response headers. Intercepting the request and changing the response headers for nginx to match those of WEBrick didn't change anything, leading me to think that it could be a keep-alive issue; setting request.KeepAlive = false; changes nothing, though - it doesn't break stuff when connecting to WEBrick, and it doesn't fix stuff when connecting to nginx.
For what it's worth, the WebException.InnerException is a NullReferenceException with the following StackTrace:
at System.Net.HttpWebRequest.CheckCacheUpdateOnResponse()
at System.Net.HttpWebRequest.CheckResubmitForCache(Exception& e)
at System.Net.HttpWebRequest.DoSubmitRequestProcessing(Exception& exception)
at System.Net.HttpWebRequest.ProcessResponse()
at System.Net.HttpWebRequest.SetResponse(CoreResponseData coreResponseData)
Headers for the (working) WEBrick connection:
########## request
GET /users/current.json HTTP/1.1
Authorization: Basic *REDACTED*
Content-Type: application/json
Accept: application/json
Accept-Charset: utf-8
Host: testbox.local:3030
If-None-Match: "84a49062768e4ca619b1c081736da20f"
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
########## response
HTTP/1.1 304 Not Modified
X-Ua-Compatible: IE=Edge
Etag: "84a49062768e4ca619b1c081736da20f"
Date: Wed, 01 Dec 2010 18:18:59 GMT
Server: WEBrick/1.3.1 (Ruby/1.8.7/2010-08-16)
X-Runtime: 0.177545
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: *REDACTED*
Headers for the (exception-throwing) nginx connection:
########## request
GET /users/current.json HTTP/1.1
Authorization: Basic *REDACTED*
Content-Type: application/json
Accept: application/json
Accept-Charset: utf-8
Host: testsystem.local:8080
If-None-Match: "a64560553465e0270cc0a23cc4c33f9f"
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
########## response
HTTP/1.1 304 Not Modified
Connection: keep-alive
Status: 304
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.0
ETag: "a64560553465e0270cc0a23cc4c33f9f"
X-UA-Compatible: IE=Edge,chrome=1
X-Runtime: 0.240160
Set-Cookie: *REDACTED*
Cache-Control: max-age=0, private, must-revalidate
Server: nginx/0.8.53 + Phusion Passenger 3.0.0 (mod_rails/mod_rack)
UPDATE:
I tried doing a quick-and-dirty manual ETag cache, but turns out that's a no-go: I get a WebException when invoking request.GetResponce(), telling me that "The remote server returned an error: (304) Not Modified." - yeah, .NET, I kinda knew that, and I'd like to (attempt to) handle it myself, grr.
UPDATE 2:
Getting closer to the root of the problem. The showstopper seems to be a difference in the response headers for the initial request. WEBrick includes a Date: Wed, 01 Dec 2010 21:30:01 GMT header, which isn't present in the nginx reply. There's other differences as well, but intercepting the initial nginx reply with fiddler and adding a Date header, the subsequent HttpWebRequests are able to process the (unmodified) nginx 304 replies.
Going to try to look for a workaround, as well as getting nginx to add the Date header.
UPDATE 3:
It seems that the serverside issue is with Phusion Passenger, they have an open issue about lack of the Date header. I'd still say that HttpWebRequest's behavior is... suboptimal.
UPDATE 4:
Added a Microsoft Connect ticket for the bug.

I think the designers find it reasonable to throw an exception when the "expected behavior"---i.e., getting a response body---cannot be completed. You can handle this somewhat intelligently as follows:
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError)
{
var statusCode = ((HttpWebResponse)ex.Response).StatusCode;
// Test against HttpStatusCode enumeration.
}
else
{
// Do something else, e.g. throw;
}
}

So, it turns out to be Phusion Passenger (or nginx, depending on how you look at it - and Thin as well) that doesn't add a Date HTTP response header, combined with what I see as a bug in .NET HttpWebRequest (in my situation there's no If-Modified-Since, thus Date shouldn't be necessary) leading to the problem.
The workaround for this particular case was to edit our Rails ApplicationController:
class ApplicationController < ActionController::Base
# ...other stuff here
before_filter :add_date_header
# bugfix for .NET HttpWebRequst 304-handling bug and various
# webservers' lazyness in not adding the Date: response header.
def add_date_header
response.headers['Date'] = Time.now.to_s
end
end
UPDATE:
Turns out it's a bit more complex than "just" setting HttpRequestCachePolicy - to repro, I also need to have manually constructed HTTP Basic Auth. So the involved components are the following:
HTTP server that doesn't include a HTTP "Date:" response header.
manual construction of HTTP Authorization request header.
use of HttpRequestCachePolicy.
Smallest repro I've been able to come up with:
namespace Repro
{
using System;
using System.IO;
using System.Net;
using System.Net.Cache;
using System.Text;
class ReproProg
{
const string requestUrl = "http://drivelog.miracle.local:3030/users/current.json";
// Manual construction of HTTP basic auth so we don't get an unnecessary server
// roundtrip telling us to auth, which is what we get if we simply use
// HttpWebRequest.Credentials.
private static void SetAuthorization(HttpWebRequest request, string _username, string _password)
{
string userAndPass = string.Format("{0}:{1}", _username, _password);
byte[] authBytes = Encoding.UTF8.GetBytes(userAndPass.ToCharArray());
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(authBytes);
}
static public void DoRequest()
{
var request = (HttpWebRequest) WebRequest.Create(requestUrl);
request.Method = "GET";
request.CachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.CacheIfAvailable);
SetAuthorization(request, "user#domain.com", "12345678");
using(var response = request.GetResponse())
using(var stream = response.GetResponseStream())
using(var reader = new StreamReader(stream))
{
string reply = reader.ReadToEnd();
Console.WriteLine("########## Server reply: {0}", reply);
}
}
static public void Main(string[] args)
{
DoRequest(); // works
DoRequest(); // explodes
}
}
}