Develop Reference

HttpClient requests not working for a particular domain in Azure

I have an Azure function that sends a request to a URL and sends back the response. This function kept failing with timeout error for URLs from a particular domain (confidential).
To debug this, I created a very minimal Azure function:
var content = string.Empty;
using (var response = await _httpClient.GetAsync(url))
{
response.EnsureSuccessStatusCode();
content = await response.Content.ReadAsByteArrayAsync();
}
return new OkObjectResult(content);
This code works fine in local. When I try using the deployed Azure function, it works for all the other domains I tried (ex: https://google.com) but it hits request timeout error for a particular domain after trying for about 90 seconds. The error happens at this particular line: _httpClient.GetAsync(url). Again, it works fine for this (confidential) domain in local.
I have tried deploying the Azure function to two completely different Azure service plans and regions. Same result. It doesn't work for URLs from the required domain. Works for URLs of other domains.
Error:
System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
Update (solution):
I tried sending a request from Postman, copied the code from there for C# and deployed it to the Azure function and it is now working for the problematic domain. Something like below:
var client = new RestClient(url);
client.Timeout = -1;
var request = new RestRequest(Method.GET);
IRestResponse response = client.Execute(request);
The key here is client.Timeout = -1, which seems to have fixed the problem.
Now, in my original code, I tried setting HttpClient's timeout to Timeout.InfiniteTimeSpan both in Startup configuration as well as at individual request level but it did not work.
services.AddHttpClient("AzureTestClient", options =>
{
options.Timeout = Timeout.InfiniteTimeSpan;
});
Am I setting the timeout wrong in the HttpClient solution?

If you are using a Consumption plan then maybe the confidential URL need to whitelist the whole Azure Data center. You can follow the guide here or consider upgrading the Consumption plan to a premium one and have a dedicated linked VNET.
Maybe your local machine is already linked to the domain/whitelisted so azure function operates from different range.
Another reason maybe the URL returns a different HttpStatusCode that is't Successful range (200-299) so it fails with "EnsureSuccessStatusCode" in the old code?
Normally for the http code initialization, I did something like that:
public void Configure(IWebJobsBuilder builder)
{
builder.Services.AddHttpClient("AzureTestClient",
options => { options.Timeout = Timeout.InfiniteTimeSpan; });
}
Then when I want to use it, I do like that in any other function and it worked:
var client = clientFactory.CreateClient("AzureTestClient");

POSTing form data using Azure data factory web activity returns different results to C# script

I am getting to the stage of hair pulling with this one, I'm hoping someone can see if I'm doing anything wrong.
I'm trying to POST some form data to website using Azure data factory web activity however whilst I get a response (I get the page and some headers) it is different to the response I get if I make the exact same request using C# and HttpClient code. I've used fiddler to view the request being post'd using my C# script and according to the request information given in data factory they are exactly the same - so same headers, same content format etc...
This POST request is to login to a website which has a custom login mechanism, so no OAuth or anything like that unfortunately. It is supposed to return a cookie, which it does if I use my C# script, but if I make the same POST request using data factory web activity then I get different html sent back (it just returns the same login screen) and also different set of response headers in the "ADFWebActivityResponseHeaders" part of the activity output!?! See below for what is returned in the web activity output response headers:-
"ADFWebActivityResponseHeaders": {
"Pragma": "no-cache",
"Vary": "Accept-Encoding",
"X-Frame-Options": "DENY",
"Cache-Control": "no-store, must-revalidate, no-cache, post-check=0, pre-check=0",
"Date": "Wed, 09 Sep 2020 08:09:30 GMT",
"Server": "Microsoft-IIS/8.5"
}
If I do this via C# I also get a 'Set-Cookie' as well (strangely if I make a 'GET' request for the homepage of this site I do get a 'Set-Cookie' in the response!!!), but never when doing this via data factory. I'm struggling to see how this is possible unless data factory is modifying my request in some fashion? Below is my C# code, pretty simple/standard:-
var handler = new HttpClientHandler();
handler.CookieContainer = new CookieContainer();
handler.UseCookies = true;
handler.UseDefaultCredentials = false;
// Create our http client which will perform our web requests
var HttpClient = new HttpClient(handler);
HttpClient.BaseAddress = new Uri("**REMOVED**");
// Some of the extracts take a LONG time, so set the timeout for default of 30mins
HttpClient.Timeout = TimeSpan.FromMinutes(30);
// Set the 'form' parameters we're going to POST to the server in the request
var parameters = new Dictionary<string, string>
{
{ "username", "**REMOVED**" },
{ "password", "**REMOVED**" }
};
// URL encode the parameters
var content = new FormUrlEncodedContent(parameters);
// Submit our POST with the parameters
var response = await HttpClient.PostAsync("**REMOVED**", content);
Running this code and using fiddler I see the following request with headers, these are the only headers:-
Content-Length: 80
Content-Type: application/x-www-form-urlencoded
username=REMOVED&password=REMOVED
and in the 'input' side of the web activity is the details of the request, I've added the headers in the web activity and these are correct:-
"method": "POST",
"headers": {
"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": 80
},
"body": "username=REMOVED&password=REMOVED"
Note that in the data factory I'm using a self hosted integration runtime as this website blocks addresses that do not come from the specific IP addresses used externally by our on-prem network/firewall. I know that is not the problem as I'm getting a response with the normal login page from the site (if I use the Azure integration runtime I get a denied response).
Here is a screen shot of the web activity in data factory:-
Really hope someone out there can see what I'm missing or whatever...

Turns out this does work and will list the cookies in the JSON output from the activity as shown below (note this is to be found in the output of the ADF activity, so you would pick up the cookie from the output a bit like... #activity('Login and get cookie').output.ADFWebActivityResponseHeaders["Set-Cookie"] )
However, in my case the url I was POSTing to was responding with a 302 (moved temporarily) but the 'Location' header which should be there is not in the ADFWebActivityResponseHeaders - which is why I missed it. I tried using Chrome with the developer tools and looked at the response directly which is where I found the 302 response code. After that, I just used the new URL given in the response headers (i.e. the url in the 'Location') that I found when using the browser dev tools.
Unfortunately at the time of writing, the Azure data factory HTTP activity does not follow redirects (and doesn't list all the response headers either!) so if anyone encounters the same problem they will need to manually find out and get the url's for any redirects. In other words, try using a tool like browser/postman and look at the response if it doesn't work in ADF... you might find there is a redirect going on :-)
There is a feature request logged for this here, be sure to add your vote :)
edited to update the Azure feedback change of URL after MS decided to change things on the feedback site!?!

What's is the equivalent to this code in C#

Situation
I'm tyring to send a certain message to every friend of my LINE's developer account.
I have no idea how to do that, but I already created it in PHP.
But again I don't know how to do this in C#.
Here is some information about my editor and stuff.
Editor: Visual Studio 2017
Template: I got "LINE Bot C# Template" in https://marketplace.visualstudio.com/items?itemName=pierre3.LINEBotCSharpTemplate
Project: I created the project with "LINEBotApplication"
What I want to do
I want to convert the code below to C#.
<?php
// HTTP REQUEST
$ch = curl_init('https://api.line.me/v2/bot/message/push');
$options = [
CURLOPT_CUSTOMREQUEST => 'POST',
CURLOPT_HTTPHEADER => $headers,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_BINARYTRANSFER => true,
CURLOPT_HEADER => true,
CURLOPT_POSTFIELDS => $post,
];
curl_setopt_array($ch, $options);
// To execute
$result = curl_exec($ch);
// To check eroors
$errno = curl_errno($ch);
if ($errno) {
return;
}
// To get HTTP status
$info = curl_getinfo($ch);
$httpStatus = $info['http_code'];
$responseHeaderSize = $info['header_size'];
$body = substr($result, $responseHeaderSize);
// If 200
echo $httpStatus . ' ' . $body;
What I'm not sure
I don't know how to fully convert the given code.
I may be able to convert "curl_init" method, by looking at
How to send data using the webrequest class
But with the remaining code I need assistance to convert it.
If you need more, I'd love to share.

中井さん、StackOverflowへようこそ！
I don't know much about PHP, but it seems to me your code is not doing anything other than sending a simple HTTP POST request. You can achieve the same thing in C# using a WebRequest. This should help:
https://learn.microsoft.com/en-us/dotnet/api/system.net.webrequest?view=netframework-4.8
EDIT:
Regarding what you asked in your comment, you can't just add the options to the WebRequest as an array like in PHP. Those API's are just different. For example you'd make a WebRequest like this
WebRequest myWebRequest=WebRequest.Create("https://api.line.me/v2/bot/message/push");
then instead of
CURLOPT_HTTPHEADER => $headers,
You would do
myWebRequest.Headers.Add([header name], [header value]);
for every header you need to add. (Or maybe there's a way of adding a collection of headers)
And so on.
Also, in researching how to do a POST with a WebRequest, it seems quite clumsy. So you might want to use HttpClient instead. Here's more on which one to choose.
https://www.infoworld.com/article/3198673/my-two-cents-on-webclient-vs-httpclient-vs-httpwebrequest.html
Bottom line is, it's not that straightforward to just rewrite something from PHP to C#. Those are two very different beasts. So you need to do a bit more studying.
Or maybe someone will come along and give you the answer :)

Welcome to SO and C#. I haven't worked with LINE Bot but I thought I'd provide a generic example which might help understanding working with HTTP request in C#.
I have created this sample app for you as an example, using .NET Core and HttpClient. Just create a simple console app and paste the following code into Program.cs file:
using System;
using System.Net.Http;
using System.Threading.Tasks;
namespace ConsoleApp1
{
class Program
{
static async Task Main()
{
var client = new HttpClient();
var content = new StringContent("{ \"your\": \"content\", \"maybe\": \"json\" }");
content.Headers.Add("X-SOME-RANDOM", "header-value");
var response = await client.PostAsync("https://httpbin.org/post", content);
var httpStatus = response.StatusCode;
var body = await response.Content.ReadAsStringAsync();
Console.WriteLine($"{httpStatus} {body}");
}
}
}
I hope it helps a bit. Have fun!
PS Thanks to httpbin.org for the handy service.

API GET Request Returns HTML/Text instead of JSON

This is my first time working with a RESTful API and Xamarin and etc. I have so far made a simple REST API. I have written a GET call to it that, if I write http://localhost:[num]/api/Name, it will return a JSON file of the matching Emu's information. I have tested this with Postman, so I know that it works.
I have now written an app that will call this API in order to catch this information and then display it. So far, I've got it connected to the server hosting my API, but I'm unable to get it to return JSON. Instead it seems to be returning text/HTTP.
From what I've searched up on previous Stack Overflow threads, it seems that I was missing Headers requesting that reply be in a JSON format. When I added in code that was on the official .NET documentation on Microsoft's website, it gave me issues with my Json Deserialiser. I have also added in the information in the header to make sure that it returns json.
Here is the code for the function:
async private void Submit_OnClicked(object sender, EventArgs e)
{
var nameValue = EmuName.Text;
var baseAddr = new Uri("http://my_url/HelloEmu/");
var client = new HttpClient { BaseAddress = baseAddr };
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
string url = (string)nameValue;
var returnedJson = await client.GetStringAsync(url);
Models.EmuItemModel MyEmu = JsonConvert.DeserializeObject<Models.EmuItemModel>(returnedJson);
ReturnedName.Text = MyEmu.Name;
ReturnedAge.Text = MyEmu.Age.ToString();
ReturnedWeight.Text = MyEmu.Weight.ToString();
My code actually faults on the line ReturnedWeight.Text = MyEmu.Weight.ToString()
But I'm guessing the more majour issue is occuring during deserialisng the object, because it seemingly "skips" over the preceeding two lines when I run it in the debugger.
When I run it in Visual Studio 2019, the value of "returnedJson" is this:
"<html><head><meta http-equiv=\"refresh\" content=\"0;url=http://lookup.t-mobile.com/search/?q=http://my_url/HelloEmu/Keith&t=0\"/></head><body><script>window.location=\"http://lookup.t-mobile.com/search/?q=\"+escape(window.location)+\"&r=\"+escape(document.referrer)+\"&t=0\";</script></body></html>"
I think this is an HTML output. I would love any hints about what on earth I'm doing wrong!
EDIT: Since it almost seems like the HTML is returning an error message, perhaps it could do with my url??? I've published the website using the File system method. So to access the API in Postman I'll use http://localhost:[port]/api/values, calling my website in a regular ol' browser makes it go http://my_url/HelloEmu. I get a 403 "no directory" method in return...
EDIT: Here is the Postman code:
enter image description here

Usually it happens because there are missing headers or some other malformed request, Download RestSharp DLL from NuGet, and then you can use the following, in postman, go to "Code":
And choose C# you will see a code snippet (Example):

Getting "A From phone number is required" in JSON call to Twilio

I'm trying to use JSON over HTTP to communicate with the Twilio API (I'm running on asp.net vNext - and their C# library doesn't yet support coreclr)
I've got the following code, but every time I call the API I get the same 400 - Bad request response:
{"code": 21603, "message": "A 'From' phone number is required.", "more_info": "https://www.twilio.com/docs/errors/21603", "status": 400}
I'm using the following code to setup the request and make the call - the auth succeeds and I get the above error:
var byteArray = Encoding.ASCII.GetBytes("{AccountSid}:{AuthToken}");
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", Convert.ToBase64String(byteArray));
var jsonStr = JsonConvert.SerializeObject(new {
From = WebUtility.UrlEncode("+15005550003"), // A test number twilio supply
To = WebUtility.UrlEncode("{my phone number}"),
Body = WebUtility.UrlEncode("Hi there from a pure json rest client...")
});
var response = await client.PostAsync("https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/SMS/Messages.json", new StringContent(jsonStr));

After contacting support and dealing with a very helpful guy called Joshua, we have figured it out.
In Twilio docs you can either use a url with "messages.json" or "messages.xml" - I took that to mean: whichever url you use, the message (request and response) will have to follow that format.
Unfortunately that is not true - all requests to Twilio, must be of Content-Type: application/x-www-form-urlencoded
e.g. To=0123456789&From=9876543210&Body=Hi
in C# (HttpClient) that looks like:
_client.PostAsync(_url, new FormUrlEncodedContent(new Dictionary<string, string>(){
{ "To", "0123456789" },
{ "From", "9876543210" },
{ "Body", "Hi" } }));

I'm Megan from the Developer Community team at Twilio.
It looks like the number you are using, "+15005550003" is one of Twilio's magic input numbers that actually represents the "To" field, thus the error you are receiving. Depending on what you want to test, you might try passing in "+15005550006" instead.
You can see more about what each magic input represents in the docs here.
Hope this helps!

If using an SMS / Messaging Service which allows you to send from multiple phone numbers and you get this error,
then replace
{ "From","<number>" }
with
{ "MessagingServiceSid","<sid>" }
the SID can be found on the messaging services page
https://www.twilio.com/console/sms/services