Develop Reference

HttpClient causing captcha while same page in browser doesn't

I wrote a little tool to check the availability of a product (yes, the PS5) by checking the products shop page:
var client = new HttpClient();
HttpResponseMessage response = await client.GetAsync("https://www.mediamarkt.de/de/product/_sony-playstation®5-2661938.html");
HttpContent responseContent = response.Content;
using (var reader = new StreamReader(await responseContent.ReadAsStreamAsync()))
{
var output = reader.ReadToEndAsync();
Console.WriteLine(output.Result);
}
For some reason the result page is requesting me to do a captcha while calling the exact same URL in my browser giving me the correct page without captcha.
What is the reason of this behaviour and how do I avoid it?

This is not a direct answer but a workaround
This website is protected by Cloudflare, which shows you recaptcha that only solvable in javascript environment. Obviously, HttpClient does not have such. While there are some solutions for this in other languages, I could not find any for C#. Will show an example in Selenium, web testing framework, that uses web browser driver (in my case Chrome).
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Support.UI;
using System;
class Program
{
public static void Main(string[] args)
{
using (var driver = new ChromeDriver())
{
driver.Url = "https://www.mediamarkt.de/de/product/_sony-playstation®5-2661938.html";
// selenium does not behave well when element you are looking for is not visible,
// this method helps us to close cookie banner that blocks the view
CloseCookieBannerIfAppears(driver);
var buyButton = By.XPath("//div[contains(#class, \"Badge\")]").FindElement(driver);
Console.WriteLine(buyButton.Text); // Ausverkauft
}
}
private static void CloseCookieBannerIfAppears(IWebDriver driver)
{
var buttonInAcceptCookieBannerSelector = By.XPath("//button[#id=\"privacy-layer-accept-all-button\"]");
var waitForCookieBanner = new WebDriverWait(driver, TimeSpan.FromSeconds(5));
if (waitForCookieBanner.Until(x => x.FindElements(buttonInAcceptCookieBannerSelector).Count > 0))
{
driver.FindElement(buttonInAcceptCookieBannerSelector)
.Click();
}
}
}
Also looks like they have unprotected API, so you should be able to get this data directly as well. You can see that there is id parameter both in your link and in api call - _sony-playstation®5-2661938.html vs productId=2661938
using Newtonsoft.Json.Linq;
using System;
using System.Net.Http;
using System.Threading.Tasks;
class Program
{
public static async Task Main(string[] args)
{
var httpClient = new HttpClient();
var response = await httpClient.GetAsync("https://delivery-prod-teasermanagement.cloud.mmst.eu/api/teaser/find?productId=2661938");
var content = await response.Content.ReadAsStringAsync();
var status = JArray.Parse(content)[0]["promotionData"]["badge"];
Console.WriteLine(status); // Ausverkauft
}
}
Maybe there are some other edge cases, but you should be able to get the point.

error while testing the Echo Chat Bot on Azure

I'm following
https://towardsdatascience.com/creating-a-serverless-python-chatbot-api-in-microsoft-azure-from-scratch-in-9-easy-steps-2f1913fc9581
in creating a python chatbot, and my source code is as below, however when I try to run it, I'm getting an error. According to repositories that error is the adapter's on_error handler receives any exceptions thrown by bot's turn logic. If there is an exception thrown, the handler deletes the conversation state for the current conversation to prevent the bot from getting stuck in an error-loop caused by being in a bad state. any expert can help with this?
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Bot.Builder;
using Microsoft.Bot.Schema;
using System.Net;
using System;
using System.Net.Http;
using System.Text; // for class Encoding
using System.IO;
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
namespace Microsoft.BotBuilderSamples.Bots
{
public class EchoBot : ActivityHandler
{
public class FlaskRequestModel
{
[JsonProperty("text")]
public string Text {get; set;}
}
protected override async Task OnMessageActivityAsync(ITurnContext<IMessageActivity> turnContext, CancellationToken cancellationToken)
{
var replyText = $"{turnContext.Activity.Text}";
//if (!replyText.ToLower().Contains("Hey Bot")){ # Optional bit of code that only sends the sends the message to the back end if it contains a particular keyword
// return;
//}
var replyTextModel = new FlaskRequestModel()
{
Text = replyText
};
var jsonObject = JsonConvert.SerializeObject(replyTextModel);
var request = new HttpRequestMessage()
{
Content = new StringContent(jsonObject, Encoding.UTF8, "application/json"),
Method = HttpMethod.Post,
RequestUri = new Uri("yudao.azurewebsites.net"), // <- Replace the URL with the the URL for your function app
};
var httpClient = new HttpClient();
// httpClient.DefaultRequestHeaders.Add("API-Key","your API-key"); <- required if your HTTP trigger authorization was set to something other than Anonymous
var response = await httpClient.SendAsync(request, HttpCompletionOption.ResponseContentRead);
if (response.IsSuccessStatusCode)
{
var responseString = await response.Content.ReadAsStringAsync();
await turnContext.SendActivityAsync(MessageFactory.Text(responseString, responseString), cancellationToken);
}
else
{
await turnContext.SendActivityAsync(MessageFactory.Text("failure", "failure"), cancellationToken);
var responseString = await response.Content.ReadAsStringAsync();
await turnContext.SendActivityAsync(MessageFactory.Text(responseString, responseString), cancellationToken);
}
}
}
}

try this using regex:
A2 = "12 15|||Pform|||their|||REQUIRED|||-NONE-|||0"
intval=[val for val in A2.split('|||') if bool(re.search(pattern='\d+', string=val))]
Output:
['12 15', '0']

Create Work Item using Azure DevOps Rest API using C#

I am unable to create Work Item using Azure DevOps REST API as mentioned in Work Items - Create
Request:
https://dev.azure.com/{organization}/MyTestProject/_apis/wit/workitems/$Task?api-version=6.0-preview.3
Request Body:
[
{
"op": "add",
"path": "/fields/System.Title",
"value": "Task2"
}
]
Code to Get Response (Note this code works for all other POST Requests):
using (HttpResponseMessage response = client.SendAsync(requestMessage).Result)
{
response.EnsureSuccessStatusCode();
JsonResponse = await response.Content.ReadAsStringAsync();
}
Response: 400
Can someone please suggest?

It might be helpful to see your full example. However, here is a working example with Newtonsoft.Json (do not forget to create your PAT create personal access token):
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleApp2
{
class Program
{
static void Main(string[] args)
{
string PAT = "<personal access token>"; //https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=preview-page
string requestUrl = "https://dev.azure.com/<my_org>/<my_project>/_apis/wit/workitems/$Task?api-version=5.0";
try
{
List<Object> flds = new List<Object>
{
new { op = "add", path = "/fields/System.Title", value = "Title" }
};
string json = JsonConvert.SerializeObject(flds);
HttpClientHandler _httpclienthndlr = new HttpClientHandler();
using (HttpClient client = new HttpClient(_httpclienthndlr))
{
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", Convert.ToBase64String(
System.Text.ASCIIEncoding.ASCII.GetBytes(
string.Format("{0}:{1}", "", PAT))));
var request = new HttpRequestMessage(new HttpMethod("PATCH"), requestUrl)
{
Content = new StringContent(json, Encoding.UTF8, "application/json-patch+json")
};
HttpResponseMessage responseMessage = client.SendAsync(request).Result;
}
}
catch (Exception ex)
{
}
}
}
}
Additionally, you can consider to use .NET client libraries for Azure DevOps and TFS. Here is the example: Create a bug in Azure DevOps Services using .NET client libraries

application/json-patch+json is required.

Make my own API capable of requesting from another API

I have an existing and functioning API, which now have to be able to get data from another external API. How do i do that best?
I have tried with using HTTPClient, but i can't seem to get it to work. The error i get:
"No MediaTypeFormatter is available to read an object of type 'IList`1' from content with media type 'text/html'." -> I get this error on line 37. Can you spot it and/or tell me how I can do this differently, taking into account that all i want is the data (From the external API) and not to display it using a view, as this is an API?
Code below. I have also created a Pastebin: https://pastebin.com/MuKjEVys
using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.AspNetCore.Mvc;
using Newtonsoft.Json;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Net;
namespace API.Controllers
{
[ApiVersion("1.0")]
[Route("api/v{version:apiVersion}/[controller]")]
[ApiController]
public class ExternalApiController : Controller
{
private string ExternalApiLink = "https://blablabla.com/api";
private string ExternalApiLinkGet = "/module/1/";
[HttpGet("getdata")]
public ActionResult<ExternalApi> GetDataFromExternal()
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(ExternalApiLink);
var requestApi = client.GetAsync(ExternalApiLinkGet);
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", "XXXX");
requestApi.Wait();
var resultFromApi = requestApi.Result;
if (resultFromApi.IsSuccessStatusCode)
{
var readResponse = resultFromApi.Content.ReadAsAsync<IList<ExternalApi>>();
readResponse.Wait();
var data = readResponse.Result;
return Json(data);
}else
{
return NotFound();
}
}
}
}
}

Your response content seems to be json, while the content-type is text/html. If that is the case, the first thing to do would be to call the party that is exposing the service and have them fix it. In the meantime you could just read the content of the response as a string, and deserialize that string:
// Note that I made this method async.
public async Task<IActionResult> GetDataFromExternal()
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(ExternalApiLink);
// note that I moved this line above the GetAsync method.
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", "XXXX");
// note that I'm disposing the response
using (var response = await client.GetAsync(ExternalApiLinkGet))
{
if (response.IsSuccessStatusCode)
{
// Since the response content is json, but the content-type
// is text/html, simply read the content as a string.
string content = await response.ReadAsStringAsync();
// You can return the actual received content like below,
// or you may deserialize the content before returning to make
// sure it is correct, using JsonConvert.DeserializeObject<List<ExternalApi>>()
// var data = JsonConvert.DeserializeObject<List<ExternalApi>>(content);
// return Json(data);
return Content(content, "application/json");
}
else
{
return NotFound();
}
}
}
}

Calling an API asynchronously and parsing JSON

I come from an iOS (Swift) background. In one of my Swift apps, I have this class that calls an API. I'm trying to port it to C# (Windows Form application) but I'm hitting several snags. First here's the Swift code. Nothing fancy. One method does a POST request to login to the API and the other function executes a GET method to retrieve the JSON response for a user profile. Both these methods are asynchronous.
import Foundation
class API {
private let session = NSURLSession.sharedSession()
private let baseURL = "https://www.example.com/api/"
func login(userID userID: String, password: String, completion: (error: NSError?) -> ()) {
let url = NSURL(string: baseURL + "login")!
let params = ["username": userID, "password": password]
let request = NSMutableURLRequest(URL: url)
request.HTTPMethod = "POST"
request.encodeParameters(params) // encodeParameters is an extension method
session.dataTaskWithRequest(request, completionHandler: { data, response, error in
if let httpResponse = response as? NSHTTPURLResponse {
if httpResponse.statusCode != 200 {
completion(error: error)
} else {
completion(error: nil)
}
}
}).resume()
}
func fetchUser(completion: (user: User?, error: NSError?) -> ()) {
let url = NSURL(string: baseURL + "profile")!
let request = NSURLRequest(URL: url)
session.dataTaskWithRequest(request, completionHandler: { data, response, error in
if let error = error {
completion(user: nil, error: error)
} else {
// Parsing JSON
var jsonDict = [String: AnyObject]()
do {
jsonDict = try NSJSONSerialization.JSONObjectWithData(data, options: []) as! [String: AnyObject]
} catch {
print("Error occurred parsing data: \(error)")
completion(user: nil, error: error)
}
let user = User()
user.name = jsonDict["name"] as! String
user.age = jsonDict["age"] as! Int
completion(user: user, error: nil)
}
}).resume()
}
}
Here's my attempt to convert this to C#.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Drawing;
using System.Threading.Tasks;
using System.Windows.Forms;
using System.Net.Http;
using System.Runtime.Serialization.Json;
using System.Text;
using System.Xml.Linq;
using System.Xml.XPath;
namespace MyTrayApp
{
public partial class Form1 : Form
{
private string baseURL = "https://www.example.com/api/";
public Form1()
{
InitializeComponent();
}
private async void Form1_Load(object sender, EventArgs e)
{
await login("myusername", "mypassword");
await fetchUser();
}
async Task login(string userID, string password)
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(baseURL);
var parameters = new Dictionary<string, string>
{
{ "username", userID },
{ "password", password }
};
var encodedParameters = new FormUrlEncodedContent(parameters);
var response = await client.PostAsync("login", encodedParameters);
string responseString = await response.Content.ReadAsStringAsync();
//Console.WriteLine(responseString);
}
}
async Task fetchUser()
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(baseURL);
var response = await client.GetAsync("profile");
response.EnsureSuccessStatusCode();
var responseString = await response.Content.ReadAsStringAsync();
var jsonReader = JsonReaderWriterFactory.CreateJsonReader(Encoding.UTF8.GetBytes(responseString.ToCharArray()), new System.Xml.XmlDictionaryReaderQuotas());
var root = XElement.Load(jsonReader);
Console.WriteLine(root.XPathSelectElement("//name").Value);
//Console.WriteLine(responseString);
}
}
}
}
These are the problems I'm having.
In my Swift methods, they have completion handlers. How can I do the same in C#?
In Swift, you get an NSData object and you can pass it to NSJSONSerialization to create a JSON object. In my current implementation, I get an XML exception at XElement.Load(jsonReader);. I'm not sure if this is the correct way to do this even. I found tons of different solutions here on SO. But some are for Metro apps, some are for web it's all too overwhelming. Also most solutions are on using third-party libraries like JSON.NET. I'm trying to achieve this without third-party libraries.

In my Swift methods, they have completion handlers. How can I do the
same in C#?
The point of wiring up a completion handler is so that you don't tie up a thread while waiting for the HTTP call to complete. The beauty of async/await is that you don't have to do this in C#. The await keyword instructs the compiler to literally rewrite the rest of the method as a callback. The current thread is freed as soon as await is encountered, preventing your UI from freezing up. You have written your async code correctly; it will behave asynchronously even though it looks synchronous.
Your second question is a bit broad, but I will make 2 suggestions:
Don't use XElement when dealing with JSON data. That part of an Microsoft's XML parsing library (one of them) and has nothing to do with JSON.
I'm not sure why achieving this without a 3rd-party library is important. I know people have their reasons, but Json.NET in particular has become so popular and ubiquitous that Microsoft itself has baked it into their ASP.NET MVC and Web API frameworks. That said, if you must avoid it, here is how you would deserialize JSON using only Microsoft libraries.