I would like to convert several data sets with a service in the background through Nominatim server. Now I've invested a lot of time to get an answer from the nominatim with the Http client class. Unfortunately, I only ever get a 404 error
Here is my sample code:
namespace ConsoleApp7
{
internal class Program
{
static void Main(string[] args)
{
Task t = new Task(DownloadPageAsync);
t.Start();
Console.WriteLine("Downloading page...");
Console.ReadLine();
}
static async void DownloadPageAsync()
{
// ... Target page.
string page = "https://nominatim.openstreetmap.org/reverse?
format=geocodejson&lat=60.2299&lon=11.1663";
// ... Use HttpClient.
using (HttpClient client = new HttpClient())
using (HttpResponseMessage response = await
client.GetAsync(page))
using (HttpContent content = response.Content)
{
// ... Read the string.
string result = await content.ReadAsStringAsync();
// ... Display the result.
if (result != null)
{
Console.WriteLine(result);
}
}
}
}
}
I'm happy about any help!
Many thanks
The problem is that website doesn't allow any default http request without user agent. You can solve this adding a useragent to your request. Code like below:
static void Main(string[] args)
{
Task t = new Task(DownloadPageAsync);
t.Start();
Console.WriteLine("Downloading page...");
Console.ReadLine();
}
static async void DownloadPageAsync()
{
// ... Target page.
string page = "https://nominatim.openstreetmap.org/reverse?format=geocodejson&lat=60.2299&lon=11.1663";
// ... Use HttpClient.
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (compatible; AcmeInc/2.0)");
using (HttpResponseMessage response = await
client.GetAsync(page))
using (HttpContent content = response.Content)
{
// ... Read the string.
string result = await content.ReadAsStringAsync();
// ... Display the result.
if (result != null)
{
Console.WriteLine(result);
}
}
}
}
You simply need to set a User-Agent
var page = "https://nominatim.openstreetmap.org/reverse?format=geocodejson&lat=60.2299&lon=11.1663";
// ... Use HttpClient.
using var client = new HttpClient();
client.DefaultRequestHeaders.Add("User-Agent", "WHATEVER VALUE");
using var response = await client.GetAsync(page);
using var content = response.Content;
// ... Read the string.
var result = await content.ReadAsStringAsync();
// ... Display the result.
if (result != null) Console.WriteLine(result);
Related
I am using Google's GeoCoding API. I have two methods where one works and the other doesn't and I can't seem to figure out why:
string address = "1400,Copenhagen,DK";
string GoogleMapsAPIurl = "https://maps.googleapis.com/maps/api/geocode/json?address={0}&key={1}";
string GoogleMapsAPIkey = "MYSECRETAPIKEY";
string requestUri = string.Format(GoogleMapsAPIurl, address.Trim(), GoogleMapsAPIkey);
// Works fine
using (var client = new HttpClient())
{
using (HttpResponseMessage response = await client.GetAsync(requestUri))
{
var responseContent = response.Content.ReadAsStringAsync().Result;
response.EnsureSuccessStatusCode();
}
}
// Doesn't work
using (HttpClient client = new HttpClient())
{
client.BaseAddress = new Uri("https://maps.googleapis.com/maps/api/", UriKind.Absolute);
client.DefaultRequestHeaders.Add("key", GoogleMapsAPIkey);
using (HttpResponseMessage response = await client.GetAsync("geocode/json?address=1400,Copenhagen,DK"))
{
var responseContent = response.Content.ReadAsStringAsync().Result;
response.EnsureSuccessStatusCode();
}
}
My last method with GetAsync where I am sending a query string doesn't work and I am in doubt why it is so. When I introduce BaseAddress on the client the GetAsync somehow doesn't send to the correct URL.
I don't suggest adding API key into globals. Maybe you'll need to send some HTTP request outside of the API and the key will be leaked.
Here's the example that works.
using Newtonsoft.Json;
public class Program
{
private static readonly HttpClient client = new HttpClient();
private const string GoogleMapsAPIkey = "MYSECRETAPIKEY";
static async Task Main(string[] args)
{
client.BaseAddress = new Uri("https://maps.googleapis.com/maps/api/");
try
{
Dictionary<string, string> query = new Dictionary<string, string>();
query.Add("address", "1400,Copenhagen,DK");
dynamic response = await GetAPIResponseAsync<dynamic>("geocode/json", query);
Console.WriteLine(response.ToString());
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
Console.ReadKey();
}
private static async Task<string> ParamsToStringAsync(Dictionary<string, string> urlParams)
{
using (HttpContent content = new FormUrlEncodedContent(urlParams))
return await content.ReadAsStringAsync();
}
private static async Task<T> GetAPIResponseAsync<T>(string path, Dictionary<string, string> urlParams)
{
urlParams.Add("key", GoogleMapsAPIkey);
string query = await ParamsToStringAsync(urlParams);
using (HttpResponseMessage response = await client.GetAsync(path + "?" + query, HttpCompletionOption.ResponseHeadersRead))
{
response.EnsureSuccessStatusCode();
string responseText = await response.Content.ReadAsStringAsync();
return JsonConvert.DeserializeObject<T>(responseText);
}
}
}
Ciao, the problem is related with key parameter on URL. Change your code like this:
using (HttpClient client = new HttpClient())
{
client.BaseAddress = new Uri("https://maps.googleapis.com/maps/api/");
using (HttpResponseMessage response = await client.GetAsync("geocode/json?address=1400,Copenhagen,DK&key=" + GoogleMapsAPIkey))
{
var responseContent = response.Content.ReadAsStringAsync().Result;
response.EnsureSuccessStatusCode();
}
}
As google sheets said:
After you have an API key, your application can append the query parameter key=yourAPIKey to all request URLs. The API key is safe for embedding in URLs; it doesn't need any encoding.
I was following this example to use the VSTS REST API:
https://learn.microsoft.com/en-us/rest/api/vsts/search/code%20search%20results/fetch%20code%20search%20results?view=vsts-rest-4.1
The following url points directly to MyService.MyController in my org's VSTS:
https://my-corp.visualstudio.com/DefaultCollection/USOpsInfrastructure/USOpsInfrastructure%20Team/_git/MyService?path=%2FMyService.UI%2FControllers%2FMyController.cs&version=GBmaster
I tried to follow the example code with my implentation code below but a 404 response is returned without a detailed message. Any idea what the issue might be or how to debug?
private static async Task FindMyControllerRefs()
{
var baseUrl = "https://my-corp.almsearch.visualstudio.com";
using (var client = GetHttpClient(baseUrl))
{
var request = "{ \"searchText\": \"MyController\" }";
var projectUri = "/USOpsInfrastructure/USOpsInfrastructure%20Team/_git/MyService";
var searchUri = "/_apis/search/codesearchresults?api-version=4.1-preview.1";
var fullUri = projectUri + searchUri;
var response = await client.PostAsJsonAsync(fullUri, request);
//process the response
if (response.IsSuccessStatusCode)
{
var result = (await response.Content.ReadAsAsync<string>());
}
else //not 200
{
var message = await response.Content.ReadAsStringAsync();
}
}
}
private static HttpClient GetHttpClient(string baseUrl)
{
var client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true });
client.BaseAddress = new Uri(baseUrl);
client.Timeout = Timeout.InfiniteTimeSpan;
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(
new MediaTypeWithQualityHeaderValue("application/json"));
return client;
}
The Code search REST API should be :
POST https://XXX.almsearch.visualstudio.com/{Project}/_apis/search/codesearchresults?api-version=4.1-preview.1
In you scenario please confirm what's the team project name: USOpsInfrastructure or MyService. If you are not sure about that, then you can have a try for each of them.
Besides, base on my test you need to add the "$top": 10 (you can change the number as needed) in the request body:
var request = "{ \"searchText\": \"MyController\",\"$top\": 10 }";
Below code works on my side: (Project name is "Git" in below sample)
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;
namespace CodeSearch
{
class Program
{
public static void Main()
{
Task t = CodeSearch();
Task.WaitAll(new Task[] { t });
}
private static async Task CodeSearch()
{
try
{
var username = "username";
var password = "Password/PAT";
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Add(
new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic",
Convert.ToBase64String(
System.Text.ASCIIEncoding.ASCII.GetBytes(
string.Format("{0}:{1}", username, password))));
string url = "https://{instance}.almsearch.visualstudio.com/Git/_apis/search/codesearchresults?api-version=4.1-preview.1";
var content = new StringContent("{\"searchText\": \"OwinStartupAttribute\", \"$top\": 10}", Encoding.UTF8, "application/json");
using (HttpResponseMessage response = client.PostAsync(url, content).Result)
{
response.EnsureSuccessStatusCode();
string responseBody = await response.Content.ReadAsStringAsync();
Console.WriteLine(responseBody);
}
Console.ReadLine();
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
Console.ReadLine();
}
}
}
I am doing HTTP get request using HttpClient in C# console app . I am not getting expected response with one get request.
Get Request is like
http://example.com/xyz/SearchProduct?productNo=11210&1d6rstc9xc=5jyi27htzk
I am getting some vague response but when i do same get request with fiddler it is giving expected response.
How can I get expected response from httpClient.GetAsync(url)?
code is :-
var httpClient = new HttpClient();
var url = "http://example.com/xyz/SearchProduct?productNo=11210&1d6rstc9xc=5jyi27htzk";
HttpResponseMessage response1 = await httpClient.GetAsync(url);
if (response1.IsSuccessStatusCode)
{
HttpContent stream = response1.Content;
Task<string> data = stream.ReadAsStringAsync();
}
You should read as string that way:
string result = await stream.ReadAsStringAsync();
instead of that:
Task<string> data = stream.ReadAsStringAsync();
Here full code example and another example
This is a full method using async/await approach.
private static async Task<string> GetRequestContentAsString(string url)
{
var data = string.Empty;
using (var httpClient = new HttpClient())
{
var response = await httpClient.GetAsync(url);
if (response.IsSuccessStatusCode)
{
var stream = response.Content;
data = await stream.ReadAsStringAsync();
}
}
return data;
}
This method is called this way:
var content = await GetRequestContentAsString("http://www.bing.com");
I have an incoming POST request from a program that consists of JSON data.
This is my server code:
static HttpListener _httpListener = new HttpListener();
static void ResponseThread()
{
while (true)
{
HttpListenerContext context = _httpListener.GetContext(); // get a context
// Now, you'll find the request URL in context.Request.Url
HttpListenerRequest request = context.Request;
string test = "";
using (http://System.IO (http://System.IO).Stream body = request.InputStream) // here we have data
{
using (http://System.IO (http://System.IO).StreamReader reader = new http://System.IO (http://System.IO).StreamReader(body, request.ContentEncoding))
{
test = reader.ReadToEnd();
}
}
Console.WriteLine(test);
byte[] _responseArray = Encoding.UTF8.GetBytes(test); // get the bytes to response
context.Response.OutputStream.Write(_responseArray, 0, _responseArray.Length); // write bytes to the output stream
context.Response.KeepAlive = false; // set the KeepAlive bool to false
context.Response.Close(); // close the connection
Console.WriteLine("Respone given to a request.");
}
}
static void Main(string[] args)
{
Console.WriteLine("Starting server...");
_httpListener.Prefixes.Add("http://localhost:5000/ (http://localhost:5000/)"); // add prefix "http://localhost:5000/ (http://localhost:5000/)"
_httpListener.Start(); // start server (Run application as Administrator!)
Console.WriteLine("Server started.");
Thread _responseThread = new Thread(ResponseThread);
_responseThread.Start(); // start the response thread
}
This is the posting code i'm using outside of the server code in a different project
static string httpPost(string json)
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create("http://localhost:5000/");
httpWebRequest.ContentType = "text/json";
httpWebRequest.Method = "POST";
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
}
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
return result;
}
}
and I want to display the "test" variable in my browser but for right now it isn't working at all. Nothing is displayed but if I just send some html it works. Is there anyway to get this working or parse it out so that it does work?
To augment my comment on your question, here's a "Hello World" of sorts using the Owin self hosted server. It mimics what your question was trying to do. First install NuGet package Microsoft.Owin.SelfHost. Then include the System.Net.Http and System.Net.Http.Formatting .Net framework references. Included also is a code example to call the self-hosted server.
using Microsoft.Owin;
using Microsoft.Owin.Hosting;
using Owin;
using System;
using System.IO;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
namespace StackOverflowAnswer
{
class Startup
{
public void Configuration(IAppBuilder app)
{
app.Use<RespondToRequestMiddleware>();
}
}
class RespondToRequestMiddleware : OwinMiddleware
{
public RespondToRequestMiddleware(OwinMiddleware next)
: base(next)
{
}
public async override Task Invoke(IOwinContext context)
{
// Perform request stuff...
// Could verify that the request Content-Type == application/json
// Could verify that the request method was POST.
bool isPost = context.Request.Method == "POST";
string test = null;
if (isPost)
{
using (StreamReader reader = new StreamReader(context.Request.Body))
{
test = await reader.ReadToEndAsync();
}
}
await Next.Invoke(context); // pass off request to the next middleware
if (isPost)
{
// Perform response stuff...
context.Response.ContentType = "application/json";
context.Response.StatusCode = 200;
await context.Response.WriteAsync(test);
Console.WriteLine("Response given to a request.");
}
}
}
class Program
{
static void Main(string[] args)
{
string url = "http://localhost:5000/";
using (WebApp.Start<Startup>(url))
{
Console.WriteLine($"Listening on {url}...");
using (HttpClient httpClient = new HttpClient())
{
string json = "{\"key\":\"value\", \"otherKey\":\"secondValue\"}";
// Typically use the extensions in System.Net.Http.Formatting in order to post a strongly typed object with HttpClient.PostAsJsonAsync<T>(url)
StringContent content = new StringContent(json, Encoding.UTF8, "application/json");
HttpResponseMessage response = httpClient.PostAsync(url, content).Result; // IMPORTANT: use await in an async method in the real world.
if (response.IsSuccessStatusCode)
{
string responseJson = response.Content.ReadAsStringAsync().Result; // Again: use await in an async method in the real world.
Console.WriteLine(responseJson); // In your method, return the string.
}
else
{
Console.WriteLine($"Unsuccessful {response.StatusCode} : {response.ReasonPhrase}");
}
}
Console.ReadLine(); // keep console from closing so server can keep listening.
}
}
}
}
In action:
Check out the Microsoft Owin/Katana site for more info.
I'm working with Windows Phone Runtime API.
I declare a timer, which every 2 seconds does async http connection in Listen method.
Timer t = new Timer(Listen, null, 0, 2000);
Listen method:
private async void Listen(object state)
{
string url = "http://www.mywebpage.com?data=my_data";
string responseBodyAsText = null;
var response = new HttpResponseMessage();
var httpClient = new HttpClient();
try
{
response = await httpClient.GetAsync(new Uri(url));
responseBodyAsText = await response.Content.ReadAsStringAsync();
}
catch
{
//...
}
Debug.WriteLine(responseBodyAsText);
httpClient.Dispose();
}
My problem is that responseBodyAsText contains always the same data (given the same uri) and not as I would expect different data according to my external actions (modifying web page or different results with the same uri).
Does HttpClient remembers content during liftime of application? How can I solve this problem?
HttpClient does have caching on by default. You can turn it off by passing it an HttpBaseProtocolFilter:
var filter = new HttpBaseProtocolFilter
{
CacheControl.ReadBehavior = HttpCacheReadBehavior.MostRecent,
CacheControl.WriteBehavior = HttpCacheWriteBehavior.NoCache;
}
Side note: You could also, instead of a Timer, use Task.Delay to achieve the timer behavior (it internally uses one):
private async Task ListenAsync()
{
while (someCondition)
{
string url = "http://www.mywebpage.com?data=my_data";
string responseBodyAsText = null;
var response = new HttpResponseMessage();
using (var httpClient = new HttpClient())
{
try
{
response = await httpClient.GetAsync(new Uri(url));
responseBodyAsText = await response.Content.ReadAsStringAsync();
}
catch
{
//...
}
Debug.WriteLine(responseBodyAsText);
await Task.Delay(2000);
}
}