I started down the path of using HttpClient as I thought the service I was accessing is a REST service. Turns out it's a JSON service running on port 80 but is a socket application.
The HttpClient opens the remote port but when it sends the JSON request it never gets a response. I was having the hardest time getting fiddler to get a response back as well. But I was able to get wget and curl to send/receiving a response. That's when I talked to the original developer and he mentioned that it wasn't a true "REST" service, but just a socket application that sends/receives JSON.
Is there something I can do to tweak HttpClient to access a socket application or am I going to have to take a step back and use WebSockets?
This is the test code that sends/receives the JSON packet.
private async Task ProcessZone(string szIPAddress)
{
string responseData = string.Empty;
Uri baseAddress = new Uri(#"http://" + szIPAddress + "/player");
try
{
using (var httpClient = new HttpClient { BaseAddress = baseAddress })
{
var _req = new SendRequest();
_req.id = "rec-100";
_req.url = "/stable/av/";
_req.method = "browse";
var _json = JsonConvert.SerializeObject(_req);
using (var content = new StringContent(_json,Encoding.UTF8, "application/json"))
{
using (var response = await httpClient.PostAsync(baseAddress, content))
{
responseData = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
}
}
}
var _Response = JsonConvert.DeserializeObject<Response>(responseData);
var item = new ZoneInfo();
item.szIPAddress = szIPAddress;
item.szZoneName = _Response.result.item.title;
lstZones.Add(item);
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
private async Task ProcessZones()
{
foreach (var item in ZoneSearch)
{
await ProcessZone(item.IPAddress);
}
}
The connection hangs on this line:
using (var response = await httpClient.PostAsync(baseAddress, content))
I should also mention that the code above does work fine on a true rest service...
That's when I talked to the original developer and he mentioned that it wasn't a true "REST" service, but just a socket application that sends/receives JSON.
Knowing the protocol is the first step towards making a working client.
Is there something I can do to tweak HttpClient to access a socket application or am I going to have to take a step back and use WebSockets?
Neither, unfortunately. HttpClient - as the name implies - only works with HTTP services. Since the server is not an HTTP server, it won't work with HttpClient. WebSockets have a rather confusing name, since they are not raw sockets but instead use the WebSocket protocol, which require an HTTP handshake to set up. Since the server is not an HTTP/WebSocket server, it won't work with WebSockets.
Your only choices are to either pressure the developer to write a real REST service (which makes your job orders of magnitude easier), or use raw sockets (e.g., Socket). Correctly using raw sockets is extremely difficult, so I recommend you pressure the developer to write a REST service like the entire rest of the world does today.
Related
I must individually test each server of a cluster. Each server have one IP address but I must use same URL. I am using a console project.
static string RequestGet(string requestUrl, string ipspecify)
{
HttpWebRequest request = HttpWebRequest.CreateHttp(requestUrl);
// Submit the request, and get the response body.
string responseBodyFromRemoteServer;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
responseBodyFromRemoteServer = reader.ReadToEnd();
}
}
return responseBodyFromRemoteServer;
}
Locally, I can modify my host file, but I want to do this automatically through my program.
RequestGet("https://toto.org/myservice", "172.2.240.16")
RequestGet("https://toto.org/myservice", "172.2.240.17")
IP address is server address.
What is the solution?
Turns out, as of .NET 5, this is possible using HttpClient. SocketsHttpHandler, the default handler used by HttpClient, gained a ConnectCallback property which lets you override how a connection to the remote machine is established.
Here's where it's called, which you can use as inspiration for writing your own. The following seems to work fine:
public static async Task Main()
{
using var client = new HttpClient(new SocketsHttpHandler() { ConnectCallback = ConnectCallback });
await client.GetStringAsync("http://example.com");
}
private static async ValueTask<Stream> ConnectCallback(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
{
var endPoint = new DnsEndPoint("1.2.3.4", context.DnsEndPoint.Port);
var socket = new Socket(SocketType.Stream, ProtocolType.Tcp) { NoDelay = true };
await socket.ConnectAsync(endPoint, cancellationToken);
return new NetworkStream(socket, ownsSocket: true);
}
The request is made to 1.2.3.4 (or whatever IP you specify), but everything above the level of the TCP socket itself, including TLS, the Host header, etc, is still set to example.com, meaning that you shouldn't get certificate errors.
I am sending cURL request using HttpClient through the method described here under.
The parameter used for this method are:
SelectedProxy = a custom class that stores my proxy's parameters
Parameters.WcTimeout = the timeout
url, header, content = the cURL request (based on this tool to convert to C# https://curl.olsh.me/).
const SslProtocols _Tls12 = (SslProtocols)0x00000C00;
const SecurityProtocolType Tls12 = (SecurityProtocolType)_Tls12;
ServicePointManager.SecurityProtocol = Tls12;
string source = "";
using (var handler = new HttpClientHandler())
{
handler.UseCookies = usecookies;
WebProxy wp = new WebProxy(SelectedProxy.Address);
handler.Proxy = wp;
using (var httpClient = new HttpClient(handler))
{
httpClient.Timeout = Parameters.WcTimeout;
using (var request = new HttpRequestMessage(new HttpMethod(HttpMethod), url))
{
if (headers != null)
{
foreach (var h in headers)
{
request.Headers.TryAddWithoutValidation(h.Item1, h.Item2);
}
}
if (content != "")
{
request.Content = new StringContent(content, Encoding.UTF8, "application/x-www-form-urlencoded");
}
HttpResponseMessage response = new HttpResponseMessage();
try
{
response = await httpClient.SendAsync(request);
}
catch (Exception e)
{
//Here the exception happens
}
source = await response.Content.ReadAsStringAsync();
}
}
}
return source;
If I am running this without proxy, it works like a charm.
When I send a request using a proxy which I tested first from Chrome, I have the following error on my try {} catch {}. Here is the error tree
{"An error occurred while sending the request."}
InnerException {"Unable to connect to the remote server"}
InnerException {"A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond [ProxyAdress]"}
SocketErrorCode: TimedOut
By using a Stopwatch I see that the TimedOut occurred after around 30 sec.
I tried a few different handler based on the following links What's the difference between HttpClient.Timeout and using the WebRequestHandler timeout properties?, HttpClient Timeout confusion or with the WinHttpHandler.
It's worth noting that WinHttpHandler allow for a different error code, i.e. Error 12002 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The operation timed out'. The underlying reason is the same though it helped to target where it bugs (i.e. WinInet) which confirms also what #DavidWright was saying regarding that timeouts from HttpClient manages a different part of the request sending.
Hence my issue is coming from the time it takes to establish a connection to the server, which triggers the 30sec timeout from WinInet.
My question is then How to change those timeout?
On a side note, it's worth noting that Chrome, which uses WinInet, does not seem to suffer from this timeout, nor Cefsharp on which a big part of my app is based, and through which the same proxies can properly send requests.
So thanks to #DavidWright I understand a few things:
Before that the HttpRequestMessage is sent and the timeout from HttpClient starts, a TCP connection to the server is initiated
The TCP connection has its own timeout, defined at OS level, and we do not identified a way to change it at run time from C# (question pending if anyone want to contribute)
Insisting on trying to connect works as each try benefits from previous tries, though proper exception management & manual timeout counter needs to be implemented (I actually considered a number of tries in my code, assuming each try is around 30sec)
All this together ended up in the following code:
const SslProtocols _Tls12 = (SslProtocols)0x00000C00;
const SecurityProtocolType Tls12 = (SecurityProtocolType)_Tls12;
ServicePointManager.SecurityProtocol = Tls12;
var sp = ServicePointManager.FindServicePoint(endpoint);
sp.ConnectionLeaseTimeout = (int)Parameters.ConnectionLeaseTimeout.TotalMilliseconds;
string source = "";
using (var handler = new HttpClientHandler())
{
handler.UseCookies = usecookies;
WebProxy wp = new WebProxy(SelectedProxy.Address);
handler.Proxy = wp;
using (var client = new HttpClient(handler))
{
client.Timeout = Parameters.WcTimeout;
int n = 0;
back:
using (var request = new HttpRequestMessage(new HttpMethod(HttpMethod), endpoint))
{
if (headers != null)
{
foreach (var h in headers)
{
request.Headers.TryAddWithoutValidation(h.Item1, h.Item2);
}
}
if (content != "")
{
request.Content = new StringContent(content, Encoding.UTF8, "application/x-www-form-urlencoded");
}
HttpResponseMessage response = new HttpResponseMessage();
try
{
response = await client.SendAsync(request);
}
catch (Exception e)
{
if(e.InnerException != null)
{
if(e.InnerException.InnerException != null)
{
if (e.InnerException.InnerException.Message.Contains("A connection attempt failed because the connected party did not properly respond after"))
{
if (n <= Parameters.TCPMaxTries)
{
n++;
goto back;
}
}
}
}
// Manage here other exceptions
}
source = await response.Content.ReadAsStringAsync();
}
}
}
return source;
On a side note, my current implementation of HttpClient may be problematic in the future. Though being disposable, HttpClient should be defined at App level through a static, and not within a using statement. To read more about this go here or there.
My issue is that I want to renew the proxy at each request and that it is not set on a per request basis. While it explains the reasdon of the new ConnectionLeaseTimeout parameter (to minimize the time the lease remains open) it is a different topic
I have had the same problem with HttpClient. Two things need to happen for SendAsync to return: first, setting up the TCP channel over which the communication occurs (the SYN, SYN/ACK, ACK handshake, if you're familiar with that) and second getting back the data that constitutes the HTTP response over that TCP channel. HttpClient's timeout only applies to the second part. The timeout for the first part is governed by the OS's network subsystem, and it's quite difficult to change that timeout in .NET code.
(Here's how you can reproduce this effect. Set up a working client/server connection between two machines, so you know that name resolution, port access, listening, and client and server logic all works. Then unplug the network cable on the server and re-run the client request. It will time out with the OS's default network timeout, regardless of what timeout you set on your HttpClient.)
The only way I know around this is to start your own delay timer on a different thread and cancel the SendAsync task if the timer finishes first. You can do this using Task.Delay and Task.WaitAny or by creating a CancellationTokenSource with your desired timeone (which essentially just does the first way under the hood). In either case you will need to be careful about cancelling and reading exceptions from the task that loses the race.
I'm using hybrid connections to request data from a listener. If I can write and read to the connection, how can I know that the response I've read from the connection matches the request I've given it? For example:
private HybridConnectionClient _client = new HybridConnectionClient(***);
public override async Task<RelayResponse> SendAsync(RelayRequest request)
{
var stream = await _client.CreateConnectionAsync();
var writer = new StreamWriter(stream) { AutoFlush = true };
var reader = new StreamReader(stream);
var reqestSerialized = JsonConvert.SerializeObject(request);
await writer.WriteLineAsync(reqestSerialized);
string responseSerialized = await reader.ReadLineAsync();
var response = JsonConvert.DeserializeObject<RelayResponse>(responseSerialized);
return response;
}
If the listener on this connection is reading and responding to many requests at the same time, is there anyway to know that the next Readline() we do on the client side to get the response is the one that is associated with the request? Or is that something that has to be managed?
Understanding a bit more about azure relay hybrid connections, I understand this now.
There isn't really any concept of a synchronous request/response in the framework, but if you use a new connection for each request, and respond on that same connection in the listener, you can be sure the response is for the request you sent.
Spawn a new connection for each request, then make sure the response is written to that connection. So looking at Microsoft's listener example code, whenever listener.AcceptConnectionAsync() fires do all the message response on relayConnection, then go back to waiting at await listener.AcceptConnectionAsync();
while (true)
{
var relayConnection = await listener.AcceptConnectionAsync();
if (relayConnection == null)
{
break;
}
ProcessMessagesOnConnection(relayConnection, cts);
}
I have tried to create a simple console application.
We have a call system from 8x8 that provide a web streaming API but their documentation is very limited and nothing in C#.
The api service streams call statuses in near real time and I would like to get that 'stream' and be able to read and process it in realtime if possible. The response or Content Type is 'text/html'. But the actual body of the response can be declared as json - sample below:
{"Interaction":{"attachedData":{"attachedDatum":[{"attachedDataKey":"#pri","attachedDataValue":100},{"attachedDataKey":"callingName","attachedDataValue":999999999999},{"attachedDataKey":"cha","attachedDataValue":99999999999},{"attachedDataKey":"cnt","attachedDataValue":0},{"attachedDataKey":"con","attachedDataValue":0},{"attachedDataKey":"med","attachedDataValue":"T"},{"attachedDataKey":"pho","attachedDataValue":9999999999},{"attachedDataKey":"phoneNum","attachedDataValue":9999999999},{"attachedDataKey":"tok","attachedDataValue":999999999}]},"event":"InteractionCreated","inboundChannelid":9999999999,"interactionEventTS":9999999,"interactionGUID":"int-15b875d0da2-DJOJkDhDsrh3AIaFP8VkICv9t-phone-01-testist","resourceType":0}}
I have seen several posts concerning httpClient and the GetAsync methods but none of these appear to work as they appear to be for calls when a response is made, not something that constantly has a response.
Using fiddler for the call it does not appear to close so the stream is constantly running, so fiddler does not display any data until a separate user or instance connects.
When I use a browser the content is 'streamed' to the page and updates automatically and shows all the content (as above).
The api contains authentication so when another client connects and retrieves data the connected client closes and finally I am able to see the data that was gathering.
This is the code so and does return the big stream when another client connects but ideally I want a real time response and appears to just get stuck in the GETASYNC method:
var response = await client.GetAsync(address, HttpCompletionOption.ResponseHeadersRead);
if (response.IsSuccessStatusCode)
{
var responseContent = response.Content;
string responseString = await responseContent.ReadAsStringAsync();
Console.WriteLine(responseString);
}
Hopefully that's enough information for one of you clever people to help me in my predicament.
I was also having an issue consuming their streaming API and the examples I found that worked with the Twitter and CouchBase streaming API's did not work with 8x8. Both Twitter and CouchBase send line terminators in their pushes so the solution relied on ReadLine to pull in the feed. Since 8x8 does not send terminators you'll need to use ReadBlock or better ReadBlockAsync.
The following code shows how to connect using credentials and consume their feed:
private static async Task StreamAsync(string url, string username, string password)
{
var handler = new HttpClientHandler()
{
Credentials = new NetworkCredential {UserName = username, Password = password},
PreAuthenticate = true
};
// Client can also be singleton
using (var client = new HttpClient(handler))
{
client.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
var request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.Connection.Add("keep-alive");
using (var response = await client.SendAsync(
request,
HttpCompletionOption.ResponseHeadersRead))
{
using (var body = await response.Content.ReadAsStreamAsync())
{
using (var reader = new StreamReader(body))
{
while (!reader.EndOfStream)
{
var buffer = new char[1024];
await reader.ReadBlockAsync(buffer, 0, buffer.Length);
Console.WriteLine(new string(buffer));
}
}
}
}
}
}
I am testing a REST API post, and it works well when I try it on Postman. However, in some scenario (related to the posting XML data) if I post with HttpClient API, I would receive the following error:
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
But the same XML content works fine on Postman with status OK and proper response.
What is the differences between using the C# HttpClient API and the postman testing? How can I configure my API call to match with the behavior on postman?
Here I attached the source code, and the Postman screenshot
public void createLoan()
{
string baseCreateLoanUrl = #"https://serverhost/create?key=";
var strUCDExport = XDocument.Load(#"C:\CreateLoan_testcase.xml");
using (var client = new HttpClient())
{
var content = new StringContent(strUCDExport.ToString(), Encoding.UTF8, Mediatype);
string createLoanApi = string.Concat(baseCreateLoanUrl, APIKey);
try
{
var response = client.PostAsync(createLoanApi, content).Result;
}
catch (Exception ex)
{
MessageBox.Show("Error Happened here...");
throw;
}
if (response.IsSuccessStatusCode)
{
// Access variables from the returned JSON object
string responseString = response.Content.ReadAsStringAsync().Result;
JObject jObj = JObject.Parse(responseString);
if (jObj.SelectToken("failure") == null)
{
// First get the authToken
string LoanID = jObj["loanStatus"]["id"].ToString();
MessageBox.Show("Loan ID: " + LoanID);
}
else
{
string getTokenErrorMsg = string.Empty;
JArray errorOjbs = (JArray) jObj["failure"]["errors"];
foreach (var errorObj in errorOjbs)
{
getTokenErrorMsg += errorObj["message"].ToString() + Environment.NewLine;
}
getTokenErrorMsg.Dump();
}
}
}
Thanks for Nard's comment, after comparing the header, I found the issue my client header has this:
Expect: 100-continue
While postman doesn't has.
Once I removed this by using the ServicePointManager:
ServicePointManager.Expect100Continue = false;
Everything seems fine now. Thanks all the input!
My gut tells me it's something simple. First, we know the API works, so I'm thinking it's down to how you are using the HttpClient.
First things first, try as suggested by this SO answer, creating it as a singleton and drop the using statement altogether since the consensus is that HttpClient doesn't need to be disposed:
private static readonly HttpClient HttpClient = new HttpClient();
I would think it would be either there or an issue with your content encoding line that is causing issues with the API. Is there something you are missing that it doesn't like, I bet there is a difference in the requests in Postman vs here. Maybe try sending it as JSON ala:
var json = JsonConvert.SerializeObject(strUCDExport.ToString());
var content = new StringContent(json, Encoding.UTF8, Mediatype);
Maybe the header from Postman vs yours will show something missing, I think the real answer will be there. Have fiddler running in the background, send it via Postman, check it, then run your code and recheck. Pay close attention to all the attribute tags on the header from Postman, the API works so something is missing. Fiddler will tell you.
I was struggling with this for 2 days when I stumbled over Fiddler which lets you record the traffic to the service. After comparing the calls I saw that I had missed a header in my code.