Develop Reference

HTTP GET instead of HTTP Connect in .NET / C#

how do I do HTTP tunneling without using the HTTP Connect method but using HTTP Get method instead. https://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_tunneling I tried IHttpClientFactory and RestSharp both are establishing HTTP Connect. In the node.js request module, there is a param called tunnel which when false establish the connection using HTTP Get, how do I do something similar in .NET. Would really appreciate the help, have been looking for this for a while
Capture1 - C# when using a proxy server, http client creates a tunnel using HTTP Connect
Frame 331: 159 bytes on wire (1272 bits), 159 bytes captured (1272 bits) on interface \Device\NPF_{262229C1-C486-4F85-BCC8-BFC96981C755}, id 0
Ethernet II, Src: IntelCor_37:28:37 (dc:41:a9:37:28:37), Dst: TaicangT_80:60:30 (18:45:93:80:60:30)
Internet Protocol Version 4, Src: {myLocalIp}, Dst: {proxyServerIp}
Transmission Control Protocol, Src Port: 51694, Dst Port: 8888, Seq: 1, Ack: 1, Len: 105
Hypertext Transfer Protocol
CONNECT {host}:443 HTTP/1.1\r\n
Host: {host}:443\r\n
User-Agent: RestSharp/106.11.8.0\r\n
\r\n
[Full request URI: {url}:443]
[HTTP request 1/1]
[Response in frame: 341]
Capture2 - node.js with tunnel:false, this request also goes through the same proxy server but via a normal HTTP GET method and no tunnel
Frame 92: 393 bytes on wire (3144 bits), 393 bytes captured (3144 bits) on interface \Device\NPF_{262229C1-C486-4F85-BCC8-BFC96981C755}, id 0
Ethernet II, Src: IntelCor_37:28:37 (dc:41:a9:37:28:37), Dst: TaicangT_80:60:30 (18:45:93:80:60:30)
Internet Protocol Version 4, Src: {myLocalIp}, Dst: {proxyServerIp}
Transmission Control Protocol, Src Port: 51842, Dst Port: 8888, Seq: 1, Ack: 1, Len: 339
Hypertext Transfer Protocol
GET {url} HTTP/1.1\r\n
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15\r\n
x-cache-proxyname: i-05ad6154426f07671\r\n
host: {host}\r\n
Connection: close\r\n
\r\n
[Full request URI: {url}]
[HTTP request 1/1]
[Response in frame: 96]

Return Unauthorized (401) only when calling from code (c#)

UPDATE
As #Alexandru Clonțea suggested, I checked the fiddler log and found:
In both success or fail cases, there are actually 2 requests being sent. The first request are mostly the same for both cases, it's something like:
GET http://myservice.com/handler?param1=something&param2=somethingelse HTTP/1.1
Authorization: Basic xxxxxx
Accept: application/json, application/xml, text/json, text/x-json,
text/javascript, text/xml
User-Agent: RestSharp/100.0.0.0
Host: myservice.com
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
The response for them are the same, which is:
HTTP/1.1 301 Moved Permanently
Content-Type: text/html; charset=utf-8
Location: /handler/?param1=something&param2=somethingelse
Date: Sat, 08 Sep 2018 01:50:16 GMT
Content-Length: 115
Moved Permanently.
I have noticed that it always try to redirect the call to /handler/?param1=something&param2=somethingelse, and that's because of the setup of the server code. it's actually working as expected. The difference is in the second request. The second request of the failure case (which is the c# code) doesn't have the authorization header and that's why it failed. Now, my question will be, why does the second request miss the authorization header? How can I fix it? Below is an example of the failed request:
GET http://myservice.com/handler/?param1=something&param2=somethingelse HTTP/1.1
Accept: application/json, application/xml, text/json, text/x-json,
text/javascript, text/xml
User-Agent: RestSharp/100.0.0.0
Accept-Encoding: gzip, deflate
Host: myservice.com
Backgroud:
I have a service written in GO deployed on a server. It requires a basic authentication. For example, I can call it successfully with the following request:
GET /handler/?
param1=something&param2=somethingelse HTTP/1.1
> Host: myservice.com
> Authorization: Basic xxxxxx
> User-Agent: RestClient/5.16.6
> Accept: */*
The request above is made by a rest api client tool (like postman), and it's working fine. It's also working fine if I call it from a browser.
Problem:
Now, I try to make the same call to the same service using c# code, and I have it as:
// pass cert validation
ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
HttpClient client = new HttpClient();
var byteArray = Encoding.ASCII.GetBytes(username + ":" + password);
var auth = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", Convert.ToBase64String(byteArray));
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.Authorization = auth;
var response = client.SendAsync(request).Result; // don't need async
But in this case, I am getting Unauthorized (401) back. I have checked into the actually request that was sent by the code, it had exactly the same authorization header as the one shows above (Authorization: Basic xxxxxx, and the xxxxxx is the same as above) and same uri as well. Actually, everything it sent looks the same as when I used the rest api client tool, but it was just failed in code.
when I check the log on the server side, I see the log below when it returns 401:
[GIN-debug] redirecting request 301: /handler --> /hanlder/?param1=something&param2=somethingelse
but I don't see this log when the call is from the rest api client tool (or browser)
As you may know from the log, the server-side code is using the go gin framework. But since it works fine in other cases, I don't think it's a problem with the server-side code.
Back to the C# code, I have tried to use the HttpWebRequest with NetworkCredential instead of the HttpClient, and I also try to use client.DefaultRequestHeaders.Authorization = auth, but I was still getting the same error.
I am wondering if someone has seen this before or could help? It will be really appreciated.

As a workaround, I can modify the request to be http://myservice.com/handler/?param1=something&param2=somethingelse so that no redirection is needed. Thus, it will be correctly authorized.
But still, haven't figure out how to make the second request to be sent with the authorize header yet.

RestSharp - Service Unavailable - Maximum number of active clients reached

I'm using RestSharp to communication with an Web Service.
I use this code
public static object GetTagValue(string url, string tagname, out string resp)
{
object result = null;
resp = string.Empty;
string theReq = string.Format("tags/{0}", tagname);
var client = new RestClient(url);
var request = new RestRequest(theReq, Method.GET);
request.RequestFormat = DataFormat.Json;
IRestResponse response = client.Execute(request);
resp = response.Content;
if (!string.IsNullOrWhiteSpace(resp))
{
dynamic json = JValue.Parse(resp);
if (null != json.value)
{
result = json.value;
}
}
return result;
}
Call to the server
GET http://ame-hp/tags/int32 HTTP/1.1
Accept: application/json, application/xml, text/json, text/x-json,
text/javascript, text/xml
User-Agent: RestSharp/105.2.3.0
Host: ame-hp
Accept-Encoding: gzip, deflate
Response from the server for a working call:
HTTP/1.1 200 Ok
Server: Internet Pack HTTP Server
Connection: Close
Set-Cookie: SID=f11985564d;Expires=Fri, 27 Jan 2017 07:52:17
GMT;Path=/
Content-Type: application/json
Content-Length: 133
{"quality":"Good","description":"","name":"int32","value":0,"dataType":"int32","controllers":[],"initialValue":null,"readonly":false}
It's working but after two calls the service answers with this
{"Code":503,"Message":"Service Unavailable - Maximum number of active clients reached."}
Third call to server
GET http://ame-hp/tags/int32 HTTP/1.1
Accept: application/json, application/xml, text/json, text/x-json,
text/javascript, text/xml
User-Agent: RestSharp/105.2.3.0
Host: ame-hp
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Response from the server for all calls from now and until the server is restarted
HTTP/1.1 503 Service Unavailable - Maximum number of active clients
reached.
Server: Internet Pack HTTP Server
Connection: Keep-Alive
Content-Type: application/json
Content-Length: 88
{"Code":503,"Message":"Service Unavailable - Maximum number of active
clients reached."}
So I assume that the service has a limit of two clients.
But why do there exist two active clients?
Either the Server or RestSharp is not closing the connection, but which?
Is there something I can do in RestSharp to close the connection?

The problem was, as I assumed, that the server only allows 2 clients. On the first connection a Session Cookie is sent back and this has to be used for the rest of the calls.
In RestSharp you only need to add one line to make this happen
After creating the client (which by the way you need to reuse for all the calls). You add this line:
client.CookieContainer = new System.Net.CookieContainer();
The initialization of the client would be
client = new RestClient();
client.CookieContainer = new System.Net.CookieContainer();
Then you can use
client.BaseUrl = new Uri(url);
To set the url you want to call

What is the exact HTTP content used to call a WCF web service?

I have a WCF service, which was created using the Visual Studio 2013 New Project wizard. I can call this service using the WCF Test Client which works fine. However, I'd like to call the service from the command line using curl.
What I've done is copied the Request content from the WCF Test Client, which looks like:
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Header>
<Action s:mustUnderstand="1" xmlns="http://schemas.microsoft.com/ws/2005/05/addressing/none">http://tempuri.org/IService1/TestStr</Action>
</s:Header>
<s:Body>
<TestStr xmlns="http://tempuri.org/" />
</s:Body>
</s:Envelope>
And stuck it in a file called Test.txt. Then, from the command line I did:
curl --verbose --header "Content-Type:text/xml; charset=utf-8" -X POST -d #Test.txt http://localhost:64777/Service1.svc
What I get back is:
* Adding handle: conn: 0x50e5a0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x50e5a0) send_pipe: 1, recv_pipe: 0
* About to connect() to localhost port 64777 (#0)
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 64777 (#0)
> POST /Service1.svc HTTP/1.1
> User-Agent: curl/7.33.0
> Host: localhost:64777
> Accept: */*
> Content-Type:text/xml; charset=utf-8
> Content-Length: 343
>
* upload completely sent off: 343 out of 343 bytes
< HTTP/1.1 400 Bad Request
< Cache-Control: private
* Server Microsoft-IIS/8.0 is not blacklisted
< Server: Microsoft-IIS/8.0
< X-AspNet-Version: 4.0.30319
< X-SourceFiles: =?UTF-8?B?YzpcdXNlcnNcbWNocmlzdGVuc2VuXGRvY3VtZW50c1x2aXN1YWwgc3R1ZGlvIDIwMTNcUHJvamVjdHNcV2NmR2VuZXJpY3NcV2NmR2VuZXJpY3NcU2VydmljZTEuc3Zj?=
< X-Powered-By: ASP.NET
< Date: Tue, 06 May 2014 18:47:29 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
What's the difference between how the WCF Test Client calls my web service and how Curl calls my web service? Do I have the URL wrong? Am I missing some HTTP headers? Thanks!
Update:
I've tried the URL http://localhost:64777/Service1.svc/TestStr as well. I've also tried encoding the file:
%3Cs%3AEnvelope%20xmlns%3As%3D%22http%3A%2F%2Fschemas.xmlsoap.org%2Fsoap%2Fenvelope%2F%22%3E%0A%20%20%3Cs%3AHeader%3E%0A%20%20%20%20%3CAction%20s%3AmustUnderstand%3D%221%22%20xmlns%3D%22http%3A%2F%2Fschemas.microsoft.com%2Fws%2F2005%2F05%2Faddressing%2Fnone%22%3Ehttp%3A%2F%2Ftempuri.org%2FIService1%2FTestStr%3C%2FAction%3E%0A%20%20%3C%2Fs%3AHeader%3E%0A%20%20%3Cs%3ABody%3E%0A%20%20%20%20%3CTestStr%20xmlns%3D%22http%3A%2F%2Ftempuri.org%2F%22%20%2F%3E%0A%20%20%3C%2Fs%3ABody%3E%0A%3C%2Fs%3AEnvelope%3E
Both just result in Bad Request.

You can use Fiddler to capture the exact HTTP request. WCF tracing unfortunately does not capture the exact bytes sent over the wire. It reports a modified copy of the actual XML sent. Fiddler is not aware of web services so it does none of that.

It looks like you created a SOAP web service but I don't see any of the SOAP headers included in your example of the request data. In your web service code - add a line to capture the actual request being received by the service through the OperationContext object: OperationContext.Current.RequestContext.RequestMessage.ToString()
You should be able to capture the data from the RequestMessage property, save that to a file, and then submit using CURL.
(You might have to strip out the <Action> elements as I think those are added by WCF and might cause problems if you try to submit a request with them included)

SSL Handshake Timeout

I have 2 client authentication certificates issued by the same certificate authority. One of them enables me to connect to a HTTPS webservice, but the other does not when I use code similar to the following:
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create( endPointUrl );
X509Store store = new X509Store( StoreName.My, StoreLocation.LocalMachine );
store.Open( OpenFlags.MaxAllowed );
X509CertificateCollection col = (X509CertificateCollection)store.Certificates.Find( X509FindType.FindBySerialNumber, certificateSerialNumber, true );
httpWebRequest.ClientCertificates.Add( col[0] );
httpWebRequest.Method = "POST";
httpWebRequest.ContentType = contentType;
httpWebRequest.KeepAlive = false;
httpWebRequest.Timeout = 3000;
httpWebRequest.ContentLength = message.Length;
httpRequestStream = httpWebRequest.GetRequestStream();
When attempting to get the request stream, I get an InvalidOperationException with the message "The operation has timed out".
I've used System.Net.trace when attempting to connect with the certificate that fails and the log shows a connection timeout before "Attempting to restart the session using the user-provided certificate" and just after the first InitializeSecurityContext.
Wireshark show the following:
"TCP","j-link > https [SYN] Seq=0 Win=65535 Len=0 MSS=1260 SACK_PERM=1"
"TCP","https > j-link [SYN, ACK] Seq=0 Ack=1 Win=32768 Len=0 MSS=1380"
"TCP","j-link > https [ACK] Seq=1 Ack=1 Win=65535 Len=0"
"TLSv1","Client Hello"
"TLSv1","Server Hello"
"TCP","[TCP segment of a reassembled PDU]"
"TCP","j-link > https [ACK] Seq=78 Ack=2521 Win=65535 Len=0"
"TLSv1","Certificate, Certificate Request, Server Hello Done"
"TCP","j-link > https [ACK] Seq=78 Ack=3187 Win=64869 Len=0"
"TCP","j-link > https [FIN, ACK] Seq=78 Ack=3187 Win=64869 Len=0"
"TCP","https > j-link [ACK] Seq=3187 Ack=79 Win=32768 Len=0"
"TLSv1","Alert (Level: Warning, Description: Close Notify)"
"TCP","j-link > https [RST, ACK] Seq=79 Ack=3194 Win=0 Len=0"
I can connect using OpenSSL from the command line using both certificates after exporting them and converting them to the PEM format.
Any suggestions would be greatly appreciated.

Thank you to Shawn's question which helped me fix the timeout problem, which was due to the connection taking over 60 seconds to fail.
I then got a "The request was aborted: Could not create SSL/TLS secure channel." error, which was solved by using the Windows HTTP Services Certificate Configuration Tool and information I obtained here.