I execute a lot of request to some the resources with HttpClient.
To avoid licks I use it as single instance. Something like that...
I want to use proxy, so how I can use different proxies for each request?
Thanks!
public class Program
{
private static HttpClient Client = new HttpClient();
public static void Main(string[] args)
{
Console.WriteLine("Starting connections");
for(int i = 0; i<10; i++)
{
var result = Client.GetAsync("http://aspnetmonsters.com").Result;
Console.WriteLine(result.StatusCode);
}
Console.WriteLine("Connections done");
Console.ReadLine();
}
}
Ah, I misread the question.
It's not about how to use a random IWebProxy with HttpClientHandler, but how to work around being unable to reset the proxy property of the same HttpClientHandler once the first request has started.
The problem is that you can't reset the proxy of the HttpClientHandler...
System.InvalidOperationException: 'This instance has already started
one or more requests.
Properties can only be modified before sending the first request.'
But it's still rather easy.
Proxy property of HttpClientHandler takes an object that implements IWebProxy.
IWebProxy interface has a method GetProxy that return the Uri of the proxy.
So you can make your own class that implements this interface and control how it returns the Uri of the proxy with GetProxy.
You can make it wrap another IWebProxy, and in GetProxy it would return GetProxy of the inner IWebProxy.
This way, you won't have to change the Proxy property of the HttpClientHandler, you can just change the inner IWebProxy.
Implementation:
public class WebProxyService
: System.Net.IWebProxy
{
protected System.Net.IWebProxy m_proxy;
public System.Net.IWebProxy Proxy
{
get { return this.m_proxy ??= System.Net.WebRequest.DefaultWebProxy; }
set { this.m_proxy = value; }
}
System.Net.ICredentials System.Net.IWebProxy.Credentials
{
get { return this.Proxy.Credentials; }
set { this.Proxy.Credentials = value; }
}
public WebProxyService()
{ } // Constructor
public WebProxyService(System.Net.IWebProxy proxy)
{
this.Proxy = proxy;
} // Constructor
System.Uri System.Net.IWebProxy.GetProxy(System.Uri destination)
{
return this.Proxy.GetProxy(destination);
}
bool System.Net.IWebProxy.IsBypassed(System.Uri host)
{
return this.Proxy.IsBypassed(host);
}
}
And then usage goes like this:
public class AlternatingProxy
{
public static async System.Threading.Tasks.Task Test()
{
string url = "http://aspnetmonsters.com";
System.Net.WebProxy[] proxies = new[] {
null,
new System.Net.WebProxy("104.238.172.20", 8080),
new System.Net.WebProxy("104.238.167.193", 8080),
new System.Net.WebProxy("136.244.102.38", 8080),
new System.Net.WebProxy("95.179.202.40", 8080)
};
System.Random rnd = new System.Random();
WebProxyService proxyService = new WebProxyService();
using (System.Net.Http.HttpClient hc = new System.Net.Http.HttpClient(
new System.Net.Http.HttpClientHandler { UseProxy = true, Proxy = proxyService }
))
{
// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent
hc.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148");
hc.DefaultRequestHeaders.Add("Accept-Language", "fr-FR, fr;q=0.9, en;q=0.8, it;q=0.7, *;q=0.5");
hc.DefaultRequestHeaders.Add("Referer", "https://www.baidu.com");
hc.DefaultRequestHeaders.ConnectionClose = true;
for (int i = 0; i < 10; ++i)
{
proxyService.Proxy = proxies[rnd.Next(proxies.Length)];
string response = await hc.GetStringAsync(url);
}
}
} // End Task Test
} // End Class TestMe
Edit:
If you want to use a singleton, maybe this is an idea:
public class WebProxyService
: System.Net.IWebProxy
{
protected System.Net.IWebProxy m_proxy;
public System.Net.IWebProxy Proxy
{
get { return this.m_proxy ??= System.Net.WebRequest.DefaultWebProxy; }
set { this.m_proxy = value; }
}
System.Net.ICredentials System.Net.IWebProxy.Credentials
{
get { return this.Proxy.Credentials; }
set { this.Proxy.Credentials = value; }
}
public WebProxyService()
{ } // Constructor
public WebProxyService(System.Net.IWebProxy proxy)
{
this.Proxy = proxy;
} // Constructor
protected System.Func<System.Net.WebProxy>[] proxies = new System.Func<System.Net.WebProxy>[] {
delegate(){ return new System.Net.WebProxy("104.238.172.20", 8080); },
delegate (){ return new System.Net.WebProxy("104.238.167.193", 8080);},
delegate(){ return new System.Net.WebProxy("136.244.102.38", 8080);},
delegate(){ return new System.Net.WebProxy("95.179.202.40", 8080);}
};
System.Uri System.Net.IWebProxy.GetProxy(System.Uri destination)
{
return proxies[RandomGen2.Next(proxies.Length)]().GetProxy(destination);
}
bool System.Net.IWebProxy.IsBypassed(System.Uri host)
{
return this.Proxy.IsBypassed(host);
}
private static class RandomGen2
{
private static System.Random _global = new System.Random();
[System.ThreadStatic]
private static System.Random _local;
public static int Next(int maxValue)
{
System.Random inst = _local;
if (inst == null)
{
int seed;
lock (_global) seed = _global.Next();
_local = inst = new System.Random(seed);
}
return inst.Next(maxValue);
}
}
} // End Class WebProxyService
Edit 2:
It's still not thread-safe if you change the proxy.
Therefore, using a fixed immutable proxy-list and blocking the set-property.
This way, it should be thread-safe.
public class WebProxyService
: System.Net.IWebProxy
{
protected System.Net.IWebProxy[] m_proxyList;
public System.Net.IWebProxy Proxy
{
get
{
// https://devblogs.microsoft.com/pfxteam/getting-random-numbers-in-a-thread-safe-way/
if (this.m_proxyList != null)
return this.m_proxyList[ThreadSafeRandom.Next(this.m_proxyList.Length)];
return System.Net.WebRequest.DefaultWebProxy;
}
set
{
throw new System.InvalidOperationException("It is not thread-safe to change the proxy-list.");
}
}
System.Net.ICredentials System.Net.IWebProxy.Credentials
{
get { return this.Proxy.Credentials; }
set { this.Proxy.Credentials = value; }
}
public WebProxyService()
{
} // Constructor
public WebProxyService(System.Net.IWebProxy[] proxyList)
{
this.m_proxyList = proxyList;
} // Constructor
System.Uri System.Net.IWebProxy.GetProxy(System.Uri destination)
{
return this.Proxy.GetProxy(destination);
}
bool System.Net.IWebProxy.IsBypassed(System.Uri host)
{
return this.Proxy.IsBypassed(host);
}
} // End Class WebProxyService
Old answer:
----
Using a proxy with HttpClient in ASP.NET-Core is actually quite simple.
All you need to do is set the handler in the HttpClient-constructor.
Then set the proxy-property of the handler for each request.
Like this:
public class Program
{
public static async System.Threading.Tasks.Task Main(string[] args)
{
string url = "http://aspnetmonsters.com";
System.Net.WebProxy[] proxies = new[] {
null,
new System.Net.WebProxy("104.238.172.20", 8080),
new System.Net.WebProxy("104.238.167.193", 8080),
new System.Net.WebProxy("136.244.102.38", 8080),
new System.Net.WebProxy("95.179.202.40", 8080)
};
System.Random rnd = new System.Random();
using (System.Net.Http.HttpClientHandler handler = new System.Net.Http.HttpClientHandler()
{
Proxy = new System.Net.WebProxy("http://127.0.0.1:8888"),
UseProxy = true,
})
{
using (System.Net.Http.HttpClient hc = new System.Net.Http.HttpClient(handler))
{
System.Console.WriteLine("Starting connections");
for (int i = 0; i < 10; i++)
{
handler.Proxy = proxies[rnd.Next(proxies.Length)];
await hc.GetAsync(url);
// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent
hc.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148");
hc.DefaultRequestHeaders.Add("Accept-Language", "fr-FR, fr;q=0.9, en;q=0.8, it;q=0.7, *;q=0.5");
hc.DefaultRequestHeaders.Add("Referer", "https://www.baidu.com");
using (System.Net.Http.HttpResponseMessage response = await hc.GetAsync(url))
{
// using (var fs = new System.IO.MemoryStream())
// { await response.Content.CopyToAsync(fs); }
byte[] ba = await response.Content.ReadAsByteArrayAsync();
} // End Using response
} // Next i
System.Console.WriteLine("Ending connections");
} // End Using hc
} // End Using handler
System.Console.WriteLine("--- Press any key to continue --- ");
System.Console.ReadKey();
} // End Task Main
} // End Class Program
You will need to implement IWebProxy.
here is a very row sample.
First implement IWebProxy
public class MyProxy : IWebProxy {
public MyProxy() { credentials = new NetworkCredential( user, password ); }
private NetworkCredential credentials;
public ICredentials Credentials
{
get = > credentials;
set = > throw new NotImplementedException();
}
private Uri proxyUri;
public Uri GetProxy( Uri destination )
{
return proxyUri; // your proxy Uri
}
public bool IsBypassed( Uri host )
{
return false;
}
private const string user = "yourusername";
private const string password = "password";}
Then provide it to handler in HttpClient
public class MyHttpClient {
internal static HttpResult httpMethod( ... )
{
var _client = client();
try
{
var message = new HttpRequestMessage( method, url );
message.Content = new StringContent( content, Encoding.UTF8, "application/json" );
var result = _client.SendAsync( message ).Result;// handle result
}
catch( Exception e ){}
}
private static HttpClient client()
{
var httpClientHandler = new HttpClientHandler() { Proxy = new MyProxy() };
var httpClient = new MyClient( new Uri( "baseurl" ), httpClientHandler );
return httpClient;
So basically to be able to change proxies you will need a reference on the HttpClientHandler.
A simple example can be found here: C# use proxy with HttpClient request
and another one here: Simple C# .NET 4.5 HTTPClient Request Using Basic Auth and Proxy
I would suggest to keep the HttpClientHandler on a private field and use the reference to change the proxy everytime you need it.
Keep in mind though that if you need to use different proxies simultaneously you would need to have multiple instances of the HttpClientHandler class.
If you need me to make a sample code for that. Ping me.
Thank you.
Related
I am using mock library in my .Net unit test and getting an error
cannot be accessed with an instance reference instead use type name.
I am getting this error at following line in my test method where it is calling cq.Instance. I am new to mock library. Could somebody let me know how do I call the static method?
attributeValue.Setup(cq => cq.Instance().CallQueryAsync(request, 1)).Returns(attrValue);
Actual method to be tested
public static async Task<AttributeValueList> GetAttributeSecDateValueList(int attrId)
{
try
{
var request = AttributeValue.ResolveRequest(attrId);
var response = await AsyncProxy<AttributeValue>.Instance().CallQueryAsync(request, (int)AttributeValue.OperationType.GetSecDateValues);
var coll = new AttributeValueList();
coll.AddRange(response);
return coll;
}
catch (Exception e)
{
throw e;
}
}
Proxy class
public class AsyncProxy<RT> : IDisposable
where RT : class, new()
{
readonly WebServiceProxy<RT> _wsProxy;
private AsyncProxy(WebServiceProxy<RT> webServiceProxy)
{
_wsProxy = webServiceProxy;
}
public static async Task<IEnumerable<RT>> Load(object parameters)
{
return await Instance().CallQueryAsync(parameters);
}
public static AsyncProxy<RT> Instance()
{
return new AsyncProxy<RT>(WebServiceProxy<RT>.Instance());
}
/// <summary>
/// Return result set of Poco as smartCollection
/// </summary>
public async Task<SmartCollection<RT>> CallQueryAsync(object request, int? uniqueIdentifier = null, bool isLongRunning = false, [CallerMemberName]string memberName = "")
{
//#if DEBUG
// var stopwatch = new Stopwatch();
// stopwatch.Start();
//#endif
try
{
// We want to get rid of the proxy as soon as we are done with it
using (_wsProxy)
{
var awaited = await _wsProxy.CallQueryAsync(request, uniqueIdentifier, isLongRunning);
if (awaited == null)
return null;
var observableCollection = new SmartCollection<RT>();
foreach (var item in awaited)
observableCollection.Add(item as RT);
return observableCollection;
}
}
finally
{
Dispose();
//#if DEBUG
// stopwatch.Stop();
// Debug.WriteLine(null);
// Debug.WriteLine($"****>>>>> AsyncProxy {memberName} took {stopwatch.ElapsedMilliseconds} ms <<<<<<<<****");
//#endif
}
}
}
test method
[TestMethod]
public void test()
{
Task<SmartCollection<AttributeValue>> attrValue = null;
var request = new AttributeValue();
var attributeValue = new Mock<AsyncProxy<AttributeValue>>();
attributeValue.Setup(cq => cq.Instance().CallQueryAsync(request, 1)).Returns(attrValue);
}
Im using selenium chrome driver combined with cefsharp, the page im working on sends some kind of ping request(fetch/xhr) everytime an element is clicked the problem is that those request aren't initiated by selenium chrome driver so there is no wayy for me to track them and wait for the finish. Is there any way for selenium or js to track those requests?
With Selenium 4 you actually can intercept requests
Call Enable to start requests tracking
Call WaitForRequestResponse to wait for the specific request
If you can same requests you can clean already catch requests with Clear
public class WebDriverNetworkController
{
private const string RequestWillBeSent = "requestWillBeSent";
private const string ResponseReceived = "responseReceived";
private readonly IDriverProvider _driverProvider;
private readonly ILogger _log;
private readonly ConcurrentDictionary<DevToolsSession, List<HttpRequestModel>> _sentRequests = new();
public WebDriverNetworkController(IDriverProvider driverProvider, ILogger log)
{
_driverProvider = driverProvider;
_log = log;
}
private void HandleDevToolsEvent(object sender, DevToolsEventReceivedEventArgs e)
{
var session = sender as DevToolsSession;
switch (e.EventName)
{
case RequestWillBeSent:
{
OnRequestWillBeSent(session, e.EventData);
break;
}
case ResponseReceived:
{
OnResponseReceived(session, e.EventData);
break;
}
}
}
public void Enable()
{
_log.Information("Enabling network tracing");
var session = GetDevToolsSession();
session.DevToolsEventReceived += HandleDevToolsEvent;
var enableTask = session.Domains.Network.EnableNetwork();
enableTask.Wait();
}
public void Disable()
{
_log.Information("Disabling network tracing");
var session = GetDevToolsSession();
session.DevToolsEventReceived -= HandleDevToolsEvent;
var disableTask = session.Domains.Network.DisableNetwork();
disableTask.Wait();
}
public void Clear()
{
_log.Information("Clearing captured network tracing requests");
var sentRequests = GetRequests();
sentRequests.Clear();
}
public HttpRequestModel[] GetAllRequests() => GetRequests().ToArray();
public HttpRequestModel WaitForRequestResponse(HttpMethod method, string uri)
{
_log.Debug($"Waiting for {method.Method} request to {uri}");
var session = GetDevToolsSession();
HttpRequestModel request = null;
var waiter = new ConditionWaiter
{
Message = $"{method.Method} request to {uri}"
};
waiter.Until(() =>
{
request = GetLastSentRequest(method, uri, session);
return request?.Response != null;
});
return request;
}
private HttpRequestModel GetLastSentRequest(HttpMethod method, string uri, DevToolsSession session) =>
GetRequests(session).LastOrDefault(request => request.Method == method && request.Uri.EndsWith(uri));
private DevToolsSession GetDevToolsSession()
{
var driver = _driverProvider.GetDriver();
if (driver is IDevTools devTools)
{
return devTools.GetDevToolsSession();
}
throw new WebDriverException($"Could not cast {driver.GetType().Name} to {nameof(IDevTools)}");
}
private List<HttpRequestModel> GetRequests(DevToolsSession session = null) =>
_sentRequests.GetOrAdd(session ?? GetDevToolsSession(), _ => new());
private void OnRequestWillBeSent(DevToolsSession session, JToken eventData)
{
var requestModel = new HttpRequestModel
{
Id = eventData.SelectToken("requestId", true)!.Value<string>(),
Method = new(eventData.SelectToken("request.method", true)!.Value<string>()!),
Uri = eventData.SelectToken("request.url", true)!.Value<string>(),
Body = eventData.SelectToken("request")!.Value<string>("postData")
};
var sentRequests = _sentRequests.GetOrAdd(session, _ => new());
sentRequests.Add(requestModel);
}
private void OnResponseReceived(DevToolsSession session, JToken eventData)
{
var requestId = eventData.SelectToken("requestId", true)!.Value<string>();
var responseModel = new HttpResponseModel
{
Uri = eventData.SelectToken("response.url", true)!.Value<string>(),
StatusCode = (HttpStatusCode)eventData.SelectToken("response.status", true)!.Value<long>()
};
var sentRequests = _sentRequests[session];
var request = sentRequests.Last(request => request.Id == requestId);
request.Response = responseModel;
}
}
The code above is not ready to be used right away but should give a general understanding of the approach.
Task is to call a third party service/wrapper multiple times and get the response. Below is the sample code and I am trying to figured out how to create singleton instance for the service.
class Program
{
static void Main(string[] args)
{
List<string> trips = new List<string>();
trips.Add("ABC");
trips.Add("XYZ");
foreach (string s in trips) {
Test.TestMethod(s);
}
}
}
public static class Test
{
public static bool TestMethod(string trip)
{
BridgeApiClient bridgeApiClient = new BridgeApiClient("http://localhost/Service.svc", "username", "password");
TripRequest tr = new TripRequest();
tr.TripNumber = trip;
var response = bridgeApiClient.GetTrip(tr);
return true;
}
}
You can just declare a static member variable and use that:
public static class Test
{
private static readonly BridgeApiClient bridgeApiClient = new BridgeApiClient("http://localhost/Service.svc", "username", "password");
public static bool TestMethod(string trip)
{
TripRequest tr = new TripRequest();
tr.TripNumber = trip;
var response = bridgeApiClient.GetTrip(tr);
return true;
}
}
EDIT: For each request, a new instance of controller is created. However, this is not true with Attribute classes. Once they are created, it is used for multiple requests. I hope it helps.
I wrote my own WebAPI (using latest version of WebAPI and .net framework) caching action filter. I am aware about CacheCow & this. However, i wanted mine anyways.
However, there is some issue with my code because i don't get exepected output when i use it in my project on live server. On local machine everything works fine.
I used below code in my blog RSS generator and i cache the data for each category. There are around 5 categories (food, tech, personal etc).
Issue: When i navigate to say api/GetTech it returns me the rss feed items from personal blog category. When i navigate to say api/GetPersonal , it returns me api/Food
I am not able to find the root cause but I think this is due to use of static method/variable. I have double checked that my _cachekey has unique value for each category of my blog.
Can someone point out any issues with this code esp when we have say 300 requests per minute ?
public class WebApiOutputCacheAttribute : ActionFilterAttribute
{
// Cache timespan
private readonly int _timespan;
// cache key
private string _cachekey;
// cache repository
private static readonly MemoryCache _webApiCache = MemoryCache.Default;
/// <summary>
/// Initializes a new instance of the <see cref="WebApiOutputCacheAttribute"/> class.
/// </summary>
/// <param name="timespan">The timespan in seconds.</param>
public WebApiOutputCacheAttribute(int timespan)
{
_timespan = timespan;
}
public override void OnActionExecuting(HttpActionContext ac)
{
if (ac != null)
{
_cachekey = ac.Request.RequestUri.PathAndQuery.ToUpperInvariant();
if (!_webApiCache.Contains(_cachekey)) return;
var val = (string)_webApiCache.Get(_cachekey);
if (val == null) return;
ac.Response = ac.Request.CreateResponse();
ac.Response.Content = new StringContent(val);
var contenttype = (MediaTypeHeaderValue)_webApiCache.Get("response-ct") ?? new MediaTypeHeaderValue("application/rss+xml");
ac.Response.Content.Headers.ContentType = contenttype;
}
else
{
throw new ArgumentNullException("ac");
}
}
public override void OnActionExecuted(HttpActionExecutedContext actionExecutedContext)
{
if (_webApiCache.Contains(_cachekey)) return;
var body = actionExecutedContext.Response.Content.ReadAsStringAsync().Result;
if (actionExecutedContext.Response.StatusCode == HttpStatusCode.OK)
{
lock (WebApiCache)
{
_wbApiCache.Add(_cachekey, body, DateTime.Now.AddSeconds(_timespan));
_webApiCache.Add("response-ct", actionExecutedContext.Response.Content.Headers.ContentType, DateTimeOffset.UtcNow.AddSeconds(_timespan));
}
}
}
}
The same WebApiOutputCacheAttribute instance can be used to cache multiple simultaneous requests, so you should not store cache keys on the instance of the attribute. Instead, regenerate the cache key during each request / method override. The following attribute works to cache HTTP GET requests.
using System;
using System.Linq;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Web.Http.Controllers;
using System.Web.Http.Filters;
using Newtonsoft.Json;
// based on strathweb implementation
// http://www.strathweb.com/2012/05/output-caching-in-asp-net-web-api/
public class CacheHttpGetAttribute : ActionFilterAttribute
{
public int Duration { get; set; }
public ILogExceptions ExceptionLogger { get; set; }
public IProvideCache CacheProvider { get; set; }
private bool IsCacheable(HttpRequestMessage request)
{
if (Duration < 1)
throw new InvalidOperationException("Duration must be greater than zero.");
// only cache for GET requests
return request.Method == HttpMethod.Get;
}
private CacheControlHeaderValue SetClientCache()
{
var cachecontrol = new CacheControlHeaderValue
{
MaxAge = TimeSpan.FromSeconds(Duration),
MustRevalidate = true,
};
return cachecontrol;
}
private static string GetServerCacheKey(HttpRequestMessage request)
{
var acceptHeaders = request.Headers.Accept;
var acceptHeader = acceptHeaders.Any() ? acceptHeaders.First().ToString() : "*/*";
return string.Join(":", new[]
{
request.RequestUri.AbsoluteUri,
acceptHeader,
});
}
private static string GetClientCacheKey(string serverCacheKey)
{
return string.Join(":", new[]
{
serverCacheKey,
"response-content-type",
});
}
public override void OnActionExecuting(HttpActionContext actionContext)
{
if (actionContext == null) throw new ArgumentNullException("actionContext");
var request = actionContext.Request;
if (!IsCacheable(request)) return;
try
{
// do NOT store cache keys on this attribute because the same instance
// can be reused for multiple requests
var serverCacheKey = GetServerCacheKey(request);
var clientCacheKey = GetClientCacheKey(serverCacheKey);
if (CacheProvider.Contains(serverCacheKey))
{
var serverValue = CacheProvider.Get(serverCacheKey);
var clientValue = CacheProvider.Get(clientCacheKey);
if (serverValue == null) return;
var contentType = clientValue != null
? JsonConvert.DeserializeObject<MediaTypeHeaderValue>(clientValue.ToString())
: new MediaTypeHeaderValue(serverCacheKey.Substring(serverCacheKey.LastIndexOf(':') + 1));
actionContext.Response = actionContext.Request.CreateResponse();
// do not try to create a string content if the value is binary
actionContext.Response.Content = serverValue is byte[]
? new ByteArrayContent((byte[])serverValue)
: new StringContent(serverValue.ToString());
actionContext.Response.Content.Headers.ContentType = contentType;
actionContext.Response.Headers.CacheControl = SetClientCache();
}
}
catch (Exception ex)
{
ExceptionLogger.Log(ex);
}
}
public override void OnActionExecuted(HttpActionExecutedContext actionExecutedContext)
{
try
{
var request = actionExecutedContext.Request;
// do NOT store cache keys on this attribute because the same instance
// can be reused for multiple requests
var serverCacheKey = GetServerCacheKey(request);
var clientCacheKey = GetClientCacheKey(serverCacheKey);
if (!CacheProvider.Contains(serverCacheKey))
{
var contentType = actionExecutedContext.Response.Content.Headers.ContentType;
object serverValue;
if (contentType.MediaType.StartsWith("image/"))
serverValue = actionExecutedContext.Response.Content.ReadAsByteArrayAsync().Result;
else
serverValue = actionExecutedContext.Response.Content.ReadAsStringAsync().Result;
var clientValue = JsonConvert.SerializeObject(
new
{
contentType.MediaType,
contentType.CharSet,
});
CacheProvider.Add(serverCacheKey, serverValue, new TimeSpan(0, 0, Duration));
CacheProvider.Add(clientCacheKey, clientValue, new TimeSpan(0, 0, Duration));
}
if (IsCacheable(actionExecutedContext.Request))
actionExecutedContext.ActionContext.Response.Headers.CacheControl = SetClientCache();
}
catch (Exception ex)
{
ExceptionLogger.Log(ex);
}
}
}
Just replace the CacheProvider with your MemoryCache.Default. In fact, the code above uses the same by default during development, and uses azure cache when deployed to a live server.
Even though your code resets the _cachekey instance field during each request, these attributes are not like controllers where a new one is created for each request. Instead, the attribute instance can be repurposed to service multiple simultaneous requests. So don't use an instance field to store it, regenerate it based on the request each and every time you need it.
Is there any way to use a socks proxy with WebClient? Specifically with the DownloadString method that it provides?
I don't want to use any third party stuff like privoxy, freecap whatever and I can't use commercial libraries like those from Chilkat. I tried using stuff from http://www.mentalis.org/ in fact I used their WebRequest implementation but they don't seem to have something similar for WebClient.
SOCKS is not supported directly by the WebRequest/WebResponse classes and by extension, the WebClient class (it relies on WebRequest to do its work).
It really can't, as it works on the transport layer (TCP/IP) and not through a simple redirect to a server that forwards HTTP requests (which is the level that the WebRequest/WebResponse classes work on).
You can create a specialized derivation of WebRequest/WebResponse (that uses ProxySocket to handle the low-level handshaking and then) and then create a specialized WebClient class which overrides the GetWebRequest and GetWebResponse methods.
Once you have that class substituted for your WebClient instances, it should work as normal (you might have to set up the proxy in each case where you use it though).
Here is how I ended up doing it, thank you casperOne for the answer
public class SocksWebClient : WebClient
{
public IProxyDetails ProxyDetails { get; set; }
public string UserAgent { get; set; }
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest result = null;
if (ProxyDetails != null)
{
if (ProxyDetails.ProxyType == ProxyType.Proxy)
{
result = (HttpWebRequest)WebRequest.Create(address);
result.Proxy = new WebProxy(ProxyDetails.FullProxyAddress);
if (!string.IsNullOrEmpty(UserAgent))
((HttpWebRequest)result).UserAgent = UserAgent;
}
else if (ProxyDetails.ProxyType == ProxyType.Socks)
{
result = SocksHttpWebRequest.Create(address);
result.Proxy = new WebProxy(ProxyDetails.FullProxyAddress);
//TODO: implement user and password
}
else if (ProxyDetails.ProxyType == ProxyType.None)
{
result = (HttpWebRequest)WebRequest.Create(address);
if (!string.IsNullOrEmpty(UserAgent))
((HttpWebRequest)result).UserAgent = UserAgent;
}
}
else
{
result = (HttpWebRequest)WebRequest.Create(address);
if (!string.IsNullOrEmpty(UserAgent))
((HttpWebRequest)result).UserAgent = UserAgent;
}
return result;
}
}
The SocksHttpWebRequest class is taken from the blog linked to by #casperOne, the code for which is as follows:
using System;
using System.Collections.Specialized;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Text;
using Org.Mentalis.Network.ProxySocket;
namespace Ditrans
{
public class SocksHttpWebRequest : WebRequest
{
#region Member Variables
private readonly Uri _requestUri;
private WebHeaderCollection _requestHeaders;
private string _method;
private SocksHttpWebResponse _response;
private string _requestMessage;
private byte[] _requestContentBuffer;
// darn MS for making everything internal (yeah, I'm talking about you, System.net.KnownHttpVerb)
static readonly StringCollection validHttpVerbs =
new StringCollection { "GET", "HEAD", "POST", "PUT", "DELETE", "TRACE", "OPTIONS" };
#endregion
#region Constructor
private SocksHttpWebRequest(Uri requestUri)
{
_requestUri = requestUri;
}
#endregion
#region WebRequest Members
public override WebResponse GetResponse()
{
if (Proxy == null)
{
throw new InvalidOperationException("Proxy property cannot be null.");
}
if (String.IsNullOrEmpty(Method))
{
throw new InvalidOperationException("Method has not been set.");
}
if (RequestSubmitted)
{
return _response;
}
_response = InternalGetResponse();
RequestSubmitted = true;
return _response;
}
public override Uri RequestUri
{
get { return _requestUri; }
}
public override IWebProxy Proxy { get; set; }
public override WebHeaderCollection Headers
{
get
{
if (_requestHeaders == null)
{
_requestHeaders = new WebHeaderCollection();
}
return _requestHeaders;
}
set
{
if (RequestSubmitted)
{
throw new InvalidOperationException("This operation cannot be performed after the request has been submitted.");
}
_requestHeaders = value;
}
}
public bool RequestSubmitted { get; private set; }
public override string Method
{
get
{
return _method ?? "GET";
}
set
{
if (validHttpVerbs.Contains(value))
{
_method = value;
}
else
{
throw new ArgumentOutOfRangeException("value", string.Format("'{0}' is not a known HTTP verb.", value));
}
}
}
public override long ContentLength { get; set; }
public override string ContentType { get; set; }
public override Stream GetRequestStream()
{
if (RequestSubmitted)
{
throw new InvalidOperationException("This operation cannot be performed after the request has been submitted.");
}
if (_requestContentBuffer == null)
{
_requestContentBuffer = new byte[ContentLength];
}
else if (ContentLength == default(long))
{
_requestContentBuffer = new byte[int.MaxValue];
}
else if (_requestContentBuffer.Length != ContentLength)
{
Array.Resize(ref _requestContentBuffer, (int) ContentLength);
}
return new MemoryStream(_requestContentBuffer);
}
#endregion
#region Methods
public static new WebRequest Create(string requestUri)
{
return new SocksHttpWebRequest(new Uri(requestUri));
}
public static new WebRequest Create(Uri requestUri)
{
return new SocksHttpWebRequest(requestUri);
}
private string BuildHttpRequestMessage()
{
if (RequestSubmitted)
{
throw new InvalidOperationException("This operation cannot be performed after the request has been submitted.");
}
var message = new StringBuilder();
message.AppendFormat("{0} {1} HTTP/1.0\r\nHost: {2}\r\n", Method, RequestUri.PathAndQuery, RequestUri.Host);
// add the headers
foreach (var key in Headers.Keys)
{
message.AppendFormat("{0}: {1}\r\n", key, Headers[key.ToString()]);
}
if (!string.IsNullOrEmpty(ContentType))
{
message.AppendFormat("Content-Type: {0}\r\n", ContentType);
}
if (ContentLength > 0)
{
message.AppendFormat("Content-Length: {0}\r\n", ContentLength);
}
// add a blank line to indicate the end of the headers
message.Append("\r\n");
// add content
if(_requestContentBuffer != null && _requestContentBuffer.Length > 0)
{
using (var stream = new MemoryStream(_requestContentBuffer, false))
{
using (var reader = new StreamReader(stream))
{
message.Append(reader.ReadToEnd());
}
}
}
return message.ToString();
}
private SocksHttpWebResponse InternalGetResponse()
{
var response = new StringBuilder();
using (var _socksConnection =
new ProxySocket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
{
var proxyUri = Proxy.GetProxy(RequestUri);
var ipAddress = GetProxyIpAddress(proxyUri);
_socksConnection.ProxyEndPoint = new IPEndPoint(ipAddress, proxyUri.Port);
_socksConnection.ProxyType = ProxyTypes.Socks5;
// open connection
_socksConnection.Connect(RequestUri.Host, 80);
// send an HTTP request
_socksConnection.Send(Encoding.ASCII.GetBytes(RequestMessage));
// read the HTTP reply
var buffer = new byte[1024];
var bytesReceived = _socksConnection.Receive(buffer);
while (bytesReceived > 0)
{
response.Append(Encoding.ASCII.GetString(buffer, 0, bytesReceived));
bytesReceived = _socksConnection.Receive(buffer);
}
}
return new SocksHttpWebResponse(response.ToString());
}
private static IPAddress GetProxyIpAddress(Uri proxyUri)
{
IPAddress ipAddress;
if (!IPAddress.TryParse(proxyUri.Host, out ipAddress))
{
try
{
return Dns.GetHostEntry(proxyUri.Host).AddressList[0];
}
catch (Exception e)
{
throw new InvalidOperationException(
string.Format("Unable to resolve proxy hostname '{0}' to a valid IP address.", proxyUri.Host), e);
}
}
return ipAddress;
}
#endregion
#region Properties
public string RequestMessage
{
get
{
if (string.IsNullOrEmpty(_requestMessage))
{
_requestMessage = BuildHttpRequestMessage();
}
return _requestMessage;
}
}
#endregion
}
}
Note that, as #casperOne pointed out, this makes use of a (free) third party library called ProxySocket.
I came across this aswell and found the nice BetterHttpClient
It derives from WebClient and allows you to specify a socks proxy:
BetterHttpClient.HttpClient client = new BetterHttpClient.HttpClient(new BetterHttpClient.Proxy("IP address", port, BetterHttpClient.ProxyTypeEnum.Socks));
I was looking for library in order to do this. finally i found the MihaZupan/HttpToSocks5Proxy it's a real lifesaver. just like this you can use it:
using MihaZupan;
var proxy = new HttpToSocks5Proxy("127.0.0.1", 1080,
"username", "password" // optional
);
var handler = new HttpClientHandler { Proxy = proxy };
HttpClient httpClient = new HttpClient(handler, true);
var result = await httpClient.SendAsync(
new HttpRequestMessage(HttpMethod.Get, "https://httpbin.org/ip"));
Console.WriteLine("HTTPS GET: " + await result.Content.ReadAsStringAsync());
If you want a more convenient, WebClient-esque API for reading the content, you can wrap the response-read calls into an extension method:
public static class ExtensionMethods
{
public static async Task<string> DownloadStringAsync(this HttpClient client, string url)
{
var response = await client.SendAsync(new HttpRequestMessage(
HttpMethod.Get, url));
return await response.Content.ReadAsStringAsync();
}
}
For future readers:
Since .NET 6, socks proxies are supported natively with HttpClient.
var handler = new HttpClientHandler
{
Proxy = new WebProxy("socks5://127.0.0.1", 9050)
};
var httpClient = new HttpClient(handler);
Note:
In time of writing this answer, WebClient is considered as obsolete. HttpClient should be used instead.
Try Yove.Proxy. Example:
using (var w = new WebClient())
{
using (var proxy = new ProxyClient("67.201.33.10", 25283, ProxyType.Socks5))
{
w.Proxy = proxy;
Console.WriteLine(w.DownloadString("https://api.ipify.org"));
}
}