I have written a little tool in C# (WinForm) using Visual Studio 2012.
Its purpose is to let users check a given City and ZIP code against a database to validate if the city/zip combination is legit.
Everything works well so far (see the simplyfied code below). Since it is possible that users carry out more than one validation simultaneously, I've used a queue pattern (to make sure, I am sending the multiple requests in the right order, as they have been submitted).
Now, I need to modify my code to run 3 web requests to achive the same result.
Instead of just querying a webservice directly with city and zip code, I now need to make 3 requests in consecutive order:
send a code to a webserver X to request a key
send the obtained key to webserver X to validate city and zip code
send the validated result to webserver Y to save the city and zip code.
Obviously, I have to wait for the 1st request to be answered before I can proceed to carry out the 2nd request, and so on... And I need to check if the 2nd request returned true/valid otherwise I do not need to save/carry out the third request.
I need help on how to include these two more web requests in my code without breaking my queue mechanism. One item in my queue should now contain the 3 requests whilst making sure I am not requesting step 2 or 3 prior to step 1 or step 2 having delivered a (valid) result.
Here is what I've got so far:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Mail;
using System.Text;
using System.Threading;
using System.Web;
using System.Xml;
using System.Globalization;
using System.Security.Cryptography;
using System.Windows.Forms;
using CheckAddress.Util;
namespace CheckAddress.Api
{
public class ApiHelper
{
#region singleton
private static ApiHelper _instance;
private static Object _lock = new object();
public static ApiHelper Instance
{
get
{
lock (_lock)
{
if (_instance == null)
{
_instance = new ApiHelper();
}
}
return _instance;
}
}
#endregion
private Thread _queueThread;
private bool _running;
private Queue<Address> _queue;
private ApiHelper()
{
_running = false;
_queue = new Queue<Address>();
MaxThreads = SettingsFile.Instance.MaxThreads;
MaxAttempts = SettingsFile.Instance.MaxAttempts;
ThreadPool.SetMaxThreads(MaxThreads, MaxThreads);
}
public int MaxThreads { get; set; }
public int MaxAttempts { get; set; }
public void Add(Address a)
{
lock (_queue)
{
_queue.Enqueue(a);
}
}
private void AddressSendProc(object state)
{
bool isValid = false;
foreach (Address.CityZIP theCityZIP in a.listOfAddresses)
{
while (!success && tries < MaxAttempts)
{
try
{
var req = (HttpWebRequest)WebRequest.Create("http://domain.com/geo.php?city=auburn&zip=95602");
req.Method = "GET";
req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0";
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
req.ContentType = "application/x-www-form-urlencoded";
req.Timeout = 3000;
var resp = (HttpWebResponse)req.GetResponse();
var ms = new MemoryStream();
var rs = resp.GetResponseStream();
if (rs != null)
{
var buf = new byte[4096];
int len = 0;
while ((len = rs.Read(buf, 0, buf.Length)) > 0)
{
ms.Write(buf, 0, len);
}
rs.Close();
}
var xml = new XmlDocument();
xml.LoadXml(Encoding.UTF8.GetString(ms.ToArray()));
var nodes = xml.SelectNodes("/response/valid");
if (nodes.Count == 1)
{
isValid = true;
}
int code = 0;
Int32.TryParse(nodes[0].InnerText, out code);
if (code == 200)
{
success = true;
}
}
catch (Exception ex)
{
/* */
}
tries++;
}
}
}
private void ApiThreadProc()
{
while (_running)
{
try
{
lock (_queue)
{
while (_queue.Count > 0)
{
ThreadPool.QueueUserWorkItem(AddressSendProc, _queue.Dequeue());
}
}
Thread.Sleep(1000);
}
catch (Exception) {/*ignore*/}
}
}
public void Start()
{
_running = true;
_queueThread = new Thread(ApiThreadProc);
_queueThread.Start();
}
public void Stop()
{
_running = false;
if (_queueThread != null)
{
_queueThread.Interrupt();
}
}
}
}
Related
I have a code snippet which is acting like a Grpc Client in Unity. The code style is designed for a console application, which can be called in Main method, blocking it and receiving data all the time. Now, I want to use it in Unity and obviously I want my app to run in Unity at the same time. Also, my end goal is to have something that works like a Udp Client. You call it one time, and all the time will receive data for you, without blocking any part of the host application.
The most important part of this design is that, if there is any event, I will get update, if there is no new event, accordingly, I am not receiving any data. And it happens all in ObserveEvents(channel).Wait();. The problem is Wait();. Which is all the time, keep the main thread, working thread, listening to updates. In Play mode Unity does not respond anymore!
I can bypass it and say, I don't need such a design, I can receive events every other second or every some frames. By doing that, I have my Unity application, but I am losing so many frame rates, regardless of the fact that my data are NOT flowing smoothly to my host application in Unity.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Grpc.Core;
using UnityEngine;
namespace Scripts
{
public class GrpcChannel
{
public void GrpcServer(string ip, int port)
{
var channel = new Channel(ip, port, ChannelCredentials.Insecure);
ObserveEvents(channel).Wait();
channel.ShutdownAsync().Wait();
}
private async Task ObserveEvents(Channel channel)
{
Debug.Log("Observing events...");
var eventService = new EventService.EventServiceClient(channel);
var request = new RegisterOnEvent();
using (var call = eventService.OnEvent(request))
{
var responseStream = call.ResponseStream;
while (await responseStream.MoveNext())
{
//var receivedEvent = responseStream.Current;
// on change of any data, this method will be fired
GetJointPosition(channel, "Flower_id_22134");
}
}
}
private void GetJointPosition(Channel channel, string flowerName)
{
var client = new JointPositionService.JointPositionServiceClient(channel);
var request = new GetJointPositionRequest
{
FlowerName = flowerName
};
var response = client.GetJointPositionAsync(request);
SavePositions(response.ResponseAsync.Result.Positions);
}
private void SavePositions(IEnumerable<JointPosition> positions)
{
var jointPositions = positions.ToList();
for (var i = 0; i < Instance.Ref.AxesValues.Length; i++)
{
var axeValueDegree = jointPositions[i].Value * 180 / Math.PI;
Instance.Ref.AxesValues[i] = (float)axeValueDegree;
}
}
}
}
And I am calling it like:
var grpcChannel = new GrpcChannel();
grpcChannel.GrpcServer("192.168.123.16", 30201);
in Update() method. Unfortunately, it does not work in Start() method. And yes, apparently, every frame it needs to create a new Channel, otherwise it will not work.
And the current implementation is like this, which is calling it every 7 frames without using that special wait for events design:
using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Grpc.Core;
using TMPro;
using UnityEngine;
namespace Assets.Scripts
{
public class AxisValuesService : MonoBehaviour
{
public TextMeshProUGUI[] AxesValuesTexts;
[HideInInspector] public Dictionary<uint, float> AxisValues = new Dictionary<uint, float>();
[HideInInspector] private int counter = 0;
private void Update()
{
counter++;
if (counter == 7)
{
try
{
var channel = new Channel("192.168.123.16", 30201, ChannelCredentials.Insecure);
GetJointPosition(channel, "Flower_id_22134");
//ObserveEvents(channel).Wait();
channel.ShutdownAsync().Wait();
}
catch (RpcException e)
{
Debug.LogError("Connection Error: " + e);
}
counter = 0;
}
}
private void GetJointPosition(Channel channel, string robotName)
{
// Request Axis Values
var client = new JointPositionService.JointPositionServiceClient(channel);
var request = new GetJointPositionRequest { RobotName = robotName };
var response = client.GetJointPositionAsync(request);
// Fill Dictionary
foreach (var i in response.ResponseAsync.Result.Positions)
{
double value = toDegree((double)i.Value);
AxisValues[i.Index] = (float)Math.Round(value, 2);
}
try
{
AxesValuesTexts[0].text = AxisValues[1].ToString();
AxesValuesTexts[1].text = AxisValues[2].ToString();
AxesValuesTexts[2].text = AxisValues[3].ToString();
AxesValuesTexts[3].text = AxisValues[4].ToString();
AxesValuesTexts[4].text = AxisValues[5].ToString();
AxesValuesTexts[5].text = AxisValues[6].ToString();
}
catch (Exception e)
{
Debug.Log("Dictionary problem.");
}
}
private double toDegree(double rad)
{
return (float)(180 / Math.PI) * rad;
}
}
}
My Question is that, first, if this method is completely async, why does it still block the application in Unity, also how I can redesign it to achieve something like Udp or Tcp style?
Thanks to #Ilian, #zambari, #Jan Tattermusch and of course my colleague Rainer, the creator of the Grpc Api connection interface for us. I changed my structure to some degree that now works very performant both on Sender and Receiver computer. Below, I am going to explain what I have changed.
I have two classes, both are attached to one gameObject in Unity hierarchy: GrpcSetup.cs and AxeValuesConnectionInterface.cs. I commented on scripts, I hope it helps.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Google.Protobuf.WellKnownTypes;
using Grpc.Core;
using FlowerShop.Grpc.Service.Joint;
using FlowerShop.Grpc.Service.Joint.Event;
using FlowerShop.Grpc.Service.System;
using UnityEngine;
namespace Scripts
{
public class GrpcSetup : MonoBehaviour
{
private int _loopController = 1;
private Channel _grpcChannel;
private EventService.EventServiceClient _eventService;
private RegisterOnEvent _request;
private IAsyncStreamReader<Any> _responseStream;
private Any _receivedEvent;
private JointPositionChangedEvent _positionChangedEvent;
// this method will be called only one time in start method in another class
// .. I am initializing a channel for Grpc and some required properties for my ObserveEvents method
public void GrpcChannelInitialization()
{
_grpcChannel = new Channel("192.168.100.16", 50001, ChannelCredentials.Insecure);
_eventService = new EventService.EventServiceClient(_grpcChannel);
_request = new RegisterOnEvent();
}
// this method will be called in Update in another class
public async void GrpcUpdateMethod()
{
try
{
// to get the initial axesVales only one time
if (_loopController == 1)
{
await GetJointPositionOnDemand(_grpcChannel, "Flower_li35443");
_loopController = 0;
}
// receiving Events only when they are available
await ObserveEvents();
}
catch (RpcException e)
{
Debug.LogError("Connection Error: " + e);
}
}
// this method will be called every frame, the good thing about it is that, I will only receive events,
// .. when there are some available.
private async Task ObserveEvents()
{
using (var call = _eventService.OnEvent(_request))
{
_responseStream = call.ResponseStream;
if (await _responseStream.MoveNext())
{
Debug.Log("New Event is available.");
_receivedEvent = call.ResponseStream.Current;
if (_receivedEvent.TypeUrl.EndsWith(JointPositionChangedEvent.Descriptor.FullName))
{
_positionChangedEvent = _receivedEvent.Unpack<JointPositionChangedEvent>();
_positionChangedEvent.Positions.ToList().ForEach(i =>
Instance.Ref.AxesValues[i.Index - 1] = (float) Math.Round(i.Value * Mathf.Rad2Deg, 2));
}
}
}
}
// if I want to get Joint values whenever I like, regardless of ObserveEvents method architecture
// .. at this moment, I am calling it, one time in Update method
private async Task GetJointPositionOnDemand(Channel channel, string flowerName)
{
var client = new JointPositionService.JointPositionServiceClient(channel);
var requestLocal = new GetJointPositionRequest {FlowerName= flowerName};
var response = await client.GetJointPositionAsync(requestLocal);
foreach (var i in response.Positions)
{
var value = i.Value * Mathf.Rad2Deg;
Instance.Ref.AxesValues[i.Index - 1] = (float) Math.Round(value, 2);
}
}
// this will be called in Unity reserved method: OnApplicationQuit
// .. so we are trying to get rid of our opened channel
public async Task ChannelShutDown()
{
await _grpcChannel.ShutdownAsync();
}
}
}
and my AxeValuesConnectionInterface.cs goes like this:
using System.Threading.Tasks;
using UnityEngine;
namespace Scripts
{
[RequireComponent(typeof(GrpcSetup))]
public class AxeValuesConnectionInterface : MonoBehaviour
{
private GrpcSetup _grpcSetup;
private float _timeElapsed;
private int _loopController = 2;
private int _loopController1 = 1;
private int _loopController2 = 1;
private int _counterLoop;
private void Start()
{
_grpcSetup = GetComponent<GrpcSetup>();
}
private void Update()
{
GrpcInitialization();
GrpcUpdateMethods();
}
private void OnApplicationQuit()
{
Task.Run(_grpcSetup.ChannelShutDown);
}
private void GrpcInitialization()
{
if (_loopController2 != 1) return;
if (Instance.Ref.ConnectionInterface != Instance.Ci.Grpc) return;
_grpcSetup.GrpcChannelInitialization();
_loopController2 = 0;
}
private void GrpcUpdateMethods()
{
if (Instance.Ref.ConnectionInterface != Instance.Ci.Grpc || !Instance.Ref.FlowerIsPresent) return;
Task.Run(() => _grpcSetup.GrpcUpdateMethod());
}
}
}
I have troubles with my .NET web scraping software for http://mydataprovider.com/ service due to Memory Leak.
How my app works: it checks 10000 proxy servers for LIVE status.
Many proxies are broken so I have to filter them and to leave only active proxies (timeout response for live proxy is 3 seconds).
And I have to do it quickly (1 process starts ~80 threads).
I used WebClient class Firstly, but Timeout property does not effect right when I set it. I used HttpWebRequest Timeout, but it also did not help me with timeout.
I discovered at SO that I could use ThreadPool.RegisterWaitForSingleObject class for right Timeout processing (find below class HttpWebRequest_BeginGetResponse what I developed ) but it has troubles with memory leak and I did not find way how to fix it,
I tested in with .net 4.0 & 4.6.2 - behaviours are the same....
If any idea, help me, please.
Here is Code of class that is responsible for proxy activities:
using System;
using System.Net;
using System.IO;
using System.Text;
using System.Threading;
using System.Collections.Generic;
namespace ECommercePriceWebTaskManager
{
//read this http://stackoverflow.com/questions/1783031/c-sharp-asynchronous-operation
/*
BeginInvoke You tell the program what you need to be done (the delegate), what to call when it's done (callback), and what to do it with (state). You get back an IAsyncResult, which is the object that you need to give it back in order to receive your result. You can then do other stuff, or use the WaitHandle in the IAsyncResult to block until the operation's done.
Callback: When the asynchronous operation finishes, it will call this method, giving you the same IAsyncResult as before. At this point, you can retrieve your state object from it, or pass the IAsyncResult to EndInvoke.
EndInvoke: This function takes the IAsyncResult and finds the result of the operation. If it hasn't finished yet, it'll block until it does, which is why you usually call it inside the callback.
This is a pattern that's often used all over the framework, not just on function delegates. Things like database connections, sockets, etc. all often have Begin/End pairs.
*/
public class HttpWebRequest_BeginGetResponse_RequestState
{
public ManualResetEvent allDone = new ManualResetEvent(false);
public byte[] BufferRead;
public HttpWebRequest request;
public HttpWebResponse response;
public Stream responseStream;
public string Html;
public IAsyncResult ResponseIAsyncResult = null;
public IAsyncResult ReadIAsyncResult = null;
public List<Exception> Exceptions = new List<Exception>();
}
public class HttpWebRequest_BeginGetResponse
{
const int BUFFER_SIZE = 10240;
const int DefaultTimeout = 5 * 1000;
List<byte> _bytes = new List<byte>();
Encoding _encoding = Encoding.UTF8;
HttpWebRequest_BeginGetResponse_RequestState _requestState = new HttpWebRequest_BeginGetResponse_RequestState();
RegisteredWaitHandle RWH_GetResponse = null;
RegisteredWaitHandle RWH_Read = null;
public string Load(string url, WebProxy wp, Encoding en)
{
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.Proxy = wp;
string respUrl;
return Load(httpWebRequest, en, out respUrl);
}
public string Load(HttpWebRequest httpWebRequest, Encoding en, out string respUrl)
{
respUrl = "";
_encoding = en;
try
{
_requestState.request = httpWebRequest;
_requestState.ResponseIAsyncResult = (IAsyncResult)httpWebRequest.BeginGetResponse(new AsyncCallback(GetResponse), _requestState);
RWH_GetResponse = ThreadPool.RegisterWaitForSingleObject(_requestState.ResponseIAsyncResult.AsyncWaitHandle, new WaitOrTimerCallback(GetResponseTimeout), _requestState, DefaultTimeout, true);
_requestState.allDone.WaitOne();
if (_requestState.response != null)
{
if (_requestState.response.ResponseUri != null)
{
respUrl = _requestState.response.ResponseUri.AbsolutePath;
}
}
}
catch (Exception e)
{
AddException(e);
}
AbortAll();
if (_requestState.Exceptions.Count > 0)
{
throw new Exception("BeginGetResponse .... ");
//throw new AggregateException(_requestState.Exceptions);
}
return _requestState.Html;
}
private void GetResponseTimeout(object state, bool timedOut)
{
lock (this)
{
if (timedOut)
{
AbortAll();
AddException(new Exception("BeginGetResponse timeout (Internal)"));
_requestState.allDone.Set();
}
}
}
private void GetResponse(IAsyncResult asynchronousResult)
{
lock (this)
{
try
{
_requestState.response = (HttpWebResponse)_requestState.request.EndGetResponse(asynchronousResult);
if (_requestState.allDone.WaitOne(0, false))
{
AbortAll();
return;
}
_requestState.responseStream = _requestState.response.GetResponseStream();
_requestState.BufferRead = new byte[BUFFER_SIZE];
_requestState.ReadIAsyncResult = _requestState.responseStream.BeginRead(_requestState.BufferRead, 0, BUFFER_SIZE, new AsyncCallback(Read), _requestState);
RWH_Read = ThreadPool.RegisterWaitForSingleObject(_requestState.ReadIAsyncResult.AsyncWaitHandle, new WaitOrTimerCallback(ReadTimeout), _requestState, 1000, true);
return;
}
catch (Exception e)
{
AddException(e);
}
AbortAll();
_requestState.allDone.Set();
}
}
private void ReadTimeout(object state, bool timedOut)
{
lock (this)
{
if (timedOut)
{
AbortAll();
AddException(new Exception("ReadTimeoutCallback timeout (Internal)"));
_requestState.allDone.Set();
}
}
}
private void AbortAll()
{
try
{
if (_requestState.responseStream != null)
{
_requestState.responseStream.Close();
}
}
catch { }
try
{
if (_requestState.response != null)
{
_requestState.response.Close();
}
}
catch { }
try
{
if (_requestState.request != null)
{
_requestState.request.Abort();
}
}
catch { }
if (RWH_GetResponse != null)
RWH_GetResponse.Unregister(_requestState.ResponseIAsyncResult.AsyncWaitHandle);
if (RWH_Read != null)
RWH_Read.Unregister(_requestState.ReadIAsyncResult.AsyncWaitHandle);
}
void AddException(Exception ex)
{
_requestState.Exceptions.Add(ex);
}
private void Read(IAsyncResult asyncResult)
{
lock (this)
{
try
{
int read = _requestState.responseStream.EndRead(asyncResult);
if (_requestState.allDone.WaitOne(0, false))
{
AbortAll();
return;
}
if (read > 0)
{
for (var i = 0; i < read; i++)
{
_bytes.Add(_requestState.BufferRead[i]);
}
if (RWH_Read != null)
{
RWH_Read.Unregister(_requestState.ReadIAsyncResult.AsyncWaitHandle);
}
_requestState.ReadIAsyncResult = _requestState.responseStream.BeginRead(_requestState.BufferRead, 0, BUFFER_SIZE, new AsyncCallback(Read), _requestState);
RWH_Read = ThreadPool.RegisterWaitForSingleObject(_requestState.ReadIAsyncResult.AsyncWaitHandle, new WaitOrTimerCallback(ReadTimeout), _requestState, 1000, true);
return;
}
else
{
_requestState.Html = _encoding.GetString(_bytes.ToArray());
}
}
catch (Exception e)
{
AddException(e);
}
AbortAll();
_requestState.allDone.Set();
}
}
}
}
Sometimes I can get a strange exception, look at the next image, please:
How I use HttpWebRequest_BeginGetResponse class :
var hb = new HttpWebRequest_BeginGetResponse ();
hb.Load("http://your_url_here.com");
That code was called from ~80 threads in 1 process.
I am trying to just request the webpage by using webclient DownloadString method with proxy.
The following is my code:
using (WebClient wc = new WebClient())
{
try
{
wc.Encoding = Encoding.UTF8;
wc.Proxy = new WebProxy(myProxy);
string result = wc.DownloadString(url);
}
catch (Exception ex)
{
//log exception
}
}
I have few proxies and almost all of them work well with above code.
However, sometimes the response return time is very very long (over an hour) and I believe it's due to my proxy slow issue.
But I don't understand that why it didn't throw exception since the webclient class should have an default timeout (I search default timeout that it should be 100 seconds).
Therefore I want to ask how to avoid this case?
Thanks everyone.
It will be complicated as you need two threads - one for download and one for cancellation on timeout.
wc.Encoding = Encoding.UTF8;
wc.Proxy = new WebProxy(myProxy);
string result = null;
var waitCancel = new CancellationTokenSource();
wc.DownloadStringCompleted += (sender, e) =>
{
if (e.Cancelled) return;
waitCancel.Cancel();
result = e.Result;
};
wc.DownloadStringAsync(url);
waitCancel.Token.WaitHandle.WaitOne(30 * 1000);
if (waitCancel.IsCancellationRequested == false)
{
wc.CancelAsync();
}
Console.WriteLine("Result: " + result);
First you need to make use of the "new" HttpClient for .Net, below I will illustrate in two steps
HttpClient has handlers you can place in a chain and then control how the whole client behaves, so in this case we will add a progress handler that will dictate behavior in regard to the speed of the transfer
if the transfer is too slow then cancel it and start a new one with different proxy in ur case ...
Create the progress handler (to tell us how the progress is going .. If too slow then throw an exception)
Type :
ProgressMessageHandler
"https://msdn.microsoft.com/en-us/library/system.net.http.handlers.progressmessagehandler(v=vs.118).aspx"
Note : Package (System.Net.Http.Formatting.Extension) required
We will wrap it with an own object to erase testing and configuring to certain expected transfer rate
Code :
public class MyOwnProgressHandlerContainer
{
private readonly long _expectedBytesTransferred;
private long _lastRecoredBytesTransferred = 0;
public MyOwnProgressHandlerContainer(long expectedBytesTransferred)
{
_expectedBytesTransferred = expectedBytesTransferred;
}
public void ReceivedProgressHandler(object sender, HttpProgressEventArgs e)
{
// you can uses e.ProgressPercentage but this is calculated based on content length
// http header which is very much ignored nowadays
if (_lastRecoredBytesTransferred != 0 && e.BytesTransferred < (_lastRecoredBytesTransferred + _expectedBytesTransferred))
throw new Exception("Too Slow, Abort !");
_lastRecoredBytesTransferred = e.BytesTransferred;
}
public void SendProgressHandler(object sender, HttpProgressEventArgs e)
{
// write stuff to handle here when sending data (mainly for post or uploads)
Console.WriteLine("Sent data ...");
}
}
Create the HttpClient and inject the handlers that we wrapped :)
Type :
HttpClient
https://msdn.microsoft.com/en-us/library/system.net.http.httpclient(v=vs.118).aspx
Code :
// set expected rate
int expectedTransferRate = 25000; // AKA 2.5Mbps
// create own handler instance
MyOwnProgressHandlerContainer myHandler = new MyOwnProgressHandlerContainer(expectedTransferRate);
// create "ProgressMessageHandler"
ProgressMessageHandler progresshandler = new ProgressMessageHandler();
// Hookup event (send)
progresshandler.HttpSendProgress += myHandler.SendProgressHandler; // these are delegates so they can be a part of a stat-full class
// Hookup event (Receive)
progresshandler.HttpReceiveProgress += myHandler.ReceivedProgressHandler; // these are delegates so they can be a part of a stat-full class
// Create proxy handler
HttpClientHandler httpClientProxyHandler =
new HttpClientHandler
{
Proxy = new WebProxy("http://localhost:8888", false),
UseProxy = true
};
// Create client from factory with progress and "proxy" in your case
using (HttpClient httpClient = HttpClientFactory.Create(progresshandler, httpClientProxyHandler))
{
try
{
string downloadResult =
httpClient
// get result (progress handlers are notified based on sent / received data)
.GetAsync("https://httpbin.org/image/svg")
// could be a stream to read file content
.Result.Content.ReadAsStringAsync().Result;
Console.WriteLine(downloadResult);
}
catch (Exception)
{
// inspected if the exception is the same as the on u throw in MyOwnProgressHandlerContainer
throw;
}
}
WebClient doesnt have Timeout. You must extend it like this:
using System;
using System.Net;
namespace ConsoleApplication1
{
internal class Program
{
private static void Main(string[] args)
{
using (var webClient = new WebClientTimeOut())
{
webClient.Encoding = Encoding.UTF8;
webClient.Proxy = new WebProxy(myProxy);
try
{
var response = webClient.DownloadString("http://www.go1ogle.com");
Console.WriteLine(response);
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
}
}
public class WebClientTimeOut : WebClient
{
protected override WebRequest GetWebRequest(Uri uri)
{
var w = base.GetWebRequest(uri);
w.Timeout = 1000; // Timeout 1 second
return w;
}
}
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Facebook;
using System.Net;
using System.IO;
namespace WebSite_Login_And_Browsing
{
class Posts
{
public string PostId { get; set; }
public string PostStory { get; set; }
public string PostMessage { get; set; }
public string PostPicture { get; set; }
public string UserId { get; set; }
public string UserName { get; set; }
}
class FacebookPosts
{
static string accesstoken;
//static string token = "2f89d691b5f39";
static string token = "1186840401345424|GoJRCpM";
static string mytoken = "CAACEdEose0cBACPu39NSSalHCGFGDGRKZAvwiTuzG8PHlNRJwbyMVugovDxgL7CT3a1QbRuVDZALXxWU0ntwSrDyq75LIIuzFpBtx47cJYCY2OiA21lpTRKt2bB0t5HrsQYIXHXhmU7GnavWZCzqN8yeuv5NWXxTIOfVCZAZArjYNiPWhZBqZAZAO03s6FKNIulm4kjzXvp4QKiahAlcyaZBg";
static string mytokenaslip = "CAACEdEose0cBABmWuBI9p9dpPxEsMJoFZAG3kScx61kZAImNBgt52kVrd8WWPRpwjWP8nCPX69zdLuFyVQHzxYfMk85ZBZC4BIajVWXNLo7OI7yaCbNIwqkcdwpabQVFZBRWt0rzTQrQr6ZBij45XnrQyEUqFKP4gADeO4Fl9yRaZAZCOFtV3b84sWUFEgwaKbZAPY4BCljVjWQZDZD";
public static void RetrievePosts()
{
try
{
var client = new FacebookClient(mytokenaslip);
dynamic result = client.Get("/me/posts");
List<Posts> postsList = new List<Posts>();
//all the posts and their information (like pictures and links) is strored in result.data not in result
for (int i = 0; i < result.data.Count; i++)
{
Posts posts = new Posts();
posts.PostId = result.data[i].id;
if (object.ReferenceEquals(result.data[i].story, null))
posts.PostStory = "this story is null";
else
posts.PostStory = result.data[i].story;
if (object.ReferenceEquals(result.data[i].message, null))
posts.PostMessage = "this message is null";
else
posts.PostMessage = result.data[i].message;
posts.PostPicture = result.data[i].picture;
posts.UserId = result.data[i].from.id;
posts.UserName = result.data[i].from.name;
postsList.Add(posts);
}
}
catch (Exception err)
{
//throw;
string myerr = err.ToString();
}
}
}
}
I'm getting 25 results in the List postsList
How do i loop now asgain to get the next page with the next 25 results and add it to postsList and loop over and over again untill there are no more results ?
What i want to do is to delete automatic every 50 minutes the last old 25 posts.
In my other class in my project i'm posting automatic to my wall a post every minute. After 50 minutes i want to delete the last old 25 posts so on my wall will be all the time with 25 posts only.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
using mshtml;
using HtmlAgilityPack;
using System.Net;
using System.IO;
namespace WebSite_Login_And_Browsing
{
public partial class Facebook_Post : Form
{
WebBrowser wb = new WebBrowser();
int postsCounter = 0;
StreamWriter w = new StreamWriter(#"e:\posts.txt");
WebBrowser webBrowser1;
public Facebook_Post()
{
InitializeComponent();
webBrowser1 = new WebBrowser();
webBrowser1.DocumentCompleted += webBrowser1_DocumentCompleted;
webBrowser1.ScriptErrorsSuppressed = true;
webBrowser1.Navigate("https://www.facebook.com/");
label4.Text = DateTime.Now.ToString();
w.WriteLine(label4.Text.ToString());
w.WriteLine(Environment.NewLine);
label5.Visible = false;
label2.Visible = false;
}
private void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
try
{
if (e.Url.AbsoluteUri != webBrowser1.Url.AbsoluteUri)
{
return;
}
wb = webBrowser1;
foreach (HtmlElement he in wb.Document.All.GetElementsByName("xhpc_message"))
{
he.SetAttribute("value", RandomString(10));
}
var elems = wb.Document.GetElementsByTagName("button");
foreach (HtmlElement elem in elems)
{
if (elem.InnerText == "Post")
{
elem.InvokeMember("click");
}
}
sent = true;
postsCounter += 1;
label2.Text = postsCounter.ToString();
label2.Visible = true;
timer1.Enabled = true;
webBrowser1.Dispose();
if (postsCounter == 720)
{
w.WriteLine(postsCounter.ToString());
w.WriteLine(Environment.NewLine);
label5.Text = DateTime.Now.ToString();
label5.Visible = true;
w.WriteLine(label5.Text.ToString());
w.Close();
}
}
catch(Exception err)
{
string myerr = err.ToString();
}
}
private void button1_Click(object sender, EventArgs e)
{
List<string> results = new List<string>();
HtmlElementCollection elems = wb.Document.GetElementsByTagName("INPUT");
foreach (HtmlElement elem in elems)
{
String nameStr = elem.GetAttribute("value");
results.Add(nameStr);
}
}
bool sent = false;
int count = 0;
private void timer1_Tick(object sender, EventArgs e)
{
try
{
count += 1;
if (sent == true && count >= 60)
{
count = 0;
timer1.Enabled = false;
webBrowser1 = new WebBrowser();
if (webBrowser1.IsBusy == false)
{
webBrowser1.DocumentCompleted += webBrowser1_DocumentCompleted;
webBrowser1.Navigate("https://www.facebook.com/");
}
sent = false;
}
}
catch(Exception err)
{
string myerr = err.ToString();
}
}
private StringBuilder builder;
private static Random random = new Random((int)DateTime.Now.Ticks);
private string RandomString(int size)
{
try
{
builder = new StringBuilder();
char ch;
for (int i = 0; i < size; i++)
{
ch = Convert.ToChar(Convert.ToInt32(Math.Floor(26 * random.NextDouble() + 65)));
builder.Append(ch);
}
}
catch(Exception err)
{
string myerr = err.ToString();
}
return builder.ToString();
}
}
}
I believe this is what you're looking for:
var client = new FacebookClient(mytokenaslip);
//1-25
dynamic result = client.Get("/me/posts", new { limit = "25", offset = "0"});
//26-50
dynamic result = client.Get("/me/posts", new { limit = "25", offset = "25"});
You can also chose to get more than 25 posts at once.
//51-100
dynamic result = client.Get("/me/posts", new { limit = "50", offset = "50"});
You can use a "recursive function" to get all entries, and the "next" parameter in the API result includes the API call for the next batch of results: https://developers.facebook.com/docs/graph-api/using-graph-api#paging
Be careful though, you may hit an API limit if you try to do this too fast and if there are too many results. Also, since you want to delete old entries and deleting one entry is one API call, you should try with a timeout after each call just to make sure not to hit a limit.
Make sure you learn and understand how recursive functions work, here´s one of countless threads about that: Help with Creating a Recursive Function C#
I'm trying to write a remote control application to Windows Phone 7.
All goes OK, but i can not make IP scan over net.
Connected param in Socket class works badly. To use it i need to connect to any IP, then send anymessage - it's Ok. But timeout is too long - about 10 secs(for turned off Pcs).
I need to scan about 256 IPs, and 2560 seconds - is too long.
I tried to write timeout by my self. It works, but after some time application stops.
I can not understand why. No exceptions. Phone just stops attempts to connect and visual studio crashes sometimes.
If there is overflow - where can it be?
Here is some code:
(TryHost is runnung in FOR cycle. Only one start at a time, Searcher.AddHost() runs next iteration of TryHost)
AsynchronousClient:
public void TryHost(string host)
{
SocketAsyncEventArgs socketEventArg = new SocketAsyncEventArgs();
DnsEndPoint hostEntry = new DnsEndPoint(host, _port);
// Create a socket and connect to the server
Socket sock = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
socketEventArg.Completed += new EventHandler<SocketAsyncEventArgs>(SocketEventArg_Completed1);
socketEventArg.RemoteEndPoint = hostEntry;
sock.NoDelay = true;
socketEventArg.UserToken = sock;
try
{
sock.ConnectAsync(socketEventArg);
}
catch (SocketException ex)
{}
}
void SocketEventArg_Completed1(object sender, SocketAsyncEventArgs e)
{
switch (e.LastOperation)
{
case SocketAsyncOperation.Connect:
ProcessConnect1(e);
break;
case SocketAsyncOperation.SendTo:
connected = false;
ProcessSend1(e);
break;
case SocketAsyncOperation.Receive:
sended = false;
ProcessReceive1(e);
break;
default:
throw new Exception("Problems");
}
}
private void ProcessReceive1(SocketAsyncEventArgs e)
{
if (e.SocketError == SocketError.Success)
{
// Received data from server
string dataFromServer = Encoding.UTF8.GetString(e.Buffer, 0, e.BytesTransferred);
if (dataFromServer.IndexOf("<EOF>") > -1)
{
Searcher.AddHost(dataFromServer);
}
}
else
{
Searcher.AddHost(null);
}
}
private void ProcessSend1(SocketAsyncEventArgs e)
{
if (e.SocketError == SocketError.Success)
{
Socket sock = e.UserToken as Socket;
sended = true;
sock.ReceiveAsync(e);
Thread.Sleep(500);
if (sended)
{
sock.Dispose();
Searcher.AddHost(null);
}
}
else
{
Searcher.AddHost(null);
}
}
private void ProcessConnect1(SocketAsyncEventArgs e)
{
if (e.SocketError == SocketError.Success)
{
byte[] buffer = Encoding.UTF8.GetBytes("Knock" + "!" + sessionId + "<EOF>");
e.SetBuffer(buffer, 0, buffer.Length);
sock = e.UserToken as Socket;
connected = true;
sock.SendToAsync(e);
e.ConnectSocket.NoDelay = false;
Thread.Sleep(500);
if (connected)
{
sock.Dispose();
}
}
else
{
sock.Dispose();
Searcher.AddHost(null);
}
}
i'm using static class Searcher. UI button starts search from StartSearch() metod wich prepares class to search, and run methods in AsyncClient variable. When AssyncClient request is timed out or answered by host, AssyncClient runs AddHost() method
and Searcher prepares next IP, then start TryHost() again
static Searcher:
using System;
using System.Net;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Documents;
using System.Windows.Ink;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Animation;
using System.Windows.Shapes;
using System.Collections.Generic;
using System.Threading;
namespace PivotApp1 {
class PartedIp
{
public string FirstPart;
public string LastPart;
public PartedIp(string first, string last)
{
FirstPart = first;
LastPart = last;
}
}
public class SearchArgs : EventArgs
{
private List<string> message;
public SearchArgs(List<string> msg)
{
message = msg;
}
public List<string> Message
{
get { return message; }
}
}
public static class Searcher
{
static bool searching = false;
static string phoneIp = "";
static string ipTemplate = "";
static int currentLookingIp;
static int stopIp;
static List<string> answers = new List<string>();
static AsynchronousClient newC;
static int chBound = 255;
static int lwBound = 255;
public static event EventHandler SearchComplete = delegate { };
public static void SayItsOver(List<string> message)
{
SearchComplete(null, new SearchArgs(message));
}
static AsynchronousClient searcher;
static string CheckIp()
{
string phoneIp = null;
try
{
MyIPAddress finder = new MyIPAddress();
finder.Find((address) =>
{
phoneIp = address == null ? "Unknown" : address.ToString();
});
}
catch (Exception e)
{
throw new Exception();
}
if (phoneIp == "Unknown")
{
return null;
}
else
{
return phoneIp;
}
}
static PartedIp PrepareForSearch(string phoneIp)
{
IPAddress ipForTest = new IPAddress(10);
if (IPAddress.TryParse(phoneIp, out ipForTest))
{
string[] splittedIp = phoneIp.Split('.');
PartedIp phonePartedIp = new PartedIp(splittedIp[0] + '.' + splittedIp[1] + '.' + splittedIp[2] + '.', splittedIp[3]);
return phonePartedIp;
}
else return null;
}
public static void StartSearch(AsynchronousClient newC)
{
phoneIp = CheckIp();
if (phoneIp != null)
{
searching = true;
newC = newC1;
PartedIp partedIp = PrepareForSearch(phoneIp);
ipTemplate = partedIp.FirstPart;
stopIp = Int32.Parse(partedIp.LastPart);
currentLookingIp = stopIp - 1;
newC.TryHost(ipTemplate + currentLookingIp);
}
else
Deployment.Current.Dispatcher.BeginInvoke(() => {
MessageBox.Show("Error in Ip detection");
});
}
static void NextHost()
{
if (searching)
{
currentLookingIp--;
if (currentLookingIp == 0)
currentLookingIp = 255;
}
}
static public void AddHost(string answer)
{
if (searching)
{
if (answer != null)
{
answers.Add(answer);
}
NextHost();
if (currentLookingIp == stopIp)
{
SayItsOver(answers);
searching = false;
}
else
{
newC.TryHost(ipTemplate + currentLookingIp);
}
}
}
} }
I debugged this a lot of time. My start ip = 192.168.0.103. My Server Ip = 192.168.0.100.
Search working perfectly, until 192.168.0.19(no PCs here). Even if i start my search from 192.168.0.20 it crashed.
Another dead zone is 192.168.0.1 and next 192.168.0.255.
Also - manual starting of TryHost() works fine. Problems begin when I'm trying to automate search.
What's wrong?
I spent a few days on it and just do not know what to do.