C# Async WebRequests: Perform Action When All Requests Are Completed

C# Async WebRequests: Perform Action When All Requests Are Completed - c#

I have this basic scraping console application in C# that Asynchronously uses WebRequest to get html from a list of sites. It works fine, but how do I set up a trigger that goes off when every site in the list has been processed?
I've spent a couple hours researching various solutions online, including the MS docs, but none of them provide a straight forward answer via code. I've read about the IAsyncResult.AsyncWaitHandle but I have no clue how to integrate it into my code. I'd just like to call a custom function when all threads complete processing or timeout.
One trick is that I never know ahead of time how many sites are in my list (it's user defined), so I need a solution that's robust enough to wait for 5 events for 100,000 events to complete.
Thanks. Working code below:
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Net;
using System.Threading;
namespace AsyncApp_01
{
class Program
{
static void Main(string[] args)
{
ArrayList alSites = new ArrayList();
alSites.Add("http://www.google.com");
alSites.Add("http://www.lostspires.com");
ScanSites(alSites);
Console.Read();
}
private static void ScanSites(ArrayList sites)
{
foreach (string uriString in sites)
{
WebRequest request = HttpWebRequest.Create(uriString);
request.Method = "GET";
object data = new object(); //container for our "Stuff"
// RequestState is a custom class to pass info to the callback
RequestState state = new RequestState(request, data, uriString);
IAsyncResult result = request.BeginGetResponse(new AsyncCallback(UpdateItem), state);
//Register the timeout callback
ThreadPool.RegisterWaitForSingleObject(result.AsyncWaitHandle, new WaitOrTimerCallback(ScanTimeoutCallback), state, (30 * 1000), true);
}
}
private static void UpdateItem(IAsyncResult result)
{
// grab the custom state object
RequestState state = (RequestState)result.AsyncState;
WebRequest request = (WebRequest)state.Request;
// get the Response
HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result);
Stream s = (Stream)response.GetResponseStream();
StreamReader readStream = new StreamReader(s);
// dataString will hold the entire contents of the requested page if we need it.
string dataString = readStream.ReadToEnd();
response.Close();
s.Close();
readStream.Close();
Console.WriteLine(dataString);
}
private static void ScanTimeoutCallback(object state, bool timedOut)
{
if (timedOut)
{
RequestState reqState = (RequestState)state;
if (reqState != null)
{
reqState.Request.Abort();
}
Console.WriteLine("aborted- timeout");
}
}
class RequestState
{
public WebRequest Request; // holds the request
public object Data; // store any data in this
public string SiteUrl; // holds the UrlString to match up results (Database lookup, etc).
public RequestState(WebRequest request, object data, string siteUrl)
{
this.Request = request;
this.Data = data;
this.SiteUrl = siteUrl;
}
}
}
}
Bonus points for anyone who can also tell me how to limit the number of concurrent threads. For example, if I have 100 sites to process, how do I set it up so that 10 sites get processed at a time, but not more. I don't want to open 100 threads.

Here's a quick sample I threw together. I removed the WebClient implementation, as it seems like you're using the WebRequest one. I'm also making use of .Net 4's ConcurrentBag:
public class Scraper
{
private readonly IEnumerable<string> _sites;
private readonly ConcurrentBag<string> _data;
private volatile int _count;
private readonly int _total;
public Scraper(IEnumerable<string> sites)
{
_sites = sites;
_data = new ConcurrentBag<string>();
_total = sites.Count();
}
public void Start()
{
foreach (var site in _sites)
{
ScrapeSite(site);
}
}
private void ScrapeSite(string site)
{
var req = WebRequest.Create(site);
req.BeginGetResponse(AsyncCallback, req);
}
private void AsyncCallback(IAsyncResult ar)
{
Interlocked.Increment(ref _count);
var req = ar.AsyncState as WebRequest;
var result = req.EndGetResponse(ar);
var reader = new StreamReader(result.GetResponseStream());
var data = reader.ReadToEnd();
this.OnSiteScraped(req.RequestUri.AbsoluteUri, data);
_data.Add(data);
if (_count == _total)
{
OnScrapingComplete();
}
}
private void OnSiteScraped(string site, string data)
{
var handler = this.SiteScraped;
if (handler != null)
{
handler(this, new SiteScrapedEventArgs(site, data));
}
}
private void OnScrapingComplete()
{
var handler = this.ScrapingComplete;
if (handler != null)
{
handler(this, new ScrapingCompletedEventArgs(_data));
}
}
public event EventHandler<SiteScrapedEventArgs> SiteScraped;
public event EventHandler<ScrapingCompletedEventArgs> ScrapingComplete;
}
public class SiteScrapedEventArgs : EventArgs
{
public string Site { get; private set; }
public string Data { get; private set; }
public SiteScrapedEventArgs(string site, string data)
{
this.Site = site;
this.Data = data;
}
}
OK, I created some basic classes, and this should do the trick. If this isn't enough, I'm sorry, I simply can't help you:
public class RankedPage
{
public int Rank { get; set; }
public string Site { get; set; }
}
public class WebRequestData
{
public WebRequest WebRequest { get; set; }
public RankedPage Page { get; set; }
}
public class Scraper
{
private readonly IEnumerable<RankedPage> _sites;
private readonly ConcurrentBag<KeyValuePair<RankedPage,string>> _data;
private volatile int _count;
private readonly int _total;
public Scraper(IEnumerable<RankedPage> sites)
{
_sites = sites;
_data = new ConcurrentBag<KeyValuePair<RankedPage, string>>();
_total = sites.Count();
}
public void Start()
{
foreach (var site in _sites)
{
ScrapeSite(site);
}
}
private void ScrapeSite(RankedPage site)
{
var req = WebRequest.Create(site.Site);
req.BeginGetResponse(AsyncCallback, new WebRequestData{ Page = site, WebRequest = req});
}
private void AsyncCallback(IAsyncResult ar)
{
Interlocked.Increment(ref _count);
var webRequestData = ar.AsyncState as WebRequestData;
var req = webRequestData.WebRequest;
var result = req.EndGetResponse(ar);
var reader = new StreamReader(result.GetResponseStream());
var data = reader.ReadToEnd();
this.OnSiteScraped(webRequestData.Page, data);
_data.Add(new KeyValuePair<RankedPage, string>(webRequestData.Page,data));
if (_count == _total)
{
OnScrapingComplete();
}
}
private void OnSiteScraped(RankedPage page, string data)
{
var handler = this.SiteScraped;
if (handler != null)
{
handler(this, new SiteScrapedEventArgs(page, data));
}
}
private void OnScrapingComplete()
{
var handler = this.ScrapingComplete;
if (handler != null)
{
handler(this, new ScrapingCompletedEventArgs(_data));
}
}
public event EventHandler<SiteScrapedEventArgs> SiteScraped;
public event EventHandler<ScrapingCompletedEventArgs> ScrapingComplete;
}
public class SiteScrapedEventArgs : EventArgs
{
public RankedPage Site { get; private set; }
public string Data { get; private set; }
public SiteScrapedEventArgs(RankedPage site, string data)
{
this.Site = site;
this.Data = data;
}
}
public class ScrapingCompletedEventArgs : EventArgs
{
public IEnumerable<KeyValuePair<RankedPage,string >> SiteData { get; private set; }
public ScrapingCompletedEventArgs(IEnumerable<KeyValuePair<RankedPage, string>> siteData)
{
this.SiteData = siteData;
}
}

Related

Calling two methods of the same object in C# - chain like syntax

Cosidering I can't change class ReadTheFile, is there a way how to "chain" instance methods so I don't have to reference myObject multiple times? I am looking for something like
myObject.Read().Open(param1);
I can write myObject.Open(param1).Read(); which compiles however Read() is not executed.
//myObject.Open(param1).Read(); does execute but it's executed as StreamReader method not ReadTheFile method. Overlooked VS help...
class TestT211
{
static void Main(string[] args)
{
var myObject = new ReadTheFile();
myObject.Read(myObject.Open(#"C:\file.txt"));
}
}
public class ReadTheFile
{
private int _lineCounter = 0;
private string _lineOfText;
public StreamReader Open(string path)
{
return new StreamReader(path);
}
public void Read(StreamReader sr)
{
while ((_lineOfText = sr.ReadLine()) != null) {
Console.WriteLine(_lineOfText);
}
}
}

If you want to have a syntax like a Fluent API you need to change some point of your class. First you need to have Open return the current instance and then use that instance to call Read. But this assumes that you keep the StreamReader as an internal variable of the class
public class ReadTheFile : IDisposable
{
private int _lineCounter = 0;
private string _lineOfText;
private StreamReader _sr = null;
public ReadTheFile Open(string path)
{
_sr = new StreamReader(path);
return this;
}
public void Read()
{
if(_sr == null) return;
while ((_lineOfText = _sr.ReadLine()) != null) {
Console.WriteLine(_lineOfText);
}
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool disposing)
{
if (disposing)
{
if(_sr != null)
{
_sr.Close();
_sr = null;
}
}
}
}
and now you can write
using(ReadTheFile myObject = new ReadTheFile())
myObject.Open(#"C:\file.txt").Read();
Notice that having a StreamReader between your internal variables requires you to implement the IDisposable interface to properly close and dispose the Stream

If you want to add more functionality to class without access to it, you can use extensions. For example to use ReadTheFile to read StreamReader.
public static class StreamReaderExtension
{
public static StreamReader ReadEx(this StreamReader sr)
{
var readTheFile = new ReadTheFile();
readTheFile.Read(sr);
return sr;
}
}
and than call it with
myObject.Open(#"C:\file.txt").ReadEx();
You can add more methods, but cannot override the instance ones so you have to create a new name or new signature.

Using simple builder (it's overkill) but you could achive what you want:
class TestT211
{
static void Main(string[] args)
{
FileReaderBuilder.New.Open(#"C:\file.txt").Read();
}
}
public class ReadTheFile
{
private int _lineCounter = 0;
private string _lineOfText;
public StreamReader Open(string path)
{
return new StreamReader(path);
}
public void Read(StreamReader sr)
{
while ((_lineOfText = sr.ReadLine()) != null)
{
Console.WriteLine(_lineOfText);
}
}
}
public class FileReaderBuilder
{
private readonly ReadTheFile _file;
private StreamReader _streamReader;
private FileReaderBuilder()
{
_file = new ReadTheFile();
}
public FileReaderBuilder Open(string path)
{
_streamReader = _file.Open(path);
return this;
}
public FileReaderBuilder Read()
{
if (_streamReader == null)
{
throw new ArgumentNullException(nameof(_streamReader));
}
_file.Read(_streamReader);
return this;
}
public static FileReaderBuilder New => new FileReaderBuilder();
}

Handle all Events for an aggregate

Please see my first Persistent Subscription below:
namespace PersistentSubscription
{
internal class Program
{
private static void Main()
{
var subscription = new PersistentSubscriptionClient();
subscription.Start();
}
}
public class PersistentSubscriptionClient
{
private IEventStoreConnection _conn;
private const string STREAM = "$ce-customer";
private const string GROUP = "a_test_group";
private const int DEFAULTPORT = 1113;
private static readonly UserCredentials User = new UserCredentials("admin", "changeit");
private EventStorePersistentSubscriptionBase _subscription;
public void Start()
{
var settings = ConnectionSettings.Create();
using (_conn = EventStoreConnection.Create(settings, new IPEndPoint(IPAddress.Loopback, DEFAULTPORT)))
{
_conn.ConnectAsync().Wait();
CreateSubscription();
ConnectToSubscription();
Console.WriteLine("waiting for events. press enter to exit");
Console.ReadLine();
}
}
private void ConnectToSubscription()
{
var bufferSize = 10;
var autoAck = true;
Action<EventStorePersistentSubscriptionBase, ResolvedEvent> eventAppeared = EventAppeared;
_subscription = _conn.ConnectToPersistentSubscription(STREAM, GROUP, eventAppeared, SubscriptionDropped, User, bufferSize, autoAck);
}
private void SubscriptionDropped(EventStorePersistentSubscriptionBase eventStorePersistentSubscriptionBase,
SubscriptionDropReason subscriptionDropReason, Exception ex)
{
ConnectToSubscription();
}
private static void EventAppeared(EventStorePersistentSubscriptionBase eventStorePersistentSubscriptionBase,
ResolvedEvent resolvedEvent)
{
MemoryStream stream = new MemoryStream(resolvedEvent.Event.Data);
IFormatter formatter = new BinaryFormatter();
stream.Seek(0, SeekOrigin.Begin);
try
{
CustomerCreated customerCreated = (CustomerCreated)formatter.Deserialize(stream);
Console.WriteLine(customerCreated);
}
catch (Exception e)
{
var test = "test";
}
}
private void CreateSubscription()
{
PersistentSubscriptionSettings settings = PersistentSubscriptionSettings.Create()
.DoNotResolveLinkTos()
.StartFromCurrent();
try
{
_conn.CreatePersistentSubscriptionAsync(STREAM, GROUP, settings, User).Wait();
}
catch (AggregateException ex)
{
if (ex.InnerException.GetType() != typeof(InvalidOperationException)
&& ex.InnerException?.Message != $"Subscription group {GROUP} on stream {STREAM} already exists")
{
throw;
}
}
}
}
}
and my first client below:
using System;
using System.IO;
using System.Net;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;
using EventStore.ClientAPI;
namespace WritingEvents
{
class Program
{
static void Main(string[] args)
{
const int DEFAULTPORT = 1113;
var settings = ConnectionSettings.Create();
using (var conn = EventStoreConnection.Create(settings, new IPEndPoint(IPAddress.Loopback, DEFAULTPORT)))
{
conn.ConnectAsync().Wait();
CustomerCreated c1 = new CustomerCreated { Id = Guid.NewGuid(), Name = "Maria" };
EventData customerCreated1 = GetEventDataFor(c1);
conn.AppendToStreamAsync("customer-100", ExpectedVersion.Any, customerCreated1).Wait();
}
}
private static EventData GetEventDataFor(CustomerCreated customerCreated)
{
IFormatter formatter = new BinaryFormatter();
MemoryStream stream = new MemoryStream();
formatter.Serialize(stream, customerCreated);
byte[] customerCreatedEventByteArray = stream.ToArray();
return new EventData(
Guid.NewGuid(),
"eventType",
true,
customerCreatedEventByteArray,
null
);
}
}
[Serializable]
public class CustomerCreated
{
public Guid Id { get; set; }
public string Name { get; set; }
}
}
I run the server and then then the client. I see an error when deserializing the CustomerCreated event on the server side. The error is: "End of stream was encountered before parsing was completed".
If I change this line:
private const string STREAM = "$ce-customer";
to this:
private const string STREAM = "customer-100";
Then deserialization works correctly on the server side.
How do I handle all customer events - not just customer 100?
I have --run-projections=all when starting Event Store. I have also enabled all projections:

This question helped me: Using the Event Store Client API (.NET), how to I write to a stream and link one event to another?
I simply had to change this:
PersistentSubscriptionSettings settings = PersistentSubscriptionSettings.Create()
.DoNotResolveLinkTos() //Specifically this line
.StartFromCurrent();
to this:
PersistentSubscriptionSettings settings = PersistentSubscriptionSettings.Create()
.ResolveLinkTos() //Specifically this line
.StartFromCurrent();
DoNotResolveLinkTos gets a link to the original event, whereas ResolveLinkTos gets the actual event itself. Therefore I was trying to deserialize the link object, which was causing the exception.

Make SQL Calls on WindowsServices

I'm making an application as follows: I have a webservice running on a local server, this webservice returns json like this:
[{"Id":1,"Titulo":"Live to win","Link":"https://www.youtube.com/embed/DPHlGVe8wxI","BandaId":1,"BandaNome":"Paul Stanley","GeneroId":1,"GeneroNome":"Rock","DtCriacao":"2017-03-23T16:42:54","CriadorId":1,"CriadorNome":"Márcio Eric","Ativo":false},{"Id":2,"Titulo":"Welcome to the jungle","Link":null,"BandaId":2,"BandaNome":"Guns n´ roses","GeneroId":1,"GeneroNome":"Rock","DtCriacao":"2017-03-23T16:42:54","CriadorId":2,"CriadorNome":"Usuário Teste","Ativo":true},{"Id":3,"Titulo":"Something just like this","Link":null,"BandaId":3,"BandaNome":"The Chainsmokers","GeneroId":3,"GeneroNome":"Indie","DtCriacao":"2017-03-23T16:42:54","CriadorId":1,"CriadorNome":"Márcio Eric","Ativo":true},{"Id":4,"Titulo":"Beliver","Link":null,"BandaId":4,"BandaNome":"Imagine Dragons","GeneroId":3,"GeneroNome":"Indie","DtCriacao":"2017-03-23T16:42:54","CriadorId":1,"CriadorNome":"Márcio Eric","Ativo":true},{"Id":5,"Titulo":"Radioactive","Link":null,"BandaId":4,"BandaNome":"Imagine Dragons","GeneroId":3,"GeneroNome":"Indie","DtCriacao":"2017-03-23T16:42:54","CriadorId":1,"CriadorNome":"Márcio Eric","Ativo":true},{"Id":6,"Titulo":"Friends - Original Mix","Link":null,"BandaId":5,"BandaNome":"Steener","GeneroId":2,"GeneroNome":"EDM","DtCriacao":"2017-03-23T16:42:54","CriadorId":2,"CriadorNome":"Usuário Teste","Ativo":true},{"Id":7,"Titulo":"Amanheceu","Link":null,"BandaId":6,"BandaNome":"Scalene","GeneroId":3,"GeneroNome":"Indie","DtCriacao":"2017-03-23T16:42:54","CriadorId":2,"CriadorNome":"Usuário Teste","Ativo":true},{"Id":8,"Titulo":"Sonhador II","Link":null,"BandaId":6,"BandaNome":"Scalene","GeneroId":3,"GeneroNome":"Indie","DtCriacao":"2017-03-23T16:42:54","CriadorId":2,"CriadorNome":"Usuário Teste","Ativo":true},{"Id":9,"Titulo":"Amianto","Link":null,"BandaId":7,"BandaNome":"Supercombo","GeneroId":3,"GeneroNome":"Indie","DtCriacao":"2017-03-23T16:42:54","CriadorId":1,"CriadorNome":"Márcio Eric","Ativo":true},{"Id":10,"Titulo":"Monstros","Link":null,"BandaId":7,"BandaNome":"Supercombo","GeneroId":3,"GeneroNome":"Indie","DtCriacao":"2017-03-23T16:42:54","CriadorId":1,"CriadorNome":"Márcio Eric","Ativo":true},{"Id":11,"Titulo":"Piloto Automático","Link":null,"BandaId":7,"BandaNome":"Supercombo","GeneroId":3,"GeneroNome":"Indie","DtCriacao":"2017-03-23T16:42:54","CriadorId":2,"CriadorNome":"Usuário Teste","Ativo":true},{"Id":12,"Titulo":"Eutanásia","Link":null,"BandaId":7,"BandaNome":"Supercombo","GeneroId":3,"GeneroNome":"Indie","DtCriacao":"2017-03-23T16:42:54","CriadorId":1,"CriadorNome":"Márcio Eric","Ativo":true},{"Id":13,"Titulo":"Shots - Broiler Remix","Link":null,"BandaId":4,"BandaNome":"Imagine Dragons","GeneroId":3,"GeneroNome":"Indie","DtCriacao":"2017-03-24T16:55:46","CriadorId":1,"CriadorNome":"Márcio Eric","Ativo":true}]
But now I need to create a Windows Services that communicates with this WebService and what it receives in a database, I spent all day doing this webservice in several ways, but I did not find anything about good practices with webservices or etc. I'll show you what I did, but I'd like some tips on windows services, thank you
Here is my code
Service:
private Timer _worker;
private readonly int _interval = Convert.ToInt32(ConfigurationManager.AppSettings["timer"]);
private readonly string _connection = ConfigurationManager.AppSettings["connection"];
//private readonly Connector _usuarioConnector;
private readonly Connector _bandaConnector;
//private readonly Connector _generoConnector;
private readonly Connector _musicaConnector;
private SqlConnection conn;
public Service1()
{
//_usuarioConnector = new Connector("UsuarioBase");
_bandaConnector = new Connector("BandaBase");
//_generoConnector = new Connector("GeneroBase");
_musicaConnector = new Connector("MusicaBase");
InitializeComponent();
}
protected override void OnStart(string[] args)
{
_worker = new Timer((Update), null, 0, _interval);
}
protected override void OnStop()
{
}
private void Update(Object state)
{
using (SqlConnection connection = new SqlConnection(
_connection))
{
SqlCommand command = new SqlCommand("insert into Genero(nome, descricao, dtcriacao, criadorid, ativo) values('teste', 'teste', getdate(), 1, 1)", connection);
command.Connection.Open();
command.ExecuteNonQuery();
EventLog.WriteEntry("Query executada", EventLogEntryType.Warning);
}
}
Loader
public class Loader
{
public static List<Usuario> LoadUsuarios(Connector usuarioConnector)
{
return (List<Usuario>)Newtonsoft.Json.JsonConvert.DeserializeObject(usuarioConnector.GetData(), typeof(List<Usuario>));
}
public static List<Banda> LoadBandas(Connector bandaConnector)
{
var bandasDtos = (List<BandaDto>)Newtonsoft.Json.JsonConvert.DeserializeObject(bandaConnector.GetData(), typeof(List<BandaDto>));
return bandasDtos.Select(dto => dto.ConvertToBanda()).ToList();
}
//public static List<Genero> LoadGeneros(Connector generoConnector)
//{
//var musicasDtos = (List<MusicaDto>)Newtonsoft.Json.JsonConvert.DeserializeObject(musicaConnector.GetData(), typeof(List<MusicaDto>));
//return musicasDtos.Select(dto => dto.ConvertToMusica()).ToList();
//}
public static List<Musica> LoadMusicas(Connector musicaConnector)
{
var musicasDtos = (List<MusicaDto>)Newtonsoft.Json.JsonConvert.DeserializeObject(musicaConnector.GetData(), typeof(List<MusicaDto>));
return musicasDtos.Select(dto => dto.ConvertToMusica()).ToList();
}
}
Connector
private string Host { get; }
private WebRequest _request;
public WebResponse Response;
public string ConnectionStatus;
private Stream _dataStream;
private StreamReader _reader;
public Connector(string config)
{
Host = ConfigurationManager.AppSettings[config];
}
public string GetData()
{
StartConnection();
_dataStream = Response.GetResponseStream();
_reader = new StreamReader(_dataStream);
var toReturn = _reader.ReadToEnd();
EndConnection();
return toReturn;
}
private void StartConnection()
{
_request = WebRequest.Create(Host);
_request.Credentials = CredentialCache.DefaultCredentials;
Response = _request.GetResponse();
ConnectionStatus = (((HttpWebResponse)Response).StatusDescription);
}
private void EndConnection()
{
_reader.Close();
Response.Close();
}
Edit1 In the current code it is doing a select, however in what I intend to do it will do inserts

I discovered, unfortunately the timer was not working, so I just changed it to a thread.

Validation Exception Queue using MSMQ in C#

I'm new in Microsoft Message Queue in Windows Server, I need to push, if the EmployeeID is NULL.
The Employee Model Class is
public class Employee
{
public string EmployeeID { get; set; }
public string EmployeeName { get; set; }
}
public void ValidationProcess(Employee emp)
{
if((emp != null) || (emp.EmployeeID == null))
{
// Push into Validation Exception Queue using MSMQ
}
}
Once the Data pushed into that Validation Exception Queue, it should be processed by separate process. Every 1hr the process need to initiate and it should call the following method
public void ValidationExceptionProcess(object obj)
{
// Some Inner Process
// Log the Error
}
Kindly guide me how to create and process it.

First Step:
Install MSMQs as a windows feature on the server/pc
Then:
- Create the queue if it does not exist
- Push the message in the queue asynchronously
Useful guide
Code example for pushing and retrieving messages from msmq:
public class ExceptionMSMQ
{
private static readonly string description = "Example description";
private static readonly string path = #".\Private$\myqueue";
private static MessageQueue exceptionQueue;
public static MessageQueue ExceptionQueue
{
get
{
if (exceptionQueue == null)
{
try
{
if (MessageQueue.Exists(path))
{
exceptionQueue = new MessageQueue(path);
exceptionQueue.Label = description;
}
else
{
MessageQueue.Create(path);
exceptionQueue = new MessageQueue(path);
exceptionQueue.Label = description;
}
}
catch
{
throw;
}
finally
{
exceptionQueue.Dispose();
}
}
return exceptionQueue;
}
}
public static void PushMessage(string message)
{
ExceptionQueue.Send(message);
}
private static List<string> RetrieveMessages()
{
List<string> messages = new List<string>();
using (ExceptionQueue)
{
System.Messaging.Message[] queueMessages = ExceptionQueue.GetAllMessages();
foreach (System.Messaging.Message message in queueMessages)
{
message.Formatter = new XmlMessageFormatter(
new String[] { "System.String, mscorlib" });
string msg = message.Body.ToString();
messages.Add(msg);
}
}
return messages;
}
public static void Main(string[] args)
{
ExceptionMSMQ.PushMessage("my exception string");
}
}
An other widely used way to do that would also be to use out of the box loggers which already contains this functionality like Enterprise Library or NLog which provide easy interfaces to do that.
For retrieving messages I would recommend a separate windows service which would periodically read messages and process them. An good example on how to do that is given here: Windows service with timer
Update: Windows Service Example:
MSMQConsumerService.cs
public partial class MSMQConsumerService : ServiceBase
{
private System.Timers.Timer timer;
public MSMQConsumerService()
{
InitializeComponent();
}
protected override void OnStart(string[] args)
{
this.timer = new System.Timers.Timer(30000D); // 30000 milliseconds = 30 seconds
this.timer.AutoReset = true;
this.timer.Elapsed += new System.Timers.ElapsedEventHandler(this.ProcessQueueMessages);
this.timer.Start();
}
protected override void OnStop()
{
this.timer.Stop();
this.timer = null;
}
private void ProcessQueueMessages(object sender, System.Timers.ElapsedEventArgs e)
{
MessageProcessor.StartProcessing();
}
}
and the MessageProcessor.cs
public class MessageProcessor
{
public static void StartProcessing()
{
List<string> messages = ExceptionMSMQ.RetrieveMessages();
foreach(string message in messages)
{
//write message in database
}
}
}

Design solution for working with multiple instantiations at the same time

I don't know if my title is correct. But here's what I want to know.
I have a Download class that returns certain events and has a couple of methods. Each instance of Download class can download a single file. And all those events and methods are related to the file being downloaded.
As it's a multi file downloader, multiple instantiations are required when more than a single file needs to be downloaded.
Each download has a download id, but that is not supplied to the Download class, to keep it independent from the other classes.
Now getting all the info from each instance of the file download and being able to control a single download, is the problem. How do I know which download is which?
Any solutions? Or design patterns you could recommend? I've hit a roadblock.
Download class:
using System;
using System.Diagnostics;
using System.IO;
using System.Net;
using System.Threading;
namespace Mackerel_Download_Manager
{
public class Download
{
public event EventHandler<DownloadStatusChangedEventArgs> ResumablityChanged;
public event EventHandler<DownloadProgressChangedEventArgs> ProgressChanged;
public event EventHandler Completed;
public bool stop = true; // by default stop is true
public bool paused = false;
SemaphoreSlim pauseLock = new SemaphoreSlim(1);
string filename;
public void DownloadFile(Uri DownloadLink, string Path)
{
filename = System.IO.Path.GetFileName(Path);
stop = false; // always set this bool to false, everytime this method is called
var fileInfo = new FileInfo(Path);
long existingLength = 0;
if (fileInfo.Exists)
existingLength = fileInfo.Length;
var request = (HttpWebRequest)HttpWebRequest.Create(DownloadLink);
request.Proxy = null;
request.AddRange(existingLength);
try
{
using (var response = (HttpWebResponse)request.GetResponse())
{
long fileSize = existingLength + response.ContentLength; //response.ContentLength gives me the size that is remaining to be downloaded
bool downloadResumable; // need it for not sending any progress
if ((int)response.StatusCode == 206) //same as: response.StatusCode == HttpStatusCode.PartialContent
{
//Console.WriteLine("Resumable");
downloadResumable = true;
}
else // sometimes a server that supports partial content will lose its ability to send partial content(weird behavior) and thus the download will lose its resumability
{
if (existingLength > 0)
{
if (ResumeUnsupportedWarning() == false) // warn and ask for confirmation to continue if the half downloaded file is unresumable
{
return;
}
}
existingLength = 0;
downloadResumable = false;
}
OnResumabilityChanged(new DownloadStatusChangedEventArgs(downloadResumable));
using (var saveFileStream = fileInfo.Open(downloadResumable ? FileMode.Append : FileMode.Create, FileAccess.Write))
using (var stream = response.GetResponseStream())
{
byte[] downBuffer = new byte[4096];
int byteSize = 0;
long totalReceived = byteSize + existingLength;
var sw = Stopwatch.StartNew();
while (!stop && (byteSize = stream.Read(downBuffer, 0, downBuffer.Length)) > 0)
{
saveFileStream.Write(downBuffer, 0, byteSize);
totalReceived += byteSize;
float currentSpeed = totalReceived / (float)sw.Elapsed.TotalSeconds;
OnProgressChanged(new DownloadProgressChangedEventArgs(totalReceived, fileSize, (long)currentSpeed));
pauseLock.Wait();
pauseLock.Release();
}
sw.Stop();
}
}
if (!stop) OnCompleted(EventArgs.Empty);
}
catch (WebException e)
{
System.Windows.MessageBox.Show(e.Message, filename);
}
}
public void pause()
{
if (!paused)
{
paused = true;
// Note this cannot block for more than a moment
// since the download thread doesn't keep the lock held
pauseLock.Wait();
}
}
public void unpause()
{
if (paused)
{
paused = false;
pauseLock.Release();
}
}
public void StopDownload()
{
stop = true;
this.unpause(); // stop waiting on lock if needed
}
public bool ResumeUnsupportedWarning()
{
var messageBoxResult = System.Windows.MessageBox.Show("When trying to resume the download , Mackerel got a response from the server that it doesn't support resuming the download. It's possible that it's a temporary error of the server, and you will be able to resume the file at a later time, but at this time Mackerel can download this file from the beginning.\n\nDo you want to download this file from the beginning?", filename, System.Windows.MessageBoxButton.YesNo);
if (messageBoxResult == System.Windows.MessageBoxResult.Yes)
{
return true;
}
else
{
return false;
}
}
protected virtual void OnResumabilityChanged(DownloadStatusChangedEventArgs e)
{
var handler = ResumablityChanged;
if (handler != null)
{
handler(this, e);
}
}
protected virtual void OnProgressChanged(DownloadProgressChangedEventArgs e)
{
var handler = ProgressChanged;
if (handler != null)
{
handler(this, e);
}
}
protected virtual void OnCompleted(EventArgs e)
{
var handler = Completed;
if (handler != null)
{
handler(this, e);
}
}
}
public class DownloadStatusChangedEventArgs : EventArgs
{
public DownloadStatusChangedEventArgs(bool canResume)
{
ResumeSupported = canResume;
}
public bool ResumeSupported { get; private set; }
}
public class DownloadProgressChangedEventArgs : EventArgs
{
public DownloadProgressChangedEventArgs(long totalReceived, long fileSize, long currentSpeed)
{
BytesReceived = totalReceived;
TotalBytesToReceive = fileSize;
CurrentSpeed = currentSpeed;
}
public long BytesReceived { get; private set; }
public long TotalBytesToReceive { get; private set; }
public float ProgressPercentage
{
get
{
return ((float)BytesReceived / (float)TotalBytesToReceive) * 100;
}
}
public float CurrentSpeed { get; private set; } // in bytes
public TimeSpan TimeLeft
{
get
{
var bytesRemainingtoBeReceived = TotalBytesToReceive - BytesReceived;
return TimeSpan.FromSeconds(bytesRemainingtoBeReceived / CurrentSpeed);
}
}
}
}
Download class is instantiated inside a Mackerel class, that starts the download for the given downloads.
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Windows;
namespace Mackerel_Download_Manager
{
public static class Mackerel
{
//Main Menu functions
public static void ResumeDownload(string[] DownloadIDs)
{
foreach (var DownloadID in DownloadIDs)
{
var itemToResume = Downloads.DownloadEntries.Where(download => download.DownloadID == DownloadID).FirstOrDefault();
if (itemToResume.Running == false)
{
itemToResume.Running = true;
var download = new Download();
download.DownloadFile(itemToResume.DownloadLink, itemToResume.SaveTo);
var window = new Dialogs.DownloadProgress(itemToResume);
window.Show();
double progress = 0;
itemToResume.Status = string.Format("{0:0.00}%", progress);
Downloads.DownloadEntries.CollectionChanged += delegate
{
if (!itemToResume.Running) window.Close();
};
}
}
}
public static void StopDownload(string[] DownloadIDs)
{
foreach (var DownloadID in DownloadIDs)
{
var itemToStop = Downloads.DownloadEntries.Where(download => download.DownloadID == DownloadID).FirstOrDefault();
if (itemToStop.Running == true)
itemToStop.Running = false;
}
}
public static void StopAllDownloads()
{
foreach (var itemToStop in Downloads.DownloadEntries.Where(download => download.Running == true))
itemToStop.Running = false;
}
public static void RemoveDownload(string[] DownloadIDs) // this method is able to delete multiple downloads
{
foreach (var DownloadID in DownloadIDs)
{
// delete from the download list
var selectedDownload = Downloads.DownloadEntries.Where(download => download.DownloadID == DownloadID).FirstOrDefault();
var selectedDownloadIndex = Downloads.DownloadEntries.IndexOf(selectedDownload);
Downloads.DownloadEntries.RemoveAt(selectedDownloadIndex);
//delete from the harddrive
if (File.Exists(selectedDownload.SaveTo))
File.Delete(selectedDownload.SaveTo);
}
Downloads.Serialize(); // save current state of object
}
public static void RemoveCompletedDownloads() // this method just removes all completed downloads from Mackerel's download list (it doesn't delete them from the hard drive)
{
foreach (var itemToRemove in Downloads.DownloadEntries.Where(download => download.Status == "Complete").ToList())
{
Downloads.DownloadEntries.Remove(itemToRemove);
}
}
// Context Menu
public static void OpenDownloadProperties(string DownloadID) // Open "Download Properties" for the given download ID
{
var DownloadProperties = new Dialogs.Context_Menu.DownloadProperties(DownloadID);
DownloadProperties.Owner = Application.Current.MainWindow; // so that this dialog centers to its parent window, as its window is set to WindowStartupLocation="CenterOwner"
DownloadProperties.ShowDialog();
}
}
}
Full source code is here: https://github.com/Expenzor/mackerel-download-manager

Sounds like your making the download itself an object but are using "downloader" as the name instead. I would maybe suggest an array of downloaded objects or similar. The downloader class can have a method which creates a new object - perhaps a download object. Using an array or linked list etc will give you the opportunity to address the object and call them independently - know which is which.
Posting your code would help as well.

Develop Reference

C# (C-Sharp) is a programming language developed by Microsoft that runs on the .NET Framework.

C# Async WebRequests: Perform Action When All Requests Are Completed - c#

Related

Calling two methods of the same object in C# - chain like syntax

Handle all Events for an aggregate

Make SQL Calls on WindowsServices

Validation Exception Queue using MSMQ in C#

Design solution for working with multiple instantiations at the same time

Categories

Resources