Develop Reference

Cancel Site Crawl on Abot

I have a list of domains that are crawled using Abot, the aim is that when it finds an amazon link on one of the sites, it quits, then moves onto the next site. But I can't seem to see who I can exit the page crawl. e.g.
https://github.com/sjdirect/abot
static Main(string[] args)
{
var domains= new List<string> { "http://domain1", "http://domain2" };
foreach (string domain in domains)
{
var config = new CrawlConfiguration
{
MaxPagesToCrawl = 100,
MinCrawlDelayPerDomainMilliSeconds = 3000
};
var crawler = new PoliteWebCrawler(config);
crawler.PageCrawlCompleted += PageCrawlCompleted;
var uri = new Uri(domain);
var crawlResult = crawler.Crawl(uri);
}
}
private static void PageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
{
var crawledPage = e.CrawledPage;
var crawlContext = e.CrawlContext;
var document = crawledPage.AngleSharpHtmlDocument;
var anchors = document.QuerySelectorAll("a").OfType<IHtmlAnchorElement>();
var hrefs = anchors.Select(x => x.Href).ToList();
var regEx= new Regex(#"https?:\/\/(www|smile)\.amazon(\.co\.uk|\.com).*");
var resultList = hrefs.Where(f => regEx.IsMatch(f)).ToList();
if (resultList.Any())
{
//NEED TO EXIT THE SITE CRAWL HERE
}
}

I would suggest the following...
var myCancellationToken = new CancellationTokenSource();
crawler.CrawlAsync(someUri, myCancellationToken);
private static void PageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
{
//More performant (since the parsing has already been done by Abot)
var hasAmazonLinks = e.CrawledPage.ParsedLinks
.Any(hl => hl.HrefValue.AbsoluteUri
.ToLower()
.Contains("amazon.com"));
if (hasAmazonLinks)
{
//LOG SOMETHING BEFORE YOU STOP THE CRAWL!!!!!
//Option A: Preferred method, Will clear all scheduled pages and cancel any threads that are currently crawling
myCancellationToken.Cancel();
//Option B: Same result as option A but no need to do anything with tokens. Not the preferred method.
e.CrawlContext.IsCrawlHardStopRequested = true;
//Option C: Will clear all scheduled pages but will allow any threads that are currently crawling to complete. No cancellation tokens needed. Consider it a soft stop to the crawl.
e.CrawlContext.IsCrawlStopRequested = true;
}
}

PoliteWebCrawler is designed to start crawling and dig deeper into the website URLs. If you just want to get the content of a URL (for example first page of a website) you can use PageRequester which is designed for such jobs.
var pageRequester = new PageRequester(new CrawlConfiguration(), new WebContentExtractor());
var crawledPage = await pageRequester.MakeRequestAsync(new Uri("http://google.com"));
Log.Logger.Information("{result}", new
{
url = crawledPage.Uri,
status = Convert.ToInt32(crawledPage.HttpResponseMessage.StatusCode)
});
By the way, if you want to stop a crawler during the process, you can use one of these two methods:
//1. hard crawl stop
crawlContext.CancellationTokenSource.Cancel();
//2. soft stop
crawlContext.IsCrawlStopRequested = true;

How to chain together multiple NAudio ISampleProvider Effects

I have some DSP effects coded in the ISampleProvider model. To apply one effect I do this and it works fine.
string filename = "C:\myaudio.mp3";
MediaFoundationReader mediaFileReader = new MediaFoundationReader(filename);
ISampleProvider sampProvider = mediaFileReader.ToSampleProvider();
ReverbSampleProvider reverbSamplr = new ReverbSampleProvider(sampProvider);
IWavePlayer waveOutDevice.Init(reverbSamplr);
waveOutDevice.Play();
How can I apply multiple effects to the same input file simultaneously?
For example, if i have a Reverb effect and Distortion effect providers, how can I chain them together to apply them at the same time to one input file?

Effects can be chained together by passing one as the "source" for the next. So if you wanted your audio to go first through a reverb, and then distortion, you might do something like this, passing the original audio into the Reverb effect, the output of the reverb into the distortion effect and then sending the distortion to the waveOut device.
var reverb = new ReverbSampleProvider(sampProvider);
var distortion = new DistortionSampleProvider(reverb);
waveOutDevice.Init(distortion);
(n.b. NAudio does not come with built in reverb/distortion effects - you must make these yourself or source them from elsewhere)

Mark's answer is correct, but that approach is a pain if you're copy and pasting things around in different orders, because you have to change the variables that you're passing through.
For example, if you start with:
var lpf = new LowPassEffectStream(input);
var reverb = new ReverbEffectStream(lpf);
var stereo = new StereoEffectStream(reverb);
var vol = new VolumeSampleProvider(stereo);
waveOutDevice.Init(vol);
And you want to swap reverb and stereo, a quick copy-paste leaves you with the input variables backwards:
var lpf = new LowPassEffectStream(input);
var stereo = new StereoEffectStream(reverb); // <--
var reverb = new ReverbEffectStream(lpf); // <--
var vol = new VolumeSampleProvider(stereo);
waveOutDevice.Init(vol);
It also makes it easy to fix a parameter but forget to fix another, e.g. fixing the stereo effect to have lpf as its input, but forgetting to fix the reverb effect. This often results in skipped effects in the chain leading to frustrated debugging when the effect appears not to work.
To make things easier and less error-prone when I'm stacking effects together and re-ordering them, I created the following helper class:
class EffectChain : ISampleProvider
{
public EffectChain(ISampleProvider source)
{
this._sourceStream = source;
}
private readonly ISampleProvider _sourceStream;
private readonly List<ISampleProvider> _chain = new List<ISampleProvider>();
public ISampleProvider Head
{
get
{
return _chain.LastOrDefault() ?? _sourceStream;
}
}
public WaveFormat WaveFormat
{
get
{
return Head.WaveFormat;
}
}
public void AddEffect(ISampleProvider effect)
{
_chain.Add(effect);
}
public int Read(float[] buffer, int offset, int count)
{
return Head.Read(buffer, offset, count);
}
}
You can use it like this:
var effectChain = new EffectChain(input);
var lpf = new LowPassEffectStream(effectChain.Head);
effectChain.AddEffect(lpf);
var stereo = new StereoEffectStream(effectChain.Head);
effectChain.AddEffect(stereo);
var reverb = new ReverbEffectStream(effectChain.Head);
effectChain.AddEffect(reverb);
var vol = new VolumeSampleProvider(effectChain.Head);
effectChain.AddEffect(vol);
waveOutDevice.Init(effectChain);
This allows you to quickly re-order effects in the chain, as each effect takes the effect chain's head as an input. If you don't add any effects it just acts as a pass-through. You could easily expand this class to have more methods for managing the contained effects if you wanted to, but as it stands it works quite cleanly.

How to properly write a fade-out to a WAV file using NAudio?

I'm using NAudio to convert & trim some audio files, and I'm trying to add a fade-out to the last few seconds of each file.
I have checked this question, this, and this, but all the answers are talking about playing the wav file with fade, while I need to actually write that fade to an output file.
So, is there any way to do this using NAudio? If not, I'm open to other suggestions.
Edit: This is what I've tried:
private void PerformFadeOut(string inputPath, string outputPath)
{
WaveFileReader waveSource = new WaveFileReader(inputPath);
ISampleProvider sampleSource = waveSource.ToSampleProvider();
OffsetSampleProvider fadeOutSource = new OffsetSampleProvider(sampleSource);
// Assume the length of the audio file is 122 seconds.
fadeOutSource.SkipOver = TimeSpan.FromSeconds(120); // Hard-coded values for brevity
// Two seconds fade
var fadeOut = new FadeInOutSampleProvider(fadeOutSource);
fadeOut.BeginFadeOut(2000);
Player = new WaveOut();
Player.Init(fadeOut);
Player.Play();
}
When I play the audio after applying the fade using Player.Play() -as shown in the method above-, it works perfectly as expected, and I can hear the fade. Now, I would like to export this result to an output WAV file.
I tried doing that by adding the following line:
WaveFileWriter.CreateWaveFile(outputPath, waveSource);
However, the output file doesn't have any fade applied to it. So, what am I missing here?

Okay, let's wrap everything up in case someone encounters the same issue in the future:
With the great help of #yms, I managed to write the fade to a file by using:
WaveFileWriter.CreateWaveFile(outputPath, new SampleToWaveProvider(fadeOut));
But that caused the wave writer to only write the last two seconds which makes sense, so I tried using the DelayFadeOutSampleProvider class instead of FadeInOutSampleProvider. With that I was able to write the whole file, but it ended up causing the fading to start in a wrong position (it's more obvious when saving though. Not when playing).
I generated a 10 seconds wav file with Audacity, and used the following method for testing:
private static void PerformFadeOut(string inputPath, string outputPath, bool playNoSave = false)
{
WaveFileReader waveSource = new WaveFileReader(inputPath);
ISampleProvider sampleSource = waveSource.ToSampleProvider();
// Two seconds fade
var fadeOut = new DelayFadeOutSampleProvider(sampleSource);
fadeOut.BeginFadeOut(8000, 2000);
if(playNoSave)
{
// Here the fade is played exactly where expected (#00:08)
var player = new WaveOut();
player.Init(fadeOut);
player.Play();
}
else
{
// But when saving, the fade is applied #00:04!
WaveFileWriter.CreateWaveFile(outputPath, new SampleToWaveProvider(fadeOut));
}
}
Here's the file, before & after writing the fade-out:
As shown above, the fade-out doesn't start at the right position.
After some investigation in the DelayFadeOutSampleProvider, I found a bug in the Read method, so I modified it like this:
public int Read(float[] buffer, int offset, int count)
{
int sourceSamplesRead = source.Read(buffer, offset, count);
lock (lockObject)
{
if (fadeOutDelaySamples > 0)
{
int oldFadeOutDelayPos = fadeOutDelayPosition;
fadeOutDelayPosition += sourceSamplesRead / WaveFormat.Channels;
if (fadeOutDelayPosition > fadeOutDelaySamples)
{
int normalSamples = (fadeOutDelaySamples - oldFadeOutDelayPos) * WaveFormat.Channels;
int fadeOutSamples = (fadeOutDelayPosition - fadeOutDelaySamples) * WaveFormat.Channels;
// apply the fade-out only to the samples after fadeOutDelayPosition
FadeOut(buffer, offset + normalSamples, fadeOutSamples);
fadeOutDelaySamples = 0;
fadeState = FadeState.FadingOut;
return sourceSamplesRead;
}
}
if (fadeState == FadeState.FadingIn)
{
FadeIn(buffer, offset, sourceSamplesRead);
}
else if (fadeState == FadeState.FadingOut)
{
FadeOut(buffer, offset, sourceSamplesRead);
}
else if (fadeState == FadeState.Silence)
{
ClearBuffer(buffer, offset, count);
}
}
return sourceSamplesRead;
}
And now everything works just fine.
Here's my fork of the whole class if someone is interested, and I already asked the author (#mark-heath) to update the original gist with this fix.

You original code was using the original waveSource as input, which is why you had no fade.
The following alternative uses the fadeOut object:
WaveFileWriter.CreateWaveFile16(outputPath, fadeOut);
The signature of CreateWaveFile16 would be:
public static void CreateWaveFile16(string filename, ISampleProvider sourceProvider)
This can be seen in the source code of the class WaveFileWriter.
Another alternative is to use the class SampleToWaveProvider and covert your fadeOut object into an IWaveProvider, and that allows you to use the regular CreateWaveFile method.
WaveFileWriter.CreateWaveFile(outputPath, new SampleToWaveProvider(fadeOut))
As you know, in all cases your output file will only contain the last k seconds corresponding to the fadeout, a different class is needed if you want the whole file with a fadeout.

How can I avoid a memory leak in my .net Application (using VLC)

Im building a digital signage application. I need to show a mix of images and videos (3 images and then a video - then repeat this). I am using a WPF application for this. I ran out of luck using the "MediaElement" in WPF - I had some files that would not play (even the default "wildlife.wmv" file in some situations). I turned to VLC and now my application only runs for ~3 hours before i run out of memory / my vlc player going black.
I have a VLC component wrapped inside a Windows Forms component. The Windows Forms component is then added to my WPF application.
My code is shown below. Im loading this using reflection - I found this using the least amount of memory. Been going about with this for hours.
//code
string path = Assembly.GetExecutingAssembly().Location;
string directory = new System.IO.FileInfo(path).Directory.FullName;
string newPath = System.IO.Path.Combine(directory, "MedianVLCLibrary.dll");
if (System.IO.File.Exists(newPath))
{
Assembly vlcAssembly = Assembly.LoadFrom(newPath);
myVlcType = vlcAssembly.GetType("MedianVLCLibrary.VLCUserControl");
}
else
{
MedianLog.Log.Instance.LogFatal("Could not fild MedianVLCLibrary.dll");
throw new FileNotFoundException(newPath);
}
obj = Activator.CreateInstance(myVlcType);
this.presentationGrid.Children.Add((UIElement)obj); //adding the ui element to the WPF grid
MethodInfo playMethod = myVlcType.GetMethod("Play");
playMethod.Invoke(obj, new object[] { file });
EventInfo completedEvent = myVlcType.GetEvent("PlayCompleted");
Delegate completedDelegate = Delegate.CreateDelegate(completedEvent.EventHandlerType, this, "PlayerCompleted");
completedEvent.AddEventHandler(obj, completedDelegate);
And then im doing my cleanup in my "PlayComplete" method before im invoking a callback method.
obj = null;
myVlcType = null;
vlcAssembly = null;
this.presentationGrid.Children.Clear();
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
FinishedCallback();
I have made a wrapper around VLC, using samples found online. Please see code below.
public partial class VLCUserControl : UserControl, IDisposable
{
AxVLCPlugin2 vlc;
public VLCUserControl()
{
InitializeComponent();
vlc = new AxVLCPlugin2();
vlc.BeginInit();
windowsFormsHost.Child = vlc;
vlc.EndInit();
}
public void Play(string path)
{
var uri = new Uri(path);
var convertedURI = uri.AbsoluteUri;
vlc.playlist.add(convertedURI, null, null);
vlc.playlist.play();
vlc.MediaPlayerEndReached += Vlc_MediaPlayerEndReached;
}
private void Vlc_MediaPlayerEndReached(object sender, EventArgs e)
{
vlc.playlist.items.clear();
vlc.MediaPlayerEndReached -= Vlc_MediaPlayerEndReached;
if (PlayCompleted != null)
{
PlayCompleted();
}
//vlc = null;
GC.Collect();
}
public void Dispose()
{
vlc.Dispose();
}
public event Action PlayCompleted;
}

Multithreading with Windows Store Applications

We are currently creating a Windows Store Application which gains information from an RSS feed and inputs this information into an ObservableCollection. The issue we are having is when the information is being gained, the Applications UI becomes unresponsive.
In order to get around this, I thought about creating a new thread and calling the method within this. Though, after some research we realised that this was no longer possible in Windows Store Apps. How can we get around this?
The method that collects the information is below.
public void getFeed()
{
setupImages();
string[] feedUrls = new string[] {
"http://www.igadgetos.co.uk/blog/category/gadget-news/feed/",
"http://www.igadgetos.co.uk/blog/category/gadget-reviews/feed/",
"http://www.igadgetos.co.uk/blog/category/videos/feed/",
"http://www.igadgetos.co.uk/blog/category/gaming/feed/",
"http://www.igadgetos.co.uk/blog/category/jailbreak-2/feed/",
"http://www.igadgetos.co.uk/blog/category/kickstarter/feed/",
"http://www.igadgetos.co.uk/blog/category/cars-2/feed/",
"http://www.igadgetos.co.uk/blog/category/software/feed/",
"http://www.igadgetos.co.uk/blog/category/updates/feed/"
};
{
try
{
XNamespace dc = "http://purl.org/dc/elements/1.1/";
XNamespace content = "http://purl.org/rss/1.0/modules/content/";
foreach (var feedUrl in feedUrls)
{
var doc = XDocument.Load(feedUrl);
var feed = doc.Descendants("item").Select(c => new ArticleItem() //Creates a copy of the ArticleItem Class.
{
Title = c.Element("title").Value,
//There are another 4 of these.
Post = stripTags(c.Element(content + "encoded").Value) }
).OrderByDescending(c => c.PubDate);
this.moveItems = feed.ToList();
foreach (var item in moveItems)
{
item.ID = feedItems.Count;
feedItems.Add(item);
}
}
lastUpdated = DateTime.Now;
}
catch
{
MessageDialog popup = new MessageDialog("An error has occured downloading the feed, please try again later.");
popup.Commands.Add(new UICommand("Okay"));
popup.Title = "ERROR";
popup.ShowAsync();
}
}
}
How would we be able to cause the Application to not freeze as we gain this information, as Threading is not possible within Windows Store Applications.
E.g - We planned to use;
Thread newThread = new Thread(getFeed);
newThread.Start

You need to use the well documented async pattern for your operations that happen on the UI thread. The link given by Paul-Jan in the comments is where you need to start. http://msdn.microsoft.com/en-us/library/windows/apps/hh994635.aspx