Develop Reference

Capturing data from browser page inspector in C# .NET Core Console Application

My C# .NET Core console application is a simple web crawler. On pages where the needed data is contained in the source code, I am able to access the needed data. In pages where the data can be copied from the window, viewed in the browser's Page Inspector, but NOT in the source code, I'm stuck.
Please provide code examples of how I can acquire this data.
My current capture code is below:
var htmlCode = string.empty;
using (WebClient client = new WebClient()) // WebClient class inherits IDisposable
{
// Get the file content without saving it
htmlCode = client.DownloadString("https://www.wedj.com/dj-photo-video.nsf/firstdance.html");
}
Using the above code, you receive the source code as seen here:
The data shown in image 1, as seen from the browser inspector is hidden inside of
<div class="entry row">

There are few ways to implement what you need (considering a C# console application).
Maybe the easiest one is to use tools that interact with an instance of a browser, i.e. Selenium (used for unit tests).
So:
Install Selenium.WebDriver nuget package
Install a browser where your application will run (let's suppose chrome)
Download the browser driver (chromedriver)
Write something like:
IWebDriver driver = null;
try
{
ChromeOptions options = new ChromeOptions();
options.AddArguments("--incognito");
driver = new ChromeDriver(options);
driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(5);
driver.Url = "https://www.wedj.com/dj-photo-video.nsf/firstdance.html";
var musicTable = driver.FindElement(By.Id("musicTable"));
// interact with driver to get data from the page.
}
finally
{
if (driver != null)
driver.Dispose();
}
Otherwise, you need to investigate a little bit more how the webpage works.
As far as I can see, the page loads a javascript, https://www.wedj.com/dj-photo-video.nsf/musiclist.js, that it is responsible to load the list of music from server.
This js script basically load data from following url: https://www.wedj.com/gbmusic.nsf/musicList?open&wedj=1&list=category_firstdance&count=100 (you can open it also in a browser). Excluding "(" and ")", the result is a json you can parse (maybe using newtonsoft.json package):
{
"more": "yes",
"title": "<h1>Most Requested Wedding First Dance Songs<\/h...",
"event": "<table class='musicTable g6-table-all g6-small' id='musicTable' borde..."
}
The event property contains the data you need (you can use HtmlAgilityPack nuget package to parse it).
Pro Selenium:
easy to interact with
the behavior is the same of what you see by the browser
Cons Selenium:
you need chrome or another browser installed
the browser is running when you interact with it
the browser download the full page (images, html, js, css...)
Pro manual:
you load only what you need
no dependencies to external programs (i.e. browsers)
Cons manual:
you need to understand how html/js works
you need to manually parse the json/html
In this specific case, I prefer the second options.

Read about Selenium Automation tool for C#, but it'll open every web page that you want to scrap and then e.g return source code or perform some actions on that webpage.
Generally this tool is not (afaik) for web crawlers, but can be good at the beginning, especially if your dotnet core app is sitting on some virtual machine / docker.
But care, it may be risky to open not-safe pages via browser.

You might watn to try pupeteer sharp.
It allows you to get the current HTML state.
using (var page = await browser.NewPageAsync())
{
await page.GoToAsync("http://www.spapage.com");
var result = await page.GetContentAsync();
}
https://github.com/kblok/puppeteer-sharp

Debenu PDF works on localhost fails on azure

I'm using Debenu PDF Library to extract pages from a pdf file. Here is how I do it,
var pdflib = new PDFLibrary(LoadDll());
int result = pdflib.UnlockKey("UNLOCKKEYISHERE");
AzureStorageWrapper sasWrapper = new AzureStorageWrapper(ConfigurationManager.AppSettings["AzureStorageConnection"], "cont1");
byte[] blob = sasWrapper.GetBlob("file.pdf");
pdflib.LoadFromString(blob, "");
byte[] page = pdflib.RenderPageToString(50, 1, 1);
When I run this on my local machine it works perfectly (first page of the PDF file comes to the page byte array).
But after publishing this as an Azure Web App page gets a some useless string of bytes.
Can anyone explain to me why this is?

You've hit an Azure App Service sandbox runtime exectution limitation.
From https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#unsupported-frameworks:
Unsupported frameworks
Here is a list of frameworks and scenarios that have been found to be not be usable due to one or more of the restrictions above. It's conceivable that some will be supported in the future as the sandbox evolves.
PDF generators failing due to restriction mentioned above:
EVOPDF
Rotativa
wkhtmltopdf
Syncfusion
Siberix
NReco (uses wkhtmltopdf)
Spire.PDF
Other scenarios:
PhantomJS/Selenium: tries to connect to local address, and also uses GDI+.
It's probably safe to say you can add Debenu PDF to that list.
There is however a mention that some GDI calls may work on Basic/Standard pricing tiers in App Service. It's worth giving that a go if you're currently on Free/Shared.
Alternatives to App Service for running stuff that uses GDI/GDI+:
A Web/Worker Role (Cloud Services)
Service Fabric
IaaS Virtual Machine
Similar question here:
What could cause EvoPDF "unable to render html" exception when deployed to Azure Website

How can I force HTTP-only mode in the C# AWS SDK?

It looks like there used to be an option in the AmazonS3Config class to specify the communication protocol to use, but I don't see that anymore. Where did it go?
I'm trying to do some benchmarking, and one of the things I want to test is HTTP vs HTTPS. Since a lot of our objects are fairly small (less than 512k), I'm wondering if the HTTPS handshaking is contributing to our slowness in uploading.

The configuration is now called UseHttp. It's in the 'Amazon.Runtime.ClientConfig' class that is the base for all of the client-specific config classes.
var config = new AmazonS3Config { UseHttp = true, ... };
var client = new AmazonS3Client(config);
This works for all clients (unless they don't support http. You can still set it, but it will be ignored).

Unable to seek/replay video (.ogv) files served from http server [C#]

ISSUE:
Video player (native player in webkit based browser) is not giving option to replay and I cannot seek video while playing it. Once played, I have to refresh page to play it again.
If I give a web url for ogv video in my html page, same player works fine (replays and seek is successful).
Hence I suspect issue with my algorithm of serving files (especially .ogv files).
What I am doing:
I have an HTTP fileserver (completely in C#, using .Net framework classes - HttpListenerContext, HttpListenerRequest, HttpListenerResponse, etc). This server is serving files (of all types and extensions - text, video, audio, images) to clients. All files are available on server which are requested by clients.
Algorithm to serve files from server:
Server gets the file name from URL.
Reads the complete file in a buffer(byte[] array) (using File.ReadAllBytes(file_path_on_server)),
Array is assigned to HttpListenerResponse.OutputStream to return the file (contents) to client.
Corresponding HttpListenerResponse.ContentType (MimeType) and HttpListenerResponse.ContentLength64 is assigned (size of array).
Response header is added as "no-cache".
HttpListenerResponse.ContentEncoding is UTF8.
Close response object to send it to client.
This works fine and files are reaching clients for processing including video files (.ogv) but video is not seekable/replayable in player.
Already tried
Tried removing assignment of HttpListenerResponse.ContentEncoding to UTF8.

It seems that you server returns the file the same way as if it's just downloaded, so seek/replay most likely will not be available until the whole file is loaded. If you want to support true video streaming you need another server-side implementation. Live Streaming within IIS Media Services may help you to support Apple HLS Streaming or similar (depending on your player), also i saw somewhere an IIS extension/module that supports http pseudo streaming with seeking for some video formats.

Creating a ASP.NET application converting text to speech

I seek some insight in creating an application that converts text to speech in ASP.NET. From my initial research, it appears that:
MS SAPI requires the client to download an ActiveX component and can support large amounts of text to be converted. Our clients are not willing to install any components on their systems, so this approach may or may not fly.
I do understand with .NET 3.0, we have the System.Speech.Synthesis namespace. Does the conversion take place on the server? If so, how would I serve it to the client?
Our requirements are ability to convert large amount of text, should be scalable and reliable. Which technology is "production ready" capable of serving a large number of requests in a short time interval.
Any thoughts are appreciated.

By default, ASP.Net applications don't run with sufficient permissions to access Speech Synthesis, and attempting to run Larsenal's code will fail with a security error.
I was able to get around this in an app by having a separate WCF service running on the server, as a regular Windows Service. The ASP.Net application then communicated with that service. That service just wrapped Larsenal's code, returning an array of bytes, given a string of text.
Also, one megabyte of text? That's a good-sized novel.
Edit, 11-12-09, answering some comments:
System.Speech can either return an array of bytes, or save to a wav file, which you can then feed to a media player embedded on the user's page. When I built my talking web page, it worked like this:
1) Page.aspx includes an 'embed' tag that puts a Windows Media Player on the page. The source is "PlayText.aspx?Textid=whatever".
2) PlayText.aspx loads the appropriate text, and communicates (via WCF) to the speechreader service, handing it the text to read.
3) The Speechreader service creates a MemoryStream and calls SpeechSynthesiser.SetOutputToWaveStream, and then returns the stream as a single array of bytes. This array is Response.Write()-ed to the client.
Here's the meat of the SpeechReader service:
byte[] ITextReader.SpeakText(string text)
{
using (SpeechSynthesizer s = new SpeechSynthesizer())
{
using (MemoryStream ms = new MemoryStream())
{
s.SetOutputToWaveStream(ms);
s.Speak(text);
return ms.GetBuffer();
}
}
}
I'm pretty sure that on the back end, this returns an enormous XML array-of-bytes, and is horribly inefficient. I just did it as a proof of concept, and so didn't research that. If you intend to use this in production, make sure that it's not internally returning something like this:
<byte>23</byte>
<byte>42</byte>
<byte>117</byte>
...

With the SpeechSynthesizer, you can output to a WAV file. You could then have a secondary process compress or convert to another format if needed. All this could be done on the server and then sent up through the browser.
This CodeProject article is a good introduction to .NET Speech Synthesis.
If you want to see how it performs with a LOT of text.... Add a reference to System.Speech and then use the following as a starting point:
using System;
using System.Speech.Synthesis;
namespace SpeakToMe
{
class Program
{
static void Main(string[] args)
{
SpeechSynthesizer synth = new SpeechSynthesizer();
synth.SetOutputToWaveFile("c:\\test.wav");
synth.Speak("Hello, world.");
synth.SetOutputToDefaultAudioDevice();
Console.ReadLine();
}
}
}
A quick test on a file of 44,700 words (238KB) on my relatively fast machine...
Completed in 55 seconds
Generated a 626 MB WAV file

I searched for "Convert Text Into Speech In Asp.Net" in Google and found a very nice and usefull link:
http://codeprojectdownload.com/asp-net-2/convert-text-into-speech-in-asp-net/#.T0ScXIfXDZE
It may also be useful to you.

I achieved this by using codeBehind to run a javascript function that runs the text-to-speech command:
codeBehind:
Page.ClientScript.RegisterStartupScript(
GetType(),
"anythingHere",
"nameOfFunction();",
true);
javascript:
<script>
function nameOfFunction()
{//start
var msg = new SpeechSynthesisUtterance('READ ME!');
window.speechSynthesis.speak(msg);
}//end
</script>

I wrote an article on this on my blog: http://weblogs.asp.net/ricardoperes/archive/2014/04/08/speech-synthesis-with-asp-net-and-html5.aspx. I used AJAX and Data URIs to send voice data back and forth between the client and the server.