C# console application code doesn't execute after await - c#

I'm trying to make a webscraper where I get all the download links for the css/js/images from a html file.
Problem
The first breakpoint does hit, but the second one not after hitting "Continue".
Picture in Visual Studio
Code I'm talking about:
private static async void GetHtml(string url, string downloadDir)
{
//Get html data, create and load htmldocument
HttpClient httpClient = new HttpClient();
//This code gets executed
var html = await httpClient.GetStringAsync(url);
//This code not
Console.ReadLine();
var htmlDocument = new HtmlDocument();
htmlDocument.LoadHtml(html);
//Get all css download urls
var linkUrl = htmlDocument.DocumentNode.Descendants("link")
.Where(node => node.GetAttributeValue("type", "")
.Equals("text/css"))
.Select(node=>node.GetAttributeValue("href",""))
.ToList();
//Downloading css, js, images and source code
using (var client = new WebClient())
{
for (var i = 0; i <scriptUrl.Count; i++)
{
Uri uri = new Uri(scriptUrl[i]);
client.DownloadFile(uri,
downloadDir + #"\js\" + uri.Segments.Last());
}
}
Edit
Im calling the getHtml method from here:
private static void Start()
{
//Create a list that will hold the names of all the subpages
List<string> subpagesList = new List<string>();
//Ask user for url and asign that to var url, also add the url to the url list
Console.WriteLine("Geef url van de website:");
string url = "https://www.hethwc.nl";
//Ask user for download directory and assign that to var downloadDir
Console.WriteLine("Geef locatie voor download:");
var downloadDir = #"C:\Users\Daniel\Google Drive\Almere\C# II\Download tests\hethwc\";
//Download and save the index file
var htmlSource = new System.Net.WebClient().DownloadString(url);
System.IO.File.WriteAllText(#"C:\Users\Daniel\Google Drive\Almere\C# II\Download tests\hethwc\index.html", htmlSource);
// Creating directories
string jsDirectory = System.IO.Path.Combine(downloadDir, "js");
string cssDirectory = System.IO.Path.Combine(downloadDir, "css");
string imagesDirectory = System.IO.Path.Combine(downloadDir, "images");
System.IO.Directory.CreateDirectory(jsDirectory);
System.IO.Directory.CreateDirectory(cssDirectory);
System.IO.Directory.CreateDirectory(imagesDirectory);
GetHtml("https://www.hethwc.nu", downloadDir);
}

How are you calling GetHtml? Presumably that is from a sync Main method, and you don't have any other non-worker thread in play (because your main thread exited): the process will terminate. Something like:
static void Main() {
GetHtml();
}
The above will terminate the process immediately after GetHtml returns and the Main method ends, which will be at the first incomplete await point.
In current C# versions (C# 7.1 onwards) you can create an async Task Main() method, which will allow you to await your GetHtml method properly, as long as you change GetHtml to return Task:
async static Task Main() {
await GetHtml();
}

Related

Copy and paste the link from clipboard in selenium C#

I'm having the following issue:
I'm saving the locators as
public static By CopyUrl = By.XPath("//a[#data-key='UrlLink']");
In the test, I cannot GetText(), or store the whole CopyUrl in a var, since it's void. Driver.Click(XPath.MainMenu.Tabs.CopyUrl);
How can I store the link which is saved in clipboard after I click on it, and paste it in a new tab.
I tried:
var elem =Driver.SwitchTo().NewWindow(WindowType.Tab).Navigate().GoToUrl("myCopiedUrl").ToString();
Driver.SwitchTo().NewWindow(WindowType.Tab).Navigate().GoToUrl(elem);
It is not working for you since both Click() and Navigate().GoToUrl() methods are returning void.
In order to get the current URL you should use this method:
String currentURL = driver.Url;
or
String currentURL = driver.getCurrentUrl();
or
String currentURL = driver.getLocation();
[Test, Order(01234)]
public void t_Document_CopyUrl_Downloaded()
{
NavigateToYourPage();
UploadNewDocument();
NavigateToTheSpecificTab();
//Click on "Copy URL"
string fileName = Driver.FindElement(XPath.Input).GetValue();
IWebElement copyUrl = Driver.FindElement(XPath.CopyUrl);
string currentUrl = copyUrl.GetAttribute("textContent");
Driver.Click(XPath.CopyUrl);
Driver.WaitForNoSpinner();
// Opens a new tab and switches to new tab
Driver.SwitchTo().NewWindow(WindowType.Tab);
Driver.Navigate().GoToUrl(currentUrl);
Driver.Navigate().GoToUrl("chrome://downloads/");
string downloadedFileName = Driver.ExecuteJavaScript<string>("return document.querySelector('downloads-manager').shadowRoot.querySelector('downloads-item').shadowRoot.querySelector('#name').textContent;");
Driver.SwitchTo().Window(Driver.WindowHandles[0]);
Assert.IsTrue(downloadedFileName.StartsWith(fileName));
}

Cancel Site Crawl on Abot

I have a list of domains that are crawled using Abot, the aim is that when it finds an amazon link on one of the sites, it quits, then moves onto the next site. But I can't seem to see who I can exit the page crawl. e.g.
https://github.com/sjdirect/abot
static Main(string[] args)
{
var domains= new List<string> { "http://domain1", "http://domain2" };
foreach (string domain in domains)
{
var config = new CrawlConfiguration
{
MaxPagesToCrawl = 100,
MinCrawlDelayPerDomainMilliSeconds = 3000
};
var crawler = new PoliteWebCrawler(config);
crawler.PageCrawlCompleted += PageCrawlCompleted;
var uri = new Uri(domain);
var crawlResult = crawler.Crawl(uri);
}
}
private static void PageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
{
var crawledPage = e.CrawledPage;
var crawlContext = e.CrawlContext;
var document = crawledPage.AngleSharpHtmlDocument;
var anchors = document.QuerySelectorAll("a").OfType<IHtmlAnchorElement>();
var hrefs = anchors.Select(x => x.Href).ToList();
var regEx= new Regex(#"https?:\/\/(www|smile)\.amazon(\.co\.uk|\.com).*");
var resultList = hrefs.Where(f => regEx.IsMatch(f)).ToList();
if (resultList.Any())
{
//NEED TO EXIT THE SITE CRAWL HERE
}
}
I would suggest the following...
var myCancellationToken = new CancellationTokenSource();
crawler.CrawlAsync(someUri, myCancellationToken);
private static void PageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
{
//More performant (since the parsing has already been done by Abot)
var hasAmazonLinks = e.CrawledPage.ParsedLinks
.Any(hl => hl.HrefValue.AbsoluteUri
.ToLower()
.Contains("amazon.com"));
if (hasAmazonLinks)
{
//LOG SOMETHING BEFORE YOU STOP THE CRAWL!!!!!
//Option A: Preferred method, Will clear all scheduled pages and cancel any threads that are currently crawling
myCancellationToken.Cancel();
//Option B: Same result as option A but no need to do anything with tokens. Not the preferred method.
e.CrawlContext.IsCrawlHardStopRequested = true;
//Option C: Will clear all scheduled pages but will allow any threads that are currently crawling to complete. No cancellation tokens needed. Consider it a soft stop to the crawl.
e.CrawlContext.IsCrawlStopRequested = true;
}
}
PoliteWebCrawler is designed to start crawling and dig deeper into the website URLs. If you just want to get the content of a URL (for example first page of a website) you can use PageRequester which is designed for such jobs.
var pageRequester = new PageRequester(new CrawlConfiguration(), new WebContentExtractor());
var crawledPage = await pageRequester.MakeRequestAsync(new Uri("http://google.com"));
Log.Logger.Information("{result}", new
{
url = crawledPage.Uri,
status = Convert.ToInt32(crawledPage.HttpResponseMessage.StatusCode)
});
By the way, if you want to stop a crawler during the process, you can use one of these two methods:
//1. hard crawl stop
crawlContext.CancellationTokenSource.Cancel();
//2. soft stop
crawlContext.IsCrawlStopRequested = true;

Extent Reports version 3.0.2 - AppendExisting

Below is the code I am trying to use to append all tests to a single report. However, latest test is replacing all the older test reports. So, it's not appending to a single report for some reason. Can you please help me out here?
var htmlReporter = new ExtentHtmlReporter(ResourcesConfig.ReportPath);
extent = new ExtentReports();
extent.AttachReporter(htmlReporter);
htmlReporter.LoadConfig(ResourcesConfig.ReportXMLPath);
**htmlReporter.AppendExisting = true;**
I had a lot of trouble with this as well as the documentation doesn't explain much. I have one method called ReportCreation which runs for every test case and in that method i have the following:
public static ExtentReports ReportCreation(){
System.out.println(extent);
if (extent == null) {
extent = new ExtentReports();
htmlReports = new ExtentHtmlReporter(fileName+ n + "\\extentReportFile.html");
htmlReports.config().setReportName("Pre release Smoke test");
htmlReports.config().setTheme(Theme.STANDARD);
htmlReports.config().setTestViewChartLocation(ChartLocation.BOTTOM);
extent.attachReporter(htmlReports);
}
else {
htmlReports = new ExtentHtmlReporter(fileName+ n+ "\\extentReportFile.html");
htmlReports.setAppendExisting(true);
extent.attachReporter(htmlReports);
}
return extent;
}
So when the first unit test is run, it will create the html report, but the second unit test will see that the report has already been generated and so use the existing one.
I have created a random number generator so that it goes to a different report on every run
public static Random rand = new Random();
public static int n = rand.nextInt(10000)+1;
I was facing the same issue. My solution was using .NET Core so ExtentReports 3 and 4 were not supported.
Instead, I wrote code to merge the results from previous html file to the new html file.
This is the code I used:
public static void GenerateReport()
{
// Publish test results to extentnew.html file
extent.Flush();
if (!File.Exists(extentConsolidated))
{
// Rename extentnew.html to extentconsolidated.html after execution of 1st batch
File.Move(extentLatest, extentConsolidated);
}
else
{
// Append test results to extentconsolidated.html from 2nd batch onwards
_ = AppendExtentHtml();
}
}
public static async Task AppendExtentHtml()
{
var htmlconsolidated = File.ReadAllText(extentConsolidated);
var htmlnew = File.ReadAllText(extentLatest);
var config = Configuration.Default;
var context = BrowsingContext.New(config);
var newdoc = await context.OpenAsync(req => req.Content(htmlnew));
var newlis = newdoc.QuerySelector(#"ul.test-list-item");
var consolidateddoc = await context.OpenAsync(req => req.Content(htmlconsolidated));
var consolidatedlis = consolidateddoc.QuerySelector(#"ul.test-list-item");
foreach (var li in newlis.Children)
{
li.RemoveFromParent();
consolidatedlis.AppendElement(li);
}
File.WriteAllText(extentConsolidated, consolidateddoc.DocumentElement.OuterHtml);
}
This logic bypasses any Extent Report reference and treats the result file as any other html.
Hope this helps.

How to get list of files from a specific GitHub repo given a link in C#?

How can I get a list of file from GitHub link?
For example, from this GitHub repository link: https://github.com/crs2007/ActiveReport/tree/master/ActiveReport/SQLFiles
We can see that there are SQL text files:
I would like to get a list of these files:
CorruptionIssues.sql
DBAlert.sql
DataBaseInfo.sql
DatabaseDefaultLogin.sql
DatabaseFiles.sql
Deadlock.sql
DefaultTrace.sql
DiskInfo.sql
InstalledPrograms.sql
.etc...sql
How can I do that?
You should be able to use GitHub Contents API
By making a request like:
curl https://api.github.com/repos/crs2007/ActiveReport/contents/ActiveReport
Github will return JSON containing directory contents.
You can do this in C# in multiple ways, using something like Octokit is probably recommended as they ironed out most issues you're likely to encounter. In case you can't use an external library, the example below shows how to use plain HttpClient to achieve the same, albeit with a lot more plumbing involved:
class Program
{
static void Main()
{
Task.Factory.StartNew(async () =>
{
var repoOwner = "crs2007";
var repoName = "ActiveReport";
var path = "ActiveReport";
var httpClientResults = await ListContents(repoOwner, repoName, path);
PrintResults("From HttpClient", httpClientResults);
var octokitResults = await ListContentsOctokit(repoOwner, repoName, path);
PrintResults("From Octokit", octokitResults);
}).Wait();
Console.ReadKey();
}
static async Task<IEnumerable<string>> ListContents(string repoOwner, string repoName, string path)
{
using (var client = GetGithubHttpClient())
{
var resp = await client.GetAsync($"repos/{repoOwner}/{repoName}/contents/{path}");
var bodyString = await resp.Content.ReadAsStringAsync();
var bodyJson = JToken.Parse(bodyString);
return bodyJson.SelectTokens("$.[*].name").Select(token => token.Value<string>());
}
}
static async Task<IEnumerable<string>> ListContentsOctokit(string repoOwner, string repoName, string path)
{
var client = new GitHubClient(new ProductHeaderValue("Github-API-Test"));
// client.Credentials = ... // Set credentials here, otherwise harsh rate limits apply.
var contents = await client.Repository.Content.GetAllContents(repoOwner, repoName, path);
return contents.Select(content => content.Name);
}
private static HttpClient GetGithubHttpClient()
{
return new HttpClient
{
BaseAddress = new Uri("https://api.github.com"),
DefaultRequestHeaders =
{
// NOTE: You'll have to set up Authentication tokens in real use scenario
// NOTE: as without it you're subject to harsh rate limits.
{"User-Agent", "Github-API-Test"}
}
};
}
static void PrintResults(string source, IEnumerable<string> files)
{
Console.WriteLine(source);
foreach (var file in files)
{
Console.WriteLine($" -{file}");
}
}
}

Window Phone 8.1: Confusing about file location and writing data

I'm trying to read and write data with json file.
I created some class.
public class SimpleTask{...}
public class DayTask{...}
public class DataModel
{
...
private async Task GetSimpleTaskAsync()
{
if (_daytask.Count != 0)
return;
string fileName = "a.json";
Uri appUri = new Uri("ms-appx:///"+ fileName);
StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(appUri);
string jsonText = await FileIO.ReadTextAsync(file);
JsonObject jsonObject = JsonObject.Parse(jsonText);
JsonArray jsonArray = jsonObject["DayTasks"].GetArray();
foreach (JsonValue daytaskValue in jsonArray)
{
JsonObject daytaskObject = daytaskValue.GetObject();
ObservableCollection<SimpleTask> simpletask = new ObservableCollection<SimpleTask>();
foreach (JsonValue simpletaskValue in daytaskObject["Tasks"].GetArray())
{
JsonObject simpletaskObject = simpletaskValue.GetObject();
simpletask.Add(new SimpleTask( simpletaskObject["StartTime"].GetString(),
simpletaskObject["EndTime"].GetString(),
simpletaskObject["Description"].GetString()));
}
DayTask daytask = new DayTask(daytaskObject["Day"].GetString(),simpletask);
this.DayTasks.Add(daytask);
}
}
}
As you can see, i have a method that gets data form a.json file. I created a.json file:
In the MainPage.xaml.cs, there is a method which calls GetDayTaskAysnc() method and retrieves data :
private async void ReadData1(object sender, TappedRoutedEventArgs e)
{
string test = String.Empty;
var daytask = await DataModel.GetDayTaskAsync();
foreach (var tasks in daytask)
{
test += String.Format("Day:{0}:\n", tasks.Day);
foreach (var simpletask in tasks.Tasks)
{
test += String.Format("\tStart Time: {0}\n", simpletask.StartTime);
test += String.Format("\tEnd Time: {0}\n", simpletask.EndTime);
test += String.Format("\tDescription Time: {0}\n", simpletask.Description);
}
}
TextBlock.Text = test;
}
It worked fine ! But i want to write data to the same file, so i added data in hardcore way:
private List<DayTask> creatList()
{
List<DayTask> DayTasks = new List<DayTask>();
ObservableCollection<SimpleTask> simpletask1 = new ObservableCollection<SimpleTask>();
simpletask1.Add(new SimpleTask("6AM","7AM","Breakfast"));
simpletask1.Add(new SimpleTask("8AM", "9AM", "Game"));
ObservableCollection<SimpleTask> simpletask2 = new ObservableCollection<SimpleTask>();
simpletask2.Add(new SimpleTask("6AM", "7AM", "Sleep"));
simpletask2.Add(new SimpleTask("8AM", "9AM", "School"));
DayTasks.Add(new DayTask ("3/8/2014",simpletask1));
DayTasks.Add(new DayTask("4/8/2014", simpletask2));
return DayTasks;
}
private async void WriteData(object sender, TappedRoutedEventArgs e)
{
string json = "a.json";
List<DayTask> daytasks = creatList();
var serializer = new DataContractJsonSerializer(typeof(List<DayTask>));
var stream = await ApplicationData.Current.LocalFolder.OpenStreamForWriteAsync(json, CreationCollisionOption.ReplaceExisting);
using (stream)
{
serializer.WriteObject(stream, daytasks);
}
TextBlock.Text = "Write to Json file succeeded";
}
When i ran my app with window phone emulator, Firstly, it wrote to the file. Then i clicked read data button to ensure data written correctly, the emulator showed data from a.json file without being modified by WriteData() method. I continued to creat the second read data method:
private async void ReadData2(object sender, TappedRoutedEventArgs e)
{
string test = String.Empty;
string json = "a.json";
string content = String.Empty;
List<DayTask> mytasks = new List<DayTask>();
var deserializer = new DataContractJsonSerializer(typeof(List<DayTask>));
var stream = await ApplicationData.Current.LocalFolder.OpenStreamForReadAsync(json);
using (stream)
{
mytasks = (List<DayTask>)deserializer.ReadObject(stream);
}
foreach (var tasks in mytasks)
{
test += String.Format("Day:{0}:\n", tasks.Day);
foreach (var simpletask in tasks.Tasks)
{
test += String.Format("\tStart Time: {0}\n", simpletask.StartTime);
test += String.Format("\tEnd Time: {0}\n", simpletask.EndTime);
test += String.Format("\tDescription Time: {0}\n", simpletask.Description);
}
}
TextBlock.Text = test;
}
I deployed my app several times, and this is my result:
ReadData2() : 'System.IO.FileNotFoundException'
WriteData() -> ReadData1(): Data from a.json was shown
WriteData() -> ReadData2(): Data from creatList() was shown
WriteData() -> ReadData1(): Data from a.json was shown -> ReadData2(): Data from creatList() was shown
So that, i have some question:
Do i have 2 json files, one i created by adding into my project and the other one i created when ran WriteData() method ? What is their paths ?
If my file is data.json in DataSource folder, how can i write data to it ? I can read data from it using uri like GetSimpleTaskAsync() but i don't know how to write to it correctly. (I tried to convert object into string to write but can't read it again, i guess i wrote it in wrong way)
Sorry for my long post and my bad english :) Thank you very much
But i want to write data to the same file, so i added data in hardcore way:
Your are making confusion between ms-appx:/// and ms-appdata:/// folders (or ApplicationData.Current.LocalFolder )
The ms-appx folder is read-only. You can't write to it. (or you could edit your app code without passing through the certification process)
The file you wrote must be into the ms-appdata folder.

Categories

Resources