Develop Reference

Scraping html list data from a dynamic server

Hallo guys!
Sorry for the dump question, this is my last resort. I swear i triend countless of other Stackoverflow questions, different Frameworks, etc., but those didnt seem to help.
Ich have the following Problem:
A website displays a list of data (there is a TON of div, li, span etc. tags infront, its a big HTML.)
Im writing a tool that fetches data from a specific list inside a ton of other div tags, downloads it and outputs an excel file.
The website im trying to access, is dynamic. So you open the website, it loads a little bit, and then the list appears (probably some JS and stuff).
When i try to download the website via a webRequest in C#, the html I get ist almost empty with a ton on white spaces, lots of non-html stuff, some garbage data as well.
Now: Im pretty used to C#, HTMLAgillityPack, and countless other libraries, not so much in web related stuff tho. I tried CefSharp, Chromium etc. all of those stuff, but couldnt get them to work properly unfortunately.
I want to have a HTML in my program to work with that looks exactly like the HTML that you see when
you open the dev console in chrome wenn visting the website mentined above.
The HTML parser works flwalessly there.
This is how I image how the code could look like simplified.
Extreme C# pseudocode:
WebBrowserEngine web = new WebBrowserEngine()
web.LoadURLuntilFinished(url); // with all the JS executed and stuff
String html = web.getHTML();
web.close();
My Goal would be that the string html in the pseudocode looks exactly like the one in the Chrome dev tab.
Maybe there is a solution posted somewhere else but i swear i coudlnt find it, been looking for days.
Andy help is greatly appreciated.

#SpencerBench is spot on in saying
It could be that the page is using some combination of scroll state, element visibility, or element positions to trigger content loading. If that's the case, then you'll need to figure out what it is and trigger it programmatically.
To answer the question for your specific use case, we need to understand the behaviour of the page you want to scrape data from, or as I asked in the comments, how do you know the page is "finished"?
However, it's possible to give a fairly generic answer to the question which should act as a starting point for you.
This answer uses Selenium, a package which is commonly used for automating testing of web UIs, but as they say on their home page, that's not the only thing it can be used for.
Primarily it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should) also be automated as well.
The web site I'm scraping
So first we need a web site. I've created one using ASP.net core MVC with .net core 3.1, although the web site's technology stack isn't important, it's the behaviour of the page you want to scrape which is important. This site has 2 pages, unimaginatively called Page1 and Page2.
Page controllers
There's nothing special in these controllers:
namespace StackOverflow68925623Website.Controllers
{
using Microsoft.AspNetCore.Mvc;
public class Page1Controller : Controller
{
public IActionResult Index()
{
return View("Page1");
}
}
}
namespace StackOverflow68925623Website.Controllers
{
using Microsoft.AspNetCore.Mvc;
public class Page2Controller : Controller
{
public IActionResult Index()
{
return View("Page2");
}
}
}
API controller
There's also an API controller (i.e. it returns data rather than a view) which the views can call asynchronously to get some data to display. This one just creates an array of the requested number of random strings.
namespace StackOverflow68925623Website.Controllers
{
using Microsoft.AspNetCore.Mvc;
using System;
using System.Collections.Generic;
using System.Text;
[Route("api/[controller]")]
[ApiController]
public class DataController : ControllerBase
{
[HttpGet("Create")]
public IActionResult Create(int numberOfElements)
{
var response = new List<string>();
for (var i = 0; i < numberOfElements; i++)
{
response.Add(RandomString(10));
}
return Ok(response);
}
private string RandomString(int length)
{
var sb = new StringBuilder();
var random = new Random();
for (var i = 0; i < length; i++)
{
var characterCode = random.Next(65, 90); // A-Z
sb.Append((char)characterCode);
}
return sb.ToString();
}
}
}
Views
Page1's view looks like this:
#{
ViewData["Title"] = "Page 1";
}
<div class="text-center">
<div id="list" />
<script src="~/lib/jquery/dist/jquery.min.js"></script>
<script>
var apiUrl = 'https://localhost:44394/api/Data/Create';
$(document).ready(function () {
$('#list').append('<li id="loading">Loading...</li>');
$.ajax({
url: apiUrl + '?numberOfElements=20000',
datatype: 'json',
success: function (data) {
$('#loading').remove();
var insert = ''
for (var item of data) {
insert += '<li>' + item + '</li>';
}
insert = '<ul id="results">' + insert + '</ul>';
$('#list').html(insert);
},
error: function (xht, status) {
alert('Error: ' + status);
}
});
});
</script>
</div>
So when the page first loads, it just contains an empty div called list, however the page loading trigger's the function passed to jQuery's $(document).ready function, which makes an asynchronous call to the API controller, requesting an array of 20,000 elements. While the call is in progress, "Loading..." is displayed on the screen, and when the call returns, this is replaced by an unordered list containing the received data. This is written in a way intended to be friendly to developers of automated UI tests, or of screen scrapers, because we can tell whether all the data has loaded by testing whether or not the page contains an element with the ID results.
Page2's view looks like this:
#{
ViewData["Title"] = "Page 2";
}
<div class="text-center">
<div id="list">
<ul id="results" />
</div>
<script src="~/lib/jquery/dist/jquery.min.js"></script>
<script>
var apiUrl = 'https://localhost:44394/api/Data/Create';
var requestCount = 0;
var maxRequests = 20;
$(document).ready(function () {
getData();
});
function getDataIfAtBottomOfPage() {
console.log("scroll - " + requestCount + " requests");
if (requestCount < maxRequests) {
console.log("scrollTop " + document.documentElement.scrollTop + " scrollHeight " + document.documentElement.scrollHeight);
if (document.documentElement.scrollTop > (document.documentElement.scrollHeight - window.innerHeight - 100)) {
getData();
}
}
}
function getData() {
window.onscroll = undefined;
requestCount++;
$('results2').append('<li id="loading">Loading...</li>');
$.ajax({
url: apiUrl + '?numberOfElements=50',
datatype: 'json',
success: function (data) {
var insert = ''
for (var item of data) {
insert += '<li>' + item + '</li>';
}
$('#loading').remove();
$('#results').append(insert);
if (requestCount < maxRequests) {
window.setTimeout(function () { window.onscroll = getDataIfAtBottomOfPage }, 1000);
} else {
$('#results').append('<li>That\'s all folks');
}
},
error: function (xht, status) {
alert('Error: ' + status);
}
});
}
</script>
</div>
This gives a nicer user experience because it requests data from the API controller in multiple smaller chunks, so the first chunk of data appears fairly quickly, and once the user has scrolled down to somewhere near the bottom of the page, the next chunk of data is requested, until 20 chunks have been requested and displayed, at which point the text "That's all folks" is added to the end of the unordered list. However this is more difficult to interact with programmatically because you need to scroll the page down to make the new data appear.
(Yes, this implementation is a bit buggy - if the user gets to the bottom of the page too quickly then requesting the next chunk of data doesn't happen until they scroll up a bit. But the question isn't about how to implement this behaviour in a web page, but about how to scrape the displayed data, so please forgive my bugs.)
The scraper
I've implemented the scraper as a xUnit unit test project, just because I'm not doing anything with the data I've scraped from the web site other than Asserting that it is of the correct length, and therefore proving that I haven't prematurely assumed that the web page I'm scraping from is "finished". You can put most of this code (other than the Asserts) into any type of project.
Having created your scraper project, you need to add the Selenium.WebDriver and Selenium.WebDriver.ChromeDriver nuget packages.
Page Object Model
I'm using the Page Object Model pattern to provide a layer of abstraction between functional interaction with the page and the implementation detail of how to code that interaction. Each of the pages in the web site has a corresponding page model class for interacting with that page.
First, a base class with some code which is common to more than one page model class.
namespace StackOverflow68925623Scraper
{
using System;
using OpenQA.Selenium;
using OpenQA.Selenium.Support.UI;
public class PageModel
{
protected PageModel(IWebDriver driver)
{
this.Driver = driver;
}
protected IWebDriver Driver { get; }
public void ScrollToTop()
{
var js = (IJavaScriptExecutor)this.Driver;
js.ExecuteScript("window.scrollTo(0, 0)");
}
public void ScrollToBottom()
{
var js = (IJavaScriptExecutor)this.Driver;
js.ExecuteScript("window.scrollTo(0, document.body.scrollHeight)");
}
protected IWebElement GetById(string id)
{
try
{
return this.Driver.FindElement(By.Id(id));
}
catch (NoSuchElementException)
{
return null;
}
}
protected IWebElement AwaitGetById(string id)
{
var wait = new WebDriverWait(Driver, TimeSpan.FromSeconds(10));
return wait.Until(e => e.FindElement(By.Id(id)));
}
}
}
This base class gives us 4 convenience methods:
Scroll to the top of the page
Scroll to the bottom of the page
Get the element with the supplied ID, or return null if it doesn't exist
Get the element with the supplied ID, or wait for up to 10 seconds for it to appear if it doesn't exist yet
And each page in the web site has its own model class, derived from that base class.
namespace StackOverflow68925623Scraper
{
using OpenQA.Selenium;
public class Page1Model : PageModel
{
public Page1Model(IWebDriver driver) : base(driver)
{
}
public IWebElement AwaitResults => this.AwaitGetById("results");
public void Navigate()
{
this.Driver.Navigate().GoToUrl("https://localhost:44394/Page1");
}
}
}
namespace StackOverflow68925623Scraper
{
using OpenQA.Selenium;
public class Page2Model : PageModel
{
public Page2Model(IWebDriver driver) : base(driver)
{
}
public IWebElement Results => this.GetById("results");
public void Navigate()
{
this.Driver.Navigate().GoToUrl("https://localhost:44394/Page2");
}
}
}
And the Scraper class:
namespace StackOverflow68925623Scraper
{
using OpenQA.Selenium.Chrome;
using System;
using System.Threading;
using Xunit;
public class Scraper
{
[Fact]
public void TestPage1()
{
// Arrange
var driver = new ChromeDriver();
var page = new Page1Model(driver);
page.Navigate();
try
{
// Act
var actualResults = page.AwaitResults.Text.Split(Environment.NewLine);
// Assert
Assert.Equal(20000, actualResults.Length);
}
finally
{
// Ensure the browser window closes even if things go pear-shaped
driver.Quit();
}
}
[Fact]
public void TestPage2()
{
// Arrange
var driver = new ChromeDriver();
var page = new Page2Model(driver);
page.Navigate();
try
{
// Act
while (!page.Results.Text.Contains("That's all folks"))
{
Thread.Sleep(1000);
page.ScrollToBottom();
page.ScrollToTop();
}
var actualResults = page.Results.Text.Split(Environment.NewLine);
// Assert - we expect 1001 because of the extra "that's all folks"
Assert.Equal(1001, actualResults.Length);
}
finally
{
// Ensure the browser window closes even if things go pear-shaped
driver.Quit();
}
}
}
}
So, what's happening here?
// Arrange
var driver = new ChromeDriver();
var page = new Page1Model(driver);
page.Navigate();
ChromeDriver is in the Selenium.WebDriver.ChromeDriver package and implements the IWebDriver interface from the Selenium.WebDriver package with the code to interact with the Chrome browser. Other packages are available containing implementations for all popular browsers. Instantiating the driver object opens a browser window, and calling its Navigate method directs the browser to the page we want to test/scrape.
// Act
var actualResults = page.AwaitResults.Text.Split(Environment.NewLine);
Because on Page1, the results element doesn't exist until all the data has been displayed, and no user interaction is required in order for it to be displayed, we use the page model's AwaitResults property to just wait for that element to appear and return it once it has appeared.
AwaitResults returns an IWebElement instance representing the element, which in turn has various methods and properties we can use to interact with the element. In this case we use its Text property which returns the element's contents as a string, without any markup. Because the data is displayed as an unordered list, each element in the list is delimited by a line break, so we can can use String's Split method to convert it to a string array.
Page2 needs a different approach - we can't use the presence of the results element to determine whether the data has all been displayed, because that element is on the page right from the start, instead we need to check for the string "That's all folks" which is written right at the end of the last chunk of data. Also the data isn't loaded all in one go, and we need to keep scrolling down in order to trigger the loading of the next chunk of data.
// Act
while (!page.Results.Text.Contains("That's all folks"))
{
Thread.Sleep(1000);
page.ScrollToBottom();
page.ScrollToTop();
}
var actualResults = page.Results.Text.Split(Environment.NewLine);
Because of the bug in the UI that I mentioned earlier, if we get to the bottom of the page too quickly, the fetch of the next chunk of data isn't triggered, and attempting to scroll down when already at the bottom of the page doesn't raise another scroll event. That's why I'm scrolling to the bottom of the page and then back to the top - that way I can guarantee that a scroll event is raised. You never know, the web site you're trying to scrape data from may itself be buggy.
Once the "That's all folks" text has appeared, we can go ahead and get the results element's Text property and convert it to a string array as before.
// Assert - we expect 1001 because of the extra "that's all folks"
Assert.Equal(1001, actualResults.Length);
This is the bit that won't be in your code. Because I'm scraping a web site which is under my control, I know exactly how much data it should be displaying so I can check that I've got all the data, and therefore that my scraping code is working correctly.
Further reading
Absolute beginner's introduction to Selenium: https://www.guru99.com/selenium-csharp-tutorial.html
(A curiosity in that article is the way that it starts by creating a console application project and later changes its output type to class library and manually adds the unit test packages, when the project could have been created using one of Visual Studio's unit test project templates. It gets to the right place in the end, albeit via a rather odd route.)
Selenium documentation: https://www.selenium.dev/documentation/
Happy scraping!

If you need to fully execute the web page, then a complete browser like CefSharp is your only option.
It could be that the page is using some combination of scroll state, element visibility, or element positions to trigger content loading. If that's the case, then you'll need to figure out what it is and trigger it programmatically. I know that CefSharp can simulate user actions like clicking, scrolling, etc.

How to prevent textbox from copying letters into that?

I am currently working on a C# MVC project. While entering user details into database I need to customize my MobilePhone field to only accept numbers. After some searching I found the below code :
$(document).on("keypress","#MobilePhone", function (e) {
var regex = new RegExp("^[0-9]\d*$");
var str = String.fromCharCode(!e.charCode ? e.which : e.charCode);
if (regex.test(str)) {
return true;
}
e.preventDefault();
return false;
});
This code works for me, It only allows numbers to be entered in the Textbox.
But there is a problem, If a user copy some text and then paste the content in the Textbox nothing happens. Then if I press submitt button it submits and occur error.
So then I found this question :Disable Copy or Paste action for text box?
Answer to the question is :
$('#email').bind("cut copy paste",function(e) {
e.preventDefault();
});
But after I tried this I can not copy even numbers to the textbox. Is there any way I can prevent copying of alphabets and special characters only.

Just add some checks in your binding to prevent cut / copy / paste a non-number : https://jsfiddle.net/hswtutd9/
$(function() {
$("#email").bind("cut copy paste", function(e) {
const data = e.originalEvent.clipboardData.getData("Text")
if(! /\d./.test(data)) {
e.preventDefault()
}
})
})

why are you using text as your input type ????
if you are using strongly typed view ie editor for then just use data annotation
[DataType(DataType.PhoneNumber)]
public string PhoneNumber{get;set;} //i've used string here believing you initially made it as string and hence not effecting the code elsewhere
if you are using html inputs try
input type ="tel" note some brawser does not support tel for them i would prefer number

You can put the phone number validation code in a function and call if both places like:
function IsValidPhoneNumber(number) {
var regex = new RegExp("^[0-9]\d*$");
if (regex.test(number)) {
return true;
}
return false;
}
and now you can call it both places like:
$(document).on("keypress","#MobilePhone", function (e) {
if(!IsValidPhoneNumber($(this).val())) {
e.preventDefault();
return false;
}
}
$('#MobilePhone').bind("cut copy paste",function(e) {
if(!IsValidPhoneNumber($(this).val())) {
e.preventDefault();
return false;
}
});
or more better would be in a single event:
$(document).on("cut copy paste keypress","#MobilePhone", function (e) {
if(!IsValidPhoneNumber($(this).val())) {
e.preventDefault();
return false;
}
}
Now it would allow copying if the value satisfies the regular expression, you might need to tweak the function to check the whole number but this should give you idea how you can allow it there.
Hope it helped!

C# Windows Store WebView get user-agent

Is there a way to extract the user-agent string that the WebView control uses? If so, I would greatly appreciate it if anyone can give me a method to do so. Using the following does not seem to work:
var userAgent = new StringBuilder(256);
int length = 0;
UrlMkGetSessionOption(UrlMonOptionUserAgent, userAgent, userAgent.Capacity - 1, ref length, 0);

I take that back, using UrlMkGetSessionOption as mentioned in the code above does work.

I currently use this method, adapted from a method given for windows phone originally. It gives the correct result, and gets it straight from a real instance of a WebView object, so gives me more confidence in it having the correct value.
private static string s_userAgent = null;
// Get the default UserAgent which webviews use on this platform.
public async Task<string> GetUserAgent()
{
if (s_userAgent == null)
{
const string Html = #"<!DOCTYPE HTML PUBLIC ""-//W3C//DTD HTML 4.01 Transitional//EN"">
<html>
<head>
<script language=""javascript"" type=""text/javascript"">
function notifyUA() { window.external.notify(navigator.userAgent); }
</script>
</head>
<body onload=""notifyUA();""></body>
</html>";
SemaphoreSlim signal_done = new SemaphoreSlim(0, 1);
var wv = new WebView();
wv.ScriptNotify += (sender, args) =>
{
s_userAgent = args.Value;
// set signal, to show we've done
signal_done.Release();
};
wv.NavigateToString(Html);
// wait for signal
await signal_done.WaitAsync();
Debug.WriteLine("GetUserAgent() called. User agent from WebView: \n{0}", s_userAgent);
}
return s_userAgent;
}

This started as a comment but became too long. To expand on his (#Rexfelis) own answer:
I've found that there can be a difference in what UrlMkGetSessionOption returns depending on where you are in the application lifecycle and if a WebView has been initialized yet in a XAML view.
If you call it before component initialization, it will be missing WebView/3.0 (at least in Windows 10); after initialization it will have that text and results in the same string as the answer by #SimonTillson.
If you need to know the right user agent before component initialization, you have to new up a WebView and navigate before querying UrlMkGetSessionOption; e.g. var wv = new WebView(); wv.NavigateToString(...);. It seems that the user agent is modified on first navigation to include WebView/3.0.

Xilium.CefGlue how to execute JavaScript with return Value?

I'm quite new in programming multi-threading and I could not understand from the xelium example how I could execute a javascript and get the return value.
I have tested:
browser.GetMainFrame().ExecuteJavaScript("SetContent('my Text.')", null, 0);
the javascript is executed, but I this function don’t allow me to get the return value.
I should execute the following function to get all the text the user have written in the box..
browser.GetMainFrame().ExecuteJavaScript("getContent('')", null, 0);
the function TryEval should do this…
browser.GetMainFrame().V8Context.TryEval("GetDirtyFlag", out returninformation , out exx);
But this function can’t be called from the browser, I think it must be called from the renderer? How can I do so?
I couldn’t understand the explanations about CefRenderProcessHandler and OnProcessMessageReceived.. How to register a Scriptable Object and set my javascript & parameters?
Thx for any suggestions how I could solve this!

I have been struggling with this as well. I do not think there is a way to do this synchronously...or easily :)
Perhaps what can be done is this:
From browser do sendProcessMessage with all JS information to renderer
process. You can pass all kinds of parameters to this call in a structured way so encapsulating the JS method name and params in order should not be difficult to do.
In renderer process (RenderProcessHandler onProcessMessageReceived method) do TryEval on the V8Context and get the return value via out parameters and sendProcessMessage back to the
browser process with the JS return value (Note that this supports ordinary return semantics from your JS method).You get the browser instance reference in the onProcessMessageReceived so it is as easy as this (mixed pseudo code)
browser.GetMainFrame().CefV8Context.tryEval(js-code,out retValue, out exception);
process retValue;
browser.sendProcessMessage(...);
Browser will get a callback in the WebClient in onProcessMessageReceived.
There is nothing special here in terms of setting up JS. I have for example a loaded html page with a js function in it. It takes a param as input and returns a string. in js-code parameter to TryEval I simply provide this value:
"myJSFunctionName('here I am - input param')"
It is slightly convoluted but seems like a neat workable approach - better than doing ExecuteJavaScript and posting results via XHR on custom handler in my view.
I tried this and it does work quite well indeed....and is not bad as it is all non-blocking. The wiring in the browser process needs to be done to process the response properly.
This can be extended and built into a set of classes to abstract this out for all kinds of calls..
Take a look at the Xilium demo app. Most of the necessary wiring is already there for onProcessMessage - do a global search. Look for
DemoRendererProcessHandler.cs - renderer side this is where you will invoke tryEval
DemoApp.cs - this is browser side, look for sendProcessMessage - this will initiate your JS invocation process.
WebClient.cs - this is browser side. Here you receive messages from renderer with return value from your JS
Cheers.

I resolved this problem by returning the result value from my JavaScript function back to Xilium host application via an ajax call to a custom scheme handler. According to Xilium's author fddima it is the easiest way to do IPC.
You can find an example of how to implement a scheme handler in the Xilium's demo app.

Check out this post: https://groups.google.com/forum/#!topic/cefglue/CziVAo8Ojg4
using System;
using System.Windows.Forms;
using Xilium.CefGlue;
using Xilium.CefGlue.WindowsForms;
namespace CefGlue3
{
public partial class Form1 : Form
{
private CefWebBrowser browser;
public Form1()
{
InitializeCef();
InitializeComponent();
}
private static void InitializeCef()
{
CefRuntime.Load();
CefMainArgs cefArgs = new CefMainArgs(new[] {"--force-renderer-accessibility"});
CefApplication cefApp = new CefApplication();
CefRuntime.ExecuteProcess(cefArgs, cefApp);
CefSettings cefSettings = new CefSettings
{
SingleProcess = false,
MultiThreadedMessageLoop = true,
LogSeverity = CefLogSeverity.ErrorReport,
LogFile = "CefGlue.log",
};
CefRuntime.Initialize(cefArgs, cefSettings, cefApp);
}
private void Form1_Load(object sender, EventArgs e)
{
browser = new CefWebBrowser
{
Visible = true,
//StartUrl = "http://www.google.com",
Dock = DockStyle.Fill,
Parent = this
};
Controls.Add(browser);
browser.BrowserCreated += BrowserOnBrowserCreated;
}
private void BrowserOnBrowserCreated(object sender, EventArgs eventArgs)
{
browser.Browser.GetMainFrame().LoadUrl("http://www.google.com");
}
}
}
using Xilium.CefGlue;
namespace CefGlue3
{
internal sealed class CefApplication : CefApp
{
protected override CefRenderProcessHandler GetRenderProcessHandler()
{
return new RenderProcessHandler();
}
}
internal sealed class RenderProcessHandler : CefRenderProcessHandler
{
protected override void OnWebKitInitialized()
{
CefRuntime.RegisterExtension("testExtension", "var test;if (!test)test = {};(function() {test.myval = 'My Value!';})();", null);
base.OnWebKitInitialized();
}
}
}

Is there a way to drop static declaration from page method?

For a site I'm developing I have two html buttons, not ASP because I do not want them to postback. For the submit button I am calling a javascript function that implements PageMethods to call a C# method from the codebehind. Here is the code for the buttons and the javascript.
<fieldset id="Fieldset">
<button onclick="SendForm();">Send</button>
<button onclick="CancelForm();">Cancel</button>
</fieldset>
<asp:ScriptManager ID="ScriptManager1" EnablePageMethods="true" EnablePartialRendering="true" runat="server" />
<script type="text/javascript">
function SendForm() {
var email = $get("txtEmail").value;
PageMethods.SendForm(email, OnSucceeded, OnFailed);
}
function OnSucceeded() {
$get("Fieldset").innerHTML = "<p>Thank you!</p>";
}
function OnFailed(error) {
alert(error.get_message());
}
</script>
The codebehind method shown here:
[WebMethod]
public static void SendForm(string email)
{
if (string.IsNullOrEmpty(email))
{
throw new Exception("You must supply an email address.");
}
else
{
if (IsValidEmailAddress(email))
{
bool[] desc = new bool[14];
bool[] local = new bool[14];
bool[] other = new bool[14];
for (int i = 1; i <= 14; i++)
{
desc[i] = ((CheckBox)Page.FindControl("chkDesc" + i.ToString())).Checked;
local[i] = ((CheckBox)Page.FindControl("chkLocal" + i.ToString())).Checked;
other[i] = ((CheckBox)Page.FindControl("chkOther" + i.ToString())).Checked;
/* Do stuff here */
}
}
else
{
throw new Exception("You must supply a valid email address.");
}
}
}
does not work unless it is declared as static. Declaring it as static blocks me from checking the checkboxes on the page because it generates a "An object reference is required for the non-static field, method, or property" error. So my problem can be fixed from either of two directions. A) Is there a way I can have this method work without declaring it as static? B) How do I check the checkboxes if the method is static.

It has to be static, no way around that; But you can access the Page like this
Page page = HttpContext.Current.Handler as Page;
and do FindControl on this page instance.
desc[i] = ((CheckBox)page.FindControl("chkDesc" + i.ToString())).Checked;

Page Methods are a special case of the legacy ASMX web service technology. They allow you to place the service in the codebehind class for the page, and keep you from needing a separate project for the service.
But they will never be able to access anything on the page itself. You'll have to do that from the client side, and pass the values of the check boxes to the service.

If you need to check the checkboxes, then you need to either use an UpdatePanel to do your AJAX stuff, or return something from your page method (ideally a string) and check the checkboxes based on what's returned in javascript on client.