I wrote a console app that will read a text file of links and test them using the HttpClient class to check if the links exist. It works wonderfully at home when tested against common links such as google.com.
When I run the app at under my work intranet, though, I get "Forbidden" errors when it checks links on our company Sharepoint and "Unauthorized" errors when it checks links on the Azure Website. My hope was that running it under my authorized Windows desktop PC would be all I needed for it to work, but nope.
Any hints one how to pass my Network credentials when I access the links with HttpClient?
EDIT: Added code to handle passed console argument (string of authentication cookies)
using System;
using System.Threading.Tasks;
using System.Net.Http;
using System.Net.Http.Headers;
namespace linkbot
{
class Program
{
private static async Task ProcessRepositories(string the_url, string the_cookies)
{ int the_index=the_url.IndexOf("/",9);
string base_string=the_url;
string the_rest="/";
if (the_index>=0)
{
base_string=the_url.Substring(0,the_index);
the_rest=the_url.Substring(the_index);
}
try {
var baseAddress = new Uri(base_string);
using (var handler = new HttpClientHandler { UseCookies = false })
using (var client = new HttpClient(handler) { BaseAddress = baseAddress })
{
var message = new HttpRequestMessage(HttpMethod.Get, the_rest);
message.Headers.Add("Cookie", the_cookies);
var result = await client.SendAsync(message);
result.EnsureSuccessStatusCode();
}
Write("\n" + the_url + " - WORKED!");
}
catch(Exception e )
{
Write("\nFailed: " + the_url + "||||||" +e.ToString() );
//throw e;
}
}
static async Task Main(string[] args)
{
if (args.Length<2){
Console.Write("\n###################\nLinkChecker by Sean J. Miller 2022\n\nusage:\n linkchecker.exe <linkfile> <cookies>\nwhere <linkfile> contains a text file of fully specified links (URLs) with one link per line. Example, https://www.google.com\n\nAn output file is generated titled brokenlinks.txt in the same directory where LinkChecker was launched. <cookies> should be a string such as \"cookie1=value1;cookie2=value2;cookie3=value3;\"\n###################\n\n\n");
return;
}
System.IO.File.Delete("brokenlinks.txt");
System.IO.File.WriteAllText("brokenlinks.txt", "Last Check Started " + (DateTime.Now).ToString());
int counter=0;
int total_lines=TotalLines(#args[0]);
Console.Write("Started " + (DateTime.Now).ToString() + "\n");
foreach (string line in System.IO.File.ReadLines(#args[0]))
{
Console.Write("Processing Link " + (++counter) + "/" + total_lines + "\n");
await ProcessRepositories(line, args[1]);
}
Console.Write("Finished " + (DateTime.Now).ToString() + "\n");
Write("\n");
}
static void Write(string the_text)
{
//Console.Write(the_text);
try
{
System.IO.File.AppendAllText("brokenlinks.txt", the_text);
}
catch (Exception ex)
{
throw ex;
}
}
static int TotalLines(string filePath)
{
using (StreamReader r = new StreamReader(filePath))
{
int i = 0;
while (r.ReadLine() != null) { i++; }
return i;
}
}
}
}
You want to set UseDefaultCredentials to true to use the current logged-on user credentials in your request. You can do that by instantiating your HttpClient like so:
private static readonly HttpClient client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true });
Related
i am struggling with consuming the data i get from my CouchDB database.
I am trying to consume new data that comes to the specific view.
CouchDB offers an option for feed=continous, but i tested it and dont get any data, same in postman.
But if i change it to feed=eventsource i can see the changes in the console. But i dont know how to handle the events.
I opened a method with the right connection, but im stuck now, any help would be great.
public async Task ObserveDbAndTrigger()
{
var url = "http://localhost:5984/MyDB/_changes?feed=eventsource&filter=_view&view=MyView&include_docs=true&attachments=true&heartbeat=1000&since=0";
using (var client = new HttpClient())
{
client.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
client.DefaultRequestHeaders.Add("Accept", "application/json");
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", Convert.ToBase64String(System.Text.ASCIIEncoding.ASCII.GetBytes($"user:password" + $"")));
var request = new HttpRequestMessage(HttpMethod.Get, url);
// handle the incoming events and work with the incoming data
}
}
Any suggestions ?
Clearly there's work to be done. Normally I shy away from answering such questions as posed because it seems like a code service request, but I believe this answer may benefit others beyond the OP.
Here is an extremely naïve bit of code meant to illustrate event delegation and the simplicity of communicating with CouchDB over TCP.
Ultimately this demonstrates the publish/subscribe pattern, which is a reasonable fit. I tested this against CouchDB 2.3 on Windows. The code is hardwired to localhost:5984 because whatever.
class NaiveChangeWatcher
{
static void Main(string[] args)
{
if (args.Length >= 4)
{
// set up server info.
string db = args[0];
string auth = "Basic " + Convert.ToBase64String(ASCIIEncoding.ASCII.GetBytes(String.Join(":", args[1], args[2])));
string query = db + "/_changes?feed=continuous&since=0&heartbeat=" + args[3];
// init the publisher
ChangesPublisher pub = new ChangesPublisher();
// let's subscribe to the OnChange event which writes event data to the console.
pub.OnChange += (sender, e) => Console.WriteLine(e.Value);
pub.OnException += (sender, e) => Console.WriteLine(e.Value.ToString() + "\r\n\r\nPress a key to exit.");
//// start publishing.
Task.Run(async () =>
{
await pub.Begin("localhost", 5984, query, auth, int.Parse(args[3]));
});
// Press a key when bored of it all
Console.ReadKey();
// stop the publisher gracefully
pub.Stop = true;
}
else
{
Console.WriteLine("usage: NaiveChangeWatcher db_name username password timeout_millis");
}
}
//
// The ChangesPublisher notifies subscribers of new data from the changes feed
// via the ChangeEvent. The publisher will trigger an OnException event in the
// event of an exception prior to ending its task.
//
public class ChangesPublisher
{
// Set to true to stop publishing. This causes the Begin method to complete.
public bool Stop { get; set; }
// The event posted when data from the server arrived
public class ChangeEvent : EventArgs
{
public string Value { get; set; }
public ChangeEvent(string value)
{
Value = value;
}
}
// Event triggered when the subscriber croaks by exception
public class ExceptionEvent : EventArgs
{
public Exception Value { get; set; }
public ExceptionEvent(Exception value)
{
Value = value;
}
}
// Subscription to changes from the _changes endpoint
public event EventHandler<ChangeEvent> OnChange = delegate { };
// Subscription to publisher exit on error
public event EventHandler<ExceptionEvent> OnException = delegate { };
public async Task Begin(string serverAddr, int port, string query, string auth, int timeout)
{
using (var client = new TcpClient())
{
string request = String.Join("\r\n", new List<string> {
String.Format("GET /{0} HTTP/1.1",query),
"Authorization: " + auth,
"Accept: application/json",
"Host: " + serverAddr,
"Connection: keep-alive",
"\r\n"
});
try
{
await client.ConnectAsync(serverAddr, port);
using (NetworkStream stream = client.GetStream())
{
StreamWriter writer = new StreamWriter(stream);
await writer.WriteAsync(request);
await writer.FlushAsync();
// read lines from the server, ad nauseum.
StreamReader reader = new StreamReader(stream);
while (!Stop)
{
string data = await reader.ReadLineAsync();
// emit a change event
OnChange(this, new ChangeEvent(data));
}
}
}
catch (Exception e)
{
OnException(this, new ExceptionEvent(e));
}
}
}
}
}
I have a C#-based Selenium test successfully running in a Github Actions container. It worked until recently, where presumably Github upgraded the ChromeDriver version of the ubuntu-latest image to 87.
After this(started failing on Dec. 2nd), I got the following:
Starting ChromeDriver 85.0.4183.87 (<uuid>-refs/branch-heads/4183#{#1689}) on port 35539
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
X SomeTestThatShouldpass [5s 387ms]
Error Message:
System.InvalidOperationException : session not created: This version of ChromeDriver only supports Chrome version 85 (SessionNotCreated)
Following this, I upgraded the ChromeDriver dependency in my *.csproj:
Before:
<PackageReference Include="Selenium.WebDriver.ChromeDriver" Version="85.0.4183.8700" />
After:
<PackageReference Include="Selenium.WebDriver.ChromeDriver" Version="87.0.4280.8800" />
However, after bumping the version I'm left with the very same error as above. It still starts ChromeDriver 85, and still brings forth an error in relation to version mismatches.
Apparently, bumping the dependency and building the .NET application again did not upgrade the ChromeDriver version.
How do I upgrade the ChromeDriver version from 85 to 87?
class Program
{
static void Main(string[] args)
{
string ChromeDriverPath = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
string ChromDriverVersion = GetChromDriverVersion(ChromeDriverPath);
string ChromeWebVersion = GetWebChromeVersion();
if (ChromDriverVersion != ChromeWebVersion)
{
var urlToDownload = GetURLToDownload(ChromeWebVersion);
KillAllChromeDriverProcesses();
DownloadNewVersionOfChrome(urlToDownload, ChromeDriverPath);
string extract = ExtructZip(ChromeDriverPath);
}
}
static string GetChromDriverVersion(string ChromeDriverePath)
{
string driverversion = "";
if (File.Exists(ChromeDriverePath + "\\chromedriver.exe"))
{
IWebDriver driver = new ChromeDriver(ChromeDriverePath);
ICapabilities capabilities = ((ChromeDriver)driver).Capabilities;
driverversion = ((capabilities.GetCapability("chrome") as Dictionary<string, object>)["chromedriverVersion"]).ToString().Split(' ').First();
driver.Dispose();
}
else
{
Console.WriteLine("ChromeDriver.exe missing !!");
}
return driverversion;
}
static string GetChromeWebPath()
{
//Path originates from here: https://chromedriver.chromium.org/downloads/version-selection
using (RegistryKey key = Registry.LocalMachine.OpenSubKey("SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\App Paths\\chrome.exe"))
{
if (key != null)
{
Object o = key.GetValue("");
if (!String.IsNullOrEmpty(o.ToString()))
{
return o.ToString();
}
else
{
throw new ArgumentException("Unable to get version because chrome registry value was null");
}
}
else
{
throw new ArgumentException("Unable to get version because chrome registry path was null");
}
}
}
static string GetWebChromeVersion()
{
string productVersionPath = GetChromeWebPath();
if (String.IsNullOrEmpty(productVersionPath))
{
throw new ArgumentException("Unable to get version because path is empty");
}
if (!File.Exists(productVersionPath))
{
throw new FileNotFoundException("Unable to get version because path specifies a file that does not exists");
}
var versionInfo = FileVersionInfo.GetVersionInfo(productVersionPath);
if (versionInfo != null && !String.IsNullOrEmpty(versionInfo.FileVersion))
{
return versionInfo.FileVersion;
}
else
{
throw new ArgumentException("Unable to get version from path because the version is either null or empty: " + productVersionPath);
}
}
static string GetURLToDownload(string version)
{
if (String.IsNullOrEmpty(version))
{
throw new ArgumentException("Unable to get url because version is empty");
}
//URL's originates from here: https://chromedriver.chromium.org/downloads/version-selection
string html = string.Empty;
string urlToPathLocation = #"https://chromedriver.storage.googleapis.com/LATEST_RELEASE_" + String.Join(".", version.Split('.').Take(3));
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlToPathLocation);
request.AutomaticDecompression = DecompressionMethods.GZip;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream))
{
html = reader.ReadToEnd();
}
if (String.IsNullOrEmpty(html))
{
throw new WebException("Unable to get version path from website");
}
return "https://chromedriver.storage.googleapis.com/" + html + "/chromedriver_win32.zip";
}
static void KillAllChromeDriverProcesses()
{
var processes = Process.GetProcessesByName("chromedriver");
foreach (var process in processes)
{
try
{
process.Kill();
}
catch
{
}
}
}
static void DownloadNewVersionOfChrome(string urlToDownload,string ChromDriverPath)
{
if (String.IsNullOrEmpty(urlToDownload))
{
throw new ArgumentException("Unable to get url because urlToDownload is empty");
}
using (var client = new WebClient())
{
if (File.Exists(ChromDriverPath + "\\chromedriver.zip"))
{
File.Delete(ChromDriverPath + "\\chromedriver.zip");
}
client.DownloadFile(urlToDownload, "chromedriver.zip");
if (File.Exists(ChromDriverPath + "\\chromedriver.zip") && File.Exists(ChromDriverPath + "\\chromedriver.exe"))
{
File.Delete(ChromDriverPath + "\\chromedriver.exe");
}
}
}
static string ExtructZip(string ChromeDriverPath)
{
string errorMsg = "";
string zipFileName = "";
try
{
zipFileName= System.IO.Path.GetDirectoryName(Assembly.GetEntryAssembly().Location) + "\\chromedriver.zip";
string orgFileName = "";
using (ZipFile zip = ZipFile.Read(zipFileName))
{
foreach (ZipEntry e in zip)
{
e.Extract(ChromeDriverPath, ExtractExistingFileAction.OverwriteSilently);
orgFileName = e.FileName;
}
}
File.Delete(zipFileName);
errorMsg = "Done";
}
catch (Exception ex)
{
errorMsg = "ExtructZip file " + zipFileName.Split('\\').Last() + " - " + ex.Message;
}
return errorMsg;
}
}
}
https://github.com/rosolko/WebDriverManager.Net
Use WebDriver manager to automatically install the version , you want
using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using WebDriverManager;
using WebDriverManager.DriverConfigs.Impl;
namespace Test
{
[TestFixture]
public class Tests
{
private IWebDriver _webDriver;
[SetUp]
public void SetUp()
{
new DriverManager().SetUpDriver(new ChromeConfig(), "2.25");
_webDriver = new ChromeDriver();
}
[TearDown]
public void TearDown()
{
_webDriver.Quit();
}
[Test]
public void Test()
{
_webDriver.Navigate().GoToUrl("https://www.google.com");
Assert.True(_webDriver.Title.Contains("Google"));
}
}
}
I am using CSOM to connect to SharePoint and load files from Document Libraries. My problem is when I create ClientContext object with the full URL that ends with Forms/AllItems.aspx or SitePages/Home.aspx or other bits, it will throw invalid URL exception.
For example, this will work:
ClientContext ctx = new ClientContext("https://mycompany.sharepoint.com/Extranet/Test%20AutoAppraisal");
And this will not work:
ClientContext ctx = new ClientContext("https://mycompany.sharepoint.com/Extranet/Test%20AutoAppraisal/SitePages/Home.aspx");
Is there a safe way to trim SharePoint URL similar to this new Microsoft Flow feature (Automatic trimming of SharePoint URLs)?
My current solution, removing segment by segment until the Uri is valid, but this is time consuming for client experience.
private static Uri TryResolvingUrl(Uri path)
{
string host = null;
while (true)
{
try
{
ClientContext ctx = new ClientContext(path)
{
Credentials = _credential,
RequestTimeout = -1
};
ctx.Load(ctx.Web, web => web.ServerRelativeUrl);
ctx.ExecuteQueryRetry();
return path;
}
catch (Exception e)
{
if (host == null)
{
host = path.Scheme + "://" + path.Authority;
}
if (path.Segments.Length == 1)
{
throw e; // we wont be able to use this url!
}
string newUrl = host;
for (int i = 0; i < path.Segments.Length - 1; i++)
{
newUrl += path.Segments[i];
}
path = new Uri(newUrl.Unescape());
}
}
}
The following code snippet for your reference.
string siteUrl = "https://mycompany.sharepoint.com/Extranet/Test%20AutoAppraisal/Account%20Managemenet%20Archive/Forms/AllItems.aspx";
var urlArray = siteUrl.Split('/');
siteUrl = urlArray[0]+"//"+urlArray[2]+"/"+urlArray[3]+"/"+urlArray[4];
using (ClientContext ctx = new ClientContext(siteUrl))
{
}
I am able to do queries using OAuth2 and this:
/rest-1.oauth.v1/Data/Story?sel=Name,Number&Accept=text/json
However I am unable to get the OAuth2 and the new query1.v1 to work against the Sumnmer2013 VersionOne. I am getting (401) Unauthorized and specified method is not supported using two different URLs.
Here is the code that includes the working /rest-1.oauth.v1 and the non-working query1.v1 and non-working query.legacy.v1. Scroll towards the bottom of the code to see the Program Main (starting point of code)
Please advise on what I am missing here.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using OAuth2Client;
namespace ExampleMemberListCSharp
{
class Defaults
{
public static string Scope = "apiv1";
//public static string EndpointUrl = "http://localhost/VersionOne.Web";
public static string EndpointUrl = "https://versionone-test.web.acme.com/summer13_demo";
public static string ApiQueryWorks = "/rest-1.oauth.v1/Data/Member?Accept=text/json";
public static string ApiQuery = "/rest-1.oauth.v1/Data/Story?sel=Name,Number&Accept=text/json";
}
static class WebClientExtensions
{
public static string DownloadStringOAuth2(this WebClient client, IStorage storage, string scope, string path)
{
var creds = storage.GetCredentials();
client.AddBearer(creds);
try
{
return client.DownloadString(path);
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError)
{
if (((HttpWebResponse)ex.Response).StatusCode != HttpStatusCode.Unauthorized)
throw;
var secrets = storage.GetSecrets();
var authclient = new AuthClient(secrets, scope);
var newcreds = authclient.refreshAuthCode(creds);
var storedcreds = storage.StoreCredentials(newcreds);
client.AddBearer(storedcreds);
return client.DownloadString(path);
}
throw;
}
}
public static string UploadStringOAuth2(this WebClient client, IStorage storage
, string scope, string path, string pinMethod, string pinQueryBody)
{
var creds = storage.GetCredentials();
client.AddBearer(creds);
client.UseDefaultCredentials = true;
try
{
return client.UploadString(path, pinMethod, pinQueryBody);
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError)
{
if (((HttpWebResponse)ex.Response).StatusCode != HttpStatusCode.Unauthorized)
throw;
var secrets = storage.GetSecrets();
var authclient = new AuthClient(secrets, scope);
var newcreds = authclient.refreshAuthCode(creds);
var storedcreds = storage.StoreCredentials(newcreds);
client.AddBearer(storedcreds);
client.UseDefaultCredentials = true;
return client.UploadString(path, pinMethod, pinQueryBody);
}
throw;
}
}
}
class AsyncProgram
{
private static async Task<string> DoRequestAsync(string path)
{
var httpclient = HttpClientFactory.WithOAuth2("apiv1");
var response = await httpclient.GetAsync(Defaults.EndpointUrl + Defaults.ApiQuery);
var body = await response.Content.ReadAsStringAsync();
return body;
}
public static int MainAsync(string[] args)
{
var t = DoRequestAsync(Defaults.EndpointUrl + Defaults.ApiQuery);
Task.WaitAll(t);
Console.WriteLine(t.Result);
return 0;
}
}
class Program
{
static void Main(string[] args)
{
IStorage storage = Storage.JsonFileStorage.Default;
using (var webclient = new WebClient())
{
// this works:
var body = webclient.DownloadStringOAuth2(storage, "apiv1", Defaults.EndpointUrl + Defaults.ApiQuery);
Console.WriteLine(body);
}
IStorage storage2 = Storage.JsonFileStorage.Default;
using (var webclient2 = new WebClient())
{
// This does NOT work. It throws an exception of (401) Unauthorized:
var body2 = webclient2.UploadStringOAuth2(storage2, "apiv1", Defaults.EndpointUrl + "/query.v1", "SEARCH", QueryBody);
// This does NOT work. It throws an exception of The remote server returned an error: (403): Forbidden."
var body3 = webclient2.UploadStringOAuth2(storage2, "apiv1", Defaults.EndpointUrl + "/query.legacy.v1", "SEARCH", QueryBody);
// These do NOT work. Specified method is not supported:
var body4 = webclient2.UploadStringOAuth2(storage2, "apiv1", Defaults.EndpointUrl + "/oauth.v1/query.legacy.v1", "SEARCH", QueryBody);
var body5 = webclient2.UploadStringOAuth2(storage2, "apiv1", Defaults.EndpointUrl + "/oauth.v1/query.legacy.v1", "SEARCH", QueryBody);
}
Console.ReadLine();
AsyncProgram.MainAsync(args);
}
public const string QueryBody = #"
from: Story
select:
- Name
";
}
}
At this time, the query.v1 endpoint requires the query-api-1.0 scope to be granted.
You'll have to add that to your scope list (it can be simply space-separated e.g. apiv1 query-api-1.0) and visit the grant URL again to authorize the permissions.
This somewhat vital piece of information doesn't seem to appear in the docs on community.versionone.com, so it looks like an update is in order.
Also, only the rest-1.oauth.v1 and query.v1 endpoints respond to OAuth2 headers at this time. A future release will see it apply to all endpoints and remove the endpoint duplication for the two types of authentication
I have had problems in the past trying to use an HTTP method other than POST to transmit the query. Security software, IIS settings, and proxies may all handle such requests in unexpected ways.
I'm pulling data off the Rally server at https://rally1.rallydev.com using C# and the Rally.RestAPI.dll. The server was recently upgraded to webservice v2.0 and I'm having some problems getting tasks for a user story. I know the way child collections are presented were changed in the API with the move to 2.0, but what I'm trying isn't working.
v2.0 removed the ability to return child collections in the same
response for performance reasons. Now fetching a collection will return
an object with the count and the url from which to get the collection
data.A separate request is needed to get to the elements of the
collection.
Here is the code that iterates over user story results, accesses "Tasks" collection on stories and issues a separate request to access individual Task attributes:
using System;
using System.Collections.Generic;
using System.Collections;
using System.Linq;
using System.Text;
using Rally.RestApi;
using Rally.RestApi.Response;
namespace aRestApp_CollectionOfTasks
{
class Program
{
static void Main(string[] args)
{
//Initialize the REST API
RallyRestApi restApi;
restApi = new RallyRestApi("user#co.com", "secret", "https://rally1.rallydev.com", "v2.0");
//Set our Workspace and Project scopings
String workspaceRef = "/workspace/11111"; //please replace this OID with an OID of your workspace
String projectRef = "/project/22222"; //please replace this OID with an OID of your project
bool projectScopingUp = false;
bool projectScopingDown = true;
Request storyRequest = new Request("HierarchicalRequirement");
storyRequest.Workspace = workspaceRef;
storyRequest.Project = projectRef;
storyRequest.ProjectScopeUp = projectScopingUp;
storyRequest.ProjectScopeDown = projectScopingDown;
storyRequest.Fetch = new List<string>()
{
"Name",
"FormattedID",
"Tasks",
"Estimate"
};
storyRequest.Query = new Query("LastUpdateDate", Query.Operator.GreaterThan, "2013-08-01");
QueryResult queryStoryResults = restApi.Query(storyRequest);
foreach (var s in queryStoryResults.Results)
{
Console.WriteLine("----------");
Console.WriteLine("FormattedID: " + s["FormattedID"] + " Name: " + s["Name"]);
//Console.WriteLine("Tasks ref: " + s["Tasks"]._ref);
Request taskRequest = new Request(s["Tasks"]);
QueryResult queryTaskResult = restApi.Query(taskRequest);
if (queryTaskResult.TotalResultCount > 0)
{
foreach (var t in queryTaskResult.Results)
{
var taskEstimate = t["Estimate"];
var taskName = t["Name"];
Console.WriteLine("Task Name: " + taskName + " Estimate: " + taskEstimate);
}
}
else
{
Console.WriteLine("no tasks found");
}
}
}
}
}