Develop Reference

How to Retrieve file from URL which Requires Authentication?

I'm making a tool in Unity to retrieve data from a server. The server's interface can provide URLs that we can later click on which will return an XML or CSV file with the results of that query from that server. But, it requires Basic Authentication. When clicking the links, it simply pops up a login screen before giving me the results. If I try what I [think] I know in Unity (starting with WebRequest.GetResponse()) it simply fails and says I am not authorized. It does not show the popup for authentication. So how do I let that login popup appear when accessing with Unity and await the login results to get the file? Or is there some standardized way to provide that info in the link itself?

Here is some code that should you get started. Just fill in the request link and username, password. please see the comments in the code to see what it does.
//try just in case something went wrong whith calling the api
try
{
//Use using so that if the code end the client disposes it self
using (HttpClient client = new HttpClient())
{
//Setup authentication information
string yourusername = "username";
string yourpwd = "password";
//this is when you expect json to return from the api
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
//add the authentication to the request
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic",
Convert.ToBase64String(
System.Text.ASCIIEncoding.ASCII.GetBytes($"{yourusername}:{yourpwd}")));
//api link used to make the call
var requestLink = $"apiLink";
using (HttpResponseMessage response = client.GetAsync(requestLink).Result)
{
//Make sure the request was successfull before proceding
response.EnsureSuccessStatusCode();
//Get response from website and convert to a string
string responseBody = response.Content.ReadAsStringAsync().Result;
//now you have the results
}
}
}
//Catch the exception if something went from and show it!
catch (Exception)
{
throw;
}

This is what I ended up going with after looking at the comments above. Let me know if I'm doing anything terribly inefficient!
String username = "Superman"; // Obviously handled secretly
String pw = "ILoveLex4evar!"; // Obviously handled secretly
String url = "https://www.SuperSecretServer.com/123&stuff=?uhh";
String encoded = System.Convert.ToBase64String(System.Text.Encoding.GetEncoding("ISO-8859-1").GetBytes(username + ":" + pw));
CookieContainer myContainer = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Headers.Add("Authorization", "Basic " + encoded);
try
{
using (WebResponse response = request.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
using (FileStream xml = File.Create("filepath/filename.xml"))
{
byte[] buffer = new byte[BufferSize];
int read;
while ((read = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
xml.Write(buffer, 0, read);
}
}
}
}
}

Logging into Jenkins Programmatically in C#

I have jobs in Jenkins that i cannot access unless i log in first using a my username and password.
For example if i try to access "localhost:xxx/job/some_job_1" i will get a 404 Error unless i log in first. And i say this because i have tried the following using WebRequest class:
string formParams = "j_username=bobbyLee&j_password=SecretPassword25&from=%2F&json=%7B%22j_username%22%3A+%bobbyLee%22%2C+%22j_password%22%3A+%22SecretPassword%25%22%2C+%22remember_me%22%3A+false%2C+%22from%22%3A+%22%2F%22%7D&Submit=log+in";
// ***this is the exact string that is sent when i log in normally, obtained using Fiddler***
string formUrl = "http://serverName:PortNum/j_acegi_security_check";
// ***I have also tried http://serverName:PortNum/login***
string cookieHeader;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
string pageSource;
string getUrl = "http://serverName:portNum/job/some_job/";
WebRequest getRequest = WebRequest.Create(getUrl);
getRequest.Headers.Add("Cookie", cookieHeader);
WebResponse getResponse = getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
The response that i get back from the POST request is "HTML OK", and cookieHeader is not null. But when i then try to make a GET request to get what i want, i get a 404 error when attempting to access the job "http://serverName:portNum/job/some_job/", as if i didn't log in successfully.
So what is the correct way to log into Jenkins from c#, and get the HTML source code of the jobs that only appears after logging in?

The RESTAPI is your best friend here.
It is an incredibly rich source of information. I have written a system that will show an entire program of work on a page with full deployment traceability.
I am going to assume you have some security in place in your Jenkins instance which means requests need to be authenticated.
I use the following class for this:
using System;
using System.Net;
using System.Text;
namespace Core.REST
{
public class HttpAdapter
{
private const string ApiToken = "3abcdefghijklmnopqrstuvwxyz12345"; // you will need to change this to the real value
private const string UserName = "restapi";
public string Get(string url)
{
try
{
const string credentials = UserName + ":" + ApiToken;
var authorization = Convert.ToBase64String(Encoding.ASCII.GetBytes(credentials));
using (var wc = new WebClient())
{
wc.Headers[HttpRequestHeader.Authorization] = "Basic " + authorization;
var htmlResult = wc.DownloadString(string.Format(url));
return htmlResult;
}
}
catch (WebException e)
{
Console.WriteLine("Could not retrieve REST API response");
throw e;
}
}
}
}
restapi is a dedicated user I created. I think I gave it admin access just so I didn't have to worry about it. I was admin but all the other developers and testers in the 3 crews had highly controlled and limited access to only what they needed and nothing more. It is also better practice to have a dedicated users for functions like this.
I constructed my c# classes to consume (deserialise) data from any page that supports the api/json suffix.

.NETCore HttpWebRequest - Old Way isn't Working

Before I upgraded to the newest .NetCore I was able to run the HttpWebRequest, add the headers and content Type and pull the stream of the JSON file from Twitch. Since the upgrade this is not working. I receive a Web Exception each time I go to get the response Stream. Nothing has changed with twitch because it still works with the old Bot. The old code is below:
private const string Url = "https://api.twitch.tv/kraken/streams/channelname";
HttpWebRequest request;
try
{
request = (HttpWebRequest)WebRequest.Create(Url);
}
request.Method = "Get";
request.Timeout = 12000;
request.ContentType = "application/vnd.twitchtv.v5+json";
request.Headers.Add("Client-ID", "ID");
try
{
using (var s = request.GetResponse().GetResponseStream())
{
if (s != null)
using (var sr = new StreamReader(s))
{
}
}
}
I have done some research and found that I may need to start using either an HttpClient or HttpRequestMessage. I have tried going about this but when adding headers content type the program halts and exits. after the first line here: (when using HttpsRequestMessage)
request.Content.Headers.ContentType.MediaType = "application/vnd.twitchtv.v5+json";
request.Content.Headers.Add("Client-ID", "rbp1au0xk85ej6wac9b8s1a1amlsi5");

You are trying to add a ContentType header, but what you really want is to add an Accept header (your request is a GET and ContentType is used only on requests which contain a body, e.g. POST or PUT).
In .NET Core you need to use HttpClient, but remember that to correctly use it you need to leverage the use of async and await.
Here it is an example:
using System.Net.Http;
using System.Net.Http.Headers;
private const string Url = "https://api.twitch.tv/kraken/streams/channelname";
public static async Task<string> GetResponseFromTwitch()
{
using(var client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/vnd.twitchtv.v5+json"));
client.DefaultRequestHeaders.Add("Client-ID", "MyId");
using(var response = await client.GetAsync(Url))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync(); // here we return the json response, you may parse it
}
}
}

Download "Visual Studio Team Services" Charts via WebRequest

I need to upgrade an old TFS 2013 class for Visual Studio Team Services.
To get the Burndown-Chart I used to download the image via HttpWebRequest direcly from the url.
Somehow Iam not able to do this in VSTS. I always get the error message "invalid parameters". Everything else works fine. (I had to setup the Alternate authentication credentials in my profile to get it working for my application)
Here my code:
public Image GetChart(string uri)
{
HttpWebRequest httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(uri);
httpWebRequest.Credentials = new NetworkCredential("MyUserNameForApplication", "MyPWForApplication");
using (HttpWebResponse httpWebReponse = (HttpWebResponse)httpWebRequest.GetResponse())
{
using (Stream stream = httpWebReponse.GetResponseStream())
{
return Image.FromStream(stream); //Error occourse
}
}
}
The url which gets passed as a parameter usally looks like this:
https://YourVSName.visualstudio.com/DefaultCollection/a5d2310b-d3f8-4365-b693-3826ab60e939/_api/_teamChart/Burndown?chartOptions={%22Width%22%3A1248%2C%22Height%22%3A161%2C%22ShowDetails%22%3Atrue%2C%22Title%22%3A%22%22}&counter=1&iterationPath=Developing\Sprint+1&__v=5
What I think the problem is:
First I thought this might be a security issue, because this code is able to download normal google images. And when I try to get the content of the url It returns a lot of code with a message in it:
Microsoft Internet Explorer's Enhanced Security Configuration is currently enabled on your environment. This enhanced level of security prevents our web integration experiences from displaying or performing correctly. To continue with your operation please disable this configuration or contact your administrator
I set my Internet security settings to the lowest level and still the same result.
Another reason why this might not working is, because the url linking to the burndown-chart doesnt contain an Image extension. Iam not quite shure here this effects the result.
Or that the parameters which are getting past in the url are incorrect...
What I have tried so far:
I have used bunch of other code to get the image from that link. For example using WebClient or tried to upload cookies (credentials) to the tfs and than tried to connect.
My Question
Is it possible to get that image from the chart via url, and if so, how?
Thanks for any kind of help :).
EDIT
Currently Iam using this code (Thanks to #Eddie - MSFT):
public static async void GetChart(string uri,string username, string password)
{
try
{
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Add(
new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic",
Convert.ToBase64String(
System.Text.ASCIIEncoding.ASCII.GetBytes(
string.Format("{0}:{1}", username, password))));
using (HttpResponseMessage response = client.GetAsync(uri).Result)
{
response.EnsureSuccessStatusCode();
var responseStream = await response.Content.ReadAsStreamAsync();
var img = Image.FromStream(responseStream);
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
static void Main(string[] args)
{
string uri = "https://Name.visualstudio.com/DefaultCollection/a5d2310b-d3f8-4365-b693-3826ab60e939/_api/_teamChart/Burndown?chartOptions=%7B%22Width%22%3A1248%2C%22Height%22%3A636%2C%22ShowDetails%22%3Atrue%2C%22Title%22%3A%22%22%7D&counter=1&iterationPath=Developing%5CSprint+1&__v=5";
TFSHelper.TFSHelper.GetChart(uri, username,pw)
}

I use "httpclient" with alternative credential to do this:
using System.Text;
using System.Threading.Tasks;
using System.Drawing;
using System.Net.Http;
using System.Net.Http.Headers;
using System.IO;
namespace GetImageA
{
class Program
{
static void Main(string[] args)
{
Uri uri = new Uri("your image uri");
GetImage(uri);
}
public static void GetImage(Uri uri)
{
var username = "username";
var password = "password";
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic",
Convert.ToBase64String(
ASCIIEncoding.ASCII.GetBytes(
string.Format("{0}:{1}", username, password))));
Stream str = client.GetStreamAsync(uri).Result;
Image im = Image.FromStream(str);
im.Save("E:\\image.png");
}
}
}
}

Did you try the authentication with your personal access token instead of username and password, something like this?
I am using the below code to download the attachments, inline images of the Work items from VSTS.
try
{
var personalaccesstoken = "Your_VSTS_Personal_Access_Token";
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Add(
new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic",
Convert.ToBase64String(
System.Text.ASCIIEncoding.ASCII.GetBytes(
string.Format("{0}:{1}", "", personalaccesstoken))));
using (HttpResponseMessage response = client.GetAsync(uri).Result)
{
response.EnsureSuccessStatusCode();
string responseBody = await response.Content.ReadAsStringAsync();
Console.WriteLine(responseBody);
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}

WCF HTTP GET api

In my .NET project, I have to use HTTP GET request to get weather info for my city from API. Because of my JavaScript background I thought "OK, so all I need is something like app.get(url, body)", so I started with something like this:
using (var client = new WebClient())
{
var responseString = client.DownloadString("http://www.webservicex.net/globalweather.asmx/GetWeather?CityName=" + city + "&CountryName=" + country);
string xmlString = DecodeXml(responseString);
return xmlString;
}
Unfortunately for me it turned out, that I have to use WCF to get the data. I searched the web for some tutorials, but I couldn't find anything with getting the data from outer sources, just creating own API.
I'm not a native speaker, so maybe I'm just out of words to look for the solution, but it would be awesome if you could give me some advice.

Assuming you are using Visual Studio. Add Service Reference, and then type "http://www.webservicex.net/globalweather.asmx" into the address and hit Go. It'll auto-generate the end point for you to use.
Then the code is something like:
ServiceReference1.GlobalWeatherSoapClient client = new ServiceReference1.GlobalWeatherSoapClient("GlobalWeatherSoap");
string cities = client.GetCitiesByCountry("Hong Kong");

If you want to just use HTTP GET, you can do something like this:
var city = "Dublin";
var country = "Ireland";
WebRequest request = WebRequest.Create(
"http://www.webservicex.net/globalweather.asmx/GetWeather?CityName=" +
city +
"&CountryName=" + country);
request.Credentials = CredentialCache.DefaultCredentials;
WebResponse response = request.GetResponse();
Console.WriteLine(((HttpWebResponse)response).StatusDescription);
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
Console.WriteLine(responseFromServer);
reader.Close();
response.Close();
Console.ReadLine();
Please note, I have not HTML decoded the response here, you can simply use HttpUtility.HtmlDecode for that.
Also, you will need to include the following using statements:
using System.IO;
using System.Net;