Develop Reference

How to Retrieve file from URL which Requires Authentication?

I'm making a tool in Unity to retrieve data from a server. The server's interface can provide URLs that we can later click on which will return an XML or CSV file with the results of that query from that server. But, it requires Basic Authentication. When clicking the links, it simply pops up a login screen before giving me the results. If I try what I [think] I know in Unity (starting with WebRequest.GetResponse()) it simply fails and says I am not authorized. It does not show the popup for authentication. So how do I let that login popup appear when accessing with Unity and await the login results to get the file? Or is there some standardized way to provide that info in the link itself?

Here is some code that should you get started. Just fill in the request link and username, password. please see the comments in the code to see what it does.
//try just in case something went wrong whith calling the api
try
{
//Use using so that if the code end the client disposes it self
using (HttpClient client = new HttpClient())
{
//Setup authentication information
string yourusername = "username";
string yourpwd = "password";
//this is when you expect json to return from the api
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
//add the authentication to the request
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic",
Convert.ToBase64String(
System.Text.ASCIIEncoding.ASCII.GetBytes($"{yourusername}:{yourpwd}")));
//api link used to make the call
var requestLink = $"apiLink";
using (HttpResponseMessage response = client.GetAsync(requestLink).Result)
{
//Make sure the request was successfull before proceding
response.EnsureSuccessStatusCode();
//Get response from website and convert to a string
string responseBody = response.Content.ReadAsStringAsync().Result;
//now you have the results
}
}
}
//Catch the exception if something went from and show it!
catch (Exception)
{
throw;
}

This is what I ended up going with after looking at the comments above. Let me know if I'm doing anything terribly inefficient!
String username = "Superman"; // Obviously handled secretly
String pw = "ILoveLex4evar!"; // Obviously handled secretly
String url = "https://www.SuperSecretServer.com/123&stuff=?uhh";
String encoded = System.Convert.ToBase64String(System.Text.Encoding.GetEncoding("ISO-8859-1").GetBytes(username + ":" + pw));
CookieContainer myContainer = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Headers.Add("Authorization", "Basic " + encoded);
try
{
using (WebResponse response = request.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
using (FileStream xml = File.Create("filepath/filename.xml"))
{
byte[] buffer = new byte[BufferSize];
int read;
while ((read = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
xml.Write(buffer, 0, read);
}
}
}
}
}

Microsoft Translator Bad request (400) issue

I've seen threads on this issue but my problem is particularly confusing. I have a free 2 million character subscription, a valid client id and secret. When I run my code I get to call the API a few times successfully (the most I've seen is 75 consecutive successful calls). Then every other call returns a Bad request response: The remote server returned an error: (400) Bad Request.
I create the token once with my credentials and never create it again. I loop through a file, parse it, and submit every parsed string for translation by calling the API. It seems that I reach some sort of limit that I'm now aware of.
When looking at my account, it doesn't seem to be discounting the characters that I've translated already which would make me highly suspicious that I have the wrong credentials when creating the token. I quadruple-checked that and everything seems to be ok.
Any guidance on what I may be missing here would be much appreciated.
Here's the code that creates the token. I do think though that there may be an unknown limitation that I'm not aware of with the free subscription.
static void gettoken()
{
//Get access token
string clientID = "my client id";
string clientSecret = "my secret";
String strTranslatorAccessURI = "https://datamarket.accesscontrol.windows.net/v2/OAuth2-13";
String strRequestDetails = string.Format("grant_type=client_credentials&client_id={0}&client_secret={1}&scope=http://api.microsofttranslator.com", clientID, clientSecret);
System.Net.WebRequest webRequest = System.Net.WebRequest.Create(strTranslatorAccessURI);
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.Method = "POST";
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(strRequestDetails);
webRequest.ContentLength = bytes.Length;
using (System.IO.Stream outputStream = webRequest.GetRequestStream())
{
outputStream.Write(bytes, 0, bytes.Length);
}
System.Net.WebResponse webResponse = webRequest.GetResponse();
System.Runtime.Serialization.Json.DataContractJsonSerializer serializer = new System.Runtime.Serialization.Json.DataContractJsonSerializer(typeof(AdmAccessToken));
AdmAccessToken token = (AdmAccessToken)serializer.ReadObject(webResponse.GetResponseStream());
MyGlobals.headerValue = "Bearer " + token.access_token;
}
And here's the code that calls the API itself. I call the API method from a loop.
static void RunBing(string sterm)
{
//Submit the translation request
string txtToTranslate = sterm;
string uri = "http://api.microsofttranslator.com/v2/Http.svc/Translate?text=" + txtToTranslate + "&from=en&to=es";
System.Net.WebRequest translationWebRequest = System.Net.WebRequest.Create(uri);
translationWebRequest.Headers.Add("Authorization", MyGlobals.headerValue);
System.Net.WebResponse response = null;
try {
response = translationWebRequest.GetResponse();
}
catch (Exception e)
{
Console.WriteLine("Term failed: " + sterm);
Console.WriteLine(e);
return;
}
System.IO.Stream stream = response.GetResponseStream();
System.Text.Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
System.IO.StreamReader translatedStream = new System.IO.StreamReader(stream, encode);
System.Xml.XmlDocument xTranslation = new System.Xml.XmlDocument();
xTranslation.LoadXml(translatedStream.ReadToEnd());
MyGlobals.xlation = xTranslation.InnerText;
}
After several successful calls to the API, I start to get the following message:
System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.HttpWebRequest.GetResponse()
at Translate.TranslateText.Program.RunBing(String sterm)

OneLogin Create Session Login Token API returns status 400 with message: Bad Request

I am developing a C# application which needs to use the onelogin API to retrieve a session token. I am able to authenticate and and create a token with the following code:
WebRequest Authrequest = WebRequest.Create("https://api.us.onelogin.com/auth/oauth2/token");
Authrequest.Method = "POST";
Authrequest.ContentType = "application/json";
Authrequest.Headers.Add("cache-control", "no-cache");
Authrequest.Headers.Add("Authorization: client_id:XXXXXXX7bbf2c50200d8175206f664dc28ffd3ec66eef0bfedb68c3366420dc, client_secret:XXXXXXXXXX6ba2802187feb23f6450c6812b8e6639361d24aa83f12010f ");
using (var streamWriter = new StreamWriter(Authrequest.GetRequestStream()))
{
string Authjson = new JavaScriptSerializer().Serialize(new
{
grant_type = "client_credentials"
});
streamWriter.Write(Authjson);
}
WebResponse AuthReponse;
AuthReponse = Authrequest.GetResponse();
Stream receiveStream = AuthReponse.GetResponseStream ();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader (receiveStream);
JObject incdata = JObject.Parse(readStream.ReadToEnd());
string sToken = incdata["data"][0]["access_token"].Value<string>();
AuthReponse.Close();
However, when running the Create Session Login Token with the following code, it only returns a 400 error, and the message has no detail. Just Bad Request:
//Get the session token for the specified user, using the token recieved from previous web request
WebRequest request = WebRequest.Create("https://api.us.onelogin.com/api/1/login/auth");
request.Method = "POST";
request.ContentType = "application/json";
request.Headers.Add("authorization", "bearer:" + sToken);
using (var streamWriter2 = new StreamWriter(request.GetRequestStream()))
{
string json = JsonConvert.SerializeObject(new
{
username_or_email = sUsername,
password = sPassword,
subdomain = "comp-alt-dev"
});
streamWriter2.Write(json);
}
WebResponse response;
response = request.GetResponse();
string streamText = "";
var responseStream = response.GetResponseStream();
using (responseStream)
{
var streamReader = new StreamReader(responseStream);
using (streamReader)
{
streamText = streamReader.ReadToEnd();
streamReader.Close();
//
}
responseStream.Close();
}
Any ideas?
-Thank you

Also for anyone who may be getting this error. in C# the email is case sensitive. I tried User.email.com. In onelogin it was saved as user#email.com. changing the c# to lower case fixed it.

Can you let us know what payload you're sending across the wire to the .../1/login/auth endpoint as well as the response (either as others have suggested as packet snoop, or just as a debug output from the code)
400 means either bad json or the endpoint requires MFA, so this will narrow it down.
~thanks!

Just joining the troubleshooting effort =) -- I can replicate a 400 Bad Request status code with a "bad request" message when the request body contains a username_or_email and/or subdomain value that does not exist, or if the request body is empty.

Can you post what goes over the wire to the OneLogin endpoint...
OK Thanks. So it appears your subdomain does not exist. If you give me an email in the account I can find the correct subdomain value for you.

Login and download image with C#

I need some images from a portal and they are only accessible if I login to the portal.
I need to do it with a C# program. I don't know what username field and password field are, because they use POST method. After loging in I want to enter some URLs that contain the images I want.
What should I do?
For logging in I'm using:
HttpWebRequest httpWReq = (HttpWebRequest)WebRequest.Create(#"http://mysite.com");
ASCIIEncoding encoding = new ASCIIEncoding();
string postData = "UsernameFieldName=Something";
postData += "&PasswordFieldName=SomethingElse";
byte[] data = encoding.GetBytes(postData);
httpWReq.Method = "POST";
httpWReq.ContentType = "application/x-www-form-urlencoded";
httpWReq.ContentLength = data.Length;
using (Stream stream = httpWReq.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}
HttpWebResponse response = (HttpWebResponse)httpWReq.GetResponse();
string responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
Then for downloading image which is in another page I use:
using (var client = new WebClient())
{
string FileName = #"image.jpg";
client.DownloadFile("http://mysite.com/Image?imgCode=12345", FileName);
}

I don’t have a complete solution but here are some details to get you started.
Figure out what are post field simply by looking at page source
Once you send request for login you’ll also need to find a way to accept authentication cookie and then send it in all subsequent requests because their application most probably uses cookies.
After you are logged in you can download images like this
string imageFile = #"c:\image.jpg";
using (System.Net.WebClient client = new System.Net.WebClient())
{
client.DownloadFile("http://www.somewebsite.com/someimage.jpg", imageFile);
}
Here are couple examples to get you started with http posts in C#
HTTP request with post

How do you login to a webpage and retrieve its content in C#?

How do you login to a webpage and retrieve its content in C#?

That depends on what's required to log in. You could use a webclient to send the login credentials to the server's login page (via whatever method is required, GET or POST), but that wouldn't persist a cookie. There is a way to get a webclient to handle cookies, so you could just POST the login info to the server, then request the page you want with the same webclient, then do whatever you want with the page.

Look at System.Net.WebClient, or for more advanced requirements System.Net.HttpWebRequest/System.Net.HttpWebResponse.
As for actually applying these: you'll have to study the html source of each page you want to scrape in order to learn exactly what Http requests it's expecting.

How do you mean "login"?
If the subfolder is protected on the OS level, and the browser pops of a login dialog when you go there, you will need to set the Credentials property on the HttpWebRequest.
If the website has it's own cookie-based membership/login system, you will have to use HttpWebRequest to first response to the login form.

string postData = "userid=ducon";
postData += "&username=camarche" ;
byte[] data = Encoding.ASCII.GetBytes(postData);
WebRequest req = WebRequest.Create(
URL);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = data.Length;
Stream newStream = req.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
StreamReader reader = new StreamReader(req.GetResponse().GetResponseStream(), System.Text.Encoding.GetEncoding("iso-8859-1"));
string coco = reader.ReadToEnd();

Use the WebClient class.
Dim Html As String
Using Client As New System.Net.WebClient()
Html = Client.DownloadString("http://www.google.com")
End Using

You can use the build in WebClient Object instead of crating the request yourself.
WebClient wc = new WebClient();
wc.Credentials = new NetworkCredential("username", "password");
string url = "http://foo.com";
try
{
using (Stream stream = wc.OpenRead(new Uri(url)))
{
using (StreamReader reader = new StreamReader(stream))
{
return reader.ReadToEnd();
}
}
}
catch (WebException e)
{
//Error handeling
}

Try this:
public string GetContent(string url)
{
using (System.Net.WebClient client =new System.Net.WebClient())
{
return client.DownloadString(url);
}
}