Develop Reference

How to Retrieve file from URL which Requires Authentication?

I'm making a tool in Unity to retrieve data from a server. The server's interface can provide URLs that we can later click on which will return an XML or CSV file with the results of that query from that server. But, it requires Basic Authentication. When clicking the links, it simply pops up a login screen before giving me the results. If I try what I [think] I know in Unity (starting with WebRequest.GetResponse()) it simply fails and says I am not authorized. It does not show the popup for authentication. So how do I let that login popup appear when accessing with Unity and await the login results to get the file? Or is there some standardized way to provide that info in the link itself?

Here is some code that should you get started. Just fill in the request link and username, password. please see the comments in the code to see what it does.
//try just in case something went wrong whith calling the api
try
{
//Use using so that if the code end the client disposes it self
using (HttpClient client = new HttpClient())
{
//Setup authentication information
string yourusername = "username";
string yourpwd = "password";
//this is when you expect json to return from the api
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
//add the authentication to the request
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic",
Convert.ToBase64String(
System.Text.ASCIIEncoding.ASCII.GetBytes($"{yourusername}:{yourpwd}")));
//api link used to make the call
var requestLink = $"apiLink";
using (HttpResponseMessage response = client.GetAsync(requestLink).Result)
{
//Make sure the request was successfull before proceding
response.EnsureSuccessStatusCode();
//Get response from website and convert to a string
string responseBody = response.Content.ReadAsStringAsync().Result;
//now you have the results
}
}
}
//Catch the exception if something went from and show it!
catch (Exception)
{
throw;
}

This is what I ended up going with after looking at the comments above. Let me know if I'm doing anything terribly inefficient!
String username = "Superman"; // Obviously handled secretly
String pw = "ILoveLex4evar!"; // Obviously handled secretly
String url = "https://www.SuperSecretServer.com/123&stuff=?uhh";
String encoded = System.Convert.ToBase64String(System.Text.Encoding.GetEncoding("ISO-8859-1").GetBytes(username + ":" + pw));
CookieContainer myContainer = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Headers.Add("Authorization", "Basic " + encoded);
try
{
using (WebResponse response = request.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
using (FileStream xml = File.Create("filepath/filename.xml"))
{
byte[] buffer = new byte[BufferSize];
int read;
while ((read = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
xml.Write(buffer, 0, read);
}
}
}
}
}

C#: HttpWebRequest POST data not working

I am developing a C# wpf application that has a functionality of logging into my website and download the file. This said website has an Authorize attribute on its action. I need 2 cookies for me to able to download the file, first cookie is for me to log in, second cookie(which is provided after successful log in) is for me to download the file. So i came up with the flow of keeping my cookies after my httpwebrequest/httpwebresponse. I am looking at my posting flow as maybe it is the problem. Here is my code.
void externalloginanddownload()
{
string pageSource = string.Empty;
CookieContainer cookies = new CookieContainer();
HttpWebRequest getrequest = (HttpWebRequest)WebRequest.Create("login uri");
getrequest.CookieContainer = cookies;
getrequest.Method = "GET";
getrequest.AllowAutoRedirect = false;
HttpWebResponse getresponse = (HttpWebResponse)getrequest.GetResponse();
using (StreamReader sr = new StreamReader(getresponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
var values = new NameValueCollection
{
{"Username", "username"},
{"Password", "password"},
{ "Remember me?","False"},
};
var parameters = new StringBuilder();
foreach (string key in values.Keys)
{
parameters.AppendFormat("{0}={1}&",
HttpUtility.UrlEncode(key),
HttpUtility.UrlEncode(values[key]));
}
parameters.Length -= 1;
HttpWebRequest postrequest = (HttpWebRequest)WebRequest.Create("login uri");
postrequest.CookieContainer = cookies;
postrequest.Method = "POST";
using (var writer = new StreamWriter(postrequest.GetRequestStream()))
{
writer.Write(parameters.ToString());
}
using (WebResponse response = postrequest.GetResponse()) // the error 500 occurs here
{
using (var streamReader = new StreamReader(response.GetResponseStream()))
{
string html = streamReader.ReadToEnd();
}
}
}

When you get the WebResponse, the cookies returned will be in the response, not in the request (oddly enough, even though you need to CookieContainer on the request).
You will need to add the cookies from the response object to your CookieContainer, so it gets sent on the next request.
One simple way:
for(var cookie in getresponse.Cookies)
cookies.Add(cookie)
Since the cookies in response is already a cookies container, you can do this (might help to check for null in case all cookies were already there)
if (response.Cookies != null) cookies.Add(response.Cookies)
You may also have trouble with your POST as you need to set ContentType and length:
myWebRequest.ContentLength = parameters.Length;
myWebRequest.AllowWriteStreamBuffering = true;
If you have any multibyte characters to think about, you may have to address that as well by setting the encoding to UTF-8 on the request and the stringbuilder, and converting string to bytes and using that length.
Another tip: some web server code chokes if there is no user agent. Try:
myWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
And just in case you have any multibyte characters, it is better to do this:
var databytes = System.Text.Encoding.UTF8.GetBytes(parameters.ToString());
myWebRequest.ContentLength = databytes.Length;
myWebRequest.ContentType = "application/x-www-form-urlencoded; charset=utf-8";
using (var stream = myWebRequest.GetRequestStream())
{
stream.Write(databytes, 0, databytes.Length);
}

In C# Application (Server side Web API) Enable the C++ Exception and Common Language Run time Exceptions using (Ctrl+Alt+E) what is the Server side Exception it's throw.
First you check data is binding Properly. After you can see what it is Exact Exception. the Internal Server Error Mostly throw the data is not correct format and not properly managed Exception.

Microsoft Translator Bad request (400) issue

I've seen threads on this issue but my problem is particularly confusing. I have a free 2 million character subscription, a valid client id and secret. When I run my code I get to call the API a few times successfully (the most I've seen is 75 consecutive successful calls). Then every other call returns a Bad request response: The remote server returned an error: (400) Bad Request.
I create the token once with my credentials and never create it again. I loop through a file, parse it, and submit every parsed string for translation by calling the API. It seems that I reach some sort of limit that I'm now aware of.
When looking at my account, it doesn't seem to be discounting the characters that I've translated already which would make me highly suspicious that I have the wrong credentials when creating the token. I quadruple-checked that and everything seems to be ok.
Any guidance on what I may be missing here would be much appreciated.
Here's the code that creates the token. I do think though that there may be an unknown limitation that I'm not aware of with the free subscription.
static void gettoken()
{
//Get access token
string clientID = "my client id";
string clientSecret = "my secret";
String strTranslatorAccessURI = "https://datamarket.accesscontrol.windows.net/v2/OAuth2-13";
String strRequestDetails = string.Format("grant_type=client_credentials&client_id={0}&client_secret={1}&scope=http://api.microsofttranslator.com", clientID, clientSecret);
System.Net.WebRequest webRequest = System.Net.WebRequest.Create(strTranslatorAccessURI);
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.Method = "POST";
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(strRequestDetails);
webRequest.ContentLength = bytes.Length;
using (System.IO.Stream outputStream = webRequest.GetRequestStream())
{
outputStream.Write(bytes, 0, bytes.Length);
}
System.Net.WebResponse webResponse = webRequest.GetResponse();
System.Runtime.Serialization.Json.DataContractJsonSerializer serializer = new System.Runtime.Serialization.Json.DataContractJsonSerializer(typeof(AdmAccessToken));
AdmAccessToken token = (AdmAccessToken)serializer.ReadObject(webResponse.GetResponseStream());
MyGlobals.headerValue = "Bearer " + token.access_token;
}
And here's the code that calls the API itself. I call the API method from a loop.
static void RunBing(string sterm)
{
//Submit the translation request
string txtToTranslate = sterm;
string uri = "http://api.microsofttranslator.com/v2/Http.svc/Translate?text=" + txtToTranslate + "&from=en&to=es";
System.Net.WebRequest translationWebRequest = System.Net.WebRequest.Create(uri);
translationWebRequest.Headers.Add("Authorization", MyGlobals.headerValue);
System.Net.WebResponse response = null;
try {
response = translationWebRequest.GetResponse();
}
catch (Exception e)
{
Console.WriteLine("Term failed: " + sterm);
Console.WriteLine(e);
return;
}
System.IO.Stream stream = response.GetResponseStream();
System.Text.Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
System.IO.StreamReader translatedStream = new System.IO.StreamReader(stream, encode);
System.Xml.XmlDocument xTranslation = new System.Xml.XmlDocument();
xTranslation.LoadXml(translatedStream.ReadToEnd());
MyGlobals.xlation = xTranslation.InnerText;
}
After several successful calls to the API, I start to get the following message:
System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.HttpWebRequest.GetResponse()
at Translate.TranslateText.Program.RunBing(String sterm)

HttpWebRequest-The remote server returned an error: (400) Bad Request

I am getting The remote server returned an error: (400) Bad Request error while running the following code.
I am trying to upload xml file on the http server.
My xml file contains tag for the username,password and domain and when i am trying to connect is manually i am able to connect it,but using same credentials when i am trying to connect it through this code, i am getting 400 Bad Request error.
Please suggest me how to overcome this issue.
Thanks
`
public static void UploadHttp(string xml)
{
string txtResults = string.Empty;
try
{
string url = "http://my.server.com/upload.aspx ";
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.KeepAlive = false;
request.SendChunked = true;
request.AllowAutoRedirect = true;
request.Method = "Post";
request.ContentType = "text/xml";
var encoder = new UTF8Encoding();
var data = encoder.GetBytes(xml);
request.ContentLength = data.Length;
var reqStream = request.GetRequestStream();
reqStream.Write(data, 0, data.Length);
reqStream.Close();
WebResponse response = null;
response = request.GetResponse();
var reader = new StreamReader(response.GetResponseStream());
var str = reader.ReadToEnd();
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError)
{
HttpWebResponse err = ex.Response as HttpWebResponse;
if (err != null)
{
string htmlResponse = new StreamReader(err.GetResponseStream()).ReadToEnd();
txtResults = string.Format("{0} {1}", err.StatusDescription, htmlResponse);
}
}
else
{
}
}
catch (Exception ex)
{
txtResults = ex.ToString();
}
}`

Are you sure you should be using POST not PUT?
POST is usually used with application/x-www-urlencoded formats. If you are using a REST API, you should maybe be using PUT? If you are uploading a file you probably need to use multipart/form-data. Not always, but usually, that is the right thing to do..
Also you don't seem to be using the credentials to log in - you need to use the Credentials property of the HttpWebRequest object to send the username and password.

400 Bad request Error will be thrown due to incorrect authentication entries.
Check if your API URL is correct or wrong. Don't append or prepend spaces.
Verify that your username and password are valid. Please check any spelling mistake(s) while entering.
Note: Mostly due to Incorrect authentication entries due to spell changes will occur 400 Bad request.

What type of authentication do you use?
Send the credentials using the properties Ben said before and setup a cookie handler.
You already allow redirection, check your webserver if any redirection occurs (NTLM auth does for sure). If there is a redirection you need to store the session which is mostly stored in a session cookie.

//use "ASCII" or try with another encoding scheme instead of "UTF8".
using (StreamWriter postStream = new StreamWriter(request.GetRequestStream(), System.Text.Encoding.UTF8))
{
postStream.Write(postData);
postStream.Close();
}

Why does sending post data with WebRequest take so long?

I am currently creating a C# application to tie into a php / MySQL online system. The application needs to send post data to scripts and get the response.
When I send the following data
username=test&password=test
I get the following responses...
Starting request at 22/04/2010 12:15:42
Finished creating request : took 00:00:00.0570057
Transmitting data at 22/04/2010 12:15:42
Transmitted the data : took 00:00:06.9316931 <<--
Getting the response at 22/04/2010 12:15:49
Getting response 00:00:00.0360036
Finished response 00:00:00.0360036
Entire call took 00:00:07.0247024
As you can see it is taking 6 seconds to actually send the data to the script, I have done further testing bye sending data from telnet and by sending post data from a local file to the url and they dont even take a second so this is not a problem with the hosted script on the site.
Why is it taking 6 seconds to transmit the data when it is two simple strings?
I use a custom class to send the data
class httppostdata
{
WebRequest request;
WebResponse response;
public string senddata(string url, string postdata)
{
var start = DateTime.Now;
Console.WriteLine("Starting request at " + start.ToString());
// create the request to the url passed in the paramaters
request = (WebRequest)WebRequest.Create(url);
// set the method to post
request.Method = "POST";
// set the content type and the content length
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = postdata.Length;
// convert the post data into a byte array
byte[] byteData = Encoding.UTF8.GetBytes(postdata);
var end1 = DateTime.Now;
Console.WriteLine("Finished creating request : took " + (end1 - start));
var start2 = DateTime.Now;
Console.WriteLine("Transmitting data at " + start2.ToString());
// get the request stream and write the data to it
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteData, 0, byteData.Length);
dataStream.Close();
var end2 = DateTime.Now;
Console.WriteLine("Transmitted the data : took " + (end2 - start2));
// get the response
var start3 = DateTime.Now;
Console.WriteLine("Getting the response at " + start3.ToString());
response = request.GetResponse();
//Console.WriteLine(((WebResponse)response).StatusDescription);
dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
var end3 = DateTime.Now;
Console.WriteLine("Getting response " + (end3 - start3));
// read the response
string serverresponse = reader.ReadToEnd();
var end3a = DateTime.Now;
Console.WriteLine("Finished response " + (end3a - start3));
Console.WriteLine("Entire call took " + (end3a - start));
//Console.WriteLine(serverresponse);
reader.Close();
dataStream.Close();
response.Close();
return serverresponse;
}
}
And to call it I use
private void btnLogin_Click(object sender, EventArgs e)
{
// string postdata;
if (txtUsername.Text.Length < 3 || txtPassword.Text.Length < 3)
{
MessageBox.Show("Missing your username or password.");
}
else
{
string postdata = "username=" + txtUsername.Text +
"&password=" + txtPassword.Text;
httppostdata myPost = new httppostdata();
string response = myPost.senddata("http://www.domainname.com/scriptname.php", postdata);
MessageBox.Show(response);
}
}

Make sure you explicitly set the proxy property of the WebRequest to null or it will try to autodetect the proxy settings which can take some time.

Chances are that because, in your test, you only call this once, the delay you see is the C# code being JIT compiled.
A better test would be to call this twice, and discard the timings from the first time and see if they are better.
An even better test would be to discard the first set of timings, and then run this many times and take an average, although for a very loose "indicative" view, this is probably not necessary.
As an aside, for this sort of timing, you are better off using the System.Diagnostics.Stopwatch class over System.DateTime.
[EDIT]
Also, noting Mant101's suggestion about proxies, if the setting no proxy fails to resolve things, you may wish to set up Fiddler and set your request to use Fiddler as its proxy. This would allow you to intercept the actual http calls so you can get a better breakdown of the http call timings themselves from outside the framework.