Develop Reference

Issue in Resumable Upload on Google Drive

I am trying to upload large file on Google drive performing Resumable upload.
Here is the code flow
Step 1 : Creating file on Google Drive using Drive service and initiating the resumable upload session
using put request
String fileID = _DriveService.Files.Insert(googleFileBody).Execute().Id;
//Initiating resumable upload session
String UploadUrl = null;
String _putUrl = "https://www.googleapis.com/upload/drive/v2/files/" + fileID + "?uploadType=resumable";
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(_putUrl);
httpRequest.Headers["Authorization"] = "Bearer " + AccessToken;
httpRequest.Method = "PUT";
requestStream = httpRequest.GetRequestStream();
_webResponse = (HttpWebResponse)httpRequest.GetResponse();
if (_webResponse.StatusCode == HttpStatusCode.OK)
{
//Getting response OK
UploadUrl = _webResponse.Headers["Location"].ToString();
}
Step 2 : Uploading chunks to using UploadUrl . The byte array is in multiple of 256kb and call to this function is in the loop for every chunk
private void AppendFileData(byte[] chunk)
{
try
{
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(UploadUrl);
httpRequest.ContentLength = chunk.Length;
httpRequest.Headers["Content-Range"] = "bytes " + startOffset + "-" + endOffset+ "/" + sourceFileSize;
httpRequest.ContentType= MimeType;
httpRequest.Method = "PUT";
MemoryStream stream =new MemoryStream(chunk);
using (System.IO.Stream requestStream = httpRequest.GetRequestStream())
{
stream.CopyTo(requestStream);
requestStream.Flush();
requestStream.Close();
}
HttpWebResponse httpResponse = (HttpWebResponse)(httpRequest.GetResponse()); // Throws exception as
//System.Net.WebException: The remote server returned an error: (308) Resume Incomplete.
//at System.Net.HttpWebRequest.GetResponse()
// There is no data getting appended to file
// Still executing the append for remaining chunks
}
catch(System.Net.WebException ex)
{
}
}
For my last chunk which is not multiple of 256KB I am getting error as
System.Net.WebException: The remote server returned an error: (400)
Bad Request. at System.Net.HttpWebRequest.GetResponse()
What I am doing wrong in this code? Please suggest.
Thanks in advance
Mayuresh.

Try checking if the last chunk is passing the correct size and not the entire array as stated in this forum. Ali has stated in that forum that "one potential issue is this: if you are sending a byte array which is half empty for the last request (i.e. the buffer has been read in less than the chunk size)." Here is a sample implementation of resumable upload. Hope this helps.

HttpWebRequest not downloading file correctly

I can't use WebClient, before anyone suggests that because it makes my legit application seem like a virus to McAfee. So please don't suggest that.
I have a binary.txt file stored on my server. It is approximately 1,240kb. However, HttpWebRequest downloads random amounts from 1,300kb to 1,700kb.
HttpWebRequest httpRequest = (HttpWebRequest)
WebRequest.Create("http://deviantsmc.com/binary.txt");
httpRequest.Method = WebRequestMethods.Http.Get;
HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse();
Stream httpResponseStream = httpResponse.GetResponseStream();
byte[] buffer = new byte[1240];
int bytesRead = 0;
StringBuilder sb = new StringBuilder();
//FileStream fileStream = File.Create(#"tcontent.txt");
while ((bytesRead = httpResponseStream.Read(buffer, 0, 1240)) != 0)
{
sb.Append(Encoding.ASCII.GetString(buffer));
//fileStream.Write(buffer, 0, bytesRead);
}
File.WriteAllText(#"tcontent1.txt", sb.ToString());
(The contents of the binary.txt file on the server are in ASCII, therefore I get the Encoding string to ASCII as well).
This is how I encoded that text file (on that server)
My file is basically this:
byte[] bytes = File.ReadAllBytes("binary.txt");
String encBytes = Encoding.ASCII.GetString(bytes);
File.WriteAllText(file("binary.txt"), encBytes);
I contacted the AV company about the WebDownloader being seen as some malicious import in C#, but they didn't get back to me, so I'm forced to use HttpWebRequest.

If your only goal is to fetch that binary and write it to disk you can simply copy the stream to a file with the CopyTo method that exists on a Stream object.
Your file looks like a broken zip file btw, given the first characters that are PK, and is used in the zip specificaton.
From wikipedia:
Viewed as an ASCII string this reads "PK", the initials of the inventor Phil Katz. Thus, when a .ZIP file is viewed in a text editor the first two bytes of the file are usually "PK".
I used 7-zip to open your file as the Windows default didn't accept it as a valid file. It contains a manifest.mf file but the content itself seems missing. the file itself has a size of 1.269.519 bytes.
HttpWebRequest httpRequest = (HttpWebRequest)
WebRequest.Create("http://deviantsmc.com/binary.txt");
httpRequest.Method = WebRequestMethods.Http.Get;
HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse();
Stream httpResponseStream = httpResponse.GetResponseStream();
// create and open a FileStream, using calls dispose when done
using(var fs= File.Create(#"c:\temp\bin.7z"))
{
// Copy all bytes from the responsestream to the filestream
httpResponseStream.CopyTo(fs);
}

I think you should write the entire binary data to the local file, instead of encoding it segmentally, you can try this:
HttpWebRequest httpRequest = (HttpWebRequest)
WebRequest.Create("http://deviantsmc.com/binary.txt");
httpRequest.Method = WebRequestMethods.Http.Get;
HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse();
Stream httpResponseStream = httpResponse.GetResponseStream();
using (BinaryReader responseReader = new BinaryReader(httpResponseStream.GetResponseStream()))
{
byte[] bytes = responseReader.ReadBytes((int)response.ContentLength);
using (BinaryWriter sw = new BinaryWriter(File.OpenWrite("tcontent1.txt")))
{
sw.Write(bytes);
sw.Flush();
sw.Close();
}
}
Hope this can help

Upload to S3 from HTTPWebResponse.GetResponseStream() in c#

I am trying to upload from an HTTP stream directly to S3, without storing in memory or as a file first. I am already doing this with Rackspace Cloud Files as HTTP to HTTP, however the AWS authentication is beyond me so am trying to use the SDK.
The problem is the upload stream is failing with this exception:
"This stream does not support seek operations."
I've tried with PutObject and TransferUtility.Upload, both fail with the same thing.
Is there any way to stream into S3 as the stream comes in, rather than buffering the whole thing to a MemoryStream or FileStream?
or is there any good examples of doing the authentication into S3 request using HTTPWebRequest, so I can duplicate what I do with Cloud Files?
Edit: or is there a helper function in the AWSSDK for generating the authorization header?
CODE:
This is the failing S3 part (both methods included for completeness):
string uri = RSConnection.StorageUrl + "/" + container + "/" + file.SelectSingleNode("name").InnerText;
var req = (HttpWebRequest)WebRequest.Create(uri);
req.Headers.Add("X-Auth-Token", RSConnection.AuthToken);
req.Method = "GET";
using (var resp = req.GetResponse() as HttpWebResponse)
{
using (Stream stream = resp.GetResponseStream())
{
Amazon.S3.Transfer.TransferUtility trans = new Amazon.S3.Transfer.TransferUtility(S3Client);
trans.Upload(stream, config.Element("root").Element("S3BackupBucket").Value, container + file.SelectSingleNode("name").InnerText);
//Use EITHER the above OR the below
PutObjectRequest putReq = new PutObjectRequest();
putReq.WithBucketName(config.Element("root").Element("S3BackupBucket").Value);
putReq.WithKey(container + file.SelectSingleNode("name").InnerText);
putReq.WithInputStream(Amazon.S3.Util.AmazonS3Util.MakeStreamSeekable(stream));
putReq.WithMetaData("content-length", file.SelectSingleNode("bytes").InnerText);
using (S3Response putResp = S3Client.PutObject(putReq))
{
}
}
}
And this is how I do it successfully from S3 to Cloud Files:
using (GetObjectResponse getResponse = S3Client.GetObject(new GetObjectRequest().WithBucketName(bucket.BucketName).WithKey(file.Key)))
{
using (Stream s = getResponse.ResponseStream)
{
//We can stream right from s3 to CF, no need to store in memory or filesystem.
var req = (HttpWebRequest)WebRequest.Create(uri);
req.Headers.Add("X-Auth-Token", RSConnection.AuthToken);
req.Method = "PUT";
req.AllowWriteStreamBuffering = false;
if (req.ContentLength == -1L)
req.SendChunked = true;
using (Stream stream = req.GetRequestStream())
{
byte[] data = new byte[32768];
int bytesRead = 0;
while ((bytesRead = s.Read(data, 0, data.Length)) > 0)
{
stream.Write(data, 0, bytesRead);
}
stream.Flush();
stream.Close();
}
req.GetResponse().Close();
}
}

As no-one answering seems to have done it, I spent the time working it out based on guidance from Steve's answer:
In answer to this question "is there any good examples of doing the authentication into S3 request using HTTPWebRequest, so I can duplicate what I do with Cloud Files?", here is how to generate the auth header manually:
string today = String.Format("{0:ddd,' 'dd' 'MMM' 'yyyy' 'HH':'mm':'ss' 'zz00}", DateTime.Now);
string stringToSign = "PUT\n" +
"\n" +
file.SelectSingleNode("content_type").InnerText + "\n" +
"\n" +
"x-amz-date:" + today + "\n" +
"/" + strBucketName + "/" + strKey;
Encoding ae = new UTF8Encoding();
HMACSHA1 signature = new HMACSHA1(ae.GetBytes(AWSSecret));
string encodedCanonical = Convert.ToBase64String(signature.ComputeHash(ae.GetBytes(stringToSign)));
string authHeader = "AWS " + AWSKey + ":" + encodedCanonical;
string uriS3 = "https://" + strBucketName + ".s3.amazonaws.com/" + strKey;
var reqS3 = (HttpWebRequest)WebRequest.Create(uriS3);
reqS3.Headers.Add("Authorization", authHeader);
reqS3.Headers.Add("x-amz-date", today);
reqS3.ContentType = file.SelectSingleNode("content_type").InnerText;
reqS3.ContentLength = Convert.ToInt32(file.SelectSingleNode("bytes").InnerText);
reqS3.Method = "PUT";
Note the added x-amz-date header as HTTPWebRequest sends the date in a different format to what AWS is expecting.
From there it was just a case of repeating what I was already doing.

Take a look at Amazon S3 Authentication Tool for Curl. From that web page:
Curl is a popular command-line tool for interacting with HTTP
services. This Perl script calculates the proper signature, then calls
Curl with the appropriate arguments.
You could probably adapt it or its output for your use.

I think the problem is that according to the AWS Documentation Content-Length is required and you don't know what the length is until the stream has finished.
(I would guess the Amazon.S3.Util.AmazonS3Util.MakeStreamSeekable routine is reading the whole stream into memory to get around this problem which makes it unsuitable for your scenario.)
What you can do is read the file in chunks and upload them using MultiPart upload.
PS, I assume you know the C# source for the AWSSDK for dotnet is on Github.

This is a true hack (which would probably break with a new implementation of the AWSSDK), and it requires knowledge of the length of the file being requested, but if you wrap the response stream as shown with this class (a gist) as shown below:
long length = fileLength;
you can get file length in several ways. I am uploading from a dropbox link, so they give me the
length along with the url. Alternatively, you can perform a HEAD request and get the Content-Length.
string uri = RSConnection.StorageUrl + "/" + container + "/" + file.SelectSingleNode("name").InnerText;
var req = (HttpWebRequest)WebRequest.Create(uri);
req.Headers.Add("X-Auth-Token", RSConnection.AuthToken);
req.Method = "GET";
using (var resp = req.GetResponse() as HttpWebResponse)
{
using (Stream stream = resp.GetResponseStream())
{
//I haven't tested this path
Amazon.S3.Transfer.TransferUtility trans = new Amazon.S3.Transfer.TransferUtility(S3Client);
trans.Upload(new HttpResponseStream(stream, length), config.Element("root").Element("S3BackupBucket").Value, container + file.SelectSingleNode("name").InnerText);
//Use EITHER the above OR the below
//I have tested this with dropbox data
PutObjectRequest putReq = new PutObjectRequest();
putReq.WithBucketName(config.Element("root").Element("S3BackupBucket").Value);
putReq.WithKey(container + file.SelectSingleNode("name").InnerText);
putReq.WithInputStream(new HttpResponseStream(stream, length)));
//These are necessary for really large files to work
putReq.WithTimeout(System.Threading.Timeout.Infinite);
putReq.WithReadWriteTimeout(System.Thread.Timeout.Infinite);
using (S3Response putResp = S3Client.PutObject(putReq))
{
}
}
}
The hack is overriding the Position and Length properties, and returning 0 for Position{get}, noop'ing Position{set}, and returning the known length for Length.
I recognize that this might not work if you don't have the length or if the server providing the source does not support HEAD requests and Content-Length headers. I also realize it might not work if the reported Content-Length or the supplied length doesn't match the actual length of the file.
In my test, I also supply the Content-Type to the PutObjectRequest, but I don't that that is necessary.

As sgmoore said, the problem is that your content length is not seekable from the HTTP response. However HttpWebResponse does have a content length property available. So you can actually form your Http post request to S3 yourself instead of using the Amazon library.
Here's another Stackoverflow question that managed to do that with what looks like full code to me.

Copy a PDF Stream to File

I'm calling a routine in PHP (TCPDF) from C# via WebRequest using StreamReader. The PDF file is returned as a stream and stored in a string (obv). I know the data being returned to the string is actually a PDF file, as I've tested it in PHP. I'm having a hard time writing the string to a file and actually getting a valid PDF in C#. I know it has something to do with the way I'm trying to encode the file, but the several things I've tried have resulted in 'Not today, Padre' (i.e. they didn't work)
Here's the class I'm using to perform the request (thanks to user 'Paramiliar' for the example I'm using/borrowed/stole):
public class httpPostData
{
WebRequest request;
WebResponse response;
public string senddata(string url, string postdata)
{
// create the request to the url passed in the paramaters
request = (WebRequest)WebRequest.Create(url);
// set the method to POST
request.Method = "POST";
// set the content type and the content length
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = postdata.Length;
// convert the post data into a byte array
byte[] byteData = Encoding.UTF8.GetBytes(postdata);
// get the request stream and write the data to it
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteData, 0, byteData.Length);
dataStream.Close();
// get the response
response = request.GetResponse();
dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
// read the response
string serverresponse = reader.ReadToEnd();
//Console.WriteLine(serverresponse);
reader.Close();
dataStream.Close();
response.Close();
return serverresponse;
}
} // end class httpPostData
...and my call to it
httpPostData myPost = new httpPostData();
// postData defined (not shown)
string response = myPost.senddata("http://www.example.com/pdf.php", postData);
In case it isn't clear, I'm stuck writing string response to a valid .pdf file. I've tried this (Thanks to user Adrian):
static public void SaveStreamToFile(string fileFullPath, Stream stream)
{
if (stream.Length == 0) return;
// Create a FileStream object to write a stream to a file
using (FileStream fileStream = System.IO.File.Create(fileFullPath, (int)stream.Length))
{
// Fill the bytes[] array with the stream data
byte[] bytesInStream = new byte[stream.Length];
stream.Read(bytesInStream, 0, (int)bytesInStream.Length);
// Use FileStream object to write to the specified file
fileStream.Write(bytesInStream, 0, bytesInStream.Length);
}
}
..and the call to it:
string location = "C:\\myLocation\\";
SaveStreamToFile(location, response); // <<-- this throws an error b/c 'response' is a string, not a stream. New to C# and having some basic issues with things like this
I think I'm close...a nudge in the right direction would be greatly appreciated.

You can use WebClient. Use the method DownloadFile, or the async ones.
Have fun!
Fernando.-
Sorry, I haven't read your comments till now.
I guess you have already done this...
But this may help you (just replace urls and paths) (from: http://msdn.microsoft.com/en-us/library/ez801hhe.aspx )
string remoteUri = "http://www.contoso.com/library/homepage/images/";
string fileName = "ms-banner.gif", myStringWebResource = null;
// Create a new WebClient instance.
WebClient myWebClient = new WebClient();
// Concatenate the domain with the Web resource filename.
myStringWebResource = remoteUri + fileName;
Console.WriteLine("Downloading File \"{0}\" from \"{1}\" .......\n\n", fileName, myStringWebResource);
// Download the Web resource and save it into the current filesystem folder.
myWebClient.DownloadFile(myStringWebResource,fileName);
Console.WriteLine("Successfully Downloaded File \"{0}\" from \"{1}\"", fileName, myStringWebResource);
Console.WriteLine("\nDownloaded file saved in the following file system folder:\n\t" + Application.StartupPath);

HttpWebRequest POST Login error 405

Im trying to make a small login system but i have this problem that when the project runs on VS2010 it says :
The remote server returned an error: (405) Method Not Allowed.
Heres my code:
//Our URL
string uri = "https://************************/ValidateUsername";
//Our postvars
byte[] buffer = Encoding.ASCII.GetBytes( "username=user" );
//Initialization
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Accept = "application/json;odata=verbose";
request.Headers.Add("Language", "es-MX");
request.Headers.Add("Application", "-------------------");
request.Headers.Add("Version", "1.0");
//Our method is POST, otherway buffer would be useless
request.Method = "POST";
//We use form contentType, for the postvars
request.ContentType = "application/x-www-form-urlencoded";
//The lenght of the content its set by postvars (buffer) lenght
request.ContentLength = buffer.Length;
//We open a stream for writing the postvars
Stream PostData = request.GetRequestStream();
//Now we write, and afterwards, we close.
PostData.Write(buffer, 0, buffer.Length);
When i hover the pointer over "PostData" and then i go to the length and position attributes, i can read :
Length = 'PostData.Length' threw an exception of type 'System.NotSupportedException'
base {System.SystemException} = {"This stream does not support seek operations."}
im not sure if this is the real problem, but im trying to give the mos of information possible.
PostData.Close();
//Get the response Handle
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
//lets show info about the response
Console.WriteLine("Estatus de la respuesta:" + response.StatusCode);
Console.WriteLine("Servidor : " + response.Server);
//Now we read the response (the string), and output it
//Stream answer = response.GetResponseStream();
//StreamReader _answer = new StreamReader(answer);
//Console.WriteLine("Respuesta: " + _answer.ReadToEnd());
At some forum i read that maybe the "_answer.ReadToEnd()" could be the problem but even thought i commented , the problem still arises.
Im doing this so i can try it later on windows phone, dont know if im wasting my time , because they are not related.
I hope someone can help. Thanks in advance.