I'm working on a quick wrapper for the skydrive API in C#, but running into issues with downloading a file. For the first part of the file, everything comes through fine, but then there start to be differences in the file and shortly thereafter everything becomes null. I'm fairly sure that it's just me not reading the stream correctly.
This is the code I'm using to download the file:
public const string ApiVersion = "v5.0";
public const string BaseUrl = "https://apis.live.net/" + ApiVersion + "/";
public SkyDriveFile DownloadFile(SkyDriveFile file)
{
string uri = BaseUrl + file.ID + "/content";
byte[] contents = GetResponse(uri);
file.Contents = contents;
return file;
}
public byte[] GetResponse(string url)
{
checkToken();
Uri requestUri = new Uri(url + "?access_token=" + HttpUtility.UrlEncode(token.AccessToken));
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUri);
request.Method = WebRequestMethods.Http.Get;
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
byte[] contents = new byte[response.ContentLength];
responseStream.Read(contents, 0, (int)response.ContentLength);
return contents;
}
This is the image file I'm trying to download
And this is the image I am getting
These two images lead me to believe that I'm not waiting for the response to finish coming through, because the content-length is the same as the size of the image I'm expecting, but I'm not sure how to make my code wait for the entire response to come through or even really if that's the approach I need to take.
Here's my test code in case it's helpful
[TestMethod]
public void CanUploadAndDownloadFile()
{
var api = GetApi();
SkyDriveFolder folder = api.CreateFolder(null, "TestFolder", "Test Folder");
SkyDriveFile file = api.UploadFile(folder, TestImageFile, "TestImage.png");
file = api.DownloadFile(file);
api.DeleteFolder(folder);
byte[] contents = new byte[new FileInfo(TestImageFile).Length];
using (FileStream fstream = new FileStream(TestImageFile, FileMode.Open))
{
fstream.Read(contents, 0, contents.Length);
}
using (FileStream fstream = new FileStream(TestImageFile + "2", FileMode.CreateNew))
{
fstream.Write(file.Contents, 0, file.Contents.Length);
}
Assert.AreEqual(contents.Length, file.Contents.Length);
bool sameData = true;
for (int i = 0; i < contents.Length && sameData; i++)
{
sameData = contents[i] == file.Contents[i];
}
Assert.IsTrue(sameData);
}
It fails at Assert.IsTrue(sameData);
This is because you don't check the return value of responseStream.Read(contents, 0, (int)response.ContentLength);. Read doesn't ensure that it will read response.ContentLength bytes. Instead it returns the number of bytes read. You can use a loop or stream.CopyTo there.
Something like this:
WebResponse response = request.GetResponse();
MemoryStream m = new MemoryStream();
response.GetResponseStream().CopyTo(m);
byte[] contents = m.ToArray();
As LB already said, you need to continue to call Read() until you have read the entire stream.
Although Stream.CopyTo will copy the entire stream it does not ensure that read the number of bytes expected. The following method will solve this and raise an IOException if it does not read the length specified...
public static void Copy(Stream input, Stream output, long length)
{
byte[] bytes = new byte[65536];
long bytesRead = 0;
int len = 0;
while (0 != (len = input.Read(bytes, 0, Math.Min(bytes.Length, (int)Math.Min(int.MaxValue, length - bytesRead)))))
{
output.Write(bytes, 0, len);
bytesRead = bytesRead + len;
}
output.Flush();
if (bytesRead != length)
throw new IOException();
}
Related
I have a below scenario.
Client send request to Server-1 for file download
Server-1 send request to Server-2 for file.
To make this work I need to create a mechanism where once client send request to the Server-1, Server-1 will request to Server-2 which will send file as response output-stream in chunks. Server-1 will send this file chunks to client browser continuously as it keep receiving from server-2.
I have done code as below, theoretically it looks fine but still it is not working.
It is not downloading entire file in client browser, it seems like last chunk is not transferred to the Server-1 or it is not downloading to client browser from Server-1
Server-1 Code (Where client request for File download)
private void ProccesBufferedResponse(HttpWebRequest webRequest, HttpContext context)
{
char[] responseChars = null;
byte[] buffer = null;
if (webRequest == null)
logger.Error("Request string is null for Perfios Docs Download at ProccesBufferedResponse()");
context.Response.Buffer = false;
context.Response.BufferOutput = false;
try
{
WebResponse webResponse = webRequest.GetResponse();
context.Response.ContentType = "application/pdf";
context.Response.AddHeader("Content-disposition", webResponse.Headers["Content-disposition"]);
StreamReader responseStream = new StreamReader(webResponse.GetResponseStream());
while (!responseStream.EndOfStream)
{
responseChars = new char[responseStream.ToString().ToCharArray().Length];
responseStream.Read(responseChars, 0, responseChars.Length);
buffer = Encoding.ASCII.GetBytes(responseChars);
context.Response.Clear();
context.Response.OutputStream.Write(buffer, 0, buffer.Length);
context.Response.Flush();
}
}
catch (Exception ex)
{
throw;
}
finally
{
context.Response.Flush();
context.Response.End();
}
}
Server-2 Code (Where Server-1 will send request for file)
private void DownloadInstaPerfiosDoc(int CompanyID, string fileName, string Foldertype)
{
string folderPath;
string FilePath;
int chunkSize = 1024;
int startIndex = 0;
int endIndex = 0;
int length = 0;
byte[] bytes = null;
DirectoryInfo dir;
folderPath = GetDocumentDirectory(CompanyID, Foldertype);
FilePath = folderPath + "\\" + fileName;
dir = new DirectoryInfo(folderPath);
HttpContext.Current.Response.Buffer = false;
HttpContext.Current.Response.BufferOutput = false;
if (dir.Exists && dir.GetFiles().Length > 0)
{
foreach (var file in dir.GetFiles(fileName))
{
FilePath = folderPath + "\\" + file.Name;
FileStream fsReader = new FileStream(FilePath, FileMode.Open, FileAccess.Read);
HttpContext.Current.Response.ContentType = "application/pdf";
HttpContext.Current.Response.AddHeader("Content-disposition", string.Format("attachment; filename = \"{0}\"", fileName));
int totalChunks = (int)Math.Ceiling((double)fsReader.Length / chunkSize);
for (int i = 0; i < totalChunks; i++)
{
startIndex = i * chunkSize;
if (startIndex + chunkSize > fsReader.Length)
endIndex = (int)fsReader.Length;
else
endIndex = startIndex + chunkSize;
length = (int)endIndex - startIndex;
bytes = new byte[length];
fsReader.Read(bytes, 0, bytes.Length);
HttpContext.Current.Response.Clear();
HttpContext.Current.Response.OutputStream.Write(bytes, 0, bytes.Length);
HttpContext.Current.Response.Flush();
}
}
}
}
Please help me to resolve this issue.
It is possible and feasible. I'll give a pseudo procedure for you to understand the overall idea.
Server1
download action gets hit
create a request to server2
get the response stream of your server2 request
read the response stream in desired chunk sizes until it's consumed completely
write each chunk (as soon as you read) to current response stream
Server2
download action gets hit
write your stream onto your current response stream however you like
I need to download large json files from url. It uses post method, with parameter, username, password, etc.
As it takes quite long time, I try to put a progressbar, so users can see how far it has gone, and how much is left.
Problem
I am not able to retrieve the content length of the Json file from url before the download. Please see error commented in the code below. Any suggestions ?
public void downloadJson(string Url, string UserName, string Password, string FileDownload)
{
HttpWebRequest httpRequest;
HttpWebResponse httpResponse;
int Size;
var json = string.Format("{{\"user\":\"{0}\",\"pwd\":\"{1}\",\"DS\":\"KG\"}}", UserName, Password);
httpRequest = (HttpWebRequest)WebRequest.Create(Url);
httpRequest.Method = WebRequestMethods.Http.Post;
httpRequest.ContentLength = json.Length;
httpRequest.ContentType = "application/json";
httpRequest.Timeout = 600 * 60 * 1000;
var data = Encoding.ASCII.GetBytes(json); // or UTF8
using (var s = httpRequest.GetRequestStream())
{
s.Write(data, 0, data.Length);
s.Close();
}
httpResponse = (HttpWebResponse)httpRequest.GetResponse();
Size = (int)httpResponse.ContentLength;
Stream rs = httpResponse.GetResponseStream();
//**********************************************
//Here is the error
//the below progress bar would never work
//Because Size returned from above is always -1
//**********************************************
progressBar.Invoke((MethodInvoker)(() => progressBar.Maximum = Size));
using (FileStream fs = File.Create(FileDownload))
{
byte[] buffer = new byte[16 * 1024];
int read;
int position;
using (rs)
{
while ((read = rs.Read(buffer, 0, buffer.Length)) > 0)
{
fs.Write(buffer, 0, read);
position = (int)fs.Position;
progressBar.Invoke((MethodInvoker)(() => progressBar.Value = position));
Console.WriteLine ("Bytes Received: " + position.ToString());
}
}
fs.Close();
}
The error could be here;
Size = (int)httpResponse.ContentLength;
I believe when you declared like
int Size;
You wanted to assign the integer datatype Size
But it seems like Size is not behaving like an integer datatype.
Try changing Size to something like siz and try it again.
I am trying to build an application that downloads a small binary file (20-25 KB) from a custom webserver using httpwebrequests.
This is the server-side code:
Stream UpdateRequest = context.Request.InputStream;
byte[] UpdateContent = new byte[context.Request.ContentLength64];
UpdateRequest.Read(UpdateContent, 0, UpdateContent.Length);
String remoteVersion = "";
for (int i = 0;i < UpdateContent.Length;i++) { //check if update is necessary
remoteVersion += (char)UpdateContent[i];
}
byte[] UpdateRequestResponse;
if (remoteVersion == remotePluginVersion) {
UpdateRequestResponse = new byte[1];
UpdateRequestResponse[0] = 0; //respond with a single byte set to 0 if no update is required
} else {
FileInfo info = new FileInfo(Path.Combine(Directory.GetCurrentDirectory(), "remote logs", "PointAwarder.dll"));
UpdateRequestResponse = File.ReadAllBytes(Path.Combine(Directory.GetCurrentDirectory(), "remote logs", "PointAwarder.dll"));
//respond with the updated file otherwise
}
//this byte is past the threshold and will not be the same in the version the client recieves
Console.WriteLine("5000th byte: " + UpdateRequestResponse[5000]);
//send the response
context.Response.ContentLength64 = UpdateRequestResponse.Length;
context.Response.OutputStream.Write(UpdateRequestResponse, 0, UpdateRequestResponse.Length);
context.Response.Close();
After this the array UpdateRequestResponse contains the entire file and has been sent to the client.
The client runs this code:
//create the request
WebRequest request = WebRequest.Create(url + "pluginUpdate");
request.Method = "POST";
//create a byte array of the current version
byte[] requestContentTemp = version.ToByteArray();
int count = 0;
for (int i = 0; i < requestContentTemp.Length; i++) {
if (requestContentTemp[i] != 0) {
count++;
}
}
byte[] requestContent = new byte[count];
for (int i = 0, j = 0; i < requestContentTemp.Length; i++) {
if (requestContentTemp[i] != 0) {
requestContent[j] = requestContentTemp[i];
j++;
}
}
//send the current version
request.ContentLength = requestContent.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(requestContent, 0, requestContent.Length);
dataStream.Close();
//get and read the response
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
byte[] responseBytes = new byte[response.ContentLength];
responseStream.Read(responseBytes, 0, (int)response.ContentLength);
responseStream.Close();
response.Close();
//if the response containd a single 0 we are up-to-date, otherwise write the content of the response to file
if (responseBytes[0] != 0 || response.ContentLength > 1) {
BinaryWriter writer = new BinaryWriter(File.Open(Path.Combine(Directory.GetCurrentDirectory(), "ServerPlugins", "PointAwarder.dll"), FileMode.Create));
writer.BaseStream.Write(responseBytes, 0, responseBytes.Length);
writer.Close();
TShockAPI.Commands.HandleCommand(TSPlayer.Server, "/reload");
}
The byte array responseBytes on the client should be identical to the array UpdateRequestResponse on the server, but it isn't. after about 4000 bytes every byte after that is set to 0 rather than what it should be (responseBytes[3985] is the last non-zero byte).
Does this happen because httpWebRequest has a size limit? I can't see any bug in my code that could be causing it and the same code works in other instances where I only have to pass around short sequences of data (less than 100 bytes).
The MSDN pages don't mention any size limit like this.
It's not that it has any artificial limit, this is a byproduct of the Streaming nature of what you're attempting to do. I have a feeling the following line is the offender:
responseStream.Read(responseBytes, 0, (int)response.ContentLength);
I've had this issue in the past (with TCP streams), it doesn't read all of the contents of the array, because they haven't all been sent over the wire yet. This is what I would try instead.
for (int i = 0; i < response.ContentLength; i++)
{
responseBytes[i] = responseStream.ReadByte();
}
That way, it will make sure to read all the way until the end of the stream.
EDIT
usr's BinaryReader based solution is much more efficient. Here is the relevant solution:
BinaryReader binReader = new BinaryReader(responseStream);
const int bufferSize = 4096;
byte[] responseBytes;
using (MemoryStream ms = new MemoryStream())
{
byte[] buffer = new byte[bufferSize];
int count;
while ((count = binReader.Read(buffer, 0, buffer.Length)) != 0)
ms.Write(buffer, 0, count);
responseBytes = ms.ToArray();
}
You are assuming that Read is reading as many bytes as you request. But the requested count is just an upper limit. You must tolerate reading small chunks.
You can use var bytes = new BinaryReader(myStream).ReadBytes(count); to read an exact number. Don't call ReadByte too often because that is very CPU intensive.
The best solution would be to step away from the fairly manual HttpWebRequest and use HttpClient or WebClient. All of this is automated for you and you get back a byte[].
I have code which downloads file(zip file) from ftp server. But when I change files inside zip file, it is downloading from memory. Telling that I mean when I download zip file in which there are file 1, file 2, file 3. Next time I change zip file inside with file 1, file 2, file 3(but new data in files) and upload it to ftp. When I download from FTP directly using WS_FTP Pro I can see new files. But when I use my code to download zip file I get that file from memory. How can I refresh memory stream so when I download new zip file, I get new files inside zip file.
Mн code is.
public static bool downloadFromWeb(string URL, string file, string targetFolder)
{
try
{
byte[] downloadedData;
downloadedData = new byte[0];
//open a data stream from the supplied URL
WebRequest webReq = WebRequest.Create(URL + file);
WebResponse webResponse = webReq.GetResponse();
Stream dataStream = webResponse.GetResponseStream();
//Download the data in chuncks
byte[] dataBuffer = new byte[1024];
//Get the total size of the download
int dataLength = (int)webResponse.ContentLength;
//lets declare our downloaded bytes event args
ByteArgs byteArgs = new ByteArgs();
byteArgs.downloaded = 0;
byteArgs.total = dataLength;
//we need to test for a null as if an event is not consumed we will get an exception
if (bytesDownloaded != null) bytesDownloaded(byteArgs);
//Download the data
MemoryStream memoryStream = new MemoryStream();
memoryStream.SetLength(0);
while (true)
{
//Let's try and read the data
int bytesFromStream = dataStream.Read(dataBuffer, 0, dataBuffer.Length);
if (bytesFromStream == 0)
{
byteArgs.downloaded = dataLength;
byteArgs.total = dataLength;
if (bytesDownloaded != null) bytesDownloaded(byteArgs);
//Download complete
break;
}
else
{
//Write the downloaded data
memoryStream.Write(dataBuffer, 0, bytesFromStream);
byteArgs.downloaded = bytesFromStream;
byteArgs.total = dataLength;
if (bytesDownloaded != null) bytesDownloaded(byteArgs);
}
}
//Convert the downloaded stream to a byte array
downloadedData = memoryStream.ToArray();
//Release resources
dataStream.Close();
memoryStream.Close();
//Write bytes to the specified file
FileStream newFile = new FileStream(targetFolder + file, FileMode.Create);
newFile.Write(downloadedData, 0, downloadedData.Length);
newFile.Close();
return true;
}
catch (Exception)
{
//We may not be connected to the internet
//Or the URL may be incorrect
return false;
}
}
Please indicate me where I should change to download always new zip file from FTP.
I have added couple strings of code in my previous code. It tells browser not to cache. Here is my code with changes.
public static bool downloadFromWeb(string URL, string file, string targetFolder)
{
try
{
byte[] downloadedData;
downloadedData = new byte[0];
// Set a default policy level for the "http:" and "https" schemes.
HttpRequestCachePolicy policy = new HttpRequestCachePolicy(HttpRequestCacheLevel.Default);
HttpWebRequest.DefaultCachePolicy = policy;
//open a data stream from the supplied URL
WebRequest webReq = WebRequest.Create(URL + file);
// Define a cache policy for this request only.
HttpRequestCachePolicy noCachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.NoCacheNoStore);
webReq.CachePolicy = noCachePolicy;
WebResponse webResponse = webReq.GetResponse();
Stream dataStream = webResponse.GetResponseStream();
//Download the data in chuncks
byte[] dataBuffer = new byte[1024];
//Get the total size of the download
int dataLength = (int)webResponse.ContentLength;
//lets declare our downloaded bytes event args
ByteArgs byteArgs = new ByteArgs();
byteArgs.downloaded = 0;
byteArgs.total = dataLength;
//we need to test for a null as if an event is not consumed we will get an exception
if (bytesDownloaded != null) bytesDownloaded(byteArgs);
//Download the data
MemoryStream memoryStream = new MemoryStream();
memoryStream.SetLength(0);
while (true)
{
//Let's try and read the data
int bytesFromStream = dataStream.Read(dataBuffer, 0, dataBuffer.Length);
if (bytesFromStream == 0)
{
byteArgs.downloaded = dataLength;
byteArgs.total = dataLength;
if (bytesDownloaded != null) bytesDownloaded(byteArgs);
//Download complete
break;
}
else
{
//Write the downloaded data
memoryStream.Write(dataBuffer, 0, bytesFromStream);
byteArgs.downloaded = bytesFromStream;
byteArgs.total = dataLength;
if (bytesDownloaded != null) bytesDownloaded(byteArgs);
}
}
//Convert the downloaded stream to a byte array
downloadedData = memoryStream.ToArray();
//Release resources
dataStream.Close();
memoryStream.Close();
//Write bytes to the specified file
FileStream newFile = new FileStream(targetFolder + file, FileMode.Create);
newFile.Write(downloadedData, 0, downloadedData.Length);
newFile.Close();
return true;
}
catch (Exception)
{
//We may not be connected to the internet
//Or the URL may be incorrect
return false;
}
}
I believe your get-request is being cached by the WebRequest. Try adding a random query parameter and see if it helps.
WebRequest webReq = WebRequest.Create(URL + file + "?nocache=" + DateTime.Now.Ticks.ToString());
i am using: `private void get_stocks_data()
{
byte[] result;
byte[] buffer = new byte[4096];
WebRequest wr = WebRequest.Create("http://www.tase.co.il/TASE/Pages/ExcelExport.aspx?sn=he-IL_ds&enumTblType=AllSecurities&Columns=he-IL_Columns&Titles=he-IL_Titles&TblId=0&ExportType=1");
using (WebResponse response = wr.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
using (MemoryStream memoryStream = new MemoryStream())
{
int count = 0;
do
{
count = responseStream.Read(buffer, 0, buffer.Length);
memoryStream.Write(buffer, 0, count);
} while (count != 0);
result = memoryStream.ToArray();
write_data_to_excel(result);
}
}
}`
to download the excel file,
And this method to fill the file on my computer:
private void write_data_to_excel(byte[] input)
{
StreamWriter str = new StreamWriter("stockdata.xls");
for (int i = 0; input.Length > i; i++)
{
str.WriteLine(input[i].ToString());
}
str.Close();
}
The result is that i get a lot of numbers...
What am i doing wrong? the file i am downloadin is excel version 2003, on my computer i have 2007...
Thanks.
I would suggest that you use WebClient.DownloadFile() instead.
This is a higher level method that will abstract from creating the request manually, dealing with encoding, etc.
Problem is in your Write_data_to_excel function
as you are using StreamWriter.WriteLine method it needs string,
you are passing byte as string so your binary value say 10 will be now string 10
try FileStream f = File.OpenWrite("stockdata.xlsx");
f.Write(input,0,input.Length); this will work.