Develop Reference

How can i download a file without file name and file extension in the url

My problem is that I dont Know how i can download a File withknowing the file name or the file extension in the url, like this http://findicons.com/icon/download/235456/internet_download/128/png?id=235724
I hope you can help me

You could inspect the Content-Disposition response header using an HTTP request to get the filename. This would be a more general solution, so even if the filename is not contained in the URL, it would work:
var url = "http://findicons.com/icon/download/235456/internet_download/128/png?id=235724";
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
var fn = response.Headers["Content-Disposition"].Split(new string[] { "=" }, StringSplitOptions.None)[1];
string basePath = #"X:\Folder\SubFolder"; // Change accordingly...
var responseStream = response.GetResponseStream();
using (var fileStream = File.Create(Path.Combine(basePath, fn)))
{
responseStream.CopyTo(fileStream);
}
}
The above code uses certain methods and functions, you can find more information here:
HttpWebRequest - Usage example here
Saving a stream to a file - see this answer. Just note that when saving an HTTP response stream, you don't need to seek to the beginning, as it already is at the beginning and doing so will throw an exception. So, to be on the safe side, use it like I have in the code above.
Hope this answer helps you :)

It's possible to get the filename since the server is sending the Content-Disposition header. Here's a code example on how to get the filename using the HttpClient class:
var url = "http://findicons.com/icon/download/235456/internet_download/128/png?id=235724";
using (var client = new HttpClient())
using (var response = await client.GetAsync(url))
{
// make sure our request was successful
response.EnsureSuccessStatusCode();
// read the filename from the Content-Disposition header
var filename = response.Content.Headers.ContentDisposition.FileName;
// read the downloaded file data
var stream = await response.Content.ReadAsStreamAsync();
// Where you want the file to be saved
var destinationFile = Path.Combine("C:\\local\\directory", filename);
// write the steam content into a file
using (var fileStream = File.Create(destinationFile))
{
stream.CopyTo(fileStream);
}
}

I've had a hard time solving this issue myself and have come to a solution solving some issues in relation of obtaining the file name automatically.
Some headers does not include the Content Disposition as suggested in
https://stackoverflow.com/a/37228939/8805908, but the ones that do is still used.
I was wondering how Chrome, firefox etc. could obtain the name of the file though this information is not available through any header entry. I found that the links left without the information could be derived via its URL, from which i am using the code of:
http://codesnippets.fesslersoft.de/how-to-get-the-filename-of-url-in-c-and-vb-net/
My conclusion so far is; check the header for Content Disposition, if this does not contain any information check the URL for any file matches. So far I have not found a download link that I have not been able to retrieve the file name of.
I hope this may solve some issues.
--- Edit 12-06-2018
The solution using these methods are satisfying the following links:
5 test cases

Reading a web page in a foreign language with StreamReader

I'm trying to fetch a web page that is a mix of English and Korean. The browser can fetch and display the page just fine, but when I try to grab it programmatically I can't get the Korean characters to display properly.
I know that you can specify an Encoding in the StreamReader but I haven't found one that works yet.
This is the code that I'm using to read the response:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader sr = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding(response.CharacterSet));
response.CharacterSet returns UTF8. I've also tried all of the basic encoding options - ASCII, BigEndian, Default, Unicode, UTF32, UTF7, and manually adding Encoding.UTF8.
I've also tried going about it through the CultureInfo:
CultureInfo kr = CultureInfo.GetCultureInfo("ko");
StreamReader sr = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding(kr.TextInfo.ANSICodePage));
using both "ko" and "ko-KR". I get varied results from all these different types, but none of them are correct.
I've also tried the code page directly:
StreamReader sr = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding(949));
response.ContentEncoding returns an empty string. I'm running out of ideas.
Edit: Here is an example of what I'm expecting:
프로젝트:
and here is what I'm getting:
//ASCII == ??????
//BigEndian == ़汩湫â¨ç‰¥æ˜½âˆ¯æ©³â½¤ç°æ”
//Default == íâ€â€žÃ«Â¡Å“ì Â트:
//Unicode == íâ€â€žÃ«Â¡Å“ì Â트
//UTF32 == ���������ï
//UTF7 == 프로ì Â트
//UTF8 == 프로ì 트

FWIW: a stream reader is likely not going to work well.
Prefer using HttpWebRequest Class to do browser requests (or you will start feeling sorry very soon when you get 302 responses or gzipped and/or chunked encoding)
I promoted this to an answer, as it might very well be the problem you're having already. I don't know what the response you are getting looks like, of course

After transmission of text files blank lines appears in them

EDIT: Filezilla caused the problem, when i download files back from server it added new lines. I'm sorry for confusion.
This method upload files to ftp server and it's work fine, but in text files uploaded to server blank lines appear after every line("cr lf" appear), for example:
File:
First line
Second line
Third line
Uploaded file:
First line
Second line
Third line
Origin and uploaded files accordingly have different sizes, non-text files are the same.
Code:
private void sendFile(string In, string Out)
{
FtpWebRequest request = (FtpWebRequest) WebRequest.Create("ftp://domain//" + Out);
request.Method = WebRequestMethods.Ftp.UploadFile;
request.Credentials = new NetworkCredential("username", "password");
FileStream sourceStream = new FileStream(In, FileMode.Open, FileAccess.Read, FileShare.Read);
byte[] fileContents = new byte[sourceStream.Length];
sourceStream.Read(fileContents, 0, (int) sourceStream.Length);
sorceStream.Close();
request.ContentLength = fileContents.Length;
Stream requestStream = request.GetRequestStream();
requestStream.Write(fileContents, 0, fileContents.Length);
requestStream.Close();
}
How can i fix this?

EDIT: As the answer below doesn't seem to have helped (but I'm leaving it there for posterity as it shows better code) here are the next diagnostics steps I'd check:
How are you viewing the files? If at all possible, get onto the server directly rather than fetching the files again via a web browser or whatever.
What's the type of FTP server you're connecting to? Maybe there's a known issue.
Have you tried looking at what's actually being sent via Wireshark?
Have you tried sending the same files via a normal FTP client?
You should set FtpWebRequest.UseBinary to true in order to preserve the exact file contents. Otherwise the two systems will try to figure out line endings themselves, changing line terminators as they see fit. I very rarely think that's a good idea. (EDIT: UseBinary is actually true by default, but this sounds like the kind of problem introduced by using text mode... it certainly does no harm to make this explicit.)
Additionally:
You should be disposing of your FileStream via a using statement
You should be disposing of the request stream via a using statement
You should be taking note of the result of Stream.Read - it needn't always read the whole of the requested data in one go
You can either use File.ReadAllBytes to simply read the complete file data in one go, or use Stream.CopyTo (if you're using .NET 4) to copy the FileStream to the request stream (which won't set the content length, of course; I don't know whether this is a problem)
You're never calling GetResponse; it's unclear exactly what happens if you never fetch the response of an FtpWebRequest
Your parameter names don't match .NET naming conventions, and aren't very descriptive
So I would probably use:
private void SendFile(string inputFile, string outputPath)
{
FtpWebRequest request = (FtpWebRequest) WebRequest.Create
("ftp://domain//" + outputPath);
request.Method = WebRequestMethods.Ftp.UploadFile;
request.UseBinary = true;
request.Credentials = new NetworkCredential("username", "password");
byte[] fileContents = File.ReadAllBytes(inputFile);
request.ContentLength = fileContents.Length;
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(fileContents, 0, fileContents.Length);
}
// This *may* be necessary in order to validate that everything has happened
using (WebResponse response = request.GetResponse())
{
}
}

Its strange. I face the same problem and I was unable to fix it until I did not provide an extension in file. For Example if my file name was
abcfile
then I make it abcfile.dat and after that it shows me the uploaded file as actual file. I again upload file with abcfile.txt but this time again empty line problem appear in my uploaded file.
I suggest that you must provide extension to your file any except .txt.

The system that you're sending to uses different line endings to what your system uses. I can assume, because you get an extra line, that you're on Windows, and it uses CRLF endings. The system you're sending to recognises CR and LF as separate endings, so you get the extra lines.
For text, truncate the LF or the CR, see what happens. I have no clue about the differing file sizes.

In the top menu of FileZilla, set:
Transfer menu > Transfer type > binary

In the top menu of FileZilla, set:
Transfer menu > Transfer type > binary
It's working for me.

How to read the response stream before the Http response completes

When making a request using HttpWebRequest object, I need to call the method GetResponse() to send the request and get the response back.
The problem with this method is that it doesn't return the response object until all data has been received. Say I am downloading a 100 MB file, I won't be able to read it until the response finish and all the 100 MB is downloaded.
What I want is to be able to read the response stream bytes as soon as they arrive, without waiting for the response to complete.
I know I can use the Range Http header, but it won't work on my situation.

I think this is very close to what #Zachary suggests. And it (seems to) work(s); actually I think using using as #Zachary does is even "nicer".
My main point being I cannot see the blocking behaviour of GetResponse() you (seem to) describe.
In addition the following code only roughly shows how everything works; it will not read the stream to the end for example (unless by coincidence :)). But it should work if you copy-n-paste it into an empty "Console Application"-project in Visual Studio.
You can try using some "shorter" URL for a test. The example here starts downloading an ISO of the debian distribution (a bit more than 600 MByte). Sorry debian, I did not mean to steal your bandwidth. -> Btw: is there something sensible one can use to test such a scenario?
The Code is strongly inspired by C# - How to read a continuous stream of XML over HTTP.
namespace StreamReadWebRequest
{
using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;
class Program
{
static void Main(string[] args)
{
HttpWebRequest req;
HttpWebResponse res = null;
try
{
req = (HttpWebRequest)WebRequest.Create(
"http://cdimage.debian.org/debian-cd/5.0.4/i386/iso-cd/debian-504-i386-CD-1.iso");
res = (HttpWebResponse)req.GetResponse();
Stream stream = res.GetResponseStream();
byte[] data = new byte[4096];
int read;
while ((read = stream.Read(data, 0, data.Length)) > 0)
{
Process(data, read);
}
}
finally
{
if (res != null)
res.Close();
}
Console.In.Read();
}
private static void Process(byte[] data, int read)
{
Console.Out.Write(ASCIIEncoding.ASCII.GetString(data));
}
}
}

I was looking for the same thing: server streams chunked XML data and I needed a C# client that could access this data while server is streaming. I tried many different ways to access the source (WebChannelFactory, WebClient, HttpWebRequest/Response, TcpClient) but failed so far. Finding this thread I focused on HttpWebRequest/Response where I have the same problem that following line is blocking:
HttpWebResponse resp = (HttpWebResponse)request.GetResponse();
As Artiom Chilaru stated, if it's blocking: something is wrong, because it should not. Now focusing on trying to replicate default behavior with downloading large .ISO files I found out that Fiddler was blocking the GetResponse() method!
However there is no problem to open Fiddler once the stream has been set up (i.e. GetResponse() has already been called), but during the HTTP GET if you find GetResponse() is blocking try to close Fiddler and see if your application now continuous it's normal flow (i.e. reading the stream).

If you set the buffer size on your read, you can read in the data in chunks... example...
// Get the response stream
using(Stream resStream = response.GetResponseStream())
{
string parseString = null;
int count = 0;
do
{
// Read a chunk of data
count = resStream.Read(buf, 0, buf.Length);
if (count != 0)
{
// Convert to ASCII
parseString = Encoding.ASCII.GetString(buf, 0, count);
// Append string to results
sb.Append(tempString);
}
}
while (count > 0);
}

I'm not sure what you have on your side, but I know for a fact (and I'm sure many people will agree here) that GetResponse() will NOT download the whole file back. It will send the request, wait for the response, and get the response headers.
After you have the response, you can easily get the response stream with GetResponseStream(), which is the actual data stream that's downloading from the server. And you can easily access the response stream BEFORE the whole file is downloaded. This is 100% true and tested.
If you're not getting the same behaviour (which is really strange, and shouldn't happen) could you add a code example that is not working as I explained above?
Also, do test the example posted by scherand. It just proves once again that it works just fine, without any special hacks.

How can I write an XML on my hard drive to GetRequestStream

I need to post raw xml to a site and read the response. With the following code I keep getting an "Unknown File Format" error and I'm not sure why.
XmlDocument sampleRequest = new XmlDocument();
sampleRequest.Load(#"C:\SampleRequest.xml");
byte[] bytes = Encoding.UTF8.GetBytes(sampleRequest.ToString());
string uri = "https://www.sample-gateway.com/gw.aspx";
req = WebRequest.Create(uri);
req.Method = "POST";
req.ContentLength = bytes.Length;
req.ContentType = "text/xml";
using (var requestStream = req.GetRequestStream())
{
requestStream.Write(bytes, 0, bytes.Length);
}
// Send the data to the webserver
rsp = req.GetResponse();
XmlDocument responseXML = new XmlDocument();
using (var responseStream = rsp.GetResponseStream())
{
responseXML.Load(responseStream);
}
I am fairly certain my issue is what/how I am writing to the requestStream so..
How can I modify that code so that I may write an xml located on the hard drive to the request stream?

ok instead of doing sampleRequest.ToString(), you should use sampleRequest.OuterXml, and that would do the magic, you were sending "System.Xml.XmlDocument" instead of the Xml
XmlDocument sampleRequest = new XmlDocument();
sampleRequest.Load(#"C:\SampleRequest.xml");
//byte[] bytes = Encoding.UTF8.GetBytes(sampleRequest.ToString());
byte[] bytes = Encoding.UTF8.GetBytes(sampleRequest.OuterXml);

Two things:
First, whenever you're trying to diagnose a problem with an HTML response, you should always examine what the response stream actually contains. If you had in this case, you would have seen that it contains System.Xml.XmlDocument, which would have told you what was wrong pretty much immediately.
Second, in an application with any kind of transaction volume, you're not going to want to load a static XML file into an XmlDocument before putting it in the response stream; your program's spending time and memory building something that you don't need. (It's even worse than that in your case; your approach not only parses the XML into a DOM object, it then makes an in-memory copy of its OuterXml property when you encode it as UTF-8. Also, do you really need to be doing that?) Instead, you should create a FileStream object and use one of the techniques in this answer to copy it to the response stream.