Develop Reference

Testing FTP Output of a Stream

Brand new to StackOverflow.... I'm writing a console app in C#. I'd like to verify that an Excel spreadsheet is streaming correctly using FTP.
The application calls a stored procedure, populates a datatable with the result set, reads the datatable into a stream and then FTP's that stream onto a 3rd party site. I'd like to upload the stream somewhere besides that actual 3rd party site just to verify everything shows properly on the other end. Looking for help with how to do that.
I've tried setting up a local FTP but received the following error when trying to create the FTPWebRequest:
"The requested URI is invalid for this FTP command."
FtpWebRequest myWebRequest = (FtpWebRequest)WebRequest.Create("ftp address");
I also tried Console.Out with no luck.
This is what my code to upload looks like:
StreamReader sourceStream;
using (sourceStream = new StreamReader(path))
{
fileContents = Encoding.UTF8.GetBytes(sourceStream.ReadToEnd());
}
myWebRequest.ContentLength = fileContents.Length;
using (Stream requestStream = myWebRequest.GetRequestStream())
{
requestStream.Write(fileContents, 0, fileContents.Length);
}
using (FtpWebResponse response = (FtpWebResponse)myWebRequest.GetResponse())
{
Console.WriteLine($"Upload File Complete, status {response.StatusDescription}");
}

Get content of HttpWebResponse while debugging

I'm trying to run sequential requests to a web api url every 10 seconds to log changes in the data returned. The code snippet looks like this:
using (Stream objStream = response.GetResponseStream())
{
query result = (query)serializer.Deserialize(objStream);
Console.WriteLine(result.results.quote.Name + " " + result.results.quote.Ask);
objStream.Flush();
objStream.Close();
}
Every now and then an InvalidOperationException is thrown when running the deserialiation with the message saying that the XML document is badly formated. In an effort to isolate the problem I'm trying to find the "raw" response content in debug mode using the autos/locals/watch view, but I really can't find it.
I can find the response header and a lot of other information and as far as I can see this looks okay with one exception; the content-length which shows -1. I'm not sure if this is something that I should care about really but since I can't find the response "body" I can't help being suspicious about it.
So my real question here is: how can I find the "body" inside a HttpWebResponse or Stream object?
And the side question: Is the content-length with value -1 something to be bothered about.

If you read the entire contents from the stream and store it in a variable before deserializing it, you should be able to see the contents while debugging

For debugging i would suggest you replicate the response into string and that way you watch it.
using (Stream objStream = response.GetResponseStream())
{
StreamReader sr = new StreamReader(objStream);
string response = sr.ReadToEnd();
objStream.Seek(0,SeekOrigin.Begin); // Get the pointer back to the begining.
query result = (query)serializer.Deserialize(objStream);
Console.WriteLine(result.results.quote.Name + " " + result.results.quote.Ask);
objStream.Flush(); // remove
objStream.Close();//remove
}
I would also recommend to remove:
objStream.Flush();
objStream.Close();
when using 'using' statement it calls Dispose() (IDisposable()), which will eventauly close the stream by itself .

After transmission of text files blank lines appears in them

EDIT: Filezilla caused the problem, when i download files back from server it added new lines. I'm sorry for confusion.
This method upload files to ftp server and it's work fine, but in text files uploaded to server blank lines appear after every line("cr lf" appear), for example:
File:
First line
Second line
Third line
Uploaded file:
First line
Second line
Third line
Origin and uploaded files accordingly have different sizes, non-text files are the same.
Code:
private void sendFile(string In, string Out)
{
FtpWebRequest request = (FtpWebRequest) WebRequest.Create("ftp://domain//" + Out);
request.Method = WebRequestMethods.Ftp.UploadFile;
request.Credentials = new NetworkCredential("username", "password");
FileStream sourceStream = new FileStream(In, FileMode.Open, FileAccess.Read, FileShare.Read);
byte[] fileContents = new byte[sourceStream.Length];
sourceStream.Read(fileContents, 0, (int) sourceStream.Length);
sorceStream.Close();
request.ContentLength = fileContents.Length;
Stream requestStream = request.GetRequestStream();
requestStream.Write(fileContents, 0, fileContents.Length);
requestStream.Close();
}
How can i fix this?

EDIT: As the answer below doesn't seem to have helped (but I'm leaving it there for posterity as it shows better code) here are the next diagnostics steps I'd check:
How are you viewing the files? If at all possible, get onto the server directly rather than fetching the files again via a web browser or whatever.
What's the type of FTP server you're connecting to? Maybe there's a known issue.
Have you tried looking at what's actually being sent via Wireshark?
Have you tried sending the same files via a normal FTP client?
You should set FtpWebRequest.UseBinary to true in order to preserve the exact file contents. Otherwise the two systems will try to figure out line endings themselves, changing line terminators as they see fit. I very rarely think that's a good idea. (EDIT: UseBinary is actually true by default, but this sounds like the kind of problem introduced by using text mode... it certainly does no harm to make this explicit.)
Additionally:
You should be disposing of your FileStream via a using statement
You should be disposing of the request stream via a using statement
You should be taking note of the result of Stream.Read - it needn't always read the whole of the requested data in one go
You can either use File.ReadAllBytes to simply read the complete file data in one go, or use Stream.CopyTo (if you're using .NET 4) to copy the FileStream to the request stream (which won't set the content length, of course; I don't know whether this is a problem)
You're never calling GetResponse; it's unclear exactly what happens if you never fetch the response of an FtpWebRequest
Your parameter names don't match .NET naming conventions, and aren't very descriptive
So I would probably use:
private void SendFile(string inputFile, string outputPath)
{
FtpWebRequest request = (FtpWebRequest) WebRequest.Create
("ftp://domain//" + outputPath);
request.Method = WebRequestMethods.Ftp.UploadFile;
request.UseBinary = true;
request.Credentials = new NetworkCredential("username", "password");
byte[] fileContents = File.ReadAllBytes(inputFile);
request.ContentLength = fileContents.Length;
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(fileContents, 0, fileContents.Length);
}
// This *may* be necessary in order to validate that everything has happened
using (WebResponse response = request.GetResponse())
{
}
}

Its strange. I face the same problem and I was unable to fix it until I did not provide an extension in file. For Example if my file name was
abcfile
then I make it abcfile.dat and after that it shows me the uploaded file as actual file. I again upload file with abcfile.txt but this time again empty line problem appear in my uploaded file.
I suggest that you must provide extension to your file any except .txt.

The system that you're sending to uses different line endings to what your system uses. I can assume, because you get an extra line, that you're on Windows, and it uses CRLF endings. The system you're sending to recognises CR and LF as separate endings, so you get the extra lines.
For text, truncate the LF or the CR, see what happens. I have no clue about the differing file sizes.

In the top menu of FileZilla, set:
Transfer menu > Transfer type > binary

In the top menu of FileZilla, set:
Transfer menu > Transfer type > binary
It's working for me.

C# with command line FTP help

I've been trying for a while now to set up an ftp connection with our mainframe via c#.  So far I've been able to connect and upload a file to the mainframe, which is great, but the file has the wrong record length and is set up as variable, instead of fixed record length.  After quite a bit a research I have come to the same conclusion as most people, that the ftpwebresponse function will not allow "qoute SITE" commands to be issued to the mainframe. If i'm wrong please don't hesitate to correct me, heres the code i'm using:
private void ftpButton_Click(object sender, EventArgs e)
{
string ftphost = "ftpSite";
string user = "myUser";
string pwd= "myPass";
string ftpfileName = "test.file2";
string inputfilePath="d:\\documents and settings\\gheff\\My Documents\\Visual Studio 2005\\Projects\\biWeeklyJob\\BiWeekly\\bin\\Debug\\file2.TXT";
string ftpfullpath = String.Format("ftp://{0}//'{1}'", ftphost,ftpfileName);
try
{
FtpWebRequest ftp = (FtpWebRequest)FtpWebRequest.Create(ftpfullpath);
ftp.Credentials = new NetworkCredential(user,pwd);
ftp.KeepAlive = true;
ftp.UseBinary = false; //Use ascii.
ftp.Method = WebRequestMethods.Ftp.UploadFile;
FileStream fs = File.OpenRead(inputfilePath);
byte[] buffer = new byte[fs.Length];
fs.Read(buffer, 0, buffer.Length);
fs.Close();
Stream ftpstream = ftp.GetRequestStream();
ftpstream.Write(buffer, 0, buffer.Length);
ftpstream.Close();
}
catch (WebException ex)
{
String status = ((FtpWebResponse)ex.Response).StatusDescription;
throw new Exception(status);
}
}
So undeterred, I tried to think of another approach, I have successfully used the command line to upload a file, change the file properties and also changed the filetype so I can send JCL jobs to the mainframe.  I am now tyring to implement this into my existing c# application, but proving harder than I thought.
I am by no means an expert in c#, but can use answers to then go and do more research to gain a better understanding.
I have seen this piece of code C# cmd excute commands, but looking around it seems that only one command can be issued from this.
So my question is, I know it took a while to get there, is it possible to run the follow commands on the command prompt without the use of a batch file as i'm looking some feedback as the process runs?
open "cmd.exe"
type "ftp"
type "FTPserver"
Type "USERNAME"
type "PASSWORD"
Then once connection has been established
then run pre-defined commands i.e upload files, upload and run JCL jobs.
I think what i'm looking for is somthing that will write text to cmd.exe but keep the session?
I hope this makes sense.
Any help would be greatly appreciated.
Thanks

So here is code you are asking for nevertheless I doubt that it solve your problem.
var cmd= new Process();
cmd.StartInfo.FileName = "cmd.exe";
cmd.StartInfo.UseShellExecute = false;
cmd.StartInfo.RedirectStandardInput = true;
cmd.Start();
using (var stdin = cmd.StandardInput)
{
stdin.WriteLine("ftp");
stdin.WriteLine("FTPserver");
stdin.WriteLine("USERNAME");
stdin.WriteLine("PASSWORD");
}
cmd.WaitForExit();
cmd.Close();

Try this cmd:
FTP -s:ftpcommandfilename url
sense the follow mode can run one command only, you can write the ftp cmd in a file and use this cmd to execute them.
WriteLine("FTP -s:ftpcommandfilename url");
By the way, I am also finding the multi-ftp-cmd-line-in-code mode, if you have got it, sharing?

I used this CodeProject project a few years ago for a simple application that needed to send some files. The thing that may help you is that it uses a socket to talk to the other server. You can send raw FTP commands which should sidestep the problems you're having with the FtpWebRequest object, without having to "shell out" and use the ftp command line program.

Another approach you could try is to submit a jcl job to allocate the dataset with the record length, block size, et al, that your process needs.
Once that job complets, "put" your data into the dataset you just allocated correctly. The z/OS ftp server should pick up the attributes from your existing dataset.

Have you considered using a commercial FTP client that supports a COM interface and is able to send raw protocol commands? This might not be economical if you are writing software to redistribute but if you are only trying to automate a recurring task for an internal IT project it could definately be worth it in terms of your time as a developer. Robo-FTP might be a good choice for connecting to a mainframe because it has built in ASCII to EBCDIC translation.

How to read the response stream before the Http response completes

When making a request using HttpWebRequest object, I need to call the method GetResponse() to send the request and get the response back.
The problem with this method is that it doesn't return the response object until all data has been received. Say I am downloading a 100 MB file, I won't be able to read it until the response finish and all the 100 MB is downloaded.
What I want is to be able to read the response stream bytes as soon as they arrive, without waiting for the response to complete.
I know I can use the Range Http header, but it won't work on my situation.

I think this is very close to what #Zachary suggests. And it (seems to) work(s); actually I think using using as #Zachary does is even "nicer".
My main point being I cannot see the blocking behaviour of GetResponse() you (seem to) describe.
In addition the following code only roughly shows how everything works; it will not read the stream to the end for example (unless by coincidence :)). But it should work if you copy-n-paste it into an empty "Console Application"-project in Visual Studio.
You can try using some "shorter" URL for a test. The example here starts downloading an ISO of the debian distribution (a bit more than 600 MByte). Sorry debian, I did not mean to steal your bandwidth. -> Btw: is there something sensible one can use to test such a scenario?
The Code is strongly inspired by C# - How to read a continuous stream of XML over HTTP.
namespace StreamReadWebRequest
{
using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;
class Program
{
static void Main(string[] args)
{
HttpWebRequest req;
HttpWebResponse res = null;
try
{
req = (HttpWebRequest)WebRequest.Create(
"http://cdimage.debian.org/debian-cd/5.0.4/i386/iso-cd/debian-504-i386-CD-1.iso");
res = (HttpWebResponse)req.GetResponse();
Stream stream = res.GetResponseStream();
byte[] data = new byte[4096];
int read;
while ((read = stream.Read(data, 0, data.Length)) > 0)
{
Process(data, read);
}
}
finally
{
if (res != null)
res.Close();
}
Console.In.Read();
}
private static void Process(byte[] data, int read)
{
Console.Out.Write(ASCIIEncoding.ASCII.GetString(data));
}
}
}

I was looking for the same thing: server streams chunked XML data and I needed a C# client that could access this data while server is streaming. I tried many different ways to access the source (WebChannelFactory, WebClient, HttpWebRequest/Response, TcpClient) but failed so far. Finding this thread I focused on HttpWebRequest/Response where I have the same problem that following line is blocking:
HttpWebResponse resp = (HttpWebResponse)request.GetResponse();
As Artiom Chilaru stated, if it's blocking: something is wrong, because it should not. Now focusing on trying to replicate default behavior with downloading large .ISO files I found out that Fiddler was blocking the GetResponse() method!
However there is no problem to open Fiddler once the stream has been set up (i.e. GetResponse() has already been called), but during the HTTP GET if you find GetResponse() is blocking try to close Fiddler and see if your application now continuous it's normal flow (i.e. reading the stream).

If you set the buffer size on your read, you can read in the data in chunks... example...
// Get the response stream
using(Stream resStream = response.GetResponseStream())
{
string parseString = null;
int count = 0;
do
{
// Read a chunk of data
count = resStream.Read(buf, 0, buf.Length);
if (count != 0)
{
// Convert to ASCII
parseString = Encoding.ASCII.GetString(buf, 0, count);
// Append string to results
sb.Append(tempString);
}
}
while (count > 0);
}

I'm not sure what you have on your side, but I know for a fact (and I'm sure many people will agree here) that GetResponse() will NOT download the whole file back. It will send the request, wait for the response, and get the response headers.
After you have the response, you can easily get the response stream with GetResponseStream(), which is the actual data stream that's downloading from the server. And you can easily access the response stream BEFORE the whole file is downloaded. This is 100% true and tested.
If you're not getting the same behaviour (which is really strange, and shouldn't happen) could you add a code example that is not working as I explained above?
Also, do test the example posted by scherand. It just proves once again that it works just fine, without any special hacks.