I have an iPhone/iPad application made in MonoTouch, which is already in the apple store. However there's still an issue that hasn't been completely solved. I need to be able to upload large files to our servers. Sizes can vary from 2-100 Mb in size. I have tried several approaches:
Using SOAP Web Services, sending the byte array: a) Sending the whole file. Crashed. b) Splitting in chunks of 1Mb. Crashed after 10-15 chunks (varies).
Using WebClient.UploadFile. Works most of the time with smaller files < 5 Mb, but every now and then it crashes and everything larger than 10-12 Mb crashes. On the server, I have a aspx page that receives the information sent through a POST
Here is the actual code from the device:
WebClient wc = new WebClient ();
string sLFN = sLocalFileName;
FileInfo fi = new FileInfo (sLFN)
string sUri = getUri ();
byte [] f = wc.UploadFile (sUri, sLFN)
I believe that the memory capacity of the iPhone / iPad is something that directly affects this.
Does anybody has any recomendation or suggested approach that I should follow?
Thanks in advance
Edgar Herrador
I'd recommend using Streaming & Chunking. You're probably exceeding the message size when passing large files.
Are you getting any exceptions ?
Have you tried WCF streaming (http://msdn.microsoft.com/en-us/library/ms733742.aspx)?
Related
I am using Discord.net to log images in a channel on a Discord server. Most of the time I need to send a couple of images (4 or 5 on average) at once.
Since the Discord Desktop app allows you to send up to 10 files at once I was wondering if this can also be done using C#.
Currently, I am sending one message per image.
The code looks like this:
DiscordSocketClient client = new DiscordSocketClient();
await client.LoginAsync(TokenType.Bot, token);
await client.StartAsync();
When the Bot has started, this code runs:
DiscordSocketClient channel = client.GetChannel(channelId) as SocketTextChannel;
Finally, to send a file I run this:
channel.SendFileAsync(filepath, message); // Both filepath and message are strings.
This works, but when it sends, for example, five images, five messages are sent on discord.
It would be nice if I could send one message that has all of those images attached. (Just like in the Desktop app)
I know that I could use an Embed to send two images at once, but that way I would still be limited to two images per message. (One as Embed-image and one as Embed-thumbnail)
This is still not close to the 10 files that are possible in the app and, in addition to that, the images would not be displayed equally.
Another option would be to create a new, bigger image that combines all of the images I want to send. Then I could send this single image by saving it as a .png and then using the method that i have already implemented.
The problem with this is that this is a workaround, rather than a real solution to the problem, as I dont really like the idea of not having the images uploaded as individual files.
Tl dr:
Is there a way to send multiple files or images in a single message using Discord.net?
Thanks in advance!
You can use SendFilesAsync! You must update to the latest version of Discord.net, when installing via Nuget make sure you uninstall all Discord.net extensions such as Discord.Net.Commands before updating.
List<FileAttachment> filestosend = new List<FileAttachment>();
filestosend.Add(new FileAttachment(#"c:\file.png", "file.png", "file description"));
filestosend.Add(new FileAttachment(#"c:\file2.png", "file2.png", "file description"));
await channel.SendFilesAsync(filestosend, "text");
Documentation can be found here
Edit for clarity: do not confuse SendFileAsync with SendFilesAsync
ISSUE:
Video player (native player in webkit based browser) is not giving option to replay and I cannot seek video while playing it. Once played, I have to refresh page to play it again.
If I give a web url for ogv video in my html page, same player works fine (replays and seek is successful).
Hence I suspect issue with my algorithm of serving files (especially .ogv files).
What I am doing:
I have an HTTP fileserver (completely in C#, using .Net framework classes - HttpListenerContext, HttpListenerRequest, HttpListenerResponse, etc). This server is serving files (of all types and extensions - text, video, audio, images) to clients. All files are available on server which are requested by clients.
Algorithm to serve files from server:
Server gets the file name from URL.
Reads the complete file in a buffer(byte[] array) (using File.ReadAllBytes(file_path_on_server)),
Array is assigned to HttpListenerResponse.OutputStream to return the file (contents) to client.
Corresponding HttpListenerResponse.ContentType (MimeType) and HttpListenerResponse.ContentLength64 is assigned (size of array).
Response header is added as "no-cache".
HttpListenerResponse.ContentEncoding is UTF8.
Close response object to send it to client.
This works fine and files are reaching clients for processing including video files (.ogv) but video is not seekable/replayable in player.
Already tried
Tried removing assignment of HttpListenerResponse.ContentEncoding to UTF8.
It seems that you server returns the file the same way as if it's just downloaded, so seek/replay most likely will not be available until the whole file is loaded. If you want to support true video streaming you need another server-side implementation. Live Streaming within IIS Media Services may help you to support Apple HLS Streaming or similar (depending on your player), also i saw somewhere an IIS extension/module that supports http pseudo streaming with seeking for some video formats.
I have created a simple WCF service to prototype file uploading. The service:
[ServiceContract]
public class Service1
{
[OperationContract]
[WebInvoke(Method = "POST", UriTemplate = "/Upload")]
public void Upload(Stream stream)
{
using (FileStream targetStream = new FileStream(#"C:\Test\output.txt", FileMode.Create, FileAccess.Write))
{
stream.CopyTo(targetStream);
}
}
}
It uses webHttpBinding with transferMode set to "Streamed" and maxReceivedMessageSize, maxBufferPoolSize and maxBufferSize all set to 2GB. httpRuntime has maxRequestLength set to 10MB.
The client issues HTTP requests in the following way:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(#"http://.../Service1.svc/Upload");
request.Method = "POST";
request.SendChunked = true;
request.AllowWriteStreamBuffering = false;
request.ContentType = MediaTypeNames.Application.Octet;
using (FileStream inputStream = new FileStream(#"C:\input.txt", FileMode.Open, FileAccess.Read))
{
using (Stream outputStream = request.GetRequestStream())
{
inputStream.CopyTo(outputStream);
}
}
Now, finally, what's wrong:
When uploading the file 100MB big, the server returns HTTP 400 (Bad request). I've tried to enable WCF tracing, but it shows no error. When I increase httpRuntime.maxRequestLength to 1GB, the file gets uploaded without problems. The MSDN says that maxRequestLength "specifies the limit for the input stream buffering threshold, in KB".
This leads me to believe that the whole file (all 100MB of it) is first stored in "input stream buffer" and only then it is available to my Upload method on server. I can actually see that the size of file on server does not gradually increase (as I would expect), instead, in the moment it is created it is already 100MB big.
The question: How can I get this to work so that the "input stream buffer" is reasonably small (say, 1MB) and when it overflows, my Upload method gets called? In other words, I want the upload to be truly streamed without having to buffer the whole file anywhere.
EDIT:
I now discovered the httpRuntime contains another setting that is relevant here - requestLengthDiskThreshold. It seems that when the input buffer grows beyond this threshold, it is no longer stored in memory, but instead, on filesystem. So at least the whole 100MB big file is not kept in memory (this is what I was most afraid of), however, I still would like to know whether there is some way to avoid this buffer altogether.
If you are using .NET 4 and hosting your service in IIS7+, you may be affected an ASP.NET bug which is described in the following blog post:
http://blogs.microsoft.co.il/blogs/idof/archive/2012/01/17/what-s-new-in-wcf-4-5-improved-streaming-in-iis-hosting.aspx
Basically, for streamed requests, the ASP.NET handler in IIS will buffer the whole request before handing over control to WCF. And this handler obeys the maxRequestLength limit.
As far as I know, there is no workaround for the bug and you have the following options:
upgrade to .NET 4.5
self-host your service instead of using IIS
use a binding that is not based on HTTP, so that the ASP.NET handler is not involved
This may be a bug in the streaming implementation. I found a MSDN article that suggests doing exactly what you are describing at http://social.msdn.microsoft.com/Forums/en-US/wcf/thread/fb9efac5-8b57-417e-9f71-35d48d421eb4/. Unfortunately the Microsoft employee suggesting the fix found a bug in the implementation and didn't follow up with details on a fix.
That said it looks like the implementation is broken which you could test by profiling your code with a memory profiler and verifying whether or not the entire file is being stored in memory. If the entire file is being stored in memory, you won't be able to fix this issue, unless somebody finds a configuration issue with your code.
That said, while using requestLengthDiskThreshold could technically work, it will dramatically increase your write times as each file will have to be written first as temp data, read from temp data, written again as final, and finally the temp data deleted. As you have already said you are dealing with extremely large files so I doubt such a solution is acceptable.
Your best bet is to use a chunking framework and manually reconstruct the file. I found instructions on how to write such logic at http://aspilham.blogspot.com/2011/03/file-uploading-in-chunks-using.html but have not had the time to check it for accuracy.
I'm sorry I can't tell you why your code isn't working as documented, but something similar to the 2nd example should be able to work without ballooning your memory footprint.
My client is uploading file more then 1 GB through application. I know i can only upload only 100mb using asp.net MVC application.
public static byte[] ReadStream(Stream st)
{
st.Position = 0;
byte[] data = new byte[st.Length];
.
.
.
.
}
i am getting error at byte[] data = new byte[st.Length]; because st.Length=1330768612
Error - "Exception of type 'System.OutOfMemoryException' was thrown."
Is there any way i can upload more then 1gb file?
Why we can define maxRequestLength= 0 - 2097151 in webconfig,
IMO you need to use the right tool for the job. Http was simply not intended to transfer large files like this. Why dont you use ftp instead, and maybe you could then build a web interface around that.
The error shown to you suggests the server has not enough memory to process the file in memory. Validate if your server has enough memory to allocate such a big array/file.
You could also try to process chuncks of the stream. The fact that you get an out of memory suggests that the file is sent to the server, but the server cannot process the file.
I really think it has to do with the size of the array you allocate. It just won't fit in the memory of you machine (of in the memory assigned to .NET).
The error says that you run out of memory while trying to allocate a 1GB byte array in memory. This is not related to MVC. You should also note that the memory limit for 32bit processes is 2GB. If your server runs a 32bit OS and you allocate 1GB of that for a single upload you will quickly deplete the available memory.
Instead of trying to read the entire stream in memory, use Stream.Read to read the data in chuncks using a reasonably sized buffer and store the chuncks to a file stream with a Write call. Not only will you avoid OutOfMemoryExceptions, your code will also run much faster, because you won't have to wait to load the entire 1GB before storing it to a file.
The code can be as simple as this:
public static void SaveStream(Stream st,string targetFile)
{
byte[] inBuffer = new byte[10000];
using(FileStream outStream=File.Create(targetFile,20000))
using (BinaryWriter wr = new BinaryWriter(outStream))
{
st.Read(inBuffer, 0, inBuffer.Length);
wr.Write(inBuffer);
}
}
You can tweak the buffer sizes to balance throughput (how quickly you upload and save) vs scalability (how many clients you can handle).
I remember on an older project we had to work out a way to allow the user to upload a 2-4gb file for our ASP.NET web application.
If I recall correctly, we used the 'File Upload Control' and edited the web.config to allow for a greater file size:
<system.web>
<httpRuntime executionTimeout="2400" maxRequestLength="40960" />
</system.web>
Another option would be to use the HttpModule:
http://dotnetslackers.com/Community/blogs/haissam/archive/2008/09/12/upload-large-files-in-asp-net-using-httpmodule.aspx
I would suggest using FTP or write a small desktop application.
HTTP was never intended to send such large files.
Here's a Microsoft Knowledge base answer for you
I seek some insight in creating an application that converts text to speech in ASP.NET. From my initial research, it appears that:
MS SAPI requires the client to download an ActiveX component and can support large amounts of text to be converted. Our clients are not willing to install any components on their systems, so this approach may or may not fly.
I do understand with .NET 3.0, we have the System.Speech.Synthesis namespace. Does the conversion take place on the server? If so, how would I serve it to the client?
Our requirements are ability to convert large amount of text, should be scalable and reliable. Which technology is "production ready" capable of serving a large number of requests in a short time interval.
Any thoughts are appreciated.
By default, ASP.Net applications don't run with sufficient permissions to access Speech Synthesis, and attempting to run Larsenal's code will fail with a security error.
I was able to get around this in an app by having a separate WCF service running on the server, as a regular Windows Service. The ASP.Net application then communicated with that service. That service just wrapped Larsenal's code, returning an array of bytes, given a string of text.
Also, one megabyte of text? That's a good-sized novel.
Edit, 11-12-09, answering some comments:
System.Speech can either return an array of bytes, or save to a wav file, which you can then feed to a media player embedded on the user's page. When I built my talking web page, it worked like this:
1) Page.aspx includes an 'embed' tag that puts a Windows Media Player on the page. The source is "PlayText.aspx?Textid=whatever".
2) PlayText.aspx loads the appropriate text, and communicates (via WCF) to the speechreader service, handing it the text to read.
3) The Speechreader service creates a MemoryStream and calls SpeechSynthesiser.SetOutputToWaveStream, and then returns the stream as a single array of bytes. This array is Response.Write()-ed to the client.
Here's the meat of the SpeechReader service:
byte[] ITextReader.SpeakText(string text)
{
using (SpeechSynthesizer s = new SpeechSynthesizer())
{
using (MemoryStream ms = new MemoryStream())
{
s.SetOutputToWaveStream(ms);
s.Speak(text);
return ms.GetBuffer();
}
}
}
I'm pretty sure that on the back end, this returns an enormous XML array-of-bytes, and is horribly inefficient. I just did it as a proof of concept, and so didn't research that. If you intend to use this in production, make sure that it's not internally returning something like this:
<byte>23</byte>
<byte>42</byte>
<byte>117</byte>
...
With the SpeechSynthesizer, you can output to a WAV file. You could then have a secondary process compress or convert to another format if needed. All this could be done on the server and then sent up through the browser.
This CodeProject article is a good introduction to .NET Speech Synthesis.
If you want to see how it performs with a LOT of text.... Add a reference to System.Speech and then use the following as a starting point:
using System;
using System.Speech.Synthesis;
namespace SpeakToMe
{
class Program
{
static void Main(string[] args)
{
SpeechSynthesizer synth = new SpeechSynthesizer();
synth.SetOutputToWaveFile("c:\\test.wav");
synth.Speak("Hello, world.");
synth.SetOutputToDefaultAudioDevice();
Console.ReadLine();
}
}
}
A quick test on a file of 44,700 words (238KB) on my relatively fast machine...
Completed in 55 seconds
Generated a 626 MB WAV file
I searched for "Convert Text Into Speech In Asp.Net" in Google and found a very nice and usefull link:
http://codeprojectdownload.com/asp-net-2/convert-text-into-speech-in-asp-net/#.T0ScXIfXDZE
It may also be useful to you.
I achieved this by using codeBehind to run a javascript function that runs the text-to-speech command:
codeBehind:
Page.ClientScript.RegisterStartupScript(
GetType(),
"anythingHere",
"nameOfFunction();",
true);
javascript:
<script>
function nameOfFunction()
{//start
var msg = new SpeechSynthesisUtterance('READ ME!');
window.speechSynthesis.speak(msg);
}//end
</script>
I wrote an article on this on my blog: http://weblogs.asp.net/ricardoperes/archive/2014/04/08/speech-synthesis-with-asp-net-and-html5.aspx. I used AJAX and Data URIs to send voice data back and forth between the client and the server.