Develop Reference

Why does NAudio read zeros to buffer after playing the file but not before?

The following code will successfully, load, play, edit audio samples and (almost) write to audio file. I say almost because when I comment out the "Play" code it works, but leaving it in causes the buffer read:
audioFile.Read(buffer, 0, numSamples);
to result in zeros.
Do I need to reset the audioFile somehow? All the examples I've found don't mention any need for this.
using System;
using NAudio.Wave;
namespace NAudioTest
{
class TestPlayer
{
static void Main(string[] args)
{
string infileName = "c:\\temp\\pink.wav";
string outfileName = "c:\\temp\\pink_out.wav";
// load the file
var audioFile = new AudioFileReader(infileName);
// play the file
var outputDevice = new WaveOutEvent();
outputDevice.Init(audioFile);
outputDevice.Play();
//Since Play only means "start playing" and isn't blocking, we can wait in a loop until playback finishes....
while (outputDevice.PlaybackState == PlaybackState.Playing) { System.Threading.Thread.Sleep(1000); }
// edit the samples in file
int fs = audioFile.WaveFormat.SampleRate;
int numSamples = (int)audioFile.Length / sizeof(float); // length is the number of bytes - 4 bytes in a float
float[] buffer = new float[numSamples];
audioFile.Read(buffer, 0, numSamples);
float volume = 0.5f;
for (int n = 0; n < numSamples; n++) { buffer[n] *= volume; }
// write edited samples to new file
var writer = new WaveFileWriter(outfileName,audioFile.WaveFormat);
writer.WriteSamples(buffer,0,numSamples);
}
}
}

You must call Dispose on your writer before it is a valid WAV file. I recommend that you put it in a using block.
using(var writer = new WaveFileWriter(outfileName,audioFile.WaveFormat))
{
writer.WriteSamples(buffer,0,numSamples);
}

Play video frame by frame performance issues

I want to play a video (mostly .mov with Motion JPEG) in frame by frame mode with changing framerate. I have a function who gives me a framenumber and then I have to jump there. It will be mostly in one direction but can skip a few frames from time to time; also the velocity is not constant.
So I have a timer asking every 40ms about a new framenumber and setting the new position.
My first approach now is with DirectShow.Net (Interop.QuartzTypeLib). Therefore I render and open the video and set it to pause to draw the picture in the graph
FilgraphManagerClass media = new FilgraphManagerClass();
media.RenderFile(FileName);
media.pause();
Now I will just set a new position
media.CurrentPosition = framenumber * media.AvgTimePerFrame;
Since the video is in pause mode it will then draw every requested new position (frame). Works perfectly fine but really slow... the video keeps stuttering and lagging and its not the video source; there are enough frames recorded to play a fluent video.
With some performance tests I found out that the LAV-Codec is the bottleneck here. This is not included directly in my project since its a DirectShow-Player it will be cast through my codec pack I installed on my PC.
Ideas:
Using the LAV-Codec by myself directly in C#. I searched but everyone is using DirectShow it seems, building their own filters and not using existing ones directly in the project.
Instead of seeking or setting the time, can I get single frames just by the framenumber and draw them simply?
Is there a complete other way to archive what I want to do?
Background:
This project has to be a train simulator. We recorded real time videos of trains driving from inside the cockpit and know which frame is what position. Now my C# programm calculates the position of the train in dependence of time and acceleration, gives back the appropriate framenumber and draw this frame.
Additional Information:
There is another project (not written by me) in C/C++ who uses DirectShow and the avcodec-LAV directly with a similar way I do and it works fine! Thats because I had the idea to use a codec / filter like the avrcodec-lav by myself. But I can't find an interop or interface to work with C#.
Thanks everyone for reading this and trying to help! :)

Obtaining specific frame by seeking filter graph (the entire pipeline) is pretty slow since every seek operation involves the following on its backyard: flushing everything, possibly re-creating worker threads, seeking to first key frame/splice point/clean point/I-Frame before the requested time, start of decoding starting from found position skipping frames until originally requested time is reached.
Overall, the method works well when you scrub paused video, or retrieve specific still frames. When however you try to play this as smooth video, it eventually causes significant part of the effort to be wasted and spent on seeking within video stream.
Solutions here are:
re-encode video to remove or reduce temporal compression (e.g. Motion JPEG AVI/MOV/MP4 files)
whenever possible prefer to skip frames and/or re-timestamp them according to your algorithm instead of seeking
have a cached of decoded video frames and pick from there, populate them as necessary in worker thread
The latter two are unfortunately hard to achieve without advanced filter development (where continuous decoding without interruption by seeking operations is the key to achieving decent performance). With basic DirectShow.Net you only have basic control over streaming and hence the first item from the list above.

Wanted to post a comment instead of an answer, but don't have the reputation. I think your heading in the wrong direction with Direct Show. I've been messing with motion-jpeg for a few years now between C# & Android, and have gotten great performance with built-in .NET code (for converting byte-array to Jpeg frame) and a bit of multi-threading. I can easily achieve over 30fps from multiple devices with each device running in it's own thread.
Below is an older version of my motion-jpeg parser from my C# app 'OmniView'. To use, just send the network stream to the constructor, and receive the OnImageReceived event. Then you can easily save the frames to the hard-drive for later use (perhaps with the filename set to the timestamp for easy lookup). For better performance though, you will want to save all of the images to one file.
using OmniView.Framework.Helpers;
using System;
using System.IO;
using System.Text;
using System.Windows.Media.Imaging;
namespace OmniView.Framework.Devices.MJpeg
{
public class MJpegStream : IDisposable
{
private const int BUFFER_SIZE = 4096;
private const string tag_length = "Content-Length:";
private const string stamp_format = "yyyyMMddHHmmssfff";
public delegate void ImageReceivedEvent(BitmapImage img);
public delegate void FrameCountEvent(long frames, long failed);
public event ImageReceivedEvent OnImageReceived;
public event FrameCountEvent OnFrameCount;
private bool isHead, isSetup;
private byte[] buffer, newline, newline_src;
private int imgBufferStart;
private Stream data_stream;
private MemoryStream imgStreamA, imgStreamB;
private int headStart, headStop;
private long imgSize, imgSizeTgt;
private bool useStreamB;
public volatile bool EnableRecording, EnableSnapshot;
public string RecordPath, SnapshotFilename;
private string boundary_tag;
private bool tagReadStarted;
private bool enableBoundary;
public volatile bool OututFrameCount;
private long FrameCount, FailedCount;
public MJpegStream() {
isSetup = false;
imgStreamA = new MemoryStream();
imgStreamB = new MemoryStream();
buffer = new byte[BUFFER_SIZE];
newline_src = new byte[] {13, 10};
}
public void Init(Stream stream) {
this.data_stream = stream;
FrameCount = FailedCount = 0;
startHeader(0);
}
public void Dispose() {
if (data_stream != null) data_stream.Dispose();
if (imgStreamA != null) imgStreamA.Dispose();
if (imgStreamB != null) imgStreamB.Dispose();
}
//=============================
public void Process() {
if (isHead) processHeader();
else {
if (enableBoundary) processImageBoundary();
else processImage();
}
}
public void Snapshot(string filename) {
SnapshotFilename = filename;
EnableSnapshot = true;
}
//-----------------------------
// Header
private void startHeader(int remaining_bytes) {
isHead = true;
headStart = 0;
headStop = remaining_bytes;
imgSizeTgt = 0;
tagReadStarted = false;
}
private void processHeader() {
int t = BUFFER_SIZE - headStop;
headStop += data_stream.Read(buffer, headStop, t);
int nl;
//
if (!isSetup) {
byte[] new_newline;
if ((nl = findNewline(headStart, headStop, out new_newline)) >= 0) {
string tag = Encoding.UTF8.GetString(buffer, headStart, nl - headStart);
if (tag.StartsWith("--")) boundary_tag = tag;
headStart = nl+new_newline.Length;
newline = new_newline;
isSetup = true;
return;
}
} else {
while ((nl = findData(newline, headStart, headStop)) >= 0) {
string tag = Encoding.UTF8.GetString(buffer, headStart, nl - headStart);
if (!tagReadStarted && tag.Length > 0) tagReadStarted = true;
headStart = nl+newline.Length;
//
if (!processHeaderData(tag, nl)) return;
}
}
//
if (headStop >= BUFFER_SIZE) {
string data = Encoding.UTF8.GetString(buffer, headStart, headStop - headStart);
throw new Exception("Invalid Header!");
}
}
private bool processHeaderData(string tag, int index) {
if (tag.StartsWith(tag_length)) {
string val = tag.Substring(tag_length.Length);
imgSizeTgt = long.Parse(val);
}
//
if (tag.Length == 0 && tagReadStarted) {
if (imgSizeTgt > 0) {
finishHeader(false);
return false;
}
if (boundary_tag != null) {
finishHeader(true);
return false;
}
}
//
return true;
}
private void finishHeader(bool enable_boundary) {
int s = shiftBytes(headStart, headStop);
enableBoundary = enable_boundary;
startImage(s);
}
//-----------------------------
// Image
private void startImage(int remaining_bytes) {
isHead = false;
imgBufferStart = remaining_bytes;
Stream imgStream = getStream();
imgStream.Seek(0, SeekOrigin.Begin);
imgStream.SetLength(imgSizeTgt);
imgSize = 0;
}
private void processImage() {
long img_r = (imgSizeTgt - imgSize - imgBufferStart);
int bfr_r = Math.Max(BUFFER_SIZE - imgBufferStart, 0);
int t = (int)Math.Min(img_r, bfr_r);
int s = data_stream.Read(buffer, imgBufferStart, t);
int x = imgBufferStart + s;
appendImageData(0, x);
imgBufferStart = 0;
//
if (imgSize >= imgSizeTgt) processImageData(0);
}
private void processImageBoundary() {
int t = Math.Max(BUFFER_SIZE - imgBufferStart, 0);
int s = data_stream.Read(buffer, imgBufferStart, t);
//
int nl, start = 0;
int end = imgBufferStart + s;
while ((nl = findData(newline, start, end)) >= 0) {
int tag_length = boundary_tag.Length;
if (nl+newline.Length+tag_length > BUFFER_SIZE) {
appendImageData(start, nl+newline.Length - start);
start = nl+newline.Length;
continue;
}
//
string v = Encoding.UTF8.GetString(buffer, nl+newline.Length, tag_length);
if (v == boundary_tag) {
appendImageData(start, nl - start);
int xstart = nl+newline.Length + tag_length;
int xsize = shiftBytes(xstart, end);
processImageData(xsize);
return;
} else {
appendImageData(start, nl+newline.Length - start);
}
start = nl+newline.Length;
}
//
if (start < end) {
int end_x = end - newline.Length;
if (start < end_x) {
appendImageData(start, end_x - start);
}
//
shiftBytes(end - newline.Length, end);
imgBufferStart = newline.Length;
}
}
private void processImageData(int remaining_bytes) {
if (EnableSnapshot) {
EnableSnapshot = false;
saveSnapshot();
}
//
try {
BitmapImage img = createImage();
if (EnableRecording) recordFrame();
if (OnImageReceived != null) OnImageReceived.Invoke(img);
FrameCount++;
}
catch (Exception) {
// output frame error ?!
FailedCount++;
}
//
if (OututFrameCount && OnFrameCount != null) OnFrameCount.Invoke(FrameCount, FailedCount);
//
useStreamB = !useStreamB;
startHeader(remaining_bytes);
}
private void appendImageData(int index, int length) {
Stream imgStream = getStream();
imgStream.Write(buffer, index, length);
imgSize += (length - index);
}
//-----------------------------
private void recordFrame() {
string stamp = DateTime.Now.ToString(stamp_format);
string filename = RecordPath+"\\"+stamp+".jpg";
//
ImageHelper.Save(getStream(), filename);
}
private void saveSnapshot() {
Stream imgStream = getStream();
//
imgStream.Position = 0;
Stream file = File.Open(SnapshotFilename, FileMode.Create, FileAccess.Write);
try {imgStream.CopyTo(file);}
finally {file.Close();}
}
private BitmapImage createImage() {
Stream imgStream = getStream();
imgStream.Position = 0;
return ImageHelper.LoadStream(imgStream);
}
//-----------------------------
private Stream getStream() {return useStreamB ? imgStreamB : imgStreamA;}
private int findNewline(int start, int stop, out byte[] data) {
for (int i = start; i < stop; i++) {
if (i < stop-1 && buffer[i] == newline_src[0] && buffer[i+1] == newline_src[1]) {
data = newline_src;
return i;
} else if (buffer[i] == newline_src[1]) {
data = new byte[] {newline_src[1]};
return i;
}
}
data = null;
return -1;
}
private int findData(byte[] data, int start, int stop) {
int data_size = data.Length;
for (int i = start; i < stop-data_size; i++) {
if (findInnerData(data, i)) return i;
}
return -1;
}
private bool findInnerData(byte[] data, int buffer_index) {
int count = data.Length;
for (int i = 0; i < count; i++) {
if (data[i] != buffer[buffer_index+i]) return false;
}
return true;
}
private int shiftBytes(int start, int end) {
int c = end - start;
for (int i = 0; i < c; i++) {
buffer[i] = buffer[end-c+i];
}
return c;
}
}
}

C# task async await smart card - UI thread blocked

I'm new to C#, and I'm trying to use task async await for a WinsForm GUI. I've read so many tutorials about it, but all of them implement tasks differently. Some tasks use functions, and others just put the code in to execute. Some use Task.Run() or just await. Furthermore, all the examples I've seen are of functions that are included in the UI class. I'm trying to run functions that are in classes that are within my UI. I'm just really confused now, and don't know what's right/wrong.
What I'm trying to do is write a file to an EEPROM, using the SpringCard API/ PC/SC library. I parse the file into packets and write it to the smart card. I also want to update a status label and progress bar. A lot of things can go wrong. I have flags set in the smart card, and right now I just a while loop running until it reads a certain flag, which will obviously stall the program if it's forever waiting for a flag.
I guess I'm just confused about how to set it up. Help. I've tried using Tasks. Here is my code so far.
/* Initialize open file dialog */
OpenFileDialog ofd = new OpenFileDialog();
ofd.Multiselect = false;
ofd.Filter = "BIN Files (.bin)|*.bin|HEX Files (.hex)|*.hex";
ofd.InitialDirectory = "C:";
ofd.Title = "Select File";
//Check open file dialog result
if (ofd.ShowDialog() != DialogResult.OK)
{
if (shade != null)
{
shade.Dispose();
shade = null;
}
return;
}
//progform.Show();
Progress<string> progress = new Progress<string>();
file = new ATAC_File(ofd.FileName);
try
{
cardchannel.DisconnectReset();
Task upgrade = upgradeASYNC();
if(cardchannel.Connect())
{
await upgrade;
}
else
{
add_log_text("Connection to the card failed");
MessageBox.Show("Failed to connect to the card in the reader : please check that you don't have another application running in background that tries to work with the smartcards in the same time");
if (shade != null)
{
shade.Dispose();
shade = null;
}
cardchannel = null;
}
}
private async Task upgradeASYNC()
{
int i = 0;
int totalpackets = 0;
add_log_text("Parsing file into packets.");
totalpackets = file.parseFile();
/*progress.Report(new MyTaskProgressReport
{
CurrentProgressAmount = i,
TotalProgressAmount = totalpackets,
CurrentProgressMessage = "Sending upgrade file..."
});*/
ST_EEPROMM24LR64ER chip = new ST_EEPROMM24LR64ER(this, cardchannel, file, EEPROM.DONOTHING);
bool writefile = chip.WriteFileASYNC();
if(writefile)
{
add_log_text("WRITE FILE OK.");
}
else
{
add_log_text("WRITE FILE BAD.");
}
}
In the file class:
public int parseFile()
{
FileStream fs = new FileStream(filename, FileMode.Open, FileAccess.Read);
BinaryReader br = new BinaryReader(fs);
FileInfo finfo = new FileInfo(filename);
int readbytecount = 0;
int packetcount = 0;
int numofbytesleft = 0;
byte[] hash = new byte[4];
byte[] packetinfo = new byte[4];
byte[] filechunk = null;
/* Read file until all file bytes read */
while (size_int > readbytecount)
{
//Initialize packet array
filechunk = new byte[MAXDATASIZE];
//read into byte array of max write size
if (packetcount < numoffullpackets)
{
//Initialize packet info array
packetinfo[0] = (byte)((size_int + 1) % 0x0100); //packetcountlo
packetinfo[1] = (byte)((size_int + 1) / 0x0100); //packetcounthi
packetinfo[2] = (byte)((packetcount + 1) / 0x0100); //packetcounthi
packetinfo[3] = (byte)((packetcount + 1) % 0x0100); //packetcountlo
//read bytes from file into packet array
bytesread = br.Read(filechunk, 0, MAXDATASIZE);
//add number of bytes read to readbytecount
readbytecount += bytesread;
}
//read EOF into byte array of size smaller than max write size
else if (packetcount == numoffullpackets)
{
//find out how many bytes left to read
numofbytesleft = size_int - (MAXDATASIZE * numoffullpackets);
//Initialize packet info array
packetinfo[0] = (byte)((size_int + 1) / 0x0100); //packetcounthi
packetinfo[1] = (byte)((size_int + 1) % 0x0100); //packetcountlo
packetinfo[2] = (byte)((packetcount + 1) / 0x0100); //packetcounthi
packetinfo[3] = (byte)((packetcount + 1) % 0x0100); //packetcountlo
//Initialize array and add byte padding, MAXWRITESIZE-4 because the other 4 bytes will be added when we append the CRC
//filechunk = new byte[numofbytesleft];
for (int j = 0; j < numofbytesleft; j++)
{
//read byte from file
filechunk[j] = br.ReadByte();
//add number of bytes read to readbytecount
readbytecount++;
}
for (int j = numofbytesleft; j < MAXDATASIZE; j++)
{
filechunk[j] = 0xFF;
}
}
else
{
MessageBox.Show("ERROR");
}
//calculate crc32 on byte array
int i = 0;
foreach (byte b in crc32.ComputeHash(filechunk))
{
hash[i++] = b;
}
//Append hash to filechunk to create new byte array named chunk
byte[] chunk = new byte[MAXWRITESIZE];
Buffer.BlockCopy(packetinfo, 0, chunk, 0, packetinfo.Length);
Buffer.BlockCopy(filechunk, 0, chunk, packetinfo.Length, filechunk.Length);
Buffer.BlockCopy(hash, 0, chunk, (packetinfo.Length + filechunk.Length), hash.Length);
//Add chunk to byte array list
packetcount++;
PacketBYTE.Add(chunk);
}
parseCMD();
return PacketBYTE.Count;
}
In the EEPROM class:
public bool WriteFileASYNC()
{
int blocknum = ATAC_CONSTANTS.RFBN_RFstartwrite;
byte[] response = null;
CAPDU[] EEPROMcmd = null;
int packetCount = 0;
log("ATTEMPT: Read response funct flag.");
do
{
StopRF();
Thread.SpinWait(100);
StartRF();
log("ATTEMPT: Write function flag.");
while (!WriteFlag(ATAC_CONSTANTS.RFBN_functflag, EEPROM.UPLOADAPP)) ;
} while (ReadFunctFlag(ATAC_CONSTANTS.RFBN_responseflag, 0) != EEPROM.UPLOADAPP);
for (int EEPROMcount = 0; EEPROMcount < file.CmdBYTE.Count; EEPROMcount++)
{
string temp = "ATTEMPT: Write EEPROM #" + EEPROMcount.ToString();
log(temp);
EEPROMcmd = file.CmdBYTE[EEPROMcount];
while (EEPROMcmd[blocknum] != null)
{
if (blocknum % 32 == 0)
{
string tempp = "ATTEMPT: Write packet #" + packetCount.ToString();
log("ATTEMPT: Write packet #");
packetCount++;
}
do
{
response = WriteBinaryASYNC(EEPROMcmd[blocknum]);
} while (response == null);
blocknum++;
}
log("ATTEMPT: Write packet flag.");
while (!WriteFlag(ATAC_CONSTANTS.RFBN_packetflag, ATAC_CONSTANTS.RFflag)) ;
log("ATTEMPT: Write packet flag.");
do
{
StopRF();
Thread.SpinWait(300);
StartRF();
} while (!ReadFlag(ATAC_CONSTANTS.RFBN_packetresponseflag, ((blocknum/32) - 1)*(EEPROMcount+1)));
blocknum = ATAC_CONSTANTS.RFBN_RFstartwrite;
}
return true;
}

Tasks are threads.
When you write this:
Task upgrade = upgradeASYNC();
you are simply executing upgradeASYNC in a new thread.
When you write this:
await upgrade;
You are only waiting for the new thread to finish (before going to the next instruction).
And this method
private async Task upgradeASYNC()
returns a Task object only because you add the async keyword. But in the body of this method there is no await. So it just runs synchronously, like any thread job.
I don't have time to rewrite your code, i let that to another stackoverflow user. You should learn and work harder ;)

How to execute all process parallel inside for loop in winforms C# application

I have created on function which will split single fine into multiple file. For example i have one file which contain 100 pages now i would like to create new files for every 15 pages, it means it will create 7 files each file has 15 pages. (100/15 = 7)
Now my problem is i have implemented logic for splitting files using Thread, ISynchronizeInvoke and delegate to smooth process and user experience. It is working aspected, but i would like to perform each split in parallel or simultaneously instead of one by one process.
This code has been written in Splitter.cs file
I have written following code:
#region Private Variables
private NotifyProgress _notifyDelegate;
private Thread _thread;
private ISynchronizeInvoke _synchronizingObject;
//this is the definition of the progress delegate - it defines the "signature" of the routine...
public delegate void NotifyProgress(int TotalFiles, int ProcessFileIndex, int TotalPages, int PageIndex, string Size);
#endregion
private void splitFiles()
{
//Intialize a new PdfReader instance with the contents of the source Pdf file:
PdfReader Reader = new PdfReader(PDFFile);
Reader.RemoveUnusedObjects();
Reader.ConsolidateNamedDestinations();
for (int f = 0; f < Files.Count(); f++)
{
List<int> Pages = Files[f].Pages;
string FileName = (f + 1).ToString();
string folderPath = Path.Combine(OutputFolderPath, GeneratedFilesFolder);
string gereatedPath = Path.Combine(folderPath, string.Format(FileNameFormat, FileName));
//This method will create folder if the path doesn't exist
CreateFolder(folderPath);
PdfImportedPage importedPage = null;
Document currentDocument = new Document();
PdfSmartCopy pdfWriter = null;
bool bIsFirst = true;
long _size = 0;
for (int p = 0; p < Pages.Count; p++)
{
NotifyUI(Files.Count(), f, Pages.Count, p, _size);
if (bIsFirst)
{
bIsFirst = false;
currentDocument = new Document(Reader.GetPageSizeWithRotation(1));
pdfWriter = new PdfSmartCopy(currentDocument, new FileStream(gereatedPath, FileMode.Create));
pdfWriter.SetFullCompression();
//pdfWriter.CompressionLevel = PdfStream..BEST_COMPRESSION;
pdfWriter.PdfVersion = Reader.PdfVersion;
currentDocument.Open();
}
_size += pdfWriter.CurrentDocumentSize;
importedPage = pdfWriter.GetImportedPage(Reader, Pages[p]);
pdfWriter.AddPage(importedPage);
}
currentDocument.Close();
pdfWriter.Close();
FileInfo _f = new FileInfo(gereatedPath);
NotifyUI(Files.Count(), f, Pages.Count, Pages.Count - 1, _f.Length);
}
}
private void NotifyUI(int TotalFiles, int ProcessFileIndex, int TotalPages, int PageIndex, long Size)
{
//this method will fail because we're not telling the delegate which thread to run in...
object[] args = { TotalFiles, ProcessFileIndex + 1, TotalPages, PageIndex + 1, CalculateFileSize(Size) };
//call the delegate, specifying the context in which to run...
_synchronizingObject.Invoke(_notifyDelegate, args);
}
This code has been written in Form.cs file
private void DelegateProgress(int TotalFiles, int ProcessFileIndex, int TotalPages, int PageIndex, string Size)
{
if (splitter != null && PDFSplitter.TotalPages > 0)
{
this.Invoke((MethodInvoker)delegate
{
int IndividualProgress = PageIndex * 100 / TotalPages;
lstFiles.Items[ProcessFileIndex - 1].SubItems[1].Text = Size;
TextProgressBar pb = (TextProgressBar)lstFiles.GetEmbeddedControl(3, ProcessFileIndex - 1);
pb.Text = string.Format("{0:00} %", IndividualProgress);
pb.Value = IndividualProgress;
int OverallProgress = ProcessFileIndex * 100 / TotalFiles;
ProgressStripItem statusProgrss = (ProgressStripItem)tsStatus.Items[3];
statusProgrss.TextProgressBar.Value = OverallProgress;
statusProgrss.TextProgressBar.Text = string.Format("{0:00}%", OverallProgress);
if (OverallProgress >= 100 && IndividualProgress >= 100)
{
tslblMessage.Text = "File has been split successfully.";
DialogResult dr = MessageBox.Show(tslblMessage.Text + "\nDo you want to open split files folder?", "Split Completed", MessageBoxButtons.YesNo, MessageBoxIcon.Information);
if (dr == System.Windows.Forms.DialogResult.Yes)
{
OpenSplitFilePath();
}
for (int i = 0; i < lstFiles.Items.Count; i++)
{
lstFiles.RemoveEmbeddedControl(lstFiles.GetEmbeddedControl(3, i));
}
lstFiles.Items.Clear();
splitter = null;
}
});
}
}

Have you tried Parallel For loop? Here's a simple example.
Parallel.For(0, 10, i =>
{
//What you would like to do simultaneously.
System.Diagnostics.Debug.WriteLine(i);
});
If you ran try building and compile this simple code, you would notice that the output will be something like this.
8
4
5
7
2
3
9
1
6

I hope this can be achieved by parallel For Each. Refer the following link to know how to use parallel for each. http://msdn.microsoft.com/en-us/library/dd460720.aspx

Most efficient way for reading IMDB movies list

I am reading IMDB movies listing from a text file on my harddrive (originally available from IMDB site at ftp://ftp.fu-berlin.de/pub/misc/movies/database/movies.list.gz).
It takes around 5 minutes on my machine (basic info: Win7 x64bit, 16GB RAM, 500 GB SATA Hardisk 7200 RPM) to read this file line by line using code below.
I have two questions:
Is there any way I can optimize code to improve the read time?
Data access don't need to be sequential as I won't mind reading data from top to bottom / bottom to top or any order for that matter as long as it read one line at a time. I am wondering is there a way to read in multiple directions to improve the read time?
The application is a Windows Console Application.
Update: Many responses correctly pointed out that Writing to the Console takes substantial time. Considering that the displaying of data on the Windows Console is now desirable but not mandatory.
//Code Block
string file = #"D:\movies.list";
FileStream fs = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.None, 8, FileOptions.None);
using (StreamReader sr = new StreamReader(fs))
{
while (sr.Peek() >= 0)
{
Console.WriteLine(sr.ReadLine());
}
}

I'm not certain whether this is more efficient or not, but an alternate method would be to use File.ReadAllLines:
var movieFile = File.ReadAllLines(file);
foreach (var movie in movieFile)
Console.WriteLine(movie);

I am not a c# developer, but how about doing a bulk insert into database using the file(which will be one time). Then you can reuse the data and export as well.

In .net 4 you can use File.ReadLines for lazy evaluation and thus lower RAM usage when working on large files.
You can do linq operation directly on files and this along with File.ReadLines would improve load time.
For better understanding you can check, Read text file word-by-word using LINQ
You can also do comparison as well but putting time intervals.
However if you making web app you can read whole file on application start event and cache them in application pool for better performanace.

First of all, if you don't care about printing out the list to console, please edit your question.
Second, I created a timing program to test the speeds of the different methods suggested:
class Program
{
private static readonly string file = #"movies.list";
private static readonly int testStart = 1;
private static readonly int numOfTests = 2;
private static readonly int MinTimingVal = 1000;
private static string[] testNames = new string[] {
"Naive",
"OneCallToWrite",
"SomeCallsToWrite",
"InParallel",
"InParallelBlcoks",
"IceManMinds",
"TestTiming"
};
private static double[] avgSecs = new double[numOfTests];
private static int[] testIterations = new int[numOfTests];
public static void Main(string[] args)
{
Console.WriteLine("Starting tests...");
Debug.WriteLine("Starting tests...");
Console.WriteLine("");
Debug.WriteLine("");
//*****************************
//The console is the bottle-neck, so we can
//speed-up redrawing it by only showing 1 line at a time.
Console.WindowHeight = 1;
Console.WindowWidth = 50;
Console.BufferHeight = 100;
Console.BufferWidth = 50;
//******************************
Action[] actionArray = new Action[numOfTests];
actionArray[0] = naive;
actionArray[1] = oneCallToWrite;
actionArray[2] = someCallsToWrite;
actionArray[3] = inParallel;
actionArray[4] = inParallelBlocks;
actionArray[5] = iceManMinds;
actionArray[6] = testTiming;
for (int i = testStart; i < actionArray.Length; i++)
{
Action a = actionArray[i];
DoTiming(a, i);
}
printResults();
Console.WriteLine("");
Debug.WriteLine("");
Console.WriteLine("Tests complete.");
Debug.WriteLine("Tests complete.");
Console.WriteLine("Press Enter to Close Console...");
Debug.WriteLine("Press Enter to Close Console...");
Console.ReadLine();
}
private static void DoTiming(Action a, int num)
{
a.Invoke();
Stopwatch watch = new Stopwatch();
Stopwatch loopWatch = new Stopwatch();
bool shouldRetry = false;
int numOfIterations = 2;
do
{
watch.Start();
for (int i = 0; i < numOfIterations; i++)
{
a.Invoke();
}
watch.Stop();
shouldRetry = false;
if (watch.ElapsedMilliseconds < MinTimingVal) //if the time was less than the minimum, increase load and re-time.
{
shouldRetry = true;
numOfIterations *= 2;
watch.Reset();
}
} while (shouldRetry);
long totalTime = watch.ElapsedMilliseconds;
double avgTime = ((double)totalTime) / (double)numOfIterations;
avgSecs[num] = avgTime / 1000.00;
testIterations[num] = numOfIterations;
}
private static void printResults()
{
Console.WriteLine("");
Debug.WriteLine("");
for (int i = testStart; i < numOfTests; i++)
{
TimeSpan t = TimeSpan.FromSeconds(avgSecs[i]);
Console.WriteLine("ElapsedTime: {0:N4}, " + "test: " + testNames[i], t.ToString() );
Debug.WriteLine("ElapsedTime: {0:N4}, " + "test: " + testNames[i], t.ToString() );
}
}
public static void naive()
{
FileStream fs = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.None, 8, FileOptions.None);
using (StreamReader sr = new StreamReader(fs))
{
while (sr.Peek() >= 0)
{
Console.WriteLine( sr.ReadLine() );
}
}
}
public static void oneCallToWrite()
{
FileStream fs = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.None, 8, FileOptions.None);
using (StreamReader sr = new StreamReader(fs))
{
StringBuilder sb = new StringBuilder();
while (sr.Peek() >= 0)
{
string s = sr.ReadLine();
sb.Append("\n" + s);
}
Console.Write(sb);
}
}
public static void someCallsToWrite()
{
FileStream fs = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.None, 8, FileOptions.None);
using (StreamReader sr = new StreamReader(fs))
{
StringBuilder sb = new StringBuilder();
int count = 0;
int mod = 10000;
while (sr.Peek() >= 0)
{
count++;
string s = sr.ReadLine();
sb.Append("\n" + s);
if (count % mod == 0)
{
Console.Write(sb);
sb = new StringBuilder();
}
}
Console.Write( sb );
}
}
public static void inParallel()
{
string[] wordsFromFile = File.ReadAllLines( file );
int length = wordsFromFile.Length;
Parallel.For( 0, length, i => {
Console.WriteLine( wordsFromFile[i] );
});
}
public static void inParallelBlocks()
{
string[] wordsFromFile = File.ReadAllLines(file);
int length = wordsFromFile.Length;
Parallel.For<StringBuilder>(0, length,
() => { return new StringBuilder(); },
(i, loopState, sb) =>
{
sb.Append("\n" + wordsFromFile[i]);
return sb;
},
(x) => { Console.Write(x); }
);
}
#region iceManMinds
public static void iceManMinds()
{
string FileName = file;
long ThreadReadBlockSize = 50000;
int NumberOfThreads = 4;
byte[] _inputString;
var fi = new FileInfo(FileName);
long totalBytesRead = 0;
long fileLength = fi.Length;
long readPosition = 0L;
Console.WriteLine("Reading Lines From {0}", FileName);
var threads = new Thread[NumberOfThreads];
var instances = new ReadThread[NumberOfThreads];
_inputString = new byte[fileLength];
while (totalBytesRead < fileLength)
{
for (int i = 0; i < NumberOfThreads; i++)
{
var rt = new ReadThread { StartPosition = readPosition, BlockSize = ThreadReadBlockSize };
instances[i] = rt;
threads[i] = new Thread(rt.Read);
threads[i].Start();
readPosition += ThreadReadBlockSize;
}
for (int i = 0; i < NumberOfThreads; i++)
{
threads[i].Join();
}
for (int i = 0; i < NumberOfThreads; i++)
{
if (instances[i].BlockSize > 0)
{
Array.Copy(instances[i].Output, 0L, _inputString, instances[i].StartPosition,
instances[i].BlockSize);
totalBytesRead += instances[i].BlockSize;
}
}
}
string finalString = Encoding.ASCII.GetString(_inputString);
Console.WriteLine(finalString);//.Substring(104250000, 50000));
}
private class ReadThread
{
public long StartPosition { get; set; }
public long BlockSize { get; set; }
public byte[] Output { get; private set; }
public void Read()
{
Output = new byte[BlockSize];
var inStream = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
inStream.Seek(StartPosition, SeekOrigin.Begin);
BlockSize = inStream.Read(Output, 0, (int)BlockSize);
inStream.Close();
}
}
#endregion
public static void testTiming()
{
Thread.Sleep(500);
}
}
Each of these tests print the file out to console.
When run under default Console settings, each test took between 5:30 and 6:10 (Min:Sec).
After considering the Console properties, by making Console.WindowHeight = 1, that is, only 1 line is shown at a time, (you can scroll up and down to see the most recent 100 lines), and I achieved a speed-up.
Currently, the task completes in just a little over 2:40 (Min:Sec) for most methods.
Try it out on your computer and see how it works for you.
Interestingly enough, the different methods were basically equivalent, with the OP's code being basically the fastest.
The timing code warms-up the code then runs it twice and averages the time it takes, it does this for each method.
Feel free to try out your own methods and time them.

The answer to this question really depends on what it is you will be doing with the data. If your intention truly is to just read in the file and dump the contents to the console screen, then it would be better to use the StringBuilder Class to build up a string of, say 1000 lines, then dump the contents to the screen, reset the string then read in another 1000 lines, dump them, etc etc...
However if you are trying to build something that is part of a larger project and you are using .NET 4.0, you can use the MemoryMappedFile Class to read the file and create a CreateViewAccessor to create a "window" that operates on just a portion of the data instead of reading in the entire file.
Another option would be to make Threads that read different parts of the file all at once, then puts it all together in the end.
If you can be more specific as to what you plan to do with this data, I can help you more. Hope this helps!
EDIT:
Try this code out man. I was able to read the whole list in literally 3 seconds time using Threads:
using System;
using System.IO;
using System.Text;
using System.Threading;
namespace ConsoleApplication36
{
class Program
{
private const string FileName = #"C:\Users\Public\movies.list";
private const long ThreadReadBlockSize = 50000;
private const int NumberOfThreads = 4;
private static byte[] _inputString;
static void Main(string[] args)
{
var fi = new FileInfo(FileName);
long totalBytesRead = 0;
long fileLength = fi.Length;
long readPosition = 0L;
Console.WriteLine("Reading Lines From {0}", FileName);
var threads = new Thread[NumberOfThreads];
var instances = new ReadThread[NumberOfThreads];
_inputString = new byte[fileLength];
while (totalBytesRead < fileLength)
{
for (int i = 0; i < NumberOfThreads; i++)
{
var rt = new ReadThread { StartPosition = readPosition, BlockSize = ThreadReadBlockSize };
instances[i] = rt;
threads[i] = new Thread(rt.Read);
threads[i].Start();
readPosition += ThreadReadBlockSize;
}
for (int i = 0; i < NumberOfThreads; i++)
{
threads[i].Join();
}
for (int i = 0; i < NumberOfThreads; i++)
{
if (instances[i].BlockSize > 0)
{
Array.Copy(instances[i].Output, 0L, _inputString, instances[i].StartPosition,
instances[i].BlockSize);
totalBytesRead += instances[i].BlockSize;
}
}
}
string finalString = Encoding.ASCII.GetString(_inputString);
Console.WriteLine(finalString.Substring(104250000, 50000));
}
private class ReadThread
{
public long StartPosition { get; set; }
public long BlockSize { get; set; }
public byte[] Output { get; private set; }
public void Read()
{
Output = new byte[BlockSize];
var inStream = new FileStream(FileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
inStream.Seek(StartPosition, SeekOrigin.Begin);
BlockSize = inStream.Read(Output, 0, (int)BlockSize);
inStream.Close();
}
}
}
}
You will need to change the FileName to match the location of your movies.list file. Also, you can adjust the total number of threads. I used 4, but you can decrease or increase this at will. You can also change the Block Size...This is how much data each thread reads in. Also, I'm assuming its an ASCII text file. If its not, you need to change the encoding type to UTF8 or whatever encoding the file is in. Good luck!