I have an ObservableCollection <T>. I want to insert the various elements in it and then save the newly created file in LocalStorage. How can I do that?
SQLiteAsyncConnection conn = new SQLiteAsyncConnection(Path.Combine(ApplicationData.Current.LocalFolder.Path, "Database.db"), true);
await conn.CreateTableAsync<Musei>();
var Dbase = Path.Combine(ApplicationData.Current.LocalFolder.Path, "Database.db");
var con = new SQLiteAsyncConnection(Dbase, true);
var query = await con.Table<Musei>().ToListAsync();
ObservableCollection<Musei> favMusei = new ObservableCollection<Musei>();
if (query.Count > 0)
{
favMusei.Clear();
foreach (Musei museifav in query)
{
favMusei.Add(museifav);
}
}
I'm using a json file to store in the memory. JSON is a light weight message exchange format and is widely used. You have to do some slight modifications in the code if you want some different file format.
Your collection would be serialized to the memory at the time of saving and has to be deserialized when reading it back from memory.
Add your own generic implementation of the collection. To create your situation i'm using a simple ObservableCollection<int>. And don't forget to initialize the collection to some meaningful values, here i'm using the default constructor initialization.
using System.Collections.ObjectModel;
using System.Runtime.Serialization.Json;
using Windows.Storage;
//Add your own generic implementation of the collection
//and make changes accordingly
private ObservableCollection<int> temp;
private string file = "temp.json";
private async void saveToFile()
{
//add your items to the collection
temp = new ObservableCollection<int>();
var jsonSerializer = new DataContractJsonSerializer(typeof(ObservableCollection<int>));
using (var stream = await ApplicationData.Current.LocalFolder.OpenStreamForWriteAsync(file, CreationCollisionOption.ReplaceExisting))
{
jsonSerializer.WriteObject(stream, temp);
}
}
private async Task getFormFile()
{
var jsonSerializer = new DataContractJsonSerializer(typeof(ObservableCollection<int>));
try
{
using (var stream = await ApplicationData.Current.LocalFolder.OpenStreamForReadAsync(file))
{
temp = (ObservableCollection<int>)jsonSerializer.ReadObject(stream);
}
}
catch
{
//if some error is caught while reading data from the file then initializing
//the collection to default constructor instance is a good choice
//again it's your choice and may differ in your scenario
temp = new ObservableCollection<int>();
}
}
To add some functionality to the code you can also have an ensureDataLoaded() function which would ensure that the data has been read from the JSON file.
public async Task ensureDataLoaded()
{
if (temp.Count == 0)
await getFormFile();
return;
}
Before using the global variable temp (having the ObservableCollection) call the ensureDataLoaded function. It would avoid some unnecessary NullPointerExceptions.
Related
I am trying to process a very large amount of data (~1000 seperate files, each of them ~30 MB) in order to use as input to the training phase of a machine learning algorithm. Raw data files formatted with JSON and I deserialize them using JsonSerializer class of Json.NET. Towards the end of the program, Newtonsoft.Json.dll throwing 'OutOfMemoryException' error. Is there a way to reduce the data in memory, or do I have to change all of my approach (such as switching to a big data framework like Spark) to handle this problem?
public static List<T> DeserializeJsonFiles<T>(string path)
{
if (string.IsNullOrWhiteSpace(path))
return null;
var jsonObjects = new List<T>();
//var sw = new Stopwatch();
try
{
//sw.Start();
foreach (var filename in Directory.GetFiles(path))
{
using (var streamReader = new StreamReader(filename))
using (var jsonReader = new JsonTextReader(streamReader))
{
jsonReader.SupportMultipleContent = true;
var serializer = new JsonSerializer();
while (jsonReader.Read())
{
if (jsonReader.TokenType != JsonToken.StartObject)
continue;
var jsonObject = serializer.Deserialize<dynamic>(jsonReader);
var reducedObject = ApplyFiltering(jsonObject) //return null if the filtering conditions are not met
if (reducedObject == null)
continue;
jsonObject = reducedObject;
jsonObjects.Add(jsonObject);
}
}
}
//sw.Stop();
//Console.WriteLine($"Elapsed time: {sw.Elapsed}, Elapsed mili: {sw.ElapsedMilliseconds}");
}
catch (Exception ex)
{
Console.WriteLine($"Error: {ex}")
return null;
}
return jsonObjects;
}
Thanks.
It's not really a problem with Newtonsoft. You are reading all of these objects into one big list in memory. It gets to a point where you ask the JsonSerializer to create another object and it fails.
You need to return IEnumerable<T> from your method, yield return each object, and deal with them in the calling code without storing them in memory. That means iterating the IEnumerable<T>, processing each item, and writing to disk or wherever they need to end up.
Refer to some EPPLUS sample code, there are just creating one epplus object for one activity.
ex
using (ExcelPackage package = new ExcelPackage(newFile))
{...
// activity
}
it means that after activity is finished, object is disposed automatically.
and for the next, object will be created again for doing activity again.
And i want to create just one EPPLUS object for many times of activity, i want to create one EPPLUS object can be used many times, not using "using" statement.
this is my code
public partial class FMain : Form
{
...
ExcelPackage pack;
FileInfo InfoPathFile;
public StringPathFile = ""
...
public FMain()
{
...
}
private void NewDialog_FileOk(object sender, CancelEventArgs e)
{
if(pack != null)
pack.Dispose();
StringPathFile = NewDialog.FileName;
InfoPathFile = new FileInfo(StringPathFile);
pack = new ExcelPackage(InfoPathFile);
...
}
private void SaveData(float[] Sens, string tt, string dd)
{
var ExSheet = pack.Workbook.Worksheets["Data"];
ExSheet.Cells["A" + rowExcel].Value = Numb;
ExSheet.Cells["B" + rowExcel].Value = Sens[0];
ExSheet.Cells["C" + rowExcel].Value = Sens[1];
ExSheet.Cells["D" + rowExcel].Value = Sens[2];
ExSheet.Cells["E" + rowExcel].Value = Sens[3];
ExSheet.Cells["F" + rowExcel].Value = tt;
ExSheet.Cells["G" + rowExcel].Value = dd;
//pack.SaveAs(InfoPathFile);
pack.Save();
}
I want to write to excel many times, using just one EPPLUS object, i dont want to create epplus object every time i do an activity. Using my code, i can just write once to excel file, and second writing process is failed.
Can i do that?
The problem you are having is calling the Save() will automatically close the package so the next time you write to it it will generate an error. EPPlus isnt really meant to do "incremental" saves like that - its more designed to sit on a server, have the client tell it to generate a file all at once, and send it to the client.
I think the best bet would be to keep a copy of it in memory and incrementally write the file. You could do something like this via MemoryStream. So create class-level MemoryStreamvar and use that to hold the work-in-progress Excel Package. This hopefully demonstrates that concept:
[TestMethod]
public void Multi_Save_Test()
{
//http://stackoverflow.com/questions/28007087/how-to-write-to-excel-many-times-using-one-object-of-epplus-in-c-sharp
var existingFile = new FileInfo(#"c:\temp\temp.xlsx");
if (existingFile.Exists)
existingFile.Delete();
//Use memstream and create the package but WITHOUT the FI so it is a memory stream as well
//Avoid using and call manual dispose
var holdingstream = new MemoryStream();
var pack = new ExcelPackage();
var ExSheet = pack.Workbook.Worksheets.Add("Data");
ExSheet.Cells["A1"].Value = "wer";
ExSheet.Cells["B1"].Value = "sdf";
//Do an incremental save to the file and copy the stream before closing - ORDER COUNTS!
pack.SaveAs(existingFile);
holdingstream.SetLength(0);
pack.Stream.Position = 0;
pack.Stream.CopyTo(holdingstream);
//*********************************************************
//reopen the holding stream, make a change, and resave it
pack.Load(holdingstream);
ExSheet = pack.Workbook.Worksheets["Data"];
ExSheet.Cells["A2"].Value = "wer";
ExSheet.Cells["B2"].Value = "sdf";
//Another incremental change
pack.SaveAs(existingFile);
holdingstream.SetLength(0);
pack.Stream.Position = 0;
pack.Stream.CopyTo(holdingstream);
//*********************************************************
//reopen the holding stream, make a change, and resave it
pack.Load(holdingstream);
ExSheet = pack.Workbook.Worksheets["Data"];
ExSheet.Cells["A3"].Value = "wer";
ExSheet.Cells["B3"].Value = "sdf";
//Another incremental change
pack.SaveAs(existingFile);
holdingstream.SetLength(0);
pack.Stream.Position = 0;
pack.Stream.CopyTo(holdingstream);
//*********************************************************
//reopen the holding stream, make a change, and do a FINAL save
pack.Load(holdingstream);
ExSheet = pack.Workbook.Worksheets["Data"];
ExSheet.Cells["A4"].Value = "wer";
ExSheet.Cells["B4"].Value = "sdf";
//All done so only need to save it to the file
pack.SaveAs(existingFile);
//cleanup
pack.Dispose();
holdingstream.Dispose();
}
I'm new to C# and Windows Phone developing, I need to do app for school project.
I have simple class, and I want to save it to have access later, on next start of application.
What is the best (and easiest) method to do that? Using file, database or some other application memory?
Here is my class:
public class Place
{
private string name;
private string description;
private int distance;
private bool enabled;
private GeoCoordinate coordinates;
}
I need to store multiple instances of class.
There is no "best" way to do it; it depends on how you're going to use it.
A simple way is to use serialization, to XML, JSON, binary or whatever you want. I personally like JSON, as it's very lightweight and easy to read for a human. You can use the JSON.NET library to serialize objects to JSON.
For instance, if you want to serialize a collection of Place to a file, you can do something like that:
static async Task SavePlacesAsync(ICollection<Place> places)
{
var serializer = new JsonSerializer();
var folder = ApplicationData.Current.LocalFolder;
var file = await folder.CreateFileAsync("places.json", CreationCollisionOption.ReplaceExisting);
using (var stream = await file.OpenStreamForWriteAsync())
using (var writer = new StreamWriter(stream))
{
serializer.Serialize(writer, places);
}
}
And to read it back from the file:
static async Task<ICollection<Place>> LoadPlacesAsync()
{
try
{
var serializer = new JsonSerializer();
var folder = ApplicationData.Current.LocalFolder;
var file = await folder.GetFileAsync("places.json");
using (var stream = await file.OpenStreamForReadAsync())
using (var reader = new StreamReader(stream))
using (var jReader = new JsonTextReader(reader))
{
return serializer.Deserialize<ICollection<Place>>(jReader, places);
}
}
catch(FileNotFoundException)
{
return new List<Place>();
}
}
I think the easiest way to make persistent your object is to store them in a file. However this is not the best way due to the time spent in IO operations, low security, etc.
Here you have a nice example:
How to quickly save/load class instance to file
After studying the manual of Visual C#, I'm starting to program a simple app for Windows 8. I'm using Visual Studio 2013 Express and .NET Framework 4.5.1. In the code I've written up to now, in the code-behind of a page I create a list of people:
private Dictionary<string, People> listPeople = new Dictionary<string, People>();
After this, I wish this list would fill a ComboBox control of another page. The solution I thought is to save the Dictionary<string, People> variable in roaming, and then use it where I need. In this way also would solve the problem of maintaining the list of people saved even when the app is terminated.
How can I do?
did you already try it like that?
var applicationData = Windows.Storage.ApplicationData.Current;
applicationData.RoamingSettings.Values["PeopleList"] = listPeople;
var applicationData = Windows.Storage.ApplicationData.Current;
var listPeople = (Dictionary<string, People>)applicationData.RoamingSettings.Values["PeopleList"];
The overwhelming advice is to serialize your setting as a string then set the value in Values (or save the values in the complex type individually).
But, I think that really ignores the fact that Values is a series of key-value pairs... If you want to support complex types, I think creating your own file in the roaming folder is a better idea. I've written a small helper class to accomplish this via JSON serialization:
public class ApplicationStorageHelper
{
private static readonly JsonSerializer jsonSerializer = JsonSerializer.Create();
public static async Task<bool> SaveData<T>(T data)
{
var file =
await ApplicationData.Current.RoamingFolder.CreateFileAsync("settings.dat", CreationCollisionOption.ReplaceExisting);
using (var stream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
using (var outputStream = stream.GetOutputStreamAt(0))
{
using (var writer = new StreamWriter(outputStream.AsStreamForWrite()))
{
var jsonWriter = new JsonTextWriter(writer);
jsonSerializer.Serialize(jsonWriter, data);
return true;
}
}
}
}
public static async Task<T> LoadData<T>()
{
try
{
var file =
await ApplicationData.Current.RoamingFolder.GetFileAsync("settings.dat");
using (var inputStream = await file.OpenSequentialReadAsync())
{
using (var reader = new StreamReader(inputStream.AsStreamForRead()))
{
var jsonReader = new JsonTextReader(reader);
return jsonSerializer.Deserialize<T>(jsonReader);
}
}
}
catch (FileNotFoundException)
{
return default(T);
}
}
}
You'll have to reference JSON.Net from Nuget or some other way.
I have an app for Windows Store and what i am trying to do is read text from a file. I have two textFields. The descriptionTextField accepts new lines.
// Read from file
public async Task ReadFile()
{
try
{
// get the file
StorageFile notesStorageFile = await localFolder.GetFileAsync("NotesData.txt");
var readThis = await FileIO.ReadLinesAsync(notesStorageFile);
foreach (var line in readThis)
{
notesRepository.Add(new Note(line.Split(';')[0], line.Split(';')[1]));
}
Debug.WriteLine("File read successfully.");
}
catch (FileNotFoundException ex)
{
Debug.WriteLine("Error1: " + ex);
}
}
Now if NotesData.txt has:
Eggs;description eggs;
it works file.
But if NotesData.txt has:
Groceries;buy 10 eggs
buy 1 kg meat;
I get the index out of bound error. I just cant figure out how to fix the ReadFile() code.
The exception appears when i am calling the method. The problem i believe is with the descriptionTextBox that can accept new lines.
NotesData.txt
Apples;description apples; // works ok
Pears; description line 1
description line 2
description line 3; // problem
Pears; description line 1; // works ok
It seems to me you're trying to read back contents of a file you have previously saved and the problems you're having are just a consequence of the format you have selected for saving the data in the first place. Looking at it, new lines are not the only difficulty you're going to be having. What if the user decides to enter a semicolon in one of the textboxes? Are you preventing that?
I suggest you abandon your own serialization format and rather use one of the existing ones. If your notesRespository is a List<Note> this could be your (de)serialization code for XML:
private async Task Save(List<Note> notesRepository)
{
var xmlSerializer = new XmlSerializer(typeof (List<Note>));
using (var stream = await ApplicationData.Current.LocalFolder.OpenStreamForWriteAsync("notes.xml", CreationCollisionOption.ReplaceExisting))
{
xmlSerializer.Serialize(stream, notesRepository);
}
}
private async Task<List<Note>> Load()
{
var xmlSerializer = new XmlSerializer(typeof(List<Note>));
using (var stream = await ApplicationData.Current.LocalFolder.OpenStreamForReadAsync("notes.xml"))
{
return (List<Note>) xmlSerializer.Deserialize(stream);
}
}
And this for JSON:
private async Task Save(List<Note> notesRepository)
{
var jsonSerializer = new DataContractJsonSerializer(typeof (List<Note>));
using (var stream = await ApplicationData.Current.LocalFolder.OpenStreamForWriteAsync("notes.json", CreationCollisionOption.ReplaceExisting))
{
jsonSerializer.WriteObject(stream, notesRepository);
}
}
private async Task<List<Note>> Load()
{
var jsonSerializer = new DataContractJsonSerializer(typeof(List<Note>));
using (var stream = await ApplicationData.Current.LocalFolder.OpenStreamForReadAsync("notes.json"))
{
return (List<Note>)jsonSerializer.ReadObject(stream);
}
}
When the repository gets too large to always load and save it as a whole you could even consider a structured storage like SQLite.
This line:
notesRepository.Add(new Note(line.Split(';')[0], line.Split(';')[1]));
assumes that you'll always have at least one semi-colon in a line. If you've got a line in your file which doesn't have that (e.g. a blank line) then it will fail.
It's not clear where that's where your problem is, because you haven't said where the exception's coming from, but that would be my first guess.
I'd also only do the split once:
string[] bits = line.Split(';');
if (bits.Length >= 2)
{
// What do you want to do with lines with more than one semi-colon?
notesRepository.Add(bits[0], bits[1]);
}
else
{
// Handle lines without a semi-colon at all.
}