Develop Reference

How to handle an IEnumerable of IDisposable objects not knowing if the results are yield or not? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
I'm looking for best practices / standard on how to deal with this situation.
We have our code (MyClass) that consumes another class (ItemGenerator).
ItemGenerator is a blackbox for us so we don't know the implementation (we do but we don't want to rely on that because it could change from underneath).
ItemGenerator has a method, GetItems(), that returns an IEnumerable of Item.
Item class implements IDisposable so we should dispose of the object when we are done.
When we (MyClass) iterates through the list of items, if an exception (any exception) occurs, we want to stop processing and release control (bubble up the exception).
My question is this:
Should we keep iterating through the items in order to dispose of all of them? It might seem silly but what happens with the rest of the items if they are not disposed of?
At the same time, based on the code below, we should definitely not iterate through the rest of the items because they are yield return. So why generate them just so we can dispose of them (it could significantly affect the performance).
The problem is that we do not know if GetItems() returns the items on demand (yield) or not. And I don't think we should care, right?
So how should we handle the situation when an exception occurs in the middle of the list (for example)?
Below is an example of the code that illustrates the gist of it.
This is our code:
public class MyClass
{
public void VerifyAllItems()
{
ItemGenerator generator = new ItemGenerator();
foreach (Item item in generator.GetItems())
{
try
{
// Do some work with "item" here. Though an exception could occur.
// If an exception occurs, we don't care about processing the rest of the items and just want to bubble up the exception
}
finally
{
// Always dispose of the
item?.Dispose();
}
}
}
}
And this is the blackbox code
public class ItemGenerator
{
private long _itemsToGenerate = 0;
public ItemGenerator()
{
_itemsToGenerate = new Random().Next(10, 100);
}
public IEnumerable<Item> GetItems()
{
while (_itemsToGenerate > 0)
{
yield return HeavyWork();
_itemsToGenerate--;
}
}
private Item HeavyWork()
{
// Doing a lot of work here
return new Item();
}
}
public class Item : IDisposable
{
private bool _isDisposed = false;
public virtual void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
private void Dispose(bool isDisposing)
{
if (!_isDisposed)
{
if (isDisposing)
{
// Dispose of any resources
}
_isDisposed = true;
}
}
}

The problem is worse than you state. Not only can you not be sure whether to enumerate the collection, you can't be sure whether to dispose of any of the items at all. Just because something implements IDisposable doesn't mean you should be disposing it, e.g. if you have a factory that always returns the same instance of something.
This kind of a problem is exactly why the code that allocates something is generally held responsible for deallocating it. In this case, the ItemGenerator creates the items, so it should dispose them.
class ItemGenerator : IDiposable
{
protected readonly List<Item> _instances = new List<Item>();
IEnumerable<Item> GetItems()
{
for ( some; condition; here; )
{
var item = new Item();
_instances.Add(item);
yield return item;
}
}
public void Dispose()
{
foreach (var item in _instances) item.Dispose();
}
}
Now all you have to do is put your ItemGenerator in a using block and you're good to go.
public void VerifyAllItems()
{
using (ItemGenerator generator = new ItemGenerator())
{
foreach (Item item in generator.GetItems())
{
try
{
// Do some work with "item" here. Though an exception could occur.
// If an exception occurs, we don't care about processing the rest of the items and just want to bubble up the exception
}
finally
{
//Don't need to dispose anything here
}
}
} //Disposal happens here because of the using statement
}
With this patterns, any items that were allocated by the ItemGenerator will get disposed when you exit the using block.
Now the caller doesn't have to care about the implementation of the item generator at all, or worry about disposing anything other than the generator itself. And of course you should dispose the generator if you're the one who allocated it.

Slightly bizarre, but...
internal class DisposingEnumerator : IEnumerator<Item>
{
private readonly List<IDisposable> deallocationQueue = new List<IDisposable>();
private readonly IEnumerable<Item> source;
private IEnumerator<Item> sourceEnumerator;
public DisposingEnumerator(IEnumerable<Item> source)
{
this.source = source;
}
public bool MoveNext()
{
if (sourceEnumerator == null)
{
sourceEnumerator = source.GetEnumerator();
}
bool hasNext = sourceEnumerator.MoveNext();
if (hasNext)
{
deallocationQueue.Add(Current);
}
return hasNext;
}
public Item Current => sourceEnumerator.Current;
object IEnumerator.Current => Current;
public void Reset()
{
throw new NotSupportedException();
}
// Will be called within "foreach" statement
// You can implement IDisposable in ItemCollection as well
public void Dispose()
{
foreach (var item in deallocationQueue)
{
item.Dispose();
}
}
}
public class ItemCollection : IEnumerable<Item>
{
private IEnumerator<Item> enumerator;
public ItemCollection(IEnumerable<Item> source)
{
this.enumerator = new DisposingEnumerator(source);
}
public IEnumerator<Item> GetEnumerator()
{
return enumerator;
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
Usage:
var items = generator.GetItems();
// Equivalent of using statement
foreach (var item in new ItemCollection(items))
{
}
Proof

Implementing a safe access container

Assume I have some class that represents a container. That container holds some public properties with get and set modifiers.
What I want is to implement some mechanism that will enable access and disable access to these properties reference at runtime.
For example, when some boolean flag is true, you can access these properties. That means that:
SomeClass.Property1;
Will not generate an exception and will return the object.
However, when it is false, the above line of code will throw an exception.
It is of course possible to be done when using some boolean key, and checking it at the gateway to every property.
My question is, is it possible to implement such mechanism that will enfoce these limitations for all the properties in the class, without the need to assert these conditions within every access to these properties.
Thanks for helping.

It looks like null object pattern might helps.
Simple code that shows how it can be used in your case. Not exactly the same as you want but it doesn't need to assert conditions with every access to object's properties and methods.
Entities:
abstract class AbstractEntity
{
public abstract void DoSomething();
public abstract void DoSomethingElse();
public abstract int Property { get; set; }
}
class RealEntity : AbstractEntity
{
public override void DoSomething()
{
Console.WriteLine("Something");
}
public override void DoSomethingElse()
{
Console.WriteLine("Something else");
}
public override int Property { get; set; }
}
class NullEntity : AbstractEntity
{
public override void DoSomething()
{
// do nothing or throw exception
}
public override void DoSomethingElse()
{
// do nothing or throw exception
}
public override int Property
{
get { throw new Exception(); }
set { throw new Exception(); }
}
}
Simple example of AccessContainer:
class AccessContainer
{
private RealEntity _entity = new RealEntity();
private NullEntity _nullEntity = new NullEntity();
private bool _access = true;
public AbstractEntity Entity
{
get => _access ? (AbstractEntity) _entity : (AbstractEntity) _nullEntity;
}
public void OpenAccess()
{
_access = true;
}
public void DenyAccess()
{
_access = false;
}
}
Usage:
var container = new AccessContainer();
container.Entity.DoSomething(); // prints something
var prop = container.Entity.Property; // access to property
container.DenyAccess();
container.Entity.DoSomething(); // do nothing
container.OpenAccess();
container.Entity.DoSomething(); // prints something again
container.DenyAccess();
var prop2 = container.Entity.Property; // exception

What you are asking for doesn't natively exist, you're going to have to write some sort of wrapping functionality to test whether accessibility is granted.
public interface IAccessOwner {
bool Accessible { get; }
}
[DebuggerDisplay("Accessible: {Accessible,nq} - Value: {ToString()}")]
[DebuggerTypeProxy(typeof(RestrictedObject<>.DebuggerProxy))]
public class RestrictedObject<T> {
private readonly IAccessOwner _owner;
private T _value;
public RestrictedObject(IAccessOwner owner, T initialValue)
: this(owner) {
_value = initialValue;
}
public RestrictedObject(IAccessOwner owner) {
_owner = owner ?? throw new ArgumentNullException(nameof(owner));
}
public T Value {
get {
ThrowIfInaccessible();
return _value;
}
set {
ThrowIfInaccessible();
_value = value;
}
}
public bool Accessible => _owner.Accessible;
public override string ToString() {
if (!Accessible)
return "<Inaccessible>"; // ToString should never throw
if (_value is { } val)
return val.ToString();
return "<null>";
}
private void ThrowIfInaccessible() {
if(!Accessible)
throw new InvalidOperationException("Not accessible!");
}
// explicit operator to cast directly to value
public static explicit operator T(RestrictedObject<T> ro) {
ro.ThrowIfInaccessible();
return ro.Value;
}
private sealed class DebuggerProxy {
public bool Accessible { get; }
public T Value { get; }
public DebuggerProxy(RestrictedObject<T> ro) {
bool acc = Accessible = ro.Accessible;
if (acc)
Value = ro._value;
}
}
}
You can then use properties of this type in your class:
public class MyClass : IAccessOwner {
private readonly RestrictedObject<int> _prop1;
private readonly RestrictedObject<string> _prop2;
public MyClass(int someVal) {
_prop1 = new RestrictedObject<int>(this, someVal);
_prop2 = new RestrictedObject<string>(this);
Accessible = true;
}
public bool Accessible { get; private set; }
// you determine how you want to toggle the above property.
// Exposing it publicly defeats the purpose of all of this,
// but for demo purposes only:
public void DenyAccess() {
Accessible = false;
}
public void AllowAccess() {
Accessible = true;
}
// these properties will throw exceptions if the owner
// (this object) is not currently accessible.
public int Prop1 {
get => _prop1.Value;
set => _prop1.Value = value;
}
public string Prop2 {
get => _prop2.Value;
set => _prop2.Value = value;
}
// alternatively return the wrapper itself
// allowing you to control the accessibility
// even after returning the object
public RestrictedObject<string> AltProp2 => _prop2;
}
You would then use it like the following (obviously exceptions will halt the execution, handling has been elided):
var mc = new MyClass(3);
Console.WriteLine(mc.Prop1); // prints 3
Console.WriteLine(mc.Prop2); // prints null
var temp = mc.AltProp2; // use the wrapper directly
mc.Prop2 = "Hello";
Console.WriteLine(mc.Prop2); // prints Hello
Console.WriteLine(temp.Value); // prints Hello
Console.WriteLine((string)temp); // explicit operator, prints Hello
mc.DenyAccess();
mc.Prop1 = 33; // throws!
Console.WriteLine(mc.Prop1); // throws!
Console.WriteLine(mc.Prop2); // throws!
Console.WriteLine(temp.Value); // throws!
Console.WriteLine((string)temp); // explicit operator, throws!
Console.WriteLine(temp); // prints "<Inaccessible>"
mc.AllowAccess();
string temp3 = (string)temp; // "Hello", explicit operator works again
mc.Prop1 = 22; // as do our setters
mc.Prop2 = "Goodbye";
if (temp.Accessible) {
Console.WriteLine(temp); // "Goodbye"
}
The only thing that won't throw an exception is the override of ToString on the RestrictedObject type itself since you should never throw from ToString. Instead we just return <Inaccessible>.
We've also changed how the RestrictedObject<T> is displayed in a debugger via the DebuggerTypeProxyAttribute. If someone tries to inspect the object's properties they will see the Accessible property and only if true will the wrapped object's Value appear. Otherwise, default(T) will be displayed (null for reference types, 0 for integral types and false for bool). Furthermore, through use of the DebuggerDisplayAttribute we've customized the display of the collapsed version of our object such that it shows the Accessible property alongside our customized ToString.
Note that this still has the drawback that if someone retrieves the inner/wrapped object and accessibility has later been denied, they still have the object. There's nothing you are going to be able to do to really guard against that case. You must also realize (and accept) that anyone using reflection could alter or access the state of the object if they really wanted to.
I will also note that this violates normal C# practices, which typically dictate that properties should not throw exceptions. Microsoft's own guidelines say as much, though they use the term "Avoid" rather than "Do Not". The framework itself is guilty of violating this "rule". If you're going to violate the principle of least surprise, at the very least have the courtesy to document this behavior for consumers of your API.

LINQ continue after Take

Say we have an IEnumerable<T> stuff;
Is there a concise way to Take n elements and then another m elements after the first, without re-evaluating?
example code:
stuff.Take(10);
stuff.Skip(10).Take(20); // re-evaluates stuff
What I was thinking was maybe this (not working code)
var it = stuff.GetEnumerator();
it.Take(10);
it.Take(20);
Edit to add to the difficulty and to clarify the complexity of what I would like to accomplish: I want to continue the query after the Take, i.e.
it.Take(10);
var cont = it.Select(Mutate);
cont.Take(20);
cont = cont.Where(Filter);
cont.Take(5);

You can use the Publish extension method in the System.Interactive NuGet package put out by Microsoft to accomplish this. This is a fantastic library that provides some 'missing' LINQ functions. From the documentation, the Publish method:
Creates a buffer with a view over the source sequence, causing each enumerator to obtain access to the remainder of the sequence from the current index in the buffer.
I.e. it allows you to partially enumerate a sequence and the next time you enumerate the sequence you will pick up where the previous enumeration left off.
var publishedSource = stuff.Publish();
var firstTenItems = publishedSource.Take(10).ToArray();
var nextTwentyTransformedItems = publishedSource.Take(20).Select(Mutate).ToArray();
// How you apply 'Where' depends on what you want to achieve.
// This returns the next 5 items that match the filter but if there are less
// than 5 items that match the filter you could end up enumerating the
// entire remainder of the sequence.
var nextFiveFilteredItems = publishedSource.Where(Filter).Take(5).ToArray();
// This enumerates _only_ the next 5 items and yields any that match the filter.
var nextOfFiveItemsThatPassFilter = publishedSource.Take(5).Where(Filter).ToArray()

If you want to just create a wrapper for IEnumerable that will handle any LINQ appended on and take one pass through the source, use this class and extension:
public static class EnumerableOnceExt {
public static EnumerableOnce<IEnumerable<T>, T> EnumerableOnce<T>(this IEnumerable<T> src) => new EnumerableOnce<IEnumerable<T>, T>(src);
}
public class EnumerableOnce<T, V> : IEnumerable<V>, IDisposable where T : IEnumerable<V> {
EnumeratorOnce<V> onceEnum;
public EnumerableOnce(T src) {
onceEnum = new EnumeratorOnce<V>(src.GetEnumerator());
}
public IEnumerator<V> GetEnumerator() {
return onceEnum;
}
IEnumerator IEnumerable.GetEnumerator() {
return onceEnum;
}
public void DoSkip(int n) {
while (n > 0 && onceEnum.MoveNext())
--n;
}
public void DoTake(int n) {
while (n > 0 && onceEnum.MoveNext())
--n;
}
#region IDisposable Support
private bool disposedValue = false; // To detect redundant calls
protected virtual void Dispose(bool disposing) {
if (!disposedValue) {
if (disposing) {
onceEnum.ActuallyDispose();
}
disposedValue = true;
}
}
// This code added to correctly implement the disposable pattern.
public void Dispose() {
Dispose(true);
}
#endregion
}
public class EnumeratorOnce<V> : IEnumerator<V> {
IEnumerator<V> origEnum;
public EnumeratorOnce(IEnumerator<V> src) {
origEnum = src;
}
public V Current => origEnum.Current;
object IEnumerator.Current => origEnum.Current;
public bool MoveNext() => origEnum.MoveNext();
public void Reset() {
origEnum.Reset();
}
public void ActuallyDispose() {
origEnum.Dispose();
}
#region IDisposable Support
protected virtual void Dispose(bool disposing) {
// don't allow disposing early
}
// This code added to correctly implement the disposable pattern.
public void Dispose() {
Dispose(true);
}
#endregion
}
Now your sample code will work if you call EnumerableOnce() to wrap the source, as long as you execute the enumerations:
var it1 = it.EnumerableOnce();
it1.Take(10).ToList();
var #continue = it1.Select(Mutate);
#continue.Take(20).ToList();
#continue = #continue.Where(Filter);
#continue.Take(5).ToList();
You can also add new methods to EnumerableOnce:
public void DoSkip(int n) {
while (n > 0 && srcEnum.MoveNext())
--n;
}
public void DoTake(int n) {
while (n > 0 && srcEnum.MoveNext())
--n;
}
And call them:
var it1 = it.EnumerableOnce();
it1.DoTake(10);
var #continue = it1.Select(Mutate);
#continue.DoSkip(20);
#continue = #continue.Where(Filter);
#continue.DoTake(5);

C# MongoDB driver only returning 100 results

I am writing a mailing label, and need to print a label for each document.
I have 829 documents on the Collection, but when I retrieve them, I only get 100 documents.
I have this LINQ code:
IMongoCollection Pessoa;
Pessoa = database.GetCollection<Pessoa>(collectionName);
return Pessoa.AsQueryable().ToList();
How to retrieve ALL the documents?

I have 829 documents on the Collection, but when I retrieve them, I only get 100 documents.
I could reproduce the issue on my side, using AsQueryable extension method on IMongoCollection collection.AsQueryable() to find documents in a collection, which always return 100 documents even though I changed Items per page setting to Unlimited on Azure portal.
Setting:
Test code:
Count documents in query explorer:
To query all the documents in a collection, as you mentioned in comment, you could try to call Find method with an empty filter.

You're probably being limited by the default cursor BatchSize.
You can modify this behaviour passing an AggregateOptions object to the AsQueryable extension and setting the BatchSize property to a large enough value.
public static IMongoQueryable<TDocument> AsQueryable<TDocument>(this IMongoCollection<TDocument> collection, AggregateOptions aggregateOptions = null)

I found the question useful, so I wrote a convenient IEnumerator:
private sealed class MongoCollectionEnumerator : IEnumerator<T> {
private IMongoCollection<T> _collection;
private IAsyncCursor<T> _cursor; // outer enumerator
private IEnumerator<T> _currentBatchEnumerator; // inner enumerator
public MongoCollectionEnumerator(IMongoCollection<T> collection) {
_collection = collection;
InternalInit();
}
#region interface implementation
T IEnumerator<T>.Current {
get {
return _currentBatchEnumerator.Current;
}
}
object IEnumerator.Current {
get {
return ThisAsTypedIEnumerator.Current;
}
}
bool IEnumerator.MoveNext() {
if (_currentBatchEnumerator != null) {
if (_currentBatchEnumerator.MoveNext()) {
return true;
}
}
// inner not initialized or already at end
if (_cursor.MoveNext()) {
// advance the outer and defer back to the inner by recursing
_currentBatchEnumerator = _cursor.Current.GetEnumerator();
return ThisAsIEnumerator.MoveNext();
}
else { // outer cannot advance, this is the end
return false;
}
}
void IEnumerator.Reset() {
InternalCleanUp();
InternalInit();
}
#endregion
#region methods private
// helper properties to retrieve an explicit interface-casted this
private IEnumerator ThisAsIEnumerator => this;
private IEnumerator<T> ThisAsTypedIEnumerator => this;
private void InternalInit() {
var filterBuilder = new FilterDefinitionBuilder<T>();
_cursor = _collection.Find(filterBuilder.Empty).ToCursor();
}
private void InternalCleanUp() {
if (_currentBatchEnumerator != null) {
_currentBatchEnumerator.Reset();
_currentBatchEnumerator = null;
}
if (_cursor != null) {
_cursor.Dispose();
_cursor = null;
}
}
#endregion
#region IDisposable implementation
private bool disposedValue = false; // To detect redundant calls
private void InternalDispose(bool disposing) {
if (!disposedValue) {
if (disposing) {
InternalCleanUp();
_collection = null;
}
disposedValue = true;
}
}
void IDisposable.Dispose() {
InternalDispose(true);
}
#endregion
}

Which is the best way to improve memory usage when you collect a large data set before processing it? (.NET)

When I have to get GBs of data, save it on a collection and process it, I have memory overflows. So instead of:
public class Program
{
public IEnumerable<SomeClass> GetObjects()
{
var list = new List<SomeClass>();
while( // get implementation
list.Add(object);
}
return list;
}
public void ProcessObjects(IEnumerable<SomeClass> objects)
{
foreach(var object in objects)
// process implementation
}
void Main()
{
var objects = GetObjects();
ProcessObjects(objects);
}
}
I need to:
public class Program
{
void ProcessObject(SomeClass object)
{
// process implementation
}
public void GetAndProcessObjects()
{
var list = new List<SomeClass>();
while( // get implementation
Process(object);
}
return list;
}
void Main()
{
var objects = GetAndProcessObjects();
}
}
There is a better way?

You ought to leverage C#'s iterator blocks and use the yield return statement to do something like this:
public class Program
{
public IEnumerable<SomeClass> GetObjects()
{
while( // get implementation
yield return object;
}
}
public void ProcessObjects(IEnumerable<SomeClass> objects)
{
foreach(var object in objects)
// process implementation
}
void Main()
{
var objects = GetObjects();
ProcessObjects(objects);
}
}
This would allow you to stream each object and not keep the entire sequence in memory - you would only need to keep one object in memory at a time.

Don't use a List, which requires all the data to be present in memory at once. Use IEnumerable<T> and produce the data on demand, or better, use IQueryable<T> and have the entire execution of the query deferred until the data are required.
Alternatively, don't keep the data in memory at all, but rather save the data to a database for processing. When processing is complete, then query the database for the results.

public IEnumerable<SomeClass> GetObjects()
{
foreach( var obj in GetIQueryableObjects
yield return obj
}

You want to yield!
Delay processing of your enumeration. Build a method that returns an IEnumerable but only returns one record at a time using the yield statement.

The best methodology in this case would be to Get and Process in chunks. You will have to find out how big a chunk to Get and Process by trial and error. So the code would be something like :
public class Program
{
public IEnumerable GetObjects(int anchor, int chunkSize)
{
var list = new List();
while( // get implementation for given anchor and chunkSize
list.Add(object);
}
return list;
}
public void ProcessObjects(IEnumerable<SomeClass> objects)
{
foreach(var object in objects)
// process implementation
}
void Main()
{
int chunkSize = 5000;
int totalSize = //Get Total Number of rows;
int anchor = //Get first row to process as anchor;
While (anchor < totalSize)
(
var objects = GetObjects(anchor, chunkSize);
ProcessObjects(objects);
anchor += chunkSize;
}
}
}