EF Core - Invalid attempt to call CheckDataIsReady when reader is closed (OutOfMemory)

.net-core ef-core-2.1 entity-framework entity-framework-core out-of-memory

Question

I have a table dbo.Cache in SQL server, with two Columns:

  • Key : varchar(50)
  • Value : nvarchar(max)

I am planning to store large strings in Value column (>30mb) and query them in multiple threads.

So the problem that when I do more then 9 queries in parallel, it starts throwing exception System.InvalidOperationException : Invalid attempt to call CheckDataIsReady when reader is closed:

[Fact]
public void TestMemory()
{
    const string key = "myKey1";
    //re-creating record with Value = large 30mb string
    using (var db = new MyDbContext())
    {
        var existingRecord = db.CachedValues.FirstOrDefault(e => e.Key == key);
        if (existingRecord!=null)
        {
            db.Remove(existingRecord);
            db.SaveChanges();
        }

        var myHugeString = new string('*',30*1024*1024);
        db.CachedValues.Add(new CachedValue() {Key = key, Value = myHugeString});
        db.SaveChanges();
    }

    //Try to load this record in parallel threads, creating new dbContext
    const int threads = 10;
    Parallel.For(1, threads, new ParallelOptions(){MaxDegreeOfParallelism = threads}, (i) =>
    {
        using (var db = new MyDbContext())
        {
            var entity = db.CachedValues.FirstOrDefault(c => c.Key == key);
        }
    });
}

Tried to execute GC.Collect(); GC.WaitForFullGCComplete(); before/after each db read - didn't not help

Tried to mimic this behavior on the lower level directly reading data through sqlDataReader.ExecuteReader(CommandBehavior.SequentialAccess) - it throws OutOfMemoryException

1
1
12/18/2018 11:25:08 PM

Popular Answer

So after investigation I figured out that it's just an OutOfMemory problem because after 8th parallel request allocating 30mb*2 (since it's unicode char) amount of memory allocated by .Net actually goes over 1.2GB in my app, which is enough for my workstation .net runtime to start chocking on lack of memory (https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/fundamentals#ephemeral-generations-and-segments).

So having that said there is not much else I can come up with just on retrying the read (mem allocate) operation until it succeed with forcing GC to collect in the catch block.

Retry.Action(() =>
{
    using (var db = new MyDbContext())
    {
        var entity = db.CachedValues.FirstOrDefault(c => c.Key == key);
    }
}).OnException((OutOfMemoryException exception) =>
{
    GC.Collect();
    GC.WaitForFullGCComplete();
    return true;

})
.OnException((InvalidOperationException exception) =>
{
    if (exception.Message != "Invalid attempt to call CheckDataIsReady when reader is closed.")
    {
        return false;
    }

    GC.Collect();
    GC.WaitForFullGCComplete();
    return true;

})
.Run();

Please let me know if you know some better solution for that

0
12/19/2018 2:45:32 PM


Related Questions





Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow