Stackoverflow Exception Entity Framework 3.1.3 when Seeding Large Dataset

entity-framework entity-framework-core

Question

When running the update-database command to my mssql server to seed a large dataset parsed from a CSV I am getting a stack overflow exception.

I seed the data from a number of legacy CSV files in OnModelCreating.

It is a large data set and contains around 9.5k lines of input which is split across a number of POCOs.

-verbose gives no further information on the issue other than it is trying to apply the migration.

I am using Visual Studio 2019 Community and EntityFrameworkCore 3.1.3.

Can I safely comment out some of the seeding (i.e. seed 50%) in the up() method of the migration?

How would I then seed the remaining data safely? (would EF on the OnModelBuilding method check the data on the next add-migration and generate the missing data? or would it assume it was seeded after checking the migrations table?

1
0
3/27/2020 4:06:35 PM

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge

Accepted Answer

Your initialcreate.Up(MigrationBuilder) method is too big and is blowing up the JITter. The StackOverflow is occuring when the MSIL for the method is compiled to machine language code.

This is because there's just too much code of the form

  migrationBuilder.InsertData(
        table: "Addresses",
        columns: new[] { "Id", "Address1", "Address2", "Address3", "Country", "DateCreated", "IsArchived", "IsPrimaryAddress", "LastModifiedDate", "LastModifiedUser", "NickName", "Notes", "PostCode", "Region", "Town" },
        values: new object[,]
        {
            { 1, "669A George Street", null, null, "", new DateTime(2020, 3, 28, 14, 58, 51, 108, DateTimeKind.Utc).AddTicks(1514), null, true, new DateTime(2020, 3, 28, 14, 58, 51, 108, DateTimeKind.Utc).AddTicks(1545), null, null, null, "AB25 3XP", "", "Aberdeen" },
            { 12307, "7 Moredon Road", null, null, "", new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8799), null, true, new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8800), null, null, null, "SN25 3DQ ", "", "Swindon" },
            { 12306, "", null, null, null, new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8735), null, false, new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8736), null, null, null, null, null, null },
            { 12305, "67 HIGH STREET, ", null, null, "", new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8690), null, true, new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8690), null, null, null, "TQ95 5NU", "", "TOTNES" },
            { 12304, "", null, null, null, new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8577), null, false, new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8579), null, null, null, null, null, null },
            { 12303, "BOWS KITCHEN, 10 MARGATE PLACE,", null, null, "", new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8530), null, true, new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8531), null, null, null, "CT9 1EN", "", "MARGATE" },
            { 12302, "", null, null, null, new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8464), null, false, new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8465), null, null, null, null, null, null },
            { 12308, "", null, null, null, new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8849), null, false, new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8850), null, null, null, null, null, null },
            { 12301, "270 ST SAVIOURS RD", null, null, "", new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8417), null, true, new DateTime(2020, 3, 28, 14, 58, 51, 198, DateTimeKind.Utc).AddTicks(8418), null, null, null, "LE5 4HF", "", "LEICESTER" },

in the method. To resolve this, the best method would be to read the data for each InserData from disk before each call. Or to embed the data in your source code as, say JSON strings and convert it to an object[,] at runtime.

1
3/28/2020 4:49:13 PM

Related Questions

Entity Framework Core 1.0 - How to seed a large amount of data

I am using an ASP.NET Core 1.0 (previously known as ASP.NET 5) application with the core framework and Entity Framework Core 1.0 (previously known as Entity Framework 7)....In this blog ...https://blogs.msdn.microsoft.com/dotnet/2015/11/18/entity-framework-7-rc1-available... the people in charge of development of EF Core 1.0 say that ...it is not r...
asp.net-core entity-framework-core seeding sqlbulkcopy
asked by diegosasw

Entity Framework bulk inserts - creating a large number of objects / entities is the slow part?

There's a lot of information and answers on here and elsewhere about how to speed up bulk inserts with EF....However, I'm only inserting about 10,000 records and the slow part of creating the entities....First, I save the external data to a datatable, then loop through and for each row create a new ...entity..., add the child entities in the loop (...
c# entity-framework entity-framework-6
asked by RemarkLima

Entity Framework Core 2.0 migrations update database with seed data throwing error

I am using EF Core 2.0 code first migration. I am trying to run the migration sin to the new database. I am using seed data. I am getting error if i use seed data and no error if i comment out the seed data method. Not sure what is the problem. Can anyone please help?...Error:...System.Data.SqlClient.SqlException (0x80131904): There is already an o...
c# entity-framework-core
asked by Mukil Deepthi

Log DbContext.SaveChanges Exception to the database in Entity Framework/Entity Framework Core

In my asp.net core web application, I am logging the exceptions to the database.Till now everything works fine but problem is that when the ...DbConext.SaveChanges... in try block throws an exception, cannot log the exception to the database as ...DbConext.SaveChanges... in ...catch block... also throws the same exception....Here is the code I hav...
asp.net-core c# entity-framework entity-framework-core
asked by TanvirArjel

Entity Framework dataset mapping

All of my DAL functions are using ...dbContext.Database.SqlQuery... to map stored procedure results in business logic objects. ...My application became more complicated and I'm looking for a modern, "up to date" way to handle the following situations. I know that I can achieve this using the low-level ADO.NET component like ...SqlDataReader... and ...
asp.net c# entity-framework entity-framework-6
asked by Omtechguy

Query with large WHERE clause causes timeout exception in EF6 with npgsql

I have a query that looks something like this:...private static IQueryable<MultiframeModule> WhereAllFramesProperties(this IQueryable<MultiframeModule> query, ICollection<Frame> frames) { return frames.Aggregate(query, (q, frame) => { return q.Where(p => p.Frames.Any(i => i.FrameData.ShaHash == frame.FrameData.ShaHash)); }); } ....
c# entity-framework entity-framework-6 npgsql
asked by Sassa

Is it possible/advisable to seed Users/Roles using the EFCore 2.1 Data Seeding system?

As of EFCore 2.1 it's possible to seed data using the DbContext OnModelCreating method....https://docs.microsoft.com/en-us/ef/core/modeling/data-seeding...However user/role creation is usually handled by a UserManager/RoleManager....Now, I can manually create a Role or User - but for example in the case of a User I need to hash the password myself ...
asp.net-core entity-framework-core
asked by David Kirkpatrick

Add-Migration with seeds throwing "no value provided" exception

I am trying to create a package that will help seed databases with data from JSON files which can be created manually or extracted from a legacy database. Currently I have created a generic class ...SeedCreator<T>... which retrieves JSON from a given file (the name of the entity plus ....json...) and deserialises it into an object of the given type...
.net-core asp.net-core-2.1 c# ef-core-2.1 entity-framework-core
asked by Tubs

Entity Framework exception: No suitable constructor found for entity type 'CultureInfo'

I have this:...model class...public class BusinessSelectList { public int Id { get; set; } public string Name { get; set; } public bool IsBusinessGroup { get; set; } [NotMapped] public bool IsSelected { get; set; } = false; } ...DbContext...public class AppDbContext : DbContext { public AppDbContext(DbContextOptions<AppDbCon...
asp.net-core-2.1 c# entity-framework-core
asked by Muflix

Entity Framework Core 3.1 is throwing "Include has been used on non entity queryable" exception

I have an ASP.NET Core 3.1 based project where I am using Entity Framework Core 3.1 as an ORM....I have the following two entity models...public class PropertyToList { public int Id { get; set; } public int ListId { get; set; } public int UserId { get; set; } public int PropertyId { get; set; } // ... Other properties removed f...
.net-core .net-core-3.1 asp.net-core entity-framework entity-framework-core
asked by user12310517

Fastest Entity Framework Extensions

Bulk Insert
Bulk Delete
Bulk Update
Bulk Merge
View more on Stack Overflow

Prime Library

Performance

Expression Evaluator

More Projects...

Related

Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow
Licensed under: CC-BY-SA with attribution
Not affiliated with Stack Overflow