EF Core DbContext Complex Many To Many Relationship

I have just switched to EFCore from EF6 and I am having trouble coming up with words for my question.

Basically, there is a Many-To-Many relationship that I managed myself in my previous DbContext, but I thought that I would put in some hard yards now and see if I can get EF to work its magic for me.

Here is the setup

My application allows for users to create their own objects. The Users create the ObjectType and then add ObjectFields to the type. Then they can create Objects of ObjectType (e.g. Term with a field of name and description). Once they have created an Object, it can be linked to folders that are in an existing system.

I keep the Objects in a table that looks similar to below

--------------------------------------------------
| ObjectID |    ObjectFieldID     | Value         |
--------------------------------------------------
|   1      |       1              | Term One Name |
--------------------------------------------------
|   1      |       2              | Term One Desc |
--------------------------------------------------
|   2      |       1              | Term Two Name |
--------------------------------------------------
|   2      |       2              | Term Two Desc |
--------------------------------------------------

Here are my classes:

Object

public class Object
{
    [Key]
    public int ObjectID { get; set; }
    [Key]
    public int ObjectFieldID { get; set; }
    public string Value { get; set; }


    public virtual ObjectField ObjectField { get; set; }

    //Object's Folders
    public virtual ICollection<FolderObject> FolderObjects { get; set; }
}

Folder

public class Folder
{
    [Key]
    public int FolderID { get; set; }

    public virtual ICollection<FolderObject> FolderObjects { get; set; }
}

Now, I am trying to make a Table that will allow me to hold the Many-To-Many relationship, but when a user creates one of their custom objects, it is composed ob multiple Objects.

Here is my class for the many to many relationship:

FolderObject

public class FolderObject
{
    public int FolderID { get; set; }
    public Folder Folder { get; set; }

    public int ObjectID { get; set; }
    public ICollection<Object> Object{ get; set; }
}

and here is how I am ATTEMPTING to configure the relationship using the Fluent API.

modelBuilder.Entity<FolderObject>()
            .HasOne<Folder>(fo => fo.Folder)
            .WithMany(f => f.FolderObject)
            .HasForeignKey(fo => fo.FolderID);

///HERE is my issue.... How do I create this relationship?
modelBuilder.Entity<FolderObject>()
            .HasOne<ICollection<Object>>(fo => fo.Object)
            .WithMany();//Error thrown here since the "One" is a Collection

This is probably fairly unusual and I can go on managing the Many-To-Many relationship in the same table manually as I was with EF6, but I was hoping that someone out there may have had a similar scenario and found a solution.

All help is appreciated. I can understand that my explanation of my setup could be confusing. Very willing to explain it further.