EF 6 Fluent API with One to Many AND Many To Many and conflicting multiplicities

I haven't been able to find an answer to this specific problem, perhaps to my newness with Entity Framework

I have two Entities: User and Facility.

The Facility table requires an Owner (user), but a Facility can also have many "FacilityUsers" assigned to a Facility.

public class User
{
    ....
    public virtual ICollection<Facility> Facilities { get; set; }
}

public Facility Facility
{
    public Guid OwnerId { get; set; }
    ...
    public virtual User Owner { get; set; }
    public virtual ICollection<User> FacilityUsers { get; set; }
}

Here is my ModelBuilder

////Many To Many: Users To Facilities
modelBuilder.Entity<User>()
    .HasMany(i => i.Facilities)
    .WithMany(u => u.FacilityUsers)
    .Map(m =>
    {
        m.MapLeftKey("UserId");
        m.MapRightKey("FacilityId");
        m.ToTable("UserFacility");
    });        

//One to Many: Facility To Owner
modelBuilder.Entity<Facility>()
    .HasRequired<User>(s => s.Owner)
    .WithMany(s => s.Facilities)
    .HasForeignKey(k => k.OwnerId);

The problem is, when I have the map Facility to Owner, I get the following error: The navigation property 'FacilityUsers' declared on type 'MEH.Web.Models.Entities.Facility' has been configured with conflicting multiplicities.

However, when I exclude the Facility to Owner mapping, it works.

//One to Many: Facility To Owner
//modelBuilder.Entity<Facility>()
//    .HasRequired<User>(s => s.Owner)
//    .WithMany(s => s.Facilities)
//    .HasForeignKey(k => k.OwnerId);

The problem is, the Facilities table then has a Owner_UserId field that is null, but the OwnerId is correctly filled in. My OCD won't let me move on until I have resolved the "Owner_UserId" problem.

Thank you, D