DbSet.Local.Any() performance issue when upgraging EF Core from 2.2.6 to 3.1.3

There is a method which is responsible for returning a single GitCommit object based on its hash from our git commit cache database table. We use Entity Framework Core, and when I've upgraded the nuget packages from 2.2.6 to 3.1.3 I've realized a strange performance issue. Beginning with the second call of the method with 2.2.6 the condition _dbContext.GitCommitCache.Local.Any() took 0-1ms to perform, on 3.1.3 is takes usually around 150-220ms. I'm running this method around 9 to 10 thousand times and its a huge performance bottleneck. Is there anything related that's changed under the hood? Does this have any trivial solutions to keep the original performance?

    public async Task<GitCommit> GetCommitAsync(string hash)
    {
        try
        {
            await _semaphore.WaitAsync();
            if (!_dbContext.GitCommitCache.Local.Any())
            {
                await _dbContext.GitCommitCache.LoadAsync();
            }
            var cacheEntry = await _dbContext.GitCommitCache.FindAsync(hash);

            if (cacheEntry != null)
            {
                return cacheEntry;
            }

            var commit = await GetCommitDataAsync(hash);

            try
            {
                _dbContext.GitCommitCache.Add(commit);
                await _dbContext.SaveChangesAsync();
            }
            catch (DbUpdateException)
            {
                _dbContext.Entry(commit).State = EntityState.Detached;
            }

            return commit;
        }
        finally
        {
            _semaphore.Release();
        }
    }