Shorter path for cached many2one and reference
With rev 4d040118e5e7, we use global cache for Many2One and Reference field but as those fields are often eager loading, they still go through the eager loading path even if they have already been read.
I think we can skip this code to speed up the reuse of cached id.