How to restrict entity reference displaying values?

How to restrict entity reference displaying values?

I have a requirement which I feel quiet complex. I am using Entity reference in X content type to reference a content type(assume Y). This works fine by displaying all the values of Y in X.
Y ...