The Baal Haturim has a different explanation in connection with the first meeting between Yosef and his brothers.
The traditional explanation as to why the brothers did not recognize Yosef, was that he now had a full beard. When they last saw him, he was a seventeen year old with only a stubble on his face. Now he was a grown man of thirty, and he did not look anything similar to the brother that they remembered.
The Baal Haturim questions this interpretation. He felt that the real reason that Yosef was not recognized, is that they never believed it was possible that their brother could have risen to such a position of prominence.
The Midrash tells us that they did search for Yosef around Egypt. But they looked for him in places of the lower class, hoping they might find him there.
It is not realistic that they would not have noticed some kind of familiar family trait in Yosef, had they really looked. They clearly negated any possibility that this could really be Yosef.
It is not uncommon for families to label a sibling or family member in a certain way. When that individual surpasses these expectations, they are in a state of shock. We must always see the good and potential in everyone. G-d has His plans, and we never know who might be blessed with wealth and success. And we are often surprised when an apparently apathetic student, eventually blossoms to become a great scholar.
Yosef’s brothers certainly placed a negative label on their brother. They learned the hard way, that they should never negate one’s potential. Had they been able to do so, they would have realized that the great Egyptian leader intimidating him, was really their brother!