The Rama comments on the Shulchan Aruch, Orach Chayim 610:4: There are those who wrote that it is customary to dress in clean, white clothes on Yom Kippur, analogous to the ministering angels, and likewise it is customary to wear a kittel which is white and clean, and it is also the clothing of the dead (the shroud), and therefore the heart of a man is humbled and broken (Hagahot Maimoniot).
On Yom Kippur we are similar to the angels as we don’t eat or drink but rather focus on spirituality. The angels are described as wearing linen (which was white) in Yechezkel 9:2 and Daniel 12:6.
Rabbi Eliyahu Kitov tells the following story in the Book of our Heritage:
A righteous person told his congregation on Yom Kippur: Children of Israel! Take to heart that it is in white garments like these we are wearing now, that we shall ascend to the World to Come to be judged and give accounting before the King of all kings, the Holy One, blessed is He. Let us then imagine that we are standing in this clothing before the Throne of Glory to be judged and give our final accounting. We should have true remorse, for one who stands before the Throne of Glory is truly remorseful…
In Yishayahu 1:18, we see the power of Tshuva (repentance):
Come, let us reach an understanding,—says the LORD. Be your sins like crimson, they can turn snow-white; be they red as dyed wool, they can become like fleece.
White is symbolic of erasing our sins and starting anew.
The Kohen Gadol (High Priest) wore white linen on Yom Kippur. By wearing white, we are emulating him as we see in Vayikra 16:4:
He shall be dressed in a sacral linen tunic, with linen breeches next to his flesh, and be girt with a linen sash, and he shall wear a linen turban. They are sacral vestments; he shall bathe his body in water and then put them on.—
In the Talmud, Shabbat 119a, Rav Hamnuna said: we honor Shabbat with food and drink but how will we honor Yom Kippur since we don’t eat or drink? The Torah said to honor it with a clean garment.
In the Mishna, Taanit 4a, we learn that Yom Kippur was one of the days that the women went out wearing white:
Rabban Shimon ben Gamliel said: There were no days as joyous for the Jewish people as the fifteenth of Av and as Yom Kippur, as on them the daughters of Jerusalem would go out in white clothes…And the daughters of Jerusalem would go out and dance in the vineyards.
The Talmud, Taanit 30b explains why Yom Kippur is a happy day:
Yom Kippur is a day of joy because it has the elements of pardon and forgiveness, and moreover, it is the day on which the last pair of tablets were given.
We see from here that by wearing white we humble ourselves yet at the same time we are festive and happy, hoping that all of our sins will be forgiven and that we will be starting off the new year with a clean slate.