Leviticus 16 is the central biblical passage for Yom Kippur. It describes the work of the high priest, the sacrifices, the entrance into the Holy of Holies and the sending away of the living goat.
Leviticus 23 places the Day of Atonement within Israel's sacred calendar. The people were commanded to rest and practice self-denial. Numbers 29 describes additional offerings.
Isaiah 58 is traditionally read on Yom Kippur. It warns against treating fasting as an outward religious act while ignoring injustice and the needs of other people.
The book of Jonah is traditionally read during the afternoon service. Its themes of repentance and mercy fit the meaning of the day.
Hebrews provides the strongest New Testament connection. Chapters 9 and 10 draw directly upon the sanctuary and sacrificial system. Jesus is presented as the greater high priest whose sacrifice accomplishes what repeated animal sacrifices could not accomplish.
Acts 27:9 refers to "the Fast." This has traditionally been understood as a reference to the Day of Atonement and helps establish the time of year in the account of Paul's voyage.