A bounce occurs whenever a user enters the page and subsequently exits without visiting another page on the website or interacting with any of the elements on the page (e.g. commenting). Because the user did not visit any other pages, this is defined as a single-page-session and all exits from these kinds of sessions would result in a bounce as well as an exit. (typically a session is 30 mins by default unless adjusted otherwise)
Exit rate, on the other hand, looks at the number of people who exit your website after landing on a page and compares it to the total number of views the page received. Exit rate is the percentage of visits that were the last in the session whereas the bounce rate is the percentage of visits that were the only one of the sessions.
This means bounces are only recorded if a user exits directly from the page they first entered while exit rates are recorded at the end of their visit to the website. Therefore, all bounces are exits but not all exits are bounces
A good analogy would be to think of throwing a ball at a wall. The ball bounces off the wall indicating that the user entered a website and left or became inactive for the whole session. If the ball sticks and then started moving to other parts of the wall (the user navigated to other parts of the website) there is no bounce recorded.
Only after navigating around the website the user then decides to leave, GA records this as an exit for that particular page. The average exit number is calculated by dividing 100% by the number of pages the user had navigated to.
In this 1st scenario, the visitor had left the page they first landed on when entering a website resulting in a single-page-session, so GA picks this up as both a bounce and an exit
An example of this would appear on the Tile as Bounce Rate = 100% and Exit Rate = 100%
In the 2nd scenario, there was an exit after page 2 (after looking at page 1 and then page 2, the user left the website) However, this does not count as a bounce because it was not an exit from a single-page-session (the user navigated to page 2 from other pages of the website) This scenario would have a 50% Exit Rate because 2 pages were visited and the user left after Page 2.
100% divided by 2 pages visited = 50% Exit Rate
This scenario would also have a 0% Bounce Rate because the user did not leave or become inactive after landing on page 1