Preheat oven to 350 degrees.

Grease a 9-inch cake pan with butter, then cut a piece of parchment paper large enough to line bottom. Place parchment in pan, easing the sides in and pressing to smooth folds along sides of pan, then butter parchment. Place melted butter in bottom of the pan and sprinkle evenly with brown sugar.

Place peaches onto clean paper towels to remove excess juice, then arrange peaches in concentric circles on top of brown sugar; you should be able to fit almost the entire bag of peach slices into pan.

In a medium bowl, whisk together cake flour, almond flour, baking powder, baking soda, and salt. In an electric mixer, beat together remaining 1/2 cup butter and granulated sugar until very light and fluffy. Add eggs, vanilla extract, and almond extract and mix until just incorporated. Alternately add flour mixture and buttermilk in 3 batches, mixing at low speed until just combined.

Spoon batter evenly over peaches, then smooth top with the back of a spoon. Transfer to oven and bake until golden, 55 minutes to 1 hour. Remove from oven and set aside briefly, then run a thin knife around edge of pan and invert a large plate on top of pan. Using pot holders to hold pan and plate together, working carefully, quickly invert pan to release cake onto plate. Lift pan off of cake and carefully peel away parchment from sides and top of cake.

Serve cake warm or at room temperature with whipped cream, vanilla ice cream, and mint leaves.