In a separate bowl, cream the butter and brown sugar with an electric mixer. Add the egg and beat until the mixture is smooth. With the mixer on low speed, add the dry ingredients, alternating with the milk. Using a wooden spoon, gently fold in the almonds and spread the mixture in the cake pan. Refrigerate for 15 minutes.