Melt butter over low heat in a small saucepan. Mix in the sugar, cinnamon, nutmeg and allspice. Add the apple slices, cover, and let simmer for 5 minutes, or until the apples are tender. Mix the apple mixture into the drained sweet potatoes along with the milk. Mix well using an electric mixer or just a fork until potatoes are mashed.
As a variation, you can mix in cloves. This is also delicious topped with brown sugar and pecans.