In a saucepan over medium heat, combine the coconut milk and sugar. Bring to a light boil, then reduce the heat and stir until the sugar is dissolved. Set aside to cool.
Once the coconut milk mixture has cooled to at least room temperature, add the milk, bananas and peanut butter to a blender.
Blend until smooth.
Pour the mixture into popsicle molds and freeze for 7 hours, or overnight.