I think a caftan can be very elegant with the right accessories. Yes, you need to be mature to wear one, but that's because it requires a certain amount of sophistication, of savoir faire, IYKWIM. Which you have in spades.
That means no fuzzy slippers, though.