Fourth castle completed today after having to go on a walkabout. Got to the boss and for the life of me could not defeat it so figured that I needed another spell to take it out, turns out I needed the reflect spell to bounce back it's attacks.
To get this I needed to find a child hidden in a cave which I accidently stumbled on but I did also find a jar of magic to give me another block so it was worth the walk. After that the boss was easy enough and after a little bit more wandering I found the fifth castle. Had a bit of a wander round it collecting keys and will hopefully complete it tomorrow.