Measurement of three-jet distributions in photoproduction at HERA