It seems that I had to disable logging to disk, and re-enable it for it to populate logging configs using newer template after upgrading to 24.1.