While testing fourth revision of my power supply board, I noticed that it was buzzing, couple of cycles of power later it stopped. After some research I discovered that the output of the 5V LDO voltage regulator was outputting 0.5V triangle wave at 800Hz. Well, this is awkward, nothing like this happened with the old board. I have like 10µF of ceramic capacitance on the output to make this not happen.
A bit of reading later and I found out that ceramic capacitors on output of low dropout linear regulators cause instability. To combat this, you can put resistors in series with your capacitors or add more load. I powered only one low power opamp from there. Adding 100Ω resistor to ground really fixed it. From the last revision I moved decoupling caps closer to the regulator so that lowered the ESR even more. In the datasheet of the regulator it says nothing about choosing output capacitor. The only place in mentions capacitors is: "Cout = 680 nF in series with 1Ω". So, everybody knows that you are supposed to put resistors in series with your ceramics?
I expect the problem to go away when adding all the other load to 5V load, but still an interesting problem.