@goldenfenix This is because you are building with a standard gcc.
You may try the patches I've put in http://moinejf.free.fr/opi2/, but some modules would not compile with loboris's .config (I don't remember which).
Otherwise, you may start with my config.
Without patch you can only compile this kernel source with included SDK binary toolchain (4.6) or for example (maybe simpler) with ubuntu 12.04and installing toolchain (4.6.4) with apt-get install.
Boards:
orangepi plus, olinuxino A20, cubieboard A10, mele A2000 .....
scaling_cur_freq changes constantly...
try to run some cpu intensive program and at the same check scaling_cur_freq, it will go to scaling_max_freq.
Or install CPU frequency applet...
480000 (480MHz) is the minimal CPU freq.