Thats probably why sorting function gets errors. Maybe this line: v.reserve(v.size() + number); is problematic for your compiler. It can be omitted and use push_back instead of emplace. But it's not important.
I also thank you, I can get rid of bitmasks finally
About shader, I thought that part: if(pixel.a != 0.0){..} is slow, but with this code result was same:
uniform sampler2D texture;
uniform vec4 color_id;
void main(){
vec4 textureColor = texture2D(texture, gl_TexCoord[0].xy).rgba;
gl_FragColor = vec4(color_id.r, color_id.g, color_id.b, textureColor.a);
}
Btw. tool for shaders which looks usable
https://github.com/dfranx/SHADERed