glsl Dithering shader

Dithering is nothing new. But not many people have used it as a stylistic choice. Not until recently, where the creator of Obra Dinn has been putting it to good use. This website was a good start on how to achieve this effect: http://devlog-martinsh.blogspot.com/2011/03/glsl-8x8-bayer-matrix-dithering.html. Using this as a starting point, and building off the pipeline for Urho3d, this is the resulting code so far:

#include "Uniforms.glsl"
#include "Samplers.glsl"
#include "Transform.glsl"
#include "ScreenPos.glsl"

#ifdef EDGE
  varying vec2 vTexCoord;
  varying vec4 vScreenPos;
#endif

#ifdef COMPILEPS

const vec4 ma = vec4(0.0,12.0,3.0,15.0);
const vec4 mb = vec4(8.0,4.0,11.0,7.0);
const vec4 mc = vec4(2.0,14.0,1.0,13.0);
const vec4 md = vec4(10.0,6.0,9.0,5.0);
const mat4 dm = mat4(ma,mb,mc,md);

float find_closest(int x, int y, float c0){
  float limit = 0.0;
  limit = ( dm[x][y] + 1.0 )/64.0;

  if(c0 < limit)
    return 0.0;
  return 1.0;
}
#endif

void VS(){
    mat4 modelMatrix = iModelMatrix;
    vec3 worldPos = GetWorldPos(modelMatrix);
    gl_Position = GetClipPos(worldPos);

    #ifdef EDGE
      vScreenPos = GetScreenPos(gl_Position);
      vTexCoord = GetTexCoord(iTexCoord);
    #endif
}

void PS(){

    #ifdef EDGE
      vec2 screenuv = vScreenPos.xy / vScreenPos.w;

      //vec4 lum = vec4(0.299, 0.587, 0.114, 0);
      vec4 lum = vec4(0.15, 0.15, 0.15, 0);

      float grayscale = dot(texture2D(sEnvMap, screenuv), lum);
      vec3 rgb = texture2D(sEnvMap, screenuv).rgb;


      vec2 xy = (vScreenPos.xy / vScreenPos.w)*(1.0/cGBufferInvSize);
      //int x = int(mod(xy.x, 8));
      //int y = int(mod(xy.y, 8));
      int x = int(mod(xy.x, 4));
      int y = int(mod(xy.y, 4));

      //vec3 finalRGB;
      //finalRGB.r = find_closest(x, y, rgb.r);
      //finalRGB.g = find_closest(x, y, rgb.g);
      //finalRGB.b = find_closest(x, y, rgb.b);

      float final = find_closest(x, y, grayscale);
      //gl_FragColor = vec4(finalRGB, 1.0);
      gl_FragColor = vec4(final,final,final,1.0);
    #endif
}

One problem that was encountered was dealing with arrays in glsl 1.10. Apparently it is not possible to create and array with default values. The was to solve this problem was to define the array. Then to assign the values. Based on the linked example above, if we are using an 8x8 array, then we are talking about 64 values. Which are be set every fragment. On a 1366x768 (which is what my lenovo laptop monitors dimensions are) that is 1,049,088 iterations. Surprisingly, even with those numbers I was getting decent (60+) fps with that resolution. However, once attempting that on a stronger desktop with 1920x1080 resolution, the frame rates dropped to an abysmal 8 fps. 2,073,600 iterations, twice as many iterations was enough to break the effect. We need something faster. According to this page http://bisqwit.iki.fi/story/howto/dither/jy/, Mr Photoshop Thomas Knoll, gets fantastic results with a 4x4 matrix. Let's attempt that instead, since 4x4 matrices are supported by glsl without the need to even use an array. Just dropping the matrix to a 4x4 sped things up a bit, but on a 1920x1080 display we are still at a slow 20 fps. And the quality took a little bit of a hit as well, but in the end speed is king, and the difference in motion is negligible. More optimization is required, but for now, this will do. Some paths to explore later will be to actually render a smaller image and scale up. To dither a full HD image might defeat the purpose. Looking further at the latter mentioned page, more research is needed to get a more pleasing color image. The current code looks decent with greyscale, but not as well with color.