Spent a little time today looking at improving one small aspect of sculpt drawing: flat-shaded faces in solid view mode with VBO enabled.
In current trunk, flat shading in sculpt mode looks quite different depending on whether VBO is enabled or not. When VBO is not enabled, and immediate mode is used to draw the mesh, each face (triangle or quad) gets drawn separately. Before drawing the face, a single averaged normal and mask value is calculated. The normal and mask values are used for each vertex of the face. In contrast, VBO drawing uses a shared vertex normal and mask value for every face that uses the vertex. This behavior is correct for smooth shading, but looks strange for flat shading.
The benefit to using a shared vertex value is lower storage requirements; the mesh’s faces can simply reference vertices with an index (and the index can usually be a short, so two bytes per index.) When flat shading is enabled, however, OpenGL uses only a single vertex’s value for the entire face. On any particular face this generally looks OK, but the problem is evident when adjacent quads appear to have the same normal despite facing different directions.
Left: current trunk. Middle: averaged normals and interpolated mask. Right: averaged normals and averaged mask.
I’ve implemented a simple fix for this, available from the sculpt-flat-shading branch on Github. If the mesh is flat shaded, the index buffer is scrapped in favor of duplicating vertex values. The GPU storage requirements are in general slightly higher, but the output looks much more correct.
In the image above you can see two variations of the fix. Both have flat-shaded normals, but the middle uses interpolated mask values while the right uses a single average mask value. Using an averaged mask value seems to look a bit better, although it’s technically a bit inaccurate as sculpt brushes operate on vertex mask values, not the average.
One further issue which affects smooth shading is that interpolation over quads doesn’t look very good with OpenGL. Mask values and normals don’t look smooth because OpenGL tessellates quads into two triangles. The vertex values are interpolated within each triangle, without regard to the missing corner. It’s technically possible to solve this with shaders. Instead of using regular vertex attributes, the vertices just get texture coordinates and the attributes are put into a texture. A bilinear (or even bicubic) interpolation can then be coded in the shader. Such fanciness is probably of questionable usefulness though, and at the very least will have to wait on the completion of other improvements to the various drawing systems in Blender.
My current plan is to wait until 2.64 is released, then commit the flat-shading fix (with averaged mask values.)