OK, time for some updates. Based in part of some useful feedback on BlenderArtists, I’ve spent the past week experimenting with various approaches to the topological update step. I think it’s now getting much closer to what people want to see; in particular the subdivision is localized to the area directly around the brush.
Until now, the topological updates were based on the paper “Freestyle: Sculpting meshes with self-adaptive topology”. This paper assumes that the entire mesh will have a consistent detail size. Note that since the algorithm collapses short edges, you can lose detail if you change your brush to a coarser detail size.
In contrast, the approach I am using now is to only do edge splits. If you create very dense topology and then set a coarse detail size, the dense topology will remain and not be collapsed. To handle situations where you want to make the topology less dense, a separate simplify brush will be added. (I haven’t done that yet, but the code is largely already written.)
In order to get a good boundary between a dense portion of the mesh and larger polygons, I used a trick that Farsthary described in his unlimited clay research: identify nearby pole vertices (in this case, defined as having nine or more adjacent edges) and split the adjacent edges. This isolates the pole vertex and prevents it from distorting the mesh. This wasn’t an issue with edge collapses, which implicitly force triangles to be relatively regular, but in a subdivision-only approach it turns out this is a really critical step. Without it the mesh quickly develops degenerate triangles that can’t be smoothed away.
Some other changes: the topology update now occurs before each sculpt step, so the result should match the brush shape a bit better. The brush also avoids the accumulation problem now, so repeatedly brushing over an area won’t continually build the surface up (unless the brush accumulate option is on, of course.)
I’m still looking into surface relaxation; I suspect a smoother result can be achieved while still maintaining detail, but haven’t found a good way to do it yet. My current short-term goal is to do some brush work: add the reduce/simplify mentioned earlier and fix the guaranteed-crash brushes like grab.
As usual, the code is available from the dyntopo branch on Github.