Got annoyed today, and finally went searching to find the cause of the weird lighting glitch on these gazebo objects.  You’ve seen this problem right from the beginning when these things were simple square-shapes, where part of the roof of one of these objects would have a weird hard edge, as you can see in the image on the left.

Basically what I’m doing here is figuring out which direction the surface is facing for each point on the model.  On some bits of the model (such as the cylinders at the bottoms of the columns), the lighting is smooth across the whole shape.  In other places on the model, there are a hard edges, or “creases” in the surface.  The general approach is to look at every edge in the model and figure out how sharp a bend is occurring across it.  By default, everything is creased.  But if the bend over a vertex is shallow enough, then I  remove the “crease”, and make the surface smooth.

As you can see here in the roof of this object, there’s a crease between the outer face of the gear-shaped roof, and the faces which connect them to the inner portion of the shape.  The problem was that here we had two different basic smooth shapes being constructed;  the wall of the roof, and the curving top part of the roof.  At the bottom part of the roof, these two bits are different enough to cause a sharp crease between them.  However, as we move up the roof, the difference between these two shapes lessens, and many vertices could legally be “uncreased” and merged into either one of those two shapes;  the side or the top of the roof.  What was happening before was that when it found a vertex that could be assigned to either, it would pick one or the other more or less at random.

My fix has been to change it to instead figure out which shapes a vertex can be merged onto, and then pick the best one, instead of picking one at random.  With only a little bit of work to explain to the system how I define “the best one”, it seems to have made a big improvement!