As a software developer, I have a solid understanding of relational databases and normalization and have designed many. But what is the best or most pragmatic answer to the whole debate of 'when to use the real candidate key, when to use an identity PK', and the byproduct concerns around joining on keys that are strings (varchar) as opposed to an identity?

I found a writeup on this exact question: Choosing a Primary Key: Natural or Surrogate?

About half way down it sums up the tradeoff Personally I have been burned more than once by choosing a natural key only to have the business change later on. My rule of thumb is to go with surrogates unless it can be guaranteed that the natural will not change definition.

an even more important rule of thumb is: never assign a surrogate key until you have also declared a UNIQUE key elsewhere in the same table

Agree with that.
I use surrogate keys on all my tables, for consistency of design and coding. But I also declare a unique natural key.