I agree with your remark, but would like to make an addition: apart from diet and exercise, there is a third variable that controls the calorie balance: culture/customs.
For example, the amount of clothing one wears and the temperature at which one keeps ones living quarters in winter affects calories 'out', as does what one does while waiting for a cab/train/appointment (standing still vs walking around). Those are areas where I think easy gains can be made, even when forgetting about the elephant in the room 'get out of your car', which, apparently, is very hard to do for many people.
And, as with all dieting related stuff, for most people, there is no need for drastic action. Gaining 3 grams of body weight per day brings you 20 kg extra body weight going from 20 years to 40 years of age; that is what happens to most people. Your diet/exercise plan, similarly, is good enough if it leads to 3 grams of weight loss a day.
Except that it's not that easy. Your body will conserve energy if you're hungry, making you feel colder and more lethargic. If you go without a sweater, you'll burn more calories and end up eating more.
And, as with all dieting related stuff, for most people, there is no need for drastic action. Gaining 3 grams of body weight per day brings you 20 kg extra body weight going from 20 years to 40 years of age; that is what happens to most people. Your diet/exercise plan, similarly, is good enough if it leads to 3 grams of weight loss a day.