There has been a couple of correct answers, but I'll try to give a simple explanation (I wont use correct terms, they exist but they could be more confusing):
1) OP:s assumption is wrong: the inverse square law applies to all light, sources and reflected.
2) Lets picture a white wall, illuminated with light (doesnt matter how). Wall is 2x2 meters with total area of 4 square meters.
3) Lets take a photo of the wall with a 100mm f/4 lens so that the wall fills the whole picture. There is certain amount of total light hitting the sensor, lets assume that equal to 1. (imaginary measure)
4) Ok, lets double our distance from the wall. Now, according to inverse square law, the amount of light should be 1/4 of what it used to be. This is correct, and at the same time the apparent dimensions of the wall have been cut by 1/2. Now the apparent area of the wall is also 1/4 (1/2*1/2) of what it used to be.
5) Humen (and cameras) see brightness of an object as (amount of light) per (apparent area). In this case, after backing down with the camera, the brightness of the wall is still the same (1/4 : 1/4 = 1). However, the size of the wall in a photo is 2x smaller (1/4 area)
6) Now, as we doubled our distance to the wall, we need to use a 200/4 lens to achieve same framing and exposure (first we had 100/4). Since the F-number (f/4 in this case) is only a ratio of focal length and actual light-gathering aperture opening, we have to take that into account.
Lets see what is the ratio of the light-collecting area of 100/4 and 200/4 lenses. Area of a circle is pi*radius^2 and as we double focal lenght and apertures radius, the area of the aperture grows 4 times larger.
In our example, this compensates the 1/4 diminishing of the light by having a 4-times larger hole to collect the light.
Case closed? 