I'm sorry, but this is the "Business of Photography" forum. We're not talking about taking happy snaps of fellow employees at the company picnic for the internal website. We're dealing with a business for a product product to be used commercially.
To think that charging for work would "cost" him anything job wise is insane. Not even earn as much as he'd get working at his desk for the same amount of time?
Why are you sorry?
Your approach to business is just different than mine. In this particular situation, I don't believe the OP should charge his employer for his photos, especially if it involves only his time and there are no hard costs involved. I further qualified my statement by adding only if the OP enjoys their job and likes their employer. In my 20+ years of experience in the professional business world, I have seen where selfless acts by an employee that go above and beyond their job description often go rewarded with much more than what the OP can hope to get paid for his time taking the photos. We're talking about four to six images of four different buildings to post on the company's website to advertise leasing the buildings, not on a public billboard on Times Square.
If the OP works a thankless job for an impersonal boss at a large corporate giant where loyalty is frequently unrewarded, then by all means charge away.
Also, had this been the boss' daughter's wedding and on a weekend, then I would agree with you. Furthermore, If the OP felt inclined to take such the hypothetical wedding job, I would recommend obtaining a signed contract with specified and agreed-upon terms. However, if the OP enjoys their job and likes their employer, I probably wouldn't take a job photographing his boss' daughter's wedding, but that's more of a personal decision.