I encountered a very similar scenario awhile back. I was shooting off the main (public) road, but my car and my tripod were probably on Chevron property. I was outside of the fence, but it was still probably their property.
Anyways, within 1.5 min of setting up my tripod, a truck drove up and I was detained by security. After a few questions about who I was, with me answering them pretty vaguely, and a few calls on the radio to his superior, I was allowed to leave. They did not try and have me remove the images though, which I wouldn't have done anyways.
I have shot several refinery locations and never had anyone bat an eyelash, except for this time.