The focal length of a simple lens is the distance from the center of the lens to the point of convergence (the sensor plane) when the lens is focused at infinity. This distance is both an optical and a physical distance.
Camera lenses, however, are not simple lenses.
The focal length is the optical, not physical, distance from the rear node of the lens to the sensor plane when the lens is focused at infinity.
Two things to keep in mind. First, don't confuse infinity focus with depth of field. They are entirely different. The lens is focused at infinity at only one position of the focus adjustment. Aperture has nothing to do with it.
Second, a complex camera lens has no optical center. Instead it has a front node and a rear node. The front node is where the "center" would be when viewed from the front of the lens, and is usually behind the rear node. Similarly, the rear node is where the "center" would be when viewed from the rear of the lens, and is usually in front of the front node. In fact, the rear node may not be within the body of the lens at all, but somewhere in front of the front element.
The focal length of a lens is a physical property of the lens itself, and has nothing whatsoever to do with the sensor size. The so-called "crop factor" affects only the angle of view, not the focal length.