Describes a graph-based approach to image processing, intended for use with images obtained from sensors having space-variant (SV) sampling grids. The connectivity graph (CG) is presented as a fundamental framework for posing image operations in any kind of SV sensor. SV sensor systems cover wide solid angles yet maintain high acuity in their central regions. Implementation of SV systems pose at least two outstanding problems. First, such a system must be active, in order to utilize its high acuity region; second, there are significant image processing problems introduced by the non-uniform pixel size, shape and connectivity. Familiar image processing operations take on new and different forms when defined on SV images. This paper provides a general method for SV image processing, based on a CG which represents the neighbor relations in an arbitrarily structured sensor. We illustrate this approach with the following applications: (1) Connected components are reduced to a graph-theoretic counterpart. (2) We show how to write local image operators in the CG that are independent of the sensor geometry. (3) We relate the CG to pyramids over irregular tesselations, and implement a local binarization operator in a 2-level pyramid. (4) We expand the CG into a transformation graph, which represents the effects of geometric transformations in SV image sensors. Using this, we define an efficient algorithm for matching in the logmap images and solve the template matching problem for SV images. The CG approach to image processing is suitable for real-time implementation.