User documentation contains documentation for running the X server and basic X clients, as well as protocol specifications and basic client programming documentation.
Developing X applications
For low-level X development, XCB, the X C Bindings provide a clean low-level protocol binding. Its older cousin Xlib (or libX11), is not recommended for new development, but is still very widely used. Xt is a similarly deprecated library for building toolkits. Documentation for Xlib and Xt is available.
Protocol documentation, man pages, etc. of the most recent X11R7 release: http://www.x.org/releases/X11R7.7/doc/index.html
Downloading, running, and developing X.Org code
The development documentation describes how to download and build the X.Org code (including the X server and drivers), and rough notes on their internals.
The X New Developer’s Guide attempts to be a gentle introduction to how the entire stack is put together and how some of the pieces work.