Notes on the Design of an X Server in Common Lisp. Alastair Bridgewater, March-April 2005. They always say 'Never Volunteer'. And in this case I volunteered. Oops. Anyway, this is going to be some vague mumblings on how an X Server should be written. Hopefully it's not going to be too far off base. To an initial approximation, an X Server consists of an inbound network interface, bearing requests; an outbound network interface, bearing events and responses; input devices, supplying events; some sort of database for storing atoms; a database for maintaining visual resource information (windows, pixmaps, etc.); a rendering interface for drawing on the display; a font system for keeping track of the screwy X font setup; and probably some other bits as well. And this is the -simplified- model. As an X Server is a fairly crucial part of the end-user experience and is expected to be fast and to remain in working order essentially forever, it would probably be a good idea to keep consing down to a minimum on most simple requests. This initial design sketch may or may not reflect this notion, but it is certainly something to keep in mind during implementation. FIXME: Should probably coalesce paired accessor functions (getter and setf pairs) into single entries. Maybe should think about converting to texinfo or something more wieldly than plain text. Protocol Error Handling Some X Protocol Requests are permitted to return errors to a client program. There are some 17 defined Errors, each with an error code between 1 and 17 inclusive. Errors are sent to the client as a 32-octet structure (those unfamiliar with the X Protocol will notice the similarity between Errors, Repsonses, and Events), with certain information about the generating request and, for 10 of the 17 Errors, the invalid datum which precipitated the error. When a Protocol Error occurs the PROTOCOL-ERROR restart is invoked, being passed the error and optionally the errant datum. This restart is bound during request processing. Some function descriptions in this spec have an entry for Exceptional Situations; this entry lists the Protocol Errors that may be raised from within the function. FIXME: Lay out the details of the PROTOCOL-ERROR restart setup. Do not specify that a restart must be used to handle the protocol-error situations. "Policy, not mechanism." ____________________________________________________________ protocol-error Inline Function error &optional (value 0) Arguments: ERROR is a symbol representing the particular Protocol Error to signal. VALUE is an optional erroneous value to return to the client. Values: Does not return. Purpose: Provides a single-point-of-entry for handling various Protocol Error situations. Client Connection Handling In general, most of the server machinery is uninterested in new client connections. Sure, the client needs to be tied in to the request processing and available for event notifications, but if the client never sends a request then it will never receive a response, event, or error back from the server. Client disconnect, however, does require a notification. A certain amount of cleanup must be done when a client disconnects, depending on the close down mode of the client A full server reset must be done when the last client disconnects if it has a close down mode of DestroyAll (the default). It is anticipated that client disconnect notification will be implemented by means of a special variable containing a list of interested functions that take some notion of the disconnecting client (at this time it is uncertain how exactly a client would be referenced) as their only argument. Each network-client has a notion of a 'current packet', which is a sequence of (unsigned-byte 8) octets representing data received from its peer (typically an X request). These octets are passed as the PACKET-DATA argument to network-client-invoke-packet-received-callback. The current packet has a fixed starting position but a variable size (that is, based on the packet contents, the network-client-packet-received-callback can say that it needs more data to complete the packet and that it should be called again when that data arrives). Packet lengths of at least 16384 octets must be supported by the system. When network-client-packet-size octets have been received for the current packet, network-client-invoke-packet-received-callback is called. The callback should, during its operation, either set a new, longer packet size, or call network-client-packet-complete to indicate that the current packet has been dealt with and that the next packet may be received. After calling network-client-packet-complete, the packet size should be set for the next (now current) packet. There is an obvious case involving receiving a packet, setting a new, shorter length for the packet, and then calling network-client-packet-complete. At this point there will be a number of octets received that are no longer part of the current packet. The correct approach would be to treat these octets as the start of the next packet. In actual practice, we will leave this case explicitly undefined until such time as we need the functionality. Packet reception of X requests is expected to start with receiving a 4-byte packet and calling the request dispatcher. The dispatcher will check the length header and set network-server-packet-size appropriately and return if more octets are expected. When a request is fully received the dispatcher will call network-client-packet-complete followed by setting network-server-packet-size to 4 to indicate that it would like to be called when the next header has been received. FIXME: Need to specify disconnect callbacks and outbound data interface. ____________________________________________________________ network-server Class The basic class representing a network server of some sort. This class is intended as an abstract type, not to be instantiated directly. ____________________________________________________________ make-network-server Function host port &key mp-style connect-callback Arguments: HOST is a host interface, as per SB-BSD-SOCKETS. PORT is the 16-bit TCP port number to listen on. MP-STYLE is required to be :SERVE-EVENT (not the default), for future compatability with possible threaded implementations. CONNECT-CALLBACK is an optional initial value for NETWORK-SERVER-CONNECT-CALLBACK. Values: Returns a network-server object. Purpose: Creating an object that handles incoming client connections on a network port. ____________________________________________________________ network-server-connect-callback Generic Function network-server Arguments: NETWORK-SERVER is a network server instance. Values: The function called when a client attempts to connect to the server, or NIL if there is no such function. Purpose: Accessor function. ____________________________________________________________ (setf network-server-connect-callback) Generic Function function network-server Arguments: FUNCTION is a function of one argument that is to be invoked when a client attempts to connect to network-server. When invoked, it will be passed NETWORK-SERVER as its sole argument. NETWORK-SERVER is a network server instance. Values: Returns FUNCTION. Purpose: Accessor function. It is an error to attempt to set FUNCTION to NIL while the server is running. ____________________________________________________________ network-server-invoke-connect-callback Generic Function network-server Arguments: NETWORK-SERVER is a network server instance. Values: Returns the value of calling the connect callback. Purpose: Signals that a client wishes to connect by calling the connect callback on network-server, passing network-server as its argument. ____________________________________________________________ network-server-start Generic Function network-server Arguments: NETWORK-SERVER is a network server instance. Values: T if starting the server is successful, otherwise NIL. Purpose: Start the network server listening for new client connections. Will signal an error if the server is already running or if connect-callback is NIL. ____________________________________________________________ network-server-stop Generic Function network-server Arguments: NETWORK-SERVER is a network server instance. Values: T if stopping the server is successful, otherwise NIL. Purpose: Stop the network server from listening for new client connections. Will signal an error if the server is not already running. ____________________________________________________________ network-server-status Generic Function network-server Arguments: NETWORK-SERVER is a network server instance. Values: :RUNNING if the server is listening for new client connections, otherwise :STOPPED. Purpose: Determine the current state of the network server. ____________________________________________________________ network-server-accept Generic Function network-server &key Arguments: NETWORK-SERVER is a network server instance. Values: A new network-client instance. Purpose: Accept a new client connection. ____________________________________________________________ network-client Protocol Class The basic class representing a connection to a client of some sort. This class is intended as an abstract type, not to be instantiated directly. In actual practice, we'll probably stick all the functionality in here for now. ____________________________________________________________ network-client-disconnect Generic Function network-client Arguments: NETWORK-CLIENT is a network client instance. Values: Returns a primary value of NIL. Purpose: Closes the connection to the remote client. ____________________________________________________________ network-client-packet-size Generic Function network-client Arguments: NETWORK-CLIENT is a network client instance. Values: The number of bytes expected for the current packet. Purpose: Accessor function. ____________________________________________________________ (setf network-client-packet-size) Generic Function new-size network-client Arguments: NEW-SIZE is the number of octets expected for the current packet. NETWORK-CLIENT is a network client instance. Values: NEW-SIZE. Purpose: Accessor function. Sets the expected length of the current packet. When the current packet is completely received then the packet received callback is invoked. ____________________________________________________________ network-client-packet-received-callback Generic Function network-client Arguments: NETWORK-CLIENT is a network client instance. Values: The function called when the current packet being received is complete (network-client-packet-size octets in length), or NIL if there is no such function. Purpose: Accessor function. ____________________________________________________________ (setf network-client-packet-received-callback) Generic Function function network-client Arguments: FUNCTION is a function of two arguments that is to be invoked when a packet being received is complete (network- client-packet-size octets in length). When invoked, it will be passed NETWORK-CLIENT and a vector of (unsigned-byte 8) values as arguments. NETWORK-CLIENT is a network client instance. Values: Returns FUNCTION. Purpose: Accessor function. It is not an error to set FUNCTION to NIL at any time, but it would be bad. Don't cross the streams. ____________________________________________________________ network-client-invoke-packet-received-callback Generic Function network-client packet-data Arguments: NETWORK-CLIENT is a network client instance. PACKET-DATA is a vector of (unsigned-byte 8) values representing the received packet. Values: Returns the value of calling the connect callback. Purpose: Signals that a packet has been received by calling the packet received callback on network-client, passing network-client and packet-data as its arguments. ____________________________________________________________ network-client-packet-complete Generic Function network-client Arguments: NETWORK-CLIENT is a network client instance. Values: NIL. Purpose: Tells the network client that we are done with the current received packet. Client State A certain amount of state needs to be kept track of for each connected client. Obvious items include the network-client instance, the current request number, the close-down mode, the assigned resource id range for resource allocation, the preferred byte order, etc. And this is just what I can think of off the top of my head. For performance reasons, we might do best to specify the client state as a Struct rather than as a Class. But we won't. So there. FIXME: Flesh out the remainder of the Client State interface. ____________________________________________________________ client-state Class The repository of basic information regarding a connected client. ____________________________________________________________ client-state-network-client Generic Function client-state Arguments: CLIENT-STATE is a client-state instance. Values: Returns the network-client instance for this client. Purpose: Accessor function. ____________________________________________________________ client-state-byte-order Generic Function client-state Arguments: CLIENT-STATE is a client-state instance. Values: Either :MSB-FIRST or :LSB-FIRST. Purpose: Accessor function. ____________________________________________________________ client-state-resource-id-base Generic Function client-state Arguments: CLIENT-STATE is a client-state instance. Values: The lowest-numbered resource in the assigned range for the client. Purpose: Accessor function. ____________________________________________________________ client-state-resource-id-mask Generic Function client-state Arguments: CLIENT-STATE is a client-state instance. Values: The bitmask indicating the size of the assigned range of Resource IDs for the client. Purpose: Accessor function. ____________________________________________________________ client-state-request-number Generic Function client-state Arguments: CLIENT-STATE is a client-state instance. Values: The sequence number of the current client request. Purpose: Accessor function. ____________________________________________________________ (setf client-state-request-number) Generic Function request-number client-state Arguments: REQUEST-NUMBER is an integer indicating the new request sequence number. CLIENT-STATE is a client-state instance. Values: Returns REQUEST-NUMBER. Purpose: Accessor function. Client Request Dispatch There are three parts to this. At the low level, there is the dispatch functionality. This request opcode to this handler function, and here's how you install a handler function. Next, there's the code to parse a request packet (reply packets are handled under the return path, along with error codes and events). Finally, there's the top level macro that ties it all together. The reason we define all three levels rather than just the top level is that extensions presumably will have their own request parsing needs. The behavior we are looking for from a define-x-request macro would be to define a function to handle a request, including the code to parse the request itself, and register it with the request dispatcher. Taking the request MapWindow for example, we would like: (define-x-request map-window (:opcode 8) ((opcode :u8 :value 8 :error :implementation) (unused :u8) (length :u16 :value 2 :error :length) (window :window)) (declare (ignore unused)) ...) to define a function like: (defun handle-x-request-map-window (client request) (let ((byte-order (client-state-byte-order client)) (#:G0001 0) (#:G0002 0) (#:G0003 0) (#:G0004 0)) (if (eq byte-order :little-endian) (progn (setf #:G0001 (request-le-u8 request 0)) (setf #:G0002 (request-le-u8 request 1)) (setf #:G0003 (request-le-u16 request 2)) (setf #:G0004 (request-le-u32 request 4))) (progn (setf #:G0001 (request-be-u8 request 0)) (setf #:G0002 (request-be-u8 request 1)) (setf #:G0003 (request-be-u16 request 2)) (setf #:G0004 (request-be-u32 request 4)))) (let ((opcode #:G0001) (unused #:G0002) (length #:G0003) (window (x-resource #:G0004))) (declare (ignore unused)) (unless (= 8 opcode) (protocol-error :implementation #:G0001)) (unless (= 2 length) (protocol-error :length #:G0003)) (unless (typep window 'window-resource) (protocol-error :window #:G0004)) ...) and to associate this function with request opcode 8 in the request dispatcher. Another example definition (without expansion this time): (define-x-request set-close-down-mode (:opcode 112) ((opcode :u8 :value 112 :error :implementation) (mode :u8 :typecheck '(integer 0 2)) ; error defaults to :value here (length :u16 :value 1 :error :length)) (setf (client-state-close-down-mode client) (elt '(:destroy :retain-permanent :retain-temporary) mode))) FIXME: Add a define-request-handler macro that does the function-definition part of define-x-request (for use by protocol extensions). ____________________________________________________________ request-be-u8 Inline Function array offset Arguments: ARRAY is an array of (unsigned-byte 8) values. OFFSET is an integer position from which to read. Values: The big-endian (unsigned-byte 8) value starting at position OFFSET within ARRAY. ____________________________________________________________ request-le-u8 Inline Function array offset Arguments: ARRAY is an array of (unsigned-byte 8) values. OFFSET is an integer position from which to read. Values: The little-endian (unsigned-byte 8) value starting at position OFFSET within ARRAY. ____________________________________________________________ request-be-s8 Inline Function array offset Arguments: ARRAY is an array of (unsigned-byte 8) values. OFFSET is an integer position from which to read. Values: The big-endian (signed-byte 8) value starting at position OFFSET within ARRAY. ____________________________________________________________ request-le-s8 Inline Function array offset Arguments: ARRAY is an array of (unsigned-byte 8) values. OFFSET is an integer position from which to read. Values: The little-endian (signed-byte 8) value starting at position OFFSET within ARRAY. ____________________________________________________________ request-be-u16 Inline Function array offset Arguments: ARRAY is an array of (unsigned-byte 8) values. OFFSET is an integer position from which to read. Values: The big-endian (unsigned-byte 16) value starting at position OFFSET within ARRAY. ____________________________________________________________ request-le-u16 Inline Function array offset Arguments: ARRAY is an array of (unsigned-byte 8) values. OFFSET is an integer position from which to read. Values: The little-endian (unsigned-byte 16) value starting at position OFFSET within ARRAY. ____________________________________________________________ request-be-s16 Inline Function array offset Arguments: ARRAY is an array of (unsigned-byte 8) values. OFFSET is an integer position from which to read. Values: The big-endian (signed-byte 16) value starting at position OFFSET within ARRAY. ____________________________________________________________ request-le-s16 Inline Function array offset Arguments: ARRAY is an array of (unsigned-byte 8) values. OFFSET is an integer position from which to read. Values: The little-endian (signed-byte 16) value starting at position OFFSET within ARRAY. ____________________________________________________________ request-be-u32 Inline Function array offset Arguments: ARRAY is an array of (unsigned-byte 8) values. OFFSET is an integer position from which to read. Values: The big-endian (unsigned-byte 32) value starting at position OFFSET within ARRAY. ____________________________________________________________ request-le-u32 Inline Function array offset Arguments: ARRAY is an array of (unsigned-byte 8) values. OFFSET is an integer position from which to read. Values: The little-endian (unsigned-byte 32) value starting at position OFFSET within ARRAY. ____________________________________________________________ request-be-s32 Inline Function array offset Arguments: ARRAY is an array of (unsigned-byte 8) values. OFFSET is an integer position from which to read. Values: The big-endian (signed-byte 32) value starting at position OFFSET within ARRAY. ____________________________________________________________ request-le-s32 Inline Function array offset Arguments: ARRAY is an array of (unsigned-byte 8) values. OFFSET is an integer position from which to read. Values: The little-endian (signed-byte 32) value starting at position OFFSET within ARRAY. ____________________________________________________________ define-x-request Macro name options fields &body body Arguments: NAME is the name of the request. OPTIONS is a list of key / value pairs. At present, the only defined option is :opcode, the opcode number of the X Request being defined, which is required. FIELDS is a list of field descriptions for decoding the request. BODY is the code to handle the request, optionally with one or more declaration forms prepended. Within BODY the symbols CLIENT and REQUEST are bound to the client-state structure and an (unsigned-byte 8) array representing the current request respectively, and the symbols naming each field are bound to their respective field values. Purpose: Used to define X Request handling functions without having to write out the code to parse the wire protocol. A field description is a list of the form (name type ), where name is the name of the field, type is the type of the field (typically one of :u8, :s8, :u16, :s16, :u32, :s32, :atom, :window, :pixmap, :drawable, :atom, etc.), and is a set of key/value pairs defining things like required values, valid ranges, protocol error types, etc. Thus far, the field options appear to be :error, which specifies a non-default protocol error to be raised when the slot fails validation; :value, which requires a slot to have a particular value in order to validate; and :typecheck, which specifies a type specifier to use in a typep test for validation. ____________________________________________________________ request-handler Function opcode Arguments: OPCODE is the (unsigned-byte 8) opcode number of the request. Values: The function which handles the request with opcode number OPCODE. Purpose: Accessor function. ____________________________________________________________ (setf request-handler) Function handler opcode Arguments: HANDLER is a function to handle the request with opcode number OPCODE. OPCODE is the (unsigned-byte 8) opcode number of the request. Values: Returns HANDLER. Purpose: Accessor function. Resource Database Resource Identifiers are represented by (unsigned-byte 29) values (the protocol spec says "a 32-bit integer with the top three bits clear"). On connection, each client is assigned a range of resource IDs to use, in the form of a base ID value and a mask indicating which bits it can change to create its own IDs. The minimal mask size allowed is 18 bits, which means that we can have (1- (expt 2 11)) => 2047 Resource ID ranges to assign (the value of 0 is used all over the place, so we would not wish to assign a range which includes it, and we require a range for the server anyway to hold the screen root windows if nothing else). A range of resource IDs is assigned when a client connects successfully and is available for reassignment when the client has disconnected and the last resource in the range is destroyed (if a client destroys all of its resources, it can still create more in its range; if a client sets its close-down-mode before disconnecting, its resources may not be destroyed immediately). Notionally, we keep a table of assigned resource ranges. This table keeps track of two bits of information for each range. First, the client to which this range has been assigned (or NIL if it hasn't been assigned or the client has disconnected). And second, if there are any resources allocated in this range. Just for fun, the spec appears to say that KillClient on a resource belonging to a connected client that has set a closdown mode other than Destroy does not destroy any resources, while KillClient on a resource belonging to a disconnected client destroys all the resources formerly belonging to that client. ____________________________________________________________ reset-resource-database Function Arguments: None. Values: Returns a primary value of NIL. Purpose: This function is called to reset the Resource Database to its initial state. That is, it destroys any and all existing resources and clears the table of assigned ranges. ____________________________________________________________ assign-resource-range Function client Arguments: CLIENT is a client-state instance to associate with the resource range. Values: Returns T if a resource range was assigned, returns NIL if there are no more unassigned resource ranges. Purpose: Assigns a resource range to a newly-connected client. Sets the client-state-resource-base and client-state-resource-mask on the client, and associates the client with said resource range. ____________________________________________________________ x-resource Function resource-id Arguments: RESOURCE-ID is an X Resource ID (unsigned-byte 29). Values: Returns the object associated with the Resource ID RESOURCE-ID or NIL if there is no such object. Purpose: Accessor function. ____________________________________________________________ (setf x-resource) Function object resource-id Arguments: OBJECT is the object to associate with the Resource ID RESOURCE-ID. RESOURCE-ID is an X Resource ID (unsigned-byte 29) to which OBJECT should be associated. Values: Returns OBJECT. Purpose: Accessor function. ____________________________________________________________ destroy-resources-for-range Function resource-id Arguments: RESOURCE-ID is an X Resource ID (unsigned-byte 29), not necessarily representing an actual resource, within the range to destroy. Values: Returns a primary value of NIL. Purpose: Used to destroy all of the resources associated with a client, either during disconnect processing or for handling KillClient on the resources formerly associated with a client that disconnected with a closedown mode other than Destroy. ____________________________________________________________ resource-client Function resource-id Arguments: RESOURCE-ID is an X Resource ID (unsigned-byte 29), not necessarily representing an actual resource, within the range for the client. Values: Returns the client-state instance associated with the range containing resource-id or NIL if the range is not associated with a client. Purpose: Used to find the client associated with a resource, primarily to support KillClient requests, although other uses may occur. Resources An X Resource is information maintained by the server on behalf of a client, either a Colormap, Cursor, Font, Graphics Context, Pixmap, or Window. There are also two aggregate resource types, Drawable and Fontable, members of which belong to one of a set of resource types. Because X Resources are all maintained within the same namespace, we need a common protocol to deal with them. The class inheritance tree for resources is as follows: +----------+ | Resource | +----------+ | +----------+ +------| Drawable | | +----------+ | | +--------+ | +------| Window | | | +--------+ | | | | +--------+ | \------| Pixmap | | +--------+ | +--------+ +------| Cursor | | +--------+ | | +----------+ +------| Fontable | | +----------+ | | +------+ | +------| Font | | | +------+ | | | | +----------+ | \------| GContext | | +----------+ | +----------+ \------| Colormap | +----------+ At the present time, the expectation is that differentiation between different resource object types will be done using typep or a similar mechanism. ____________________________________________________________ resource Protocol Class The basic class representing an X Resource. This class is intended to be an abstract type, not to be instantiated directly. All X Resource types should inherit from RESOURCE. ____________________________________________________________ drawable-resource Protocol Class The class representing an X Resource that is considered to be a 'Drawable'. This class is intended to be an abstract type, not to be instantiated directly. The Window and Pixmap types should inherit from DRAWABLE-RESOURCE. ____________________________________________________________ fontable-resource Protocol Class The class representing an X Resource that is considered to be a 'Fontable'. This class is intended to be an abstract type, not to be instantiated directly. The Font and GContext types should inherit from FONTABLE-RESOURCE. ____________________________________________________________ resource-client Generic Function resource Arguments: RESOURCE is an instance of the resource class. Values: Returns the client which created RESOURCE, or NIL if the client has since disconnected. Purpose: Accessor function. ____________________________________________________________ resource-id Generic Function resource Arguments: RESOURCE is an instance of the resource class. Values: Returns the Resource ID to which RESOURCE is associated. Purpose: Accessor function. Atom Database The Atom Database is initialized to a known state on server reset, and provides protocol requests InternAtom and GetAtomName. ____________________________________________________________ reset-atom-database Function Arguments: None. Values: Returns a primary value of NIL. Purpose: This function is called to reset the Atom Database to its initial state. That is, it clears the database and then loads the initial 68 atoms required by the X Protocol specification. ____________________________________________________________ intern-atom Function name only-if-exists Arguments: NAME is the name of the atom to intern. ONLY-IF-EXISTS indicates, if true, that a new atom should not be created if an atom named NAME is not found. Values: Returns a 32-bit unsigned integer representing the atom with the name NAME or 0 (None) if ONLY-IF-EXISTS is true and such an atom does not exist. Purpose: This function is called to find a given atom, or to create an atom with a given name if there isn't one already. Exceptional Situations: Value, Alloc. ____________________________________________________________ atom-name Function atom Arguments: ATOM is a 32-bit unsigned integer representing an atom. Values: Returns the name of the atom as a string. The consequences are undefined if this string is ever modified. Purpose: This function is called to find the name of a given atom, presumably for output to the user. Exceptional Situations: Atom. ____________________________________________________________ assert-atom-valid Function atom Arguments: ATOM is a 32-bit unsigned integer representing an atom. Values: Returns a primary value of NIL. Purpose: This function is called to verify that a given atom is within the range of currently-allocated atoms (that is, (>= 1 atom max-atoms), where max-atoms is the total number of interned atoms). Exceptional Situations: Atom. EOF