USB Speed and Version Background

USB devices are often referred to in two different ways. The first way is the USB version that they conform to. In the wild this looks like USB 1.1, USB 2.0, USB 3.0, etc.. However, devices are also referred to as ‘full-’, ‘low-’, ‘high-’, ‘super-’ speed devices. The latter description describes the maximum theoretical speed of a given device. For example, a super-speed device theoretically caps out around 5 Gbit/s, whereas a low-speed device caps out at 1.5 Mbit/s. In general, each speed usually corresponds to a specific USB protocol generation. For example, all USB 3.0 devices are super-speed devices. All 'high-speed' devices are USB 2.x devices. Full-speed devices are special in that they can either be USB 1.x or USB 2.x devices. Low-speed devices are only a USB 1.x thing, they did not jump the fire line to USB 2.x. USB 3.0 devices and ports generally have the wiring for both USB 2.0 and USB 3.0. When a USB 3.0 device is plugged into a USB 2.0 port or hub, then it will report its version as USB 2.1, to indicate that it is actually a USB 3.0 device.

USB Endpoint Background

To understand the organization of the functions that make up the hcdi operations vector, it helps to understand how USB devices are organized and work at a high level. A given USB device is made up of endpoints. A request, or transfer, is made to a specific USB endpoint. These endpoints can provide different services and have different expectations around the size of the data that'll be used in a given request and the periodicity of requests. Endpoints themselves are either used to make one-shot requests, for example, making requests to a mass storage device for a given sector, or for making periodic requests where you end up polling on the endpoint, for example, polling on a USB keyboard for keystrokes. Each endpoint encodes two different pieces of information: a direction and a type. There are two different directions: IN and OUT. These refer to the general direction that data moves relative to the operating system. For example, an IN transfer transfers data in to the operating system, from the device. An OUT transfer transfers data from the operating system, out to the device. There are four different kinds of endpoints:

BULK

These transfers are large transfers of data to or from a device. The most common use for bulk transfers is for mass storage devices. Though they are often also used by network devices and more. Bulk endpoints do not have an explicit time component to them. They are always used for one-shot transfers.

CONTROL

These transfers are used to manipulate devices themselves and are used for USB protocol level operations (whether device-specific, class-specific, or generic across all of USB). Unlike other transfers, control transfers are always bi-directional and use different kinds of transfers.

INTERRUPT

Interrupt transfers are used for small transfers that happen infrequently, but need reasonable latency. A good example of interrupt transfers is to receive input from a USB keyboard. Interrupt-IN transfers are generally polled. Meaning that a client (device driver) opens up an interrupt-IN endpoint to poll on it, and receives periodic updates whenever there is information available. However, Interrupt transfers can be used as one-shot transfers both going IN and OUT.

ISOCHRONOUS

These transfers are things that happen once per time-interval at a very regular rate. A good example of these transfers are for audio and video. A device may describe an interval as 10ms at which point it will read or write the next batch of data every 10ms and transform it for the user. There are no one-shot Isochronous-IN transfers. There are one-shot Isochronous-OUT transfers, but these are used by device drivers to always provide the system with sufficient data.

To find out information about the endpoints, USB devices have a series of descriptors that cover different aspects of the device. For example, there are endpoint descriptors which cover the properties of endpoints such as the maximum packet size or polling interval. Descriptors exist at all levels of USB. For example, there are general descriptors for every device. The USB device descriptor is described in usb_dev_descr(9S). Host controllers will look at these descriptors to ensure that they program the device correctly; however, they are more often used by client device drivers. There are also descriptors that exist at a class level. For example, the hub class has a class-specific descriptor which describes properties of the hub. That information is requested for and used by the hub driver. All of the different descriptors are gathered by the system and placed into a tree, with device descriptors, configurations, endpoints, and more. Client device drivers gain access to this tree and then use them to then open endpoints, which are called pipes in USBA (and some revisions of the USB specification). Each pipe gives access to a specific endpoint on the device which can be used to perform transfers of a specific type and direction. For example, a mass storage device often has three different endpoints, the default control endpoint (which every device has), a Bulk-IN endpoint, and a Bulk-OUT endpoint. The device driver ends up with three open pipes. One to the default control endpoint to configure the device, and then the other two are used to perform I/O. These routines translate more or less directly into calls to a host controller driver. A request to open a pipe takes an endpoint descriptor that describes the properties of the pipe, and the host controller driver goes through and does any work necessary to allow the client device driver to access it. Once the pipe is open, it either makes one-shot transfers specific to the transfer type or it starts performing a periodic poll of an endpoint. All of these different actions translate into requests to the host controller. The host controller driver itself is in charge of making sure that all of the required resources for polling are allocated with a request and then proceed to give the driver's periodic callbacks. For each of the different operations described above, there is a corresponding entry in usba_hcdi_ops(9S). For example, open an endpoint, the host controller has to implement usba_hcdi_pipe_open(9E) and for each transfer type, there is a different transfer function. One example is usba_hcdi_pipe_bulk_xfer(9E). See usba_hcdi_ops(9S) for a full list of the different function endpoints.

HCDI Initialization

hcdi drivers are traditional character device drivers. To start with, an hcdi driver should define traditional dev_ops(9S) and cb_ops(9S) structures. To get started, the device driver should perform normal device initialization in an attach(9E) entry point. For example, PCI devices should setup the device's registers and program them. In addition, all devices should configure interrupts, before getting ready to call into the USBA. Each instance of a device must be initialized and registered with the USBA. To initialize a device driver with the USBA, it must first call usba_alloc_hcdi_ops(9F). This provides a device driver with the usba_hcdi_ops(9S) structure that it must fill out. Please see usba_hcdi_ops(9S) for instructions on how it should be filled out. Once filled out, the driver should call usba_hcdi_register(9F). If the call to register fails for whatever reason, the device driver should fail its attach(9E) entry point. After this call successfully completes, the driver should assume that any of the functions it registered with the call to usba_hcdi_register(9F) will be called at this point.

Binding the Root Hub

Once this is set up, the hcdi driver must initialize its root hub by calling usba_hubdi_bind_root_hub(9F). To bind the root hub, the device driver is responsible for providing a device descriptor that represents the hardware. Depending on the hardware, this descriptor may be either static or dynamic. This device descriptor should be a packed descriptor that is the same that would be read off of the device. The device descriptor should match a hub of a USB generation equivalent to the maximum speed of the device. For example, a USB 3.0 host controller would use a USB 3.0 hub's device descriptor. Similarly, a USB 2.0 host controller would use a USB 2.0 hub's device descriptor. The descriptor first starts with a USB configuration descriptor, as defined in usb_cfg_descr(9S). It is then followed by an interface descriptor. The definition for it can be found in usb_if_descr(9S). Next is the endpoint descriptor for the single Interrupt-IN endpoint that all hubs have as defined in usb_ep_descr(9S). Finally, any required companion descriptors should be used. For example, a USB 3.x hub will have a usb_ep_ss_comp_descr(9S) appended to the structure. Note, that the structure needs to be packed, as though it were read from a device. The structures types referenced in usb_cfg_descr(9S), usb_if_descr(9S), usb_ep_descr(9S), and usb_ep_ss_comp_descr(9S) are not packed for this purpose. They should not be used as they have gaps added by the compiler for alignment. Once assembled, the device driver should call usba_hubdi_bind_root_hub(9F). This will cause an instance of the hubd(7D) driver to be attached and associated with the root controller. As such, driver writers need to ensure that all initialization is done prior to loading the root hub. Once successfully loaded, driver writers should assume that they'll get other calls into the driver's operation vector before the call to usba_hubdi_bind_root_hub(9F). If the call to usba_hubdi_bind_root_hub(9F) failed for whatever reason, the driver should unregister from USBA (see the next section), unwind all of the resources it has allocated, and return DDI_FAILURE. Otherwise, at this point it's safe to assume that the instance of the device has initialized successfully and the driver should return DDI_SUCCESS.

Driver Teardown

When a driver's detach(9E) entry point has been called, before anything else is done, the device driver should unbind its instance of the root hub and then unregister from the USBA. To unbind the root hub, the instance of the driver should call usba_hubdi_unbind_root_hub(9F). If for some reason that function does not return USB_SUCCESS, then the device driver should fail the call to detach(9E) and return DDI_FAILURE. Once the root hub has been unbound, the device driver can continue by removing its hcdi registration with USBA. To do this, the driver should call usba_hcdi_unregister(9F). As this call always succeeds, at this point, it is safe for the driver to tear down all the rest of its resources and successfully detach.

State Tracking and Minor Numbers

Because a host controller driver is also a root hub, there are a few constraints around how the device must store its per-instance state and how its minor numbers are used. hcdi drivers must not store any data with ddi_get_driver_private(9F). This private data is used by USBA. If it has been called before the device registers, then it will fail to register successfully with the USBA. However, setting it after that point will corrupt the state of the USBA and likely lead to data corruption and crashes. Similarly, part of the minor number space is utilized to represent various devices like the root hub. Whenever a device driver is presented with a dev_t and it's trying to extract the minor number, it must take into account the constant HUBD_IS_ROOT_HUB. The following shows how to perform this, given a dev_t called dev:

minor_t minor = getminor(dev) & ~HUBD_IS_ROOT_HUB;

Required Character and Device Operations

The USBA handles many character and device operations entry points for a device driver or has strict rules on what a device driver must do in them. This section summarizes those constraints. In the dev_ops(9S) structure, the following members have special significance:

devo_bus_ops

The devo_bus_ops member should be set to the symbol usba_hubdi_busops. See usba_hubdi_dev_ops(9F) for more information.

devo_power

The devo_power member should be set to the symbol usba_hubdi_root_hub_power. See usba_hubdi_dev_ops(9F) for more information.

The other standard entry points for character devices, devo_getinfo, devo_attach, and devo_detach should be implemented normally as per getinfo(9E), attach(9E), and detach(9E) respectively. The following members of the cb_ops(9S) operations vector must be implemented and set:

cb_open

The device driver should implement an open(9E) entry point that obtains access to its dev_info_t and then calls usba_hubdi_open(9F). See usba_hcdi_cb_open(9E) for more information.

cb_close

The device driver should implement a close(9E) entry point that obtains access to its dev_info_t and then calls usba_hubdi_close(9F).
See usba_hcdi_cb_close(9E) for more information.

cb_ioctl

The device driver should implement a ioctl(9E) entry point that obtains access to its dev_info_t and then calls usba_hubdi_ioctl(9F).

If the device driver wishes to have private ioctls, it may check the ioctl command before calling usba_hubdi_ioctl(9F). Because the usba_hubdi_ioctl(9F) function normally takes care of checking for the proper privileges, device drivers must verify that a caller has appropriate privileges before processing any private ioctls.

See usba_hcdi_cb_ioctl(9E) for more information.

cb_prop_op

The cb_prop_op member should be set to ddi_prop_op(9F).

cb_flag

The cb_flag member should be set to the bitwise-inclusive-OR of the D_MP flag and the D_HOTPLUG flag.

All other members of the cb_ops(9S) structure should not be implemented and set to the appropriate value, such as nodev(9F) or nochpoll(9F).

Locking

In general, the USBA calls into a device driver through one of the functions that it has register in the usba_hcdi_ops(9S) structure. However, in response to a data transfer, the device driver will need to call back into the USBA by calling usba_hcdi_cb(9F). A device driver must hold no locks across the call to usba_hcdi_cb(9F). Returning an I/O to the USBA, particularly an error, may result in another call back to one of the usba_hcdi_cb(9F) vectors. Outside of that constraint, the device driver should perform locking of its data structures. It should assume that many of its entry points will be called in parallel across the many devices that exist. There are certain occasions where a device driver may have to enter the p_mutex member of the usba_pipe_handle_data(9S) structure when duplicating isochronous or interrupt requests. The USBA should in general, not hold this lock across calls to the HCD driver, and in turn, the HCD driver should not hold this lock across any calls back to the USBA. As such, the HCD driver should make sure to incorporate the lock ordering of this mutex into its broader lock ordering and operational theory. Generally, the p_mutex mutex will be entered after any HCD-specific locks. The final recommendation is that due to the fact that the host controller driver provides services to a multitude of USB devices at once, it should strive not to hold its own internal locks while waiting for I/O to complete, such as an issued command. This is particularly true if the device driver uses coarse grained locking. If the device driver does not pay attention to these conditions, it can easily lead to service stalls.

Synchronous and Asynchronous Entry Points

The majority of the entry points that a host controller driver has to implement are synchronous. All actions that the entry point implies must be completed before the entry point returns. However, the various transfer routines: usba_hcdi_pipe_bulk_xfer(9E), usba_hcdi_pipe_ctrl_xfer(9E), usba_hcdi_pipe_intr_xfer(9E), and usba_hcdi_pipe_isoc_xfer(9E), are ultimately asynchronous entry points. Each of the above entry points begins one-shot or periodic I/O. When the driver returns USB_SUCCESS from one of those functions, it is expected that it will later call usba_hcdi_cb(9F) when the I/O completes, whether successful or not. It is the driver's responsibility to keep track of these outstanding transfers and time them out. For more information on timeouts, see the section Endpoint Timeouts. If for some reason, the driver fails to initialize the I/O transfer and indicates this by returning a value other than USB_SUCCESS from its entry point, then it must not call usba_hcdi_cb(9F) for that transfer.

Short Transfers

Not all USB transfers will always return the full amount of data requested in the transfer. Host controller drivers need to be ready for this and report it. Each request structure has an attribute to indicate whether or not short transfers are OK. If a short transfer is OK, then the driver should update the transfer length. Otherwise, it should instead return an error. See the individual entry point pages for more information.

Root Hub Management

As was mentioned earlier, every host controller is also a root hub. The USBA interfaces with the root hub no differently than any other hub. The USBA will open pipes and issue both control and periodic interrupt-IN transfers to the root hub. In the host controller driver's usba_hcdi_pipe_open(9E) entry point, it already has to look at the pipe handle it's been given to determine the attributes of the endpoint it's looking at. However, before it does that it needs to look at the USB address of the device the handle corresponds to. If the device address matches the macro ROOT_HUB_ADDR, then this is a time where the USBA is opening one of the root hub's endpoints. Because the root hub is generally not a real device, the driver will likely need to handle this in a different manner from traditional pipes. The device driver will want to check for the presence of the device's address with the following major entry points and change its behavior as described:

usba_hcdi_pipe_ctrl_xfer()

The device driver needs to intercept control transfers to the root hub and translate them into the appropriate form for the device. For example, the device driver may be asked to get a port's status. It should determine the appropriate way to perform this, such as reading a PCI memory-mapped register, and then create the appropriate response.

The device driver needs to implement all of the major hub specific request types. It is recommended that driver writers see what existing host controller drivers implement and what the hub driver currently requires to implement this.

Aside from the fact that the request is not being issued to a specific USB device, a request to the root hub follows the normal rules for a transfer and the device driver will need to call usba_hcdi_cb(9F) to indicate that it has finished.

usba_hcdi_pipe_bulk_xfer()

The root hub does not support bulk transfers. If for some reason one is requested on the root hub, the driver should return USB_NOT_SUPPORTED.

usba_hcdi_pipe_intr_xfer()

The root hub only supports periodic interrupt-IN transfers. If an interrupt-OUT transfer or an interrupt-IN transfer with the USB_ATTRS_ONE_XFER attribute is set, then the driver should return USB_NOT_SUPPORTED.

Otherwise, this represents a request to begin polling on the status endpoint for a hub. This is a periodic request, see the section Device Addressing Every USB device has an address assigned to it. The addresses assigned to each controller are independent. The root hub of a given controller always has an address of ROOT_HUB_ADDR.

In general, addresses are assigned by the USBA and stored in the usb_addr member of a usba_device_t(9S). However, some controllers, such as xHCI, require that they control the device addressing themselves to facilitate their functionality. In such a case, the USBA still assigns every device an address; however, the actual address on the bus will be different and assigned by the HCD driver. An HCD driver that needs to address devices itself must implement the usba_hcdi_device_address(9E) entry point. Endpoint Polling more on the semantics of polling and periodic requests.

Here, the device driver will need to provide data and perform a callback whenever the state of one of the ports changes on its virtual hub. Different drivers have different ways to perform this. For example, some hardware will provide an interrupt to indicate that a change has occurred. Other hardware does not, so this must be simulated.

The way that the status data responses must be laid out is based in the USB specification. Generally, there is one bit per port and the driver sets the bit for the corresponding port that has had a change.

usba_hcdi_pipe_isoc_xfer()

The root hub does not support isochronous transfers. If for some reason one is requested on the root hub, the driver should return

usba_hcdi_pipe_close()

When a pipe to the root hub is closed, the device driver should tear down whatever it created as part of opening the pipe. In addition, if the pipe was an interrupt-IN pipe, if it has not already had polling stop, it should stop the polling as part of closing the pipe.

usba_hcdi_pipe_stop_intr_polling()

When a request to stop interrupt polling comes in and it is directed towards the root hub, the device driver should cease delivering callbacks upon changes in port status being detected. However, it should continue keeping track of what changes have occurred for the next time that polling starts.

The primary request that was used to start polling should be returned, as with any other request to stop interrupt polling.

usba_hcdi_pipe_stop_isoc_polling()

The root hub does not support isochronous transfers. If for some reason it calls asking to stop polling on an isochronous transfer, the device driver should log an error and return USB_NOT_SUPPORTED.

Endpoint Polling

Both interrupt-IN and isochronous-IN endpoints are generally periodic or polled endpoints. interrupt-IN polling is indicated by the lack of the USB_ATTRS_ONE_XFER flag being set. All isochronous-IN transfer requests are requests for polling. Polling operates in a different fashion from traditional transfers. With a traditional transfer, a single request is made and a single callback is made for it, no more and no less. With a polling request, things are different. A single transfer request comes in; however, the driver needs to keep ensuring that transfers are being made within the polling bounds until a request to stop polling comes in or a fatal error is encountered. In many cases, as part of initializing the request, the driver will prepare several transfers such that there is always an active transfer, even if there is some additional latency in the system. This ensures that even if there is a momentary delay in the device driver processing a given transfer, I/O data will not be lost. The driver must not use the original request structure until it is ready to return due to a request to stop polling or an error. To obtain new interrupt and isochronous request structures, the driver should use the usba_hcdi_dup_intr_req(9F) and usba_hcdi_dup_isoc_req(9F) functions. These functions also allocate the resulting message blocks that data should be copied into. Note, it is possible that memory will not be available to duplicate such a request. In this case, the driver should use the original request to return an error and stop polling.

Request Memory and DMA

Each of the four transfer operations, usba_hcdi_pipe_ctrl_xfer(9E), usba_hcdi_pipe_bulk_xfer(9E), usba_hcdi_pipe_intr_xfer(9E), and usba_hcdi_pipe_isoc_xfer(9E) give data to hcdi drivers in the form of mblk(9S) structures. To perform the individual transfers, most systems devices will leverage DMA. Drivers should allocate memory suitable for DMA for each transfer that they need to perform and copy the data to and from the message blocks. Device drivers should not use desballoc(9F) to try and bind the memory used for DMA transfers to a message block nor should they bind the message block's read pointer to a DMA handle using ddi_dma_addr_bind_handle(9F). While this isn't a strict rule, the general framework does not assume that there are going to be outstanding message blocks that may be in use by the controller or belong to the controller outside of the boundaries of a given call to one of the transfer functions and its corresponding callback.

Endpoint Timeouts

The host controller is in charge of watching I/Os for timeouts. For any request that's not periodic (an interrupt-IN or isochronous-IN) transfer, the host controller must set up a timeout handler. If that timeout expires, it needs to stop the endpoint, remove that request, and return to the caller. The timeouts are specified in seconds in the request structures. For bulk timeouts, the request is in the bulk_timeout member of the usb_bulk_req(9S) structure. The interrupt and control transfers also have a similar member in their request structures, see usb_intr_req(9S) and usb_ctrl_req(9S). If any of the times is set to zero, the default USBA timeout should be used. In that case, drivers should set the value to the macro HCDI_DEFAULT_TIMEOUT, which is a time in seconds. Isochronous-OUT transfers do not have a timeout defined on their request structure, the usb_isoc_req(9S). Due to the periodic nature of even outbound requests, it is less likely that a timeout will occur; however, driver writers are encouraged to still set up the default timeout, HCDI_DEFAULT_TIMEOUT, on those transfers. The exact means of performing the timeout is best left to the driver writer as the way that hardware exposes scheduling of different endpoints will vary. One strategy to consider is to use the timeout(9F) function at a one second period while I/O is ongoing on a per-endpoint basis. Because the time is measured in seconds, a driver writer can decrement a counter for a given outstanding transfer once a second and then if it reaches zero, interject and stop the endpoint and clean up. This has the added benefit that when no I/O is scheduled, then there will be no timer activity, reducing overall system load.

Notable Types and Structures

The following are data structures and types that are used throughout host controller drivers:

usb_cfg_descr

The configuration descriptor. A device may have one or more configurations that it supports that can be switched between. The descriptor is documented in usb_cfg_descr(9S).

usb_dev_descr

The device descriptor. A device descriptor contains basic properties of the device such as the USB version, device and vendor information, and the maximum packet size. This will often be used when setting up a device for the first time. It is documented in usb_dev_descr(9S).

usb_ep_descr

The endpoint descriptor. An endpoint descriptor contains the basic properties of an endpoints such as its type and packet size. Every endpoint on a given USB device has an endpoint descriptor. It is documented in usb_ep_descr(9S).

usb_xep_descr

The extended endpoint descriptor. This structure is used to contain the endpoint descriptor, but also additional endpoint companion descriptors which are a part of newer USB standards. It is documented in usb_ep_xdescr(9S).

usb_bulk_req

This structure is filled out by client device drivers that want to make a bulk transfer request. Host controllers use this and act on it to perform bulk transfers to USB devices. The structure is documented in usb_bulk_req(9S).

usb_ctrl_req

This structure is filled out by client device drivers that want to make a control transfer request. Host controllers use this and act on it to perform bulk transfers to USB devices. The structure is documented in usb_ctrl_req(9S).

usb_intr_req

This structure is filled out by client device drivers that want to make an interrupt transfer request. Host controllers use this and act on it to perform bulk transfers to USB devices. The structure is documented in usb_intr_req(9S).

usb_isoc_req

This structure is filled out by client device drivers that want to make an isochronous transfer request. Host controllers use this and act on it to perform bulk transfers to USB devices. The structure is documented in usb_isoc_req(9S).

usb_flags_t

These define a set of flags that are used on certain entry points. These generally determine whether or not the entry points should block for memory allocation. Individual manual pages indicate the flags that drivers should consult.

usb_port_status_t

The usb_port_status_t determines the current negotiated speed of the device. The following are valid values that this may be:

USBA_LOW_SPEED_DEV

The device is running as a low speed device. This may be a USB 1.x or USB 2.0 device.

USBA_FULL_SPEED_DEV

The device is running as a full speed device. This may be a USB 1.x or USB 2.0 device.

USBA_HIGH_SPEED_DEV

The device is running as a high speed device. This is a USB 2.x device.

USBA_SUPER_SPEED_DEV

The device is running as a super speed device. This is a USB 3.0 device.

usb_cr_t

This is a set of codes that may be returned as a part of the call to usba_hcdi_cb(9F). The best place for the full set of these is currently in the source control headers.

Interrupts

While some hardware supports more than one interrupt queue, a single interrupt is generally sufficient for most host controllers. If the controller supports interrupt coalescing, then the driver should generally enable it and set it to a moderate rate.

driver.conf considerations

Due to the way host controller drivers need to interact with hotplug, drivers should generally set the ddi-forceattach property to one in their driver.conf(4) file.