1. Dr. Yann-Hang Lee yhlee@asu.edu (480) 727-7507
Quark, PCI, and GPIO
Computer Science & Engineering Department Arizona State University Tempe, AZ 85287
Dr. Yann-Hang Lee (480)

2. Current Processor Design
Moore’s law continues to hold true, transistor counts doubling every 18 months
But can no longer rely upon increasing clock rates and instruction-level parallelism to meet computing performance demands
Semiconductor device fabrication process
65 nm – 2006, 45 nm – 2008, 32 nm – 2010, and 22 nm – 2012
How to best exploit ever-increasing on-chip transistor counts?
Multi- & many-core (MC) devices are new technology wave
exploiting explicit parallelism in the new devices
Size and Power constraints

3. Intel Processors X86 32/64 architecture Processors for Differences
486 – first pipelined x86 design
Pentium – the first x86 superscalar CPU
Processors for
Server (Xeon), desktop (Core i3/i5/i7), mobile (Core i3/i5/i7), and embedded (Atom)
All of them support hypervisor (VM)
Differences
CPUs, memory, and interconnection bandwidth
reliability (quality of dies) and form factor
power and thermal requirements
Uses available clock cycles and power, not to push up higher clock speeds and energy needs

4. Typical x86 System Architecture
Processor
Host Bus (PSB) 100/133/200MHz 64-bit
HubLink Bus
PCI Bus 33 MHz 32-bit
AGP Bus
System Memory
Audio
USB
LAN
IDE
Keybrd
Mouse
Floppy
Serial
Parallel
Clock Gen
Host Clock
PCI Clock
USB Clock
Hublink Clock
LPC Bus
SM Bus
CNR
SIO
South Bridge (ICH)
North Bridge (MCH)
FWH
Chipset
North Bridge
South Bridge
Firmware Hub
Various chipsets available from Intel to meet performance requirements
FSB, DMI/Hub interface
System control hub (SCH) – GMCH and ICH are merged into one chip

5. Intel Quark SoC X1000 SOC – Chip size, power and pins CPU core (x86)
cache, internal memory (flash, SRAM)
IO interfaces and external buses
interconnection or switches
misc (clock, JTAG)
Chip size, power and pins
32nm process in 1st Quark
one-fifth the size and
one-tenth the power of
low-end Atom chip
393 solder balls on 15mm2
5 power rails (3.3V, 1.8V,
1.5V, 1.05V, 1.0V)

6. Quark Core Internal Architecture
32-bit RISC integer core
Single cycle execution
Instruction pipelining
Floating-point unit
Cache with cache consistency support (16-Kbyte for both data and instructions)
Memory management unit

7. 486 Pipeline

8. Host Bridge in Quark
A central hub that routes transactions to and from Quark CPU core, DRAM controller, and other functional blocks.
CPU core  PCI devices
via MMIO and IO accesses

9. Galileo Gen 2 Board 400MHz Quark SoC 256MB DDR3 Ethernet USB Host Port
MicroSD Support
I2C, SPI Support
Mini PCI Express
Serial Connectivity
GPIO
Linux on Board
Source:

10. PCI Bus Operation Bus transaction :
bus masters issue requests  arbitration  bus grant
issues address and command and begins a cycle frame (transaction)
memory, I/O, configuration read/write commands
a target is selected (device select)
it is ready to complete the data transfer phase
PCI allows the use of up to 16 different 4-bit commands
Configuration commands
Memory commands
I/O commands
Special-purpose commands
A command is presented on the C/BE# bus by the initiator during an address phase (the first assertion of FRAME#)
C/BE[3::0]#
Command Type
0000
Interrupt Acknowledge
0001
Special Cycle
0010
I/O Read
0011
I/O Write
0100
Reserved
0101
0110
Memory Read
0111
Memory Write
1000
1001
1010
Configuration Read
1011
Configuration Write
1100
Memory Read Multiple
1101
Dual Address Cycle
1110
Memory Read Line
1111
Memory Write and Invalidate

11. PCI Address Space
A PCI target can implement up to three different types of address spaces
Configuration space
Stores basic information about the device
Allows the central resource or O/S to program a device with operational settings
I/O space
– Used mainly with PC peripherals and not much else
Memory space
– Used for just about everything else
Message bus space
message bus space is through the SoC’s PCI configuration registers

12. Accessing the Address Spaces
accessed using a large variety of processor
instructions (mov, add, or, shr, push, etc.)
and virtual-to-physical address-translation
memory
space
(4GB)
accessed only by using the processor’s
special ‘in’ and ‘out’ instructions
(without any translation of port-addresses)
PCI
configuration
space
(16MB)
i/o space
(64KB)
i/o-ports 0x0CF8-0x0CFF dedicated to accessing PCI Configuration Space

13. PCI Configuration Address Space
Contains 256 bytes of basic device information,
addressable by 8-bit PCI bus, 5-bit device, and 3-bit function numbers for the device
the first 64 bytes (00h – 3Fh) make up the standard configuration header, including PCI ID, i.e. vendor ID and device ID registers, to identify the device
the remaining 192 bytes (40h – FFh) represent user-definable configuration space, such as the information specific to a PC card for use by its accompanying software driver
Also permits Plug-N-Play
base address registers allow an agent to be mapped dynamically into memory or I/O space
a programmable interrupt-line setting allows a software driver to program a PC card with an IRQ upon power-up

14. Configuration Transactions
Are generated by a host or PCI-to-PCI bridge
Use a set of IDSEL signals as chip selects
Dedicated address decoding
Each agent is given a unique IDSEL signal
Are typically single data phase
Bursting is allowed, but is very rarely used
Two types (specified via AD[1:0] in addr. phase)
Type 0: Configures agents on same bus segment
Type 1: Configures across PCI-to-PCI bridges

15. Type 00h Configuration Space Header

16. Configuration Commands
Two DWORD I/O locations are used to generate configuration transactions
0CF8h references a read/write register, CONFIG_ADDRESS.
0CFCh references a read/write register, CONFIG_DATA.
Bus enumeration
attempting to read the Vendor- and Device ID register for each combination of bus number and device number, at the device's function #0
knows a device exists, and can then program the memory mapped and I/O port addresses for the device.

17. Example Quark GIP Configuration
lspci –s 00:15.2 –vvvxxx
00: c
10:
20:
30: ff
40:
50:
60:
70:
80: 01 a
90:
a0: c 10 e0 fe d
b0:
c0: c
d0:
e0:
f0: b1 0f

18. Accessing PCI Space in Linux
linux/arch/x86/pci/early.c
u32 read_pci_config(u8 bus, u8 slot, u8 func, u8 offset) ….
outl(0x | (bus<<16) | (slot<<11) | (func<<8) | offset, 0xcf8);
v = inl(0xcfc);
linux/arch/x86/pci/direct.c
static int pci_conf1_read (int seg, int bus, int devfn, int reg, int len, u32 *value) ……
spin_lock_irqsave(&pci_config_lock, flags);
outl(PCI_CONF1_ADDRESS(bus, devfn, reg), 0xCF8);
switch (len) {
case 1:
*value = inb(0xCFC + (reg & 3));
break;
…….
spin_unlock_irqrestore(&pci_config_lock, flags);
…..

19. Example: I2C and GPIO in Quark
A PCI device: B:0, D:21, F:2
MMIO –
use two base registers in configuration
registers
I2C memory registers – BAR0+offset
I2C Master mode operation
Disable the I2C controller by writing 0 to DW_IC_ENABLE.ENABLE.
Write to the DW_IC_CON register
Write to the DW_IC_TAR register.
Enable the I2C controller by writing a 1 in DW_IC_ENABLE.
Write the transfer direction and data to be sent to the DW_IC_DATA_CMD register.
(/drivers/i2c/busses/i2c-designware-core.c)
Offset Start
Offset End
Register ID
Default Value
10h
13h
BAR0 h
14h
17h
BAR1

20. Example: Quark GIP GPIO Set
In intel_qrk_gip_gpio.c
static void __iomem *reg_base; /* base register of GPIO registers
allocated dynamically */
void __iomem *reg_data = reg_base + PORTA_DATA;
/* find port register address*/
Then,
spin_lock_irqsave(&lock, flags);
val_data = ioread32(reg_data);
if (value) // write 1 to the bit at offset
iowrite32(val_data | BIT(offset % 32), reg_data);
else
iowrite32(val_data & ~BIT(offset % 32), reg_data);
spin_unlock_irqrestore(&lock, flags);

21. Example: Quark GPIO IRQ Enable
Allows each bit of Port A to be configured for interrupts.
In drivers/mfd/intel_qrk_gip_gpio.c,
#define PORTA_INT_EN 0x30 /* Interrupt enable */
#define PORTA_INT_MASK 0x34 /* Interrupt mask */
#define PORTA_INT_TYPE_LEVEL 0x38 /* Interrupt level*/
static void intel_qrk_gpio_irq_enable(struct irq_data *d) {
void __iomem *reg_inte = reg_base + PORTA_INT_EN;
gpio = d->irq - irq_base;
spin_lock_irqsave(&lock, flags);
val_inte = ioread32(reg_inte);
iowrite32(val_inte | BIT(gpio % 32), reg_inte);
spin_unlock_irqrestore(&lock, flags); }

22. Linux Device Initialization
Devices that can or cannot be probed (discovered)
PNP devices on PCI bus and USB bus
An IO expander attached on an I2C bus
If we recognize an available device, what should be done:
find a matching driver
allocate resources for the device, including memory-mapped IO and interrupt.
run the device initialization in the driver
to reset the hardware and to register the device
to enable an access path such that the device functions can be invoked.
To give the definitions of available devices
device tree or hard-coded platform device definition
in /linux/drivers/platform/x86/quark/intel_qrk_plat_galileo_gen2.c
static struct pca953x_platform_data pcal9555a_platform_data_exp0 = {
.gpio_base = PCAL9555A_GPIO_BASE_OFFSET,
.irq_base = -1, };

23. Linux Initialization Order of Built-in Modules
Linux startup sequence of built-in modules
a mechanism to add a function to kernel initialization and to add some amount of order to those calls
In Linux/init/main.c
entry point – start_kernel() to to finish kernel initialization
process (mm_init, sched_init, time_init, ... tec.) and launch
the first init process
In rest_init() – create a kernel thread to run “kernel_init”
In kernel_init –
do_basic_setup()  do_initcalls()
 for each level, do_initcall_level(level)  do_one_initcall(*fn);
in /include/linux/init.h, the mechanism of initcall is defined,
module_init(my_finit_unc)  __define_initcall(my_finit_unc, 6)
pointer to my_init_func is added to .initcall6.init section (array of function pointers)
early_initcall
0. pure_initcall
core_initcall
postcore_initcall
arch_initcall
subsys_initcall
fs_initcall
rootfs_initcall
device_initcall
late_initcall
#define __define_initcall(fn, id) \
static initcall_t __initcall_##fn##id __used \
__attribute__((__section__(".initcall" #id ".init"))) = fn

24. PCI Device Initialization In Linux
In Linux/arch/x86/pci/init.c
arch_initcall(pci_arch_init) – pci_arch_init is a function to be called in arch_init phase
Linux/arch/x86/pci/legacy.c-- PCI bus initialization code
subsys_initcall defines a driver initialization entry point.
scan all of the PCI buses in the system looking for all PCI devices in the system (including PCI-PCI bridge devices).
If the PCI slot is occupied, builds a pci_dev data structure describing the device and links into the list of known PCI devices (pointed at by pci_devices).
In pci_dev – vendor and device id, pointers to driver and resource (start, end,…), irq, .., etc
PCI device initialization –
after identifying a device, where is its driver -- probe call to device driver (intel-qrk-gip-gpio.c)
libpci.a -- a library that allows user applications to access the PCI subsystem. (built from pciutils package)

25. PCI Device Initialization Path
x86_init.pci.init()  pci_legacy_init()
pcibios_scan_root(0)  pci_scan_bus_on_node()
 pci_scan_root_bus()  pci_create_root_bus()
 pci_scan_child_bus() {
pci_scan_slot()  pci_scan_device()  pci_device_add()
// pci_dev is created after probing pci bus
 device_add()  kobject_add()  device_add_file() // sysfs stuffs
 bus_probe_device() // probe drivers for a new device
 device_attach() …. }
 pci_bus_add_devices { // deal with bus devices
pci_create_sysfs_dev_files() // sysfs stuffs
 device_attach()  bus_for_each_drv( … __device_attach)
 driver_probe_device // call driver’s probe function

26. with I2C addresses 0x25, 0x26, and 0x27
GPIOs on Galileo Gen 2 (1)
Quark CPU core
PCAL9535
Quark GIP
controller
Quark legacy
GPIO
GPIO_SUS[5:0]
GPIO[9:0]
3 IO expanders (16 GPIOs)
with I2C addresses 0x25, 0x26, and 0x27
GPIO port
PCA9685
PWM/LED driver
(16 channels)
I2C bus
Quark processor

27. GPIOs on Galileo Gen 2 (2)
From Linux’s point of view, all GPIOs can be programmed (direction, value, and interrupt)
Questions:
how does Linux know which GPIO devices are available?
how is this numbering system defined and mapped to physical GPIO pins?
Linux GPIO #
Device
0-1
qrk_gip_gpio(8,9)
2-7
sch_gpio (0-5)
8-15
qrk_gip_gpio(0-7)
16-31
pcal9535 (0-15)
i2c address 0x25
32-47
i2c address 0x26
48-63
i2c address 0x27
64-79
pca 9685-gpio
multiplexed with pwm

28. Arduino Shield Pins on Galileo Gen 2
IO signals can come from
Quark
IO expanders
PWM/LED driver
ADC
To reach shield pins, they have to pass
multiplexer(s)
direction control
pull-up (down) resistor
Signal routing is
controlled by GPIOs

29. Galileo Gen 2 IO Multiplexing -- Example
SCL (pin 10 of J2B1 on arduino shield)
selected by AMUX1_IN
AMUX1_IN = GPIO 60

30. GPIO interface in Linux
From programmer’s view
GPIO pin numbering
enable GPIO (pin multiplexing)
set direction
read/write data
interrupt
There may be various physical connections
GPIO pins in IO expender
GPIO ports
GPIO devices on PCI bus
various GPIO controllers
GPIO driver (interface to GPIO chips) : /driver/gpio/gpiolib.c
Drivers for gpio chips
For Galileo Gen2 board:
gpio_sch.c (Poulsbo SCH) + intel_qrk_gip_gpio.c + gpio-pca953x.c

31. Linux GPIO Driver
A GPIO (General Purpose Input/Output) pin can be configured, set up its direction, and value if it is an output pin
A SoC chip may have several GPIO components
Multiple “gpio chips”
A global number in the integrated GPIO namespace, i.e., 0, 1, 2,…,n
sysfs interface to user space
GPIO framework
(gpiolib.c)
IO expanders
Quark GIP
controller
Quark legacy
GPIO
GPIO_SUS[5:0]
GPIO[9:0]
GPIO[….]
chip
drivers

32. GPIO Sysfs Interface Example of shell commands for GPIO:
echo -n $led > /sys/class/gpio/export
echo -n $DIR > /sys/class/gpio/gpio$led/direction
echo -n $ON > /sys/class/gpio/gpio$led/value
Sysfs interface -- a virtual file system
to export kernel data structures, their attributes, and the linkages between them from kernel to user space.
two methods for read and write operations
struct device_attribute {
struct attribute attr;
ssize_t (*show)(struct device *dev, ….);
ssize_t (*store)(struct device *dev, ….); };
So, in gpiolib.c, the following functions are provided
export_store(), gpio_value_store(), gpio_direction_show(), ….

33. GPIO Programming GPIO Programming Poll and GPIO interrupt
In user space – via sysfs
int fd = open("/sys/class/gpio/export", O_WRONLY); sprintf(buffer, "%d", gpio); // int gpio
write(fd, buffer, strlen(buffer));
close(fd);
In kernel space – use the exported interface of gpiolib.c
gpio_request_one(GP_3, GPIOF_OUT_INIT_LOW, "led1");
__gpio_set_value(GP_3, 0);
Poll and GPIO interrupt
configure GPIO pin as an interrupt source via  /sys/class/gpio/gpioX/edge
poll() system call to block on the gpio value until an interrupt occurs. When the content changes, poll will return POLLERR|POLLPRI.
Example code -- gpio-int-test.c

34. Programming GPIO on Galileo Gen 2
gpiochips are numbered with the starting gpio no
Programming steps:
export a gpio pin
the files are read to access gpio attributes
use file descriptors to
set pin direction
read/write value
enable edge
poll
Example: IO multiplexing
export
direction = out
value to select IOs

35. Programming Examples
A program to control the led attached at shield pin IO10.
Need 3 gpio: gpio10, gpio26 (direction), and gpio74 (mux)
A simple library to control gpio pins
gpio_func.c and gpio_func.h
mraa library (libmraa)
a C/C++ library to interface with the I/O on Galileo, Edison & other platforms, with a structured API where port names/numbering matches the board
mraa_gpio_context gpio;
gpio = mraa_gpio_init(6); // IO6
mraa_gpio_dir(gpio, MRAA_GPIO_IN);
for (;;) {
fprintf(stdout, "Gpio is %d\n", mraa_gpio_read(gpio));
sleep(1); }
mraa_gpio_close(gpio);

36. PWM – Pulse Width Modulation
Duty cycle =
(pulse width)/(pulse period)
50% -- square wave
The average power produced by the pulse signal to control
dimming of RGB LED
DC motor speed
for each channel of PCA9685
ON and OFF registers
counter (12 bits – 0 to 4095)
(source:

37. Linux PWM Driver pwmlib (core.c) and pwm chip drivers
sysfs interface to the generic PWM framework for user space access
Each probed PWM controller/chip will be exported as pwmchipN
/sys/class/pwm/
`-- pwmchipN/ for each PWM chip
|-- export (w/o) ask the kernel to export a PWM channel
|-- npwm (r/o) number of PWM channels in this PWM chip
|-- pwmX/ for each exported PWM channel
| |-- duty (r/w) duty cycle (in nanoseconds)
| |-- enable (r/w) enable/disable PWM
| |-- period (r/w) period (in nanoseconds)
| `-- polarity (r/w) polarity of PWM (normal/inversed)
`-- unexport (w/o) return a PWM channel to the kernel

38. PWM Channels on Galileo Gen 2
PCA channel, 12-bit PWM Fm+ I2C-bus LED controller
pins can be pwm or gpio(output) –pwmchip0/export or gpio/export
all 16 channel use a single period counter (i.e. pwm3’s period cannot be modified individually)
set up “period” per device at pwmchip0/device/pwm_period

39. PWM Example Code Steps: which pwm channel IO multiplexing export
int pwmGetFdPeriod(unsigned int pwm) { int fd; char path[256]; #ifdef GEN2 fd = open("/sys/class/pwm/pwmchip0/device\ /pwm_period", O_WRONLY); #else sprintf(path, PWM_PERIOD, pwm); fd= open(path, O_WRONLY); #endif return fd; } /***********/ FdPeriod = pwmGetFdPeriod(1); // pwm 1 period = /freq; // nano seconds sprintf(TString,"%d",(period)); write(FdPeriod,TString,strlen(TString)));
Steps:
which pwm channel
IO multiplexing
export
set “period”
set “duty cycle”
enable pwm
disable pwm