PCI/VME Bit3 Development

General Description
Performance
Conclusions

Code Production Index


General Description

The Bit3167 system bridges the local PCI bus services to the VME bus. A further description of the cards can be found at http://www.bit3.com. Currently, SBS does not provide a Linux software package to develop user apps. However, party software can be found at Nikhef, and Ific. The ROD environment requires a special management of the two cards (VME/PCI) due to bandwidth requirements and software latencies. This reason forces us to build a custom software package. Our current design has been splitted in two stages: The Kernel Side (Linux2.2.x) and user side. For the present, interrupts are not required to manage the system.

The kernel module basically provides the virtual mapping of the three main Bit3 components, the CSR Space, the Descriptor Page Table (DPT) and the raw PCI memory space mapped to VME. All of them, are obtained using POSIX mmap calls to the bit3drv kernel module. The management and monitoring of these devices is completely carried out in user space using the library package libVMEBit3.a which merge Bit3lib.o and Bit3extras.o objects. Bit3lib is written in C, but it is also strongly related with object oriented programming concepts. The data types interface is also compliance with the VME-ROD software, therefore the API provides five VME basic types located at Bit3lib.h: All of them are internally managed by libVMEbit3.a, specially CSR and DPT devices. Each one of VME objects provide to user a virtual pointer to access on the virtual space granted.

VMESlave And DMA_dev are involved with the system RAM memory. This matter is resolved using an extra kernel module called uiodrv which alloc, map and release System RAM on user request. The memory granted is always contiguous and it's temporally removed from the internal Linux Memory management. Local and remote applications can use safely DMAs to extract/put data from/to the PC RAM. The maximum size granted from uiodrv is 128Kbytes due internal Linux kernel constrains. However, since 128 devices could be open by the driver, if more memory is needed. Current uiodrv design only provides virtual pointers to the current calling process. I should be very grateful if somebody could think about share the memory granted by uiodrv between external processes ;-). May the shm_xxx POSIX4 calls do it in the 3.0 kernel development. The libVMEBit3.a library is designed having in mind multiple threads applications, therefore each hardware device contain a POSIX1.0 semaphore which is used as "stopper" of multiple reentrant requests on a single Device. Bit3PCI.h provides a bit-field user interface to the internal PCI/VME registers, the header defines a complete structure set which behaviour should be carefully checked the first time that you run the software. If the bit-fields structures are still functional you only needs to index an structure to monitor/set a register flag or programming a DMA PCI address.

Bit3extras object provides dump/monitor functions to each one of the Bit3 registers blocks:
Local DMA, Remote DMA, Remote CSR (VME), Local CSR (PCI) and the Local PCI Configuration Space.

Each function returns a decoded and fixed formatted data stream which could be easily parsed by other kind of applications like TCP/IP or GUIs. An optional terminal dump can be as well requested as a parameter control. Examples of using are the incomplete GUI tool Bit3Tree built with gtk-1.2.1 developed to monitoring purposes. Complete applications are Bit3Util (terminal version of Bit3Tree) and Bit3Dump which dumps to the terminal the state of a range of page descriptors on user selection.

ROD code is completely developed and linked with ific-1.0 driver And Bit3-1.0 VME library. The 1.0 version is related with the rod driver design under Linux-2.2.x which gathers the Memory/Bit3 And SLink management on the kernel side. However, a public distribution ific-1.1 is available splitting the original design in three drivers: slinkdrv, uiodrv and bit3drv. The Bit3-1.1 library have been built on top of two of them (bit3drv and uiodrv). There are not dramatic changes on the internals of the library and applications because the services from the drivers layer are the same. We have considered that to be logically linked with other modules is enough to increase the version number. The only changes made, are four defines which redeclarate the file system nodes where each user object opens a file descriptor.

Performance

The DMA_Go Latency has been measured using a digital scope taking the time that the VME AS signal is high between two DMA cycles, 43usecare required to reprogram Bit3 registers under Linux2.2.1 running on a Pentium II 266MHz.

Raw DMA bandwidths are measured taking the time between a falling edge and a rising edge of AS in a single DMA block transfer cycle. Playing as Master device the performance for read cycles is close to 21Mbytes/sec and 15Mbytes/sec for the write cycles; the VME target is the system RAM of the PowerPC 2604. Using the MVME167 card as DMA target, read cycles bursts since 22Mbytes/sec and the write cycles takes 1ms to transfer 16Kbytes therefore there's a good correlation in the numbers.

In the Bit3 Slave mode, PowerPC reads data from PC memory using DMAs at a rate of 10Mbytes/sec, the same number is obtained if the MVME167 performs the DMA transfer; Write DMA cycles from the native VME hosts shows that 16Kbytes are moved in 1250 usec for MVME167 and 1216 usec for PowerPC, in other words roughly 12,5Mbytes/sec are obtained.

Conclusions

With the above VME layer, the ROD Controller, Module and Front-End software have been fully implemented under Linux-2.2.12. Such stuff is useful at the early stages of the project development where the re-coding cycle iteration and the OS reliability are fundamental issues.

TileCal ROD
Maintaned by Juanba
Ific, University of Valencia