Poster - Indico

The AMC13XG: A New Generation Clock/Timing/DAQ Module
for (CMS) MicroTCA
Detailed documentation at:
www.amc13.info
E. Hazen, J. Rohlf, S.X. Wu, A. Heister, C. Hill, D. Zou
Boston University, Boston, MA USA
Motivation: HCAL Upgrade
Analog
Signal
Copper
80 MB/s
Fiber
1.6Gb/s
AMC13 Construction
Fiber
Myrinet
2.5 Gb/s
S-Link
400 MB/s
T3 board
HTR
GOL
DCC
FRL
Crosspoint switch or other custom board
can be installed here
ADC
Front-End
VME
T2 Clocks board
Split/Patch
Analog
Signal
Fiber
4.8Gb/s
ADC
uHTR
GBT
AMC13
T4
T3
T2
T1
Quad SFP+
Cage
FEROL
CMS /
Level 1 Trigger
EMU Track Finder:
Overlap region TF:
Barrel TF:
Global Trigger:
Spares, Test stands:
TDC
Front-End
DAQ
Interface
MicroTCA
CMS / Pixels
Observation: All subsystems need timing / DAQ services,
So why not develop a common module?
AMC13 Logic
The AMC13 provides clock, timing and DAQ service for many subdetectors and
central systems in the upgraded CMS detector. This year we have developed an
upgraded module, the AMC13XG, which supports 10 Gigabit optical fiber and
backplane interfaces. Many of these modules are now being installed in the CMS
experiment during the current LHC shutdown.
G-2 (FNAL) /
Calorimeter
G-2 (FNAL) /
Tracker
Total (est)
AMC13XG Front Panel
LED2
LED1
The AMC13XG mounts in the MCH2 site of a “dual-star” MicroTCA crate, and thus
has point-to-point connections for clock and data fabrics to each of 12 AMC slots.
The module is constructed as a stack of 3 PC boards (Tongues 1, 2, 3).
SFP0
Serial #
(MMC console)
SFP1
(DAQ Output)
JTAG
JTAG
These tests establish the reliability of the high-speed serial interconnections. In each
plot, the horizontal axis represents the time of sampling of the serial data, with 0
representing the ideal time (middle of bit period). The vertical axis represents the
threshold voltage used to characterize a bit as '1' or '0'.
3x SFP+
10Gb/s capable
USB
Functions listed
for initial HCAL
firmware
Tongue 3
JTAG
Backplane Loop-Back @ 5.0 Gb/s
AMC Slot 1 (farthest from AMC13)
SFP2
(MMC AVR-32)
(Spare)
(6) GPIO
T3 LVDS
Clock
MMC
JTAG
Sensors
MMC
JTAG
(AMC13 FPGAs)
Fabric B
TX: TTC out
RX: -spare-
Spartan 6
SPI
AT32UC3A1
SFP3
XC6SLX25T-2FG484C
Handle
GTP
SPI Flash
AMC13 T1 Board
UTCA
T2
Fanout
GTP
Top Layer 1 (signal)
Fanout
M25P128
128Mbit
HS to T2
1.8V
2.0V
PCB Stackup
UTCA connector
DDR3 power
Mux
SFP+
Nelco 4000SI-13
DDR3
2.5 Gb/s
Ethernet+
Private
1
GTL
Z0 ctrl (10G, SDRAM)
Prepreg 2.7
2
IPMB bus
Tongue 1
Divide
by 4
GP1
3
Split power
GP2
Prepreg 5.4
4
40MHz
5
6
LS to T2
CLK
7
8
Split power
GP3
Prepreg 2.7
GP4
GND
9
UTCA
T1
SFP+ TTC
Prepreg 5.4
G2
Z0 ctrl (SDRAM)
GP6
GND
Prepreg 5.4
10 G3
Z0 ctrl (10G, SDRAM)
Core 5.0
11 GP7
GND
Prepreg 2.7
12 GBL
Kintex 7
GTX
XC7K325T-2FFG900
.
.
.
GTX
GTX
16 (data)
17
(addr)
1.0V aux
3.3V Payload
AMC13 T2 Board
GTX
Fiber clock
16
Backplane clock
Memory clock
Ethernet clock
DDR3-1600
speed
128M x 16
DDR3 SDRAM
UTCA connector
Spartan 6 FPGA
128M x 16
DDR3 SDRAM
The AMC13 Machine clock recovery begins on tongue 1 with an optical receiver
feeding an Analog Devices ADN2814 Clock and Data Recovery IC. This device
process i.e. an LHC Trigger, Timing and Control (TTC) encoded stream and recovers
clock and data. The clock (160MHz for the TTC case) is fanned out and delivered to
several alternative clock inputs on the Kintex 7 FPGA on tongue 1 as well as to
Tongue 2. The recovered data is sent to the Kintex 7 for further processing.
The AMC13 clock fanout is located on tongue 2. It can take input from either a
front-panel input on tongue 3, or from the TTC receiver on tongue 1. The clock is
fanned out to 12 AMC modules via the MicroTCA backplane as well as to the Spartan
FPGA. From there it can be returned to the Kintex FPGA on tongue one for special
applications.
GTL
Z0 ctrl (TTC)
5 mil dielectric
2
Clock fanout ICs
3
GP1
GP2
GND
3 mil dielectric
Split power
9 mil dielectric
Connector to T3
4
G1
Inner signal
9 mil dielectric
5
Connector from T1
GP3
Split power
9 mil dielectric
6
Inner Signal
G2
9 mil dielectric
7
8
GND
GP4
GBL
Splitters installed on GOL links
Parallel readout in VME and uTCA
Events “tagged” by specific L1A ID
(programmed HLT to save events)
HCAL uTCA Data!
xDAQ software to control uTCA
Results:
All data match perfectly!
5 mil dielectric
Z0 ctrl (TTC)
AMC13 Software
LHC bunch structure seen in uTCA
A set of C++ classes is provided for full control of the AMC13. Command-line tools support
testing and diagnostics. CMS xDAQ applications support operation and monitoring for
CMS HCAL (currently) and generic CMS operation (soon).
AMC13 in CMS MicroTCA Crate
A HyperDAQ web display provides real-time status during data taking.
DAQ optical fibers (5/10 Gb/s)
Fiber links from detector (i.e. GBT)
TTC / TTS (160.xxx Mb/s)
Selectable level of detail
MicroTCA
Backplane
5.0 Gb.s
L1A
FIFO
Possible 2nd
DAQ fiber
Note: Data could flow
through SDRAM
AMC
Event
Builder
DAQ Tx
AMC
FIFO
DAQ
Fiber
Power
DAQ Tx
Power
Fiber Out
5.0 Gb/s
(to 10 Gb/s)
FIFO
TTC
Thickness
Overall: 1.6mm
Signal: 18μm
Power: 36μm
FR-4
1
AVR 32 (MMC)
AMC13 DAQ Path
Link Tx
(in AMC)
PCB Stackup
Fabric B (TTC)
The AMC13 local clock subsystem is quite flexible. A SiLabs Si5338B quad
programmable “fractional-N” type clock synthesizer is used to generate four
independent clocks. These can be used in addition to the TTC recovered clock to
drive the backplane links, front-panel fiber links, Gigabit Ethernet and DDR3 memory.
The AMC13 Module Management Controller (MMC) is implemented with an Atmel
32 bit AVR microcontroller, with firmware developed by colleagues at the University of
Wisconsin. In addition to required MicroTCA MMC functions, our MMC provides
several extended features, allowing for fine-grained control over fault conditions,
remote setting of module IP addresses and other parameters.
MicroTCA Crate at P5
AMC
Si5338B
Quad Clock
AMC port 1
1.2V
MCH1
SFP+ Spare
1.0V
Fabric A
5.0 Gb/s
AMC
SFP+ DAQ 1
Z0 ctrl (10G, SDRAM)
GTX
As a validation test of the MicroTCA readout system, the front-end data for the HCAL
subdetector of CMS was split using optical splitters on the fibers from the on-detector
electronics. Collider runs were taken in CMS and the data was compared byte-wise
between the legacy VME electronics and the new MicroTCA system. An exact
match was found in all data.
AMC
SFP+ DAQ 0
HCAL Slice Test
Split power
GP5
Core 5.0
GTP
DAT
Z0 ctrl (SDRAM)
Fiber Optic Loop-Back Test – In this test a 30m optical fiber is connected between
an SFP transmitter and receiver, and a PRBS is sent through the fiber.
Core 5.0
4 GPIO
reserved
IN
G1
Core 5.0
GbE from MCH
Backplane Loop-Back Test – In this test a pseudo-random bit stream (PRBS) is
sent down the backplane to an AMC slot, and electrically looped-back to the AMC13.
This corresponds to double the trace length of a normal point-to-point connection.
GND
Core 5.0
Kintex-7
160 MHz
Fanout
Clock/Data Separator
ADN2814
Thickness
Overall: 1.6mm
Signal: 18μm
Power: 36μm
AMC
Voltages
Fiber Optic Loop-Back @ 10.0Gb/s
(TTC/TTS)
AMC
Serial
Console
FPGA
JTAG
AMC
Tongue 2
110
Serial Link Tests
(DAQ Loop-back test)
USB
3 Custom firmware
(MMC Green LED)
(MMC Red LED)
AMC13 Block Diagram
3
3
3
1
20
6 TTC only, readout direct from
FEDs?
10 Custom T3 for front-panel clock
and control inputs
28 Custom firmware
CMS / TCDS
AMC
Detector
AMC
ADC
(500/1G B/s)
Power
JTAG (MMC and Xilinx)
Utility SPI
MMC serial console
30 May require 2-3 x 10Gb/s DAQ
outputs
6
36 Calo trigger:
AMC
ADC
Fiber
10 GbE
No. Special Requirements
AMC13
CMS / HCAL
Connector to T3 provides:
Clock / controls fanout
Fiber
5.0/10 Gb/s
uTCA Backplane
5.0 Gb/s (250 MB/s)
Experiment /
Subsystem
AMC
Detector
MMC functions (Wisconsin firmware)
TTC optical rx
3x SFP+ cage
Cross-over GbE from MCH1
for controls and local DAQ
AMC13
ADC
Estimated AMC13 Use through 2014
T1 base board
Provides JTAG / LEDs on front panel
Can be removed after initial programming
ADC
AMC13 Test and Production
8k event
SDRAM
IPbus control / monitor / local DAQ
This page provides an overview of the operation
of one AMC13. Color highlights various warning
and error conditions in real time
Fiber links to trigger
Ethernet
GbE
The AMC13 DAQ Path is detailed above. In response to each Level 1 trigger (L1A),
each AMC card in the MicroTCA crate sends a CRC-protected packet over the
backplane to the AMC13 at 5.0Gb/s. The packet has a header with event number, bunch
crossing number and orbit number along with an optional subsystem-specific payload.
The AMC13 event builder collects these packets and builds event fragments. The event
builder output can be sent to one, two or three 5/10Gb fiber links to CMS CDAQ, and
additionally to an SDRAM buffer which can hold several thousand events.
The SDRAM buffer may be prescaled, and may capture “windows” around events which
exhibit specific error conditions. The buffer may be read out to a host PC over the 1GbE
backplane link via the MicroTCA Carrier Hub
The AMC13 also supports 10GbE TCP/IP readout on up to 3 links. A simplified subset of
the TCP/IP protocol allows transfer at close to 1GB/s to a standard PC..
AMC13
HCAL uHTR
Fiber Link to DAQ
●
●
AMC module
5.0 Gb/s optical link with “S-Link like” protocol
Firmware developed by CMS CDAQ (both ends)
–
–
–
Error check coding, retransmission on error
Error monitoring
Full diagnostic and test capability from receive end
AMC13
Data from
FED
-DATA (64 bit)
-WEN
-UCTRL
-CLOCK
-Backpressure
-link down
FEROL
4 blocks
(4Kbytes each)
SFP itf
Main
Logic
Block is sent until it is
acknowledged
Internal
Receive CMD
+ ACk
Vadatech VT892 Crate
-Receive block
-Ack. block
-Order blocks
Send commands
(one at the time)
MCH
(commercial)

Download Report