Back to SBS Shift Takers Wiki
- 1 Description
- 2 High Voltage
- 3 GEM Gas Controls
- 4 EPICS Server Software
New EPICS controls for high voltage control/monitoring and for GEM gas system monitoring have been developed for the SBS experiments.
The High voltage for PMTs and GEMs for SBS experiments several different types of power supplies. These include:
- 5 Lecroy mainframes
- A CAEN 1527 mainframe (removed)
- A CAEN 5527 mainframe for the BigBite Hodoscope
- A Wiener Mpod mainframe for (UVA) GEM detectors
- CAEN VME V6521 cards for (INFN) GEM detectors
- CAEN NIM N1471 HV card for GEM detectors
- CAEN HiVolta for GEM detectors
- CAEN 4527 mainframe(s) (possible use for GeN-RP)
Each of these systems has a vendor or lab developed graphical or web interface. In order to simplify operations, EPICS controls have been implemented for all these systems.
When large blocks or individual channels for detectors go missing, first check if the EPICs gui is malfunctioning. If so, try the following:
Now, as far as addressing the EPICs software "symptoms" as I call it when hv channels cut off, the best thing to try first is
- ssh aslow@adaqsc
- ps -aux | grep java
- kill -9 #####
Where ##### is the number of the old java sessions still running from past instances of the EPICs gui being opened. Kill all old java sessions including the current one. Then restart the EPICs gui. Check all detector hv channels in the gui that they have the correct Vset values as sometimes the EPICs gui does not load these correctly at first.
Steve Wood added that one thing that causes problems with the HV is when there are too many copies of the java Phoebus CSS GUI running. As time goes on, the number of separate GUI’s running on adaqsc tends to go into the dozens, using up OS resources. Killing all the GUIs and just starting what is needed clears things up.
Carlos reports as well that several instances of CS-Studio can cause problems, as was reported before: https://logbooks.jlab.org/entry/4091669
User control of high voltage
EPICS Graphical interfaces have been implemented for all the HV under EPICS control. These interfaces include the ability to backup and restore HV settings.
The various types of high voltage mainframes have vendor or user supplied control interfaces that don't use EPICS. These interfaces should continue to work.
To start the HV GUI controls, logon to "aslow@adaqsc" and type the command
This will bring up a main screen from which a desired detector or sub-detector can be selected. Clicking on one of the detector buttons will give a choice of "Listed Controls & Monitor" or "Histogram Monitor." Choosing the listed controls option will bring up a table of actual voltage and currents, and fields in which the voltage, trip current, maximum voltage and ramp rates can be changed.
For a graphical view of the high voltage, choose the second option when clicking on a detector button. This will show a bar chart of voltage and current for all the channels.
Backup and Restore
High voltage settings may be backed up and restored from the HV GUI. Select "Backup" from the "Backup /Restore" button. HV backups, which are readable text files, will be placed in
~aslow/EPICS/HV/backup-dir. "Restore" should be used with care. It is recommended to only restore HV settings for a single detector or portion of a detector at a time. This may be done by restoring a copy of a backup file where all the lines except for the detector of interested have been removed.
For each HV channel, a voltage difference (VDiff) EPICS signal is provided. Alarm limits are places on these signals so that shift workers will be notified via the alarm handler if a channel trips off or if the actual voltage provided by the channel differs from the set voltage. By default, a minor alarm is raised if the monitored voltage is more than 25 volts above or below the set voltage, and a major alarm is raised if the voltage is more than 50 volts away from the set voltage. These limits may be changed for individual channels using the alarm handler.
Building GUI screens
If detector HV is changed due to bad channels or other reasons, control
screens will need to be rebuilt. The configuration files are the HV.hvc and HV.group files located in subdirectories of
~aslow/EPICS/HV. The relevant subdirectories are presently bb_gem, bb_hodo, bb_other, lhrs_fp, and sbs_hcal. The subdirectories are defined by the Makefiles in the "all" subdirectories, bb_all, sbs_all and lhrs_all.
The HV.hvc in each subdirectories defines the HV for one or more detectors. Each line of the file is of the form:
ElementName Crate# Slot# Channel# Group# Row Column
The group numbers are defined in the HV.group files. The set of HV channels in each group are accessed by corresponding button on the HV GUI. The Row and Column can indicate vertical and horizontal position of the element within a detector such as a calorimeter or they may be set to "-1 -1".
If an HV.hvc file is edited, type make within the directory containing the modified file. Then type make from
~aslow/EPICS/HV. This will rebuild the HV GUI.
After rebuilding the GUI, all running instances of the GUI should be restarted.
Every high voltage mainframe must be assigned a unique crate/mainframe number. For the Lecroy crates, this crate number is automatically assigned from the Raspberry Pi hostname. If the hostname is of the form rpiNN, then NN is the crate number. Other HV system types are assigned a crate number manually. The following mainframes are in use for the SBS experiments:
0 CAEN1527 BigBite Hodoscope (removed) 8 Lecroy HRS Left 17 Lecroy BigBite 18 Lecroy BigBite 20 Lecroy Hcal 21 Lecroy Hcal 81 CAEN 5527 BitBite Hodoscope 91 MPOD UVA GEM HV 92 CAEN V6521 INFN GEM HV 93 CAEN 4527 GeN-RP 95 CAEN N1470 GEM HV 96 CAEN HiVolta GEM HV
EPICS High Voltage PV Naming Convention
The EPICS Process Variables (PVs) for individual HV channels are chosen to have a consistent form, independent of hardware. The form is chosen to match the built in EPICS on CAEN4527 mainframes. The use of consistent PV naming convention allows for GUIs that are independent of hardware.
EPICS PVs for HV have the form
"%s%d:%.2d:%.3d:%s", prefix, crate, slot, channel, property
where prefix is a string unique to Hall A ("HAHV"), crate is a mainframe number (from 0 to 99), slot is a card # within a mainframe (0-99), channel is the channel number within a card (0-999) and property is the attribute to be written or read. For example, the monitored voltage (VMon property) for crate 7, slot 5, channel 3 will have the PV "HAHV7:05:003:VMon".
Properties common to all hardware types are:
V0Set (w) Voltage setting for a channel V0Setr (r) Readback of voltage setting VMon (r) Monitored/measured voltage I0Set (w) Setting for Current at which to trip channel I0Setr (r) Readback of trip current setting IMon (r) Measured current Pw (w) (0/1) Turn off or on the channel Pwr (r) Readback of on/off status RUp (w) Ramp rate in V/s when turning on channel Rupr (r) Readback of ramp up rate RDWn (w) Ramp down rate in V/s when turning off channel RDWnr (r) Readback of ramp up rate VDiff (r) Difference between set and monitor voltage. Used for alarms.
These property names are chosen to be consistent with the property ppnames of the CAEN4527 built in EPICS.
GEM Gas Controls
The Detector Support Group (DSG) has developed a system using small sensors to detect the gas flow of each channel of the GEM gas distribution system (also developed/designed/assembled by DSG). The SBS system is based on the BigBite system which is running in Hall A right now. The BigBite system is an 8 channel system, while the the SBS will have 42 channels.
The system was designed to provide flow values for each individual GEM layer as opposed to using a single mass flow meter, which would become cost prohibitive to provide 50 channels of flow.
These systems can be monitored at SBS GEM Flow Readout and BigBite GEM Gas Flow Readout screens.
For system details see these DSG publications:
DSG development of the GEM gas distribution and readout system
DSG prototype gas distribution system with remote monitoring
DSG Gas flow monitoring for Hall A GEM
DSG GEM gas distribution system
DSG designed flow sensor chassis
The GEM detector Gas distribution system pushes information to EPICS for monitoring. The EPICS signals are on the IOC iocGEMGas running the GEMGasApp on adaqsc.
EPICS Server Software
The master copy of the EPICS HV controls for SBS are in the directory
~aslow/EPICS on adaqsc. The
EPICS directory contains the following
- The base EPICS distribution from https://epics.anl.gov/
- Contains CAEN libraries that are needed for the communicating with SY1527 HV crates
- Contains the source code for EPICS support to provide IOCs (Input Output Controllers) for all the models of HV crates in use. Also contains configuration files to define EPICS signals for all the HV channels in use.
The base-3.15.8 and sbs-epics directories are also installed on the Raspberry Pi for each Lecroy Mainframe and on intelbbmpd for controlling CAEN VME HV cards.
sbs-epics/apps contains these source code/configuration directories:
- Mirrors CAEN 5527 crate signals to provide "fake" readback signals. (See CAEN 4527-5527 section below) May also be used with CAEN 4527 for Gen-RP
- Code and configuration for CAEN SY1527 mainframes. (Used by BigBite hodoscope)
- Code and configuration for CAEN VME HV cards. (Used by GEM detectors
- Scripts to start the IOCs for each of the HV systems
- Code and configuration for RPi enabled Lecroy Mainframtes
- 2.6-0App : A wrapper for running the EPICS IOCs
- Code and configuration for communicating with SNMP systems. Used for Wiener HV.
- Library routines needed by caenHvApp
- Soft Channel records for BB Gas system
- Serial device support. Dependency of StreamDevice
- Support for byte stream devices. Dependency of caenN1470App.
- Configuration for CAEN N1470 and HiVolta HV.
This code is compiled by running make from sbs-epics/apps. The Makefiles in some of the above source code directories are setup to compile only under certain conditions:
caenHvApp and syX527App will only be compiled if the
clas12-epics-third-party-libs directory exists.
snmpApp will only be compiled if snmp support is installed on the
operating system. (By checking if
caenV65XXApp is only compiled if LINUXVME_INC is defined.
ProcServ and ProcServMgr
On systems where the EPICS IOCS (servers) run, a cron task is
configured to periodically run ProcServMgr. ProcServMgr looks at the
sbs-epics/apps/iocBoot/procServMgr.conf to determine which IOC
tasks need to run on that system and starts any IOCs that are not
running. The IOC tasks are run by ProcServ which allows experts to
login to the IOC with the command
telnet PORT where the
PORT number is defined in
Hardware specific notes
Hall A has a large number of Lecroy Mainframes. These were controlled either by ARCnet or RS232. Several years ago, the PC boards in each of these mainframes was replaced with Raspberry Pi boards that communcate with the mainframe via a custom serial board developed by Javier Gomez and Jack Segal. Two processes run on each board. A process that communicates with the HV cards (written by J. Gomez) and a process (shim) (written by Brad Sawatzky) that mimics the original ARCnet/RS232 command set, except that it uses TCP/IP. This is further described in HV HowTo for Experts.
In 2000, Chris Slominski developed an EPICS driver for the Lecroy mainframes that communicated with ARCnet. This driver has been adapted to use the TCP/IP interface that the modified Lecroy mainframes now provide.
A single linux IOC can be configured to support multiple mainframes or an IOC can be run on each rPI board. For the SBS HV, the later method is used. Each mainframe is configured to create create EPICS signals for the installed HV cards.
To install EPICS on a Lecroy Mainframe, use the following steps.
- Setup a rpi with a hostname of rpiN, where N is not an existing HV crate number (See "Crate Numbering" section). (Unless it is replacing an existing crate.)
- Insure that the rPI board in the mainframe has the software prerequisites to compile C programs (gcc, ncurses, readline) and the perl modules needed by the shim (IO-Multiplex, Net-Server, IO-Tty, Expect). If the JAVA GUI already works with this mainframe, then these prerequisites are installed. If they are not, ask the Spectrometer Support Group to prepare an RPI with these prerequisites.
- Obtain the files scripts.tar.gz, lecroy_servers.tar.gz and epics.tar.gz and place them in the home directory of the pi account on the rPI. These can be found in the directory
- Untar all three files into the pi account home directory.
~/scripts/kill_hv_allto kill and disable old versions of the Lecroy software.
- Wait 5 minutes and check that an "i2lchv" process and a "LecroyHV_shim" procees is running.
- Check that the EPICS server is running with
telnet localhoset 20000. You should see the
dblshould show a long list of EPICS signal names starting with "HAHV91".
Lecroy Cron job
The cronjob that is setup with
crontab ~/scripts/cron.tab run the
start_hv_cron_new every four minutes. This script checks
that each of the servers, i2lchv, the shim and the epics server is
running. It will start any server that is not running.
If the EPICS server is not running,
start_hv_cron_new will first run
~/scripts/configure_epics.py. This python script does several
- Determines the mainframe/crate number from the hostname rpiN.
- Reads the file
/home/pi/scripts/LecroySlotList.txtto find which # slots are occupied and what type of module is in each slot. Currenly, only 1461N, 1461P and 1471N cards are supported.
- Using the slot information, define epics signals for every HV # channel in the crate by writing the file
The CAEN1527 was used for the BigBite hodoscope. It is instrumented with four 48 channel A1932 cards. These cards differ from most other HV cards. There is a single primary power supply and 48 secondary channels which supply the PMTs. The voltage on the primary supply can be set to an arbitrary value. The secondary channels can only be set to the primary voltage or less. Not all attributes are available on both primary and secondary channels. When an attribute doesn't exist, the GUI will show -1.0.
The A1932 cards have been moved to a CAEN 5527 system.
To troubleshoot EPICs with this hv mainframe trying disable the EPICS service and restart it; and also power cycle the crate. All of this is possible remotely via the web app. Or it can be done manually by hand following the new EPICs instructions in the timing hodoscope how-to document.
UVA GEM modules. Wiener MPOD systems are controlled by SNMP (Simple Network Management Protocol). The standard EPICS [https://groups.nscl.msu.edu/controls/files/devSnmp.html SNMP EPICS driver] is used.
The IOC for the Wiener MPOD system runs on adaqsc. It is started by ProcServMgr.
CAEN VME V6521 cards
CAEN VME V6521 cards are used for the INFN GEM detectors. These cards
do not use a custom HV mainframe, just a regular VME crate. The
EPICS driver for these cards runs on the Linux Single board computer
(intelbbmpd) in the VME crate with the cards. The driver software for
these cards is in
sbs-epics/apps/caenV65XXApp and the startup script for the IOC
sbs-epics/apps/iocBoot/iocv65xx/st.cmd. Slot number used in the EPICS PVs is
not the physical slot but rather the logical slot, where slot 0 is the
first card defined, slot 1 is the second card defined etc. The cards
are defined with the command
V65XXInit(Ncards, baseaddr, incaddr)
st.cmd. For example,
0x100000) will define "slots" 0-2 with VME addresses
0xa00000, 0xb00000, and 0xc00000.
The CAEN 4527 and CAEN5527 are relatively modern high voltage systems. A 5527 is used for the BigBite Hodoscope and 4527's may be used for some SBS experiments such as GeN-RP.
The CAEN 4527 and 5527 have built in EPICS. The EPICS server can be enabled in the mainframe's web interface. The EPICS PV name prefix (e.g. "HAHV") and mainframe number are also set in this interface.
The CAEN 4527 and 5527 EPICS interfaces do not supply separate signal names for setting and readback of setting. For example, the PV HAHV93:00:000:V0Set is used both to set the voltage and to readback the voltage setting. In order provide compatibility with the other EPICS drivers, the Alias4527 app is used. This app mirrors each of writable signal names to provide a "fake" readback signal. The Alias4527 also provides a VDiff signal for each channel for alarms.
A1932 cards do not have some fields for the secondary channels. (Current trip set point, maximum high voltage and current monitoring.) The app is also used to create dummy EPICS records for these fields to keep the GUIs happy.
These mirror signals for the 5527 crate with A1932 cards are configured in the files
A1932_secondary.substitutions. The crate number is configured in the ioc startup script