SBS EPICS

Back to SBS Shift Takers Wiki

Description

New EPICS controls for high voltage control/monitoring and for GEM gas system monitoring have been developed for the SBS experiments.

High Voltage

The High voltage for PMTs and GEMs for SBS experiments several different types of power supplies. These include:

• 5 Lecroy mainframes
• A CAEN 1527 mainframe (removed)
• A CAEN 5527 mainframe for the BigBite Hodoscope
• A Wiener Mpod mainframe for (UVA) GEM detectors
• CAEN VME V6521 cards for (INFN) GEM detectors
• CAEN NIM N1471 HV card for GEM detectors
• CAEN 4527 mainframe(s) for GEM detectors

Each of these systems has a vendor or lab developed graphical or web interface. In order to simplify operations, EPICS controls have been implemented for all these systems.

Troubleshooting EPICs

When large blocks or individual channels for detectors go missing, first check if the EPICs gui is malfunctioning. If so, try the following:

Now, as far as addressing the EPICs software "symptoms" as I call it when hv channels cut off, the best thing to try first is

• ssh aslow@adaqsc
• ps -aux | grep java
• kill -9 #####

Where ##### is the number of the old java sessions still running from past instances of the EPICs gui being opened. Kill all old java sessions including the current one. Then restart the EPICs gui. Check all detector hv channels in the gui that they have the correct Vset values as sometimes the EPICs gui does not load these correctly at first.

Steve Wood added that one thing that causes problems with the HV is when there are too many copies of the java Phoebus CSS GUI running. As time goes on, the number of separate GUI’s running on adaqsc tends to go into the dozens, using up OS resources. Killing all the GUIs and just starting what is needed clears things up.

Carlos reports as well that several instances of CS-Studio can cause problems, as was reported before: https://logbooks.jlab.org/entry/4091669

User control of high voltage

EPICS Graphical interfaces have been implemented for all the HV under EPICS control. These interfaces include the ability to backup and restore HV settings.

The various types of high voltage mainframes have vendor or user supplied control interfaces that don't use EPICS. These interfaces should continue to work.

To start the HV GUI controls, logon to "aslow@adaqsc" and type the command

 go_hv

This will bring up a main screen from which a desired detector or sub-detector can be selected. Clicking on one of the detector buttons will give a choice of "Listed Controls & Monitor" or "Histogram Monitor." Choosing the listed controls option will bring up a table of actual voltage and currents, and fields in which the voltage, trip current, maximum voltage and ramp rates can be changed.

For a graphical view of the high voltage, choose the second option when clicking on a detector button. This will show a bar chart of voltage and current for all the channels.

Backup and Restore

High voltage settings may be backed up and restored from the HV GUI. Select "Backup" from the "Backup /Restore" button. HV backups, which are readable text files, will be placed in ~aslow/EPICS/HV/backup-dir. "Restore" should be used with care. It is recommended to only restore HV settings for a single detector or portion of a detector at a time. This may be done by restoring a copy of a backup file where all the lines except for the detector of interested have been removed.

HV Alarms

For each HV channel, a voltage difference (VDiff) EPICS signal is provided. Alarm limits are places on these signals so that shift workers will be notified via the alarm handler if a channel trips off or if the actual voltage provided by the channel differs from the set voltage. By default, a minor alarm is raised if the monitored voltage is more than 25 volts above or below the set voltage, and a major alarm is raised if the voltage is more than 50 volts away from the set voltage. These limits may be changed for individual channels using the alarm handler.

Building GUI screens

If detector HV is changed due to bad channels or other reasons, control screens will need to be rebuilt. The configuration files are the HV.hvc and HV.group files located in subdirectories of ~aslow/EPICS/HV. The relevant subdirectories are presently bb_gem, bb_hodo, bb_other, lhrs_fp, and sbs_hcal. The subdirectories are defined by the Makefiles in the "all" subdirectories, bb_all, sbs_all and lhrs_all.

The HV.hvc in each subdirectories defines the HV for one or more detectors. Each line of the file is of the form:

 ElementName Crate# Slot# Channel# Group# Row Column

The group numbers are defined in the HV.group files. The set of HV channels in each group are accessed by corresponding button on the HV GUI. The Row and Column can indicate vertical and horizontal position of the element within a detector such as a calorimeter or they may be set to "-1 -1".

If an HV.hvc file is edited, type make within the directory containing the modified file. Then type make from ~aslow/EPICS/HV. This will rebuild the HV GUI.

After rebuilding the GUI, all running instances of the GUI should be restarted.

Crate Numbering

Every high voltage mainframe must be assigned a unique crate/mainframe number. For the Lecroy crates, this crate number is automatically assigned from the Raspberry Pi hostname. If the hostname is of the form rpiNN, then NN is the crate number. Other HV system types are assigned a crate number manually. The following mainframes are in use for the SBS experiments:

   0	CAEN1527   BigBite Hodoscope (removed)
   8	Lecroy	   HRS Left
   17	Lecroy	   BigBite
   18	Lecroy	   BigBite
   20	Lecroy	   Hcal
   21	Lecroy	   Hcal
   70  CAEN 4527  GEM HV (Hall A)
   71  CAEN 4527  GEM HV (EEL)
   81	CAEN 5527  BitBite Hodoscope
   91	MPOD	   UVA GEM HV
   92	CAEN V6521 INFN GEM HV
   93	CAEN 4527  GeN-RP
   95	CAEN N1470 GEM HV
   96	CAEN HiVolta GEM HV

EPICS High Voltage PV Naming Convention

The EPICS Process Variables (PVs) for individual HV channels are chosen to have a consistent form, independent of hardware. The form is chosen to match the built in EPICS on CAEN4527 mainframes. The use of consistent PV naming convention allows for GUIs that are independent of hardware.

EPICS PVs for HV have the form

"%s%d:%.2d:%.3d:%s", prefix, crate, slot, channel, property

where prefix is a string unique to Hall A ("HAHV"), crate is a mainframe number (from 0 to 99), slot is a card # within a mainframe (0-99), channel is the channel number within a card (0-999) and property is the attribute to be written or read. For example, the monitored voltage (VMon property) for crate 7, slot 5, channel 3 will have the PV "HAHV7:05:003:VMon".

Properties common to all hardware types are:

   V0Set  (w) Voltage setting for a channel
   V0Setr (r) Readback of voltage setting
   VMon   (r) Monitored/measured voltage
   I0Set  (w) Setting for Current at which to trip channel
   I0Setr (r) Readback of trip current setting
   IMon   (r) Measured current
   Pw     (w) (0/1) Turn off or on the channel
   Pwr    (r) Readback of on/off status
   RUp    (w) Ramp rate in V/s when turning on channel
   Rupr   (r) Readback of ramp up rate
   RDWn   (w) Ramp down rate in V/s when turning off channel
   RDWnr  (r) Readback of ramp up rate
   VDiff  (r) Difference between set and monitor voltage.  Used for alarms.

These property names are chosen to be consistent with the property ppnames of the CAEN4527 built in EPICS.

GEM Gas Controls

The Detector Support Group (DSG) has developed a system using small sensors to detect the gas flow of each channel of the GEM gas distribution system (also developed/designed/assembled by DSG). The SBS system is based on the BigBite system which is running in Hall A right now. The BigBite system is an 8 channel system, while the the SBS will have 42 channels.

The system was designed to provide flow values for each individual GEM layer as opposed to using a single mass flow meter, which would become cost prohibitive to provide 50 channels of flow.

These systems can be monitored at SBS GEM Flow Readout and BigBite GEM Gas Flow Readout screens.

For system details see these DSG publications:

DSG development of the GEM gas distribution and readout system

DSG prototype gas distribution system with remote monitoring

DSG Gas flow monitoring for Hall A GEM

DSG GEM gas distribution system

DSG GEM readout electronics

DSG designed flow sensor chassis

The GEM detector Gas distribution system pushes information to EPICS for monitoring. The EPICS signals are on the IOC iocGEMGas running the GEMGasApp on adaqsc.

EPICS Server Software

EPICS Software

The master copy of the EPICS HV controls for SBS are in the directory ~aslow/EPICS on adaqsc. The EPICS directory contains the following directories:

base-3.15.8 

The base EPICS distribution from https://epics.anl.gov/

clas12-epics-third-party-libs 

Contains CAEN libraries that are needed for the communicating with SY1527 HV crates

sbs-epics 

Contains the source code for EPICS support to provide IOCs (Input Output Controllers) for all the models of HV crates in use. Also contains configuration files to define EPICS signals for all the HV channels in use.

The base-3.15.8 and sbs-epics directories are also installed on the Raspberry Pi for each Lecroy Mainframe and on intelbbmpd for controlling CAEN VME HV cards.

sbs-epics/apps contains these source code/configuration directories:

Alias4527 

Mirrors CAEN 5527 crate signals to provide "fake" readback signals. (See CAEN 4527-5527 section below) May also be used with CAEN 4527 for Gen-RP

caenHvApp 

Code and configuration for CAEN SY1527 mainframes. (Used by BigBite hodoscope)

caenV65XXApp 

Code and configuration for CAEN VME HV cards. (Used by GEM detectors

iocBoot 

Scripts to start the IOCs for each of the HV systems

lecroyApp 

Code and configuration for RPi enabled Lecroy Mainframtes

procServ

2.6-0App : A wrapper for running the EPICS IOCs

snmpApp 

Code and configuration for communicating with SNMP systems. Used for Wiener HV.

syX527App 

Library routines needed by caenHvApp

GEMGasApp 

Soft Channel records for BB Gas system

asynDriver 

Serial device support. Dependency of StreamDevice

StreamDevice 

Support for byte stream devices. Dependency of caenN1470App.

caenN1470App 

Configuration for CAEN N1470 and HiVolta HV.

This code is compiled by running make from sbs-epics/apps. The Makefiles in some of the above source code directories are setup to compile only under certain conditions:

caenHvApp and syX527App will only be compiled if the clas12-epics-third-party-libs directory exists.

snmpApp will only be compiled if snmp support is installed on the operating system. (By checking if /usr/include/net-snmp/net-snmp-config.h exists.)

caenV65XXApp is only compiled if LINUXVME_INC is defined.

ProcServ and ProcServMgr

On systems where the EPICS IOCS (servers) run, a cron task is configured to periodically run ProcServMgr. ProcServMgr looks at the file sbs-epics/apps/iocBoot/procServMgr.conf to determine which IOC tasks need to run on that system and starts any IOCs that are not running. The IOC tasks are run by ProcServ which allows experts to login to the IOC with the command telnet PORT where the PORT number is defined in ProcServMgr.conf.

Hardware specific notes

Lecroy

Hall A has a large number of Lecroy Mainframes. These were controlled either by ARCnet or RS232. Several years ago, the PC boards in each of these mainframes was replaced with Raspberry Pi boards that communcate with the mainframe via a custom serial board developed by Javier Gomez and Jack Segal. Two processes run on each board. A process that communicates with the HV cards (written by J. Gomez) and a process (shim) (written by Brad Sawatzky) that mimics the original ARCnet/RS232 command set, except that it uses TCP/IP. This is further described in HV HowTo for Experts.

In 2000, Chris Slominski developed an EPICS driver for the Lecroy mainframes that communicated with ARCnet. This driver has been adapted to use the TCP/IP interface that the modified Lecroy mainframes now provide.

A single linux IOC can be configured to support multiple mainframes or an IOC can be run on each rPI board. For the SBS HV, the later method is used. Each mainframe is configured to create create EPICS signals for the installed HV cards.

To install EPICS on a Lecroy Mainframe, use the following steps.

1. Setup a rpi with a hostname of rpiN, where N is not an existing HV crate number (See "Crate Numbering" section). (Unless it is replacing an existing crate.)
2. Insure that the rPI board in the mainframe has the software prerequisites to compile C programs (gcc, ncurses, readline) and the perl modules needed by the shim (IO-Multiplex, Net-Server, IO-Tty, Expect). If the JAVA GUI already works with this mainframe, then these prerequisites are installed. If they are not, ask the Spectrometer Support Group to prepare an RPI with these prerequisites.
3. Obtain the files scripts.tar.gz, lecroy_servers.tar.gz and epics.tar.gz and place them in the home directory of the pi account on the rPI. These can be found in the directory ~/EPICS/RPI of the aslow@adaqsc account.
4. Untar all three files into the pi account home directory.
5. Type crontab -r and ~/scripts/kill_hv_all to kill and disable old versions of the Lecroy software.
6. Type crontab ~/scripts/cron.tab
7. Wait 5 minutes and check that an "i2lchv" process and a "LecroyHV_shim" procees is running.
8. Check that the EPICS server is running with telnet localhoset 20000. You should see the epics> prompt. Typing dbl should show a long list of EPICS signal names starting with "HAHV91".

Lecroy Cron job

The cronjob that is setup with crontab ~/scripts/cron.tab run the script start_hv_cron_new every four minutes. This script checks that each of the servers, i2lchv, the shim and the epics server is running. It will start any server that is not running.

If the EPICS server is not running, start_hv_cron_new will first run ~/scripts/configure_epics.py. This python script does several things.

1. Determines the mainframe/crate number from the hostname rpiN.
2. Reads the file /home/pi/scripts/LecroySlotList.txt to find which # slots are occupied and what type of module is in each slot. Currenly, only 1461N, 1461P and 1471N cards are supported.
3. Using the slot information, define epics signals for every HV # channel in the crate by writing the file LecroyHV.substitutions to /home/pi/epics/sbs-epics/apps/db/.

CAEN 1527

The CAEN1527 was used for the BigBite hodoscope. It is instrumented with four 48 channel A1932 cards. These cards differ from most other HV cards. There is a single primary power supply and 48 secondary channels which supply the PMTs. The voltage on the primary supply can be set to an arbitrary value. The secondary channels can only be set to the primary voltage or less. Not all attributes are available on both primary and secondary channels. When an attribute doesn't exist, the GUI will show -1.0.

The A1932 cards have been moved to a CAEN 5527 system.

To troubleshoot EPICs with this hv mainframe trying disable the EPICS service and restart it; and also power cycle the crate. All of this is possible remotely via the web app. Or it can be done manually by hand following the new EPICs instructions in the timing hodoscope how-to document.

Wiener MPOD

UVA GEM modules. Wiener MPOD systems are controlled by SNMP (Simple Network Management Protocol). The standard EPICS [https://groups.nscl.msu.edu/controls/files/devSnmp.html SNMP EPICS driver] is used.

The IOC for the Wiener MPOD system runs on adaqsc. It is started by ProcServMgr.

CAEN VME V6521 cards

CAEN VME V6521 cards are used for the INFN GEM detectors. These cards do not use a custom HV mainframe, just a regular VME crate. The EPICS driver for these cards runs on the Linux Single board computer (intelbbmpd) in the VME crate with the cards. The driver software for these cards is in sbs-epics/apps/caenV65XXApp and the startup script for the IOC is sbs-epics/apps/iocBoot/iocv65xx/st.cmd. Slot number used in the EPICS PVs is not the physical slot but rather the logical slot, where slot 0 is the first card defined, slot 1 is the second card defined etc. The cards are defined with the command

  V65XXInit(Ncards, baseaddr, incaddr)

in st.cmd. For example, V65XXInit(3, 0xa00000, 0x100000) will define "slots" 0-2 with VME addresses 0xa00000, 0xb00000, and 0xc00000.

CAEN 4527-5527

The CAEN 4527 and CAEN5527 are relatively modern high voltage systems. A 5527 is used for the BigBite Hodoscope and 4527's may be used for some SBS experiments such as GeN-RP.

The CAEN 4527 and 5527 have built in EPICS. The EPICS server can be enabled in the mainframe's web interface. The EPICS PV name prefix (e.g. "HAHV") and mainframe number are also set in this interface.

The CAEN 4527 and 5527 EPICS interfaces do not supply separate signal names for setting and readback of setting. For example, the PV HAHV93:00:000:V0Set is used both to set the voltage and to readback the voltage setting. In order provide compatibility with the other EPICS drivers, the Alias4527 app is used. This app mirrors each of writable signal names to provide a "fake" readback signal. The Alias4527 also provides a VDiff signal for each channel for alarms.

A1932 cards do not have some fields for the secondary channels. (Current trip set point, maximum high voltage and current monitoring.) The app is also used to create dummy EPICS records for these fields to keep the GUIs happy.

These mirror signals for the 5527 crate with A1932 cards are configured in the files sbs-epics/apps/Alias4725App/A1932_primary.substitutions and A1932_secondary.substitutions. The crate number is configured in the ioc startup script sbs-epics/iocBoot/iocAlias4527/st.cmd