Latest revision as of 10:23, 6 June 2025

<<SBS Main

The DAQ will not run and I know nothing about CODA

Figure 1. CODA GUI with ROC information printed at the bottom

Before attempting anything, try to reset and start CODA. Whenever you reset, it's a good practice to Initialize GEM VTPs to make sure they function as intended. Follow steps 1 and 3 here to initialize the VTPs. Once completed, go ahead and start CODA by pressing Configure, Download, Pre-start, and Go in that order. If the first attempt fails, try one more time.

What should I do if the second attempt also fails?

• If it is during the work day or you think it will take you more than 15 minutes to troubleshoot this yourself then call the RC immediately instead of following these steps

• First thing is to figure out which DAQ crate is causing the issue.
• Look at the CODA GUI (figure 1), and check the "Severity" information at the bottom:
  • Yellow "Warn" messages are fine.
  • Red or Orange "error" messages must be fixed.
• If there are no errors, scroll up — sometimes they are pushed upward before you notice them!
  • If there are still no obvious errors, call a DAQ expert!!!.
• Check the crate name on the left column and determine which crate it is. The table below lists which subsystem the crates belong to.
• If you want to try fixing the DAQ on your own, go HERE.
  • Do not do this if it is between 8 am and 10 pm and you think this will take you more than 10 minutes. Call the expert instead.
  • If it is nighttime, try for 20 minutes or so before calling the relevant expert.
• Otherwise, look at the table HERE and call the expert matching that subsystem.

List of crates in the DAQ

The passwords for these links are on a wall in the counting house

All network devices at https://hallaweb.jlab.org/wiki/index.php/SBS_Network_Devices_in_Hall_A

Fixing the DAQ yourself

Almost all issues are fixed by restarting the right crate. Look at the table HERE and determine what subsystem matches the crate error.

• If it is a GEM crate go to GEM Crates
• If it is a different crate follow the reset links on the table above.
• If it is a "CODA Platform" issue then the reset must be through CODA. Go to Fixing CODA Platform Issues
• VTP is not booting

Fixing CODA Platform Issues

• On the adaq@adaq2 desktop look for the folder labeled "SBS_coda_scripts" and open it.

• The following options will appear in front of you. First try clicking on "Restart CODA Components"

• If a CODA component continues being in a red "disconnected" state then instead try "Kill CODA Xterms" and then "Start Xterms".
• If things will still not work then try restarting the platform using the instructions below.

 ssh adaq@adaq2
 sudo systemctl restart platform.service

• Use the "Kill CODA Xterms" button and restart all terminals with "Start Xterms"

• If the DAQ still will not run then try this from any hadesk:

 ssh sbs-onl@eel124gemdaq
 kcoda

• This will kill other processes that may be causing an issue
• The click on the "Restart CODA Components" button
• If things are still not working then call the DAQ expert

RC GUI is Frozen

Sometime the RC GUI will be frozen and you cannot press any buttons or close it. If that happens then do the following in the terminal:

xkill

This will turn the mouse pointer into a cross ("X"). Take the pointer on top of the frozen RC GUI and then left click. The GUI should close and then you can open a new GUI using the "CODA RunControl" button without issue.

IN CASE THE ABOVE DOESN'T WORK, TRY THE FOLLOWING:

ps aux | grep ui.rcgui

You will get information in the terminal that looks like the picture below.

Inside the red rectangle is the process ID, which I will call the PID. To kill the GUI then do:

pkill -9 #PID

The GUI should close and then you can open a new GUI using the "CODA RunControl" button without issue.

Resetting sbsvme29ROC1

Open a browser and go to http://sbsvme29crate.jlab.org

Use username and password for a VXS crate

Restarting window manager

type in a terminal in DAQ VNC

 marco --replace &

replace marco by mate-panel if mate is used

GEM Crates

⚠️ Troubleshoot GEM Crates ⚠️

Looks like the problem is related to GEM crates. There are four different GEM crates, and the error you saw in the Run Control GUI should point to one or more of these crates. To fix, we will use the GEM Reset GUI, shown in Figure 01 on the right:

Figure 01: GEM Reset GUI

1. Opening GEM Reset GUI:
   • It should already be open on one of the Counting House computers. If not, you can launch it manually with the following commands:

   ssh a-onl@aonl1  
   gosbs  
   GEM_resets.sh  
       

2. Reset All GEM Crates:
   • Shift workers need to use ONLY the RED buttons on the GEM Reset GUI. DO NOT PRESS anything under "GEM Experts only".
   • Press Reset SBS GEMs once. It will stay pressed for a while — this is normal and intended.
   • It will power-cycle all GEM-related crates and will take at least 5 minutes.
   • While this happens, you'll see the GEM crates disconnecting and rebooting in the Run Control GUI. Wait until they are done.
   • After the GEMs have reset, CODA may still be showing that a ROC is disconnected. In this case, you will need to restart the coda terminals. You can do this, from the GUI called SBS_coda_scripts. Click the script to Kill CODA Xterms and then run the script to Start Xterms. Proceed to Configure the DAQ.
3. Handling VTP Initialization Errors:
   

   Figure 02: VTP Initialization Errors
   • Press init all vtp on the GEM Reset GUI.
   • This will run initialization routines on GEM electronics (vtps).
   • There are four such vtp's: vtp2, vtp3, vtp4 and vtp5.
   • If all the vtp's initialized properly you'll see four green progress bars.
   • In Figure 02, vtp5 (4th VTP) failed to initialize.
   • For single failures (e.g. vtp5): Press init vtpX.
   • If all fail to initialize: Press init all vtp again.
   • If an error does not go away even after multiple initialization and you already spend more than 10 minutes doing so (and have already tried resetting the GEM crates more than twice), CALL A GEM EXPERT!
4. Restart CODA:
   

   Figure 03: Reset CODA RCgui
   • Hopefully, you have successfully initialized all vtp's!
   • Now we can reset CODA and restart it. Press the green Reset button in the Run Control GUI (Figure 03).
   • Also press Reset in the pop-up window to confirm.
   • Click Configure, then Download (in this order).
   • To be on the safe side, press init all vtp once more to verify there are no errors before pressing Prestart.
   • If you still see initialization errors, CALL A GEM EXPERT!
   • If Prestart succeeds, no red errors should appear in the Severity column of the Run Control GUI.
   • Click Go to begin the new run.

⚠️ If the run crashed immediately with GEM related errors on the Severity column of CODA Run Control GUI, CALL A GEM EXPERT! ⚠️

Troubleshooting Missing ECAL/HCAL VTP Channels

If some channels are missing in the ECAL/HCAL Energy vs row or col plots in a 5k/50k replay (see https://logbooks.jlab.org/entry/4389163) (note: in this example, the vtp plots for ecal are showing two issues. One, the large vertical chunk is missing, meaning that either a vxs or vtp crate is not properly communicating/connected. Two, we should not see this roughly horizontal empty band of energy with a densely filled region below, this indicates something is wrong with the cluster thresholds, perhaps the prescale is set incorrectly leading to the coin matching trigger/cluster thresholds not being used properly. Best to just call an ecal expert if this is not fixed by doing the following , vtp plots for ecal should look like this roughly https://logbooks.jlab.org/entry/4391404) then do the following:

1. Stop the run (if still ongoing)
2. Go to HALOG
3. Find the "SBSTS: END of run .." entry for the corresponding run and open the attached .log file.
4. In the log file, search for the string "SSP Port Status Summary". There will be 7 matches. Go to the very last one.
5. "Channel RX" value for the first 8 lines should be around 1400ns. Any channel with a value of 16380 would be bad. Note that slot 20 ports 1 to 7 are the ECAL crates (sbsecal1 to sbsecal7) and port 8 is HCAL.
6. Once the bad crate(s) is identified, power cycle it using the link listed at https://sbs.jlab.org/wiki/index.php/Fixing_the_SBS_DAQ#List_of_crates_in_the_DAQ
7. Reset and restart the CODA following the standard procedure.