Client updates report (Incidents)

20 Apr - 22 May

RT Client Replies missed Subject
58226 Internal Check for Security Notices and System Updates
57976 NFW WARNING: Mail: Exim Panic Log on nfwgs2.nfw.internal
58244 TCV web4/mis.tcv.org.uk /boot volume filling up
57954 BD Group EFSRV-21761 eFiling high load
57997 OGT rodney1 reboot/stonith problem
57982 Sucden globalstar01/02.sucden.co.uk failed services
58220 Internal icinga error
58082 Midnight Hax woodlands.midnighthax.com high storage volume usage /home
58169 Crystal Units crylonlis2024.crystal.local backups to thames duration
58070 Internal Check for Security Notices and System Updates
57919 Imoves 17 Apr Investigate PHP-related issues with staging site
57958 Internal Check for Security Notices and System Updates
58166 Internal Check for Security Notices and System Updates
58064 OGT 01 May, 06 May rodney.internal.ogtip.com volume usage /srv/ngs
58317 OGT manuel.internal.ogtip.com restic issues
58308 Geograph geograph/prod node not ready
58081 Internal Check for Security Notices and System Updates
57944 AskBio 23 Apr, 28 Apr ZFS pool data fragmentation on talisker.synpromics.com
58113 Novadata 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58172 Crystal Units Fw: can't connect lisec server
58071 Midnight Hax CRITICAL: NRPE Disk: All Disks on woodlands.midnighthax.com
58305 Internal Check for Security Notices and System Updates
58191 DLS Plastics DOWN: dls-zimbra2.dlsmedical.internal
58307 Cirencester College ccolive.cirencester.local stars HTTP/HTTPS timeouts
58291 AskBio CRITICAL: Networking: SSH on rtp-promptdv-01.askbio.com
57998 Quickline aql-svr-zabbix-app.quickline.co.uk backups to mersey problem
58315 Internal Check for Security Notices and System Updates
58089 Internal Check for Security Notices and System Updates
58214 Midnight Hax fw.midnighthax.com volume usage /boot
58015 Tangram [FIRING: 1 for tangram/stage] ThanosCompactHalted in tcl-monitoring
58136 Internal Check for Security Notices and System Updates
58219 Internal Check for Security Notices and System Updates
58083 Lantech 01 May ipa-m3.private.ipa.ie root storage volume usage
58269 Internal Check for Security Notices and System Updates
58292 Internal DOWN: trent.tiger-computing.co.uk
58134 Internal Green monitoring check showing as a problem in Icinga
57960 Troika CRITICAL: System Health: Systemd State on tron.troika.lan
57996 Internal Check for Security Notices and System Updates
57959 OGT rodney1.internal.ogtip.com failed disk
58132 Sucden 04 May, 08 May Linux kernel security update (CVE-2026-31431)
58192 Internal Check for Security Notices and System Updates
58008 Tangram CRITICAL: AWS Site-to-Site VPN: Tunnel2 on dgx1.internal.etherapeutics.co.uk
58213 OGT CRITICAL: System Health: Log Rotation on basil.internal.ogtip.com
58224 AskBio Cristina
58117 Reigate College 01 May, 14 May Linux vulnerability: Copy fail (CVE-2026-31431)
58093 AskBio Linux vulnerability: Copy fail (CVE-2026-31431)
58293 OGT mickey/granddad.internal.ogtip.com configuration failure
58247 Internal Check for Security Notices and System Updates
57999 AskBio AskBio outage
57961 Reigate College rc-mss.reigate.ac.uk root volume usage
58225 AskBio rtp-promptdv-01 is down
58072 Koinex DOWN: koinexserver.home
58270 Troika tron.troika.lan volume usage /mnt/pve/troika_vm_v2
58059 NFW DOWN: nfwgs2.nfw.internal
58193 Internal dirtyfrag exploit
57981 Internal Check for Security Notices and System Updates
58095 Azura 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58007 Internal Check for Security Notices and System Updates
58282 OGT Restart Docker service on Basil
58251 OGT basil.internal.ogtip.com volume usage /data
58290 Internal Check for Security Notices and System Updates
58150 AskBio login to RTP server
57963 AskBio WARNING: Security: Let's Encrypt Certificates on darkmatter.synpromics.com
58009 AskBio WARNING: NRPE Disk: / on edb-lab-skye.askbio.com
58085 AskBio talisker.synpromics.com root / volume usage
58096 BD Group 04 May Linux vulnerability: Copy fail (CVE-2026-31431)
58146 Internal Check for Security Notices and System Updates
58019 Top To Toe Re: STILL DOWN: toptotoe-staging.net is still offline ([28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received))
58022 Internal Check for Security Notices and System Updates
58020 Sucden equistar03/04 megastar03/04 NTP Timeouts
58111 Midnight Hax 04 May Linux vulnerability: Copy fail (CVE-2026-31431)
57962 Cirencester College 25 Apr ccolive.cirencester.local storage volume usage /var
58217 OGT DOWN: damien.internal.ogtip.com
58284 Internal Check for Security Notices and System Updates
58205 AskBio Storage alert: /home on talisker.synpromics.com
58278 AskBio /tmp cleanup script errors on talisker
58271 NFW nfwgs2.nfw.internal volume usage /home
58075 Gravytrain CRITICAL: HTTP: www.rightsure.co.uk HTTPS on web.prod.bf2.bedfordinsurance.co.uk
58060 Global Brands Printer issue
57993 Tangram [FIRING: 1 for tangram/prod] KubeJobFailed in gitlab
58153 Gravytrain prod.gt1.grvytrn.co.uk certbot renew failure
58086 Internal [FIRING: 1 for tcl/avon] KubeJobFailed in nextcloud
57964 Trident Honda WARNING: Cluster: Csync2 on vmsvr2.trident.lan
58321 NFW DOWN: nfwgs2.nfw.internal
58100 Econic Bio Linux vulnerability: Copy fail (CVE-2026-31431)
58154 Globecast gcukscheddbdev and gcukscheddb01 insights client failure
58124 TCV 01 May, 11 May Linux vulnerability: Copy fail (CVE-2026-31431)
58229 Tangram CRITICAL: System Health: Systemd State on dgx1.internal.etherapeutics.co.uk
57994 Tangram [FIRING: 1 for tangram/stage] ThanosCompactHalted in tcl-monitoring
58273 OGT 19 May Run updates failure on blaney.internal.ogtip.com
58257 OGT blaney.internal.ogtip.com tomcat cron failure
58230 AskBio Backup failure on macleod.synpromics.com
58228 Gravytrain 18 May Certbot failed on app.prod.qs1.grvytrn.co.uk
58285 Troika Unmerged config file on tron.troika.lan
58296 Internal WARNING: Backups: BackupPC directories (NRPE) on mersey.tiger-computing.co.uk
58181 Internal Websites down on wey.tiger-computing.co.uk
58187 BD Group eFiling: File Access Error
58000 Tangram [FIRING: 1 for tangram/prod] KubePersistentVolumeFillingUp in tcl-monitoring
58197 TCV web4.tcv.org.uk HTTP/S checks timing out
58104 Globecast 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58140 Midnight Hax 04 May CRITICAL: Disk: fstab mounts on woodlands.midnighthax.com
58286 Lantech Storage utilisation warning: / on ipa-m3.private.ipa.ie
58127 Trident Honda 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58179 Internal Check for Security Notices and System Updates
58024 Tangram tangram/prod pods in non ready state
57967 Tangram 21 Apr tangram/prod pods in non ready state
58167 Crystal Units Backy2 cleanup failed on crylonlis001.crystal.local
58180 Crystal Units backy2 cleanup failure | invalid database
58160 Sucden strlb-gs2-p1l02.sucden.co.uk failing configuration
58255 Lantech ClamAV warnings this morning (15/05/2026)
58076 AskBio exim4 not starting on coll,jura
58155 Quickline 06 May aql-svr-zabbix-db.quickline.co.uk Zabbix partitioning less than 10 days
58001 Troika WARNING: Puppet: Checking-in on tron.troika.lan
57970 Internal Check for Security Notices and System Updates
58107 ITC Compliance 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58141 Sucden DOWN: patrs-gs2-p1l01.sucden.co.uk
58198 NFW CRITICAL: System Health: Systemd State on nfwgs2.nfw.internal
58161 Crystal Units FW:
58275 TCV Storage alert: /var on web4.tcv.org.uk
58091 Internal 04 May, 12 May CRITICAL: Backups: Backup to woodlands.midnighthax.com on mersey.tiger-computing.co.uk
58287 Crystal Units Storage utilisation warning: /backup on crylonlis2024.crystal.local
58171 Sucden 07 May Storage alert: / on strlb-gs2-p1l01.sucden.co.uk > 90%
58109 Koinex Linux vulnerability: Copy fail (CVE-2026-31431)
58212 Internal Check for Security Notices and System Updates
58276 OGT Storage alert: /srv/illumina_data on rodney.internal.ogtip.com
57974 Troika UNKNOWN: Hardware: IPMI on tron.troika.lan
57972 Sucden Connectivity Issue from IXN and PPS and GS2 to Azure.
57973 Internal CRITICAL: Backups: Prebackup for mersey.tiger-computing.co.uk on aql-svr-zabbix-app.quickline.co.uk
58006 AskBio Darkmatter ssh oddity
57975 OGT Storage alert: /srv/illumina_data on rodney.internal.ogtip.com
58144 Sucden 04 May CRITICAL: System Health: Systemd State on megastar04.sucden.co.uk
58078 TCV RT autoreply issues
58110 Lantech 01 May, 09 May, 15 May Linux vulnerability: Copy fail (CVE-2026-31431)
58157 Crystal Units 06 May crylonlis2024.crystal.local insights client failure
58115 OGT 01 May, 15 May Linux vulnerability: Copy fail (CVE-2026-31431)

git hash: master (6c029d07)