Client updates report (Incidents)

06 Apr - 08 May

RT Client Replies missed Subject
57976 NFW WARNING: Mail: Exim Panic Log on nfwgs2.nfw.internal
57954 BD Group EFSRV-21761 eFiling high load
57942 Internal Check for Security Notices and System Updates
57851 NFW nfwgs2.nfw.internal unmerged configuration
57857 Internal Check for Security Notices and System Updates
57997 OGT rodney1 reboot/stonith problem
57982 Sucden globalstar01/02.sucden.co.uk failed services
57938 Sucden CRITICAL: HTTP: Uyuni HTTPS on patrs-gs2-p1l01.sucden.co.uk
57849 Internal Check for Security Notices and System Updates
57941 AskBio /var on Talisker at 90%
58082 Midnight Hax woodlands.midnighthax.com high storage volume usage /home
58169 Crystal Units crylonlis2024.crystal.local backups to thames duration
58070 Internal Check for Security Notices and System Updates
57923 Internal Check for Security Notices and System Updates
57919 Imoves 15 Apr, 17 Apr Investigate PHP-related issues with staging site
57958 Internal Check for Security Notices and System Updates
57929 Trident Honda WARNING: Cluster: Pacemaker CRM on vmsvr1.trident.lan
57951 Midnight Hax Fix package double declaration on ws-trixie.midnighthax.com
58166 Internal Check for Security Notices and System Updates
57866 JISC WARNING: Cluster: ClusterCheck on ath-lin-prx-l01.int.openathens.net
58064 OGT 01 May, 06 May rodney.internal.ogtip.com volume usage /srv/ngs
57811 Sucden DOWN: globalstar{01,02}.sucden.co.uk
57870 Cirencester College All systems down
57917 Internal Error when reviewing draft schedules
57883 Koinex WARNING: Mail: Queue Length on koinexserver.home
57924 Trident Honda CRITICAL: Cluster: Pacemaker CRM on vmsvr2.trident.lan
57927 Trident Honda smbfs1.trident.lan root storage volume usage
57934 Internal Check for Security Notices and System Updates
57939 Lantech CRITICAL: HTTP: moodle.ipa.ie HTTPS on moodle.ipa.ie
57867 ITC Compliance Urgent - SSL Root Cert
57945 Internal WARNING: Special: IcingaDB State on towy2.tiger-computing.co.uk
57859 Internal 10 Apr, 15 Apr tcl/avon prometheus job missing
57930 Sucden Storage alert: /srv/uyuni/srv/www on patrs-gs2-p1l01.sucden.co.uk
57850 Sucden apt-show-versions.service failed on interstar05,06
57918 Imoves Investigate PHP-related issues with staging site
57884 Tangram CRITICAL: System Health: Systemd State on dgx1.internal.etherapeutics.co.uk
57876 OGT Storage alert: /srv/department on rodney.internal.ogtip.com
57786 Trident Honda 03 Apr VM hera - /etc/sudoers.d/azure-sudo
57781 OGT 04 Apr basil.internal.ogtip.com volume usage /var
57770 TCV 03 Apr tcv-crm.tcv.org.uk cron php script error
57877 AskBio All systems down
57855 Troika Unable to login to tricky
57863 Tangram [FIRING: 5 for tangram/prod] AWSBackupCopyJobFailedToComplete
57868 Internal Check for Security Notices and System Updates
58081 Internal Check for Security Notices and System Updates
57944 AskBio 23 Apr, 28 Apr ZFS pool data fragmentation on talisker.synpromics.com
57881 Internal Check for Security Notices and System Updates
57937 Quickline Disk critically full on aql-svr-zabbix-db
58113 Novadata 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58172 Crystal Units Fw: can't connect lisec server
57787 Internal 06 Apr Check for Security Notices and System Updates
58071 Midnight Hax CRITICAL: NRPE Disk: All Disks on woodlands.midnighthax.com
58191 DLS Plastics DOWN: dls-zimbra2.dlsmedical.internal
57998 Quickline aql-svr-zabbix-app.quickline.co.uk backups to mersey problem
58089 Internal Check for Security Notices and System Updates
58015 Tangram [FIRING: 1 for tangram/stage] ThanosCompactHalted in tcl-monitoring
58136 Internal Check for Security Notices and System Updates
58083 Lantech 01 May ipa-m3.private.ipa.ie root storage volume usage
58134 Internal Green monitoring check showing as a problem in Icinga
57960 Troika CRITICAL: System Health: Systemd State on tron.troika.lan
57996 Internal Check for Security Notices and System Updates
57959 OGT rodney1.internal.ogtip.com failed disk
58132 Sucden 04 May Linux kernel security update (CVE-2026-31431)
58192 Internal Check for Security Notices and System Updates
58008 Tangram CRITICAL: AWS Site-to-Site VPN: Tunnel2 on dgx1.internal.etherapeutics.co.uk
58117 Reigate College 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58093 AskBio Linux vulnerability: Copy fail (CVE-2026-31431)
57999 AskBio AskBio outage
57961 Reigate College rc-mss.reigate.ac.uk root volume usage
58072 Koinex DOWN: koinexserver.home
58059 NFW DOWN: nfwgs2.nfw.internal
58193 Internal dirtyfrag exploit
57981 Internal Check for Security Notices and System Updates
58095 Azura 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58007 Internal Check for Security Notices and System Updates
58150 AskBio login to RTP server
57963 AskBio WARNING: Security: Let's Encrypt Certificates on darkmatter.synpromics.com
58009 AskBio WARNING: NRPE Disk: / on edb-lab-skye.askbio.com
58085 AskBio talisker.synpromics.com root / volume usage
58096 BD Group 04 May Linux vulnerability: Copy fail (CVE-2026-31431)
58146 Internal Check for Security Notices and System Updates
58019 Top To Toe Re: STILL DOWN: toptotoe-staging.net is still offline ([28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received))
58022 Internal Check for Security Notices and System Updates
58020 Sucden equistar03/04 megastar03/04 NTP Timeouts
57947 Internal Check for Security Notices and System Updates
58111 Midnight Hax 04 May Linux vulnerability: Copy fail (CVE-2026-31431)
57962 Cirencester College 25 Apr ccolive.cirencester.local storage volume usage /var
58205 AskBio Storage alert: /home on talisker.synpromics.com
58075 Gravytrain CRITICAL: HTTP: www.rightsure.co.uk HTTPS on web.prod.bf2.bedfordinsurance.co.uk
58060 Global Brands Printer issue
57993 Tangram [FIRING: 1 for tangram/prod] KubeJobFailed in gitlab
58153 Gravytrain prod.gt1.grvytrn.co.uk certbot renew failure
58086 Internal [FIRING: 1 for tcl/avon] KubeJobFailed in nextcloud
57964 Trident Honda WARNING: Cluster: Csync2 on vmsvr2.trident.lan
58100 Econic Bio 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58154 Globecast gcukscheddbdev and gcukscheddb01 insights client failure
58124 TCV 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
57994 Tangram [FIRING: 1 for tangram/stage] ThanosCompactHalted in tcl-monitoring
58181 Internal Websites down on wey.tiger-computing.co.uk
58187 BD Group eFiling: File Access Error
58000 Tangram [FIRING: 1 for tangram/prod] KubePersistentVolumeFillingUp in tcl-monitoring
58197 TCV web4.tcv.org.uk HTTP/S checks timing out
58104 Globecast 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58140 Midnight Hax 04 May CRITICAL: Disk: fstab mounts on woodlands.midnighthax.com
58127 Trident Honda 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58179 Internal Check for Security Notices and System Updates
58024 Tangram tangram/prod pods in non ready state
57967 Tangram 21 Apr tangram/prod pods in non ready state
58167 Crystal Units Backy2 cleanup failed on crylonlis001.crystal.local
58180 Crystal Units backy2 cleanup failure | invalid database
58160 Sucden strlb-gs2-p1l02.sucden.co.uk failing configuration
58076 AskBio exim4 not starting on coll,jura
58155 Quickline 06 May aql-svr-zabbix-db.quickline.co.uk Zabbix partitioning less than 10 days
58001 Troika WARNING: Puppet: Checking-in on tron.troika.lan
57970 Internal Check for Security Notices and System Updates
58107 ITC Compliance 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58141 Sucden DOWN: patrs-gs2-p1l01.sucden.co.uk
58198 NFW CRITICAL: System Health: Systemd State on nfwgs2.nfw.internal
58161 Crystal Units FW:
58091 Internal 04 May CRITICAL: Backups: Backup to woodlands.midnighthax.com on mersey.tiger-computing.co.uk
58171 Sucden 07 May Storage alert: / on strlb-gs2-p1l01.sucden.co.uk > 90%
58109 Koinex Linux vulnerability: Copy fail (CVE-2026-31431)
57974 Troika UNKNOWN: Hardware: IPMI on tron.troika.lan
57972 Sucden Connectivity Issue from IXN and PPS and GS2 to Azure.
57973 Internal CRITICAL: Backups: Prebackup for mersey.tiger-computing.co.uk on aql-svr-zabbix-app.quickline.co.uk
58006 AskBio Darkmatter ssh oddity
57975 OGT Storage alert: /srv/illumina_data on rodney.internal.ogtip.com
58144 Sucden 04 May CRITICAL: System Health: Systemd State on megastar04.sucden.co.uk
58078 TCV RT autoreply issues
58110 Lantech 01 May Linux vulnerability: Copy fail (CVE-2026-31431)
58157 Crystal Units 06 May crylonlis2024.crystal.local insights client failure
58115 OGT 01 May Linux vulnerability: Copy fail (CVE-2026-31431)

git hash: master (6c029d07)