15 minute update

30 Mar - 01 May

RT Client Subject First work started First post-work reply Time to reply
54100 BD Group CRITICAL: HTTP: formations.elementalcosec.com HTTPS on ef-v5-nodes.c.ef-platform.internal 2025-03-10 19:39:35 None 4h 20m
58066 AskBio Install `uv` on all AskBio systems (consultancy) 2026-04-28 16:02:53 None 7h 57m
57976 NFW WARNING: Mail: Exim Panic Log on nfwgs2.nfw.internal 2026-04-21 15:48:49 2026-04-21 15:54:40 0h 06m
57769 NFW nfwgs2.nfw.internal samba cleanup script error 2026-04-01 13:00:40 2026-04-01 13:13:30 0h 13m
57954 BD Group EFSRV-21761 eFiling high load 2026-04-20 10:54:30 2026-04-20 11:11:57 0h 17m
57983 Sucden Server Decommissions 2026-04-22 09:17:24 None 1d 4h 43m
57745 JISC Access for new employee 2026-03-31 10:51:59 2026-03-31 11:07:22 0h 15m
57851 NFW nfwgs2.nfw.internal unmerged configuration 2026-04-08 08:45:19 2026-04-08 09:02:10 0h 17m
57997 OGT rodney1 reboot/stonith problem 2026-04-23 09:13:55 2026-04-23 09:52:11 0h 38m
57982 Sucden globalstar01/02.sucden.co.uk failed services 2026-04-22 08:43:13 2026-04-22 08:55:03 0h 12m
57938 Sucden CRITICAL: HTTP: Uyuni HTTPS on patrs-gs2-p1l01.sucden.co.uk 2026-04-16 11:39:45 2026-04-16 11:53:25 0h 14m
58080 TCV TCV-WEB4 PHP Max Input 2026-04-28 16:49:42 2026-04-28 16:51:11 0h 01m
57783 Reigate College CRITICAL: Networking: SSH on rc-mss.reigate.ac.uk 2026-04-02 10:11:31 2026-04-02 10:17:20 0h 06m
57782 Gravytrain CRITICAL: HTTP: www.bedfordinsuranceforbusiness.co.uk HTTPS on web.prod.bf2.bedfordinsurance.co.uk 2026-04-02 10:05:06 2026-04-02 10:23:11 0h 18m
57759 AskBio AskBio hosts down 2026-03-30 13:39:33 2026-03-30 14:39:32 0h 60m
57941 AskBio /var on Talisker at 90% 2026-04-16 10:01:19 2026-04-16 10:09:10 0h 08m
57771 Novadata purge leonis from everywhere. 2026-03-31 16:38:23 None 7h 22m
57773 OpenCorporates [FIRING:1] DatasourceError Tiger Computing (ElastiCache Connections (openc-frontend-003)) 2026-03-31 10:10:01 2026-04-01 10:22:36 0h 13m
57779 TCV WARNING: NRPE Disk: /var on web4.tcv.org.uk 2026-04-02 10:37:03 2026-04-02 10:38:14 0h 01m
57853 Tangram Expiring SSL certificate: AWS Client VPN 2026-04-10 16:36:27 2026-04-10 16:43:06 0h 07m
57952 ITC Compliance Timings for system 2026-04-17 14:40:22 2026-04-17 14:47:57 0h 08m
57919 Imoves Investigate PHP-related issues with staging site 2026-04-13 16:45:29 2026-04-14 11:18:51 1d 8h 33m
57933 Sucden interstar07-10 local firewall check (Star Echo Service) 2026-04-15 09:54:42 2026-04-15 09:59:13 0h 05m
57929 Trident Honda WARNING: Cluster: Pacemaker CRM on vmsvr1.trident.lan 2026-04-15 09:05:46 2026-04-15 09:07:56 0h 02m
57968 NFW Email Forwarding 2026-04-21 15:55:20 2026-04-21 16:10:06 0h 15m
57949 Troika tricky.troika.lan stuck updates 2026-04-17 10:07:10 None 1d 3h 53m
57951 Midnight Hax Fix package double declaration on ws-trixie.midnighthax.com 2026-04-17 14:26:02 2026-04-17 15:11:38 0h 46m
57861 Geograph Geograph Gitlab upgrade (18.10.3) 2026-04-09 10:16:11 2026-04-09 11:49:13 1h 33m
57866 JISC WARNING: Cluster: ClusterCheck on ath-lin-prx-l01.int.openathens.net 2026-04-09 10:57:15 2026-04-09 11:00:01 0h 03m
57810 Troika Independent check: tricky.troika.lan 2026-04-08 12:56:24 None 1d 1h 04m
57870 Cirencester College All systems down 2026-04-10 12:04:30 2026-04-10 12:04:41 0h 00m
57798 Koinex koinexserver.local owncloud/php version 2026-04-10 10:21:21 2026-04-10 10:34:10 0h 13m
57883 Koinex WARNING: Mail: Queue Length on koinexserver.home 2026-04-13 08:54:10 2026-04-13 09:18:43 0h 25m
57927 Trident Honda smbfs1.trident.lan root storage volume usage 2026-04-14 11:33:22 2026-04-14 11:40:31 0h 07m
57864 Sucden Interstar01/02 rebuild to ubuntu 2026-04-13 14:04:56 2026-04-13 17:42:43 3h 38m
57867 ITC Compliance Urgent - SSL Root Cert 2026-04-09 14:21:16 2026-04-09 14:24:51 0h 04m
57611 Crystal Units crylonlis2024 RHEL renewal 2026-04-29 15:04:45 None 8h 55m
57930 Sucden Storage alert: /srv/uyuni/srv/www on patrs-gs2-p1l01.sucden.co.uk 2026-04-16 10:55:35 2026-04-16 10:56:16 0h 01m
57850 Sucden apt-show-versions.service failed on interstar05,06 2026-04-08 11:47:51 None 1d 2h 12m
57862 Tangram Tangram Gitlab upgrade (18.10.3) 2026-04-10 08:02:24 2026-04-10 08:36:16 0h 34m
57874 BD Group Install PHP 8.2 on staging server 2026-04-10 14:36:17 None 9h 24m
57884 Tangram CRITICAL: System Health: Systemd State on dgx1.internal.etherapeutics.co.uk 2026-04-13 10:12:44 2026-04-13 10:15:32 0h 03m
57858 Troika tricky.troika.lan monitoring enhancements 2026-04-10 12:45:51 2026-04-13 12:08:27 2d 3h 23m
57921 Trident Honda New access key 2026-04-13 15:48:33 2026-04-13 16:09:08 0h 21m
57876 OGT Storage alert: /srv/department on rodney.internal.ogtip.com 2026-04-10 15:51:23 2026-04-10 15:52:00 0h 01m
57786 Trident Honda VM hera - /etc/sudoers.d/azure-sudo 2026-04-02 17:56:21 2026-04-02 18:03:45 0h 07m
57781 OGT basil.internal.ogtip.com volume usage /var 2026-04-07 11:52:25 2026-04-07 11:57:14 0h 05m
57774 Tangram DGX H100 Maintenance – GPU Replacement 2026-03-31 12:42:41 2026-03-31 12:44:28 0h 02m
57770 TCV tcv-crm.tcv.org.uk cron php script error 2026-04-01 15:29:08 2026-04-01 16:03:12 0h 34m
57772 Quickline Zabbix Upgrade (Consultancy) 2026-03-31 09:39:55 None 1d 4h 20m
57629 Sucden Upgrade Uyuni 2026-04-08 15:28:31 2026-04-08 16:30:52 1h 02m
57877 AskBio All systems down 2026-04-13 09:28:30 2026-04-13 10:10:06 0h 42m
57855 Troika Unable to login to tricky 2026-04-08 15:12:45 2026-04-08 15:26:06 0h 13m
57863 Tangram [FIRING: 5 for tangram/prod] AWSBackupCopyJobFailedToComplete 2026-04-09 09:59:15 2026-04-09 11:24:14 1h 25m
57944 AskBio ZFS pool data fragmentation on talisker.synpromics.com 2026-04-16 10:22:56 2026-04-17 10:02:57 2d 3h 40m
57937 Quickline Disk critically full on aql-svr-zabbix-db 2026-04-09 11:21:17 2026-04-15 07:45:51 2d 0h 25m
57925 AskBio New installs 2026-04-17 11:38:49 2026-04-17 12:05:11 0h 26m
57762 Reigate College rc-mss.reigate.ac.uk disk volume usage 2026-04-02 10:34:16 2026-04-02 10:36:56 0h 03m
57985 JISC BCP questions 2026-04-22 11:39:10 2026-04-22 13:41:43 2h 03m
57936 Crystal Units FW: Shapes drawing with spacer cutting size 2026-04-15 10:51:07 2026-04-15 11:03:48 0h 13m
57998 Quickline aql-svr-zabbix-app.quickline.co.uk backups to mersey problem 2026-04-23 10:16:12 None 1d 3h 44m
57879 Quickline Zabbix database partition error; please verify and fix as needed (workaround currently in place) 2026-04-14 13:08:18 2026-04-14 14:55:43 1h 47m
58015 Tangram [FIRING: 1 for tangram/stage] ThanosCompactHalted in tcl-monitoring 2026-04-27 10:21:15 2026-04-27 17:14:24 6h 53m
57960 Troika CRITICAL: System Health: Systemd State on tron.troika.lan 2026-04-20 11:22:46 2026-04-20 12:13:48 0h 51m
57959 OGT rodney1.internal.ogtip.com failed disk 2026-04-20 10:54:09 2026-04-20 11:09:44 0h 16m
58132 Sucden Linux kernel security update (CVE-2026-31431) 2026-05-01 13:24:02 2026-05-01 13:28:53 0h 05m
58008 Tangram CRITICAL: AWS Site-to-Site VPN: Tunnel2 on dgx1.internal.etherapeutics.co.uk 2026-04-24 09:21:15 2026-04-24 09:46:12 0h 25m
58137 ERF Electrical FW: Infor Actions Required: Impending NAT IP Address Changes 2026-05-01 10:01:42 2026-05-01 10:42:44 0h 41m
58093 AskBio Linux vulnerability: Copy fail (CVE-2026-31431) 2026-05-01 10:42:38 2026-05-01 10:44:03 0h 01m
57999 AskBio AskBio outage 2026-04-23 11:26:33 2026-04-23 11:26:38 0h 00m
57961 Reigate College rc-mss.reigate.ac.uk root volume usage 2026-04-20 12:48:07 2026-04-20 13:02:26 0h 14m
58072 Koinex DOWN: koinexserver.home 2026-04-28 09:26:23 2026-04-28 09:52:38 0h 26m
58079 AskBio pyenv on upgraded servers 2026-04-29 11:13:19 2026-04-29 11:25:07 0h 12m
58133 ATA Group [-External-] Linux kernel security update (CVE-2026-31431) 2026-04-30 11:51:01 2026-04-30 11:56:56 0h 06m
58074 Crystal Units crylonlis2024 reboot 2026-04-28 09:01:17 2026-04-28 09:10:51 0h 10m
57963 AskBio WARNING: Security: Let's Encrypt Certificates on darkmatter.synpromics.com 2026-04-20 10:07:05 2026-04-20 10:23:57 0h 17m
57992 AskBio GITLAB_TOKEN and GITLAB_USERNAME on new RTP server 2026-04-23 14:57:02 2026-04-23 15:10:35 0h 14m
58009 AskBio WARNING: NRPE Disk: / on edb-lab-skye.askbio.com 2026-04-24 12:35:45 2026-04-24 14:52:27 2h 17m
58085 AskBio talisker.synpromics.com root / volume usage 2026-04-29 14:53:10 None 9h 07m
58096 BD Group Linux vulnerability: Copy fail (CVE-2026-31431) 2026-05-01 15:15:28 2026-05-01 15:24:36 0h 09m
57977 Troika Tron VM NFS mounts 2026-04-21 16:20:53 2026-04-21 16:26:57 0h 06m
58019 Top To Toe Re: STILL DOWN: toptotoe-staging.net is still offline ([28] Timeout was reached (Operation timed out after 30001 milliseconds with 0 bytes received)) 2026-04-27 10:30:07 2026-04-27 17:00:47 6h 31m
58020 Sucden equistar03/04 megastar03/04 NTP Timeouts 2026-04-27 09:46:16 2026-04-27 09:47:53 0h 02m
57962 Cirencester College ccolive.cirencester.local storage volume usage /var 2026-04-20 13:10:33 2026-04-20 13:34:13 0h 24m
58075 Gravytrain CRITICAL: HTTP: www.rightsure.co.uk HTTPS on web.prod.bf2.bedfordinsurance.co.uk 2026-04-28 10:26:59 2026-04-28 10:43:02 0h 16m
58060 Global Brands Printer issue 2026-04-27 12:52:03 2026-04-27 12:59:50 0h 08m
58068 BD Group Update .htaccess config. 2026-04-29 13:14:51 2026-04-29 13:35:23 0h 21m
57932 Sucden Readdress globabstr01 and 02 2026-04-16 11:24:08 2026-04-16 11:28:52 0h 05m
57852 Sucden Server Decommission & replacements 2026-04-08 10:56:39 2026-04-08 11:38:25 0h 42m
57993 Tangram [FIRING: 1 for tangram/prod] KubeJobFailed in gitlab 2026-04-23 12:01:47 2026-04-23 12:04:37 0h 03m
57987 OpenCorporates Update a user for access to aws-bot2 server 2026-04-23 12:01:04 2026-04-24 14:49:53 2h 49m
58010 Trident Honda VMs Hera and Dokuwiki 2026-04-24 12:21:56 2026-04-24 12:31:03 0h 09m
57964 Trident Honda WARNING: Cluster: Csync2 on vmsvr2.trident.lan 2026-04-21 11:05:14 2026-04-21 11:13:29 0h 08m
58023 Quickline aql-svr-zabbix-app.quickline.co.uk backups to mersey problem 2026-04-27 11:32:19 2026-04-27 13:31:14 1h 59m
58100 Econic Bio Linux vulnerability: Copy fail (CVE-2026-31431) 2026-05-15 09:22:55 None 1d 4h 37m
58061 NFW DOWN: nfwgs2.nfw.internal 2026-04-27 13:46:40 2026-04-27 13:56:40 0h 10m
57784 AskBio Configure RTP server for support and functionality 2026-04-10 16:59:03 2026-04-13 11:14:18 1d 8h 15m
57994 Tangram [FIRING: 1 for tangram/stage] ThanosCompactHalted in tcl-monitoring 2026-04-23 12:05:58 2026-04-23 12:12:15 0h 06m
58011 Sucden OLM (svaolmgs201) decommission 2026-04-24 12:32:42 2026-04-30 16:36:55 4h 04m
57988 Troika Tron Rebuild 2026-04-22 14:50:07 2026-04-22 14:52:28 0h 02m
58062 AskBio python-is-python3 2026-04-27 15:36:45 2026-04-27 15:52:28 0h 16m
58000 Tangram [FIRING: 1 for tangram/prod] KubePersistentVolumeFillingUp in tcl-monitoring 2026-04-23 12:50:25 2026-04-23 13:30:46 0h 40m
58140 Midnight Hax CRITICAL: Disk: fstab mounts on woodlands.midnighthax.com 2026-05-01 16:20:28 2026-05-01 16:27:52 0h 07m
58106 Imoves Linux vulnerability: Copy fail (CVE-2026-31431) 2026-05-01 14:43:48 2026-05-05 17:13:02 2h 29m
58024 Tangram tangram/prod pods in non ready state 2026-04-27 10:10:53 2026-04-27 15:31:07 5h 20m
57967 Tangram tangram/prod pods in non ready state 2026-04-20 14:18:59 2026-04-20 14:20:32 0h 02m
58088 OGT Upgrade blaney.internal.ogtip.com to Debian 13 Trixie 2026-05-14 15:15:00 None 8h 45m
58097 Cirencester College Linux vulnerability: Copy fail (CVE-2026-31431) 2026-05-01 15:48:51 2026-05-01 16:31:01 0h 42m
58076 AskBio exim4 not starting on coll,jura 2026-04-28 14:49:46 2026-04-28 14:57:06 0h 07m
58001 Troika WARNING: Puppet: Checking-in on tron.troika.lan 2026-04-23 15:16:13 2026-04-23 15:24:34 0h 08m
58141 Sucden DOWN: patrs-gs2-p1l01.sucden.co.uk 2026-05-01 16:23:46 2026-05-01 17:08:52 0h 45m
57990 Tangram Tangram Gitlab upgrade (18.11) 2026-04-23 15:10:47 2026-04-23 15:11:24 0h 01m
58014 AskBio uv 2026-04-24 15:17:28 2026-04-24 15:28:12 0h 11m
58109 Koinex Linux vulnerability: Copy fail (CVE-2026-31431) 2026-05-01 12:41:41 2026-05-01 12:58:54 0h 17m
57965 OGT Spare disk for Rodneys 2026-04-20 11:38:28 2026-04-20 11:42:20 0h 04m
57974 Troika UNKNOWN: Hardware: IPMI on tron.troika.lan 2026-04-21 11:40:38 2026-04-21 13:02:52 1h 22m
57972 Sucden Connectivity Issue from IXN and PPS and GS2 to Azure. 2026-04-22 10:04:56 2026-04-22 10:14:34 0h 10m
57971 Sucden Rebuild HK servers 2026-04-21 09:17:29 2026-04-21 09:23:43 0h 06m
58006 AskBio Darkmatter ssh oddity 2026-04-23 17:35:14 2026-04-23 17:41:11 0h 06m
57975 OGT Storage alert: /srv/illumina_data on rodney.internal.ogtip.com 2026-04-21 14:55:20 2026-04-21 14:59:54 0h 05m
58002 Gravytrain Cancel EC2 support for prod.gt1 2026-04-24 17:56:46 2026-04-28 11:56:15 1d 7h 59m
57991 Geograph Geograph Gitlab upgrade (18.11) 2026-04-22 15:37:49 2026-04-22 17:14:34 1h 37m
58078 TCV RT autoreply issues 2026-04-28 14:54:06 2026-04-28 15:53:13 0h 59m
57763 Sucden interstar05-10.sucden.co.uk IPMI errors 2026-03-30 14:51:22 2026-03-30 15:11:48 0h 20m
58139 Sucden Create @sucfin.com accounts for TCL 2026-05-01 16:13:24 2026-05-01 16:18:16 0h 05m
57679 Rotastat Upgrade server to Debian 13 Trixie 2026-03-30 12:23:24 2026-03-30 12:49:12 0h 26m

git hash: master (f521b466)