15 minute update

23 Mar - 24 Apr

RT Client Subject First work started First post-work reply Time to reply
54100 BD Group CRITICAL: HTTP: formations.elementalcosec.com HTTPS on ef-v5-nodes.c.ef-platform.internal 2025-03-10 19:39:35 None 4h 20m
57976 NFW WARNING: Mail: Exim Panic Log on nfwgs2.nfw.internal 2026-04-21 15:48:49 2026-04-21 15:54:40 0h 06m
57744 Geograph Geograph Gitlab upgrade (18.10.1) 2026-03-27 08:16:32 2026-03-27 08:17:31 0h 01m
57769 NFW nfwgs2.nfw.internal samba cleanup script error 2026-04-01 13:00:40 2026-04-01 13:13:30 0h 13m
57671 JISC Upgrade servers to Debian 13 Trixie 2026-03-25 12:03:36 2026-03-25 12:11:22 0h 08m
57954 BD Group EFSRV-21761 eFiling high load 2026-04-20 10:54:30 2026-04-20 11:11:57 0h 17m
57983 Sucden Server Decommissions 2026-04-22 09:17:24 None 1d 4h 43m
57707 Azura Azura - Ocado vpn certificates 2026-03-25 14:35:38 2026-03-25 14:44:30 0h 09m
57745 JISC Access for new employee 2026-03-31 10:51:59 2026-03-31 11:07:22 0h 15m
57851 NFW nfwgs2.nfw.internal unmerged configuration 2026-04-08 08:45:19 2026-04-08 09:02:10 0h 17m
57997 OGT rodney1 reboot/stonith problem 2026-04-23 09:13:55 2026-04-23 09:52:11 0h 38m
57726 Ingeus Ingeus: login to Azure 2026-03-24 08:42:33 2026-03-24 08:44:37 0h 02m
57982 Sucden globalstar01/02.sucden.co.uk failed services 2026-04-22 08:43:13 2026-04-22 08:55:03 0h 12m
57938 Sucden CRITICAL: HTTP: Uyuni HTTPS on patrs-gs2-p1l01.sucden.co.uk 2026-04-16 11:39:45 2026-04-16 11:53:25 0h 14m
57700 Sucden declan and duffman decom 2026-03-19 13:14:00 2026-03-23 15:38:33 2h 25m
57783 Reigate College CRITICAL: Networking: SSH on rc-mss.reigate.ac.uk 2026-04-02 10:11:31 2026-04-02 10:17:20 0h 06m
57782 Gravytrain CRITICAL: HTTP: www.bedfordinsuranceforbusiness.co.uk HTTPS on web.prod.bf2.bedfordinsurance.co.uk 2026-04-02 10:05:06 2026-04-02 10:23:11 0h 18m
57713 OGT rodney1.internal.ogtip.com failed disk 2026-03-23 11:25:58 2026-03-23 12:03:10 0h 37m
57723 OGT OOM - blaney.internal.ogtip.com 2026-03-23 17:10:42 2026-03-23 17:22:15 0h 12m
57735 Tangram CRITICAL: AWS Site-to-Site VPN: Tunnel2 on dgx1.internal.etherapeutics.co.uk 2026-03-24 17:23:38 2026-03-24 17:24:30 0h 01m
57746 Trident Honda VM unifi 2026-03-26 09:50:32 2026-03-26 17:09:40 7h 19m
57722 Sucden iLO maintenance 2026-03-24 10:19:35 2026-03-24 10:20:40 0h 01m
57733 Tangram tangram/prod/gitlab eviction/saml errors 2026-03-25 15:33:57 2026-03-26 13:02:03 2d 1h 28m
57732 Troika Bring up VPN tunnels 2026-03-24 14:48:24 None 9h 12m
57729 Geograph geograph/prod haproxy deployments failing 2026-03-24 16:20:27 None 7h 40m
56919 Ingeus Remove Ingeus systems from support 2026-03-26 15:16:50 2026-03-26 16:12:12 0h 55m
57718 Trident Honda smbfs1.trident.lan volume usage /srv 2026-03-24 13:00:56 2026-03-24 13:02:56 0h 02m
57740 Geograph geograph/prod/gitlab pod status 2026-03-25 08:18:40 2026-03-25 14:39:44 6h 21m
57754 Tangram tangram/prod ThanosCompactIsDown 2026-03-27 09:47:40 2026-03-27 11:06:09 1h 18m
57759 AskBio AskBio hosts down 2026-03-30 13:39:33 2026-03-30 14:39:32 0h 60m
57941 AskBio /var on Talisker at 90% 2026-04-16 10:01:19 2026-04-16 10:09:10 0h 08m
57682 ERF Electrical Upgrade servers to Debian 13 Trixie 2026-03-25 12:42:25 2026-03-25 12:55:33 0h 13m
57715 Tangram tangram/prod issues 2026-03-23 08:48:15 2026-03-23 10:13:57 1h 26m
57771 Novadata purge leonis from everywhere. 2026-03-31 16:38:23 None 7h 22m
57773 OpenCorporates [FIRING:1] DatasourceError Tiger Computing (ElastiCache Connections (openc-frontend-003)) 2026-03-31 10:10:01 2026-04-01 10:22:36 0h 13m
57779 TCV WARNING: NRPE Disk: /var on web4.tcv.org.uk 2026-04-02 10:37:03 2026-04-02 10:38:14 0h 01m
57853 Tangram Expiring SSL certificate: AWS Client VPN 2026-04-10 16:36:27 2026-04-10 16:43:06 0h 07m
57664 JISC OpenAthens server patching 2026-03-26 08:17:58 2026-03-26 08:26:47 0h 09m
57743 Tangram Tangram Gitlab upgrade (18.10.1) 2026-03-26 08:14:05 2026-03-26 08:15:57 0h 02m
57719 Trident Honda VM migration 2026-03-23 16:37:26 2026-03-25 10:46:44 1d 8h 09m
57952 ITC Compliance Timings for system 2026-04-17 14:40:22 2026-04-17 14:47:57 0h 08m
57919 Imoves Investigate PHP-related issues with staging site 2026-04-13 16:45:29 2026-04-14 11:18:51 1d 8h 33m
57933 Sucden interstar07-10 local firewall check (Star Echo Service) 2026-04-15 09:54:42 2026-04-15 09:59:13 0h 05m
57727 Quickline Zabbix Upgrade 2026-03-24 11:36:20 2026-03-24 12:55:38 1h 19m
57929 Trident Honda WARNING: Cluster: Pacemaker CRM on vmsvr1.trident.lan 2026-04-15 09:05:46 2026-04-15 09:07:56 0h 02m
57673 Global Brands Upgrade servers to Debian 13 Trixie 2026-03-25 16:02:32 2026-03-25 16:05:33 0h 03m
57752 Geograph Re: CRITICAL: HTTP: Geograph staging HTTPS on staging.geograph.org.uk 2026-03-27 08:54:54 2026-03-27 09:12:32 0h 18m
57968 NFW Email Forwarding 2026-04-21 15:55:20 2026-04-21 16:10:06 0h 15m
57949 Troika tricky.troika.lan stuck updates 2026-04-17 10:07:10 None 1d 3h 53m
57951 Midnight Hax Fix package double declaration on ws-trixie.midnighthax.com 2026-04-17 14:26:02 2026-04-17 15:11:38 0h 46m
57861 Geograph Geograph Gitlab upgrade (18.10.3) 2026-04-09 10:16:11 2026-04-09 11:49:13 1h 33m
57866 JISC WARNING: Cluster: ClusterCheck on ath-lin-prx-l01.int.openathens.net 2026-04-09 10:57:15 2026-04-09 11:00:01 0h 03m
57810 Troika Independent check: tricky.troika.lan 2026-04-08 12:56:24 None 1d 1h 04m
57870 Cirencester College All systems down 2026-04-10 12:04:30 2026-04-10 12:04:41 0h 00m
57798 Koinex koinexserver.local owncloud/php version 2026-04-10 10:21:21 2026-04-10 10:34:10 0h 13m
57751 Troika Onboard server to support: tricky.troika.lan 2026-03-26 13:05:49 2026-03-26 16:38:59 3h 33m
57883 Koinex WARNING: Mail: Queue Length on koinexserver.home 2026-04-13 08:54:10 2026-04-13 09:18:43 0h 25m
57750 Sucden Relocate interstar03/04 from IXN to GS2 -build out to Ubuntu 2026-03-27 15:55:31 2026-03-27 15:59:38 0h 04m
57927 Trident Honda smbfs1.trident.lan root storage volume usage 2026-04-14 11:33:22 2026-04-14 11:40:31 0h 07m
57864 Sucden Interstar01/02 rebuild to ubuntu 2026-04-13 14:04:56 2026-04-13 17:42:43 3h 38m
57680 Trident Honda Upgrade servers to Debian 13 Trixie 2026-03-25 15:08:07 2026-03-25 15:42:41 0h 35m
57867 ITC Compliance Urgent - SSL Root Cert 2026-04-09 14:21:16 2026-04-09 14:24:51 0h 04m
57611 Crystal Units crylonlis2024 RHEL renewal 2026-04-29 15:04:45 None 8h 55m
57755 Trident Honda VM topaz 2026-03-27 15:03:44 2026-03-27 15:21:56 0h 18m
57930 Sucden Storage alert: /srv/uyuni/srv/www on patrs-gs2-p1l01.sucden.co.uk 2026-04-16 10:55:35 2026-04-16 10:56:16 0h 01m
57850 Sucden apt-show-versions.service failed on interstar05,06 2026-04-08 11:47:51 None 1d 2h 12m
57862 Tangram Tangram Gitlab upgrade (18.10.3) 2026-04-10 08:02:24 2026-04-10 08:36:16 0h 34m
57874 BD Group Install PHP 8.2 on staging server 2026-04-10 14:36:17 None 9h 24m
57884 Tangram CRITICAL: System Health: Systemd State on dgx1.internal.etherapeutics.co.uk 2026-04-13 10:12:44 2026-04-13 10:15:32 0h 03m
57858 Troika tricky.troika.lan monitoring enhancements 2026-04-10 12:45:51 2026-04-13 12:08:27 2d 3h 23m
57921 Trident Honda New access key 2026-04-13 15:48:33 2026-04-13 16:09:08 0h 21m
57876 OGT Storage alert: /srv/department on rodney.internal.ogtip.com 2026-04-10 15:51:23 2026-04-10 15:52:00 0h 01m
57786 Trident Honda VM hera - /etc/sudoers.d/azure-sudo 2026-04-02 17:56:21 2026-04-02 18:03:45 0h 07m
57781 OGT basil.internal.ogtip.com volume usage /var 2026-04-07 11:52:25 2026-04-07 11:57:14 0h 05m
57716 OGT sybil.internal.ogtip.com volume usage /Data 2026-03-25 17:19:23 2026-03-25 17:22:31 0h 03m
57774 Tangram DGX H100 Maintenance – GPU Replacement 2026-03-31 12:42:41 2026-03-31 12:44:28 0h 02m
57770 TCV tcv-crm.tcv.org.uk cron php script error 2026-04-01 15:29:08 2026-04-01 16:03:12 0h 34m
57772 Quickline Zabbix Upgrade (Consultancy) 2026-03-31 09:39:55 None 1d 4h 20m
57629 Sucden Upgrade Uyuni 2026-04-08 15:28:31 2026-04-08 16:30:52 1h 02m
57877 AskBio All systems down 2026-04-13 09:28:30 2026-04-13 10:10:06 0h 42m
57855 Troika Unable to login to tricky 2026-04-08 15:12:45 2026-04-08 15:26:06 0h 13m
57863 Tangram [FIRING: 5 for tangram/prod] AWSBackupCopyJobFailedToComplete 2026-04-09 09:59:15 2026-04-09 11:24:14 1h 25m
57944 AskBio ZFS pool data fragmentation on talisker.synpromics.com 2026-04-16 10:22:56 2026-04-17 10:02:57 2d 3h 40m
57717 AskBio edb-iona.askbio.com HTTP status for prompt-softdev.askbio.com 2026-03-24 15:49:45 2026-03-26 11:23:30 1d 9h 34m
57937 Quickline Disk critically full on aql-svr-zabbix-db 2026-04-09 11:21:17 2026-04-15 07:45:51 2d 0h 25m
57925 AskBio New installs 2026-04-17 11:38:49 2026-04-17 12:05:11 0h 26m
57762 Reigate College rc-mss.reigate.ac.uk disk volume usage 2026-04-02 10:34:16 2026-04-02 10:36:56 0h 03m
57985 JISC BCP questions 2026-04-22 11:39:10 2026-04-22 13:41:43 2h 03m
57936 Crystal Units FW: Shapes drawing with spacer cutting size 2026-04-15 10:51:07 2026-04-15 11:03:48 0h 13m
57998 Quickline aql-svr-zabbix-app.quickline.co.uk backups to mersey problem 2026-04-23 10:16:12 None 1d 3h 44m
57720 Azura UNKNOWN: Hardware: Dell OpenManage on ocado2.opentrk.com 2026-03-23 14:48:29 2026-03-27 16:21:22 1h 33m
57879 Quickline Zabbix database partition error; please verify and fix as needed (workaround currently in place) 2026-04-14 13:08:18 2026-04-14 14:55:43 1h 47m
57714 Gravytrain web.prod.bf2.bedfordinsurance.co.uk HTTP/HTTPS timeout for www.bedfordinsuranceforbusiness.co.uk 2026-03-23 10:31:52 2026-03-23 11:10:36 0h 39m
57960 Troika CRITICAL: System Health: Systemd State on tron.troika.lan 2026-04-20 11:22:46 2026-04-20 12:13:48 0h 51m
57959 OGT rodney1.internal.ogtip.com failed disk 2026-04-20 10:54:09 2026-04-20 11:09:44 0h 16m
58008 Tangram CRITICAL: AWS Site-to-Site VPN: Tunnel2 on dgx1.internal.etherapeutics.co.uk 2026-04-24 09:21:15 2026-04-24 09:46:12 0h 25m
57999 AskBio AskBio outage 2026-04-23 11:26:33 2026-04-23 11:26:38 0h 00m
57961 Reigate College rc-mss.reigate.ac.uk root volume usage 2026-04-20 12:48:07 2026-04-20 13:02:26 0h 14m
57672 Cirencester College Upgrade servers to Debian 13 Trixie 2026-03-25 15:47:14 2026-03-25 15:50:27 0h 03m
57963 AskBio WARNING: Security: Let's Encrypt Certificates on darkmatter.synpromics.com 2026-04-20 10:07:05 2026-04-20 10:23:57 0h 17m
57992 AskBio GITLAB_TOKEN and GITLAB_USERNAME on new RTP server 2026-04-23 14:57:02 2026-04-23 15:10:35 0h 14m
58009 AskBio WARNING: NRPE Disk: / on edb-lab-skye.askbio.com 2026-04-24 12:35:45 2026-04-24 14:52:27 2h 17m
57977 Troika Tron VM NFS mounts 2026-04-21 16:20:53 2026-04-21 16:26:57 0h 06m
57962 Cirencester College ccolive.cirencester.local storage volume usage /var 2026-04-20 13:10:33 2026-04-20 13:34:13 0h 24m
57932 Sucden Readdress globabstr01 and 02 2026-04-16 11:24:08 2026-04-16 11:28:52 0h 05m
57852 Sucden Server Decommission & replacements 2026-04-08 10:56:39 2026-04-08 11:38:25 0h 42m
57993 Tangram [FIRING: 1 for tangram/prod] KubeJobFailed in gitlab 2026-04-23 12:01:47 2026-04-23 12:04:37 0h 03m
57987 OpenCorporates Update a user for access to aws-bot2 server 2026-04-23 12:01:04 2026-04-24 14:49:53 2h 49m
58010 Trident Honda VMs Hera and Dokuwiki 2026-04-24 12:21:56 2026-04-24 12:31:03 0h 09m
57964 Trident Honda WARNING: Cluster: Csync2 on vmsvr2.trident.lan 2026-04-21 11:05:14 2026-04-21 11:13:29 0h 08m
57784 AskBio Configure RTP server for support and functionality 2026-04-10 16:59:03 2026-04-13 11:14:18 1d 8h 15m
57994 Tangram [FIRING: 1 for tangram/stage] ThanosCompactHalted in tcl-monitoring 2026-04-23 12:05:58 2026-04-23 12:12:15 0h 06m
58011 Sucden OLM (svaolmgs201) decommission 2026-04-24 12:32:42 2026-04-30 16:36:55 4h 04m
57988 Troika Tron Rebuild 2026-04-22 14:50:07 2026-04-22 14:52:28 0h 02m
58000 Tangram [FIRING: 1 for tangram/prod] KubePersistentVolumeFillingUp in tcl-monitoring 2026-04-23 12:50:25 2026-04-23 13:30:46 0h 40m
57967 Tangram tangram/prod pods in non ready state 2026-04-20 14:18:59 2026-04-20 14:20:32 0h 02m
58001 Troika WARNING: Puppet: Checking-in on tron.troika.lan 2026-04-23 15:16:13 2026-04-23 15:24:34 0h 08m
57990 Tangram Tangram Gitlab upgrade (18.11) 2026-04-23 15:10:47 2026-04-23 15:11:24 0h 01m
58014 AskBio uv 2026-04-24 15:17:28 2026-04-24 15:28:12 0h 11m
57965 OGT Spare disk for Rodneys 2026-04-20 11:38:28 2026-04-20 11:42:20 0h 04m
57974 Troika UNKNOWN: Hardware: IPMI on tron.troika.lan 2026-04-21 11:40:38 2026-04-21 13:02:52 1h 22m
57972 Sucden Connectivity Issue from IXN and PPS and GS2 to Azure. 2026-04-22 10:04:56 2026-04-22 10:14:34 0h 10m
57971 Sucden Rebuild HK servers 2026-04-21 09:17:29 2026-04-21 09:23:43 0h 06m
58006 AskBio Darkmatter ssh oddity 2026-04-23 17:35:14 2026-04-23 17:41:11 0h 06m
57975 OGT Storage alert: /srv/illumina_data on rodney.internal.ogtip.com 2026-04-21 14:55:20 2026-04-21 14:59:54 0h 05m
58002 Gravytrain Cancel EC2 support for prod.gt1 2026-04-24 17:56:46 2026-04-28 11:56:15 1d 7h 59m
57991 Geograph Geograph Gitlab upgrade (18.11) 2026-04-22 15:37:49 2026-04-22 17:14:34 1h 37m
57763 Sucden interstar05-10.sucden.co.uk IPMI errors 2026-03-30 14:51:22 2026-03-30 15:11:48 0h 20m
57679 Rotastat Upgrade server to Debian 13 Trixie 2026-03-30 12:23:24 2026-03-30 12:49:12 0h 26m

git hash: master (6c029d07)