The Ocean Server will be offline briefly after 9pm, Eastern time (6pm Pacific) on May 18, 2018 for a software upgrade. We expect the system to be offline for less that 30 minutes, during which a system maintenance page will appear in place of the Ocean Portal.
- Scheduled Maintenance for May 18, 2018
- Ocean Performance Issues related to Friday's Server Migration (Resolved)
- Email Delivery Issues Relating to Friday's Server Migration (Resolved)
- Scheduled Maintenance for May 4, 2018
- Scheduled Maintenance for April 9, 2018
- TELUS API Connectivity Issues April 9, 2018
- Important Note for Accuro Clinics Using the New Ocean API
- System Interruption Jan 24 10:46-10:50 AM EST
- Ocean Server Release Jan 8th 2018 Deferred to Jan 9th
- Ocean Server Release Reschedule
- Performance Issue with eRequest Inbox Tab: June 27, 2017 10:01 [Resolved]
- System Interruption Feb 28 5:15-5:21 PM EST [Resolved]
- Critical patch release Feb 16 at 12PM EST
- Service Interruption January 25, 2017 10:41-10:45 AM
- Critical patch release January 19 at 1PM EST
- Critical bug in tablet settings management [resolved]
- System Interruption October 4 10:41 EST (2 minutes)
- System Interruption Sept 14 3:43 EST (2 minutes)
- Ocean on Android 6 Advisory
- Ocean Studies Data Issue (RESOLVED)
- Critical system maintenance - Friday, February 19, 2016 at 10pm ET **COMPLETED**
- February 12 2016 4:57-5:05pm: System interruption
- Issue with Emailed Links After Datacentre Migration
- Data Centre Migration Planned for Friday January 29th at 9:00PM ** Completed **
- Ocean Compatibility for PS Suite 5.4
Tablets and Ocean Portal users experienced intermittent performance issues this morning as a result of an incorrect configuration setting on the new server. The configuration error became apparent when server load returned to normal daily volume. Ocean server was up throughout, but some requests would have failed due to timeouts. The issue has been corrected as of 11:15am ET and performance has returned to normal. Please let us know if you continue to experience issues.
We are experiencing some issues with email delivery from our new data centre from Friday night. Emails may fail to be delivered and no error is presented.
In particular, clients using the "from" address in their email settings are affected (under Admin->Site Account), and batch sending may fail on large files due to an unexpected throttling limit from the email service provider. We are working to resolve these issues throughout today. In the meantime, as usual, clinics should rely on Ocean's patient confirmation status to determine whether a message was delivered successfully.
We will post updates to this article as they become available.
May 7 5:30PM ET: We've configured an internal mail server to which we will migrate and are now waiting for our data centre provider to configure some firewall configuration changes to match. We currently expect to migrate to the internal server tomorrow night.
May 8 11:30PM ET: We've updated the Ocean server to use the new internal mail server, akin to our previous configuration at the Toronto data centre. We've tested a number of email send scenarios and it appears to be working, and no error reports have been seen from the system. We believe the email delivery issues described above are resolved and we will close this issue accordingly.
May 9 4:00PM ET: Our logs indicated that our new server was been flagged by ProofPoint.com, which is a spam blacklist. We requested removal.
May 9 4:20PM ET: ProofPoint blacklist removal was completed.
The Ocean Server will be offline briefly after 9pm, Eastern time (6pm-9pm Pacific) on May 4, 2018 for a system migration. We expect the system to be offline for less that 30 minutes, during which a system maintenance page will appear in place of the Ocean Portal.
The production Ocean servers will be migrated from our existing production data centre in Toronto to our disaster recovery (DR) data centre in Montreal. Once migration is complete, our Toronto data centre will be designated as our DR data centre and Montreal will become our production data centre. This migration is being undertaken to increase server resources and provide additional future hosting configuration flexibility. Connected devices will automatically switch over to the new data centre. Please contact firstname.lastname@example.org if any devices fail to move to the new location.
Update 2018-05-04 11:55PM ET: The migration to the Montreal data centre is complete. Service interruption was approximately 10 minutes shortly after 9:00 ET. Data replication has been reconfigured to allow for failover to our Toronto data centre if needed.
The Ocean Server will be offline briefly after business hours on April 9, 2018 for a system upgrade. We expect the system to be offline for less that 5 minutes sometime between 9pm and midnight, Eastern time (6pm-9pm Pacific).
We are seeing errors returned from the TELUS API this morning. TELUS is investigating the issue. TELUS clients that have connected an OceanConnect device to the TELUS API may experience the following:
- Patients not loaded for kiosk use
- Patients not marked as arrived
- Generated Ocean notes not being added to chart
- Ocean Form Reminders delayed
As a temporary workaround, clinics can download clinical notes as required using the "download" link in the toolbar.
Clinics that are not yet connected to the TELUS API will not be affected.
We will update this ticket as more information becomes available from TELUS.
Update 2:55PM from TELUS: "Please note that we continue to experience an issue with the SSH Tunnel service for PS Suite clients. This issue is intermittent and is causing the SSH tunnel connection to drop. This results in a “nTELUS EMR Mobile cannot connect to your EMR Server. Please see the following link for more information.” error message from the API. TELUS has a team currently troubleshooting and have made it a high priority."
Update 5:30PM: TELUS believes they have isolated the issue to a change in the latest PSS version, and a ripple effect of a handful of clinics upgrading Sunday morning that resulted in Ocean connectivity disruptions for other clinics (that were not upgraded). They are going to disconnect the upgraded clinics from the API tonight, apply a PSS patch and monitor the API tomorrow. TELUS is contacting clinics directly that need to be disconnected, so if you haven't heard from them, there is no action required.
Update April 10, 2018 9:45AM: Although we do not yet have confirmation from TELUS, it appears that the maintenance work done by TELUS last night has helped at least some clinics that have reported success with kiosks this morning. Three clinics were disconnected from the API by TELUS; we assume that TELUS has informed those clinics but will be reaching out to confirm. If you are not one of the "disconnected" clinics, please try your kiosks this morning and contact us if you have any issues.
Update April 11, 2018 9:45AM: TELUS has confirmed that the issue was related to a PSS upgrade and patches were applied to affected clinics last night to address the issue. All clinics, including those disconnected from the API yesterday, have been reconnected by TELUS and your Ocean Kiosks, Ocean Online and other Ocean technology that interfaces with a TELUS EMR should be back to normal. Let us know if otherwise.
Important note for Accuro clinics using the new Ocean API with the username "OceanAPI":
QHR has identified an issue with the Ocean API that results from multiple clinics using the same username for the API, causing the authorization to be revoked for clinics in what appears to be a random manner. In the past, usernames for different clinics have been routinely set to "OceanAPI".
QHR is working on a fix, but in the meantime, QHR support can change your username from OceanAPI to a name unique to your clinic.
If your API username is "OceanAPI" and you are on the new API, contact QHR customer support and ask them to change your API username.
We experienced a system interruption between 10:46 and 10:50 AM EST on Jan 24 due to a networking problem. We are working with our hosting provider to determine the root cause of the outage and will update this ticket when more information is available. We apologize for the inconvenience.
Due to a last minute test failure during final testing of the Ocean Server Release scheduled for last night, January 8th, we've rescheduled the deployment for tonight, January 9th at 9:30 EST PM.
The Ocean Server release originally scheduled for Sept 25th 9:00pm has been rescheduled for Sept 26th at 9:00pm.
June 27, 2017 10AM ET: We are investigating a performance issue relating to the eRequest inbox tab relating to last night's release. Users may see delays of around 5s loading the page. We will update this post with more information and an ETA for a fix shortly.
Update 12:30pm: we found two independent sources of slowness (unnecessarily queries for some hidden inbox folders plus an inefficient database query path) that we will fix tonight in a patch. The system will be offline for approximately 5-10 minutes tonight at 9:30pm ET while we add the new indexes.
We are currently experiencing a system interruption on the Ocean application due to excessive system load. We are investigating and will update this ticket shortly as more information is available. We apologize for the inconvenience.
UPDATE 5:24PM: the Ocean system is back online after a system restart. Total outage was six minutes. We will be investigating the root cause based on some captured diagnostic information and will update this ticket.
ROOT CAUSE ANALYSIS 2017-03-02 10PM: After analysis, it appears that the initial source of the excessive load was a series of directory searches that exposed a poorly indexed path. Compounding the problem was some system instrumentation designed to log "performance warnings"; due to the volume of requests that triggered this logging, it caused a knock on effect, forcing the hard restart by CognisantMD operations staff.
We are addressing the issue in three ways:
- We have put in place a measure through a patch release tonight to help protect against such events in the future
- We have some index changes planned for the next major release of the Ocean server
- We are adding protective code to reduce redundant performance warning logging in the event of a system wide slowdown
We will be releasing a critical patch release between 12:00 and 1PM EST today to address an issue that prevented EMR users without Ocean accounts from being able to send messages. We expect the Ocean system to be unavailable for approximately 30s. We apologize for any inconvenience this may cause.
The Ocean system was unavailable from 10:41 to 10:45 AM EST (4 minutes) on January 25th due to an unexpected lock related a database integrity check run manually by the Ocean system operations team. We apologize for any inconvenience it may have caused.
We will be releasing a critical patch release between 12:30 and 1PM EST today to address a couple of critical bugs from last night's release. We expect the Ocean system to be unavailable for approximately 30s. We apologize for any inconvenience this may cause.
We are investigating a bug that affects tablet group setting management introduced as part of last night's release (Dec 13). It will cause favourites layout items and rules to be duplicated in the display. If the user clicks "save", the "doubled" favourites and rules will be saved to the database and may cause the forms to appear twice to the patient.
We are working on a fix and will deploy it tonight and fix any affected settings groups. In the meantime, please be very careful when updating tablet settings groups to ensure you don't save duplicate rules. If you can wait until tomorrow to update tablet settings rules, please do so.
Please let us know if you have any questions or concerns at email@example.com, and we apologize for the inconvenience.
Update (2016-12-14 10:15PM): The fix has been deployed to our production data centre and we ran a script to repair a number of duplicated rules and favourites (we contacted those sites directly to let them know, so if you didn't hear from us, you were not affected). Again, we apologize for the inconvenience.
The Ocean system was offline for approximately 2 minutes this morning at 10:41 EST. During some routine maintenance (storage increase for backups) on our disaster recovery server, the production database server was accidentally rebooted instead of the disaster recovery database server. We apologize for any inconvenience this may have caused.
We experienced a brief service interruption on the Ocean server between 3:43 and 3:45 PM EST today for a duration of just under 2 minutes. Some users may have experienced delays or errors during certain system operations.
We are looking into the root cause and will update as more is known.
With Android 6, Google changed the way that a tablet app can identify its hardware by removing access to the wireless "mac address". Whereas Ocean previously could uniquely identify tablets in this manner, and use it for session management (like a username), this doesn't work in Android 6.
Fortunately, most tablets do not auto-update to Android 6, so the problem occurs only for certain tablets like newer Samsung and Lenovo tablets. In early July, we released a fix in version 125 of the Ocean tablet app that identifies tablets using a "hardware ID" instead. We highly recommend clients upgrade their tablets to v125 to "get ahead" of any potential Android 6 auto-upgrades.
If your tablets are running Android 6 and an Ocean app below v125, there are a few things that you might notice:
- General connectivity issues based on the sessions getting mixed up (if you have 2 or more tablets on Android 6 with Ocean below v125)
- Billing issues: if you notice redundant tablets on your month-end bill, it may be because your tablets changed identifiers and confused Ocean into thinking there were more tablets than in reality.
If you need help with either of the above, please contact firstname.lastname@example.org and we'll help sort things out. You should also review your Tablets tab in the portal to ensure you don't have legacy registrations for tablets on your account.
Finally, in order to get ahead of the issue, next week we'll force upgrade all tablets below v125 to the latest version. If you have a large number of tablets running below v125, you may wish to upgrade them in advance. See here for how to do that.
We are investigating a bug that caused Ocean Studies submissions to be lost retroactively when using older versions of Ocean Tablets. The bug was introduced in Thursday night's Ocean server upgrade as a side-effect of the feature to eliminate redundant submissions to an Ocean Study in the same patient session.
We have fixed the bug and we deployed the fix at 10am this morning. We are in the process of recovering data from backups and replication transaction logs.
At this point, we expect to have full data recovery completed by the end of the day. We apologize for the inconvenience. Please email email@example.com if you have any concerns.
2016-03-23 11:30PM: Issue resolved. We have restored all study submissions and verified against audit logs. Your Ocean Study repositories are back to normal and we do not believe that any data was lost (although some data was unavailable during the day today for export). We apologize for the inconvenience.
A critical vulnerability has been identified in a library used by the operating system running Ocean servers. The vulnerability has been identified by security researchers at Google and Red Hat and a patch has been made generally available. We have applied this patch to all Ocean servers however a reboot is required to complete the patch application. As this reboot will impact Ocean services, we are scheduling it outside business hours. We expect the reboot will take approximately 5 minutes, during which time you may encounter errors while using Ocean tablets or Ocean EMR integrations and the Ocean Portal will be unavailable. If the outage lasts longer than 5 minutes, please contact CognisantMD support.
Technical details on the vulnerability can be found here.
Although this is a critical vulnerability, we have no reason to believe that any Ocean server has been compromised in any way. Please contact support if you have any concerns.
10:40pm: reboot process complete, all servers responding
The Ocean system is currently offline due to a misconfiguration relating to the final step of migrating to our new data centre (changing the DB replication settings). We are in the process of correcting the misconfiguration.
2016-02-12 5:04pm: The system is back online for a total downtime of 7 minutes. Root cause coming shortly.
Root cause analysis:
Background: the Ocean database is replicated in real time from Toronto to Vancouver for redundancy. Since the February 2nd migration to the new Toronto datacentre, we've been running a third replicated database -- our old Toronto data centre -- as an insurance policy.
Cause: as part of the final stage of migration in our new data centre, our old Toronto data center and its replicated database was decommissioned at 5pm today. Unexpectedly, this disrupted the replication set in such a way as no database was elected "primary", leading to a situation in which the application had no active database connection. The replication set was repaired manually and the system was brought back online.
Actions: we will be investigating the underlying misconfiguration that would cause the unavailability of a secondary node to disrupt the primary.
Due to a misconfiguration related to the datacentre migration this past Friday night, links to ocean.cognisantmd.com in generated emails (e.g. web questionnaires, referral notifications, etc) sent since then were incorrectly pointing to "peer1.cognisantmd.com" instead. When clicked these links would cause web browsers to display a warning message regarding an invalid SSL certificate. The links are safe to use but patients may be understandably reluctant to follow them. The problem was corrected at 10:30am ET today (Feb 2 2016). We apologize for any inconvenience.
On Friday January 29th at 9:00pm, we will be migrating to a new data centre. The new data centre is also in Toronto and we believe the system will be more reliable and more responsive in general.
We expect no impact clients and patients except for 5-10 minutes of downtime while the new DNS settings propagate to tablets and web browsers across the Internet.
2016-01-29 10:45: The migration and subsequent testing is complete and Ocean is fully operational. Monitoring software has been repointed at the new data center. If you see a "we have moved" message, simply refresh your browser. Let us know if you see any issues.
We are aware of a few sites that are running PS Suite 5.4 and having issues with PS Suite and Ocean, particularly with concurrent access to charts in PS Suite. We are working with TELUS on updating the custom form for 5.4 and have released a few custom form iterations in an attempt to address the issues.
If you are running Ocean and PS Suite without issue, we recommend that you do not upgrade the Ocean custom form until both 5.4 and the Ocean custom form are more stable.
On the other hand, if you are experiencing issues downloading notes in 5.4 (freezing, checkout errors, etc.) OR have already upgraded to Ocean custom form 2.84 or higher, please contact us at firstname.lastname@example.org.
Update 2015-12-08: We continue to work on the issue and have built a version 3.0 of the custom form for selective distribution that addresses some of the issues and adds some error handling logic. Please contact us if you would like it; it will be made publicly available at the standard download location later this week.
Update 2015-12-11: We have now released version 3.0.1 to the public download site. If you are running PSS 5.3 or 5.4, you should upgrade to this version as soon as possible. If you are running PSS 5.2, you should hold off upgrading the Ocean custom form for the time being.