inRESONANCE Service Updates

Service updates are posted here, published automatically on the iR home page and available for subscription as an RSS feed.

  • SFO and PWS/SBWS Forms failing to generate PDF files (RESOLVED)

    Jul 5 2017 - 3:31pm

    We are investigating an issue where submitted SFO forms are not generating PDF files of the forms. This also delays the retrieval of those forms from the web into your PORTAL/KEYSTONE/SANDBOX solution.  
    We have identified an issue with our third party PDF service and are working with them to resolve the issue.

    UPDATE: [4:41PM EDT] - The core PDF generation service has been restored but there is a backlog of PDF's that still need to get created. We are in the process of clearing this backlog and will provide updates as the situation evolves.

    UPDATE: [9:36AM EDT] - The SFO PDF backlog has been cleared and the service is operating normally.

    UPDATE: [10:20AM EDT] - This issue also affected PDF versions of Applications (in PWS/SBWS) and Recommendations (in PWS). The Recommendation PDF backlog has been cleared. Application PDF's are automatically generated when previewed in PWS/SBWS. The service is now operating normally. 

     

     

     

     

  • Schoolyard Outage (RESOLVED)

    Jun 29 2017 - 6:00am

    Please be advised that we are aware that several Schoolyard websites are down. We are currently investigating the situation and will post updates with more information as it becomes available.

    UPDATE: [8:15AM EDT] - Some sites are recovering.

    UPDATE: [8:48AM EDT] - All sites have recovered.

    UPDATE: [11:00AM EDT] - While we still do not have the specifics on the root cause of the outage this morning, we do know that the problem stems from issues with our upstream infrastructure provider. We are continuing to monitor the situation and will provide more details here as they become available.

  • Some Schoolyard Sites Not Loading (RESOLVED)

    Mar 2 2017 - 7:47am

    There is an issue where some Schoolyard sites are not loading. The issue is isolated to sites with a GoDaddy hosted DNS and appears to be isolated to Europe. We are continuing to monitor the situation and will post back here as information becomes available.

    UPDATE: [9:30AM EST] - This issue is isolated to schools who use GoDaddy as their DNS provider. GoDaddy has acknowledged the issue and is working on a solution. Until then, site visitors located in some parts of Europe attempting to load affected sites may experience slow page loads or no page loads at all. Site visitors located elsewhere in the world are not affected. We are continuing to monitor the situation and will provide updates until resolution.

    UPDATE: [10:18AM EST] - GoDaddy has acknowledged a DNS issue with EU customers and continues to work towards a solution. We are continuing to monitor the situation and will provide updates until resolution.

    UPDATE: [11:40AM EST] - We are starting to see a slowdown in alerts from our site monitors in the EU which may indicate that GoDaddy is making progress on the issue. Though we are confident that this is a GoDaddy issue, we still do not have a root cause. We appreciate your patience and will continue updates until we have a full resolution.

    UPDATE: [1:00PM EST] - GoDaddy continues to work on this issue. No new news on ETA for a permanent fix. We are continuing to monitor the situation and will provide updates until resolution.

    UPDATE: [10:00PM EST] - As of about 6PM EST, the GoDaddy issue was resolved. We are still seeing some residual DNS related issues with sites and will continue to monitor.

    UPDATE: [9:30AM EST - 3-MAR] - This issue is now fully resolved.

  • Service Abnormality: Schoolyard Websites (RESOLVED)

    Feb 28 2017 - 1:58pm

    Due to an AWS S3 outage, several Schoolyard websites have been dropped into "Read from Cache" mode which means that content may not display properly or may be out of date. Users also may have trouble logging in to sites and updating content. We are working actively to resolve the issue and will report back here at 2:30PM with an update.

    UPDATE: [2:30 PM EST] - the AWS S3 outage in the US-EAST-1 data center continues to affect some Schoolyard sites. We will continue to post updates here until the issue is resolved.

    UPDATE: [2:57 PM EST] - We are continuing to monitor the Amazon (AWS) S3 outage that is affecting multiple services across the internet including some Schoolyard sites. Of note, Amazon is reporting that they have identified the root cause and are working on remediation. We will continue to post updates here until the issue is resolved.

    UPDATE: [3:23 PM EST] - There are no new updates from Amazon regarding the S3 outage. We will continue to post updates here until the issue is resolved.

    UPDATE: [4:20 PM EST] - Amazon is reporting progress with restoring S3 services but some functions are still not operational. Many Schoolyard websites are still not rendering properly. We will continue to post updates here until the issue is resolved.

    UPDATE: [5:18 PM EST] - Amazon has reported that their S3 services are fully restored and we are seeing Schoolyard sites resuming normal operation. We are continuing to test and will post updates here until the issue is fully resolved.

    UPDATE: [5:30 PM EST] - We believe that this issue is resolved and will continue to monitor.

  • Service Abnormality: Schoolyard Websites (RESOLVED)

    Nov 11 2016 - 2:21pm

    We are investigating some abnormalities in the hosting environment that may be causing interruptions in service for some SY websites. This page will be updated as we have new information. 

    UPDATE: [9:30 AM EST] This issue has been resolved. 

  • Unexpected Outage: Web Services (RESOLVED)

    Oct 25 2016 - 3:45pm

    At approximately 3:18PM Eastern Time on 25-OCT-2016, the inRESONANCE Web Services cluster went offline. This is affecting Inquiry and Application forms, Recommendations, Events, Notifications, School Forms Online (SFO), Family Access Module (FAM), Teacher Access v2 (TAM2), online Giving Forms, and BillPay. This does not affect Schoolyard websites. Our operations team is aware of the issue and is in the process of diagnosing and resolving the issue. We will post timely updates here as we bring services back online.

    UPDATE: [3:55 PM ET] The outage appears to be related to a networking issue in our data center. We are still working to determine the exact cause and fix the problem as soon as possible. Services are beginning to come back online.

    UPDATE: [4:17 PM ET] The outage is related to a networking issue in our data center. We are still working to determine the exact cause and fix the problem as soon as possible. Services are intermittent.

    UPDATE: [4:43 PM ET] The hardware, software and databases that support our web services cluster are all in tact. The current outage is strictly a services availability problem related to a networking issue in our data center. We are still working to determine the exact cause and fix the problem as soon as possible. Services are intermittent.

    UPDATE: [5:05 PM ET] We are still working to determine the exact cause and fix the problem as soon as possible. Services are intermittent.

    UPDATE: [5:31 PM ET] Services have begun to recover and we are seeing page load times coming back down to normal. Application server throughput is still unusually high as the cluster processes the backlog of requests. We are continuing to monitor the situation and are still looking for a root cause.

    UPDATE: [6:03 PM ET] Web services have again gone offline. We appreciate your patience during this outage as we work towards a resolution.

    UPDATE: [6:26 PM ET] We continue to experience network connectivity and performance issues in our web services cluster data center. These performance issues are preventing clients from connecting to the services hosted there. Our eingineers in the data center are continuing to work to restore services.

    UPDATE: [7:11 PM ET] Services have again been restored yet we still are working towards identifying a root cause and a permanent resolution.

    UPDATE: [7:45 PM ET] Services are performing as expected and we still are working towards identifying a root cause and a permanent resolution.

    UPDATE: [7:57 PM ET] While services have been restored for the moment, the root issue has still not been addressed and we expect that services may become unavailable again until a permanent resolution has been put in place. Our investigation continues and we will keep information coming as we uncover it.

    UPDATE: [8:18 PM ET] Services are still operating normally. Updates may come less frequently here until we have more information.

    UPDATE: [9:12 PM ET] Services continue to be available but are operating at less than peak performance. Application page loads may be longer than normal.

    UPDATE: [9:48 PM ET] Services are operating normally and the situation has been stabilized. Our data center was targeted by a distributed denial of service attack (DDoS) that took sime time to identify and mitigate. Note that this was an attack on the data center itself and not an attack that specifically targeted any inRESONANCE service. We are resuming normal operations but will continue to monitor the performance of our services as we do 24/7/365. If you have any concerns or encounter any performance or availability related problems with your inRESONANCE services, please open a support ticket ([email protected]). We appreciate your patience during this service interruption.

  • BROADCASTER Assets and PORTAL Direct Document Viewer Maintenance

    Aug 5 2016 - 10:15am

    The BROADCASTER Assets and PORTAL Direct Document Viewer services will be shut down for maintenance at 5PM EDT on Friday, August 5th 2016. The maintenance will last approximately 90 minutes. Neither service will be available during the maintenace window.

    UPDATE: as of Friday, August 5, 2016, 6:10PM, all services have been restored. Thanks for your patience.

  • Web Services Scheduled Maintenance March 25

    Mar 15 2016 - 2:29pm

    We have scheduled maintenance on our web cluster for Friday, March 25, 4:30 AM to 10:00 AM EDT. All inRESONANCE web services will be off line during this time. This includes Inquiry and Application forms, Recommendations, Events, Notifications, School Forms Online (SFO), Family Access Module (FAM), online Giving Forms, and BillPay. This does not affect Schoolyard websites.

    Update 9:30am EDT: The maintenance window has been extended to 11:30 AM EDT. Please watch this page for any further updates.

    Update: 11:30am EDT: The maintenance has concluded. All services have been restored.

  • SSO with Blackbaud (Whipple Hill) Service Outage (RESOLVED)

    Feb 10 2016 - 2:13pm

    Blackbaud has reported an issue where SSO authentication services are not operating as expected. As a result, end users may experience difficulties authenticating to inRESONANCE web services. We are actively working with Blackbaud to address the issue and will provide updates as they are available.

    UPDATE: As of 2:17PM ET, this issue is resolved.

  • SFO, PWS, GWS, and FAM Maintenance Window

    Sep 24 2014 - 2:49pm

    On Thursday, September 25th between 3:00 AM and 3:15 AM ET, the server cluster which hosts all instances of SFO, PWS, GWS, and FAM will undergo routine maintenance. During this maintenance, end users may experience unusual behavior including difficulty submitting forms. Users who experience trouble should wait a moment and try their form submission again. We have scheduled this maintenance for a time period which typically has the lowest user load. We appreciate your patience.

Syndicate content