{"id":3642,"date":"2024-01-04T17:19:16","date_gmt":"2024-01-05T01:19:16","guid":{"rendered":"https:\/\/jetstreamsoft.com\/portal\/?post_type=ht_kb&#038;p=3642"},"modified":"2024-08-29T10:56:19","modified_gmt":"2024-08-29T18:56:19","slug":"protected-domain-recovery-fails-on-cfo-recovery-domain","status":"publish","type":"ht_kb","link":"https:\/\/jetstreamsoft.com\/portal\/jetstream-knowledge-base\/protected-domain-recovery-fails-on-cfo-recovery-domain\/","title":{"rendered":"Protected Domain Recovery Fails on CFO Recovery Domain"},"content":{"rendered":"\n<p class=\"is-style-info wp-block-ht-blocks-messages wp-block-hb-message wp-block-hb-message--withicon\">This article applies to all product versions.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Problem<\/h2>\n\n\n\n<p class=\"\">JetStream DR at the recovery site may display a warning under the recovery CFO domain indicating \u201c<em>Protected Domain recovery not running.<\/em>\u201d The field &#8220;<em>Data (Processed\/Known Remaining)<\/em>&#8221; appears stuck with no visible change for a prolonged period of time.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img fetchpriority=\"high\" decoding=\"async\" width=\"736\" height=\"394\" src=\"https:\/\/jetstreamsoft.com\/portal\/wp-content\/uploads\/2024\/01\/image-1.png\" alt=\"\" class=\"wp-image-3645\" srcset=\"https:\/\/jetstreamsoft.com\/portal\/wp-content\/uploads\/2024\/01\/image-1.png 736w, https:\/\/jetstreamsoft.com\/portal\/wp-content\/uploads\/2024\/01\/image-1-300x161.png 300w, https:\/\/jetstreamsoft.com\/portal\/wp-content\/uploads\/2024\/01\/image-1-50x27.png 50w\" sizes=\"(max-width: 736px) 100vw, 736px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Troubleshooting<\/h2>\n\n\n\n<p class=\"\">To troubleshoot the problem, try following the below steps:<\/p>\n\n\n\n<ol start=\"1\" class=\"wp-block-list\">\n<li class=\"\">Ensure the DRVA status is \u201c<em>Running<\/em>\u201d from the JetStream DR UI <em><strong>Appliances <\/strong><\/em>tab.\n<ul class=\"wp-block-list\">\n<li class=\"\">If any connection error is reported, check the connection to the object store from the DRVA.<br>&nbsp;<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li class=\"\">Ensure the replication log volume has enough free space for operation.\n<ul class=\"wp-block-list\">\n<li class=\"\"><a href=\"https:\/\/jetstreamsoft.com\/portal\/jetstream-knowledge-base\/resize-a-replication-log-section\/\">Resize the replication log volume, if necessary.<\/a><br>&nbsp;<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li class=\"\"><a href=\"https:\/\/jetstreamsoft.com\/portal\/jetstream-knowledge-base\/recovery-virtual-appliance-rocva\/\">Check the RocVA VM<\/a> to make sure it is up and running with no errors reported.\n<ul class=\"wp-block-list\">\n<li class=\"\">Ensure there are no underlying network issues and validate the VM events.<br>&nbsp;<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li class=\"\"><a href=\"https:\/\/jetstreamsoft.com\/portal\/jetstream-knowledge-base\/collecting-support-logs\/\">Check the MSA logs<\/a> and note if there are any connection issues between the <a href=\"https:\/\/jetstreamsoft.com\/portal\/jetstream-knowledge-base\/recovery-virtual-appliance-rocva\/\">RocVA and DRVA<\/a>. <br>The following example illustrates RocVA reconnect attempts:<br><br><code>Sep 24 07:07:21 drva-live DRVA[1054]: JSS:NETWORKING:accept_connection(190): Accepted socket 35 at 4#10.245.24.81:35364<\/code><br><code>Sep 24 07:07:21 drva-live DRVA[1054]: JSS:REPLICATION:FioValidateAuth(1125): Authenticated socket 35 ctx 0x7fe1d4000df0 group 3cc|800000001 <\/code><br><code>Sep 24 07:07:21 drva-live DRVA[1054]: JSS:PLATFORM:FioReplicationLogOpenSection(1740): invoking connector->openSection, connector = 0x7fe1ac002dc0 <\/code><br><code>Sep 24 07:07:21 drva-live DRVA[1054]: JSS:PLATFORM:FioReplicationLogOpenSection(1749): connector->openSection 0 <\/code><br><code>Sep 24 07:07:21 drva-live DRVA[1054]: JSS:OC:deployOcReplicationChannel(221): Rejected deployment of OC channel for UUID 0000000400000000f6f32fe098c53b00 due to existence of entry 37 with same UUID. <\/code><br><code>Sep 24 07:07:21 drva-live DRVA[1054]: JSS:OC:FioOpenOcChannelForRecovery(1396): Failed to deploy channel 0000000400000000f6f32fe098c53b00 for GC database recovery <\/code><br><code>Sep 24 07:07:21 drva-live DRVA[1054]: JSS:REPLICATION:FioApplianceCmdOpenGCMetadataLog(972): Failed to open OC channel 0000000400000000f6f32fe098c53b00 for recovery<\/code><\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\">Example<\/h2>\n\n\n\n<p class=\"\">The following example illustrates the recovery site MSA initiating cessation of the recovery process and then addressing the event by expanding\/adding the disk of the corresponding RVM, or creating a new RVM for the newly protected VM. After that, it resumes the recovery state of the domain.<\/p>\n\n\n\n<p class=\"is-style-alert wp-block-ht-blocks-messages wp-block-hb-message wp-block-hb-message--withicon\">If a VM is unprotected, be aware the CFO domain may require some time to execute cleanup and then resume the recovery process (e.g., you may need to wait 5, or even 15 to 20 minutes before seeing a change in status).<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>2023-11-02 08:10:15,509  INFO &#91;Thread-31] (CleanupCFOProtectionStoppedVMsTask:179) |taskId=410053f1-7957-11ee-ae35-005056891e25|4-Hours-Domain02| Cleaning up RVM for protection stopped VM. pvmId=542a0901-7259-11ed-951c-005056907ae3, vm=DCWS1004\n2023-11-02 08:10:25,276 DEBUG &#91;Thread-73] (TopologyCollectionTask:4823) Removing objects: VirtualMachine:vm-4450\n2023-11-02 08:10:25,278 DEBUG &#91;Thread-73] (TopologyCollectionTask:4940) Removed vm object: VmKey &#91;vmUuid=50099923-51d7-3446-4544-9f6945874b16, vcenterUuid=738adc85-10b9-49cd-a349-6babb0042823]\n2023-11-02 08:10:25,697  INFO &#91;Thread-54] (TopologyCollectionTask:1809) Removing VM from cache. vmName=jss-rvm-DCWS1004-1464\n2023-11-02 08:10:25,700 DEBUG &#91;Thread-54] (TopologyCollectionTask:1667) Removing the VM from Host->VM cache. VmKey &#91;vmUuid=50099923-51d7-3446-4544-9f6945874b16, vcenterUuid=738adc85-10b9-49cd-a349-6babb0042823]\n2023-11-02 08:10:30,361 DEBUG &#91;Thread-31] (CleanupCFOProtectionStoppedVMsTask:231) |taskId=410053f1-7957-11ee-ae35-005056891e25|4-Hours-Domain02| Stopping recovery on ROCVA. rocvaIp=10.245.24.246\n2023-11-02 08:10:30,925  INFO &#91;Thread-31] (CleanupCFOProtectionStoppedVMsTask:253) |taskId=410053f1-7957-11ee-ae35-005056891e25|4-Hours-Domain02| Task cleanup for CleanupCFOProtectionStoppedVMsTask\n2023-11-02 08:10:34,300 DEBUG &#91;Thread-24] (ResolveRecoveryIssueTask:202) |4c387d62-7957-11ee-ae35-005056891e25|4-Hours-Domain02|initial| Initialize ResolveRecoveryIssueTask\n2023-11-02 08:10:34,303 DEBUG &#91;Thread-24] (ResolveRecoveryIssueTask:211) |4c387d62-7957-11ee-ae35-005056891e25|4-Hours-Domain02|initial| ResolveRecoveryIssueTask is ready now.<\/code><\/pre>\n\n\n\n<p class=\"\">Upon resuming recovery, the MSA log reports:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>2023-11-02 08:14:06,000 INFO &#91;Thread-23] (ResolveRecoveryIssueTask:265) |4c387d62-7957-11ee-ae35-005056891e25|4-Hours-Domain02|checkAndResolveRocvaIssue| Recovery status is already running. No action needed. status=RUNNING<\/code><\/pre>\n\n\n\n<p class=\"is-style-alert wp-block-ht-blocks-messages wp-block-hb-message wp-block-hb-message--withicon\">If the disk of a protected VM is added, expanded or removed, similar behavior as above may be witnessed. No manual intervention is required at the recovery site.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Solution<\/h2>\n\n\n\n<ol start=\"1\" class=\"wp-block-list\">\n<li class=\"\">Resolve any underlying network issues, then re-check the status of the domain.<br>&nbsp;<\/li>\n\n\n\n<li class=\"\">Perform a <strong>hard reboot of the DRVA VM<\/strong> associated with the CFO domain experiencing problems.<br>&nbsp;<\/li>\n\n\n\n<li class=\"\">If the problem persists, <strong>reboot the RocVA VM<\/strong> associated with the CFO domain experiencing problems.<\/li>\n<\/ol>\n\n\n\n<p class=\"is-style-alert wp-block-ht-blocks-messages wp-block-hb-message wp-block-hb-message--withicon\">When performing any actions, wait a few minutes for the status change to be reported by the JetStream DR UI.<\/p>\n\n\n\n<p class=\"\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Problem JetStream DR at the recovery site may display a warning under the recovery CFO domain indicating \u201cProtected Domain recovery not running.\u201d The field &#8220;Data (Processed\/Known Remaining)&#8221; appears stuck with no visible change for a prolonged period of time. Troubleshooting To troubleshoot the problem, try following the below steps: Example The following example illustrates the [&hellip;]<\/p>\n","protected":false},"author":1,"comment_status":"open","ping_status":"closed","template":"","format":"standard","meta":{"nf_dc_page":"","footnotes":""},"ht-kb-category":[92],"ht-kb-tag":[213],"class_list":["post-3642","ht_kb","type-ht_kb","status-publish","format-standard","hentry","ht_kb_category-troubleshooting","ht_kb_tag-protected-domain"],"_links":{"self":[{"href":"https:\/\/jetstreamsoft.com\/portal\/wp-json\/wp\/v2\/ht-kb\/3642","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jetstreamsoft.com\/portal\/wp-json\/wp\/v2\/ht-kb"}],"about":[{"href":"https:\/\/jetstreamsoft.com\/portal\/wp-json\/wp\/v2\/types\/ht_kb"}],"author":[{"embeddable":true,"href":"https:\/\/jetstreamsoft.com\/portal\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jetstreamsoft.com\/portal\/wp-json\/wp\/v2\/comments?post=3642"}],"version-history":[{"count":1,"href":"https:\/\/jetstreamsoft.com\/portal\/wp-json\/wp\/v2\/ht-kb\/3642\/revisions"}],"predecessor-version":[{"id":5559,"href":"https:\/\/jetstreamsoft.com\/portal\/wp-json\/wp\/v2\/ht-kb\/3642\/revisions\/5559"}],"wp:attachment":[{"href":"https:\/\/jetstreamsoft.com\/portal\/wp-json\/wp\/v2\/media?parent=3642"}],"wp:term":[{"taxonomy":"ht_kb_category","embeddable":true,"href":"https:\/\/jetstreamsoft.com\/portal\/wp-json\/wp\/v2\/ht-kb-category?post=3642"},{"taxonomy":"ht_kb_tag","embeddable":true,"href":"https:\/\/jetstreamsoft.com\/portal\/wp-json\/wp\/v2\/ht-kb-tag?post=3642"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}