Can't Retrieve Files and Can't Change Storage Deal's State from Cold to Hot

[ChristianOConnor]

I just found out that if you don't enable hot storage and allowUnfreeze in the config, apparently, you can't even retrieve your files?

I ran pow data get <CID> and it gave me this error:

⠋ Retrieving specified data...> Error! Rpc error: code = Unknown desc = cid disabled in hot storage 

So I figured that I had to change the state of my last deal to allow hot storage because, from what I'm gathering, that's the only way to be able to retrieve the files you store with a Filecoin miner. So I made a new config file.

{
  "hot":  {
    "enabled":  true,
    "allowUnfreeze":  true,
    "unfreezeMaxPrice":  "100000000000000000",
    "ipfs":  {
      "addTimeout":  "900"
    }
  },
  "cold":  {
    "enabled":  true,
    "filecoin":  {
      "replicationFactor":  "1",
      "dealMinDuration":  "518400",
      "excludedMiners":  [],
      "trustedMiners":  ["<Miner Id>", "<Miner Id>", "<Miner Id>", "<Miner Id>", "<Miner Id>",],
      "countryCodes":  [],
      "renew":  {
        "enabled":  false,
        "threshold":  "0"
      },
      "address":  "<my filecoin address with 400000000 attoFIL>",
      "maxPrice":  "100000000000000000",
      "fastRetrieval":  true,
      "dealStartOffset":  "8640",
      "verifiedDeal":  false
    }
  },
  "repairable":  false
}

The only things that I changed in the above config file from the original config file was making the "hot" section entries: "enabled" and "allowUnfreeze" true, and I set unfreezeMaxPrice to 0.1 FIL. But whenever I run the new command to update my cold storage deal pow config apply -c custom-073021-hot-config.json -o --watch <CID>, I get this:

                 JOB ID                |             STATUS             | MINER | PRICE | DEAL STATUS
---------------------------------------+--------------------------------+-------+-------+--------------
        <The Job's new CID Here>       | JOB_STATUS_FAILED executing    |       |       |
                                       | enabled hot-storage:           |       |       |
                                       | unfreezing from Cold Storage:  |       |       |
                                       | fetching from deal module: no  |       |       |
                                       | providers to retrieve the data |       |       |

How do I make the cold storage deal I made hot? Or at least allow me to have access to the files lol.

[jsign]

Even if you previously store files with the hot-layer being off, the data should we found in your go-ipfs node. If you're reaching that last error, feels like you ran GC in the go-ipfs node or simply are trying to retrieve some Cid that you never stored.

To double check manually if that Cid is in your go-ipfs node, you can connect to the ipfs container and ipfs cat <cid> and see if you instantly get some output. If that doesn't work, then most probably you never stored the data in this setup or maybe you ran ipfs garbage collection.

Also, this last error also signals that you never did a successful deal with miners with the data since it doesn't know of a miner to retrieve it.

[ChristianOConnor]

I already deleted the data from my ipfs node. I thought the whole point of this app was storing data on Filecoin so you didn't have to store it on your local computer. If you have to store all data on your local computer what's the point of putting it on Filecoin?

[ChristianOConnor]

I feel like I have to be misunderstanding this. If the files are in Cold Storage they have to be accessible somehow. Even if I deleted the files from my IPFS node, the files have to still be accessible. Otherwise there would be no point in putting files in cold storage right? If you need to already have the file to retrieve the files?

[jsign]

I thought the whole point of this app was storing data on Filecoin so you didn't have to store it on your local computer. If you have to store all data on your local computer what's the point of putting it on Filecoin?

Not saying otherwise, that's the point. As I mentioned, the no providers to retrieve the data error signals that Powergate doesn't know about miners storing that Cid. That shouldn't be the case if you store that Cid in Powergate.

If you're trying to retrieve a cid that wasn't stored by that same Powergate instance, you should import some extra information about which DealIDs are storing that data, since Filecoin doesn't have a "Cid discovery" system to know who has which data (natively).

If you want to retrieve some Cid you never stored you have to use: pow config apply with HotStorage: true and --import-deals flag to provide a list of at least one DealID in the network that is storing that data, so you avoid the no providers to retrieve the data problem.

[ChristianOConnor]

Oh awesome! I don't have my PC today but as soon as I get back to it I will try it with the --import-deals with my deal's CID. I will keep this thread updated as soon as I do. Thank you for the help.

[ChristianOConnor]

Okay so separate but very related problem... I did a hot storage deal that I configured perfectly. hot storage was enabled and "allowUnfreeze" was set to true. So I ran the command pow data get <CID> on this new hot storage deal and I got this error.

⠇ Retrieving specified data...> Error! Rpc error: code = Unknown desc = getting cid config: stored item not found

P.S. I'm using the "rootCid", should I be using "proposalCid"?

[jsign]

Should be the same Cid that you used in pow config apply.

[ChristianOConnor]

Okay so the hot storage problem is fixed. I can successfully recover my hot storage file. Idk why it wasn't working before but now I can retrieve hot storage files fine. Thanks for the help with this.

The problem is I still can't change the cold storage file to hot storage. I did exactly what you said. I used the command pow config apply -o <RootCID> -c new-config.json -i <DealId>

My config file looks like this

{
  "hot":  {
    "enabled":  true,
    "allowUnfreeze":  true,
    "unfreezeMaxPrice":  "100000000000000000",
    "ipfs":  {
      "addTimeout":  "900"
    }
  },
  "cold":  {
    "enabled":  true,
    "filecoin":  {
      "replicationFactor":  "2",
      "dealMinDuration":  "518400",
      "excludedMiners":  [],
      "trustedMiners":  ["<minerId>", "<minerId>","<minerId>", "<minerId>", "<minerId>", "<minerId>", "<minerId>", "<minerId>", "<minerId>", "<minerId>"],
      "countryCodes":  [],
      "renew":  {
        "enabled":  false,
        "threshold":  "0"
      },
      "address":  "<my wallet>",
      "maxPrice":  "100000000000000000",
      "fastRetrieval":  true,
      "dealStartOffset":  "8640",
      "verifiedDeal":  false
    }
  },
  "repairable":  false
}

I made this cold storage deal on a different wallet from a since deleted install. Does this pow config apply -o <RootCID> -c new-config.json -i <DealId> command have to come from the same wallet as the original deal? This is my relevant pow storage-job list output.

{
      "id":  "<dealID>",
      "apiId":  "<dealID>",
      "cid":  "<RootCID>",
      "status":  "JOB_STATUS_FAILED",
      "errorCause":  "executing enabled hot-storage: unfreezing from Cold Storage: fetching from deal module: no providers to retrieve the data",
      "dealInfo":  [],
      "dealErrors":  [],
      "createdAt":  "<Number>"
    },

I saved the wallet private key from my original install. Do I have to import the original private key into this install? If so how do I do this?

P.S. the deal ID I'm using is the 7 digit number in the "dealId" section of the output of pow deals storage --include-final that I copied from my original install.

[ChristianOConnor]

Do I need to do the pow config apply command from my original wallet?

Am I using the right deal ID? The 7 digit number I discussed above?

[ChristianOConnor]

I think what's causing this problem is the fact that I am using a new wallet to get an old wallet's deal's data.

Btw I did the command perfectly to move the cold storage deal into hot storage. pow config apply -o <CID> -c new-config.json --import-deals <deal number>

And this is new-config.json

{
  "hot":  {
    "enabled":  true,
    "allowUnfreeze":  true,
    "unfreezeMaxPrice":  "100000000000000000",
    "ipfs":  {
      "addTimeout":  "900"
    }
  },
  "cold":  {
    "enabled":  true,
    "filecoin":  {
      "replicationFactor":  "2",
      "dealMinDuration":  "518400",
      "excludedMiners":  [],
      "trustedMiners":  ["<Miner 1>", "<Miner 2>","<Miner 3>", "<Miner 4>", "<Miner 5>", "<Miner 6>", "<Miner 7>", "<Miner 8>", "<Miner 9>", "<Miner 10>"],
      "countryCodes":  [],
      "renew":  {
        "enabled":  false,
        "threshold":  "0"
      },
      "address":  "<new Filecoin address>",
      "maxPrice":  "100000000000000000",
      "fastRetrieval":  true,
      "dealStartOffset":  "8640",
      "verifiedDeal":  false
    }
  },
  "repairable":  false
}

This is the result of pow data log <CID>

warpanomalythree@DESKTOP-2VS3NLD:~/dev/powergate_config_files$ pow data log <CID>
> 2021-08-05T02:02:43 - Pushing new configuration...
> 2021-08-05T02:02:43 - Configuration saved successfully
> 2021-08-05T02:02:43 - Executing job <Job Id>...
> 2021-08-05T02:02:43 - Executing Hot-Storage configuration...
> 2021-08-05T02:02:43 - Fetching from the IPFS network...
> 2021-08-05T02:17:43 - Direct fetching from IPFS wasn't possible.
> 2021-08-05T02:17:43 - Unfreezing from Filecoin...
> 2021-08-05T02:18:38 - Enabled Hot-Storage excution failed.
> 2021-08-05T02:18:38 - Job <Job Id> execution failed: executing enabled hot-storage: unfreezing from Cold Storage: fetching from deal module: no providers to retrieve the data
> 2021-08-05T02:38:04 - Pushing new configuration...
> 2021-08-05T02:38:04 - Configuration saved successfully
> 2021-08-05T02:38:04 - Executing job <Job Id>...
> 2021-08-05T02:38:04 - Executing Hot-Storage configuration...
> 2021-08-05T02:38:04 - Fetching from the IPFS network...
> 2021-08-05T02:53:04 - Direct fetching from IPFS wasn't possible.
> 2021-08-05T02:53:04 - Unfreezing from Filecoin...
> 2021-08-05T02:53:53 - Enabled Hot-Storage excution failed.
> 2021-08-05T02:53:53 - Job <Job Id> execution failed: executing enabled hot-storage: unfreezing from Cold Storage: fetching from deal module: no providers to retrieve the data
> 2021-08-07T04:41:23 - Pushing new configuration...
> 2021-08-07T04:41:23 - Configuration saved successfully
> 2021-08-07T04:41:23 - Executing job <Job Id>...
> 2021-08-07T04:41:23 - Executing Hot-Storage configuration...
> 2021-08-07T04:41:23 - Fetching from the IPFS network...
> 2021-08-07T04:56:23 - Direct fetching from IPFS wasn't possible.
> 2021-08-07T04:56:23 - Unfreezing from Filecoin...
> 2021-08-07T04:56:23 - Fetching from <Miner Id>...
> 2021-08-07T04:56:23 - Received 0 B, total spent: 0FIL (ClientEventOpen/DealStatusNew)
> 2021-08-07T04:56:23 - Received 0 B, total spent: 0FIL (ClientEventDealProposed/DealStatusWaitForAcceptance)
> 2021-08-07T04:56:23 - Received 0 B, total spent: 0FIL (ClientEventDealRejected/DealStatusRetryLegacy)
> 2021-08-07T04:56:23 - Received 0 B, total spent: 0FIL (ClientEventDealProposed/DealStatusWaitForAcceptanceLegacy)
> 2021-08-07T04:56:24 - Received 0 B, total spent: 0FIL (ClientEventDealRejected/DealStatusRejected)
> 2021-08-07T04:56:24 - Enabled Hot-Storage excution failed.
> 2021-08-07T04:56:24 - Job <<Job Id>> execution failed: executing enabled hot-storage: unfreezing from Cold Storage: event error in retrieval progress: Retrieve: Retrieval Proposal Rejected: deal rejected: sh: 1: /path/to/go/bin/bitscreen: not found

The trouble is that I can't import my old wallet into the docker pow/lotus/ipfs stack. Is there any way to input the old wallet's private key into this new install? I do have the old private key as well as all of the files in the old docker images.

Btw I tried manually importing the old wallet's private key inside of the lotus running docker image but back inside of my host machine, the command pow wallet addrs didn't display this newly imported address.

[jsign]

There's currently no feature to add external wallets into Powergate.

[ChristianOConnor]

There's currently no feature to add external wallets into Powergate.

Is there a way to do this manually? Like are the wallets stored somewhere in a config file in the powergate docker image? Maybe I could add the imported wallet?