Gossip attestation fails when it arrives before the block

[KatyaRyazantseva]

Problem

When a gossip attestation arrives before the block, validation fails with UnknownHeadBlock and the attestation is permanently dropped. Example from the logs:

Feb-24 14:40:03.004 [info] (zeam): [node] received gossip attestation for slot=457 validator=5.{ .name = null } from peer=unknown_peer.{ .name = null }
Feb-24 14:40:03.004 [warning] (zeam): [node] gossip attestation validation failed slot=457 validator=5 error=error.UnknownHeadBlock
Feb-24 14:40:03.013 [info] (zeam): [validator] constructed attestation slot=457 validator=0
Feb-24 14:40:03.013 [info] (zeam): [node] adding locally produced attestation to chain: slot=457 validator=0
Feb-24 14:40:03.016 [info] (zeam): [node] published attestation to network: slot=457 validator=0.{ .name = null }
Feb-24 14:40:03.444 [info] (zeam): [node] received gossip block for slot=457 parent_root=0x36553a38255b306bcda4e4ad004d2bec5bd671386b06e3b2ea6613b9a94a440d proposer=1.{ .name = null } hasParentBlock=true from peer=unknown_peer.{ .name = null }
Feb-24 14:40:03.523 [info] (zeam): [chain] processed block with root=0xc4bb890fbaf77fb8c7f387ef65a4dc476a6400dc77dc420f90579b81ef5f5aac slot=457 processing time=0.07610655 (computed root=false computed state=true)
Feb-24 14:40:03.530 [info] (zeam): [chain] completed on block followup with states_count=20 fc_nodes_count=20
Feb-24 14:40:03.532 [info] (zeam): [node] received gossip attestation for slot=457 validator=1.{ .name = null } from peer=unknown_peer.{ .name = null }
Feb-24 14:40:03.532 [info] (zeam): [chain] processed gossip attestation for slot=457 validator=1.{ .name = null }
Feb-24 14:40:03.532 [info] (zeam): [node] received gossip attestation for slot=457 validator=2.{ .name = null } from peer=unknown_peer.{ .name = null }
Feb-24 14:40:03.533 [info] (zeam): [chain] processed gossip attestation for slot=457 validator=2.{ .name = null }
Feb-24 14:40:03.547 [info] (zeam): [node] received gossip attestation for slot=457 validator=4.{ .name = null } from peer=unknown_peer.{ .name = null }
Feb-24 14:40:03.549 [info] (zeam): [chain] processed gossip attestation for slot=457 validator=4.{ .name = null }

Potential Solution

The attestation should be cached and applied to fork choice once the referenced block arrives. The existing pending_blocks queue solves a similar problem for blocks arriving before the fork choice clock has ticked, the same pattern could be applied here. Expiry should be attestation.slot <= finalized_slot to handle the case where the referenced block never arrives.