No slot events with --initial-state

When running Teku using --initial-state it seems to load fine but then only logs epoch events. Please note, that there are no logs for slot events (i.e. silence on stdout for several minutes)

13:01:39.371 INFO  - Loading initial state from /var/teku/data/teku-state.ssz
13:01:51.218 INFO  - Loaded initial state at slot 72304 (state root = 0x81ab60d77a872e55ad51c5793481b0fe7e66739f8f55603beddb87c6d535b59f, block root = 0xd4163f729af9c92481dfcebab3ed8769041f1cc4cf053159fee702369743d288)
13:01:51.288 INFO  - Initializing storage
13:01:51.297 INFO  - Storage initialization complete
13:01:56.332 INFO  - Generated new p2p private key and storing in: generated-node-key
13:01:57.716 INFO  - PreGenesis Local ENR: enr:-KG4QEb4EK1pd8x0oAO-6GSNkbii2jPkMBOK2anIVU25-HuoT1l18DDz22HVjVU2XoxsiqI3oufeMiBKiGx8tkZ9rocChGV0aDKQ9aX9QgAAAAD__________4JpZIJ2NIJpcISsEgACiXNlY3AyNTZrMaEDMefY-hI6SP4NWnBo0ZYWUwP7EaeYwc4N1tqI72MUkf6DdGNwgiMog3VkcIIjKA
13:01:58.176 INFO  - Logging initialized @28208ms to org.eclipse.jetty.util.log.Slf4jLog
13:01:58.539 INFO  - Starting libp2p network...
13:01:58.603 INFO  - Listening for connections on: /ip4/172.18.0.2/tcp/9000/p2p/16Uiu2HAmG1oNhgvQnW4RKTr8Cp1X9ercbcLcTjRGiLkaWCGjXAkD
13:01:58.633 INFO  - Starting discovery server on UDP port 9000
13:01:58.666 INFO  - UDP discovery client started
13:01:58.763 INFO  - Begin historical sync of blocks prior to slot 72304
13:01:58.767 INFO  - Starting Javalin ...
13:01:58.972 INFO  - jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 15.0.1+9-18
13:01:59.344 INFO  - Started ServerConnector@35571023{HTTP/1.1, (http/1.1)}{0.0.0.0:5051}
13:01:59.348 INFO  - Started @29381ms
13:01:59.349 INFO  - Listening on http://localhost:5051/
13:01:59.350 INFO  - Javalin started in 591ms \o/
13:01:59.459 INFO  - Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
13:01:59.603 INFO  - Local ENR: enr:-KG4QM1CJnjnGxFB50oRxlCXVrv-1tMBkoser60jxKKr-61AY05p5C8flhUoWt8c0I7d5T3-5yIUeBgJcXx1kfV-8GwDhGV0aDKQtTA_KgAAAAD__________4JpZIJ2NIJpcISsEgACiXNlY3AyNTZrMaEDMefY-hI6SP4NWnBo0ZYWUwP7EaeYwc4N1tqI72MUkf6DdGNwgiMog3VkcIIjKA
13:04:23.443 INFO  - Epoch Event *** Epoch: 2260, Justified checkpoint: 2260, Finalized checkpoint: 2260, Finalized root: d4163f..d288
13:10:47.439 INFO  - Epoch Event *** Epoch: 2261, Justified checkpoint: 2260, Finalized checkpoint: 2260, Finalized root: d4163f..d288
13:10:47.499 INFO  - Attestation performance: expected 0, produced 0, included 0 (0%), distance 0 / 0.00 / 0, correct target 0 (0%), correct head 0 (0%)
13:17:11.437 INFO  - Epoch Event *** Epoch: 2262, Justified checkpoint: 2260, Finalized checkpoint: 2260, Finalized root: d4163f..d288
13:17:11.443 INFO  - Attestation performance: expected 0, produced 0, included 0 (0%), distance 0 / 0.00 / 0, correct target 0 (0%), correct head 0 (0%)
13:23:35.437 INFO  - Epoch Event *** Epoch: 2263, Justified checkpoint: 2260, Finalized checkpoint: 2260, Finalized root: d4163f..d288

Perhaps related to that, I further see …

13:23:39.637 WARN  - Failed to import block from peer (err: UNKNOWN_PARENT) SignedBeaconBlock{message=BeaconBlock{root=0x171f757ebfd748ab7b884821f54e1948d4468be7ad2fcf4444b9cc52f1d88526, slot=72321, proposer_index=9325, parent_root=0x323dd0e75a6c078a468787d2e1e35db6fcda46f41035407d3e707cc7b5465690, state_root=0x29afdb0f27225a2f4bf7bc04b11f462cffaf58538801239d6804a7c10fd54770, body=0xd84ceb100a10398e54ef2ee48144c6cf2e8f7fb554338a8689c7d9672ec8dc3b}, signature=0xab12e91e8ae247866e38773e8ca3037a5a874a42bcded9158219b464db2abdccc69d2330bd6fe8660be351ec7a90fb19160be072bee846b421efa1a3aa07dc06d1998223909bde405667b5435e54e23587f85f778d35c693f17be30ff72c3c1b}: DefaultEth2Peer{id=16Uiu2HAm6fLmHr3MvUHVhRbew7qD7W34CMoCLowakRRAqLVSyhex, remoteStatus=Optional[PeerStatus{forkDigest=0xb5303f2a, finalizedRoot=0xf8e0d085a7e1e94a4e66d44fb90e0b93bb8afea8d6b77f8b031d3aeb2097673b, finalizedEpoch=2261, headRoot=0xae08c48fc7a95c9ce105955b080fe82dc245516a95f4229c6da1999083c8227d, headSlot=72416}]}
13:23:44.874 WARN  - Failed to import block from peer (err: UNKNOWN_PARENT) SignedBeaconBlock{message=BeaconBlock{root=0x171f757ebfd748ab7b884821f54e1948d4468be7ad2fcf4444b9cc52f1d88526, slot=72321, proposer_index=9325, parent_root=0x323dd0e75a6c078a468787d2e1e35db6fcda46f41035407d3e707cc7b5465690, state_root=0x29afdb0f27225a2f4bf7bc04b11f462cffaf58538801239d6804a7c10fd54770, body=0xd84ceb100a10398e54ef2ee48144c6cf2e8f7fb554338a8689c7d9672ec8dc3b}, signature=0xab12e91e8ae247866e38773e8ca3037a5a874a42bcded9158219b464db2abdccc69d2330bd6fe8660be351ec7a90fb19160be072bee846b421efa1a3aa07dc06d1998223909bde405667b5435e54e23587f85f778d35c693f17be30ff72c3c1b}: DefaultEth2Peer{id=16Uiu2HAkzmMizseY9FteUySacPV1u8np4NcKGVvKavh7ZFFhZsce, remoteStatus=Optional[PeerStatus{forkDigest=0xb5303f2a, finalizedRoot=0xf8e0d085a7e1e94a4e66d44fb90e0b93bb8afea8d6b77f8b031d3aeb2097673b, finalizedEpoch=2261, headRoot=0xae08c48fc7a95c9ce105955b080fe82dc245516a95f4229c6da1999083c8227d, headSlot=72416}]}
13:23:44.876 ERROR - Encountered error while processing response
java.util.ConcurrentModificationException: null
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1631) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.IntPipeline.reduce(IntPipeline.java:491) ~[?:?]
        at java.util.stream.IntPipeline.sum(IntPipeline.java:449) ~[?:?]
        at tech.pegasys.teku.networking.eth2.rpc.core.encodings.compression.snappy.SnappyFramedCompressor$SnappyFramedDecompressor.decodeOneMessage(SnappyFramedCompressor.java:72) ~[teku-networking-eth2-20.12.0.jar:20.12.0]
        at tech.pegasys.teku.networking.eth2.rpc.core.encodings.LengthPrefixedPayloadDecoder.decodeOneMessage(LengthPrefixedPayloadDecoder.java:75) ~[teku-networking-eth2-20.12.0.jar:20.12.0]
        at tech.pegasys.teku.networking.eth2.rpc.core.RpcResponseDecoder.decodeNextResponse(RpcResponseDecoder.java:75) ~[teku-networking-eth2-20.12.0.jar:20.12.0]
        at tech.pegasys.teku.networking.eth2.rpc.core.RpcResponseDecoder.decodeNextResponses(RpcResponseDecoder.java:50) ~[teku-networking-eth2-20.12.0.jar:20.12.0]
        at tech.pegasys.teku.networking.eth2.rpc.core.Eth2OutgoingRequestHandler.processData(Eth2OutgoingRequestHandler.java:121) ~[teku-networking-eth2-20.12.0.jar:20.12.0]
        at tech.pegasys.teku.networking.p2p.libp2p.rpc.RpcHandler$Controller.lambda$channelRead0$0(RpcHandler.java:146) ~[teku-networking-p2p-20.12.0.jar:20.12.0]
        at java.util.Optional.ifPresentOrElse(Optional.java:194) [?:?]

[tdiesler]

Yes, the node is up and running in < 2min

GREAT WORK!

2020-12-12 08:06:41,019 main INFO Configuring logging for destination: console
2020-12-12 08:06:41,024 main INFO Logging includes events: true
2020-12-12 08:06:41,026 main INFO Logging includes validator duties: true
2020-12-12 08:06:41,028 main INFO Logging includes color: true
08:06:41.061 INFO  - Teku version: teku/v20.12.0/linux-aarch_64/oracle_openjdk-java-15
08:06:41.069 INFO  - This software is licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
08:06:41.380 INFO  - Loading 0 validator keys...
08:06:41.410 INFO  - Loaded 0 Validators:
08:06:41.594 INFO  - Using default implementation for ThreadExecutor
08:06:41.602 INFO  - Job execution threads will use class loader of thread: main
08:06:41.637 INFO  - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
08:06:41.638 INFO  - Quartz Scheduler v.2.3.2 created.
08:06:41.641 INFO  - RAMJobStore initialized.
08:06:41.644 INFO  - Scheduler meta-data: Quartz Scheduler (v2.3.2) 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
08:06:41.644 INFO  - Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
08:06:41.645 INFO  - Quartz scheduler version: 2.3.2
08:06:41.656 INFO  - Storing beacon chain data in: /var/teku/data/beacon
08:06:41.659 INFO  - Starting metrics http service on 0.0.0.0:8008
08:06:42.295 INFO  - Metrics service started and listening on 8008:8008
08:06:42.307 INFO  - Beacon data directory set to: /var/teku/data/beacon
08:06:43.467 INFO  - Created V5 Hot database (5) at /var/teku/data/beacon/db
08:06:43.467 INFO  - Created V5 Finalized database (5) at /var/teku/data/beacon/archive
08:06:43.620 INFO  - Loading initial state from /var/teku/data/teku-state.ssz
08:06:55.723 INFO  - Loaded initial state at slot 77951 (state root = 0xde71db9fd0774089c54415cf18a3d11778bb5690726b180814c92dbb1269670a, block root = 0x097a8537942e6ea66864824e828ce83928b32a4f7745775eb58bdc805efca1d7)
08:06:55.791 INFO  - Initializing storage
08:06:55.800 INFO  - Storage initialization complete
08:07:01.324 INFO  - Generated new p2p private key and storing in: generated-node-key
08:07:02.936 INFO  - PreGenesis Local ENR: enr:-KG4QALt96SwS3fSyPsaQPKVhVqw5QKcUMMIrcnXEOm_8eC0bo8TAH_GDhdmkG8Dz3D-hk90JoPA2csMV2R_OmZL-LAChGV0aDKQ9aX9QgAAAAD__________4JpZIJ2NIJpcISsEgACiXNlY3AyNTZrMaECBa3Jz1nE9B4hmEr3rgIucxJ6a8Vk4jdeqRWQ3MGOCyKDdGNwgiMog3VkcIIjKA
08:07:03.371 INFO  - Logging initialized @30212ms to org.eclipse.jetty.util.log.Slf4jLog
08:07:03.810 INFO  - Starting libp2p network...
08:07:03.870 INFO  - Listening for connections on: /ip4/172.18.0.2/tcp/9000/p2p/16Uiu2HAkuoqiaoWM2ZcXrbC785HukAfkQp4rcPK9a6oWfTiGgFTX
08:07:03.903 INFO  - Starting discovery server on UDP port 9000
08:07:03.938 INFO  - UDP discovery client started
08:07:04.022 INFO  - Begin historical sync of blocks prior to slot 77951
08:07:04.047 INFO  - Starting Javalin ...
08:07:04.224 INFO  - jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 15.0.1+9-18
08:07:04.594 INFO  - Started ServerConnector@6204d668{HTTP/1.1, (http/1.1)}{0.0.0.0:5051}
08:07:04.596 INFO  - Started @31440ms
08:07:04.598 INFO  - Listening on http://localhost:5051/
08:07:04.600 INFO  - Javalin started in 553ms \o/
08:07:04.644 INFO  - Local ENR: enr:-KG4QJsR_juAxqzCUPi0DL7hAjn_bvgu35-buIHeLKRTNaj8WgS5zsVJJYEAOH4F63Ek9Bw2eRbmhzm9LL592UcmDaoDhGV0aDKQtTA_KgAAAAD__________4JpZIJ2NIJpcISsEgACiXNlY3AyNTZrMaECBa3Jz1nE9B4hmEr3rgIucxJ6a8Vk4jdeqRWQ3MGOCyKDdGNwgiMog3VkcIIjKA
08:07:04.749 INFO  - Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
08:07:04.858 INFO  - Not performing attestation duties for slot 78033 because it is too far ahead of the current slot UNDEFINED
08:07:49.898 INFO  - BLS: loaded BLST library
08:07:51.137 INFO  - Sync Event  *** Current slot: 78037, Head slot: 77953, Connected peers: 10
08:08:01.957 INFO  - Sync Event  *** Current slot: 78038, Head slot: 77985, Connected peers: 9
08:08:14.200 INFO  - Sync Event  *** Current slot: 78039, Head slot: 78024, Connected peers: 8
08:08:17.588 INFO  - Chain reorg at slot 78039 from 825f17..ac75 to 7f29e8..0ab7. Common ancestor at slot 78036
08:08:17.676 INFO  - Chain reorg at slot 78039 from 7f29e8..0ab7 to e214eb..b623. Common ancestor at slot 78037
08:08:17.834 INFO  - Chain reorg at slot 78039 from e214eb..b623 to 8c1acf..ca3d. Common ancestor at slot 78038
08:08:28.335 INFO  - Slot Event  *** Slot: 78040, Block:    ... empty, Epoch: 2438, Finalized checkpoint: 2436, Finalized root: 097a85..a1d7, Peers: 18
08:08:35.367 INFO  - Chain reorg at slot 78040 from 8c1acf..ca3d to aae659..501f. Common ancestor at slot 78039

[mbaxter]

are you saying that this should be

GET /teku/v1/beacon/states/finalized

instead?

Yes, that’s right. You need to pull the “finalized” state rather than the “head” state.

Perhaps, teku could be explicit about this condition

It’s difficult to detect bad initial states in all cases, but there are probably some warnings we could add - I’ll look into it 👍