0.17.1rc1 Reindex speed (one drive)

Continuing the discussion from New 0.17.1rc1 client syncs much faster than 0.15.2!:

Full reindex with bitcoingold.conf containing:

[main]
debug=1
assumevalid=0
reindex=1

Start:

2019-09-27T16:53:01Z Bitcoin Gold version v0.17.1rc1 (release build)
2019-09-27T16:53:01Z InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2019-09-27T16:53:01Z Validating signatures for all blocks.

Completion:

2019-09-28T09:30:05Z UpdateTip: new best=00000001380ac6b33ee4a6a251bf41577b6f177d3b970ed85a9f062141b2de9c height=600939 version=0x20000000 log2_work=87.340668 tx=266526218 date='2019-09-28T08:53:30Z' progress=0.635266 cache=554.4MiB(4069226txo)
2019-09-28T09:30:05Z   - Connect postprocess: 1.00ms [113.84s (0.19ms/blk)]
2019-09-28T09:30:05Z - Connect block: 5.00ms [52884.73s (88.00ms/blk)]

Elapsed time: 17 hours 37 minutes.

Much longer to re-index from disk than the fresh sync from network, which took 9 hours.

However, I can also see that my SymLinks from my Hard Drive to my SSD for the index folders are gone; the node did all the work while reading blocks and writing indexes to the same hard drive (a slow 2TB SATA drive, WD Caviar Green WD20EARS.)

This was an unintentional test of index speed on a single drive versus sync from network. The slowdown happens because the drive must read all the blocks from the drive to process them while writing the Index & chainstate files to the same disk. The sync from network, however, never needs to read from disk. It’s getting all the data, fresh, from the network, so it only has to write to the disk (most of the time, assuming the node has enough memory to cache the chainstate data and the UTXOs within.)

Incidentally, this is also why the old “bootstrap” torrent is not useful - unless you can place the bootstrap file on a separate physical disk (or have a high-end, speedy drives), you end up with the same problem of trying to continuously read and write to the same disk. Meanwhile, the well-seeded BTG network can probably hand you the data as quick as your machine can process it. Another bonus: if you use a bootstrap, you must first download the whole bootstrap torrent, and then begin indexing & validating. This is an extra delay compared to just syncing from the network, which begins indexing & validating blocks as soon as it downloads the first ones.

1 Like

I’d see the ‘bootstrap’ advantage, in case of a limited bandwidth.
The ‘bootstrap’ could be downloaded from somewhere else, and then only sync the last 1-2 days, of blocks, from the network, after the blocks from the ‘bootstrap’ torrent, have been processed.

True, but one can also sync the whole chain to a drive from the place with good bandwidth before moving to the place with low bandwidth (example, a laptop).

But you’re right, it’s a valid use case for certain (unusual) scenarios. It’s possible to bring a flash drive with the bootstrap to remote places. OTOH, bootstraps are rarely made. You’ll be syncing many months from the network, not just a few days.

Perhaps our 0.17.1 release is a good time to produce an updated bootstrap.

1 Like

Also, if you want to run a node in a place where internet is metered and you pay for download, it may also make sense to DL a bootstrap from a place DL is free and then take it to the place where you want to run the node which does the validation.

1 Like

I didn’t get a chance to experiment technically with the bootstrap, but I imagined it as a torrent containing all the validated blockchain data, until a certain point in time, compressed in an archive file.
So, as far as I understood, I could install the core wallet, extract the data from the archived file(bootstrap) in the same folder the core wallet was installed, paste my wallet.dat file somewhere in there, and sync the last days of the blockchain, since the bootstrap-snapshot. :grin:
Maybe that’s not how it works, because I read hax has a nasty outlook over the whole bootstrap thingy (How to speed up Bitcoin Gold Core bootstrap - #3 by h4x3rotab). :slight_smile: