Clickhouse: Import compressed JSON fast
As part of the MVG Observatory Project I collect departures of the Munich public transport. Top level is the date and the subfolder includes multiple stations (with their id as the folder name). Each station includes lots of single files. Every day the top level folder gets archived into a zstd compressed file. To analyse the data, the content has to be imported into Clickhouse. The *_body.json files contain either an error message from the api or a JSON array with multiple responses, which will go into the mvg....