alt Hacker News> Uses jq for TB json files
> Hadoop: bro
> Spark: bro
> hive: bro
> data team: bro
Replies
anonymoushn • today at 12:44 PM
are those tools known for their fast json parsers?
> Uses jq for TB json files
> Hadoop: bro
> Spark: bro
> hive: bro
> data team: bro
are those tools known for their fast json parsers?
JQ is very convenient, even if your files are more than 100GB. I often need to extract one field from huge JSON line files, I just pipe jq to it to get results. It's slower, but implementing proper data processing will take more time.