Apache Hive hanging while loading data or writing to HDFS

sysadminWe noticed a problem the last couple of days where Hive was hanging while doing a “LOAD DATA” command. This would be a symptom of Hive being unable to write to HDFS in general. So, a “CREATE TABLE AS SELECT” would fail, also.

I did an strace on the hive command:

strace -f hive -e "LOAD DATA ..."

I found that hive was trying to connect to the ZooKeeper server, but that the connections were timing out.

I looked into the ZooKeeper settings to make sure that it was allowed to run enough threads to meet everyone’s needs, and it was.

However, I also found a setting that was limiting the number of connections per IP address:

maxClientCnxns=60

Because this server was also running Hue for all of our users, it was running out of connections allowed. I increased this number significantly, restarted ZooKeeper, and we haven’t had any problems since.