with ptb_word_lm.py, there is a bug dealing with cell state for the case when there is more than one hidden layer.

initialState is like state which is a return value of cell(...)
it has nLayers "LSTMStateTuple (c, h)"s: that is,
len(state) = nLayers, len(state[i]) = 2
c and h are tensors of shape [batchSz, hiddenSz], respectively
c and h are also aliases of 0 and 1, respectively
don't confuse c and h as aliases with c and h of (c, h)
feed_dict[c] = state[i].c # state[i].c indicates c of (c, h)
feed_dict[h] = state[i].h

print("feed_dict[c].shape=", feed_dict[c].shape)
print("feed_dict[c]=", feed_dict[c])

feed_dict[c] is a tensor of [batchSz, hiddenSz]

The solution is as follows:

https://stackoverflow.com/questions/39112622/how-do-i-set-tensorflow-rnn-state-when-state-is-tuple-true