Security in fog computing is multi‐faceted, and one particular challenge is establishing a secure communication channel between fog nodes and end devices. This emphasizes the importance of designing efficient and secret key distribution scheme to facilitate fog nodes and end devices to establish secure communication channels. Existing secure key distribution schemes designed for hierarchical networks may be deployable in fog computing, but they incur high computational and communication overheads and thus consume significant memory. In this paper, we propose a novel hierarchical key pre‐distribution scheme based on “Residual Design” for fog networks. The proposed key distribution scheme is designed to minimize storage overhead and memory consumption while increasing network scalability. The scheme is also designed to be secure against node capture attacks. We also demonstrate that, in an equal‐size network, our scheme reduces node storage overhead significantly. Our research paves the way for building an efficient key management framework for secure communication within the hierarchical network of fog nodes and end devices.