Zigzag antiferromagnetic quantum ground state in monoclinic honeycomb lattice antimonates<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:msub><mml:mi>A</mml:mi><mml:mn>3</mml:mn></mml:msub><mml:mi mathvariant="normal">N</mml:mi><mml:msub><mml:mi mathvariant="normal">i</mml:mi><mml:mn>2</mml:mn></mml:msub><mml:mi>Sb</mml:mi><mml:msub><mml:mi mathvariant="normal">O</mml:mi><mml:mn>6</mml:mn></mml:msub><mml:mspace width="0.28em"/><mml:mo>(</mml:mo><mml:mi>A</mml:mi><mml:mo>=</mml:mo><mml:mi>Li</mml:mi><mml:mo>,</mml:mo><mml:mspace width="0.28em"/><mml:mi>Na</mml:mi><mml:mo>)</mml:mo></mml:mrow></mml:math>

We present a comprehensive experimental and theoretical study of the electronic and magnetic properties of two quasi-two-dimensional (2D) honeycomb lattice monoclinic compounds ${A}_{3}\mathrm{N}{\mathrm{i}}_{2}\mathrm{Sb}{\mathrm{O}}_{6}\phantom{\rule{0.16em}{0ex}}(A=\mathrm{Li},\phantom{\rule{0.16em}{0ex}}\mathrm{Na})$. Magnetic susceptibility and specific heat data are consistent with the onset of antiferromagnetic long-range order with N\'eel temperatures of \ensuremath{\sim}14 and 16 K for $\mathrm{L}{\mathrm{i}}_{3}\mathrm{N}{\mathrm{i}}_{2}\mathrm{Sb}{\mathrm{O}}_{6}$ and $\mathrm{N}{\mathrm{a}}_{3}\mathrm{N}{\mathrm{i}}_{2}\mathrm{Sb}{\mathrm{O}}_{6}$, respectively. The effective magnetic moments of $4.3\phantom{\rule{0.16em}{0ex}}{\ensuremath{\mu}}_{\mathrm{B}}/\mathrm{f}.\mathrm{u}.\phantom{\rule{0.16em}{0ex}}(\mathrm{L}{\mathrm{i}}_{3}\mathrm{N}{\mathrm{i}}_{2}\mathrm{Sb}{\mathrm{O}}_{6})$ and $4.4\phantom{\rule{0.16em}{0ex}}{\ensuremath{\mu}}_{\mathrm{B}}/\mathrm{f}.\mathrm{u}.\phantom{\rule{0.16em}{0ex}}(\mathrm{N}{\mathrm{a}}_{3}\mathrm{N}{\mathrm{i}}_{2}\mathrm{Sb}{\mathrm{O}}_{6})$, where f.u. is formula units, indicate that $\mathrm{N}{\mathrm{i}}^{2+}$ is in a high-spin configuration ($S=1$). The temperature dependence of the inverse magnetic susceptibility follows the Curie-Weiss law in the high-temperature region and shows positive values of the Weiss temperature, $\ensuremath{\sim}8\phantom{\rule{0.16em}{0ex}}\mathrm{K}$ ($\mathrm{L}{\mathrm{i}}_{3}\mathrm{N}{\mathrm{i}}_{2}\mathrm{Sb}{\mathrm{O}}_{6}$) and \ensuremath{\sim}12 K ($\mathrm{N}{\mathrm{a}}_{3}\mathrm{N}{\mathrm{i}}_{2}\mathrm{Sb}{\mathrm{O}}_{6}$), pointing to the presence of nonnegligible ferromagnetic interactions, although the system orders antiferromagnetically at low temperatures. In addition, the magnetization curves reveal a field-induced (spin-flop type) transition below ${T}_{\mathrm{N}}$ that can be related to the magnetocrystalline anisotropy in these systems. These observations are in agreement with density functional theory calculations, which show that both antiferromagnetic and ferromagnetic intralayer spin exchange couplings between $\mathrm{N}{\mathrm{i}}^{2+}$ ions are present in the honeycomb planes, supporting a zigzag antiferromagnetic ground state. Based on our experimental measurements and theoretical calculations, we propose magnetic phase diagrams for the two compounds.