OP sometimes smokes weed on his deck after work to unwind and deal with his anxiety. He lives next to the couple with two kids. OP's deck was close to their backyard, which became an issue.
OP said, "Sorry, but I won't stop, and I'm well within my rights to do it." OP also told the neighbor that he would not smoke if he saw their children playing in the backyard.