The Tozama lands comprised the territories owned by lords who opposed Tokugawa Ieyasu during the battle of Sekigahara. Because of this they were shut out of the Tokugawa government.
By contrast, the Fudai lands comprised the territories owned by lords who joined Tokugawa before the battle of Sekigahara.
The Shinpan lands comprised the territories owned by lords related to the shogun Tokugawa Ieyasu. They enjoyed the best lands in all of Japan along with high prestige.
The Bakufu lands comprised the territories owned by the Shogun and staffed with Fudai/Shinpan lords on a rotating basis.
By 1664 Japan was well into its Edo period. Years earlier in 1600, the Battle of Sekigahara had determined the future of not only Japanese politics, but also Japanese society. Because there were no major conflicts after that, the rank of every lord and soldier would usually never exceed that of his ancestor at the battle. Hence, under the Tokugawa Shogunate there were few promotions or pay raises. Not only that but there were demotions for improper conduct, dress and court appearance.
This also applied to a certain extent to all social classes. If your father was a merchant, you would become a merchant and your son after you.
Due to a lack of actual war, the samurai class became dormant and indolent. Receiving government stipends of rice, they were expected to remain in peak condition while maintaining at least 3 lackeys (one for armor, another for your horse and a third to move other baggage). Over time many Samurai began to adopt more stylized forms of combat, stressing dueling rather than battlefield techniques. Whereas the main weapon on the field had been the bow and arrow, followed by the naginata and the matchlock musket; the main weapon of policing was the sword. After this period the sword would become falsely known as the main weapon of the Samurai.
For more information on this subject, see [link]
map base courtesy of wikipedia
map base source [link]
map base created by [link]