In the second half of the 14th century the measures initiated by the Florentine Republic in order to defend its territory provide some information on the material aspect of this small rural municipality. It turns out, in fact, that between 1366 and 1370, the male population of San Matteo in Granaiolo enjoyed tax exemptions for their work on the fortification or refortification of Granaiolo. We do not know whether it was a situation of constructing defenses that did not exist at all, or a renewal. In any case, in 1376, the umpteenth provision of Florence refers to the villa sive castro Granaiuoli (village or town of Granaiuoli): by the end of the 14th century, therefore, Granaiolo must have looked like a small walled village. The same document of 1376 provides interesting information on the material aspect of the castle: there is talk about the sale to the highest bidder of some houses and other buildings that were built, not in masonry, but with walls in pressed earth (de terra ad arcas). This refers evidently to the building technique known as "pisè", in use throughout the lower Valdarno until modern times, which involved the construction of pressed earth walls molded using formwork (arcas). The Florentine provision of 1376 specified that the buildings made with this technique would have required continual maintenance costs, making quite evident the reason for choosing to sell off those properties. This information on the material aspect of the buildings of Granaiolo suggests, as a hypothesis, the extreme ease with which parts of the castle, so difficult to preserve, may have undergone rapid deterioration when definitively abandoned or transformed. However, in the 15th century, the castle was still standing: the statements reported in the Florentine land registry of 1427 speak of different buildings positioned inside the castle, still surrounded by the "muro chastellano" (castle wall). Inside the castle one could even find rather articulate building complexes, such as the “podere chon chasa e uno casolare e parte di una torre” (farm with house, a farmhouse, and part of a tower). One also can intuit the existence of a certain organization of interior spaces. There must have existed a small internal road system: one road was called "Via Salimegro". One part of the castle was called "Borghetto". What was referred to as "Borgho Vecchio", on the other hand, must have been outside the castle walls. Here the church of Santa Maria was situated, attested to in written sources from the late 13th century. The small church of the village of Granaiolo remains, even if not structurally present, at least in the name of the chapel annexed to the Villa Pucci, which still bears the title of Santa Maria al Borgovecchio. The parish church of Granaiolo, however, was San Matteo, which was located downstream from the castle, near the River Elsa. The prior, charged with the management of the complex of structures of the rectory of San Matteo, lived in the castle as reported in a declaration of 1427: “casa nella quale habita il priore” (house in which the prior lives).