Development of high-performance unipolar n-type organic semiconductors still remains as a great challenge. In this work, all-acceptor bithiophene imide-based ladder-type small molecules BTIn and semiladder-type homopolymers PBTIn (n = 1–5) were synthesized, and their structure–property correlations were studied in depth. It was found that Pd-catalyzed Stille coupling is superior to Ni-mediated Yamamoto coupling to produce polymers with higher molecular weight and improved polymer quality, thus leading to greatly increased electron mobility (μe). Due to their all-acceptor backbone, these polymers all exhibit unipolar n-type transport in organic thin-film transistors, accompanied by low off-currents (10–10–10–9 A), large on/off current ratios (106), and small threshold voltages (∼15–25 V). The highest μe, up to 3.71 cm2 V–1 s–1, is attained from PBTI1 with the shortest monomer unit. As the monomer size is extended, the μe drops by 2 orders to 0.014 cm2 V–1 s–1 for PBTI5. This monotonic decrease of μe was also observed in their homologous BTIn small molecules. This trend of mobility decrease is in good agreement with the evolvement of disordered phases within the film, as revealed by Raman spectroscopy and X-ray diffraction measurements. The extension of the ladder-type building blocks appears to have a large impact on the motion freedom of the building blocks and the polymer chains during film formation, thus negatively affecting film morphology and charge carrier mobility. The result indicates that synthesizing building blocks with more extended ladder-type backbone does not necessarily lead to improved mobilities. This study marks a significant advance in the performance of all-acceptor-type polymers as unipolar electron transporting materials and provides useful guidelines for further development of (semi)ladder-type molecular and polymeric semiconductors for applications in organic electronics.