Different types of siRNA delivery vehicles including nanoparticles have been synthesized and utilized for prostate cancer gene therapy. However, one of the most common limitations being faced is the toxicity of cationic polymers toward the cells. In the current study, magnetic nanoparticles were prepared and conjugated with cationic polymer, polyethylenimine. Then polyethylene glycol was conjugated with polyethylenimine to improve the biocompatibility of nanoparticles. The transmission electron microscopy size of nanoparticles was found to be 15.82 (±9.07) nm, while hydrodynamic size was about 79.20 (±0.68) nm. Zeta potential analysis of polyethylenimine and polyethylene glycol-coated nanoparticles was +31.4 (±0.62) and +5.65 (±0.76) mV, respectively. Fourier transform infrared spectroscopy and 1H nuclear magnetic resonance confirmed the presence of polyethylene glycol and polyethylenimine polymers in magnetic nanoparticles. Cell viability test in mouse fibroblast NIH 3T3 and prostate cancer PC3 cells showed an increased in biocompatibility of functionally modified polyethylene glycol–polyethylenimine–Fe3O4 nanoparticles. siRNA targeting a disintegrin and metalloproteinase 10 (ADAM10) was successfully loaded into the polyethylene glycol–polyethylenimine–Fe3O4 nanoparticles and delivered to PC3 cells. The results clearly demonstrated a significant decrease in cell viability, which increased within a certain siRNA concentration. The inhibitory concentration (IC50) value for ADAM10 siRNA was calculated to be 15.83 nM after 72 h. Confocal microscopy confirmed the delivery of siRNA-loaded nanoparticles intracellularly to the tumor cells cytosol. This magnetic system can be used as a powerful platform to inhibit cancer cells progression.