This custom tileset does not seem to have "transition tiles", or they're not being displayed properly. These are tiles that influence how tiles are displayed on the borders of two different terrain tiles.
Go to Advanced -> Modify Tileset, select your custom tile and move it far to left (or right, I'm not sure). This should make it display below all other tiles, making the game use the transition tiles of the other textures instead.
If that doesn't work, attach the textures here, so we can see if there's something wrong with them.